diff --git a/public/banners/why-i-still-bought-a-local-gpu.png b/public/banners/why-i-still-bought-a-local-gpu.png new file mode 100644 index 0000000..6ee2bec Binary files /dev/null and b/public/banners/why-i-still-bought-a-local-gpu.png differ diff --git a/public/og/why-i-still-bought-a-local-gpu.png b/public/og/why-i-still-bought-a-local-gpu.png new file mode 100644 index 0000000..58f937d Binary files /dev/null and b/public/og/why-i-still-bought-a-local-gpu.png differ diff --git a/scripts/banner-gen/generate.mjs b/scripts/banner-gen/generate.mjs index 6796dbb..e13a67e 100644 --- a/scripts/banner-gen/generate.mjs +++ b/scripts/banner-gen/generate.mjs @@ -605,6 +605,25 @@ BANNERS['patching-workbench-26-for-mariadb'] = { ], }; +BANNERS['why-i-still-bought-a-local-gpu'] = { + titlebar: 'root@unraid — local inference', + lines: [ + { t: 'prompt', text: '$' }, { t: 'cmd', text: 'llama-server -m qwen3.8-27b-Q6_K.gguf --n-gpu-layers 99' }, + { t: 'dim', text: '24GB VRAM · ~1300 tok/s prefill · 40 tok/s decode' }, + { t: 'prompt', text: '$' }, { t: 'cmd', text: 'agent run --task build-from-spec --iterations 50' }, + { t: 'err', text: 'frontier api: 50 calls · metered · quota can change' }, + { t: 'ok', text: 'local gpu: 50 calls · $0.00 · runs as long as I want' }, + { t: 'hl', text: 'use the frontier to DECIDE, local compute to BUILD' }, + ], + flow: [ + { n: '1', label: 'frontier plans' }, + { n: '2', label: 'spec locked' }, + { n: '3', label: 'local agents run' }, + { n: '4', label: '50× iterations' }, + { n: '5', label: 'fixed cost ✓' }, + ], +}; + // ---------- read frontmatter ---------- const postPath = join(ROOT, 'src', 'content', 'posts', `${slug}.md`); let category = 'devops'; diff --git a/scripts/og-gen/generate.mjs b/scripts/og-gen/generate.mjs index df32d6b..cd127b5 100644 --- a/scripts/og-gen/generate.mjs +++ b/scripts/og-gen/generate.mjs @@ -191,7 +191,13 @@ const DEFAULT_TERMINAL = `
$engineering · devops · self-hosting
 read the full post →
`; -// ---------- read frontmatter from the post ---------- +TERMINALS['why-i-still-bought-a-local-gpu'] = ` +
$agent run --task build-from-spec --iterations 50
+
 api: 50 calls · $12.40 · quota may change
+
 local: 50 calls · $0.00 · runs on my card
+`; + +// ---------- read frontmatter ---------- const postPath = join(ROOT, 'src', 'content', 'posts', `${slug}.md`); if (!existsSync(postPath)) { console.error(`Post not found: ${postPath}`); diff --git a/src/content/posts/why-i-still-bought-a-local-gpu.md b/src/content/posts/why-i-still-bought-a-local-gpu.md new file mode 100644 index 0000000..43aba81 --- /dev/null +++ b/src/content/posts/why-i-still-bought-a-local-gpu.md @@ -0,0 +1,225 @@ +--- +title: "Why I Still Bought a GPU to Run AI When Claude and GPT Are Obviously Stronger" +description: "Claude Opus and GPT are stronger than anything I can run locally. Here is the cost, control, and uptime argument for owning a GPU anyway — with real numbers." +pubDate: 2026-09-21 +category: ai +tags: [local-llm, gpu, llama-cpp, lm-studio, agents, self-hosting, cost] +ogImage: /og/why-i-still-bought-a-local-gpu.png +banner: /banners/why-i-still-bought-a-local-gpu.png +draft: false +--- + +I have a house full of self-hosted infrastructure — Docker stacks, a NAS, a +tunnel, a small army of cron jobs. And I still pay for frontier models. + +So when people find out I run local models too, the question is always the same: +*Claude and GPT are stronger. Local models are a pain to set up, slower, and +often just worse. So why did you spend money on a GPU?* + +If you are only comparing raw model capability, I agree with you completely. +When I hit a genuinely new problem, do complex reasoning, plan a product +architecture, or kick off a long agent run, I still reach for Claude or GPT +first. I did not deploy local models because I think they have beaten the +closed frontier. + +I did it because of a different problem that only showed up *after* AI got good +enough to depend on. Everything I built started sitting on a service I do not +control. + +--- + +## The problem is not capability — it is dependency + +How many tokens you get, what a plan costs, which regions can buy the commercial +service, when the model version changes under you — none of that is the user's +decision. Anthropic has regional restrictions of its own; some markets cannot +buy Claude's commercial service directly. OpenAI's plans, quotas and model +lineup keep shifting. These are commercial services, so of course they change. + +That was fine when ChatGPT was something I opened occasionally to ask a +question. If it was unavailable today, I would ask tomorrow and lose nothing. + +It stopped being fine once I started using AI to write code, run agents, +analyze documents, handle support, organize company data, and wire models into +workflows that run every day. When AI moves from *chat tool* to *infrastructure*, +the question changes. It is no longer "which model is smarter?" It becomes: + +> "If I depend on this model today, and its terms change tomorrow, does my work +> still run?" + +That is why I think the stronger the closed models get, the more valuable a +local model becomes. Precisely *because* they are so good, people hand them more +and more work. And the deeper the dependency, the more of your operation someone +else controls. + +There is a second-order version of this that rarely gets said plainly. Every big +lab talks about AI safety, model risk, and restricting certain content, regions +or use cases — while all of them keep racing to make models stronger, because +capability *is* the competitive advantage. Nobody stops, because if you stop, +someone else does not. + +So what actually gets controlled is rarely whether AI keeps developing. It is +who may use it, how, how much, and how far the vendor is willing to go. For an +individual, that reads like a terms-of-service page. For a business, it is +**supplier risk**. If your core process and every agent, tool and data pipeline +around it are pinned to one vendor, you are not just buying tokens — you are +handing over part of the steering wheel. + +--- + +## "But local models are worse" — yes, and that is the wrong comparison + +This is the most honest objection, and it is true. Put a model that fits on a +24GB or 32GB consumer card against Claude Opus and it loses on complex tasks. No +argument. + +But the comparison most people run is wrong. The question is not *can Qwen beat +Claude*. It is: + +> **Is the capability gap large enough to justify using the most expensive model +> for this task, forever?** + +Because the trade is not symmetric. A 20–30B model is meaningfully behind the +frontier — and costs one to two orders of magnitude less per token, or runs on +hardware you already own. Different ratio, different decision. + +### I treat capability as a threshold, not a ranking + +If a task is below some capability line, it cannot be done at all — use the +strong model. But once a model crosses that line, the things that decide how I +actually work stop being benchmark scores and become cost, speed, whether it can +run for hours, and whether I control the system. + +Concretely, here is how it splits for me: + +| Task | What I reach for | +|---|---| +| Vague problem, no clear direction yet | Frontier model — a wrong early call invalidates everything downstream | +| Product architecture, data model, acceptance criteria | Frontier model | +| Scaffolding a project to a fixed spec, editing files, running tests, fixing failures | Local model | +| Long-running agents, retries, batch jobs, bulk content | Local model | + +The important shift is the second block. Once direction is settled and the agent +is *executing a spec*, a single task may call a model 20–50 times. Do all fifty +calls need the most expensive model on the market? Usually not. The agent is not +deciding what the product should be — it is following specifications that are +already written. If a local model understands those specs and knows how to use +the tools, it has crossed the working threshold. + +--- + +## Agents turn a small price difference into a large one + +With chat, you ask once and get one answer. If Claude costs many times more than +a local model, you barely notice. + +Agents are not one call. A task can invoke the model twenty times, fifty times; +it fails and reruns. The per-token price gap gets multiplied by every call. + +And anyone who has actually built these pipelines knows the expensive part is +not the final run — it is the testing before it. You rewrite the prompt, swap +tooling, re-chunk the data, adjust the workflow, and run the same thing ten or +twenty times. That is normal. + +On metered API calls, I feel it: *I am burning money right now.* On my own card, +there is still electricity and hardware cost, but running an agent five times or +fifty times does not add a line to an invoice. That difference changes my +behavior — it decides whether I dare keep testing. Most stable AI workflows are +not written correctly the first time. They get there after failing enough times +that you find out where they break. + +The value of local compute is that you move that trial-and-error into a fixed +cost. That is a different question from "how much does a million tokens cost." + +--- + +## Open models crossed the useful line, and the line keeps moving + +Three years ago, running a model on a consumer card that could genuinely do work +for you was a research project. Today, 20–30B models — quantized — handle real +work on 24GB and 32GB cards. Go up in VRAM and language, image, and now portions +of video generation become self-hostable too. + +The important part is not "open source finally won." It is that work which used +to be closed-model-only keeps falling inside the range that local hardware can +do acceptably. If three years ago two of ten jobs could run locally, and today +four or five can, that ratio alone has changed the value of owning the hardware. + +And when I buy a GPU, I am not buying a specific model. Models change. If a +better coding model ships next month, I swap weights. If a better quantization +scheme lands in six months, the same card suddenly fits a model it could not run +before. What I actually bought is **compute I control**; the models on top of it +are disposable. + +That is the structural difference. A closed model is stronger — and also exactly +the version one company currently chooses to offer you, at terms it currently +chooses. Local hardware is weaker — but as long as the card holds up, I decide +which model runs today, which version I switch to, and how long it runs. + +--- + +## Fixed compute vs. renting compute + +If AI stays a question-and-answer tool, I do not think most people need a GPU at +all. But that is clearly not the direction. Agents run for long stretches. Image +and video models want bulk generation. Companies hand more fixed processes to +AI. We are not just *using models more often* — models are becoming a continuously +working compute unit. + +At that point, owning part of the capacity and renting every execution become +**two different cost structures**. Low usage: renting is obviously more +convenient. High, steady usage: owning lets you control and predict cost. + +It also buys something no API can sell you — control. When the machine is mine, +I can point it at internal data, connect my own tools, and run agents for hours +without checking whether a vendor's plan, quota or limits changed overnight. +Once AI is inside real company processes, this stops being about saving tokens. +Some data should not leave the environment it lives in. Some agent permissions +should not be handed wholesale to an external service. If the model runs on my +own hardware, data, tools, permissions and execution can be designed together. +*That* is what I mean by control — not merely having the weights on disk. + +--- + +## What I actually evaluate now + +I no longer judge a local model by "it is dumber than Claude, therefore +worthless." Four questions, in order: + +1. **Does this model clear my working threshold** for this specific job? +2. **How many times will this task execute?** +3. **Does it need heavy trial-and-error, or long agent runs?** +4. **Does the data or system need to stay in my own environment?** + +When those stack up, a local model — even one that is not the strongest — can be +the better tool for that slice of work. And as open models improve, that slice +keeps getting bigger. + +So what I am bullish on is not one model or one GPU. It is the trend: **more AI +work becomes doable on hardware you own.** A consumer card that used to run only +language models now runs agents, image generation, video. Better models and +better inference frameworks keep giving the same metal new jobs. + +Hardware depreciates, new cards get more expensive — that is real. But the model +is not locked in. As long as the machine has enough compute, I can keep swapping +in newer open models. + +--- + +## The short version + +I no longer compare "which model is smarter." I compare **how much capability +this job needs, how many times it will run, and how much control I am willing to +leave in someone else's hands.** + +Where I need frontier reasoning, I pay for the frontier. But when work becomes +high-volume, continuous and predictable — and a local model has crossed the +usable threshold — there is no reason to keep paying frontier prices for every +single step. + +In one line: + +> Use the frontier model to decide what to build. Use local compute to build it +> a thousand times. + +That is why I bought the GPU. diff --git a/src/content/posts/zh/why-i-still-bought-a-local-gpu.md b/src/content/posts/zh/why-i-still-bought-a-local-gpu.md new file mode 100644 index 0000000..a0fbd26 --- /dev/null +++ b/src/content/posts/zh/why-i-still-bought-a-local-gpu.md @@ -0,0 +1,195 @@ +--- +title: "Claude 和 GPT 明明更強，我為什麼還是買了 GPU 跑本地 AI" +description: "Claude Opus 和 GPT 確實比我能在本地跑的任何模型都強。但我還是買了 GPU——這篇講成本、控制權與可用性，附實際的判斷標準。" +pubDate: 2026-09-21 +category: ai +tags: [local-llm, gpu, llama-cpp, lm-studio, agents, self-hosting, cost] +ogImage: /og/why-i-still-bought-a-local-gpu.png +banner: /banners/why-i-still-bought-a-local-gpu.png +draft: false +--- + +我家裡塞滿了自架的東西——Docker stack、NAS、tunnel、一堆 cron job。但我 +照樣在付頂級模型的錢。 + +所以每次別人知道我也在跑本地模型，問的都是同一句話：*Claude 和 GPT 就是 +比較強，本地模型又麻煩、又慢、效果還比較差，那你到底為什麼要花錢買 GPU？* + +如果只比模型能力，我完全同意。今天要處理一個全新的問題、做複雜推理、規劃 +產品架構、或者跑一個很長的 Agent 任務，我大部分時間還是先用 Claude、GPT。 +我部署本地模型，從來不是因為我覺得它已經打贏閉源模型。 + +而是因為 AI 越來越好用之後，出現了另一個問題：我做的每一件事，都開始建立 +在一個我自己完全不能控制的服務上。 + +--- + +## 真正的問題不是能力，是依賴 + +今天模型可以用多少、方案怎麼收費、什麼地區能不能用、版本什麼時候換掉， +最後都不是使用者決定的。Anthropic 本來就有自己的地區限制，有些市場不能 +直接買 Claude 的商業服務；OpenAI 的方案、額度跟模型也一直在調。這些是商業 +服務，變動本來就可以預期。 + +以前 ChatGPT 只是偶爾拿來問問題，今天不能用，我明天再問沒什麼差別。 + +但當我開始拿 AI 寫程式、跑 Agent、分析文件、處理客服、整理公司資料，甚至 +把它接進每天都在跑的工作流程之後，問題就變了。當 AI 從聊天工具變成工作裡 +的基礎設施，你要考慮的就不只是「哪一個模型比較強」，而是： + +> 如果我今天很依賴這個模型，但有一天它的使用條件改掉，我原本的工作還能不能繼續？ + +這就是為什麼我覺得，閉源模型越強，本地模型的重要性會越來越高。因為閉源 +模型太好用，大家就會把越來越多事情交給它；依賴越深的同時，也會越被這些 +閉源模型控制。 + +還有一件事很值得講清楚。每一家大模型公司都在談 AI 安全、模型風險，甚至 +限制某些內容、地區或使用方式，但同時每一家又都還在拼命把模型做得更強—— +因為模型能力本身就是競爭力，你不做，別人還是會做。 + +所以最後真正被控制的，往往不是 AI 要不要繼續發展，而是誰能用、怎麼用、 +用多少，以及模型公司願意提供到什麼程度。對一般人來說，這可能只是服務 +條款；對企業來說，這就是**供應商風險**。假設一間公司的核心流程，加上後面 +所有 Agent、工具、資料處理，全部綁在同一個供應商身上，那它其實不只是在買 +Token——它是在把一部分工作的主控權交出去。 + +--- + +## 「可是本地模型就是比較差啊」——對，但這是錯誤的比較方式 + +這是最誠實的反對意見，而且它是事實。拿一個能放進 24GB、32GB 消費級顯卡的 +模型，直接跟 Claude Opus 全面比較，很多複雜任務就是會輸。這沒什麼好辯的。 + +但大家很容易用了錯誤的比較方式。真正該問的不是「Qwen 能不能打贏 Claude」， +而是： + +> **這個能力差距，到底有沒有大到值得你永遠用最貴的模型來做這件事？** + +因為這個交換並不對稱。二三十 B 的模型確實落後頂級模型一截，但成本低一到 +兩個數量級，或者直接跑在你已經買的硬體上。比例不同，決策就不同。 + +### 我把模型能力看成分界線，不是排行榜 + +如果一個任務低於某個能力門檻，那就是做不了，當然用強模型。但一旦跨過 +那條線，後面真正影響我使用方式的，就不再是 benchmark 分數，而是成本、 +速度、能不能長時間運行，以及我能不能控制這套系統。 + +以我的實際分法： + +| 任務 | 我用什麼 | +|---|---| +| 問題還很模糊、連方向都不知道 | 頂級閉源模型——前面判斷錯了，後面做再多都沒意義 | +| 產品架構、資料格式、驗收條件 | 頂級閉源模型 | +| 照著已定規格建專案骨架、改檔案、跑測試、看到錯誤再修 | 本地模型 | +| 長時間 Agent、重試、批次工作、大量內容產出 | 本地模型 | + +關鍵是第二塊。一旦方向確定、Agent 只是在執行規格，一個任務可能會呼叫模型 +二十次、五十次。這五十次真的每一次都需要市面上最貴的模型嗎？通常不需要。 +Agent 已經不是在幫我決定產品要怎麼做，它是在按照前面已經寫好的規格執行。 +只要本地模型可以正確理解這些規格、知道工具怎麼用，它就已經跨過可以工作的 +門檻。 + +--- + +## Agent 會把小小的價差一直放大 + +用聊天，是問一次、回答一次。Claude 一次比本地模型貴很多，你也不會有特別 +感受。 + +但 Agent 不是一次。一個任務可能呼叫模型二十次、五十次，失敗之後還會重新 +跑。模型單次價格的差距，放進這種連續任務裡就會被一直放大。 + +而真正做過這些流程的人都知道，最花資源的往往還不是最後正式執行，而是前面 +測試。你會改 Prompt、換工具、重新切資料、調整工作流程，同一個東西跑十次、 +二十次很正常。 + +每次都是 API，我自己就會很明顯感覺到：又在燒錢。用自己那張卡，當然還是有 +電費跟硬體成本，但我今天讓一個 Agent 跑五次還是五十次，不會每跑一次就多 +一筆帳單。這對我來說很重要，因為它會直接影響我敢不敢一直連續測試。很多 +真正穩定的 AI workflow，不是第一次寫完就成功，而是讓它失敗很多次之後， +才慢慢找到什麼地方容易出問題。 + +本地算力其中一個很大的價值，就是你可以把這些大量試錯搬回自己的固定成本 +裡。這跟單純比較「一百萬 Token 多少錢」其實是兩件事。 + +--- + +## 開放模型已經跨過可用的線，而這條線還在移動 + +三年前，要在消費級顯卡上跑一個真的可以幫你工作的模型，限制很多。現在 +二三十 B 的模型經過量化之後，24GB、32GB 這種顯卡已經可以處理不少工作； +再往更大的記憶體走，語言、圖片，甚至部分影片模型都開始可以自己部署。 + +真正重要的不是「開源終於打贏閉源了」，而是原本只有閉源模型做得到的工作， +正在一個一個掉進本地可以處理、而且可以被接受的範圍。假設三年前十個工作 +只有兩個能放本地，現在已經有四五個可以做——這個比例本身就已經改變了 +本地部署的價值。 + +而且我買 GPU 的時候，不是在買 Qwen 這一顆模型。模型可以一直換，也會一直 +出現新的：如果之後出現一個更適合 Coding 的開放模型，我直接換權重；再過 +半年有更好的量化方式，同一張卡可能又可以裝進原本跑不動的模型。我真正買 +下來的，是**一塊自己可以控制的運算資源**，後面的模型可以一直替換。 + +這就是本地跟閉源服務最不一樣的地方。閉源模型能力通常比較強，但你用的是 +某家公司現在願意提供給你的版本；本地硬體能力可能沒那麼高，但只要硬體 +撐得住，我可以自己決定今天跑什麼模型、換什麼版本、要跑多久。 + +--- + +## 固定算力 vs. 每次租算力 + +如果 AI 永遠只是拿來問問題，我完全不覺得大家都需要買 GPU。但現在發展方向 +很明顯不是這樣：Agent 會長時間執行工作，圖片跟影片模型會需要大量生成， +公司也會把更多固定流程交給 AI。我們不是單純使用模型的次數變多，而是模型 +開始變成一個持續工作的運算單位。 + +這時候，「固定擁有一部分算力」跟「每一次執行都向外面租」，就會開始變成 +兩種不同的成本結構。使用量少，租一定比較方便；使用量夠大、夠固定之後， +自己有算力就能更好控制與估算成本。 + +它還多了一個閉源服務給不了的東西：主控權。當機器是我的，我要跑內部資料、 +接自己的工具、長時間執行 Agent，都不需要每次先確認供應商的方案、額度或者 +限制有沒有改。尤其公司真的開始把 AI 接進內部流程之後，這件事就不只是省 +Token——有些資料本來就不適合離開自己的環境，有些 Agent 權限也不可能直接 +全部交給外部服務。當模型跑在自己的環境裡，資料、工具、權限跟執行方式可以 +一起設計。這才是「控制權」真正指的東西，不只是把權重下載下來而已。 + +--- + +## 我現在實際怎麼判斷 + +我不會再用「它比 Claude 笨，所以沒有價值」這種方式判斷一個本地模型。我會 +依序看四件事： + +1. **這顆模型能不能跨過我這個工作的門檻？** +2. **這個任務會執行多少次？** +3. **是不是需要大量試錯，或長時間的 Agent？** +4. **這些資料跟系統，到底要不要留在自己的環境？** + +這幾個條件一疊起來，本地模型即使不是最強，還是可能比頂級閉源模型更適合 +那一段工作。而且隨著開放模型越來越強，這個範圍只會一直變大。 + +所以我真正看好的，不是某一顆模型，也不是某一張 GPU，而是這個趨勢： +**可以在自己硬體上完成的 AI 工作正在變多。** 以前買一張消費級顯示卡， +可能只能拿來跑一些語言模型；現在開始可以接 Agent、圖片、影片，後面模型 +跟推理框架繼續優化，同樣的硬體又可能得到新的用途。 + +硬體當然會折舊，新卡也可能越來越貴，但至少模型不是綁死的。只要這台機器 +還有足夠的運算能力，我就可以一直換新的開放模型。 + +--- + +## 一句話版本 + +我現在已經不會只看哪個模型比較聰明。我比較的是：**這件工作到底需要多強的 +模型、它要被執行多少次，以及我願意把多少控制權留在外部服務手上。** + +需要頂級推理的地方，我照樣花錢用最強的模型。但當工作開始變成大量、連續、 +固定，而且本地模型的能力已經跨過可使用門檻時，我就沒有理由每一個步驟都 +繼續用最貴的閉源模型。 + +濃縮成一句： + +> 用頂級模型決定要做什麼，用本地算力把它做上一千次。 + +這就是我買 GPU 的原因。