post(ai): the five-step AI roadmap — what I actually run in production (EN+ZH) + og/banner

src/content/posts/five-step-ai-roadmap-what-i-actually-run.md

@@ -0,0 +1,242 @@
+---
+title: "The Five-Step AI Roadmap Everyone Shares — and What I Actually Run in Production"
+description: "The five-step AI roadmap circulates on every feed. Here is the honest version: which steps I run in production, which one I skipped, and what each is for."
+pubDate: 2026-10-13
+category: ai
+tags: ["llm", "rag", "agents", "self-hosting", "whisper", "mem0"]
+ogImage: /og/five-step-ai-roadmap-what-i-actually-run.png
+banner: /banners/five-step-ai-roadmap-what-i-actually-run.png
+draft: false
+---
+
+There is a graphic that makes the rounds every few weeks, and by now it has
+reached my feed in translated form, reposted by accounts I don't follow. It
+lays out the path to working with large models in five steps: learn the
+fundamentals — Transformer, neural networks, attention — and stop as soon as
+you can read the diagrams; learn prompt engineering, which takes days; learn
+RAG so the model answers from your own data instead of making things up;
+learn agents, the biggest opportunity in the field; and then learn
+fine-tuning, where the advice is to skip pretraining entirely (that belongs
+to the big labs) and get comfortable with multimodal fine-tuning instead.
+There is a salary band attached to each step, and the band at the end is the
+reason the thing spreads.
+
+I am not going to argue with the list. It is not wrong — it is a syllabus.
+And a syllabus is the kind of advice that behaves perfectly in a classroom
+and then dies on contact with a system, because it tells you what to learn
+and in what order, and never tells you which of those items you will still
+be doing years later when somebody is paying you for the result. I have been
+running AI workloads of my own for a while now — not research, not model
+training, but real services other people depend on. So here is the honest
+version of that graphic, checked against what is actually running on my
+hardware tonight.
+
+## Why this matters
+
+The reason to distrust the roadmap is not that the items are wrong. It is
+that a roadmap tells you what to *study*, while employment asks what you can
+*operate*. Those are different skill sets, and the gap between them is where
+most people who "learned AI in five steps" stall: they can describe a RAG
+pipeline, but they have never had one break on them at 11pm in a way that
+made them understand retrieval, cost, ingestion and evaluation in a single
+evening.
+
+The rest of this post maps each step to something I actually run, states
+plainly which steps I have never done, and gives you the order I would pick
+if I had to start again. Every claim below is measured on one desktop with
+an RTX 3060 (12 GB) and one NAS.
+
+## Step 1: Fundamentals — enough to be dangerous, then stop
+
+The graphic's framing here is closer to right than most engineers admit:
+learn the Transformer until the diagrams make sense, then move on. My
+version stopped earlier than that. I can hold a real conversation about
+attention, tokens, context windows, quantization schemes and the KV cache,
+because those are the levers I actually turn when a model has to fit into 12
+GB of VRAM and answer fast enough to be useful in a chat window. I could not
+derive backpropagation on a whiteboard, and I would not trust a number I
+produced from memory.
+
+What that costs me: nothing measurable so far, in systems that ship. What it
+would cost me: the day I train something. If your goal is "own the AI stack
+for a small business", step 1 is a reading list, not a degree — and the
+honest reason to buy local hardware in the first place is not that a 12 GB
+card beats the frontier models, because it does not; it is what running a
+model yourself teaches you that an API never will. I wrote about that trade
+honestly in
+[why I still bought a local GPU to run AI](https://blog.hoelee.com/posts/why-i-still-bought-a-local-gpu/).
+
+## Step 2: Prompt engineering — the step with the lowest ceiling
+
+"Days to learn" is accurate, and that is the problem with it. Prompting is
+the cheapest step to enter and the first one to get automated, so it is
+where enormous numbers of people stop and start describing themselves as AI
+engineers. The honest skill is not writing a better instruction; it is
+deciding when an instruction should not exist at all.
+
+If a task has one correct answer, a deterministic input and a known output
+shape, then a script is faster, cheaper and cannot hallucinate. I have
+replaced agent prompts with forty lines of Python more than once, and
+[wrote up the case for that](https://blog.hoelee.com/posts/stop-writing-agent-prompts-for-deterministic-work/).
+What survives on the prompt side is interface design: what the model is
+allowed to see, what shape its answer must take, how much of your own
+context you are willing to hand it, and — the part the graphic never
+mentions — how you would know the output got worse. That last item is
+evaluation, and in my experience it is the difference between a demo and a
+service.
+
+## Step 3: RAG — the hard half is everything except the retrieval
+
+The one-line version everyone quotes is "give the model your data so it
+stops lying". I run exactly that: a self-hosted memory layer built from
+mem0, LiteLLM and PostgreSQL 17 with pgvector, all on my own NAS, with
+memories scoped to a `user_id` so one customer's history cannot surface in
+another's answer —
+[the whole stack is written up here](https://blog.hoelee.com/posts/self-hosting-mem0-memory-stack/),
+including the container layout and the fact that the API is two small HTTP
+calls.
+
+Three things the tutorial version of RAG skips, and each of them cost me
+real time:
+
+1. **Ingestion is the boring half that breaks.** Retrieval quality gets all
+   the attention; getting the data in reliably is where the work is. The
+   most recent input I pushed through my own speech-to-text stack was a
+   57-second video sitting on my desktop — it came back as text in about
+   eight seconds on the same 12 GB card that already serves live dictation
+   for my phone and laptop. That speed is what makes the stack usable
+   instead of a demo, and the pipeline is
+   [documented end to end here](https://blog.hoelee.com/posts/self-hosted-speech-to-text-api/).
+2. **Byte-level mistakes look like model mistakes.** My first attempt to
+   feed that same service through a workflow tool reported a clean 502
+   "conversion failed" from the audio layer. The real cause was a multipart
+   upload arriving without a `Content-Type` header, which made the broker
+   decode raw audio as UTF-8 text and destroy it before any of my code ran.
+   Nothing about that bug is AI. It is exactly the kind of failure the
+   five-step graphic cannot prepare you for.
+3. **Retrieval is not a model problem.** When an answer is wrong, the useful
+   question is whether the right chunk was ever retrieved, whether it
+   survived chunking, and whether the model was given room to use it.
+   Swapping to a bigger model has never once fixed a retrieval bug for me.
+
+Underneath that entire layer there are two calls, which is worth seeing
+once:
+
+```text
+POST http://127.0.0.1:20015/memories    # store a fact or a conversation, tagged by user_id
+GET  http://127.0.0.1:20015/search      # semantic search, scoped to the same user_id
+```
+
+Nothing exotic — which is the point. A memory layer you can describe in two
+lines is a memory layer you can debug, and the whole thing stays local.
+
+## Step 4: Agents — the opportunity is real, and the agent is the smallest part
+
+I agree with the graphic's ranking: this is where the work is. I disagree
+with what it implies by the word "agent", because an agent is a model plus a
+loop plus tools — and the part that decides whether it survives contact with
+reality is the checking around it, not the model at the centre. I've written
+about this as
+[loop engineering](https://blog.hoelee.com/posts/loop-engineering-without-a-coding-agent/),
+and the finding I keep coming back to is uncomfortable: most of my scheduled
+automation has no AI in it at all, and those are precisely the jobs that
+have never needed babysitting. The model is the part that adds judgement.
+The rest of the system is what makes the judgement safe to trust.
+
+Three things that are actually running here, so the claim is concrete rather
+than aspirational:
+
+- A Telegram support bot for my own digital-goods shop that keeps long-term
+  memory per customer, so a returning buyer is not a stranger on their
+  second message. The memory layer it writes into is the same self-hosted
+  mem0 stack from step 3.
+- A scheduled job that manages discount settings in my Shopee seller
+  account, which is the least glamorous useful agent I own: it reads the
+  current state, decides, acts, and reports — and the entire value comes
+  from the fact that it checks the result afterwards instead of assuming its
+  click landed.
+- A research run that compared twenty VPS providers in parallel by fanning
+  the work out across subagents, which is a pattern worth learning even if
+  you never build a "chatbot":
+  [the write-up is here](https://blog.hoelee.com/posts/how-i-vetted-20-vps-providers-with-parallel-subagents/).
+
+## Step 5: Fine-tuning — the step I have never done, on purpose
+
+The graphic says skip pretraining, which is correct, and then suggests
+learning multimodal fine-tuning. For anyone whose job looks like mine, I'd
+push back on the second half.
+
+I have never fine-tuned a model. Not because I couldn't, but because in the
+time I have been running these services, none of my failures have been
+model-behaviour failures. They were retrieval failures, licensing surprises,
+byte-level failures and operational failures — a job that reported success
+while the file it claimed to replace was untouched, a mount whose metadata
+lied about which file it had swapped, an API whose real model name differed
+from the one it advertised. A LoRA does not fix a schema drift.
+
+Where I would fine-tune: one narrow, well-defined task; thousands of
+labelled examples rather than dozens; an inference bill or a latency ceiling
+that a smaller tuned model would genuinely fix; and the willingness to own
+an evaluation set, because a fine-tune you cannot measure is a regression
+you cannot see. And I would rent the GPU for it long before I bought a
+second card — the training run is a project, not a workstation purchase.
+
+## What I'd do differently
+
+If I were starting from zero with that graphic in front of me, I would not
+walk it in numerical order. I would walk it in the order my problems
+arrived, and accept that the first real lesson will not come from a course:
+
+- Treat step 1 as a reading list you finish in public documentation, not a
+  curriculum. Enough theory to know why quantization and context limits
+  hurt; no more until something forces you deeper.
+- Spend a week on step 2, then immediately automate the parts of it that
+  have one correct answer, and move the rest of your attention to evaluation
+  — which the graphic omits entirely.
+- Live in steps 3 and 4, because that is where nearly all employed AI work
+  currently sits: wiring models to real data, and making the loop around
+  them trustworthy.
+- Keep step 5 in your back pocket until a specific problem names it.
+  Fine-tuning is a technique with a precondition, not a rung on a ladder.
+
+And notice the roadmap's real omission, which is not a step but half the
+job: identity, storage, deployment, cost, monitoring, and who gets woken up
+when it breaks. That is the half that decides whether a team can use you,
+and it is the half this blog mostly writes about.
+
+## The result
+
+Three of those five steps run in production on one desktop GPU and one NAS,
+and the honest scorecard is this:
+
+- A 57-second video becomes text in about eight seconds, on the same 12 GB
+  card that serves live dictation — and the speech server drops from **4,906
+  MiB to 1,321 MiB** of VRAM when I toggle it idle, because a GPU that is
+  always resident is a GPU I cannot use for anything else.
+- The memory layer answers in two small HTTP calls, scoped per user, backed
+  by PostgreSQL with pgvector on the NAS — mine, not a vendor's.
+- Most of the scheduled automation that keeps this stack alive contains no
+  model at all, and it is the most reliable part of the system.
+- Step 5 stays empty, and I would rather say that plainly than pad a
+  roadmap. The gap is real, and it is a choice.
+
+If the five-step graphic is the syllabus, this is the transcript. The
+syllabus tells you what to learn. The transcript is what a recruiter or a
+client can actually check.
+
+---
+
+## Want an AI stack that runs in your business?
+
+I run Hoelee Enterprise, a website design and development business for
+Malaysian SMEs — and the AI work above runs on the same infrastructure I
+build for clients. If you want a model answering from *your* data, on *your*
+hardware, without a per-seat subscription attached, I'd be glad to talk:
+
+- 📱 **WhatsApp:** [+60 12-797 2969](https://wa.me/60127972969)
+- 📧 **Email:**
+  [[email protected]](mailto:[email protected]?subject=AI%20stack%20for%20my%20business)
+- 🌐 **Website:** [hoelee.com](https://hoelee.com)
+
+And if you are hiring for a platform, DevOps or AI-infrastructure role, the
+same channels reach me.

src/content/posts/zh/five-step-ai-roadmap-what-i-actually-run.md

@@ -0,0 +1,103 @@
+---
+title: "全网都在转的 AI 五步学习路线 —— 我真正在生产里跑的是哪几步"
+description: "AI 五步学习路线（基础、提示词、RAG、Agent、微调）到处都在转。这是诚实版：哪几步真的在跑，哪一步我跳过了，每一步到底是为了什么。"
+pubDate: 2026-10-13
+category: ai
+tags: ["llm", "rag", "agents", "self-hosting", "whisper", "mem0"]
+ogImage: /og/five-step-ai-roadmap-what-i-actually-run.png
+banner: /banners/five-step-ai-roadmap-what-i-actually-run.png
+draft: false
+---
+
+每隔几周就会有一张图在各个信息流里转一圈，到我这里的时候已经是翻译过的版本，发它的账号我也没关注。图里把「做大模型」这条路分成五步：先学基础 —— Transformer、神经网络、注意力机制，学到能看懂图就停；再学提示词工程，几天就能上手；然后学 RAG，让模型基于你自己的数据回答，而不是张口乱编；接着学 Agent，说是这个领域最大的风口；最后是微调，这里给的建议是预训练完全不用碰（那是大厂的事），把多模态微调练熟就行。每一步后面都挂着一个薪资数字，而正是最后那个数字让这张图到处流传。
+
+我不是来跟这张图抬杠的。它没说错 —— 它是一份教学大纲。而大纲这种东西，在教室里完美无缺，一碰到真实系统就失效：它只告诉你学什么、按什么顺序学，从来不告诉你，几年之后你还在做的是其中哪几件事、有谁愿意为结果付钱。我自己跑这一类负载已经有一段时间了 —— 不是做研究，也不是训练模型，而是别人真的在用的服务。所以下面是一个诚实版本：每一步对着我今晚机器上实际在跑的东西逐条核对。
+
+## 为什么这件事值得说
+
+不信任那张图的理由，不在于条目本身错了，而在于路线图回答的是「该学什么」，而工作问的是「你能不能让东西跑起来」。这是两套不同的能力，而多数「五步学完 AI」的人就卡在这道缝里：他们能描述一条 RAG 流水线，但从来没有在晚上十一点被它搞崩过一次，也就没有在一晚上里真正搞懂检索、成本、数据接入和评估这四件事。
+
+接下来我会把五个步骤逐一对应到我实际在跑的东西上，明说哪几步我从来没做过，最后给出如果重新来一次我会选的顺序。下面所有说法，都跑在一台装了 RTX 3060（12 GB）的台式机和一台 NAS 上。
+
+## 第一步：基础 —— 够用就停
+
+图里在这点上比多数工程师愿意承认的更对：把 Transformer 学到能看懂图，然后往前走。我的进度比这还早一点就停了。我可以认真聊注意力、token、上下文窗口、量化方式和 KV cache，因为这些正是我在一台 12 GB 显卡上真正会去拧的旋钮 —— 模型要装得下，回答还得快到在聊天窗口里能用。但你让我在白板上推反向传播，我做不到，我也不会相信自己凭记忆报出来的数字。
+
+代价是什么：到目前为止，在要交付的系统里，我没为这件事付过代价。什么时候会付：等我需要训练一个模型的那天。如果你的目标是「给一家小企业把 AI 这套东西撑起来」，第一步是一份书单，不是一个学位 —— 而买本地显卡真正诚实的理由，不是 12 GB 的卡能打赢前沿模型（它打不赢），而是「自己跑一遍」能教你 API 永远教不了的东西。这个取舍我写在[为什么我还是买了一张本地显卡](https://blog.hoelee.com/posts/zh/why-i-still-bought-a-local-gpu/)里。
+
+## 第二步：提示词 —— 天花板最低的一步
+
+「几天就能上手」是准确的，而这正是它的问题。提示词是门槛最低的一步，也是最早被自动化掉的一步，所以大量的人停在这里，然后开始管自己叫 AI 工程师。真正的能力不是把指令写得更漂亮，而是判断什么任务根本不该存在指令。
+
+如果一个任务只有一个正确答案、输入是确定的、输出形状也是已知的，那脚本更快、更便宜，而且不会编造。我不止一次把 agent 的提示词换成四十行 Python，并把理由写进了[这里](https://blog.hoelee.com/posts/zh/stop-writing-agent-prompts-for-deterministic-work/)。提示词这一侧真正留下来的工作是接口设计：模型被允许看到什么、它的回答必须长成什么形状、你愿意把自己的多少上下文交出去，以及那张图从来不提的一件事 —— 你怎么知道它变差了。最后这件事就是评估，在我看来，它就是「演示」和「服务」之间的分界线。
+
+## 第三步：RAG —— 难的那一半，跟检索没关系
+
+大家都在引用的一句话是「把数据喂给模型，它就不乱说了」。我确实在跑这套东西：一个自建的记忆层，由 mem0、LiteLLM 和带 pgvector 的 PostgreSQL 17 组成，全部跑在自己的 NAS 上，记忆按 `user_id` 隔离，所以一个客户的记录不会跑到另一个客户的回答里 —— [整套栈都写在这里](https://blog.hoelee.com/posts/zh/self-hosting-mem0-memory-stack/)，包括容器怎么摆、以及它对外其实只是两个小小的 HTTP 调用。
+
+教学版的 RAG 会跳过三件事，而每一件都花掉过我真实的时间：
+
+1. **数据接入是无聊的那一半，也是最容易坏的那一半。** 检索质量拿走全部注意力，但真正的工作是把数据可靠地送进去。我最近一次推进自己语音转文字服务的东西，是桌面上的一个 57 秒视频 —— 它变成文字花了大约八秒，用的还是那张同时给手机和笔记本提供实时听写的 12 GB 显卡。正是这个速度让它从演示变成能用的服务，整条链路[写在](https://blog.hoelee.com/posts/zh/self-hosted-speech-to-text-api/)这里。
+2. **字节级的错误，看起来像模型的错误。** 我第一次把同一个服务接到工作流工具上，拿到的是一个很干净的 502「conversion failed」。真正的原因是：multipart 上传里的文件部分少了 `Content-Type`，于是中间件把原始音频当成 UTF-8 文本解码，在我的任何一行代码跑起来之前就把它毁了。这个 bug 里没有一丝 AI 成分 —— 而它恰好就是那张五步图没办法帮你准备的失败。
+3. **检索不是一个模型问题。** 回答错了，该问的是：正确的片段到底有没有被检索出来、它在切片时有没有被切坏、模型有没有拿到足够的空间去用它。对我来说，换更大的模型从来没有修好过一个检索 bug。
+
+这一层下面其实只有两个调用，值得看一眼：
+
+```text
+POST http://127.0.0.1:20015/memories    # store a fact or a conversation, tagged by user_id
+GET  http://127.0.0.1:20015/search      # semantic search, scoped to the same user_id
+```
+
+没什么花活 —— 这正是重点。一个能用两行描述完的记忆层，才是你能调试的记忆层，而且它整条都留在本地。
+
+## 第四步：Agent —— 机会是真的，但 Agent 是最小的一块
+
+图里把它排在最值钱的位置，这点我同意：活儿确实在这里。我不同意的是「agent」这个词暗示的东西。一个 agent 是模型加循环加工具，而决定它能不能活得下去的，是围着它的那套检查，不是中间那个模型。我把这套做法写成过 [loop engineering](https://blog.hoelee.com/posts/zh/loop-engineering-without-a-coding-agent/)，而里面最让我不舒服的发现是：我大部分定时任务里根本没有 AI，偏偏就是这些任务从来没有需要我半夜起来救。模型负责的是判断力，系统的其余部分负责让这份判断力可以被信任。
+
+我这边真正在跑的三件事，好让这句话不只是一句口号：
+
+- 一个给我自己数字商品店用的 Telegram 客服 bot，按客户保存长期记忆，所以回头客在发第二条消息的时候不是陌生人。它写入的记忆层，就是第三步那套自建的 mem0。
+- 一个定时任务，负责改我 Shopee 卖家后台的优惠券设置 —— 它是我最不性感但真有用的一类 agent：读当前状态、做决定、动手，然后汇报。它全部的价值都来自一个事实：它会去检查结果，而不是假设自己那一下点中了。
+- 一次把二十家 VPS 服务商并行比一轮的调研，做法是把活拆给多个子 agent 分头跑。这个模式值得学，哪怕你根本不打算做「聊天机器人」：[写法在这里](https://blog.hoelee.com/posts/zh/how-i-vetted-20-vps-providers-with-parallel-subagents/)。
+
+## 第五步：微调 —— 我故意没做的一步
+
+图里说预训练不用碰，这句是对的；但它接着说要去学多模态微调。对工作内容跟我类似的人来说，后半句我有异议。
+
+我从来没有微调过任何模型。不是做不到，而是我跑这些服务这段时间里，没有任何一次失败是「模型行为」的失败。我的失败是检索失败、许可授权意外、字节级失败和运维失败 —— 一个报了成功、但它声称替换掉的文件根本没动的任务；一个对「到底换掉了哪个文件」撒谎的挂载点；一个真实模型名跟它宣传的不一样的 API。LoRA 修不了 schema 漂移。
+
+我什么时候会去微调：任务足够窄且定义清楚；有成千上万条标注样本而不是几十条；推理账单或者延迟上限确实能被一个更小的调优模型解决；以及愿意自己维护一套评估集 —— 因为一个你无法测量的微调，就是一次你看不见的退化。而且真要做，我会先去租显卡，而不是买第二张 —— 训练是一次项目，不是一次装机。
+
+## 如果重来一次，我会怎么做
+
+如果让我拿着那张图从零开始，我不会按它的编号顺序走。我会按自己遇到的问题出现的顺序走，并且接受第一个真正的教训不会来自任何一门课：
+
+- 把第一步当成一份能在公开文档里读完的书单，而不是一门课程。懂到足以解释「为什么量化会和上下文上限互相打架」就够了，除非有事情逼你往深处走。
+- 第二步花一周，然后立刻把其中只有一个正确答案的部分自动化掉，剩下的注意力全部转到评估上 —— 那张图完全没提这件事。
+- 把时间花在第三、第四步上，因为现在市面上几乎所有有人付钱的 AI 工作都在这里：把模型接到真实数据上，以及让围着它的循环变得可信。
+- 第五步先放口袋里，等到某个具体问题点名它为止。微调是一门有前置条件的技术，不是楼梯上的一级。
+
+还要注意那张路线图真正的遗漏：那不是一个步骤，而是这个工作的一半 —— 身份、存储、部署、成本、监控，以及出事的时候谁被叫醒。决定了团队能不能用你的，正是这一半，而这个博客写的也主要是这一半。
+
+## 结果
+
+那五步里有三步跑在一张桌面显卡和一台 NAS 上，诚实的成绩单是这样的：
+
+- 一个 57 秒的视频变成文字大约八秒，用的还是那张同时提供实时听写的 12 GB 显卡 —— 而我把语音服务切成空闲时，显存从 **4,906 MiB 掉到 1,321 MiB**，因为一张常年占着的显卡，就是一张我别的什么都干不了的显卡。
+- 记忆层两个小 HTTP 调用就给出答案，按用户隔离，后端是 NAS 上带 pgvector 的 PostgreSQL —— 是我的，不是某个厂商的。
+- 维持这套栈活着的定时任务里，大部分根本不含模型，而它们恰好是整个系统里最可靠的部分。
+- 第五步仍然是空的。我宁愿直说这件事，也不愿意把一份路线图注水填满。这个缺口是真的，而且是我选的。
+
+如果那张五步图是教学大纲，这篇就是成绩单。大纲告诉你该学什么，成绩单是招聘方或者客户能真正核对的东西。
+
+---
+
+## 想让 AI 真正在你公司里跑起来？
+
+我经营 Hoelee Enterprise，一家面向马来西亚中小企业做网站设计与开发的公司 —— 上面这些 AI 的东西，跑的就是我给客户搭的同一套基础设施。如果你想要一个基于**你自己**的数据、跑在**你自己**的硬件上、不挂按人头订阅费的模型，很欢迎聊聊：
+
+- 📱 **WhatsApp：** [+60 12-797 2969](https://wa.me/60127972969)
+- 📧 **邮箱：** [[email protected]](mailto:[email protected]?subject=AI%20stack%20for%20my%20business)
+- 🌐 **网站：** [hoelee.com](https://hoelee.com)
+
+如果你是招聘平台、DevOps 或 AI 基础设施岗位，同样的渠道也能找到我。