post(ai): the five-step AI roadmap — what I actually run in production (EN+ZH) + og/banner

src/content/posts/five-step-ai-roadmap-what-i-actually-run.md

@@ -0,0 +1,242 @@
+---
+title: "The Five-Step AI Roadmap Everyone Shares — and What I Actually Run in Production"
+description: "The five-step AI roadmap circulates on every feed. Here is the honest version: which steps I run in production, which one I skipped, and what each is for."
+pubDate: 2026-10-13
+category: ai
+tags: ["llm", "rag", "agents", "self-hosting", "whisper", "mem0"]
+ogImage: /og/five-step-ai-roadmap-what-i-actually-run.png
+banner: /banners/five-step-ai-roadmap-what-i-actually-run.png
+draft: false
+---
+
+There is a graphic that makes the rounds every few weeks, and by now it has
+reached my feed in translated form, reposted by accounts I don't follow. It
+lays out the path to working with large models in five steps: learn the
+fundamentals — Transformer, neural networks, attention — and stop as soon as
+you can read the diagrams; learn prompt engineering, which takes days; learn
+RAG so the model answers from your own data instead of making things up;
+learn agents, the biggest opportunity in the field; and then learn
+fine-tuning, where the advice is to skip pretraining entirely (that belongs
+to the big labs) and get comfortable with multimodal fine-tuning instead.
+There is a salary band attached to each step, and the band at the end is the
+reason the thing spreads.
+
+I am not going to argue with the list. It is not wrong — it is a syllabus.
+And a syllabus is the kind of advice that behaves perfectly in a classroom
+and then dies on contact with a system, because it tells you what to learn
+and in what order, and never tells you which of those items you will still
+be doing years later when somebody is paying you for the result. I have been
+running AI workloads of my own for a while now — not research, not model
+training, but real services other people depend on. So here is the honest
+version of that graphic, checked against what is actually running on my
+hardware tonight.
+
+## Why this matters
+
+The reason to distrust the roadmap is not that the items are wrong. It is
+that a roadmap tells you what to *study*, while employment asks what you can
+*operate*. Those are different skill sets, and the gap between them is where
+most people who "learned AI in five steps" stall: they can describe a RAG
+pipeline, but they have never had one break on them at 11pm in a way that
+made them understand retrieval, cost, ingestion and evaluation in a single
+evening.
+
+The rest of this post maps each step to something I actually run, states
+plainly which steps I have never done, and gives you the order I would pick
+if I had to start again. Every claim below is measured on one desktop with
+an RTX 3060 (12 GB) and one NAS.
+
+## Step 1: Fundamentals — enough to be dangerous, then stop
+
+The graphic's framing here is closer to right than most engineers admit:
+learn the Transformer until the diagrams make sense, then move on. My
+version stopped earlier than that. I can hold a real conversation about
+attention, tokens, context windows, quantization schemes and the KV cache,
+because those are the levers I actually turn when a model has to fit into 12
+GB of VRAM and answer fast enough to be useful in a chat window. I could not
+derive backpropagation on a whiteboard, and I would not trust a number I
+produced from memory.
+
+What that costs me: nothing measurable so far, in systems that ship. What it
+would cost me: the day I train something. If your goal is "own the AI stack
+for a small business", step 1 is a reading list, not a degree — and the
+honest reason to buy local hardware in the first place is not that a 12 GB
+card beats the frontier models, because it does not; it is what running a
+model yourself teaches you that an API never will. I wrote about that trade
+honestly in
+[why I still bought a local GPU to run AI](https://blog.hoelee.com/posts/why-i-still-bought-a-local-gpu/).
+
+## Step 2: Prompt engineering — the step with the lowest ceiling
+
+"Days to learn" is accurate, and that is the problem with it. Prompting is
+the cheapest step to enter and the first one to get automated, so it is
+where enormous numbers of people stop and start describing themselves as AI
+engineers. The honest skill is not writing a better instruction; it is
+deciding when an instruction should not exist at all.
+
+If a task has one correct answer, a deterministic input and a known output
+shape, then a script is faster, cheaper and cannot hallucinate. I have
+replaced agent prompts with forty lines of Python more than once, and
+[wrote up the case for that](https://blog.hoelee.com/posts/stop-writing-agent-prompts-for-deterministic-work/).
+What survives on the prompt side is interface design: what the model is
+allowed to see, what shape its answer must take, how much of your own
+context you are willing to hand it, and — the part the graphic never
+mentions — how you would know the output got worse. That last item is
+evaluation, and in my experience it is the difference between a demo and a
+service.
+
+## Step 3: RAG — the hard half is everything except the retrieval
+
+The one-line version everyone quotes is "give the model your data so it
+stops lying". I run exactly that: a self-hosted memory layer built from
+mem0, LiteLLM and PostgreSQL 17 with pgvector, all on my own NAS, with
+memories scoped to a `user_id` so one customer's history cannot surface in
+another's answer —
+[the whole stack is written up here](https://blog.hoelee.com/posts/self-hosting-mem0-memory-stack/),
+including the container layout and the fact that the API is two small HTTP
+calls.
+
+Three things the tutorial version of RAG skips, and each of them cost me
+real time:
+
+1. **Ingestion is the boring half that breaks.** Retrieval quality gets all
+   the attention; getting the data in reliably is where the work is. The
+   most recent input I pushed through my own speech-to-text stack was a
+   57-second video sitting on my desktop — it came back as text in about
+   eight seconds on the same 12 GB card that already serves live dictation
+   for my phone and laptop. That speed is what makes the stack usable
+   instead of a demo, and the pipeline is
+   [documented end to end here](https://blog.hoelee.com/posts/self-hosted-speech-to-text-api/).
+2. **Byte-level mistakes look like model mistakes.** My first attempt to
+   feed that same service through a workflow tool reported a clean 502
+   "conversion failed" from the audio layer. The real cause was a multipart
+   upload arriving without a `Content-Type` header, which made the broker
+   decode raw audio as UTF-8 text and destroy it before any of my code ran.
+   Nothing about that bug is AI. It is exactly the kind of failure the
+   five-step graphic cannot prepare you for.
+3. **Retrieval is not a model problem.** When an answer is wrong, the useful
+   question is whether the right chunk was ever retrieved, whether it
+   survived chunking, and whether the model was given room to use it.
+   Swapping to a bigger model has never once fixed a retrieval bug for me.
+
+Underneath that entire layer there are two calls, which is worth seeing
+once:
+
+```text
+POST http://127.0.0.1:20015/memories    # store a fact or a conversation, tagged by user_id
+GET  http://127.0.0.1:20015/search      # semantic search, scoped to the same user_id
+```
+
+Nothing exotic — which is the point. A memory layer you can describe in two
+lines is a memory layer you can debug, and the whole thing stays local.
+
+## Step 4: Agents — the opportunity is real, and the agent is the smallest part
+
+I agree with the graphic's ranking: this is where the work is. I disagree
+with what it implies by the word "agent", because an agent is a model plus a
+loop plus tools — and the part that decides whether it survives contact with
+reality is the checking around it, not the model at the centre. I've written
+about this as
+[loop engineering](https://blog.hoelee.com/posts/loop-engineering-without-a-coding-agent/),
+and the finding I keep coming back to is uncomfortable: most of my scheduled
+automation has no AI in it at all, and those are precisely the jobs that
+have never needed babysitting. The model is the part that adds judgement.
+The rest of the system is what makes the judgement safe to trust.
+
+Three things that are actually running here, so the claim is concrete rather
+than aspirational:
+
+- A Telegram support bot for my own digital-goods shop that keeps long-term
+  memory per customer, so a returning buyer is not a stranger on their
+  second message. The memory layer it writes into is the same self-hosted
+  mem0 stack from step 3.
+- A scheduled job that manages discount settings in my Shopee seller
+  account, which is the least glamorous useful agent I own: it reads the
+  current state, decides, acts, and reports — and the entire value comes
+  from the fact that it checks the result afterwards instead of assuming its
+  click landed.
+- A research run that compared twenty VPS providers in parallel by fanning
+  the work out across subagents, which is a pattern worth learning even if
+  you never build a "chatbot":
+  [the write-up is here](https://blog.hoelee.com/posts/how-i-vetted-20-vps-providers-with-parallel-subagents/).
+
+## Step 5: Fine-tuning — the step I have never done, on purpose
+
+The graphic says skip pretraining, which is correct, and then suggests
+learning multimodal fine-tuning. For anyone whose job looks like mine, I'd
+push back on the second half.
+
+I have never fine-tuned a model. Not because I couldn't, but because in the
+time I have been running these services, none of my failures have been
+model-behaviour failures. They were retrieval failures, licensing surprises,
+byte-level failures and operational failures — a job that reported success
+while the file it claimed to replace was untouched, a mount whose metadata
+lied about which file it had swapped, an API whose real model name differed
+from the one it advertised. A LoRA does not fix a schema drift.
+
+Where I would fine-tune: one narrow, well-defined task; thousands of
+labelled examples rather than dozens; an inference bill or a latency ceiling
+that a smaller tuned model would genuinely fix; and the willingness to own
+an evaluation set, because a fine-tune you cannot measure is a regression
+you cannot see. And I would rent the GPU for it long before I bought a
+second card — the training run is a project, not a workstation purchase.
+
+## What I'd do differently
+
+If I were starting from zero with that graphic in front of me, I would not
+walk it in numerical order. I would walk it in the order my problems
+arrived, and accept that the first real lesson will not come from a course:
+
+- Treat step 1 as a reading list you finish in public documentation, not a
+  curriculum. Enough theory to know why quantization and context limits
+  hurt; no more until something forces you deeper.
+- Spend a week on step 2, then immediately automate the parts of it that
+  have one correct answer, and move the rest of your attention to evaluation
+  — which the graphic omits entirely.
+- Live in steps 3 and 4, because that is where nearly all employed AI work
+  currently sits: wiring models to real data, and making the loop around
+  them trustworthy.
+- Keep step 5 in your back pocket until a specific problem names it.
+  Fine-tuning is a technique with a precondition, not a rung on a ladder.
+
+And notice the roadmap's real omission, which is not a step but half the
+job: identity, storage, deployment, cost, monitoring, and who gets woken up
+when it breaks. That is the half that decides whether a team can use you,
+and it is the half this blog mostly writes about.
+
+## The result
+
+Three of those five steps run in production on one desktop GPU and one NAS,
+and the honest scorecard is this:
+
+- A 57-second video becomes text in about eight seconds, on the same 12 GB
+  card that serves live dictation — and the speech server drops from **4,906
+  MiB to 1,321 MiB** of VRAM when I toggle it idle, because a GPU that is
+  always resident is a GPU I cannot use for anything else.
+- The memory layer answers in two small HTTP calls, scoped per user, backed
+  by PostgreSQL with pgvector on the NAS — mine, not a vendor's.
+- Most of the scheduled automation that keeps this stack alive contains no
+  model at all, and it is the most reliable part of the system.
+- Step 5 stays empty, and I would rather say that plainly than pad a
+  roadmap. The gap is real, and it is a choice.
+
+If the five-step graphic is the syllabus, this is the transcript. The
+syllabus tells you what to learn. The transcript is what a recruiter or a
+client can actually check.
+
+---
+
+## Want an AI stack that runs in your business?
+
+I run Hoelee Enterprise, a website design and development business for
+Malaysian SMEs — and the AI work above runs on the same infrastructure I
+build for clients. If you want a model answering from *your* data, on *your*
+hardware, without a per-seat subscription attached, I'd be glad to talk:
+
+- 📱 **WhatsApp:** [+60 12-797 2969](https://wa.me/60127972969)
+- 📧 **Email:**
+  [[email protected]](mailto:[email protected]?subject=AI%20stack%20for%20my%20business)
+- 🌐 **Website:** [hoelee.com](https://hoelee.com)
+
+And if you are hiring for a platform, DevOps or AI-infrastructure role, the
+same channels reach me.