hoelee/hoelee-blog
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
050d73305d65dc0766f66b31a68a56adbf75144a
hoelee-blog/docs/consultant-review/batches/batch-04.md
T

232 KiB
Raw Blame History

博客文章顾问审阅资料包 — Batch-04

批次编号：Batch-04
包含文章数：16 篇
本批次总字符量：168,304 字（严格限制 < 200,000 字）
模式：BILINGUAL
生成时间：2026-10-27 04:23:31

外部顾问审阅前置须知与工程背景规范

本文档目标：本批次包含 blog.hoelee.com 的待审阅技术文章全文。请外部顾问在审阅正文与技术主张时，以此规范为最高评估准则。

1. 站点定位与双重业务目的

  • 受众 A（技术招聘官 / 资深技术主管）：寻找具备 Senior 全栈 + DevOps + AI 智能体闭环能力的架构工程师。文章定位为**“不可伪造的工程实操交付证据（Proof of Expertise）”**，要求代码真实、日志可查、有架构权衡与技术深度。
  • 受众 B（中小企业主 SME / 商业客户）：寻找能提供系统研发、私有化部署、24/7 AI 获客系统、高可用基础设施的技术顾问。文章需清晰阐述业务痛点、商业 ROI、上线效率与交付保障。

2. 核心人设铁律与工程硬约束（必须严格遵守）

  1. 彻底杜绝业余爱好者/玩具口吻：
    • ❌ 绝对禁用词：homelab, home lab, 家用, 小主机, 玩具, 折腾。
    • ✅ 标准替代词：the infrastructure I manage（我管理的私有基础设施）、self-hosted production node（自托管生产节点）、on-premise host（私有化服务器）。
  2. 严禁捏造虚假指标与性能数字：
    • 所有 benchmark 数据必须有明确的测试基准与单设备限制。
    • 严格区分“耗时减少比例”与“吞吐加速倍数”（例如耗时从 352s 降至 218s 是耗时减少 38.0%，吞吐提升 1.61×，严禁偷换概念混淆）。
    • 时间戳精度与物理帧对齐：视频帧率上限（如 30 fps 单帧 33.3ms）不可妄称“微秒级精确”。
  3. 去模板化叙事与真实物理锚点：
    • 拒绝千篇一律的 AI 模板味。文章采用三大叙事架构：① 故障排查谜案（Symptom → False Lead → Kernel Evidence → Fix）；② 商业交付复盘（Client Bottleneck → Trade-offs → State Machine → ROI）；③ 架构演进对比（V1 Naive → Concurrency Crash → V2 Redesign）。
    • 每篇文章必须包含至少一个物理真实世界锚点（主机真实硬件规格、生产维护窗口、真实业务场景）。
  4. 代码与配置展示规范：
    • 超过 20 行的配置或日志采用 HTML <details><summary> 原生折叠，<summary> 必须精准标注文档名与行数。

3. 顾问重点开刀维度

  • 事实与数字可核性：技术判断与性能数据是否自洽？有无过度夸大？
  • 人设与语调纯洁性：有无遗漏的 hobbyist/homelab 口吻？
  • 双轨转化引导有效性：是否有清晰的业务价值提炼与行动召唤（CTA）？
  • 代码与生产安全性：有无未脱敏的凭据、绝对内部私密路径或不安全的权限配置？

本批次文章目录索引

  1. [engineering] 30 天提交 51.8 万行，生产环境只留下 23.5 万行 (why-500000-lines-of-commit-churn-only-left-230000-in-production) — ~12,919 字符
  2. [notes] 给别人家的邮件服务器加一个域名 (adding-a-domain-to-someone-elses-mail-server) — ~10,624 字符
  3. [notes] 你好，世界 —— 关于这个博客 (hello-world) — ~2,749 字符
  4. [notes] 我的备份爬得比爬虫还慢，因为我在用别人的 API 配额 (my-backup-was-using-someone-elses-api-quota) — ~12,287 字符
  5. [notes] NocoDB SSO 是授权功能：环境变量不会告诉你的事 (nocodb-sso-is-a-licensed-feature) — ~15,352 字符
  6. [notes] 与其看供应商文档，不如直接读容器自带的 API 文档 (reading-a-containers-own-api-docs) — ~13,865 字符
  7. [notes] smartctl 退出码 32：专门跳过你最该看的硬盘的那个“错误” (smartctl-exit-code-32-skips-the-disks-that-matter) — ~8,382 字符
  8. [notes] 确定性的活，别再用 Agent Prompt 去跑 (stop-writing-agent-prompts-for-deterministic-work) — ~7,167 字符
  9. [notes] 密码正确，Synology API 登录却返回 401，问题出在哪 (synology-api-401-with-the-correct-password) — ~8,106 字符
  10. [notes] NocoDB 附件字段怎么改都不生效，和一个漏网的正则 (the-nocodb-attachment-that-wouldnt-update) — ~5,726 字符
  11. [notes] CodeIgniter 4.6 升级到 4.7：官方指南没说到的破坏性变更 (upgrading-codeigniter-46-to-47) — ~14,880 字符
  12. [notes] 为什么本地语音合成一直在念 “Percent”？语音管线里的 URL 编码与沙盒避坑记 (why-is-my-tts-model-repeating-percent) — ~11,915 字符
  13. [notes] 为什么多个 AI 一起写代码总会烂尾？传话失真与隐式决策困境 (why-multi-agent-coding-fails) — ~8,640 字符
  14. [notes] 为什么你的 Windows 桌面 MCP 调用会静默失败：攻破 10 秒空闲门禁与避坑实录 (why-your-windows-mcp-calls-silently-fail) — ~8,459 字符
  15. [tutorials] 校准 Intel Quick Sync HEVC：28 与 30 之间的悬崖，和一个毫无作用的 preset (calibrating-intel-quick-sync-hevc-quality) — ~16,121 字符
  16. [tutorials] 转成 HEVC 到底会不会让文件变小？我实测了 (does-hevc-actually-shrink-your-files) — ~11,112 字符

文章 01 / 16：30 天提交 51.8 万行，生产环境只留下 23.5 万行

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "30 天提交 51.8 万行，生产环境只留下 23.5 万行"
description: "过去 30 天，自主编程 Agent 在 5 个仓库中累计新增了 51.8 万行代码，但最终留存仅 23.5 万行。为什么在 AI 时代，验证闸门与代码剪枝比生成速度重要十倍？"
pubDate: "2026-10-18"
category: "engineering"
blueprint: ai-agent-automation
blueprintLayer: L2-llm-gateway-agents
tags: ["ai", "git", "devops", "testing", "refactoring"]
ogImage: "/og/zh-why-500000-lines-of-commit-churn-only-left-230000-in-production.jpg"
banner: "/banners/why-500000-lines-of-commit-churn-only-left-230000-in-production.jpg"
---

上周我做了一件平时几乎不会做的事：数了数自己的提交。不是数我记得写过的那些，而是数仓库里真实记录下来的那些。自建 Gitea 上 5 个生产仓库、三十天、一个人：865 次 Commit、新增 518,233 行、删除 130,892 行。

然后我数了数活下来的部分。对主生产项目的最新版本跑一次干净扫描，有效代码总量是 234,976 行；同一时间窗口里净增 58,240 行的视频转码器，今天整个仓库只有 43,050 行。

那 28 万行进哪儿去了？没有丢。它们被写出来、跑过测试、被判为不对，然后删掉——或者被一个更小、做同一件事的东西替换掉。我想把这条循环记下来，因为它这一年对我工作方式的影响，比任何一次模型升级都大。

## 一开始我不信的这几个数字

以下是 2026 年 9 月 17 日至 10 月 17 日，名下 5 个活跃仓库的精确 Git 统计：

| 仓库名称 | 30 天 Commit | 累计新增 (+Add) | 累计删除 (-Del) | 净变动 (Net) | 当前存量有效代码/文本 | 核心技术栈 |
|---|---|---|---|---|---|---|
| **numerology-report** | 432 | +331,672 | -98,937 | +232,735 | **234,976** | PHP, Markdown, JSON |
| **hevc-transcode** | 187 | +58,240 | -15,508 | +42,732 | **43,050** | Python, HTML, Shell |
| **dsm-resource-management** | 132 | +58,234 | -14,076 | +44,158 | **368,793** | Python, TypeScript, Astro |
| **hoelee-blog** | 88 | +23,603 | -709 | +22,894 | **55,933** | Astro, TypeScript, Markdown |
| **hoelee-agent-dist** | 26 | +46,484 | -1,662 | +44,822 | *(Dist)* | Python, Shell, Markdown |
| **合计** | **865** | **+518,233** | **-130,892** | **+387,341** | — | — |

其中两行你可以自己复算，这就是完整的量法：

```bash
git -C numerology-report log --since=2026-09-17 --until=2026-10-17 \
  --numstat --pretty=format: | awk '{add+=$1; del+=$2} END {print add, del}'
```

在人类团队里，这样的更替量通常意味着出事了：需求在代码底下不停移动、有人偷偷在重写、架构反复塌陷。这几种情况我一条都不占——这 5 个仓库都是我每天在跑、离不开的东西，每一次提交都发生在能读到 diff 的会话里。

生成成本变低之后，这种更替量就是重构的样子：Agent 起草一段改动，我跑闸门，闸门不同意，这段就被丢掉或者改小重写。凡是闸门写对的那几天，代码总量往下掉得比功能上来的速度还快。

## Agent 实际接手的是什么

大多数人第一次让 AI 写代码，是在一个浏览器标签页里：把函数粘进对话框，再把答案拷回编辑器。这条路的瓶颈是剪贴板，而且模型离真正重要的两样东西都很远——仓库自己的约定，以及运行它刚写出来的东西的能力。

所以我的是跑在终端里、跑在这台机器上的，手里有真工具：

1. **文件与语法工具**：直接读已有模块，用模糊匹配打补丁而不是整文件覆盖，保存前先过语法与 AST 校验。
2. **测试与执行器**：PHPUnit 单元测试、pytest 套件、无头 Chrome 探针，在前台跑起来并把输出读回来。
3. **版本控制上下文**：读 `git diff`、显式暂存指定路径、翻一个文件的历史看上一个决定为什么这么做。

这套组合很快，比我亲手打第一版快一个数量级；但它本身也是让代码库腐烂得更快的方法——这一半通常没人提醒。

## 臃肿是从哪里长出来的

不受约束的模型天生爱写代码。给它一个细小的边界情况，它的第一反应是新建一个私有辅助类、把一个已有工具函数复制一份改个参数，或者给一个库套上三层抽象——那三层只有在这段 diff 里才讲得通。放它自己跑一周，就是几千行语气笃定、看起来合理、其实没必要的代码。

我这么讲是因为我自己的历史里就有那一周。每天读这些 diff，最后落成两条规矩：

- 新建一个类，要跟一个我本来就有的标准库调用竞争位置。
- 生成出来的东西默认有罪，直到某个确定性检查说它没问题。模型说「看起来对」不算证据，红的测试才算。

## 真正把代码删掉的是三道闸门

在最近一次主生产系统的 V2 架构重构中，三十天里删掉了 98,000 多行。这些几乎不是我坐下来专门去删的，而是三道闸门不断把回来的东西打回，最后留下的就是更小的版本：

### 1. 用确定性脚手架，而不是问「这段看着对不对」

让模型审查自己写的代码，多数时候产出的是自信，不是正确。有用的做法是把它接到一个没法被说服成绿灯的东西上：

- PHP 应用里，所有公开路由参数必须带显式默认值，由专用的反射探针（`ControllerAutoRouteSafetyTest.php`）守着——这个测试存在，是因为一条裸路由在生产上真的 500 过。
- 报告生成管线里，无头 PDF 解析器先数一遍渲染出的汉字数量，数不对就不许构建通过；这样字体或排版改动如果悄悄漏字，就是构建失败，而不是发出一份坏 PDF。
- 转码队列里，台账比的是精确字节数而不是 MB 估算，因为圆整过的数字把一次体积回退藏了好几个星期。

闸门红了，Agent 要么改到过，要么把这段撤掉。没有任何东西能豁免。

### 2. 把剪枝当成日常动作，而不是一次大扫除

每次会话都会分出一段时间专门删东西：这个 200 行的辅助类，能不能换成一行已有的原生调用？这个新的映射层，是不是在重复 `app/Libraries/` 里放了一年的某个函数？上面那 13 万行删除不是重写的副产品，而是活过头了的脚手架、只有一个调用方的迁移层，以及换了名字出现两遍的同一段逻辑。

### 3. 确定性的活留给确定性代码

我在 AI 项目里反复看到的坑，是让模型去做它根本不该碰的计算。如果一条 Shell 管道、一个 SQL 约束、一个幂等脚本或者一个 Cron 任务就能给出答案，那 LLM 就不该出现在那里。

模型擅长起草初始实现、生成三语词库、摸清三方 API 的脾气；但状态流转、钱、队列调度，我不会交给它——一个确定性问题上概率性的答案，是一颗带延时引信的 bug。

## 我的工作方式变了什么

我职业生涯的大部分时间里，物理瓶颈是打字速度和记语法。产出就是指标，一个高效的一周看起来就是新建了很多文件。

现在语法是免费的。我可以一边读上一次改动的 diff，一边让 Agent 起草一份带单元测试的 500 行迁移。于是瓶颈转移到了模型替我看不了的三件事上：

- **边界设计**：模块从哪里分开，哪些状态允许跨过去。
- **验证体系**：那个能抓住类型检查看不见的语义漂移的脚手架。
- **剪枝的决断力**：愿不愿意删掉那 28 万行探索性、冗余、已死掉的代码，只把 23.5 万行送进生产。

所以我现在不再把提交曲线当生产率图表看。一个月新增 50 万行，只能说明工具在转；真正说明这一个月值不值的是那删掉的 13 万行、底下的重构，和一套仍然全绿的测试。

## 需要一套能长期不倒的系统？

我运营 **Hoelee Enterprise**，为马来西亚中小企业做网站设计与系统开发。上面这套 Agent 工作流、测试脚手架和自托管基础设施，就是我给客户交付并长期维护系统时用的同一套东西。

如果你需要全栈系统研发、业务流程自动化，或者能扛住自己维护周期的自建基础设施：
- **WhatsApp:** [+60 12-797 2969](https://wa.me/60127972969)
- **Email:** [[email protected]](mailto:[email protected])
- **官网:** [hoelee.com](https://hoelee.com)

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "I Committed 518,000 Lines in 30 Days. Production Kept 235,000."
description: "Over 30 days, autonomous coding agents added 518,000 lines across 5 repos, yet production kept 235,000. Here is why verification gates matter more than generation."
pubDate: "2026-10-18"
category: "engineering"
blueprint: ai-agent-automation
blueprintLayer: L2-llm-gateway-agents
tags: ["ai", "git", "devops", "testing", "refactoring"]
ogImage: "/og/why-500000-lines-of-commit-churn-only-left-230000-in-production.jpg"
banner: "/banners/why-500000-lines-of-commit-churn-only-left-230000-in-production.jpg"
---

Last week I did something I almost never bother to do: I counted my own
commits. Not the ones I remember writing — the ones the repositories
actually recorded. Five production repos on my self-hosted Gitea, thirty
days, one person: 865 commits, 518,233 lines added, 130,892 deleted.

Then I counted what survived. A clean scan of the current revision of my
main production application returns 234,976 lines. The video transcoder that
gained 58,240 lines in the same window is 43,050 lines long today.

So where did the other 280,000 lines go? Nothing was lost. They were
written, run, judged wrong, and deleted — or replaced by something smaller
that did the same job. That loop, not the writing, is what I want to write
down, because it changed how I work this year more than any model update
did.

## The numbers I didn't believe

Here is the exact commit breakdown across the five repositories, between 17
September and 17 October 2026:

| Repository | Commits (30d) | Additions (+) | Deletions (-) | Net Change | Current Total LOC | Primary Stack |
|---|---|---|---|---|---|---|
| **numerology-report** | 432 | +331,672 | -98,937 | +232,735 | **234,976** | PHP, Markdown, JSON |
| **hevc-transcode** | 187 | +58,240 | -15,508 | +42,732 | **43,050** | Python, HTML, Shell |
| **dsm-resource-management** | 132 | +58,234 | -14,076 | +44,158 | **368,793** | Python, TypeScript, Astro |
| **hoelee-blog** | 88 | +23,603 | -709 | +22,894 | **55,933** | Astro, TypeScript, Markdown |
| **hoelee-agent-dist** | 26 | +46,484 | -1,662 | +44,822 | *(Dist)* | Python, Shell, Markdown |
| **Total** | **865** | **+518,233** | **-130,892** | **+387,341** | — | — |

Two lines are easy to check yourself; this is the whole measurement:

```bash
git -C numerology-report log --since=2026-09-17 --until=2026-10-17 \
  --numstat --pretty=format: | awk '{add+=$1; del+=$2} END {print add, del}'
```

In a human team, churn like that usually means trouble: requirements moving
under the code, a rewrite nobody admits to, an architecture that keeps
collapsing. None of those applied here. Every one of these five repos is
something I run and depend on, and every commit was authored in a session
where I could read the diff before it landed.

The churn is what refactoring looks like when generation is cheap. An agent
drafts a slice of the change, I run the gates, the gates disagree, and the
slice gets thrown away or rewritten smaller. On the days I got the gates
right, the size drop came faster than the features arrived.

## What the agents actually touch

Most people's first experience of AI code is a browser tab: paste a function
into a chat window, copy the answer back into an editor. The bottleneck
there is the clipboard. It also keeps the model far away from the two things
that matter — the codebase's own conventions and the ability to run what it
wrote.

So mine run in a terminal, on the machine, with real tools:

1. **Filesystem and syntax tools.** They read existing modules, patch by
   fuzzy match instead of overwriting whole files, and run syntax or AST
   checks before anything is saved.
2. **Test and execution runners.** PHPUnit suites, pytest harnesses, and
   headless Chrome probes, executed in the foreground with the output read
   back.
3. **Version control.** They read `git diff`, stage explicit paths, and look
   at the history of a file to see why the previous decision was made.

That combination is fast — an order of magnitude faster than me typing the
first draft. It is also, on its own, a way to make a codebase rot faster,
which is the part nobody warns you about.

## Where the bloat comes from

Unsupervised models love to write code. Give one a small edge case and its
instinct is to add a private helper class, clone an existing utility with
one extra parameter, or wrap a library in three layers of abstraction that
only make sense to the diff that introduced them. Left alone for a week,
that is a few thousand lines of confident, plausible, unnecessary code.

I know because I have that week in my own history. Two things came out of
reading these diffs every day:

- A new class has to earn its place against a standard-library call I
  already have.
- Anything generated is guilty until a deterministic check says otherwise.
  "Looks correct" from a model is not evidence; a red test is.

## The gates that actually deleted the code

During the V2 architecture overhaul of my main production application, more
than 98,000 lines went out in thirty days. Almost none of that was me
sitting down to delete things. It went because three gates kept rejecting
what came back:

### 1. Deterministic harnesses instead of "does this look right?"

Asking a model to review its own code mostly produces confidence, not
correctness. The useful move is to wire the work to something that cannot be
talked out of failing:

- In the PHP application, every public route parameter has to carry an
  explicit default, enforced by a reflection probe
  (`ControllerAutoRouteSafetyTest.php`) — that test exists because a bare
  route crashed with a 500 in production.
- In the report pipeline, a headless PDF extractor counts the rendered
  Chinese characters before a build is allowed to pass, so a font or layout
  change that silently drops glyphs fails the build instead of shipping a
  broken PDF.
- In the transcoder, the ledger compares exact byte counts rather than
  approximating in megabytes, because rounded numbers hid a size regression
  for weeks.

If a gate goes red, the agent either fixes the change or drops it. Nothing
reaching a commit is exempt from that.

### 2. Pruning as a standing task, not a cleanup project

A portion of every session goes to deleting. Can a 200-line helper be
replaced by a ten-line call into something that already exists? Is this new
mapper duplicating a function that has lived in `app/Libraries/` for a year?
The 130,000 deletions above are not an accident of a rewrite; they are
scaffolding that outlived its purpose, migration layers with one caller, and
logic that arrived twice under two names.

### 3. Keeping deterministic work deterministic

The trap I keep seeing in AI-assisted projects is a model being asked to do
arithmetic it should never touch. If a shell pipeline, a SQL constraint, an
idempotent script, or a cron job can answer it, an LLM has no business
there.

Models are good at drafting an initial implementation, generating a
three-language dictionary, or mapping a third-party API's quirks. They are
not where I put state transitions, money, or queue scheduling, because a
probabilistic answer to a deterministic question is a bug with a delay fuse.

## What changed in how I work

For most of my career the physical constraint was typing speed and
remembering syntax. Output was the metric, and a productive week looked like
a lot of new files.

Syntax is free now. I can get a 500-line migration with tests drafted while
I read the diff of the previous one. So the constraint moved to the three
things a model cannot check for me:

- **Boundary design** — where components separate, and which state is
  allowed to cross.
- **Test architecture** — the harness that catches semantic drift the type
  checker never sees.
- **Pruning discipline** — the willingness to delete 280,000 lines of
  exploratory, redundant, or dead code so 235,000 lines can go to
  production.

Which is why I no longer read a commit graph as a productivity chart. Half a
million added lines in a month tells me the tools were running. The 130,000
deleted lines, the refactor underneath them, and a suite that is still green
tell me whether the month was worth anything.

## Building something that has to stay up?

I run **Hoelee Enterprise**, a website design and development studio for
Malaysian SMEs. The agent workflow, the test harnesses and the self-hosted
infrastructure in this post are the same setup I build and maintain client
systems on.

If you need full-stack engineering, workflow automation, or infrastructure
that survives its own maintenance:
- **WhatsApp:** [+60 12-797 2969](https://wa.me/60127972969)
- **Email:** [[email protected]](mailto:[email protected])
- **Website:** [hoelee.com](https://hoelee.com)

文章 02 / 16：给别人家的邮件服务器加一个域名

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "给别人家的邮件服务器加一个域名"
description: "收信只要四条 DNS 记录，一次就通了；而从它发信、以及转发它，撞上两堵任何 DNS 记录都修不了的墙。"
pubDate: 2025-05-06
updatedDate: 2026-10-16
category: notes
blueprint: sme-digital-suite
blueprintLayer: L2-sifumail-communication
tags: ["email", "dns", "mailcow", "smtp", "deliverability"]
ogImage: /og/zh-adding-a-domain-to-someone-elses-mail-server.jpg
banner: /banners/adding-a-domain-to-someone-elses-mail-server.jpg
draft: false
---

## 这件事看起来像个 DNS 活

有个域名要收信。邮件服务器是我自己那台跑 mailcow 的 VPS，所以这就是「加域名、加记录、建信箱」——二十分钟的活，而且我做过好几次。

这次花了更久，原因是我关于邮件服务器的两个假设并不成立，另外还因为互联网有一条任何服务器配置都改不了的性质。

## 收信：四条记录，加一次真测试

收信恰好只需要四样东西，而且都住在域名自己的 DNS 里（这里是 Cloudflare）：

```text
MX      ninelore.com        → mail.hoelee.com   （优先级 10）
TXT     @                   → v=spf1 mx -all
TXT     dkim._domainkey     → （MTA 发布的那个选择器）
TXT     _dmarc              → （策略 + 报告地址）
```

配好之后，只用一种算数的方式测：**从这套环境外面**（走第三方中继）寄进新地址，然后去读服务器自己的日志行——`rspamd` 在收下这封信时记下 `RCPT_MAILCOW_DOMAIN(ninelore.com)`。收信一次就通。**收信确实是容易的那一半**，而它之所以容易，正因为它只依赖你自己能控制的记录。

## 出信才是地面移动的地方

两堵独立的墙，各有各的错误，而且都不是改一改区域文件就能修的配置失误。

**第一堵墙：那条反垃圾策略，本质上是缺一条 PTR。** 一次直投尝试被对方退回：

```text
554 5.7.1 Mail rejected due to antispam policy
```

在怪远端之前，我先排掉常见原因：四个主流黑名单全干净，域名自己的 SPF/DKIM/DMARC 也都正确。剩下的解释就是那条不在我控制里的记录——**发信 IP 没有反向 DNS**。一台收信服务器如果无法把连进来的 IP 映射回一个名字，就会把它当成一个它无法背书的来源；再正确的 SPF 也压不过这一点。

**第二堵墙：用中继绕过去也不行。** 下一个显然的动作是让所有信走那条已经能用的出信中继。结果是：

```text
550-From header sender domain not verified (gmail.com)
```

这行错误要仔细读，因为它一次打死了一整类设计。中继的「已验证寄件人」策略，检查的是 **From 头里的域名**——而**转发**出去的信保留的是原寄件人的 From。所以从你转发第一封来自外部寄件人的信开始，你就是在要求中继以 `gmail.com` 的身份发信，而它永远不会允许。mailcow 在这里还有个值得先知道的坑：它按**寄件人**域名选中继，所以一封 From 是外部域的信，根本不会走到你配的那条中继上。

## 那个并不存在的「信箱转发」功能

原计划是建好信箱，再转发一份到我的主邮箱。结果连着两个意外：

- mailcow **没有**「信箱转发」这个设置。把信箱地址写进别名的 `goto` 列表会被拒（`is_alias_or_mailbox`），因为每个信箱自带一条 self-alias 记录，代码在算目标时会跳过这个地址。
- 想「保留信箱**并且**转发」只能走 **Sieve 过滤器**——那是另一套机制、另一种失败形态——而它最后仍然落在第一堵墙和第二堵墙上，因为这封信终究得离开这台机。

所以我**什么都没启用**。一条出信路径送不出去的转发，不会安静地失败：它会把退信发回给无辜的原始寄件人（backscatter），而每一封这样的退信都在花一个本来就没有余量的 IP 的信誉。对「这个新信箱要送到我的收件箱」这件事，稳定的答案是**收件侧的 POP 拉取**：我的邮件客户端从 `mail.hoelee.com:995` 拉信，VPS 什么都不用发。

## PTR 是一个 IP 一条，不是一个域一条

这是我一直需要解释、包括对我自己解释的部分：

- 反向记录是**IP 地址**的属性。不存在「每个域一条 PTR」，下次再加域也没有任何 PTR 要动。
- 它正确的值就是邮件服务器自己对外报的名字——这里是 `mail.hoelee.com`，也就是 postfix 用作 HELO/myhostname 的那个。正向记录（`mail.hoelee.com → 41.216.187.102`）已经存在，**这一对**才是收信方要验证的东西：**forward-confirmed rDNS**。缺任何一半都一文不值。
- 那个 IP 上所有域名都用同一条反向记录来评判。一百个域名，一条 PTR。
- 谁来设：跑反向区的人。这里 `216.41.in-addr.arpa` 与 `187.216.41.in-addr.arpa` 两级的 SOA 都是 VPS 商自己的名字服务器，所以它是**服务商侧**的改动——先去 VM / 网络面板里找有没有 rDNS/PTR 输入框，没有就提工单，把上面那个确切的值给他们。

我没有假设，而是去查了当前状态：正向解析正常（`mail.hoelee.com → 41.216.187.102`），**那个 IP 的反向解析依旧什么都不返回**。所以直投仍然会被拒，唯一能用的出信路径是中继——而它那条「已验证寄件人」的要求，新域名一样得满足。

## 三件要提前知道的事，免得以后被吓到

1. **PTR 不是信誉。** 它拆掉的是一次硬拒；它不会让你被信任。新 IP 通常需要几周的日常通信，大厂才会停止把它塞进垃圾箱——而从全新 IP 起步，就是把这口钟从零开始拨。
2. **PTR 修的是你自己的出信，对转发毫无帮助。** 转发信带的是别人的 From；如果那个域发布了 `p=reject`（银行和大厂都会），那么不管你这边多干净，它在远端都会被挡。
3. **中继有一个按域的前置条件。** 一个域要走中继，就必须先在中继那边注册并验证它为寄件人。这是每个域永远多出来的一步，且与你自己的 DNS 里的一切无关。

## 如果重来一次

1. **在建域之前先定：要信箱还是要转发。** 转发是「必须成立的条件」最长的那条路；收件侧 POP 拉取几乎没有条件。
2. **在向任何人承诺邮件之前，先查发信 IP 的反向记录。** 一次反向查询就能告诉我接下来会撞上哪堵墙。
3. **把「收件侧拉取」当一等设计，而不是权宜之计。** 「这封信根本不用离开我的服务器」一次消掉三大类故障。
4. **把那四条收信记录当成一张清单。** 那一半很无聊、可验证，而且是唯一一次就通的部分。

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "Adding a Domain to Someone Else's Mail Server"
description: "Receiving mail took four DNS records and worked first try. Sending from it, and forwarding it, hit two walls that no DNS record can fix."
pubDate: 2025-05-06
updatedDate: 2026-10-16
category: notes
blueprint: sme-digital-suite
blueprintLayer: L2-sifumail-communication
tags: ["email", "dns", "mailcow", "smtp", "deliverability"]
ogImage: /og/adding-a-domain-to-someone-elses-mail-server.jpg
banner: /banners/adding-a-domain-to-someone-elses-mail-server.jpg
draft: false
---

## The task looked like a DNS job

A domain needed mail. The mail server is my own VPS running mailcow, so this
is "add the domain, add the records, create the mailbox" — the kind of task
that takes twenty minutes and that I have done several times.

It took longer, because two of the things I assumed about mail servers are
not true, and because of one property of the internet that no amount of
server configuration can change.

## Inbound: four records and a real test

Inbound mail needs exactly four things, and all four live in the domain's
own DNS (Cloudflare here):

```text
MX      ninelore.com        → mail.hoelee.com   (priority 10)
TXT     @                   → v=spf1 mx -all
TXT     dkim._domainkey     → (the selector the MTA publishes)
TXT     _dmarc              → (policy + reporting address)
```

Configured and then tested the only way that counts: mail sent **from
outside** the estate (through a third-party relay) into the new address, and
then read the server's own log line — `rspamd` recording
`RCPT_MAILCOW_DOMAIN(ninelore.com)` as the message was accepted into the
mailbox. Receiving worked on the first try. Inbound is genuinely the easy
half, and it is easy precisely because it depends only on records you
control.

## Outbound is where the ground moves

Two separate walls, each with its own error, and neither is a config mistake
you can fix by editing a zone file.

**Wall one: the anti-spam policy that is really a missing PTR.** A direct
delivery attempt to the destination provider came back:

```text
554 5.7.1 Mail rejected due to antispam policy
```

Before blaming the remote, I checked the usual causes: four major
blocklists, all clean, and the domain's own SPF/DKIM/DMARC all correct. The
remaining explanation was the one with no DNS record in my control — the
sending IP had no reverse DNS. A receiving server that cannot map the
connecting IP back to a name treats it as a source it cannot vouch for, and
no amount of correct SPF outweighs that.

**Wall two: relaying your way around it does not work either.** The obvious
next move is to send everything through the outbound relay that already
works. That attempt produced:

```text
550-From header sender domain not verified (gmail.com)
```

Read that error carefully, because it kills a whole family of designs. The
relay enforces its verified-sender policy against the **From header domain**
— and a *forwarded* message keeps the original sender's From. So the moment
you forward mail from an outside sender, you are asking your relay to send
as `gmail.com`, which it will never allow. mailcow makes this worse in a way
that is worth knowing before you try: it selects the relay by **sender**
domain, so a message with an outside From does not even reach the relay you
configured.

## The forwarding feature that does not exist

The plan was to create the mailbox and forward a copy to my main address.
That is two surprises in a row:

- mailcow has no "mailbox forwarding" setting. Writing the mailbox address
  into an alias' `goto` list is rejected (`is_alias_or_mailbox`) because
  every mailbox already carries a self-alias record and the code skips the
  address when computing destinations.
- Keeping the mailbox *and* forwarding requires a **Sieve filter**, which is
  a different mechanism with a different failure mode — and it lands on wall
  one and wall two anyway, because the message still has to leave the box.

So I enabled **nothing**. A forward whose outbound path cannot deliver does
not fail quietly: it sends the rejection back to the innocent original
sender — backscatter — and every one of those bounces spends the reputation
of an IP that has none to spare. The stable answer for "this new mailbox
should reach my inbox" turned out to be a **POP pull on the receiving
side**: my mail client fetches from `mail.hoelee.com:995` and nothing has to
be sent from the VPS at all.

## PTR is one per IP, not one per domain

This is the part I keep having to explain, including to myself:

- The reverse record is a property of the **IP address**. There is no
  per-domain PTR, and there is nothing to add the next time a domain is
  added.
- Its correct value is the name the mail server itself announces — in this
  case `mail.hoelee.com`, which is what postfix sends as its
  HELO/myhostname. The matching forward record
  (`mail.hoelee.com → 41.216.187.102`) already exists, and the pair is what
  receivers verify: **forward-confirmed rDNS**. One half without the other
  is worth nothing.
- Every domain on that IP is judged by that same reverse record. A hundred
  domains, one PTR.
- Who sets it: whoever runs the reverse zone. Here the SOA for both
  `216.41.in-addr.arpa` and `187.216.41.in-addr.arpa` is the VPS host's own
  nameserver, so the record is a provider-side change — look for an rDNS/PTR
  field in the VM or network panel first, and otherwise open a ticket with
  the exact value above.

I checked the current state rather than assuming: the forward lookup
resolves (`mail.hoelee.com → 41.216.187.102`) and the reverse lookup for
that IP still returns nothing. So direct delivery continues to be rejected,
and the only outbound path that works is the relay — with its
verified-sender requirement, which new domains have to satisfy there too.

## Three things to expect, so you are not surprised later

1. **PTR is not a reputation.** It removes a hard rejection; it does not
   make you trusted. A new IP typically needs weeks of ordinary mail before
   the big providers stop folding it into spam, and starting from a fresh IP
   means starting that clock from zero.
2. **PTR fixes your own outbound and does nothing for forwarding.** A
   forwarded message carries someone else's From, and if that domain
   publishes `p=reject` — banks and large providers do — it is blocked at
   the far end no matter how clean your setup is.
3. **Relaying has a per-domain precondition.** If a domain sends through a
   relay, that domain must be registered and verified as a sender with the
   relay. It is an extra step per domain forever, and it is independent of
   everything in your own DNS.

## What I would do differently

1. **Decide mailbox-or-forward before creating the domain.** Forwarding is
   the option with the longest list of things that must be true; a POP pull
   on the receiving side has almost none.
2. **Check the sending IP's reverse record before promising anyone mail.**
   One reverse lookup would have told me which of the two walls I was about
   to hit.
3. **Treat a receiving-side pull as a first-class design, not a
   workaround.** "The message never has to leave my server" removes three
   whole categories of failure at once.
4. **Keep the four inbound records as a checklist.** That half is boring,
   verifiable, and was the part that worked first try.

文章 03 / 16：你好，世界 —— 关于这个博客

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "你好，世界 —— 关于这个博客"
description: "这个博客是做什么的：技术写作、自托管经验，以及一份我构建与学习过程的记录。"
pubDate: 2024-09-24
updatedDate: 2026-09-05
category: notes
tags: [intro]
ogImage: /og/zh-hello-world.jpg
banner: /banners/hello-world.jpg
---

欢迎。这里写下我构建与学习的东西——主要是工程、DevOps 和自托管，中间也会
混一些 AI 自动化和 Web3 的实验。

## 这里会有什么

这里的每篇文章都有两种语言。英文版先写，中文版是完整翻译，而不是摘要——同一个 slug，放在 `/posts/zh/` 下面。

大部分文章是几种形态之一：

- **案例研究**——「我怎么做出 X」，包含架构和踩过的坑。
- **教程**——一个难题、我试过什么、最后怎么修好的。
- **踩坑与笔记**——关于那些耗掉我一整天的小事情，写下简短记录。

## 我写什么

- **全栈工程**——PHP/CodeIgniter、Java/Spring、TypeScript/React、WordPress。
- **DevOps 与自托管**——Docker、Traefik、Cloudflare 隧道、NAS/unRaid 私有基础设施。
- **AI 与自动化**——Telegram 机器人、n8n 工作流、本地 LLM、记忆系统。
- **Web3**——Solidity、Foundry、ERC-20/721（仍在学习，如实呈现）。

## 关于我

我是 Lee Teong Hoe（Mr Hoelee），一名常驻马来西亚的全栈开发者兼 DevOps 工程师。
我做 Web 应用，运维三台自管主机上的 162 个容器，并经营 Hoelee Enterprise
——一家面向马来西亚中小企业的网站设计与开发公司（邮箱托管是附带的服务，
不是主业）。

这里的文章是我的动态作品集，不是推销页：我做了什么、什么坏了、证据到底说了什么。
可以从[最新文章](/posts/zh/)开始看。

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "Hello, World — About This Blog"
description: "What this blog is for: technical writing, self-hosting experience, and a record of what I build and learn."
pubDate: 2024-09-24
updatedDate: 2026-09-05
category: notes
tags: [intro]
ogImage: /og/hello-world.jpg
banner: /banners/hello-world.jpg
---

Welcome. This is where I write about what I build and learn — mostly
engineering, DevOps, and self-hosting, with some AI automation and Web3
experiments mixed in.

## What to expect

Every post here exists in two languages. I write the English version first,
and the Chinese (简体中文) version is a full translation rather than a summary —
the same slug, served under `/posts/zh/`.

Most posts are one of a few shapes:

- **Case studies** — "how I built X", with the architecture and the mistakes.
- **Tutorials** — a hard problem, what I tried, and the fix.
- **Gotchas & notes** — short entries on the small things that cost me a day.

## What I write about

- **Full-stack engineering** — PHP/CodeIgniter, Java/Spring, TypeScript/React, WordPress.
- **DevOps & self-hosting** — Docker, Traefik, Cloudflare tunnels, NAS/unRaid homelabs.
- **AI & automation** — Telegram bots, n8n workflows, local LLMs, memory systems.
- **Web3** — Solidity, Foundry, ERC-20/721 (learning, honestly framed).

## About me

I'm Lee Teong Hoe (Mr Hoelee), a full-stack developer and DevOps engineer
based in Malaysia. I build web applications, run 162 containers
across three self-managed hosts, and run Hoelee Enterprise — a website design & development business
for Malaysian SMEs (with email hosting as a side offering, not the focus).

The posts here are my working portfolio rather than a sales page: what I
built, what broke, and what the evidence actually said. Start with the
[latest posts](/posts/).

文章 04 / 16：我的备份爬得比爬虫还慢，因为我在用别人的 API 配额

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "我的备份爬得比爬虫还慢，因为我在用别人的 API 配额"
description: "1 TB 的 restic 种子传输卡在零字节。日志里点名的那个 Google 项目不是我的：那是 rclone 的共享 OAuth 客户端，配额和全世界每个 rclone 用户共池。"
pubDate: 2026-10-15
updatedDate: 2026-10-16
category: notes
blueprint: self-hosted-infrastructure
blueprintLayer: L4-secops-cicd-dr
tags: ["rclone", "backup", "google-drive", "oauth", "restic"]
ogImage: /og/zh-my-backup-was-using-someone-elses-api-quota.jpg
banner: /banners/my-backup-was-using-someone-elses-api-quota.jpg
draft: false
---

## 订阅到期了，我得搬一个 TB

这件事以最不方便的方式找上门：承载我每晚备份的那个 OneDrive 账号已经没有订阅了，而手上还躺着一个 1.5 年没用的 Google 订阅。把备份搬过去、保住历史、别丢一晚。

第一件撞上的不是技术问题：**我记不清这套备份到底是谁在写**。rclone？Duplicati？还是群晖自带的什么？在一套 78 个 compose 栈、三台机器的环境里（就是[《一个人跑生产基础设施》](/posts/running-production-infrastructure-solo/)里那套），「我的备份是谁写的」本该是我张口就答的问题，而它当时不是。所以第一步是去读机器。

## 现场到底有什么

**两份 rclone 配置**，正是那种会咬人的细节：Backrest（restic 的前端）用的是 GUI 那份，而另外还有一份**陈旧的 CLI 副本**，被某个 cron 或某次人手 shell 用过。改了一份，等于只改了一半系统。

搬迁移本身在顺序对的时候很无聊，而我是先读后动的：

```text
rclone.conf                     追加 [gdrive-*] + [gdrive-*-crypt]
rclone/rclone.conf（CLI 副本）   同样两个 remote
Portainer stack 177             env → 新的 crypt remote；配置 :ro → :rw
restic init                     在 Google Drive 上建新仓库
Backrest SetConfig              dsm-repo 的 URI → 新位置
```

在动计划 URI 之前，我先去源码里确认了一件一切依赖它的事：**restic 仓库的身份是 `guid = sha256(id)`** —— 一个由仓库 id 派生的稳定值，**不是**从 URL 推出来的。所以把计划指到新后端，不可能悄悄把历史分叉。六个计划事后逐字节与原版一致，仓库的 `id`/`guid` 未被动过。

## 然后种子停住了，而日志里点了一个陌生人的名字

新仓库灌到 1075 GB 里的大约 2.88 GB —— 0.27% —— 然后慢成爬行。实时日志里有答案：

```text
'Read requests per minute' ... reason: RATE_LIMIT_EXCEEDED
consumer: projects/202264815644        <- 不是我的项目
quota_limit_value: 840000
```

`202264815644` 是 **rclone 自己的共享 OAuth 项目**。我那个 TB 级种子吃的是每分钟读取配额，和地球上每一个 rclone 用户共池——这就是为什么它从约 750 KB/s 掉到零，而我的 NAS、我的磁盘、我的线路全都闲着。等我去量的时候，种子已经完全停摆：**90 秒内 0 字节**、159 个错误、依旧 `rateLimitExceeded`；而六个夜间计划也全撞同一堵墙：

```text
Fatal: unable to save snapshot: unexpected HTTP response (500)
googleapi: Error 403: Quota exceeded for quota metric 'Queries'
  and limit 'Requests per minute' ...
```

在进度条里，「慢」和「被挡住」看起来一样，而它们是两个完全不同的问题。**采样 90 秒得到的零**，才是让我做出这个判断的东西。

## 你必须有自己的 OAuth 客户端，而控制台绕不过去

唯一的真修法只有一个：别再用共享客户端，用你自己的。也就是一个 Google Cloud 项目、一个 OAuth 同意屏、一个凭据。两个设置花掉了真正的时间：

- **Application type 必须选 Desktop app。** rclone 的回调是 `http://127.0.0.1:53682`，只有这个类型能不用额外配置就允许它。
- **要点「Publish app」。** 留在 Testing 状态，Google 会**每 7 天**让刷新令牌过期一次——那就把一个「能跑」的系统变成了每周一次的静默故障。

我查过能不能用机器上已经认证好的 CLI 抄近路。不能，而原因值得记下来：这里的 `gcloud` 只以一个**别人的项目里的服务账号**登录，而服务账号**无法**创建同意屏或 OAuth 客户端 ID。凭据必须属于一个真人拥有的项目。这是一条死路——我是验证出来的，不是猜的。

## 授权流程里的三个坑

1. **未经验证就发布的应用，可能会硬性堵死受限 scope。** `.../auth/drive` 是受限 scope，一个**已发布但未完成 Google 审核**的应用有时会被直接拒绝且**没有旁路**。我这一次旁路是存在的——「Google hasn't verified this app」页面，然后 **Advanced → Go to rclone-backup (unsafe)**——但那取决于 scope 与审核状态，是一枚硬币，而不是保证；而「unsafe」只是个标签，意思是从没走过付费审核。
2. **回调只接受一次请求。** 刷新页面，或者开第二个标签，都会在**没有 code** 的情况下打到 `127.0.0.1:53682`，把这次尝试弄死。我第一次就是这么死的（`no code returned`），而它看起来像凭据错了。**一个标签，别刷新。**
3. **同意屏必须在跑 rclone 的那台机器的浏览器里完成。** 回调指向 `localhost`，所以手机或另一台电脑完不成——浏览器必须和等待那个端口的进程在同一台主机上。

之后就简单了：把 token 读出来，把 `client_id` + `client_secret` + `token` 写进**两份**配置（GUI 那份和陈旧的 CLI 副本），并恢复两个对长时无人值守上传很关键的开关：`use_trash = false` 与 `stop_on_upload_limit = true`。

## 旧的那一半：靠读，不靠问

清空旧 remote 时报了错——`itemNotFound`——而空间照样回来了：

```text
Used:   1.012 TiB  ->  3.800 GiB
Free:   ~0         ->  1.007 TiB
DSM 文件夹: 在 -> 不在        回收站: 0 B
```

那个错误是 rclone 在**干完活之后**发的抱怨。另外值得记一笔：工具的文档说个人账号无法 `permanentDelete`，按它你就得手动清回收站——实际是回收站已经是空的，约 1 TB 就这么回来了，不需要任何浏览器步骤。文档描述的是 API，而 API 不总是你拿到的东西。

还有一条量出来的事，因为它改变了我下一个决定：把那台机器上共享的同步容器暂停之后，负载均值直接减半——16.70 → 8.89。这就是「网络慢」和「机器忙」的区别。

## 如果重来一次

1. **从第一天起就用自己的 OAuth 客户端。** 共享客户端 ID 是一条你排在里面、却看不到进度、也找不到人负责的队伍。
2. **「我的备份是谁写的」要写在文档里，不靠记忆。** 两份 rclone 配置，对一个一年没人画过图的系统来说是很正常的漂移量。
3. **读目的地，别读退出码。** 那次清空「失败」了，却释放了一个 TB；而一次看起来成功的改名，可能什么都没落地。跟[《它报告成功，其实什么都没变》](/posts/it-reported-success-nothing-had-changed/)是同一课——这套环境一直在教这一课。
4. **不要在注意力只剩最后百分之几的时候启动一个要跑几天的上传。** 在 0.27% 时发现 `RATE_LIMIT_EXCEEDED` 只花几分钟；跑了两天才发现，花掉的是整个窗口。

## 结果，诚实地说

管道是通的：计划现在住在 Google Drive 上、仓库身份没变、种子花的是**我的**配额而不是共享池、旧的那一侧退回了大约 1 TB。但它现在**还不是一个备份**：写这篇的时候，第一轮种子才上传了约 1075 GB 里的 2.88 GB，而且新仓库里**快照数为零**——在第一张快照落地之前，什么都还原不了。那是我盯着的数字，因为「传输在跑」和「你能还原」是两句不同的话。

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "My Backup Was Crawling Because I Was Using Someone Else's API Quota"
description: "A 1 TB restic seed stalled at zero bytes. The log named a Google project I did not own: rclone's shared OAuth client, quota pooled with every rclone user alive."
pubDate: 2026-10-15
updatedDate: 2026-10-16
category: notes
blueprint: self-hosted-infrastructure
blueprintLayer: L4-secops-cicd-dr
tags: ["rclone", "backup", "google-drive", "oauth", "restic"]
ogImage: /og/my-backup-was-using-someone-elses-api-quota.jpg
banner: /banners/my-backup-was-using-someone-elses-api-quota.jpg
draft: false
---

## The subscription ended, and I had to move a terabyte

The call came in the least convenient way: the OneDrive account holding my
nightly backups had no subscription any more, and I had a 1.5-year Google
subscription sitting unused. Move the backup, keep the history, don't lose a
night.

The first thing I hit was not a technical problem: I could not remember
*how* the backup was wired. rclone? Duplicati? Something Synology shipped?
On a three-host estate with 78 compose stacks — the one in
[Running Production Infrastructure Solo](/posts/running-production-infrastructure-solo/)
— "which tool writes my backups" is a question I should have been able to
answer from memory and could not. So step one was reading the machine.

## What was actually there

Two rclone configs, which is the kind of detail that bites: the GUI config
that Backrest (the restic front end) uses, plus a **separate stale CLI
copy** that a cron job or a human shell had been using at some point. Change
one and you have changed half the system.

The migration itself is boring when done in the right order, and I did it
read-first:

```text
rclone.conf                     append [gdrive-*] + [gdrive-*-crypt]
rclone/rclone.conf (CLI copy)   same two remotes
Portainer stack 177             env → the new crypt remote; config :ro → :rw
restic init                     new repo on Google Drive
Backrest SetConfig              dsm-repo URI → the new location
```

Before touching the plan URIs I checked one property in the source, because
everything depended on it: a restic repository's identity is
`guid = sha256(id)` — a stable value derived from the repo id, **not** from
its URL. So repointing a plan at a new backend cannot silently fork the
history. All six plans came out byte-identical to the originals, and the
repo `id`/`guid` were untouched.

## Then the seed stopped moving, and the log named a stranger

The new repo seeded to 2.88 GB of roughly 1075 GB — 0.27% — and then slowed
to a crawl. The live log had the answer:

```text
'Read requests per minute' ... reason: RATE_LIMIT_EXCEEDED
consumer: projects/202264815644        <- not my project
quota_limit_value: 840000
```

`202264815644` is **rclone's own shared OAuth project**. My terabyte seed
was drawing on a per-minute read quota pooled with every rclone user on
earth, which is why it degraded from ~750 KB/s to nothing while my NAS, my
disk and my link were all idle. By the time I measured, the seed was fully
stalled — **0 bytes in 90 seconds**, 159 errors, still `rateLimitExceeded` —
and the six nightly plans had all failed with the same wall:

```text
Fatal: unable to save snapshot: unexpected HTTP response (500)
googleapi: Error 403: Quota exceeded for quota metric 'Queries'
  and limit 'Requests per minute' ...
```

"Slow" and "blocked" look similar in a progress bar and are completely
different problems. Ninety seconds of sampled zeros is what made the call.

## You need your own OAuth client — and the console is unavoidable

There is exactly one real fix: stop using the shared client and bring your
own. That means a Google Cloud project, an OAuth consent screen, and a
credential. Two settings cost real time:

- **Application type = Desktop app.** rclone's redirect is
  `http://127.0.0.1:53682`, and that client type permits it without extra
  configuration.
- **Publish the app.** Leave it in "Testing" and Google expires your refresh
  token **every 7 days** — which turns "working" into a silent weekly
  outage.

I checked whether I could shortcut the console with the CLI that was already
authenticated here. I could not, and the reason is worth knowing: `gcloud`
was signed in only as a **service account** in an unrelated project, and
service accounts cannot create consent screens or OAuth client IDs. The
credential has to belong to a project a human owns. It was a dead end, and I
verified that instead of asserting it.

## Three traps in the authorization flow

1. **Publishing an unverified app can hard-block a restricted scope.**
   `.../auth/drive` is restricted, and an app that is published *without*
   completing Google's verification review is sometimes blocked outright
   with no bypass. In my case the bypass existed — the "Google hasn't
   verified this app" screen, then **Advanced → Go to rclone-backup
   (unsafe)** — but that is a coin flip on scope and review state, not a
   guarantee, and "unsafe" is only a label meaning the app never went
   through a paid audit.
2. **The callback accepts exactly one request.** A reload, or a second tab,
   hits `127.0.0.1:53682` with no code and kills the attempt. My first try
   died exactly that way (`no code returned`), and it looks like a
   credentials problem. One tab. Don't reload it.
3. **Finish the consent screen on the machine that runs rclone.** The
   redirect goes to `localhost`, so a phone or another PC cannot complete it
   — the browser has to be the one on the same host as the process waiting
   on that port.

Then: read the token, write `client_id` + `client_secret` + `token` into
**both** configs (the GUI one and the stale CLI copy), and restore the two
flags that matter for a long unattended upload: `use_trash = false` and
`stop_on_upload_limit = true`.

## The old side: verify by reading, not by asking

Purging the old remote reported an error — `itemNotFound` — and the space
came back anyway:

```text
Used:   1.012 TiB  ->  3.800 GiB
Free:   ~0         ->  1.007 TiB
DSM folder: present -> gone      Trashed: 0 B
```

The error was rclone complaining *after* the work finished. Also worth
recording: the tool's documentation says personal accounts cannot
`permanentDelete`, which would have meant emptying the recycle bin by hand —
in practice the trash was already empty and roughly 1 TB was reclaimed with
no browser step. Documentation describes the API; the API is not always what
you get.

One more measured thing, because it changed my next decision: pausing the
file-sync container that shared that machine halved the load average — 16.70
→ 8.89 — which is the difference between "the network is slow" and "the box
is busy".

## What I would do differently

1. **Own the OAuth client from day one.** A shared client id is a queue you
   are standing in, with no status page and no one to call.
2. **Answer "what writes my backups" from a document, not from memory.** Two
   rclone configs is a normal amount of drift for a system nobody has drawn
   in a year.
3. **Read the destination, not the exit code.** The purge "failed" and freed
   a terabyte; a successful-looking rename can land nothing. Same lesson as
   [It Reported Success. Nothing Had Changed.](/posts/it-reported-success-nothing-had-changed/)
   — this estate keeps teaching it.
4. **Never start a multi-day upload while the last few percent of your
   attention is elsewhere.** Checking for `RATE_LIMIT_EXCEEDED` at 0.27%
   costs minutes; discovering it after two days costs the window.

## The result, honestly stated

The plumbing works: the plans live on Google Drive, the repo identity is
unchanged, the seed is now spending **my** quota instead of a shared one,
and the old side gave back about a terabyte. What it is not, yet, is a
backup: at the time I wrote this, the first seed had uploaded 2.88 GB of
~1075 GB and the new repo had **zero snapshots** — nothing is restorable
until the first snapshot lands. That is the number I watch, because "the
transfer is running" and "you can restore" are different claims.

文章 05 / 16：NocoDB SSO 是授权功能：环境变量不会告诉你的事

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "NocoDB SSO 是授权功能：环境变量不会告诉你的事"
description: "NocoDB 的 OIDC 环境变量是真的，启动时也会校验，所以自托管 SSO 看起来完全受支持。但在未授权的构建上，第一次登录尝试就会让整个实例崩溃。"
pubDate: 2026-09-29
category: notes
tags: ["nocodb", "sso", "oidc", "self-hosting", "licensing", "docker"]
ogImage: /og/zh-nocodb-sso-is-a-licensed-feature.jpg
banner: /banners/nocodb-sso-is-a-licensed-feature.jpg
draft: false
---

如果你自托管 NocoDB，去搜一下「nocodb sso self-hosted」，你会找到一篇介绍 OIDC 单点登录的文档页、一组环境变量，以及一堆论坛回复告诉你「设置好这些就能用」。这些在技术上都是真的。但没有一样会告诉你真正要紧的那部分。

所以直说了：**NocoDB 的 OIDC SSO 是付费功能。**在未授权的自托管构建上，代码路径是存在的，但它会直接抛弃进程——而且失败的方式也不是一句客气的「此实例未启用 SSO」。整个服务器直接退出。

## 为什么这件事重要

有两种人会在这里踩坑。

**如果你在生产环境启用它，你就给自己造好了一个拒绝服务的触发器。**处理 OIDC 回调的路由不需要认证就能访问——它必须如此，登录流程就是这样开始的。在一个这条路由会让进程崩溃的实例上，任何请求它的人都能把你的数据库界面打挂。不是变慢，不是返回错误页：而是直接干掉容器。你甚至不必成为攻击目标；一个好奇的访客，或者一个顺着 URL 爬的链接扫描器，就够了。

**如果你正在评估 NocoDB 这个平台，这是一个你该读对的授权信号。**功能在开源镜像里是有的，变量会被强制校验，代码也一并发布。缺的只是许可。一个*已实现*、*未授权*的功能，看起来和能正常工作的功能一模一样，直到你真的去试——而这正是整个陷阱所在。

## 文档说了什么，镜像又做了什么

NocoDB 自己的文档，只要你读到授权那一行，其实写得很清楚：*「OIDC SSO 在 NocoDB Cloud（Business 及以上版本）以及获得授权的自托管部署（Business 及以上版本）中可用。」*但另一半——未授权的二进制不会优雅降级——文档里没有写，从文档里也无从发现，因为这是关于镜像本身的运行时事实。

环境变量是真的。以下是我对照随镜像发布的内容确认过的：

```yaml
NC_SSO: oidc
NC_SSO_OIDC_ISSUER: https://auth.example.com/application/o/your-app/
NC_SSO_OIDC_AUTHORIZATION_URL: https://auth.example.com/application/o/authorize/
NC_SSO_OIDC_TOKEN_URL: https://auth.example.com/application/o/token/
NC_SSO_OIDC_USERINFO_URL: https://auth.example.com/application/o/userinfo/
NC_SSO_OIDC_CLIENT_ID: ...
NC_SSO_OIDC_CLIENT_SECRET: ...
NC_OIDC_PROVIDER_NAME: Hoelee SSO
```

公开文档里只出现了 `NC_SSO` 和 provider-name 变量。其余六个按 provider 区分的变量没有文档，但**会被强制校验**：用 `NC_SSO=oidc` 启动容器、却不给任何 URL，它直接拒绝启动：

```
Open ID SSO is enabled but missing required env keys
```

正是这个细节让陷阱显得可信。启动时的校验意味着这个功能是*接好线的*，不是残留代码。那条错误消息没有任何地方暗示：没有授权，这条代码路径就不可达——它读起来就像一个改改配置就能修好的错误。

## 怎么在零风险的情况下测试

千万别在你关心的实例上测试。一个用镜像内置 SQLite 元数据存储的一次性容器就够了，大约一分钟：

```bash
docker run -d --name nc-sso-test -p 10399:8080 \
  -e 'NC_SSO=oidc' \
  -e 'NC_SSO_OIDC_ISSUER=https://auth.example.com/application/o/your-app/' \
  -e 'NC_SSO_OIDC_AUTHORIZATION_URL=https://auth.example.com/application/o/authorize/' \
  -e 'NC_SSO_OIDC_TOKEN_URL=https://auth.example.com/application/o/token/' \
  -e 'NC_SSO_OIDC_USERINFO_URL=https://auth.example.com/application/o/userinfo/' \
  -e 'NC_SSO_OIDC_CLIENT_ID=test' \
  -e 'NC_SSO_OIDC_CLIENT_SECRET=test' \
  -e 'NC_OIDC_PROVIDER_NAME=Test SSO' \
  nocodb/nocodb:<your-tag>
```

注意这里刻意省略了 `NC_DB`——没有配置外部数据库，容器会自己拉起一个 SQLite 存储，所以它不需要任何凭据、不碰生产数据，一条 `docker rm -f nc-sso-test` 就能删掉。

观察启动过程，然后请求一次 SSO 路由：

```bash
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:10399/auth/oidc
docker ps -a --filter name=nc-sso-test
```

我得到的、两次都是——第二次我加了个 `?workspaceId=` 参数，排除请求格式不对的可能：

```
### UNCAUGHT EXCEPTION ###
unhandledRejection TypeError: _0x418c23 is not a function at docker/index.js:1:27483406
```

……然后容器就不再运行了。一次未认证的 GET，退出码 1。在 `2026.09.0` 上复现，两次结果一模一样。这不是配置错误，也不是客户端把请求形状搞错了。

注意启动日志里那句解释了一切的话：

```
No license key found — running in CE mode
```

## 授权故事的另一半

如果你的反应是「行，那我买授权」，在把它写进预算之前先读读这段，因为这里还有一个同样不明显的先决条件：

```
Instance ID unavailable — PostgreSQL is required for enterprise licensing
```

我跑的社区版用 **MySQL** 作为元数据存储，而 MySQL 无法生成授权所依赖的实例 ID。所以授权不是你在现有实例上激活一下就完事——它是授权*外加一次从 MySQL 到 Postgres 的元数据迁移*，这本身就是个自带风险的项目，还得在线上实例上执行。NocoDB 为此专门发布了迁移指南，光这一点就说明这种情况有多常见。

这不是在抱怨定价模式。这是我希望能提前拥有的评估：对任何跑默认 MySQL 配置的人来说，「直接买」的真实成本是授权**加上**一次存储引擎迁移。

## 搜索结果推荐的那个 fork——先确认它还活着

不买授权搜 NocoDB SSO，搜索结果会指向 `brunostjohn/nocodb-oidc`——一个给开源构建加上 OIDC 的社区 fork。它是即插即用的镜像，正是你想要的东西。但它也被弃养了，这一点大约两分钟就能查证：

```bash
# what version of NocoDB is the fork based on?
curl -s https://raw.githubusercontent.com/brunostjohn/nocodb-oidc/main/packages/nocodb/package.json \
  | python -c "import sys,json; print(json.load(sys.stdin)['version'])"
# 0.255.2

# when was it last touched?
curl -s 'https://api.github.com/repos/brunostjohn/nocodb-oidc/commits?per_page=1' \
  | python -c "import sys,json; print(json.load(sys.stdin)[0]['commit']['author']['date'])"
# 2024-10-29
```

一个 2024 年的版本，对上一个按月发版的实例，不是即插即用——那是倒退两年的降级，还会带走你的自动化已经依赖的 schema 迁移、安全修复和 API 行为。建议的好坏只取决于 fork 有没有人继续维护，而搜索结果是不会因为仓库弃养而过时的。

**在围绕它做规划之前，先查一下 fork 的基础版本和最后一次提交的日期。**两条 curl，就能免掉你原本要在凌晨一点做的回滚。

## 那该怎么办

1. **先想清楚你需要的是应用内的 SSO，还是应用前面的 SSO。**如果你要的是「互联网够不到这个登录表单」，那么在主机名前放一个 forward-auth 代理，用开源构建就能达成。它不是应用内的单点登录——应用仍然会要自己的密码。我把这个搭建方案，以及那个让我的第一次尝试看起来成功、实际上毫无作用的 ingress 陷阱，写在了[验证完美通过、实际并未生效的 Forward-Auth Gate](/posts/authentik-forward-auth-gate-wasnt-live/)里。
2. **保留应用自己的登录，在边缘把门锁上。**对只有几个用户的内部工具来说，这才是大多数人真正需要的。
3. **如果确实需要授权构建，把迁移也计入成本，而不只是授权本身。**先确认你用的是哪种元数据存储。

## 如果重来一次，我会怎么改

**当一个功能的可用性模糊不清时，从镜像里读答案，而不是看文档。**文档描述的是意图；镜像才是真相。一条 `docker exec ... printenv | grep -iE 'NC_|OIDC'`、一次对随镜像发布代码的 grep、一个一次性容器，二十分钟就回答了读一下午文档都答不出来的问题。

还有一条更直白、也适用于 NocoDB 之外的规则：**没有文档、却被强制校验的环境变量，是一条授权边界，而不是功能开关。**如果一个变量在没填完整时拒绝启动、而文档又藏起了它一半的兄弟变量，那么缺的那块几乎从来不是配置。

## 结果

在临时容器上五分钟复现两次，两次容器都死了。还有一个我宁可别在生产环境才学到的意外发现：对 SSO 路由的一次未认证请求，就足以让整个实例停下来。

好的一面是，我不再试图让应用自己去做 SSO，而是把决定放到了它该在的位置——主机名前面，现在也能用一句话回答「NocoDB 能免费做 SSO 吗？」，而不是花一下午。

---

## 想给业务用上这套？

如果你正在评估自托管软件、需要内部工具前面的单点登录，或者想让别人去核对你的技术栈实际做了什么、对照文档声称它做了什么——这正是我做的工作。

- **WhatsApp：** [011-797 2969](https://wa.me/60127972969) ——点一下就能聊
- **Email：** [[email protected]](mailto:[email protected]?subject=Self-hosted%20SSO%20enquiry)
- **Website：** [hoelee.com](https://www.hoelee.com)

我为马来西亚的小企业搭建 authentik 单点登录、自托管 Docker 技术栈和反向代理——并且把做这些事时学到的东西写下来。

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "NocoDB SSO Is a Licensed Feature — and the Env Vars Won't Tell You"
description: "NocoDB's OIDC environment variables are real and validated at startup, so self-hosted SSO looks supported. On an unlicensed build, the first login attempt crashes the whole instance."
pubDate: 2026-09-29
category: notes
tags: ["nocodb", "sso", "oidc", "self-hosting", "licensing", "docker"]
ogImage: /og/nocodb-sso-is-a-licensed-feature.jpg
banner: /banners/nocodb-sso-is-a-licensed-feature.jpg
draft: false
---

If you self-host NocoDB and search for "nocodb sso self-hosted", you'll find a documentation page describing OIDC single sign-on, a set of environment variables, and forum answers telling you it works if you set them. All of that is technically true. None of it tells you the part that matters.

So here it is: **NocoDB's OIDC SSO is a paid feature.** On an unlicensed self-hosted build, the code path exists but abandons the process — and the failure isn't a polite "SSO is not enabled on this instance". The whole server exits.

## Why this matters

Two audiences get bitten here.

**If you enable it in production, you've built a denial-of-service trigger.** The route that handles the OIDC callback is reachable without authentication — it has to be, that's how a login flow starts. On an instance where that route crashes the process, anyone who requests it takes your database UI down. Not degrades it, not returns an error page: kills the container. You don't need to be a target; a single curious visitor, or a link scanner that follows URLs, is enough.

**If you're evaluating NocoDB as a platform, this is a licensing signal you should read correctly.** The feature is present in the open-source image, the variables are enforced, the code ships. What's missing is permission. A feature that is *implemented* but *not licensed* looks identical to a feature that works, right up until you try it — and that's the whole trap.

## What the docs say, and what the image does

NocoDB's own docs are clear if you read the licensing line: *"OIDC SSO is available on NocoDB Cloud (Business plan and above) and licensed self-hosted deployments (Business plan and above)."* The other half — that the unlicensed binary will not degrade gracefully — isn't documented, and can't be discovered from the docs, because it's a runtime fact about the image.

The environment variables are real. The ones I confirmed against the shipped image:

```yaml
NC_SSO: oidc
NC_SSO_OIDC_ISSUER: https://auth.example.com/application/o/your-app/
NC_SSO_OIDC_AUTHORIZATION_URL: https://auth.example.com/application/o/authorize/
NC_SSO_OIDC_TOKEN_URL: https://auth.example.com/application/o/token/
NC_SSO_OIDC_USERINFO_URL: https://auth.example.com/application/o/userinfo/
NC_SSO_OIDC_CLIENT_ID: ...
NC_SSO_OIDC_CLIENT_SECRET: ...
NC_OIDC_PROVIDER_NAME: Hoelee SSO
```

Only `NC_SSO` and the provider-name variable appear in the public documentation. The six per-provider variables are undocumented but **enforced**: start the container with `NC_SSO=oidc` and no URLs and it refuses to boot:

```
Open ID SSO is enabled but missing required env keys
```

That's the detail that makes the trap convincing. Validation at startup means the feature is *wired in*, not vestigial. Nothing about that error message suggests the code path is unreachable without a licence — it reads like a configuration mistake you can fix.

## How to test it without risking anything

Never test this on the instance you care about. A throwaway container with an in-image SQLite meta store is enough, and it takes about a minute:

```bash
docker run -d --name nc-sso-test -p 10399:8080 \
  -e 'NC_SSO=oidc' \
  -e 'NC_SSO_OIDC_ISSUER=https://auth.example.com/application/o/your-app/' \
  -e 'NC_SSO_OIDC_AUTHORIZATION_URL=https://auth.example.com/application/o/authorize/' \
  -e 'NC_SSO_OIDC_TOKEN_URL=https://auth.example.com/application/o/token/' \
  -e 'NC_SSO_OIDC_USERINFO_URL=https://auth.example.com/application/o/userinfo/' \
  -e 'NC_SSO_OIDC_CLIENT_ID=test' \
  -e 'NC_SSO_OIDC_CLIENT_SECRET=test' \
  -e 'NC_OIDC_PROVIDER_NAME=Test SSO' \
  nocodb/nocodb:<your-tag>
```

Note the deliberate omission of `NC_DB` — with no external database configured, the container spins up its own SQLite store, so it needs no credentials, touches no production data, and can be deleted with `docker rm -f nc-sso-test`.

Watch the startup, then request the SSO route once:

```bash
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:10399/auth/oidc
docker ps -a --filter name=nc-sso-test
```

What I got, twice — the second time with a `?workspaceId=` parameter to rule out a bad request shape:

```
### UNCAUGHT EXCEPTION ###
unhandledRejection TypeError: _0x418c23 is not a function at docker/index.js:1:27483406
```

...and the container is no longer running. Exit code 1, from an unauthenticated GET. Reproduced on `2026.09.0` with the same result both times. It isn't a configuration error, and it isn't a request the client got wrong.

Note the log line that explains it, from startup:

```
No license key found — running in CE mode
```

## The other half of the licensing story

If your reaction is "fine, I'll buy the licence", read this before you budget for it, because there's a prerequisite that isn't obvious either:

```
Instance ID unavailable — PostgreSQL is required for enterprise licensing
```

The community edition I run uses **MySQL** as its metadata store, and MySQL cannot generate the instance ID that licensing depends on. So a licence isn't something you activate on the instance you have — it's a licence *plus a metadata migration from MySQL to Postgres*, which is its own project with its own risk, done on a live instance. NocoDB publishes a migration guide for exactly this, which tells you how common the situation is.

That isn't a complaint about the pricing model. It's the evaluation I wish I'd had before spending an afternoon: the cost of "just buy it" is licence **plus** a storage-engine migration for anyone running the default MySQL setup.

## The fork that search results recommend — check it's alive first

Search results for NocoDB SSO without a licence point at `brunostjohn/nocodb-oidc`, a community fork that adds OIDC to the open-source build. It's a drop-in image and it's exactly the thing you want. It's also abandoned, and this is checkable in about two minutes:

```bash
# what version of NocoDB is the fork based on?
curl -s https://raw.githubusercontent.com/brunostjohn/nocodb-oidc/main/packages/nocodb/package.json \
  | python -c "import sys,json; print(json.load(sys.stdin)['version'])"
# 0.255.2

# when was it last touched?
curl -s 'https://api.github.com/repos/brunostjohn/nocodb-oidc/commits?per_page=1' \
  | python -c "import sys,json; print(json.load(sys.stdin)[0]['commit']['author']['date'])"
# 2024-10-29
```

A version from 2024 against an instance on a monthly release cadence is not a drop-in — it's a two-year downgrade, taking with it the schema migrations, the security fixes and the API behaviour your automations already depend on. The advice is only as good as the fork is maintained, and search results don't go stale when a repo does.

**Check the fork's base version and last commit date before you plan around it.** Two curl calls, and they replace a rollback you'd otherwise do at 1 a.m.

## What to do instead

1. **Decide whether you need SSO in the app, or in front of it.** If what you want is "the internet can't reach this login form", a forward-auth proxy in front of the hostname gets you there with the open-source build. It's not single sign-on in the app, and the app still asks for its own password — I wrote up that construction, and the ingress trap that made my first attempt look successful while being a no-op, in [The Forward-Auth Gate That Verified Perfectly — and Wasn't Live](/posts/authentik-forward-auth-gate-wasnt-live/).
2. **Keep the app's own login and lock the door at the edge.** For an internal tool with a handful of users, this is what most people actually need.
3. **If a licensed build is genuinely required, price the migration, not just the licence.** Confirm which metadata store you're on first.

## What I'd do differently

**When a feature's availability is ambiguous, read it out of the image instead of the docs.** The docs describe intent; the image is the truth. One `docker exec ... printenv | grep -iE 'NC_|OIDC'`, one grep of the shipped bundle, and one throwaway container answered in twenty minutes what an afternoon of documentation reading couldn't.

And a blunter rule that generalises past NocoDB: **an undocumented-but-enforced environment variable is a licensing boundary, not a feature flag.** If a variable rejects your startup when it's incomplete, but the documentation hides half its siblings, the missing piece is almost never configuration.

## The result

Five minutes to reproduce on a throwaway container, twice, with the container dead both times. One accidental finding I'd rather not have learned in production: an unauthenticated request to the SSO route is enough to stop the instance.

The upside is that I stopped trying to make the app do SSO, put the decision in front of the hostname where it belongs, and can now answer "can NocoDB do SSO for free?" in one sentence instead of an afternoon.

---

## Want this for your business?

If you're evaluating self-hosted software, need single sign-on in front of internal tools, or want someone to check what your stack actually does versus what its documentation claims, that's the work I do.

- **WhatsApp:** [011-797 2969](https://wa.me/60127972969) — tap to chat
- **Email:** [[email protected]](mailto:[email protected]?subject=Self-hosted%20SSO%20enquiry)
- **Website:** [hoelee.com](https://www.hoelee.com)

I set up authentik single sign-on, self-hosted Docker stacks and reverse proxies for small businesses in Malaysia — and I write up what I learn while doing it.

文章 06 / 16：与其看供应商文档，不如直接读容器自带的 API 文档

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "与其看供应商文档，不如直接读容器自带的 API 文档"
description: "当 API 文档藏在登录墙后面、甚至根本不存在时，镜像本身就是唯一的事实来源：四条 Docker 命令，就能挖出 API 规范、配置契约，以及它真正监听的端口。"
pubDate: 2025-10-01
updatedDate: 2026-09-26
category: notes
tags: [docker, openapi, api, documentation, debugging, reverse-engineering]
ogImage: /og/zh-reading-a-containers-own-api-docs.jpg
banner: /banners/reading-a-containers-own-api-docs.jpg
draft: false
---

我需要对接一个供应商自托管的 API。他们的文档是有的，但挡在账号登录后面；
而我能看到的那一页，只描述了 API *能*做什么，没说该怎么配置。与此同时，
容器就安安静静地躺在我机器上——而答案就在它里面。

现在，凡是文档单薄、要登录、或者压根没有的镜像，我都会先跑这套命令。
下面所有操作都是只读的，也不需要进入容器开 shell。

## 为什么这很重要

1. **供应商文档描述的是理想路径；容器描述的才是契约。** 确切的 env 变量名、
   默认值、暴露的端口、是否需要配置文件——这些在营销页面上都不可靠。
2. **很多镜像压根没有文档**——私有仓库、内部构建，或是别人拉了又推上去的
   社区镜像。构件本身是唯一权威。
3. **版本漂移是真实存在的。** 网站上的文档描述的是最新版本；而你正在跑的
   镜像可能是两年前的了。读构件，了解的是你实际部署的那个东西。

## 1. 不运行镜像，直接读里面的文件

最有用的一招：覆盖 entrypoint，把镜像当成文件系统来用。

```bash
# what's in there?
docker run --rm --entrypoint ls <image> -la /app

# does it ship an API spec?
docker run --rm --entrypoint cat <image> /app/public/openapi.yml | head -40
```

第二条命令直接在镜像里返回了一份完整的 OpenAPI 3.1 规范，27 KB——每个端点、
请求体 schema、认证模型，一应俱全。不用登录、不用账号、没有文档墙。

如果你还不知道文件布局，用 `--entrypoint find` 能覆盖更广：

```bash
docker run --rm --entrypoint find <image> / -maxdepth 3 \
  -name '*.yml' -o -name '*.yaml' -o -name '*.json' 2>/dev/null | head -30
```

## 2. 从 bundle 里挖出配置契约

真正会让你卡住的往往是 env 变量名——一个没有默认值的必需变量，意味着容器
启动时就挂掉，通常还带一条毫无帮助的错误信息。直接问源码吧：

```bash
docker run --rm --entrypoint grep <image> \
  -rhoE 'process\.env\.[A-Za-z_][A-Za-z0-9_]*' /app/dist | sort -u
```

```
AUTH_SECRET
DEBUG
HOST
LOKI_HOST
METRICS_TOKEN
PORT
USER_TIMEOUT
WORKER_TIMEOUT
WORKER_PATH
```

九个变量名，整个配置面就清楚了。想看默认值，用带上下文的 grep 抓你关心的
那一个：

```bash
docker run --rm --entrypoint grep <image> \
  -ohE '.{0,80}process\.env\.PORT.{0,80}' /app/dist/index.js
```

```js
serverPort: parseInt(process.env.PORT) || 3e3,
serverHost: process.env.HOST || "0.0.0.0",
workerTimeout: parseInt(process.env.WORKER_TIMEOUT) || 600*1e3,
```

`PORT` 默认是 3000，host 默认是 `0.0.0.0`，worker 每十分钟回收一次。一条命令
就消掉了三个不确定的决策点。这套办法适用于任何 Node/Python 镜像；对 Go 或
Rust 的二进制，同样的思路，只是把 grep 换成对二进制跑 `strings`。

## 3. 不拉镜像，直接查镜像配置

在慢速网络下，或者在决定下载 1 GB 镜像之前，你可以直接从 registry 读取镜像
的元数据。本地连 Docker daemon 都不需要：

```bash
REPO=synology/spreadsheet-api
TAG=3.4.1

TOKEN=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:$REPO:pull" \
  | python -c "import json,sys; print(json.load(sys.stdin)['token'])")

# manifest (follow the platform entry if it's a multi-arch index)
curl -s -H "Authorization: Bearer ***" \
  -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
  "https://registry-1.docker.io/v2/$REPO/manifests/$TAG" \
  | python -c "import json,sys; m=json.load(sys.stdin); print(m['config']['digest'])"
```

然后取回那个 config blob，把关心的字段打印出来：

```bash
curl -s -H "Authorization: Bearer ***" \
  "https://registry-1.docker.io/v2/$REPO/blobs/<config-digest>" \
  | python -c "
import json,sys
c = json.load(sys.stdin)['config']
print('Env:       ', c.get('Env'))
print('Entrypoint:', c.get('Entrypoint'))
print('Cmd:       ', c.get('Cmd'))
print('Workdir:   ', c.get('WorkingDir'))
print('Ports:     ', list((c.get('ExposedPorts') or {}).keys()))"
```

```
Env:        ['PATH=...', 'NODE_VERSION=22.18.0', 'WORKER_PATH=/app/dist/spreadsheet_worker.js']
Entrypoint: ['docker-entrypoint.sh']
Cmd:        ['node', 'dist/index.js']
Workdir:    /app
Ports:      []
```

还没下载一个字节，就有两条有用信息跳出来了：这是个 Node 服务，而且它**没有
声明任何暴露端口**——所以任何端口映射都得靠 `PORT` 变量，而不是 `EXPOSE`。
既然来了，顺带也值得查一下：tag 列表会告诉你真实的版本历史。

```bash
curl -s "https://hub.docker.com/v2/repositories/$REPO/tags/?page_size=25" \
  | python -c "import json,sys; [print(t['name'], t['last_updated'][:10]) for t in json.load(sys.stdin)['results']]"
```

## 4. 找出它真正监听的端口

`EXPOSE` 是文档，不是行为。想知道进程真正绑定的端口，要看运行中的容器内部：

```bash
docker exec <container> cat /proc/net/tcp
```

端口以十六进制写在字段 2 里——字段 4 是 `0A` 表示 `LISTEN`。解码一下：

```bash
docker exec <container> sh -c \
  "awk 'NR>1 && \$4==\"0A\" {print \$2}' /proc/net/tcp" \
  | cut -d: -f2 | while read h; do printf '%d\n' "0x$h"; done
```

```
3000
```

然后不离开容器，确认它真的在响应：

```bash
docker exec <container> sh -c 'wget -qSO- -O- http://127.0.0.1:3000/ 2>&1 | head -5'
```

```
HTTP/1.1 302 Found
  location: /docs
```

重定向到 `/docs`——原来镜像一直在提供它自己的 Swagger UI。

## 5. 读启动日志，别把 shell 挂死

把 `docker run` 直接接到 `head` 或日志过滤器上，看起来无害，实际上会挂死：
进程一直占着管道，你的命令永远不会返回。改成后台运行，读日志，然后删掉它：

```bash
docker run -d --name probe -e AUTH_SECRET=probe-only <image>
sleep 8
docker logs probe 2>&1 | head -20
docker rm -f probe
```

```
[02:06:17 UTC] INFO: Server listening at http://127.0.0.1:3000
[02:06:17 UTC] INFO: Server listening at http://172.27.0.2:3000
```

两行日志就是全部答案——而且它还告诉我：容器*有* `AUTH_SECRET` 就能正常启动，
*没有*就挂。这比我在前台运行、眼睁睁看着它崩溃后拿到的那段压缩过的堆栈信息
有用得多。

## 这不能替代什么

如果供应商自己的文档存在，还是要读——通常比钻洞翻找更快，而且它携带镜像
无法告诉你的东西，比如版本兼容性对照表。就拿这个镜像来说，供应商页面*确实*
发布了运行命令和必需的 `AUTH_SECRET`；我之所以先在镜像里找到它们，只是因为我
没把页面往下滚够。两个都用：页面看意图，构件看你要部署的确切契约。

## 结果

四条命令——`cat` 出打包的规范、`grep` 出 env 变量、从 registry 读 config blob、
`exec cat /proc/net/tcp` 找端口——把一场文档大海捞针，换成了：一份完整的
OpenAPI 规范、全部九个 env 变量、默认端口、配置默认值，以及一个 Swagger UI
地址。没进容器开 shell，不用登录，什么都没改动。

这些命令最值的一次应用，就是 Synology 的 spreadsheet API 容器——完整案例、以及逼我动用这些命令的
四个坑，写在[《Synology 的 Spreadsheet API 是一个容器，不是一个 API 端点》](/posts/synology-spreadsheet-api-is-a-container/)里。

---

*如果你正在对接一个自托管系统，而文档又戛然而止——这正是我为马来西亚中小
企业做的活：[WhatsApp](https://wa.me/60127972969)、[邮件](mailto:[email protected]?subject=API%20integration)，
或 [hoelee.com](https://hoelee.com)。*

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "Read a Container's Own API Docs Instead of the Vendor's"
description: "When the API documentation is behind a login or doesn't exist, the image is still the source of truth: four Docker commands that extract the API spec, the config contract, and the port it listens on."
pubDate: 2025-10-01
updatedDate: 2026-09-26
category: notes
tags: [docker, openapi, api, documentation, debugging, reverse-engineering]
ogImage: /og/reading-a-containers-own-api-docs.jpg
banner: /banners/reading-a-containers-own-api-docs.jpg
draft: false
---

I needed to drive a vendor's self-hosted API. Their docs existed, but behind an account login, and the page I could reach described what the API *could* do rather than how to configure it. The container, meanwhile, was sitting right there on my machine — and it contained the answers.

This is the set of commands I now run first for any image whose documentation is thin, gated, or missing. Everything here is read-only and needs no shell inside the container.

## Why this matters

1. **Vendor docs describe the happy path; the container describes the contract.** Exact env var names, defaults, exposed ports, whether a config file is required — none of that is reliably in a marketing page.
2. **Many images have no docs at all** — a private registry, an internal build, or a community image someone pulled and pushed. The artifact is the only authority.
3. **Version drift is real.** Docs on a website describe the latest release; the image you're running might be two years old. Reading the artifact tells you about the thing you actually deployed.

## 1. Read files out of the image without running it

The single most useful trick: override the entrypoint and treat the image as a filesystem.

```bash
# what's in there?
docker run --rm --entrypoint ls <image> -la /app

# does it ship an API spec?
docker run --rm --entrypoint cat <image> /app/public/openapi.yml | head -40
```

That second command returned a full OpenAPI 3.1 specification, 27 KB, inside the image — complete with every endpoint, request body schema, and the authentication model. No login, no account, no gated docs.

If you don't know the file layout yet, `--entrypoint find` covers more ground:

```bash
docker run --rm --entrypoint find <image> / -maxdepth 3 \
  -name '*.yml' -o -name '*.yaml' -o -name '*.json' 2>/dev/null | head -30
```

## 2. Extract the configuration contract from the bundle

Env var names are the part that will actually stop you — a required variable with no default means the container dies at startup, usually with an unhelpful error. Ask the source directly:

```bash
docker run --rm --entrypoint grep <image> \
  -rhoE 'process\.env\.[A-Za-z_][A-Za-z0-9_]*' /app/dist | sort -u
```

```
AUTH_SECRET
DEBUG
HOST
LOKI_HOST
METRICS_TOKEN
PORT
USER_TIMEOUT
WORKER_TIMEOUT
WORKER_PATH
```

Nine names, and now you know the whole config surface. For a default value, grep with context around the one you care about:

```bash
docker run --rm --entrypoint grep <image> \
  -ohE '.{0,80}process\.env\.PORT.{0,80}' /app/dist/index.js
```

```js
serverPort: parseInt(process.env.PORT) || 3e3,
serverHost: process.env.HOST || "0.0.0.0",
workerTimeout: parseInt(process.env.WORKER_TIMEOUT) || 600*1e3,
```

`PORT` defaults to 3000, host to `0.0.0.0`, workers recycle after ten minutes. That's three uncertain decisions removed in one command. This works for any Node/Python image; for a Go or Rust binary the same idea applies with `strings` on the binary instead.

## 3. Inspect the image config without pulling it

On a slow link, or before you commit to a 1 GB download, you can read the image's metadata straight from the registry. No local Docker daemon needed:

```bash
REPO=synology/spreadsheet-api
TAG=3.4.1

TOKEN=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:$REPO:pull" \
  | python -c "import json,sys; print(json.load(sys.stdin)['token'])")

# manifest (follow the platform entry if it's a multi-arch index)
curl -s -H "Authorization: Bearer $TOKEN" \
  -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
  "https://registry-1.docker.io/v2/$REPO/manifests/$TAG" \
  | python -c "import json,sys; m=json.load(sys.stdin); print(m['config']['digest'])"
```

Then fetch that config blob and print the interesting fields:

```bash
curl -s -H "Authorization: Bearer $TOKEN" \
  "https://registry-1.docker.io/v2/$REPO/blobs/<config-digest>" \
  | python -c "
import json,sys
c = json.load(sys.stdin)['config']
print('Env:       ', c.get('Env'))
print('Entrypoint:', c.get('Entrypoint'))
print('Cmd:       ', c.get('Cmd'))
print('Workdir:   ', c.get('WorkingDir'))
print('Ports:     ', list((c.get('ExposedPorts') or {}).keys()))"
```

```
Env:        ['PATH=...', 'NODE_VERSION=22.18.0', 'WORKER_PATH=/app/dist/spreadsheet_worker.js']
Entrypoint: ['docker-entrypoint.sh']
Cmd:        ['node', 'dist/index.js']
Workdir:    /app
Ports:      []
```

Two useful facts fell out of that before downloading a byte: it's a Node service, and it declares **no exposed ports** — so any port mapping has to come from the `PORT` variable, not from `EXPOSE`. Also worth checking while you're there: the tags list tells you the real version history.

```bash
curl -s "https://hub.docker.com/v2/repositories/$REPO/tags/?page_size=25" \
  | python -c "import json,sys; [print(t['name'], t['last_updated'][:10]) for t in json.load(sys.stdin)['results']]"
```

## 4. Find the port it actually listens on

`EXPOSE` is documentation, not behaviour. To see what the process really bound to, look inside a running container:

```bash
docker exec <container> cat /proc/net/tcp
```

The port is in hex in field 2 — `0A` in field 4 means `LISTEN`. Decode it:

```bash
docker exec <container> sh -c \
  "awk 'NR>1 && \$4==\"0A\" {print \$2}' /proc/net/tcp" \
  | cut -d: -f2 | while read h; do printf '%d\n' "0x$h"; done
```

```
3000
```

Then confirm it answers, without leaving the container:

```bash
docker exec <container> sh -c 'wget -qSO- -O- http://127.0.0.1:3000/ 2>&1 | head -5'
```

```
HTTP/1.1 302 Found
  location: /docs
```

A redirect to `/docs` — the image was serving its own Swagger UI the whole time.

## 5. Read startup logs without hanging your shell

Piping `docker run` straight into `head` or a log filter looks harmless and will hang: the process keeps the pipe open, so your command never returns. Run it detached, read the logs, then remove it:

```bash
docker run -d --name probe -e AUTH_SECRET=probe-only <image>
sleep 8
docker logs probe 2>&1 | head -20
docker rm -f probe
```

```
[02:06:17 UTC] INFO: Server listening at http://127.0.0.1:3000
[02:06:17 UTC] INFO: Server listening at http://172.27.0.2:3000
```

That's the whole answer in two lines — and it also told me the container starts fine *with* `AUTH_SECRET` and dies *without* it, which is a far better signal than the minified stack trace I got from running it in the foreground and watching it crash.

## What this doesn't replace

Read the vendor's own documentation when it exists — it's usually faster than spelunking, and it carries things an image can't tell you, like version compatibility tables. On this particular image, the vendor's page *did* publish the run command and the required `AUTH_SECRET`; I found them in the image first only because I hadn't scrolled far enough down the page. Use both: the page for intent, the artifact for the exact contract you're going to deploy.

## The result

Four commands — `cat` the bundled spec, `grep` the bundle for env vars, read the config blob from the registry, `exec cat /proc/net/tcp` for the port — replaced a documentation hunt with: a full OpenAPI spec, all nine env vars, the default port, the config defaults, and a Swagger UI URL. No shell in the container, no login, nothing mutated.

The place this paid off hardest was Synology's spreadsheet API container — the worked example, including the four failures that made me reach for these commands in the first place, is in [The Synology Spreadsheet API Is a Container, Not an API Endpoint](/posts/synology-spreadsheet-api-is-a-container/).

---

*If you're integrating a self-hosted system and the docs stop short, I do that work for small businesses in Malaysia — [WhatsApp](https://wa.me/60127972969), [email](mailto:[email protected]?subject=API%20integration), or [hoelee.com](https://hoelee.com).*

文章 07 / 16：smartctl 退出码 32：专门跳过你最该看的硬盘的那个“错误”

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "smartctl 退出码 32：专门跳过你最该看的硬盘的那个“错误”"
description: "把 smartctl 的任何非零退出码当成读取失败的监控脚本，恰好会跳过属性已经逼近阈值的那些盘。退出码 32 不是错误，而是一段历史。"
pubDate: 2026-04-14
updatedDate: 2026-09-29
category: notes
tags: [smart, smartctl, unraid, monitoring, bash, disks, self-hosting]
ogImage: /og/zh-smartctl-exit-code-32-skips-the-disks-that-matter.jpg
banner: /banners/smartctl-exit-code-32-skips-the-disks-that-matter.jpg
draft: false
---

我的硬盘健康采集脚本连续几周都是\"全绿\"。每块盘都有温度、通电小时数和 SMART 结论，指标看起来完整——直到我数了一下只有 **4 块 SSD 里的 2 块**。

不是故障，是**消失了**。脚本认定这两块盘不存在。

## 问题出在把退出码当布尔值

采集脚本遍历块设备、对每块盘跑 `smartctl`、再写 Prometheus textfile 指标。错误处理看起来挺防御：

```bash
for dev in /dev/sd?; do
  if ! smartctl -A -H "$dev" > /tmp/smart.out 2>&1; then
    continue          # "盘不支持 SMART / 读不到"
  fi
  # 解析并输出指标
done
```

`if ! cmd` 是布尔判断，但 `smartctl` 的退出状态**不是布尔值，是一个位域**。把位域当真假用，就是这次翻车的原因。

## 退出码 32 的真实含义

`man smartctl` 里这些位是独立累加的：

| 位 | 值 | 含义 |
|---|---|---|
| 0 | 1 | 命令行解析失败 |
| 1 | 2 | 设备打开失败 / 无 IDENTIFY DEVICE 结构 |
| 2 | 4 | SMART 或 ATA 命令失败、校验和错误 |
| 3 | **8** | SMART 状态为 **DISK FAILING** |
| 4 | **16** | 有预失效属性 **<= 阈值** |
| 5 | **32** | SMART 状态 **OK**，但某些属性**曾经**低于阈值 |
| 6 | 64 | 设备错误日志中有记录 |
| 7 | 128 | 自检日志中有失败记录 |

所以 `32` 的意思和\"读不到盘\"正好相反：**现在没问题，但它曾经踩过阈值。** 这正是应该持续盯着的那类盘——而我的脚本把它们丢掉了。

消失的两块盘，恰好是原始属性长期处在边缘值的那两块；退出码为 `0` 的\"健康盘\"被正常采集。**这个过滤器实际上筛选出了\"没有任何历史可报告\"的盘。**

## 修法：把\"信息位\"掩掉

对监控来说，位 32 和 64 是信息；位 8 和 16 才是该告警的。掩掉信息位，只对剩下的做判断：

```bash
smartctl -A -H -d sat "$dev" > /tmp/smart.out 2>&1
rc=$?

# fatal bits: 1 (parse), 2 (open), 4 (command/checksum), 8 (FAILING)
# informational bits: 32 (was below threshold in the past), 64 (error log has records)
fatal=$(( rc & ~(32 | 64) ))
if [ "$fatal" -ne 0 ]; then
  echo "device $dev unreadable or failing (rc=$rc)" >&2
  continue
fi

# emit the verdict AND the exit code, so the code itself is a metric
echo "disk_smart_exit_code{device=\"$dev\"} $rc"
echo "disk_smart_health{device=\"$dev\"} $(( rc & 8 ? 0 : 1 ))"
```

三点改变：

1. **退出码变成数据，而不是控制流。** 它作为指标被导出，磁盘从 `0` → `32` → `64` 的漂移会变成趋势，而不是一次静默跳过。
2. **信息位不再致命。** 有历史的盘被采集，这才是监控它们的目的。
3. **NAS 上 `-d sat` 很关键。** 在 Synology DSM（以及某些 USB 桥接）上，设备类型不对时 `smartctl` 拿不到有用输出——症状同样是\"盘消失\"，原因却不同。

## 我下次会怎么做

- **面对有\"退出码位域\"文档的工具，永远不要写 `if ! cmd`。** 先把手册里的退出码那节读完。
- **除了\"采集器有没有跑\"，还要数\"采到了几块盘\"。** 我的告警是\"采集器过期\"，真正的 bug 是\"采集器正常跑了，只报了 50% 的盘\"。一个 `disk_count` 指标就能立刻暴露。
- **把\"这个设备没有数据\"本身当成可告警状态。** 缺失的序列是不可见的，所以它存活了好几周。

## 结果

采集器从 2 块可用盘变成 4 块，每块都报温度、通电小时数、SMART 状态和原始退出码——共 122 个 textfile 指标，包括我真正想要的 btrfs 错误计数。重新出现的那两块，正是长期处在边缘属性值的盘。

一个会跳过自己最坏信号的监控流水线，比没有监控更糟——因为它一直告诉你一切正常。

## 需要为你的业务做这个吗？

如果你在跑 NAS 或服务器机架，想让硬盘健康真正\"会通知你\"——SMART 属性、温度、btrfs/RAID 错误计数，并推到 Telegram 或邮件——这类自托管监控流水线我可以帮你搭。

**WhatsApp：[+60 12-797 2969](https://wa.me/60127972969)** · **邮箱：[[email protected]](mailto:[email protected]?subject=Disk%20health%20monitoring)** · **[hoelee.com](https://hoelee.com)**

网站设计与开发是我的主业；服务器加固与自托管基础设施是它的另一半。

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "smartctl Exit Code 32: The Code That Skips the Disks You Care About"
description: "A disk-health collector that treats any non-zero smartctl exit as failure quietly skips exactly the drives with marginal attributes. Exit 32 is not an error — it is a history lesson."
pubDate: 2026-04-14
updatedDate: 2026-09-29
category: notes
tags: [smart, smartctl, unraid, monitoring, bash, disks, self-hosting]
ogImage: /og/smartctl-exit-code-32-skips-the-disks-that-matter.jpg
banner: /banners/smartctl-exit-code-32-skips-the-disks-that-matter.jpg
draft: false
---

My disk-health collector was green for weeks. Every drive reported a temperature, a
power-on-hours counter and a SMART verdict, and the textfile metrics looked complete
until I counted them: **two of the four SSDs were simply missing**.

Not failing. Missing. The script had decided they didn't exist.

## The script that "handled" errors

The collector walks the block devices, runs `smartctl` against each one, and writes
Prometheus textfile metrics. The error handling looked defensive — the classic shell
shape:

```bash
for dev in /dev/sd?; do
  if ! smartctl -A -H "$dev" > /tmp/smart.out 2>&1; then
    continue          # "disk isn't SMART-capable / can't be read"
  fi
  # parse and emit metrics
done
```

`if ! cmd` is a boolean. `smartctl`'s exit status is **not** a boolean — it's a
bitfield, and treating a bitfield as true/false is where this goes wrong.

## What exit code 32 actually means

From `man smartctl`, the bits are cumulative and independent:

| Bit | Value | Meaning |
|---|---|---|
| 0 | 1 | Command line did not parse |
| 1 | 2 | Device open failed, or no IDENTIFY DEVICE structure |
| 2 | 4 | A SMART or ATA command failed / checksum error in a SMART structure |
| 3 | **8** | SMART status check returned **DISK FAILING** |
| 4 | **16** | Pre-fail attributes found **<= threshold** |
| 5 | **32** | SMART status **OK**, but some attributes were **<= threshold at some time in the past** |
| 6 | 64 | Device error log contains records of errors |
| 7 | 128 | Device self-test log contains records of errors |

So exit `32` is the opposite of "unreadable". It means: *the disk is fine right now,
and it has been below a threshold before.* That is precisely the signal you want to
keep an eye on — and my collector was throwing it away.

The two disks that vanished were the two whose raw attributes sit at marginal values.
The healthy disks exited `0` and got collected. **The filter was selecting for the
disks with nothing to report.**

## The fix: mask the informational bits

Exit bits 32 and 64 are informational for monitoring purposes; bits 8 and 16 are the
ones that deserve an alert. Mask them off and act on what's left:

```bash
smartctl -A -H -d sat "$dev" > /tmp/smart.out 2>&1
rc=$?

# fatal bits: 1 (parse), 2 (open), 4 (command/checksum), 8 (FAILING)
# informational bits: 32 (was below threshold in the past), 64 (error log has records)
fatal=$(( rc & ~(32 | 64) ))
if [ "$fatal" -ne 0 ]; then
  echo "device $dev unreadable or failing (rc=$rc)" >&2
  continue
fi

# emit the verdict AND the exit code, so the code itself is a metric
echo "disk_smart_exit_code{device=\"$dev\"} $rc"
echo "disk_smart_health{device=\"$dev\"} $(( rc & 8 ? 0 : 1 ))"
```

Three things changed the value of this collector:

1. **The exit code became data, not control flow.** It's exported as a metric, so a
   disk drifting from `0` → `32` → `64` shows up as a trend instead of a silent skip.
2. **The informational bits stopped being fatal.** Disks with history are collected,
   which is the whole point of monitoring them.
3. **`-d sat` matters on a NAS.** On Synology DSM (and some USB bridges), a SATA disk
   behind the wrong device type returns nothing useful — the same "missing disk"
   symptom with a different cause.

## What I'd do differently

- **Never write `if ! cmd` against a tool that documents an exit-code bitfield.** Read
  the exit-code section of the man page before using the return value as a boolean.
- **Count what you collected, not just whether the collector ran.** My alert was on
  "collector stale"; the real bug was "collector ran fine and reported 50% of the
  disks". A single `disk_count` metric would have surfaced it immediately.
- **Treat "no data for this device" as its own alertable state.** A missing series is
  invisible, which is exactly why it survived for weeks.

## The result

The collector went from 2 usable disks to 4, with every drive reporting temperature,
power-on hours, SMART status and its raw exit code — 122 textfile metrics in total,
including the btrfs error counters I actually wanted. The two disks that reappeared
are the two that were already sitting at marginal attribute values.

A monitoring pipeline that skips its own worst signals is worse than no pipeline,
because it tells you everything is fine.

## Want this for your business?

If you're running a NAS or a server rack and want disk health that actually pages you
before a drive dies — SMART attributes, temperatures, btrfs/RAID error counters, and
alerts to Telegram or email — that's the kind of self-hosted monitoring pipeline I set up.

**WhatsApp: [+60 12-797 2969](https://wa.me/60127972969)** · **Email: [[email protected]](mailto:[email protected]?subject=Disk%20health%20monitoring)** · **[hoelee.com](https://hoelee.com)**

Website design and development is my main line of work; server hardening and
self-hosted infrastructure is the other half of it.

文章 08 / 16：确定性的活，别再用 Agent Prompt 去跑

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "确定性的活，别再用 Agent Prompt 去跑"
description: "我有一个定时任务连续失败 29 次，原因是我在一个只需要字符串比较的循环里塞了个模型。这是我现在用的规则。"
pubDate: 2026-10-01
category: notes
blueprint: ai-agent-automation
blueprintLayer: L1-deterministic-orchestration
tags: ["loop-engineering", "cron", "ai-agents", "automation"]
ogImage: "/og/zh-stop-writing-agent-prompts-for-deterministic-work.jpg"
banner: "/banners/stop-writing-agent-prompts-for-deterministic-work.jpg"
draft: false
---

我有一个定时任务连续失败了 29 次。没人告诉我，因为它被设计成只在有东西可报的时候才开口——而一个还没来得及开口就崩掉的运行，看起来和一个「什么都没发现」的运行一模一样。

那个任务是个版本检查：问 npm 某个我常用的 CLI 最新发布版本是多少，和本机装的版本比一下，有更新就告诉我。整个任务就这些。它就是两个字符串加一次比较。

我当初把它做成了 agent 任务，因为那阵子我脑子里的默认模式就是这个。每次 tick，模型读一遍注入的脚本输出，然后判断要不要写一条通知。它正常跑了一阵子。后来我改了全局默认模型，而这个任务会记录自己创建时的模型，于是它干脆整个不跑了，打印一行 `[drift_skip]` 就退出。连续 29 次。

修法不是写一个更好的 prompt，而是把模型从循环里删掉。

## 规则

**如果停止条件是一次比较——一个字符串、一个数字、一个状态码——那么循环里的模型只能带来新的失败模式。**

「`3.8.50` 是不是比 `3.8.50` 新」这件事里没有任何判断可做。模型并没有在决定代码决定不了的事；它带来的是一条依赖（一条模型路由、一个提供商、一份配置快照、一张 token 账单）和一种全新的出错方式。所以我把它重写成 125 行 Python，每条分支都有明确出口，再把任务切成脚本模式：脚本的 stdout 直接作为通知内容，整条路径上没有任何 LLM。

从此一路干净。Anthropic 自己的建议也是一句话——*确定性的活用脚本，跑脚本比让模型去推理步骤便宜*——但对我来说成本从来不是重点，可靠性才是。

## 模型该用在哪

我跑的 11 个循环里，现在有 8 个整条路径上没有模型。用模型的那 3 个，处理的都是我确实写不成代码的东西：

- **这封邮件重不重要？** 一个邮箱里同时有促销、DMARC 报告、容器告警，和一位来问报价的客户。抓邮件是确定性的，给它排优先级不是。
- **这张店铺优惠券是不是快超支了？** 算术很简单；判断一个发现对我已经承诺出去的钱意味着什么，不简单。
- **这个项目是不是卡住了？** 读两个 markdown 文件是脚本的活。发现「计划已经三周没动过」不是。

我给新循环做的测试是：**写下这个检查在成功和失败时分别会打印什么。** 如果两个答案都是一个数字或一个字符串，那它就是脚本。如果答案是「要看它说了什么」，那它需要模型——而这时候它就需要一个我比 agent 更信任的验证器，因为我已经无法预测输出了。

## 真正咬到我的是哪一部分

失败模式不是 drift，是**沉默**。

一个只在出事时才开口的循环，是我唯一能忍受的那种——我不想让 11 个任务每天给我发状态报告。但「健康所以安静」和「崩了所以来不及开口」，从我手机这端看是同一个现象。29 次运行的证据一直躺在那个任务的执行日志里；我却一直在相信「没有通知」这件事本身，而没去读历史。

所以现在这套脚本都多了一条出口：退出码 0 表示*量过了，这是结果*，退出码 1 表示*我量不了*——而退出码 1 永远不静默。配置缺失、状态文件解析失败、没预料到的异常：全都打印一行并退出 1。就这一处改动，本该让这个 bug 在第 1 次就暴露，而不是第 29 次。

如果你有一个循环「已经安静一阵子」了，先去翻它最近十次运行，再下结论说它正常。这条建议花了我三个星期才换到。

## 简短版

- 确定性的检查 → 脚本。判断 → 模型。别为了方便把两者混在一起。
- 给「成功则静默」这条约定开一个例外：「我量不了」必须永远大声。
- 去读 runner 自己的执行历史，而不是只看你收到的通知。

想给你的网站或网店也做一套这样的东西——可用性、证书、备份、以及只在真的有事时才 ping 你的邮件？**[WhatsApp +60 12-797 2969](https://wa.me/60127972969)** · **[[email protected]](mailto:[email protected]?subject=Self-hosted%20monitoring)** · **[hoelee.com](https://hoelee.com)**

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "Stop Writing Agent Prompts for Deterministic Work"
description: "One of my scheduled jobs failed 29 times in a row because I put a model in a loop that only needed a string comparison. Here is the rule I use now."
pubDate: 2026-10-01
category: notes
blueprint: ai-agent-automation
blueprintLayer: L1-deterministic-orchestration
tags: ["loop-engineering", "cron", "ai-agents", "automation"]
ogImage: "/og/stop-writing-agent-prompts-for-deterministic-work.jpg"
banner: "/banners/stop-writing-agent-prompts-for-deterministic-work.jpg"
draft: false
---

One of my scheduled jobs failed 29 times in a row. Nobody told me, because
it was designed to speak only when it had something to report — and a run
that dies before it can speak looks exactly like a run that found nothing
wrong.

The job was a version check: ask npm for the latest published version of a
CLI I use, compare it with the one installed here, and tell me if there is a
newer one. That is the entire task. It is two strings and an inequality.

I had built it as an agent job, because that was the pattern I had in my
head at the time. Every tick, the model would read the injected script
output and decide whether to write a notification. It worked for a while.
Then my global default model changed, and the job — which records the model
it was created with — stopped running entirely. It printed `[drift_skip]`
and exited. Twenty-nine times.

The fix was not a better prompt. It was deleting the model from the loop.

## The rule

**If the stop condition is a comparison — a string, a number, a status code — a model in that loop can only add failure modes.**

There is no judgment to make in "is `3.8.50` newer than `3.8.50`". The model
was not deciding anything the code could not; it was adding a dependency (a
model route, a provider, a config snapshot, a token bill) and a new way to
fail. So I rewrote the checker as 125 lines of Python with an explicit exit
path for every branch, and flipped the job to script mode: the script's
stdout is delivered as the notification, with no LLM anywhere in the path.

It has run clean ever since. Anthropic's own guidance says the same thing in
one line — *use scripts for deterministic work; running a script is cheaper
than reasoning through the steps* — but the cost was never the interesting
part for me. The reliability was.

## Where the model does earn its place

Of the eleven loops I run, eight now have no model in the path. The three
that do all handle something I genuinely cannot express as code:

- **Is this email important?** A mailbox produces promos, DMARC reports, container alerts and a customer asking for a quote. Fetching mail is deterministic. Ranking it is not.
- **Is this shop voucher about to overspend?** The arithmetic is trivial; deciding what a finding means for money I have already committed is not.
- **Is this project stalled?** Reading two markdown files is a script's job. Noticing that the plan has not moved in three weeks is not.

The test I apply to a new loop: **write down what the check would print on
success and on failure.** If both answers are a number or a string, it is a
script. If the answer is "it depends what it says", it needs a model — and
then it needs a verifier I trust more than the agent, because I am no longer
able to predict the output.

## The part that actually bit me

The failure mode was not the drift. It was the **silence**.

A loop that speaks only when something is wrong is the only kind of loop I
can tolerate — I do not want a daily status report from eleven jobs. But
"silent because healthy" and "silent because it crashed before it could
speak" are the same observation from my side of the phone. Twenty-nine runs
of evidence were sitting in the job's execution log the whole time; I was
trusting the absence of a notification instead of reading the history.

So every script in the fleet now has a second exit path: exit 0 means
*measured, here is what I found*, and exit 1 means *I could not measure* —
and exit 1 is never silent. Missing config, unparseable state, an unexpected
exception: all of them print a line and exit 1. That one change would have
surfaced this bug on the first run instead of the twenty-ninth.

If you have a loop that has been "quiet for a while", go and read its last
ten runs before you conclude it is working. That advice cost me three weeks.

## The short version

- Deterministic check → script. Judgment → model. Do not mix them for convenience.
- Give the silent-on-success convention one exception: "I could not measure" must always be loud.
- Read the runner's own execution history, not just your notifications.

Want a version of this for your own site or shop — uptime, certificates,
backups, mail that only pings you when it matters?
**[WhatsApp +60 12-797 2969](https://wa.me/60127972969)** ·
**[[email protected]](mailto:[email protected]?subject=Self-hosted%20monitoring)** ·
**[hoelee.com](https://hoelee.com)**

文章 09 / 16：密码正确，Synology API 登录却返回 401，问题出在哪

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "密码正确，Synology API 登录却返回 401，问题出在哪"
description: "两种原因，同一个报错：Office Suite API 对 TLS 握手失败和受 2FA 保护的账号返回完全相同的错误。三条命令就能区分，不必折腾一下午。"
pubDate: 2025-09-10
updatedDate: 2026-09-26
category: notes
tags: [synology, dsm, rest-api, tls, 2fa, debugging, authentication]
ogImage: /og/zh-synology-api-401-with-the-correct-password.jpg
banner: /banners/synology-api-401-with-the-correct-password.jpg
draft: false
---

你搭好了 Synology 的 Office Suite API，把凭据发过去，得到的却是：

```json
{"error":"Unauthorized"}
```

然后你用这组密码登录 DSM 验证——一切正常。你重新输入一遍，改用 HTTP 而不是 HTTPS，怀疑是不是账号缺了什么找不到的权限。这些全都是白费功夫，因为在这套 API 上，密码正确却返回 `401` 的原因恰好只有两个，而这两个都不是密码的问题。

## 原因一：`host` 字段必须是一个名字，且证书要能被容器接受

这个 API 是自配置的：每次登录调用时，由你来告诉*它*要对哪台 DSM 进行认证。

```bash
curl -s -X POST http://<api-host>:8791/spreadsheets/authorize \
  -H 'Content-Type: application/json' \
  -d '{"username":"acct","password":"pw","host":"192.168.1.1:5001","protocol":"https"}'
```

```json
{"error":"Unauthorized"}
```

代理会与那个 `host` 自行完成 TLS 握手。裸 IP 地址通常拿到的证书是为某个主机名签发的，于是握手失败——而失败的握手会被报告为 `Unauthorized`，和密码错误一模一样。响应里没有任何线索指向 TLS。

同样的请求、同样的凭据，把 `host` 换成一个持有有效证书的名字：

```bash
-d '{"username":"acct","password":"pw","host":"cloud.example.com","protocol":"https"}'
```

```json
{"token":"eyJhbGciOi...","host":"cloud.example.com"}
```

修复方法就这么多。**规则：`host` 的值必须是一个 FQDN，且其证书能被容器接受；如果端口不是该协议默认端口，也要一并写上。**

## 原因二：双重认证在这里永远行不通

第二个原因是结构性的。登录的 schema 只有四个字段，没有一次性验证码字段：

```yaml
AuthorizationBody:
  properties:
    username: {type: string}
    password: {type: string}
    host:     {type: string}
    protocol: {type: string}
```

没有 OTP，没有应用专用密码，也没有设备令牌交换。启用了 2FA 的账号，无论在哪个 `host` 上、无论密码多正确，都会永远返回 `401`。请改用关闭了 2FA 的专用服务账号，并把权限范围限定在它需要的文件夹上。

## 三条命令区分二者

别靠猜——用不了一分钟就能把两种原因分开。

**1. `host` 是否可达，并且是否为该名字提供了有效证书？**

```bash
curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://cloud.example.com/
```

非零的 `ssl_verify_result` 指向原因一。（在信任该证书的机器上，同一命令返回 `0`。）

**2. API 到底有没有应答？它对乱写的垃圾请求是不是也这么拒绝？**

```bash
curl -s -o - -w '\n%{http_code}\n' -X POST http://<api-host>:8791/spreadsheets/authorize \
  -H 'Content-Type: application/json' \
  -d '{"username":"nobody","password":"wrong","host":"cloud.example.com","protocol":"https"}'
```

```json
{"error":"Unauthorized"}
401
```

和你失败的那次调用输出一样——这正是关键。它证明服务是活的，并且 `401` 是它对*所有*认证失败的通用回答，TLS 失败也不例外。如果容器宕了，你得到的是连接错误而不是 `401`，所以这一步也能排除"API 没在运行"。

**3. 一个关闭了 2FA 的账号在好用的 `host` 上能成功吗？** 如果非 2FA 服务账号能用而你的管理员账号不行，那就是原因二。

## 相关的坑：几周后才冒出来的 `401`

你拿到的 token 是一个有效期 28 天的 JWT，但它绑定在 DSM 会话上。DSM 重启，或该账号被强制登出，都会让它提前失效。所以昨天还能用的调用今天返回 `401`，意思是"重新认证"，而不是"我的凭据变了"——先检查 token 的年龄，再去找配置回归的问题。

而且登录进去之后，别把下一道墙和这一道搞混：`403 Permission denied` 表示文件存在但账号拿不到（通常是文件放在个人的 `My Drive` 主目录里，其他 DSM 账号都访问不到）；`404 Spreadsheet not found` 则表示 ID 写错了。不同的问题，不同的修法。

完整的搭建过程——容器、compose 文件、可用的调用，以及我踩过的四个坑——都在 [Synology Spreadsheet API 是一个容器，不是一个 API 端点](/posts/synology-spreadsheet-api-is-a-container/) 里。

---

*我为马来西亚的中小企业构建自托管的集成和内部工具——[WhatsApp](https://wa.me/60127972969) 或 [邮件](mailto:[email protected]?subject=Synology%20API%20integration)。更多信息见 [hoelee.com](https://hoelee.com)。*

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "Why Your Synology API Login Returns 401 With the Correct Password"
description: "Two causes, one error: the Office Suite API reports a failed TLS handshake and a 2FA-protected account identically. How to tell them apart in three commands instead of an afternoon."
pubDate: 2025-09-10
updatedDate: 2026-09-26
category: notes
tags: [synology, dsm, rest-api, tls, 2fa, debugging, authentication]
ogImage: /og/synology-api-401-with-the-correct-password.jpg
banner: /banners/synology-api-401-with-the-correct-password.jpg
draft: false
---

You set up Synology's Office Suite API, you send your credentials, and you get:

```json
{"error":"Unauthorized"}
```

You then verify the password by logging into DSM with it — works fine. You retype it, you try HTTP instead of HTTPS, you wonder whether the account needs a permission you can't find. All of it is wasted motion, because on this API a `401` with a correct password has exactly two causes, and neither of them is the password.

## Cause 1: the `host` field must be a name with a certificate the container accepts

The API is self-configuring: you tell *it* which DSM to authenticate against, on every sign-in call.

```bash
curl -s -X POST http://<api-host>:8791/spreadsheets/authorize \
  -H 'Content-Type: application/json' \
  -d '{"username":"acct","password":"pw","host":"192.168.1.1:5001","protocol":"https"}'
```

```json
{"error":"Unauthorized"}
```

The proxy performs its own TLS handshake with that `host`. A bare IP address usually presents a certificate issued for a hostname, so the handshake fails — and a failed handshake is reported as `Unauthorized`, identically to a wrong password. Nothing in the response hints at TLS.

Same request, same credentials, `host` changed to a name with a valid certificate:

```bash
-d '{"username":"acct","password":"pw","host":"cloud.example.com","protocol":"https"}'
```

```json
{"token":"eyJhbGciOi...","host":"cloud.example.com"}
```

That is the whole fix. **Rule: the `host` value must be an FQDN whose certificate the container will accept, with the port included if it isn't the default for the scheme.**

## Cause 2: two-factor authentication can never work here

The second cause is structural. The sign-in schema has four fields and no one-time-code field:

```yaml
AuthorizationBody:
  properties:
    username: {type: string}
    password: {type: string}
    host:     {type: string}
    protocol: {type: string}
```

No OTP, no app password, no device-token exchange. An account with 2FA enabled returns `401` on every host, with every correct password, forever. Use a dedicated service account with 2FA off, scoped to the folders it needs.

## Telling them apart in three commands

Don't guess — separate the two in under a minute.

**1. Is the host reachable and serving a valid certificate for that name?**

```bash
curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://cloud.example.com/
```

A non-zero `ssl_verify_result` points at cause 1. (From a machine that trusts the cert, the same command returns `0`.)

**2. Does the API answer at all, and does it reject garbage the same way?**

```bash
curl -s -o - -w '\n%{http_code}\n' -X POST http://<api-host>:8791/spreadsheets/authorize \
  -H 'Content-Type: application/json' \
  -d '{"username":"nobody","password":"wrong","host":"cloud.example.com","protocol":"https"}'
```

```json
{"error":"Unauthorized"}
401
```

Same output as your failing call — which is the point. It proves the service is alive and that `401` is its generic answer for *every* authentication failure, TLS included. A container that is down gives you a connection error, not a `401`, so this also rules out "the API isn't running".

**3. Does an account with 2FA off succeed against the good host?** If a non-2FA service account works and your admin account doesn't, you have cause 2.

## The related trap: a `401` that appears weeks later

The token you get is a JWT valid for 28 days, but it's bound to a DSM session. A DSM restart, or a forced logout of that account, invalidates it early. So a `401` on a call that worked yesterday means "re-authenticate", not "my credentials changed" — check the token's age before you go looking for a config regression.

And once you're in, don't confuse the next wall with this one: `403 Permission denied` means the file exists but the account can't have it (usually a file sitting in a personal `My Drive` home folder, unreachable by any other DSM account), while `404 Spreadsheet not found` means the ID is wrong. Different problems, different fixes.

The full setup — the container, the compose file, the working calls, and the four traps I hit getting there — is in [The Synology Spreadsheet API Is a Container, Not an API Endpoint](/posts/synology-spreadsheet-api-is-a-container/).

---

*I build self-hosted integrations and internal tools for small businesses in Malaysia — [WhatsApp](https://wa.me/60127972969) or [email](mailto:[email protected]?subject=Synology%20API%20integration). More at [hoelee.com](https://hoelee.com).*

文章 10 / 16：NocoDB 附件字段怎么改都不生效，和一个漏网的正则

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "NocoDB 附件字段怎么改都不生效，和一个漏网的正则"
description: "回填 300 行数据时踩到的两个小坑：NocoDB 在附件保留 id 的情况下会无视 path 的改动，以及一个 lookahead 正则悄悄跳过了没有扩展名的 URL。"
pubDate: 2026-02-04
updatedDate: 2026-09-10
category: notes
blueprint: sme-digital-suite
blueprintLayer: L3-entitlement-backoffice
tags: [nocodb, backfill, python, regex, data-pipeline]
ogImage: /og/zh-the-nocodb-attachment-that-wouldnt-update.jpg
banner: /banners/the-nocodb-attachment-that-wouldnt-update.jpg
---

在 NocoDB 里回填 300 行数据的经历，暴露了两个小坑，每一个都耗掉了我一段时间。
它们都属于那种「只有当你修的数据恰好接近、但又没完全符合代码假设的形状时」
才会咬你的 bug。

## 坑 1：Attachment 改了 `path` 没用，只要你还留着 `id`

我想把一个 Attachment 列里的缩略图 URL 全部换成高清版。直觉上就是去改这个
列存的 JSON：

```python
img = json.loads(row["image"])          # [{"id": "...", "path": "...", "signedPath": "..."}]
img[0]["path"] = new_fullsize_url
img[0]["signedPath"] = new_fullsize_url
```

它看起来成了——PATCH 返回成功。但把行读回来，每个 `path` 还是旧的缩略图。

元凶是：这个附件带着一个 NocoDB 生成的 `id`，只要它还在，NocoDB 就会**按这个
id 去解析附件**，然后心安理得地无视新的 `path`。我以为自己在更新的那个字段，
其实是只读的，只要 `id` 还在。

修复的办法是丢掉服务端分配的字段，让 NocoDB 从头重新注册这个附件：

```python
img[0] = {
    "path": new_fullsize_url,
    "mimetype": "image/jpeg",
    "title": img[0].get("title", "image.jpg"),
}
# 不要 "id"，也不要 "signedPath" —— NocoDB 会重新分配
```

没有 `id` 之后，NocoDB 会根据 `path` 注册一个新附件，并生成一个新的签名 URL。
这下值才真正写进去了。

**规则：** 如果 NocoDB 的 Attachment 改来改去都不生效，检查一下 `id` 字段。
把它剥掉、重新注册，而不是原地打补丁。

## 坑 2：一个会悄悄跳过「无扩展名 URL」的 lookahead 正则

为了拼出高清 URL，我需要去掉 `_progressive_thumbnail` 这个后缀。第一版我用了
一个 lookahead：

```python
clean = re.sub(r"_progressive_thumbnail(?=\.\w+$)", "", url)
```

这对这种情况有效：

```
...70776f54_progressive_thumbnail.jpg   →  ...70776f54.jpg   ✅
```

但 Carousell 有**两种** URL 形态。有些缩略图以光秃秃的后缀结尾，**没有扩展名**：

```
...f94af259_progressive_thumbnail      →  (lookahead 匹配不上)  ❌
```

因为 lookahead 要求结尾是 `.\w+`，第二种形态就漏掉了——300 行里有 25 行。
这个正则「没报错」，它只是在那部分数据上什么都没做。

更稳的写法是直接 `replace`，两种形态都能覆盖：

```python
clean = url.replace("_progressive_thumbnail", "")
```

**规则：** 当你要规范化一个有多种真实形态的字符串时，除非你已经把所有形态都
枚举清楚了，否则优先用 `replace()` 而不是正则。一个悄悄跳过的正则，比一个
直接报错的更糟。

---

这里头没什么高深的东西。但这两类都是那种**只有在真实数据里才会现形**的无聊
bug——一个 API 信誓旦旦说更新成功了的附件，一个匹配了 91% 行、却对剩下 9%
一声不吭的正则。如果你给 NocoDB 回填数据时总觉得「只生效了一半」，先来查这两
个地方。

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "The NocoDB Attachment That Wouldn't Update (and a Regex That Missed)"
description: "Two small gotchas from backfilling 300 rows: NocoDB ignores a path change when the attachment keeps its id, and a lookahead regex silently skipped URLs without a file extension."
pubDate: 2026-02-04
updatedDate: 2026-09-10
category: notes
blueprint: sme-digital-suite
blueprintLayer: L3-entitlement-backoffice
tags: [nocodb, backfill, python, regex, data-pipeline]
ogImage: /og/the-nocodb-attachment-that-wouldnt-update.jpg
banner: /banners/the-nocodb-attachment-that-wouldnt-update.jpg
---

Backfilling 300 rows in NocoDB surfaced two small gotchas that each ate a chunk of
time. Both are the kind of thing that only bites you when the data you're fixing
is *almost* — but not quite — in the shape the code assumes.

## Gotcha 1: updating an Attachment's `path` doesn't matter if you keep the `id`

I wanted to swap every thumbnail URL in an Attachment column for its full-size
version. The obvious move is to edit the JSON the column stores:

```python
img = json.loads(row["image"])          # [{"id": "...", "path": "...", "signedPath": "..."}]
img[0]["path"] = new_fullsize_url
img[0]["signedPath"] = new_fullsize_url
```

It looked like it worked — the PATCH returned success. But reading the rows back,
every `path` was still the old thumbnail.

The culprit: the attachment carries an NocoDB-generated `id`, and as long as it's
present, NocoDB resolves the attachment *by that id* and happily ignores the new
`path`. The field I thought I was updating was effectively read-only while the `id`
was still there.

The fix is to drop the server-assigned fields and let NocoDB re-register the
attachment from scratch:

```python
img[0] = {
    "path": new_fullsize_url,
    "mimetype": "image/jpeg",
    "title": img[0].get("title", "image.jpg"),
}
# no "id", no "signedPath" — NocoDB assigns fresh ones
```

With `id` gone, NocoDB registers a new attachment from the `path` and generates a
fresh signed URL. The value finally sticks.

**Rule:** if a NocoDB Attachment edit isn't sticking, check for the `id` field.
Strip it and re-register instead of patching in place.

## Gotcha 2: a lookahead regex that silently skipped extension-less URLs

To build the full-size URL I needed to strip the `_progressive_thumbnail` suffix.
My first pass used a lookahead:

```python
clean = re.sub(r"_progressive_thumbnail(?=\.\w+$)", "", url)
```

That works for:

```
...70776f54_progressive_thumbnail.jpg   →  ...70776f54.jpg   ✅
```

But Carousell serves *two* URL shapes. Some thumbnails end with the bare suffix
and **no extension**:

```
...f94af259_progressive_thumbnail      →  (lookahead doesn't match)  ❌
```

Because the lookahead requires `.\w+` at the end, the second shape slipped through
unmatched — 25 of my 300 rows. The regex "worked," it just silently didn't do
anything on that slice of the data.

The robust version is a plain replacement, which handles both:

```python
clean = url.replace("_progressive_thumbnail", "")
```

**Rule:** when normalizing a string that has multiple real-world shapes, prefer
`replace()` over a regex unless you've enumerated every shape. A regex that
silently skips is worse than one that fails loudly.

---

Nothing here is exotic. But both are the *boring* kind of bug that only reveals
itself in real data — the attachment the API swears it updated, the regex that
matches 91% of the rows and says nothing about the other 9%. If backfilling data
into NocoDB ever feels like it's half-working, these two are where I'd look first.

文章 11 / 16：CodeIgniter 4.6 升级到 4.7：官方指南没说到的破坏性变更

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "CodeIgniter 4.6 升级到 4.7：官方指南没说到的破坏性变更"
description: "CodeIgniter 4.7 官方升级指南列了八项破坏性变更。真正让我应用致命崩溃的那两项不在里面——而且它们属于同一类问题。"
pubDate: 2026-09-27
category: notes
tags: [codeigniter, php, upgrade, composer, breaking-changes, config, framework]
ogImage: /og/zh-upgrading-codeigniter-46-to-47.jpg
banner: /banners/upgrading-codeigniter-46-to-47.jpg
draft: false
---

我把一个 CodeIgniter 4 应用从 4.6.3 升到了 4.7.4。我读了升级指南，把每一项文档化的破坏性变更都对着代码库核了一遍，确认没有一项影响到我。然后我跑了一下应用，它致命崩溃了两次。

两个致命错误属于同一类问题，而**升级指南里完全没提这类问题**。这是一篇短文，讲真正会坏的是什么，以及为什么指南没法提前警告你。

## 为什么这件事重要

Composer 有一条容易忘掉的作用域边界。`composer update` 只写 `vendor/`——也就是**系统作用域**，框架自己的文件，它可以随意替换。

而你的 `app/Config/*.php` 是**项目作用域**。那是你的文件。Composer 永远不会碰它们，升级指南对它们的措辞是"你可能想要合并这些变更"，而不是"这会导致崩溃"。

但当 4.7 的框架代码去读一个属性，而你 4.6 时代的配置类里根本没定义它时，你得到的是致命错误——不是废弃警告，不是 notice。指南没法穷举这件事，因为这个属性的集合是「新框架读什么」与「你的配置文件里恰好有什么」的交集，而它对后半边完全不可见。

如果你要升级一个落后不止一个 minor 版本的 CI4 应用，请为此留出预算。我花了一个小时左右，改了 14 个配置文件。

## 我先核了什么

4.7.0 文档化的破坏性变更，以及每一项是否适用：

| 文档化的变更 | 是否影响本应用 |
|---|---|
| `Model::insertBatch()` / `updateBatch()` 返回值变化 | 否——没用 Model |
| Entity 类型转换变更 | 否——没用 Entity |
| 验证规则变更（`regex_match`、`differs`） | 否 |
| 加密处理器默认值 | 否 |
| 上传文件／图片验证行为 | 否——没有上传 |
| `IncomingRequest` 内部变更 | 否 |
| 移除的 `Session` 类属性 | 否——见下文 |
| `PageCache` 构造函数签名 | 否——见下文 |

其中两项需要细看，结果两个都是**假警报**，值得讲一下，因为你如果只 grep，它们看起来和真命中一模一样：

**"移除的 `Session` 类属性"** —— 在配置里 grep `session`，立刻一片飘红。但那些属性（`sessionDriver`、`sessionCookieName` 之类）是我 `Config\App` **自己的属性**，并没有被移除。指南指的是 `Session` 类上的属性。同一个词，不同的类。

**"`PageCache` 构造函数签名变更"** —— 我确实引用了 `pagecache`，但只是把它作为 `Config\Filters` 里的一个 filter 别名。这并不等于继承或实例化这个类，而只有后者才会坏。**引用不等于子类。**

这两次核对教给我同一件事：**grep 找到的是「提及」，不是「使用」。** 要确认命中的那个类是指南说的那个类，以及你确实继承或调用了那个发生变化的东西。

## 指南里没有的两个致命错误

两个都在干净启动时冒出来。两个都是 4.7 框架代码要读、而 4.6 时代的配置文件没有定义的属性。

### 1. `Config\App::$permittedURIChars`

```
Undefined property: Config\App::$permittedURIChars
```

4.7 的 Router 要求这个属性，用它校验进来的 URI 字符。4.6 时代的 `App.php` 里没有它，于是 Router 在路由任何东西之前就抛异常了。

```php
// app/Config/App.php —— 追加到类里
/**
 * CI4 4.7 compatibility: allowed characters in a URI.
 * Required by the 4.7 Router.
 */
public string $permittedURIChars = 'a-z 0-9~%.:_\-';
```

### 2. `Config\Format::$jsonEncodeDepth`

```
Undefined property: Config\Format::$jsonEncodeDepth
```

4.7 的 `JSONFormatter` 要求它。这一个直接打断了应用的**报告引擎**——因为引擎的 JSON 响应是用 `json_decode()` 解的，失败信号在很下游才爆出来，表现为一个让人一头雾水的解析错误，而不是真正的"属性未定义"。

```php
// app/Config/Format.php —— 追加到类里
/**
 * CI4 4.7 compatibility: json_encode() depth limit.
 */
public int $jsonEncodeDepth = 512;
```

第二个值得多说一句。表面上看到的错误是：

```
Failed to parse JSON string. Malformed UTF-8 characters
```

指向编码问题。真实原因是缺一个配置属性。**当框架的错误信息描述的是症状而不是病因时，先去日志里找真正的异常，再照着你拿到的那句话行动**——我在一个从来没坏过的 UTF-8 上白白耗了时间。

## 怎么在它们咬到你之前找出剩下的

两个致命错误都是靠**运行应用**发现的。运行是可靠的方法，但你可以先一步靠"属性覆盖差异对比"把这一类问题一次挖出来。

思路是：解析 `vendor/` 里框架自带的配置默认值，再解析你项目里的配置类，列出前者有、后者缺的属性。这样你一次拿到整类问题，而不是每重启一次收获一个致命错误。

```python
import re, pathlib

sys_ = pathlib.Path('vendor/codeigniter4/framework/system/Config')
app_ = pathlib.Path('app/Config')

def props(path):
    try:
        src = path.read_text(encoding='utf-8', errors='replace')
    except FileNotFoundError:
        return set()
    # public/protected/private $name = ...
    return set(re.findall(r'(?:public|protected|private)\s+(?:[\w\\\[\]|?]+\s+)?\$(\w+)', src))

# 4.7 需要、而陈旧配置可能漏掉的属性所在类
for name in ['App', 'Cache', 'ContentSecurityPolicy', 'CURLRequest', 'DocTypes',
             'Email', 'Encryption', 'Exceptions', 'Format', 'Honeypot',
             'Migrations', 'Paths', 'Routing', 'Toolbar', 'View']:
    missing = props(sys_ / f'{name}.php') - props(app_ / f'{name}.php')
    if missing:
        print(name, '->', sorted(missing))
```

在 `composer update` **之后**、启动应用**之前**，拿新的 `vendor/` 跑一遍。它打印出来的任何东西都是潜在的致命错误。

在这个项目里它标出了 14 个配置文件。我把缺失的属性合并进每一个文件，以标记块的形式追加在类的末尾，这样上面原有的设置和行为完全不动：

```php
    // ----------------------------------------------------------------
    // CI4 4.7 compatibility — properties the 4.7 framework reads.
    // Appended as a block so existing settings above are unchanged.
    // ----------------------------------------------------------------
```

框架新带出来的两个配置文件在项目里完全没有对应物，需要新建而不是合并：`Hostnames.php` 和 `WorkerMode.php`。直接从 `vendor/codeigniter4/framework/app/Config/` 拷过去就行。

## 升级前还值得知道的几点

**`app.baseURL` 的校验变严了。** 4.7 会对 4.6 接受的值抛 `ConfigException`。`'http://localhost/'`——这个值在我本地 `.env` 里躺了好几个月、一直好好的——现在直接抛异常。生产环境没受影响，因为它用的是真实 URL，但任何用了简写 baseURL 的开发环境都会拒绝启动。

**安全上的理由是真的。** 单是 4.7.4 就修了上传文件扩展名绕过（`is_image` / `mime_in`）、`deleteBatch()` 的 SQL 注入、以及 `UploadedFile::move()` 的路径穿越。这三个在本应用里都不可利用——它没有上传、也不碰数据库——但它们会在你加上这些功能的那一刻变成真实风险。**在你需要这些功能之前先升级，是更省事的顺序。**

**PHP 最低版本现在是 8.2。** 动手前先确认你的主机，不是事后。

## 如果重来一次，我会怎么改

**先拿新的 `vendor/` 跑一遍应用，别的什么都别做。** `composer update`、启动、然后修坏掉的地方。读指南告诉你可能变什么；只有运行告诉你实际变了什么。

**把属性差异对比当成常规动作。** 它把一串串行的调试过程变成一份报告。

**把升级做成独立的一次提交。** 我就是这么做的，这让两个致命错误变得极易二分定位——一条 `git show` 就精确告诉我每个错误是哪次改动引入的。把框架升级混进功能分支，会把五分钟的诊断变成一场考古。

**读 changelog，不要只读升级指南。** 指南列的是维护者判断「可能会弄坏你」的东西。changelog 列的是实际变了的东西。这两者不是同一个集合，而这两个致命错误正好住在两者的缝隙里。

## 结果

一个 17,000 行的 CodeIgniter 应用从 4.6.3 升到 4.7.4：合并 14 个配置文件、新增 2 个配置文件、找到并修复 2 个致命错误，所有路由验证返回 200——包括一次真实的报告生成，产出 19 页 A4，与升级前的基线在字节层面可比。应用日志零错误。

---

**手上跑着旧版 CodeIgniter？** 我做这类框架升级——CI4 minor 版本升级、PHP 版本迁移，以及那些升级指南默认你自己会搞定的配置合并工作。

[WhatsApp +60 12-797 2969](https://wa.me/60127972969) · [[email protected]](mailto:[email protected]?subject=CodeIgniter%20升级) · [hoelee.com](https://hoelee.com)

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "Upgrading CodeIgniter 4.6 to 4.7: The Breaking Changes the Guide Misses"
description: "The official CodeIgniter 4.7 upgrade guide lists eight breaking changes. The two that actually fataled my app weren't in it — and both are the same class of problem."
pubDate: 2026-09-27
category: notes
tags: [codeigniter, php, upgrade, composer, breaking-changes, config, framework]
ogImage: /og/upgrading-codeigniter-46-to-47.jpg
banner: /banners/upgrading-codeigniter-46-to-47.jpg
draft: false
---

I upgraded a CodeIgniter 4 app from 4.6.3 to 4.7.4. I read the upgrade guide, checked every documented breaking change against the codebase, and confirmed none of them applied. Then I ran the app and it fataled twice.

Both fatal errors were the same kind of problem, and that kind of problem is **not mentioned anywhere in the upgrade guide**. This is a short post about what actually breaks and why the guide can't warn you.

## Why this matters

Composer has a scope boundary that is easy to forget. `composer update` writes to `vendor/` — the **system scope**, the framework's own files, which it owns and can replace freely.

Your `app/Config/*.php` files are **project scope**. They are yours. Composer will never touch them, and the upgrade guide describes them as "you may want to merge these changes" rather than "this will break".

But when 4.7's framework code reads a property that your 4.6-era config class does not define, you get a fatal — not a deprecation, not a warning. The guide cannot enumerate this, because the set of properties is the intersection of *what the new framework reads* and *what your config file happens to contain*, and it has no visibility into the second half.

If you are upgrading a CI4 app more than a minor version behind, budget for this. It took me about an hour and 14 config files.

## What I checked first

The documented breaking changes in 4.7.0, and whether each applied:

| Documented change | Affects this app? |
|---|---|
| `Model::insertBatch()` / `updateBatch()` return values | No — no Models used |
| Entity casting changes | No — no Entities |
| Validation rule changes (`regex_match`, `differs`) | No |
| Encryption handler defaults | No |
| Uploaded-file / image validation behaviour | No — no uploads |
| `IncomingRequest` internal changes | No |
| Removed `Session` class properties | No — see below |
| `PageCache` constructor signature | No — see below |

Two of those needed a closer look, and both turned out to be **false alarms** worth describing, because they look identical to a real hit if you only grep:

**"Removed `Session` class properties"** — grepping for `session` in my configs lit up immediately. But the properties in question (`sessionDriver`, `sessionCookieName`, and friends) on my `Config\App` are **`Config\App`'s own properties**, which were not removed. The guide refers to properties on the `Session` class. Same word, different class.

**"`PageCache` constructor signature changed"** — I reference `pagecache` — but only as a filter alias in `Config\Filters`. That does not extend or instantiate the class, which is the only thing that breaks. A reference is not a subclass.

Both checks taught me the same thing: **grep finds *mentions*, not *usages*.** Verify the class in the match is the class the guide means, and that you actually extend or call the thing that changed.

## The two fatals that were not in the guide

Both surfaced on a clean boot. Both are properties that 4.7 framework code reads and 4.6-era config files do not define.

### 1. `Config\App::$permittedURIChars`

```
Undefined property: Config\App::$permittedURIChars
```

Required by 4.7's Router, which validates incoming URI characters against this property. Absent from a 4.6-era `App.php`, the Router throws before it routes anything.

```php
// app/Config/App.php — append to the class
/**
 * CI4 4.7 compatibility: allowed characters in a URI.
 * Required by the 4.7 Router.
 */
public string $permittedURIChars = 'a-z 0-9~%.:_\-';
```

### 2. `Config\Format::$jsonEncodeDepth`

```
Undefined property: Config\Format::$jsonEncodeDepth
```

Required by 4.7's `JSONFormatter`. This one broke the app's **report engine**, because the engine's JSON response is decoded with `json_decode()` and the failure surfaced much further downstream as a confusing parse error rather than the real "undefined property" message.

```php
// app/Config/Format.php — append to the class
/**
 * CI4 4.7 compatibility: json_encode() depth limit.
 */
public int $jsonEncodeDepth = 512;
```

That second one is worth dwelling on. The visible error was:

```
Failed to parse JSON string. Malformed UTF-8 characters
```

Which points at encoding. The actual cause was a missing config property. **If a framework error names a symptom rather than a cause, go looking for the real exception in the log before acting on the message you were given** — I lost time chasing UTF-8 that was never broken.

## How to find the rest before they bite you

Both fatal errors were found by *running* the app. Running is the reliable method, but you can get ahead of it by diffing property coverage.

The approach: parse the framework's shipped config defaults from `vendor/`, parse your project's config classes, and list properties defined in the former but missing from the latter. That gives you the whole class of problem at once instead of one fatal per restart.

```python
import re, pathlib

sys_ = pathlib.Path('vendor/codeigniter4/framework/system/Config')
app_ = pathlib.Path('app/Config')

def props(path):
    try:
        src = path.read_text(encoding='utf-8', errors='replace')
    except FileNotFoundError:
        return set()
    # public/protected/private $name = ...
    return set(re.findall(r'(?:public|protected|private)\s+(?:[\w\\\[\]|?]+\s+)?\$(\w+)', src))

# names whose properties 4.7 requires but a stale config may omit
for name in ['App', 'Cache', 'ContentSecurityPolicy', 'CURLRequest', 'DocTypes',
             'Email', 'Encryption', 'Exceptions', 'Format', 'Honeypot',
             'Migrations', 'Paths', 'Routing', 'Toolbar', 'View']:
    missing = props(sys_ / f'{name}.php') - props(app_ / f'{name}.php')
    if missing:
        print(name, '->', sorted(missing))
```

Run it against the **new** `vendor/` after `composer update` and before you boot the app. Anything it prints is a candidate fatal.

In this project it flagged 14 config files. I merged the missing properties into each one, appended in a marked block at the end of the class so existing settings and behaviour were untouched:

```php
    // ----------------------------------------------------------------
    // CI4 4.7 compatibility — properties the 4.7 framework reads.
    // Appended as a block so existing settings above are unchanged.
    // ----------------------------------------------------------------
```

Two config files the framework now ships had no counterpart at all and needed creating rather than merging: `Hostnames.php` and `WorkerMode.php`. Copy those straight from `vendor/codeigniter4/framework/app/Config/`.

## Also worth knowing before you upgrade

**Validate `app.baseURL` properly.** 4.7 throws a `ConfigException` on values 4.6 accepted. `'http://localhost/'` — which had been sitting in my local `.env` for months, working fine — now throws. Production was unaffected because it used a real URL, but any dev environment with a shorthand baseURL will refuse to boot.

**The security case is real.** 4.7.4 alone patched an uploaded-file extension bypass (`is_image` / `mime_in`), a SQL injection in `deleteBatch()`, and a path traversal in `UploadedFile::move()`. None were exploitable in this app — it does no uploads and uses no database — but all three become live the moment those features are added. Upgrading before you need those features is the cheaper order.

**PHP floor is now 8.2.** Check your host before starting, not after.

## What I'd do differently

**Run the app against the new `vendor/` before doing anything else.** `composer update`, then boot it, then fix what breaks. Reading the guide tells you what *might* change; only running tells you what *did*.

**Do the property diff as a matter of course.** It converts a serial debugging session into one report.

**Do the upgrade as its own commit, on its own.** I did, and it made the two fatals trivially bisectable — one `git show` told me exactly which change introduced each. Mixing a framework bump into a feature branch turns a five-minute diagnosis into an archaeology exercise.

**Read the changelog, not just the upgrade guide.** The guide lists what the maintainers judged *likely* to break you. The changelog lists what changed. They are not the same set, and the gap is exactly where these two fatals lived.

## The result

4.6.3 → 4.7.4 on a 17,000-line CodeIgniter app: 14 config files merged, two new config files added, two fatals found and fixed, and every route verified returning 200 — including a real report generation producing 19 A4 pages with byte-comparable output to the pre-upgrade baseline. Zero errors in the application log.

---

**Running an older CodeIgniter app?** I do framework upgrades like this one — CI4 minor bumps, PHP version migrations, and the config-merging work that the upgrade guide assumes you will figure out yourself.

[WhatsApp +60 12-797 2969](https://wa.me/60127972969) · [[email protected]](mailto:[email protected]?subject=CodeIgniter%20upgrade) · [hoelee.com](https://hoelee.com)

文章 12 / 16：为什么本地语音合成一直在念 “Percent”？语音管线里的 URL 编码与沙盒避坑记

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "为什么本地语音合成一直在念 “Percent”？语音管线里的 URL 编码与沙盒避坑记"
description: "记录本地部署 CosyVoice 2 并通过群晖 n8n 对接时，语音合成陷入无限重复 “Percent” 的现象剖析，以及字符前端规范化与 n8n 沙盒表达式避坑。"
pubDate: 2026-10-22
category: notes
blueprint: ai-agent-automation
blueprintLayer: L1-deterministic-orchestration
tags: ["tts", "debugging", "n8n", "urlencode", "python", "ai"]
ogImage: "/og/zh-why-is-my-tts-model-repeating-percent.jpg"
banner: "/banners/why-is-my-tts-model-repeating-percent.jpg"
draft: false
---

刚把阿里巴巴开源的 CosyVoice 2 模型部署到局域网并挂载上群晖 NAS 的 n8n 自动化工作流，在控制台用 Python 脚本直调合成一切正常。然而，当我们第一次尝试通过 n8n 的 Webhook 端点发起 HTTP GET 测试时，音箱里传出来的却是一段令人窒息的“外星密码”。

原本传入的中文台词是：*“恭喜！本地 CosyVoice 2 模型通过群晖 N8N 联动测试成功！”*

但生成的音频却以极其严肃认真的语调，一口气连念了十几秒：

> *“Percent E 6, percent 81, percent 81, percent B 9, percent A 7, percent 9... percent...”*

模型没有崩溃，甚至声线极为清晰、咬字极其标准，但它在不断疯狂重复 “Percent”。

这是一起非常典型、但在语音合成自动化链路中极容易踩中的“文本前端（Text Normalization）灾难”。本文记录这起 Bug 的全链路成因与双层防御方案。

---

## 1. 现象复现：无限的 “Percent” 循环

我们在终端使用 `curl` 模拟普通外部请求调用 n8n 的 Webhook：

```bash
curl -G "http://192.168.1.1:5678/webhook/ltts" \
  --data-urlencode "text=恭喜！本地模型测试成功！" \
  --data-urlencode "voice=child" \
  -o test.wav
```

接口返回状态码 HTTP 200，并成功下载了一段 2.4MB 的标准 WAV 音频。但当我们将音频丢进本地的语音识别（STT）模型逆向转写核对内容时，转写文本原形毕露：

```json
{
  "text": "percent E 6 percent 81 percent 81 percent B 9 percent A 7 percent..."
}
```

模型并没有出现自回归幻觉，它只是在“百分之百忠实”地读出它所收到的字符串。

---

## 2. 根因剖析 1：被透传的 URL 百分号编码

追溯 HTTP 参数在网络与容器网关中的流动链路，整个过程如下：

1. **客户端转码**：当客户端通过 HTTP GET 发送带中文参数的请求时，所有非 ASCII 字符必须按照标准进行百分号编码（Percent-Encoding）。例如汉字 `恭`（UTF-8 编码为 `0xE6 0x81 0xAD`），在 URL Query 中会被编码为字符串 `"%E6%81%AD"`。
2. **n8n Webhook 接收**：n8n 收到请求后，直接将原始 Query 参数挂载到数据上下文：
   ```json
   {
     "query": {
       "text": "%E6%81%AD%E5%96%9C%EF%BC%81..."
     }
   }
   ```
3. **HTTP Request 节点转发**：在最初的简易工作流中，下游转发给本地 GPU 服务器的 JSON Body 直接使用了变量插值：
   ```json
   {
     "text": "={{ $json.query.text }}"
   }
   ```
4. **模型收到原始转码文本**：本地运行在 50000 端口的 Python FastAPI 服务直接收到了一个含有几十个 `%` 符号的纯 ASCII 字符串 `"%E6%81%AD..."`。

---

## 3. 为什么现代神经 TTS 不会自动转义？

在早期的传统规则式 TTS 中，如果传入奇怪的编码字符，系统往往会直接报错或丢弃非中英文字符。

但像 CosyVoice 2 或 GPT-SoVITS 这类新一代基于大语言模型架构（LLM + Flow Matching / Diffusion）的 TTS 系统，其文本前端包含复杂的**文本规范化模块（Text Normalization）**。为了让模型能够读懂各种数学符号、技术参数与代码片段，它的分词器将 `%` 显式映射为了发音为 **“Percent”** 的有效 Token。

因此，对 TTS 模型来说：
- 它根本不知道上游传入的是经过 URL 编码的汉字。
- 它只看到文本里写着一个个真实的百分号、大写字母 E 和数字 6。
- 于是，它严格按照规范将一段 15 个字的中文句子，当成了包含几十个数学符号和英文序列的混排公式，认认真真地读出了每一个 “Percent”。

---

## 4. 根因剖析 2：N8N 沙盒表达式的隐形暗坑

发现问题后，第一反应通常是在 n8n 的 HTTP 节点表达式中直接加一句 JS 解码：

```javascript
// 表面合理的写法：
{{ decodeURIComponent($json.query.text) }}
```

但这会立刻引发二次故障：
1. **POST 请求断流**：如果调用方改用 POST JSON 发送请求，`$json.query` 会变成 `undefined`，直接报错 `Cannot read property 'text' of undefined`。
2. **ES 语法特性支持差异**：在较老或某些特殊打包的 n8n 环境中，内嵌的 JavaScript 解释器对可选链操作符（如 `$json.query?.text`）的支持并不稳定，极易出现 `Unexpected token '.'` 语法错误。
3. **URIError 崩溃风险**：如果用户传入的文本本身包含合法的独立百分号（例如“*降价 50%*”），未经包装的原生 `decodeURIComponent("50%")` 会直接抛出 `URIError: URI malformed`，导致整个工作流执行中断。

---

## 5. 双层防御方案

为了让整条语音合成链路彻底免受各类编码问题的困扰，我们在编排层与模型层同时设置了兜底防线。

### 第一道防线：N8N Code 节点独立预清洗

在 Webhook 节点之后、HTTP 转发节点之前，插入一个纯 JavaScript **Code 节点**，统一承接 GET 与 POST 数据并做容错解码：

```javascript
const item = $input.first().json;
const query = item.query || {};
const body = item.body || {};

let text = query.text || body.text || '你好，本地语音测试。';
let voice = query.voice || body.voice || 'default';

// 仅在文本包含百分号时尝试解码，并在异常时优雅回退
if (typeof text === 'string' && text.indexOf('%') !== -1) {
  try {
    text = decodeURIComponent(text.replace(/\+/g, ' '));
  } catch (err) {
    // 包含非法百分号组合时保持原文，避免流程崩溃
  }
}

return [{
  json: {
    text: text.trim(),
    voice: String(voice).trim()
  }
}];
```

### 第二道防线：后端 Python 服务幂等兜底

在本地运行的 FastAPI 接口实现中（`webui.py`），对传入的 `text` 参数增加一层无条件但幂等的 `urllib.parse.unquote`：

```python
import urllib.parse

@app.post("/tts")
async def api_tts(request: Request):
    body = await request.json()
    raw_text = body.get("text", "").strip()
    
    # 无论上游是否已经解码，在此做一次幂等安全解码
    text = urllib.parse.unquote(raw_text).strip()
    voice = body.get("voice", "default")
    
    # 送入推理管线...
```

`unquote` 是幂等的：如果文字已经是纯中文，处理后原样返回；如果有任何遗漏的 URL 编码，会被彻底还原为真实汉字，绝不把 `%` 留给下层的分词与发音模块。

---

## 6. 验证与总结

在部署好双层防护后，我们重新使用同样的带 URL 编码命令请求接口：

```bash
curl -G "http://192.168.1.1:5678/webhook/ltts" \
  --data-urlencode "text=恭喜！本地模型通过 N8N 联动测试成功！" \
  --data-urlencode "voice=child" \
  -o test_clean.wav
```

重新将生成的音频送入语音识别转写：

```json
{
  "text": "恭喜！本地模型通过N8N联动测试成功！"
}
```

所有的 “Percent” 杂音彻底消失，取而代之的是清脆甜美的正常女童声。

**工程经验**：
在搭建涉及“网络网关 → 自动化编排器 → AI 神经推理模型”的多层系统时，绝不能假设到达 API 边界的数据格式天然符合预期。在网关层做安全清洗、在模型接口层做幂等兜底，两行轻量代码就能彻底杜绝 AI 模型在半夜把网络请求头念给用户听的尴尬事故。

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "Why Is My TTS Model Repeating \"Percent\"? The URL-Encoding Trap in Speech Pipelines"
description: "Why our newly deployed local TTS engine synthesized endless 'percent' repetitions instead of spoken Chinese, and how character normalization and n8n sandbox limits collided."
pubDate: 2026-10-22
category: notes
blueprint: ai-agent-automation
blueprintLayer: L1-deterministic-orchestration
tags: ["tts", "debugging", "n8n", "urlencode", "python", "ai"]
ogImage: "/og/why-is-my-tts-model-repeating-percent.jpg"
banner: "/banners/why-is-my-tts-model-repeating-percent.jpg"
draft: false
---

We had just wired Alibaba's open-source CosyVoice 2 into our self-hosted automation cluster. The direct Python test scripts worked cleanly, but when we fired our first test request through a Synology-hosted n8n Webhook, the audio response sounded like an alien countdown.

Instead of the Chinese phrase *"恭喜！本地模型通过 N8N 联动测试成功！"* (Congratulations! Local model test via n8n succeeded!), the speaker rattled off a robotic avalanche:

> *"Percent E 6, percent 81, percent 81, percent B 9, percent A 7, percent 9... percent..."*

For fifteen agonizing seconds, the model faithfully enunciated dozens of hexadecimal percentage codes.

Here is the post-mortem of why this happened, why neural speech text frontends amplify subtle encoding oversights, and the two defensive layers required to make speech webhooks robust.

---

## 1. The Symptom: The "Percent" Loop

We tested the n8n endpoint via a standard HTTP GET request from a terminal:

```bash
curl -G "http://192.168.1.1:5678/webhook/ltts" \
  --data-urlencode "text=恭喜！本地模型测试成功！" \
  --data-urlencode "voice=child" \
  -o test.wav
```

The server returned HTTP 200 and a valid 2.4MB WAV audio file. But when passed to a speech-to-text transcriber to inspect the synthesized words, the transcription revealed the disaster:

```json
{
  "text": "percent E 6 percent 81 percent 81 percent B 9 percent A 7 percent..."
}
```

The model had not hallucinated; it had pronounced the exact string it received with meticulous phonetic clarity.

---

## 2. Root Cause 1: Double-Encoding & Missing Query Decoding

To understand why, you have to trace how query parameters travel through a webhook:

1. When `curl` (or a web browser) sends a GET request, Unicode characters are percent-encoded into ASCII octets (UTF-8 bytes formatted as `%XX`). For example, the Chinese character `恭` (Unicode `U+606D`) becomes `%E6%81%AD`.
2. When the n8n Webhook node received the GET query parameters, it passed the raw string directly into the node context:
   ```json
   {
     "query": {
       "text": "%E6%81%AD%E5%96%9C%EF%BC%81..."
     }
   }
   ```
3. In our initial n8n HTTP Request node, the outgoing JSON body was constructed with a direct template string:
   ```json
   {
     "text": "={{ $json.query.text }}"
   }
   ```
4. n8n serialized this raw percent-encoded string into the JSON payload forwarded to our local FastAPI service on `127.0.0.1:50000`.
5. The Python backend received the string `"%E6%81%AD..."` literally.

---

## 3. Why LLM-Based Speech Frontends Don't Auto-Decode

In traditional text processing, non-ASCII characters might trigger an encoding error or render as replacement diamonds (``). 

However, modern neural TTS systems like CosyVoice 2 and GPT-SoVITS rely on **Text Normalization (TN)** modules and byte/token vocabularies designed to read arbitrary technical jargon, symbols, and mathematical notation.

When the Text Normalizer encountered `%`:
- It did **not** treat `%E6%81` as an escape sequence.
- It treated `%` as the literal mathematical symbol **"percent"**.
- It treated `E`, `6`, and `8` as individual alphanumeric characters.

Consequently, a 12-character Chinese sentence was translated into a 36-token alphanumeric tongue-twister. The model simply did what it was trained to do: speak whatever was written.

---

## 4. Root Cause 2: The N8N Sandbox Trap

We attempted to fix this inside n8n by adding an inline expression:

```javascript
// The trap inside n8n node expression:
{{ decodeURIComponent($json.query.text) }}
```

This immediately broke on two edge cases:
1. **POST requests don't have `$json.query`**: Accessing `$json.query.text` on a POST JSON call threw `Cannot read property 'text' of undefined`.
2. **Optional Chaining (`?.`) syntax limitations**: In older embedded V8 runtimes used by certain n8n releases, modern ECMAScript features like `$json.query?.text` can cause silent parse failures or throw `Unexpected token '.'`.
3. **Double-decoding URI errors**: If the input string contains an unescaped `%` (such as *"The price dropped by 50%"*), a naked `decodeURIComponent("50%")` throws an unhandled `URIError: URI malformed`, terminating the entire workflow.

---

## 5. The Two-Layer Defense

To make the speech pipeline bulletproof against any client library, we implemented defensive decoding at both the orchestration layer and the model backend.

### Layer 1: N8N Code Node Pre-Processing

Between the Webhook node and the HTTP Request node, we inserted a dedicated JavaScript **Code Node** that safely extracts and sanitizes incoming text from either GET queries or POST bodies:

```javascript
const item = $input.first().json;
const query = item.query || {};
const body = item.body || {};

let text = query.text || body.text || '语音测试';
let voice = query.voice || body.voice || 'default';

// Safe percent-decoding with error fallback
if (typeof text === 'string' && text.indexOf('%') !== -1) {
  try {
    text = decodeURIComponent(text.replace(/\+/g, ' '));
  } catch (err) {
    // Retain original string if it is not valid percent-encoding
  }
}

return [{
  json: {
    text: text.trim(),
    voice: String(voice).trim()
  }
}];
```

### Layer 2: Backend Python Fallback

On the receiving FastAPI side (`webui.py`), we added an unconditional `unquote` step in the request handler:

```python
import urllib.parse

@app.post("/tts")
async def api_tts(request: Request):
    body = await request.json()
    raw_text = body.get("text", "").strip()
    
    # Defensive unquote handles both encoded and pre-decoded strings
    text = urllib.parse.unquote(raw_text).strip()
    voice = body.get("voice", "default")
    
    # Pass clean string to the audio generator...
```

`urllib.parse.unquote` is idempotent: if the string is already plain Chinese or English, it returns it unchanged. If residual percent signs from an upstream proxy survive, they are decoded before hitting the tokenizer.

---

## 6. The Result

After deploying both layers, we re-ran the exact same `curl` query:

```bash
curl -G "http://192.168.1.1:5678/webhook/ltts" \
  --data-urlencode "text=恭喜！本地模型通过 N8N 联动测试成功！" \
  --data-urlencode "voice=child" \
  -o test_clean.wav
```

We verified the output through our local Whisper/SenseVoice ASR engine:

```json
{
  "text": "恭喜！本地模型通过N8N联动测试成功！"
}
```

The "percent" loop vanished, replaced by crisp, natural speech.

When building AI pipelines that bridge HTTP gateways with neural text frontends, never assume the string arriving at your API boundary is in its final semantic form. Defensive decoding at both ends of the bridge costs fractions of a millisecond—and prevents your AI from reading URL headers as bedtime stories.

文章 13 / 16：为什么多个 AI 一起写代码总会烂尾？传话失真与隐式决策困境

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "为什么多个 AI 一起写代码总会烂尾？传话失真与隐式决策困境"
description: "为什么让多个自主 AI Agent 协同开发必然导致分支踩踏与架构漂移？深度复盘「传话游戏」与「隐式决策冲突」两大不可逾越的机制死穴。"
pubDate: 2025-07-29
updatedDate: 2026-10-25
category: notes
blueprint: ai-agent-automation
blueprintLayer: L2-llm-gateway-agents
tags: ["ai", "agents", "architecture", "debugging"]
ogImage: "/og/zh-why-multi-agent-coding-fails.jpg"
banner: "/banners/why-multi-agent-coding-fails.jpg"
draft: false
---

## 美好的愿景 vs 残酷的现实

多 Agent 协同编程的蓝图听起来极具诱惑力：
- 将一份庞大的用户需求拆分成若干子任务；
- 启动一个主管 Agent；
- 由主管同时分发任务：一个 Agent 去写数据库迁移，一个写后端 REST 路由，另一个写前端界面；
- 开发者只需喝杯咖啡，三个自主进程并行提交，几分钟就能交付一套全栈特性。

如果你真正在哪怕稍微有点规模的实际代码库中实操过，你一定清楚最终的下场：数据库迁移里的字段和 ORM 模型对不上；后端接口返回的是 `snake_case`，前端却死死按 `camelCase` 解析；而 Git 在共享的路由注册与依赖文件上直接爆出了十几个冲突，整个工作树彻底卡死。

多 Agent 蜂群协同写代码的失败，根本不是模型“智商不够”，而是撞上了两个无法逾越的系统动力学死穴：**传话游戏（The Telephone Game）** 与 **隐式架构决策漂移（Implicit Architectural Drift）**。

---

## 1. 传话游戏：逐级抽象导致的信息大滑坡

大家小时候都玩过传话游戏：一句话悄悄传过 5 个人的耳朵，到了最后一个人嘴里，往往变成了完全荒诞的另一句话。

在多 Agent 协同架构中，这种信息失真会以极其机械的方式必然发生，因为分发任务必然伴随序列化与有损抽象：

```
[人类原始意图：边界防御、性能瓶颈、历史遗留妥协、未成文惯例]
                                 │
                                 ▼ (Prompt 序列化有损压缩)
                        [主管 Agent]
                                 │
                                 ▼ (拆解任务，生成摘要任务便签)
                       [执行子 Agent]
```

当主管 Agent 向子 Agent 分派任务时，它无法把自己的全部工作记忆（Context）整体灌过去，而只能写出一张简短的“任务便签”。

此时子 Agent 面临着极度不对称的信息黑盒：
- 它只知道便签上写的那几句冷冰冰的字面指令；
- 它完全不知道人类在几轮前反复强调过的深层业务痛点；
- 它不理解为什么主管在前几轮调试中坚决否决了另外三种技术方案。

子 Agent 确实极其忠诚且完美地完成了便签上的字面要求，但把它放回全局系统中，却成了一个格格不入的异类。

---

## 2. 隐式架构决策漂移：Flappy Bird 困局

Cognition（Devin 开发团队）曾在 2025 年的一场技术讲座中举过一个绝妙的例子：**用两个 Agent 协同开发一款类似《Flappy Bird》（像素鸟）的小游戏**。

主管 Agent 将分工划分得极其干净利落：
- **Agent A**：负责背景画面、障碍物水管与滚动视角渲染；
- **Agent B**：负责玩家操控的小鸟、飞行物理引擎与碰撞检测逻辑。

两个 Agent 都领到了清晰的便签，开始全速并行敲代码。

最终拼装代码时，荒诞的一幕发生了：
- Agent A 自行选用了类似超级马里奥的 16-bit 像素复古瓦片风；
- Agent B 则在 HTML5 Canvas 上用三次贝塞尔曲线画出了一只极简、高保真的矢量扁平小鸟。

两边各自都没有犯错，两边的单元测试全部顺利通过。但因为编写每一行代码都在暗中包含成百上千个“未显式声明的微小决策”——命名规范、视觉风格、错误处理语义、异步约定——两个没有进行锁步同步的 Agent，必然会朝着完全相反的架构方向渐行渐远。

---

## 3. 协同开销彻底吞噬吞吐量

推崇多 Agent 的最常见理由是“并行”：同时开 10 个 Agent，吞吐量理论上应该翻 10 倍。

但在软件开发领域，这一假设直接违背了系统理论的基本规律（阿姆达尔定律在可变状态上的体现）：

```
实际吞吐收益 = 并行执行速度 - (协同沟通损耗 + Git 分支合并开销)
```

当 5 个 Agent 在同一个代码库里自由穿梭时：
1. **文件并发写争抢**：多个 Agent 频繁修改同一个依赖配置文件、全局类型中心与路由字典。
2. **状态无效化**：Agent A 重构了接口，而 Agent B 还在基于旧接口痛苦地生成实现类。
3. **死循环合并**：为了消除三方合并冲突，自主 Agent 经常会简单粗暴地抹掉同伴的代码，甚至凭空捏造未导出的变量名，只为了强行换取一个 `exit 0` 的假象。

正如 Factory 工程师 Luke Veenhuis 的一针见血的总结：*“协同开销最终吞掉了所有速度收益，顺便把你的 Token 烧了个精光。”*

---

## 破局：多 Agent 真正适用的三大场景

这是否意味着多 Agent 在软件工程中毫无用处？完全不是。关键在于确立核心铁律：**绝对不能让多个 Agent 对同一个代码状态拥有并发写权限**。

多 Agent 协同在满足以下三个条件时，才能真正发挥百倍威力：
1. **分块且只读（Sharded & Read-Only）**：如批量抓取第三方文档、多源资料深度调研、检索开源代码依赖，这些操作完全没有写入冲突。
2. **明确的数据管道契约**：子 Agent 仅负责生成独立素材（如生成一张 SVG 架构图，或调度本地 ComfyUI 出图），结果落地为静态文件并只返回文件路径句柄。
3. **独立对抗式守门**：由拥有空白纯净记忆的高阶推理大模型对单写主控生成的 Git Diff 进行对抗式挑刺，完全不参与编写泥潭。

**写操作永远单线程，只读与外部素材放手并行**——这才是规避蜂群混乱、稳定交付高质量软件的唯一工程正道。

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "Why Multi-Agent Coding Fails: The Telephone Game and Architectural Drift"
description: "Why swarms of autonomous coding agents step on each other, play the telephone game, and make implicit conflicting decisions that break repositories."
pubDate: 2025-07-29
updatedDate: 2026-10-25
category: notes
blueprint: ai-agent-automation
blueprintLayer: L2-llm-gateway-agents
tags: ["ai", "agents", "architecture", "debugging"]
ogImage: "/og/why-multi-agent-coding-fails.jpg"
banner: "/banners/why-multi-agent-coding-fails.jpg"
draft: false
---

## The Vision vs. The Reality

The pitch for multi-agent software engineering is undeniably seductive: 
- Break a massive user story into subtasks.
- Spawn a manager agent.
- Let the manager spin up a worker agent for the database, another for the REST API, and a third for the frontend.
- Sip your coffee while three autonomous processes concurrently commit code and ship a full-stack feature.

If you have actually attempted this on a non-trivial production codebase, you already know the ending: the database migration doesn't match the ORM models, the API route returns snake_case while the frontend expects camelCase, and Git aborts with 18 merge conflicts across shared registration files.

The failure of multi-agent coding swarms is rarely due to raw model intelligence. It is caused by two fundamental mechanics: **The Telephone Game** and **Implicit Architectural Drift**.

---

## 1. The Telephone Game: Lossy Human-to-Agent Delegation

In human childhood games, a sentence whispered through five children turns into absurd gibberish by the time it reaches the end of the line.

In multi-agent systems, the exact same distortion occurs because delegation requires serialization:

```
[Human Intent: Edge cases, performance bounds, legacy quirks, unspoken conventions]
                                 │
                                 ▼ (Prompt Serialization)
                     [Lead Orchestrator Agent]
                                 │
                                 ▼ (Subtask Breakdown & Summarized Prompt)
                       [Worker Subagent]
```

When the orchestrator delegates a subtask to Subagent B, it does not transfer its full internal working memory. It writes a brief, lossy instruction slip.

The subagent operates under severe information asymmetry:
- It knows only what was written on the prompt slip.
- It lacks awareness of the subtle constraints mentioned in earlier conversational turns.
- It does not understand why the lead agent rejected three previous architectural approaches.

The subagent executes flawlessly according to its prompt slip, yet fails completely within the holistic system.

---

## 2. Implicit Architectural Drift: The Flappy Bird Dilemma

Cognition (the team behind Devin) highlighted a brilliant analogy during a 2025 tech lecture: **Building a Flappy Bird Clone with Two Agents**.

Suppose the manager agent divides the work cleanly:
- **Agent A**: Build the background scenery, pipe obstacles, and scrolling camera.
- **Agent B**: Build the player character (the bird), flight physics, and collision detection.

Both agents receive clear prompt instructions. Both execute in parallel.

When the two halves are assembled:
- Agent A chose a 16-bit retro Super Mario visual aesthetic with tiled pixel assets.
- Agent B built a sleek, flat minimalist vector bird using HTML5 canvas bezier curves.

Neither agent made a mistake. Both followed instructions. But because every line of software code embodies hundreds of unstated, implicit decisions—conventions, styling, naming, error semantics—two agents working without lockstep synchronization will drift apart in opposite architectural directions.

---

## 3. Coordination Overhead Devours Throughput

The most common justification for multi-agent systems is parallelism: if you run 10 agents simultaneously, you should get 10x throughput.

In software development, this assumption violates basic systems theory (Amdahl's Law applied to state mutation):

```
Throughput Gains = Parallel Execution Speed - Coordination & Merge Overhead
```

When 5 agents modify the same repository:
1. **File Contention**: Agents edit the same package manifests, configuration dictionaries, and routing tables.
2. **Lock Invalidation**: Agent A refactors a shared interface while Agent B is halfway through generating implementations against the old interface.
3. **Merge Loops**: An agent trying to resolve a 3-way git merge will often hallucinate variable names or silently wipe out a teammate's changes just to produce a clean exit code.

As Factory engineer Luke Veenhuis famously summarized: *"Coordination overhead ends up eating up the speed gains, all the while burning through tokens."*

---

## The Path Forward: Where Multi-Agent Actually Works

Does this mean multi-agent architectures have no place in engineering? Absolutely not. It simply means **agents must never share concurrent write access to the same state**.

Multi-agent architecture is exceptional when tasks satisfy three criteria:
1. **Sharded and Read-Only**: Tasks like scraping API documentation, verifying test logs, or querying dependencies can run completely in parallel.
2. **Clear Schema Boundaries**: Subagents generate isolated assets (e.g., an SVG diagram or an image via local diffusion) that are written to deterministic paths and returned as static handles.
3. **Adversarial Gatekeeping**: A fresh, unpolluted reasoning model reviews the single writer's output without participating in the authoring struggle.

Restrict file writes to a single thread, use subagents strictly for read-only parallel work, and your pipeline will ship code reliably without the chaos.

文章 14 / 16：为什么你的 Windows 桌面 MCP 调用会静默失败：攻破 10 秒空闲门禁与避坑实录

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "为什么你的 Windows 桌面 MCP 调用会静默失败：攻破 10 秒空闲门禁与避坑实录"
description: "排查 windows-mcp 调用返回 CONTROL_PREEMPTED 的深层机制，以及为什么必须通过环境变量彻底禁用桌面控制门禁。"
pubDate: 2026-03-12
updatedDate: 2026-10-25
category: notes
tags: ["windows", "mcp", "debugging", "ai-agents", "troubleshooting"]
ogImage: /og/zh-why-your-windows-mcp-calls-silently-fail.jpg
banner: /banners/why-your-windows-mcp-calls-silently-fail.jpg
draft: false
---

你刚装好 `windows-mcp`，将其加到 Agent 的配置中，然后让它发送一条 Windows 系统通知，或者点击某个打开的应用窗口按钮。

工具调用在不到 200 毫秒内就返回了，大模型语气轻松地向你汇报：*“通知已成功发送。”*

**然而屏幕上什么都没有发生。**

当你把 MCP 协议传输的底层原始 JSON 结果打印出来时，既没有看到标准的 Python 异常，也没有看到 Win32 系统错误，而是看到这样一段内容：

```json
{
  "code": "CONTROL_PREEMPTED",
  "status": {
    "state": "user",
    "generation": 3,
    "wait_seconds": 9.8639,
    "executed_steps": 0
  }
}
```

或者在不同的时序下，后台进程直接抛出一串异常堆栈：

```text
RuntimeError: AI control indicator is stopping
AI control indicator failed; releasing physical input
```

这就是导致大部分桌面 MCP 调用静默失败的隐蔽真凶。

---

## 根因剖析：桌面控制门禁（Desktop Control Policy）

`windows-mcp` 内部设计了一套**桌面所有权控制机制**（位于 `src/windows_mcp/desktop/control_overlay.py` 与 `tools/control_policy.py`）。

开发者为了避免 AI 在物理桌面上与正在操作鼠标键盘的真人“抢夺输入焦点”，实现了一个空闲检测看门狗：

1. 在执行任何具有副作用的输入工具（`Click`、`Type`、`Move`、`Shortcut`、`Notification` 等）前，看门狗会通过 Win32 API `GetLastInputInfo` 读取最后一次物理输入的时间戳。
2. 如果人类在过去 **10 秒**内（`WINDOWS_MCP_CONTROL_IDLE_TIMEOUT`）移动过鼠标或敲击过键盘，服务端判定此时“人类正在使用电脑”。
3. 此时它**既不排队也不等待**，而是立刻主动放弃执行，直接返回 `CONTROL_PREEMPTED`（控制权被抢占），工具实际执行步骤数为 0。

但现实使用场景是：你在终端敲完回车下达指令，或者在聊天框点下发送——**这意味着你刚刚操作过键盘鼠标**。你的系统空闲时间必然小于 1 秒，因此 Agent 随后的第一次工具调用几乎 100% 触发抢占拦截，导致操作直接被吞。

---

## 为什么 `--no-control` 参数不管用

翻看该项目的 README 或文档，你可能会看到关于 `--control / --no-control` 的说明。

很多人第一反应是给命令加上 `--no-control`：

```bash
uvx windows-mcp serve --no-control
```

但服务会在启动瞬间直接崩溃报错：

```text
Usage: windows-mcp serve [OPTIONS]
Error: No such option '--no-control'.
```

这是因为该项目基于 Click 库构建 CLI，`--no-control` 只绑在顶层或其他模式上，`serve` 子命令的参数解析器中根本没有注册这个 CLI 标志。

---

## 真正有效的解法：WINDOWS_MCP_CONTROL=false

通过审查代码可以发现，配置加载模块（`src/windows_mcp/config.py`）会优先读取系统环境变量。

要彻底绕过这道 10 秒空闲限制，允许 Agent 在下达指令后立即获得执行权限，只需注入环境变量：

```bash
WINDOWS_MCP_CONTROL=false
```

将其填入 Agent 的配置文件 `env` 字段：

```json
{
  "command": "uvx",
  "args": ["windows-mcp", "serve", "--exclude-tools", "PowerShell,Registry"],
  "env": {
    "WINDOWS_MCP_CONTROL": "false"
  }
}
```

此时控制层覆盖（Overlay）、空闲看门狗以及拦截抢占逻辑将被完全旁路关闭。

---

## 验证与验收

为了验证问题已彻底根除，在鼠标持续移动的状态下通过原生 Python MCP 客户端发起 Toast 通知测试：

```python
import asyncio
from mcp.client.stdio import stdio_client, StdioServerParameters
from mcp.client.session import ClientSession

async def main():
    params = StdioServerParameters(
        command="uvx",
        args=["windows-mcp", "serve", "--exclude-tools", "PowerShell,Registry"],
        env={"WINDOWS_MCP_CONTROL": "false", "ANONYMIZED_TELEMETRY": "false"}
    )
    async with stdio_client(params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            res = await session.call_tool("Notification", arguments={
                "title": "Windows-MCP Ready",
                "message": "Immediate execution verified.",
                "app_id": "Hermes Agent"
            })
            for c in res.content:
                print(c.text)

asyncio.run(main())
```

终端实时输出：
```text
Notification sent: "Windows-MCP Ready" - Immediate execution verified.
```

无需苦等 10 秒倒数，再无 `CONTROL_PREEMPTED`，所有桌面动作即时执行。

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "Why Your Windows Desktop MCP Calls Silently Fail: Taming the 10-Second Idle Gate"
description: "Why actions in windows-mcp return CONTROL_PREEMPTED or fail silently, and the exact flag needed to disable the desktop control gate."
pubDate: 2026-03-12
updatedDate: 2026-10-25
category: notes
tags: ["windows", "mcp", "debugging", "ai-agents", "troubleshooting"]
ogImage: /og/why-your-windows-mcp-calls-silently-fail.jpg
banner: /banners/why-your-windows-mcp-calls-silently-fail.jpg
draft: false
---

You install `windows-mcp`, register it in your agent config, and ask your model to trigger a native Windows notification or click a button in an open desktop application.

The tool call finishes in under 200 milliseconds. The agent happily reports: *"I have sent the notification."*

**Nothing appeared on screen.**

When you inspect the raw JSON response returned by the MCP protocol, you don't find a standard Python exception or Win32 error. Instead, you see this:

```json
{
  "code": "CONTROL_PREEMPTED",
  "status": {
    "state": "user",
    "generation": 3,
    "wait_seconds": 9.8639,
    "executed_steps": 0
  }
}
```

Or, in slightly different timing conditions, your stdio process dumps a stack trace:

```text
RuntimeError: AI control indicator is stopping
AI control indicator failed; releasing physical input
```

Here is the root cause of this failure mode and the exact configuration setting that fixes it.

---

## The Culprit: The Desktop Control Gate

`windows-mcp` is designed around a concept called the **Desktop Control Policy** (implemented in `src/windows_mcp/desktop/control_overlay.py` and `tools/control_policy.py`).

The developers wanted to prevent an AI agent from fighting the human user for physical mouse and keyboard ownership. To achieve this, the server installs an idle detection watchdog:

1. Before executing any state-changing input tool (`Click`, `Type`, `Move`, `Shortcut`, `Notification`, etc.), it checks Windows user input timers (`GetLastInputInfo`).
2. If the user moved the physical mouse or pressed a keyboard key within the last **10 seconds** (`WINDOWS_MCP_CONTROL_IDLE_TIMEOUT`), the server considers the human to be actively using the machine.
3. Instead of waiting for you to stop typing, it **aborts immediately**, returning `CONTROL_PREEMPTED` and leaving the action unexecuted.

If you are chatting with your agent in a desktop window or typing commands in a terminal, **you just touched the keyboard**. Your idle timer is zero. Therefore, almost every single immediate tool call the agent issues will be preempted by default.

---

## Why `--no-control` Fails

If you inspect the repository's README or CLI documentation, you might see references to `--control / --no-control`.

Naturally, you might try adding `--no-control` to your server startup command:

```bash
uvx windows-mcp serve --no-control
```

The server instantly crashes on startup with:

```text
Usage: windows-mcp serve [OPTIONS]
Error: No such option '--no-control'.
```

The CLI parser for the `serve` subcommand only exposes transport, network, and tool exclusion flags (`--transport`, `--host`, `--port`, `--exclude-tools`). It does not bind the control flag to the CLI argument group for `serve`.

---

## The Real Fix: WINDOWS_MCP_CONTROL=false

The configuration loader (`src/windows_mcp/config.py`) reads desktop control settings directly from environment variables.

To completely bypass the 10-second idle lockout and allow the agent to execute actions immediately upon request, set:

```bash
WINDOWS_MCP_CONTROL=false
```

When injected into your MCP client's environment block:

```json
{
  "command": "uvx",
  "args": ["windows-mcp", "serve", "--exclude-tools", "PowerShell,Registry"],
  "env": {
    "WINDOWS_MCP_CONTROL": "false"
  }
}
```

The control overlay, idle watchdog, and preemptive preemption logic are completely disabled.

---

## Verification

To confirm the fix, send a toast notification through the raw MCP client while actively moving your mouse:

```python
import asyncio
from mcp.client.stdio import stdio_client, StdioServerParameters
from mcp.client.session import ClientSession

async def main():
    params = StdioServerParameters(
        command="uvx",
        args=["windows-mcp", "serve", "--exclude-tools", "PowerShell,Registry"],
        env={"WINDOWS_MCP_CONTROL": "false", "ANONYMIZED_TELEMETRY": "false"}
    )
    async with stdio_client(params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            res = await session.call_tool("Notification", arguments={
                "title": "Windows-MCP Ready",
                "message": "Immediate execution verified.",
                "app_id": "Hermes Agent"
            })
            for c in res.content:
                print(c.text)

asyncio.run(main())
```

Output:
```text
Notification sent: "Windows-MCP Ready" - Immediate execution verified.
```

No 10-second wait, no `CONTROL_PREEMPTED`, and no silent drops.

文章 15 / 16：校准 Intel Quick Sync HEVC：28 与 30 之间的悬崖，和一个毫无作用的 preset

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "校准 Intel Quick Sync HEVC：28 与 30 之间的悬崖，和一个毫无作用的 preset"
description: "我在真实 1080p 文件上量了 QSV HEVC：ICQ 18–22 会把瘦文件撑大，质量悬崖落在 28 与 30 之间，而 fast 与 medium 逐字节同大小。"
pubDate: 2026-10-14
updatedDate: 2026-10-16
category: tutorials
tags: ["ffmpeg", "hevc", "qsv", "transcoding", "video"]
ogImage: /og/zh-calibrating-intel-quick-sync-hevc-quality.jpg
banner: /banners/calibrating-intel-quick-sync-hevc-quality.jpg
draft: false
---

## 一个我回答不了的问题：到底该出货哪个数字？

我的转码队列按文件逐个决定用哪个编码器：源文件本来就高效就用 `copy`、GPU 走 NVENC、CPU 走 x265——现在又多了一个 Intel Quick Sync。每一档都有两个决定质量的旋钮，除了 QSV，其他每一档我都有在自己片库上量过的数字。

QSV 一个都没有。而代码拒绝猜：出货默认值是 `qsv_quality = 0` / `qsv_preset = ""`，含义是「没人标定过这一档」，引擎把它当成硬排除——于是 QSV 可以存在于代码里，却在任何机器上都不会被选中，包括真的有这块显卡的那台。

所以任务不是「把 QSV 打开」，而是「为两个旋钮产出量过的数字，让这个编码器能被诚实地选中」。

## 问题一：硬件不在跑这个程序的机器里

队列跑在 Windows 11 虚拟机里。那台 VM 直通了一张 RTX 3060，没有任何 Intel 核显——这正是一直没量过 QSV 的原因：最顺手的主意就是在本地跑一次、把数字读出来。

核显在宿主上（unRaid，i9-13900K），而且空闲、无人占用：

```text
00:02.0 VGA compatible controller: Intel Raptor Lake-S GT1 [UHD Graphics 770]
  driver: i915            refcount: 0
/dev/dri/card0  /dev/dri/renderD128        （存在，全局可读）
没有任何运行中的容器声明 dri 设备
virsh dumpxml Win11: hostdev = 01:00.0, 01:00.1（3060 + 它的 HDMI 音频）
```

两个结论都很有用。VM 里那句 `Error creating a MFX session` 是**预期行为**，不是一个要去查的 bug——那台 VM 永远看不到这块核显。而量 QSV 的唯一办法，是在宿主上、在容器里、用 Linux 侧去量。

## 用量尺的那个人自己的闸门来量，而不是用 ffmpeg

这是让上面这些数字真正有价值的部分。我没有自己写 ffmpeg 命令、自己算 SSIM——整场标定跑的是**将来真正干活的那个引擎本体**，带着它自己的校验和自己的判收闸门：

```bash
python3 library_run.py --work-dir /work --tier qsv --force-tier qsv \
  --qsv-quality Q --qsv-preset P --path <真原件> --keep-local --overwrite
```

`--keep-local` 的意思是：编码、校验、过闸门，但**不落盘**——我收集的是判决，不是一批新文件。素材通过已有的 SMB 挂载只读进入，而且每个源档在编码前后都做了哈希核对：未变。

镜像是 `linuxserver/ffmpeg`（ffmpeg 9.0.2、iHD 26.3.5、oneVPL 2.17）加 `python3`，再加 **numpy**——后者不是可选项。没有它闸门会打 `flat-frame scan skipped: No module named 'numpy'`，悄悄改变近平坦帧的判定方式，也就改变了判决。

`-global_quality` 在 `hevc_qsv` 上对应 `RateControlMethod: ICQ`，这就是被标定的旋钮。

## 质量阶梯：在瘦源上，低 ICQ 会把文件**变大**

第一轮，源是一支真实的 1080p25、951 kbps（bpp 0.0184——已经很瘦、榨过一遍的那种）：

| ICQ | 引擎判决 | mean SSIM | 视频 = 源 |
|---|---|---|---|
| 18 | **FAIL size** — 176.8% | — | — |
| 20 | **FAIL size** — 141.8% | — | — |
| 22 | **FAIL size** — 107.6% | — | — |
| 24 | ok | 0.9967 | 81.1% |
| 26 | ok | 0.9959 | 61.0% |
| 28 | ok | 0.9951 | 46.4% |
| 30 | **FAIL quality** | 0.9942 | — |
| 32 | **FAIL quality** | 0.9942 | — |

这张表掉出三个结论。

第一个反直觉的：**低 ICQ 不等于安全**。ICQ 18–22 产出的文件比源还**大**，于是被引擎的体积闸门（源视频码率的 100%）拒掉。这不是 QSV 独有的毛病——NVENC 对瘦输入用低 `cq` 也是同一个形状——但这里的边界更靠上：可用区间从 24 起，不是 18。

第二个，**28 与 30 之间有一道悬崖**。30 和 32 都质量失败，而且失败在**首次尝试**。这句话很重要，因为引擎在质量失败时会用 `global_quality - 2` 重试一次——ICQ 更低、质量更高——并把重试写进日志。一行写着「ok at quality+2」的日志其实是**首次尝试失败**，不是通过。我第一遍在自己的笔记里把这条读反了；规矩是读首次尝试，不读救济结果。

第三个，地板决定取舍：在 **0.995 的 mean SSIM 下限**下，q28 的余量是 0.0001，q26 的余量是 0.0010。

## 完整文件确认了这张阶梯——而且有一个捷径站得住

摘录（180 秒无损切片）是「不用跑十次完整编码就能排出候选顺序」的办法。它只有在与完整文件结果一致时才成立，所以我把真原件端到端跑了一遍：

| 源 | ICQ | preset | 视频 = 源 | 文件 | SSIM min / mean | 编码 |
|---|---|---|---|---|---|---|
| 254 MB，2023 秒，872 kbps | 26 | medium | **61.0%** | 168,435,266 B（66%） | 0.9938 / **0.9960** | 360 秒 ≈ 5.6× 实时 |
| 同一支 | 28 | medium | 46.3% | 54% | 0.9927 / 0.9952 | 330 秒 |
| 315 MB，3280 秒，546 kbps | 26 | medium | 50.4% | 60% | 0.9965 / **0.9977** | 600 秒 ≈ 5.5× 实时 |

第一支的 180 秒摘录量到 mean 0.9959，完整文件量到 0.9960。这个接近程度足以让人相信「摘录用来排质量档」这条路——而且现在这是有证据的，不是靠希望。

## 那个毫无作用的 preset

摘录那一层给出的故事很干净：preset 只影响速度——veryfast 30 秒 / 61.6%、fast 30 秒 / 61.0%、medium 60 秒 / 61.0%、slow 与 veryslow 60 秒。结论似乎就是选 veryfast，每支文件省 30 秒。

然后完整文件否掉了它。同一支源、同一个 ICQ 26，一次用 `fast`、一次用 `medium`：

```text
fast:    168,435,266 B   编码 360 秒   mean SSIM 0.9960
medium:  168,435,266 B   编码 360 秒   mean SSIM 0.9960
```

逐字节同大小、同编码时间、同质量。摘录层的 30 秒对 60 秒，是每一轮固定开销造成的差别，不是编码速度——这正是那种「如果只走到一步就停」会写进配置文件、再写进一篇博客的数字。

所以：**摘录只能排质量，永远不能排速度。** 速度必须在完整文件上量，固定开销在那里才会消失。

## 最后落了什么

`qsv_quality = 26`、`qsv_preset = medium`——源视频码率的 61.0%，mean SSIM 0.9960（min 0.9938），约 5.6× 实时；在片库里最瘦的一支上是 50.4%、mean 0.9977。q28 相对下限的余量只有 0.0002，而一次质量失败的代价是两次整片编码再加一次搁置，所以那多出来的 15 个百分点不值得换这条尾部风险。

另外两条值得记下来的量测，它们回答的是「到底是 QSV 的问题还是文件的问题」：

- **跨编码器对照。** 一支 720p、节奏不均匀的源，QSV q24 量到 mean 0.9154、QSV q28 量到 0.9146、x265 crf26 量到 0.9164——引擎的对齐搜索同时报 `no alignment explains the mismatch`。这条拒绝属于源档，不属于 Quick Sync。
- **并发。** 引擎拒绝在共享 work root 上起第二个编码器：`another transcode is already running (pid 7, /work/run-qsv.lock, 3 s old)`。换成各自独立的 work root，两个作业能跑——但墙钟 201 秒对单人 107 秒。两倍时间，零吞吐收益。QSV 一次只跑一个作业。

## 四个坑，每个都花掉一轮

1. **永远不要把共享片库挂到容器的 `/lib`。** 我把 SMB 共享挂进了容器的 `/lib`——而那是容器自己的动态加载器目录。八个作业全在 1 秒内死掉，报 `exec /bin/bash: no such file or directory`，读起来像镜像坏了，而不是挂载点错了。挂 `/library` 就正常。
2. **合成源会给出与事实相反的 ICQ 结论。** `hevc_qsv` 的码率控制对 `rgb24` 输入会被驱动拒绝（`Selected ratecontrol mode is unsupported`），所以用 `testsrc`/`mandelbrot` 自检会得出相反的结论。要么用真文件，要么加 `format=nv12`。
3. **已经是 HEVC 的源根本没法标定。** 闸门会回 `SKIP already HEVC hvc1`，`--overwrite` 也不解除。这个片库里肥档 1080p H.264 原件早就转完了，剩下还在肥档的都是 HEVC——所以那一档已经没有可量的素材了。这是缺输入，不是漏做任务。
4. **读首次尝试，不读重试**（见上面那道悬崖）。

## 如果重来一次

1. **速度要在你真正会跑的那个产物上量**，不要在它的一片切片上量。整件事里我唯一报错的数字，就是信了摘录的墙钟。
2. **先让机器证明它有没有这块硬件，再排测试计划。** 六分钟的 `lspci` 加 `virsh dumpxml`，就把「为什么 QSV 在 VM 里失败」这个悬了几周的问题结掉了。
3. **把闸门留在回路里。** 因为每一次运行都由引擎自己的校验判收，上面这些表格是决策，不是观点——将来在生产里会拒掉坏文件的，正是这个把我的 ICQ 30 拒掉的闸门。
4. **把诚实的适用范围写进文档。** 这些数字是 UHD 770 上 Linux iHD 26.3.5 + oneVPL 2.17 的数；生产目标是同一块硅片上的 Windows MSDK 路径，我不能声称两者一致。这里站得住的是：命令在真硬件上跑得通、质量↔体积曲线的位置、以及锁的行为——留给将来的，只是「在 Windows+Intel 上按同一张表复测一次」，而不是又一个下午。

## 结果

QSV 从「存在于代码里、永远选不到」变成了一个带公开数字的已标定档，安全规则也跟着改了：原来那条钉住「**出货默认值必须代表未标定**」的测试，现在钉的是「**出货默认值必须是量过的值**」，同时保留「空值 = 硬排除」的全部断言。在没装 Intel 显卡的机器上，doctor 那行也从信息级的「等硬件」变成了警告——这本身是另一个小教训：一个已标定的编码器自检失败该长什么样，以及为什么机器上不存在的硬件应该进一行汇总，而不是进警告。

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "Calibrating Intel Quick Sync HEVC: A Cliff at 28, and a Preset That Does Nothing"
description: "I measured QSV HEVC on real 1080p files: ICQ 18–22 inflate thin sources, the quality cliff sits between 28 and 30, and fast equals medium byte for byte."
pubDate: 2026-10-14
updatedDate: 2026-10-16
category: tutorials
tags: ["ffmpeg", "hevc", "qsv", "transcoding", "video"]
ogImage: /og/calibrating-intel-quick-sync-hevc-quality.jpg
banner: /banners/calibrating-intel-quick-sync-hevc-quality.jpg
draft: false
---

## The question I could not answer: which number do I ship?

My transcoding queue decides, per file, which encoder does the work: `copy`
when the source is already efficient, NVENC on the GPU, x265 on the CPU —
and now Intel Quick Sync. Each tier has two knobs that decide quality, and
every tier except QSV had numbers I had measured on my own library.

QSV had none. And the code refused to guess: a shipping default of
`qsv_quality = 0` / `qsv_preset = ""` means *nobody has calibrated this*,
and the engine treats that as a hard exclusion — QSV can exist in the code
and never be selected on any machine, including the one that has the
hardware.

So the task was not "turn on QSV". It was "produce measured numbers for two
knobs, so the encoder becomes selectable honestly".

## Problem one: the hardware is not in the machine where the app runs

The queue runs on Windows 11 in a VM. That VM has an RTX 3060 passed through
and no Intel graphics of any kind — which is exactly why QSV had never been
measured: the obvious move is to run one test locally and read the numbers
off.

The iGPU is on the host instead (unRaid, i9-13900K), idle and unclaimed:

```text
00:02.0 VGA compatible controller: Intel Raptor Lake-S GT1 [UHD Graphics 770]
  driver: i915            refcount: 0
/dev/dri/card0  /dev/dri/renderD128        (present, world-accessible)
no running container declares a dri device
virsh dumpxml Win11: hostdevs = 01:00.0, 01:00.1 (the 3060 + its HDMI audio)
```

Two conclusions, both useful. The VM's `Error creating a MFX session` is
**expected** and not a bug to chase — the VM can never see that iGPU. And
the only way to measure QSV is on the Linux side, in a container, on the
host.

## Measure with the engine's own gate, not with ffmpeg

This is the part that makes the numbers worth anything. I did not write
ffmpeg commands and compute SSIM myself — the calibration runs **the engine
that will do the work**, with its own verification and its own accept/reject
gate:

```bash
python3 library_run.py --work-dir /work --tier qsv --force-tier qsv \
  --qsv-quality Q --qsv-preset P --path <real source> --keep-local --overwrite
```

`--keep-local` means encode, verify, run the gate, and then *do not* land
the file — the verdict is what I am collecting, not a new library. Sources
came in read-only over an existing SMB mount, and I hashed each source
before and after: unchanged.

The image is `linuxserver/ffmpeg` (ffmpeg 9.0.2, iHD 26.3.5, oneVPL 2.17)
plus `python3` — plus **numpy**, which is not optional. Without it the gate
prints `flat-frame scan skipped: No module named 'numpy'` and quietly
changes how near-flat frames are judged, which changes verdicts.

`-global_quality` maps to `RateControlMethod: ICQ` on `hevc_qsv`, which is
the knob being calibrated.

## The ladder: on a thin source, low ICQ makes the file *bigger*

First run, on a real 1080p25 source at 951 kbps (0.0184 bits per pixel — a
"thin" file, the kind that is already squeezed):

| ICQ | engine verdict | mean SSIM | video = source |
|---|---|---|---|
| 18 | **FAIL size** — 176.8% | — | — |
| 20 | **FAIL size** — 141.8% | — | — |
| 22 | **FAIL size** — 107.6% | — | — |
| 24 | ok | 0.9967 | 81.1% |
| 26 | ok | 0.9959 | 61.0% |
| 28 | ok | 0.9951 | 46.4% |
| 30 | **FAIL quality** | 0.9942 | — |
| 32 | **FAIL quality** | 0.9942 | — |

Three things fall out of that table.

The counterintuitive one: **low ICQ does not mean safe.** ICQ 18–22 produce
files *larger* than the source, so the engine's size ceiling (100% of the
source's video bitrate) rejects them. QSV is not alone in this — NVENC at a
low `cq` does the same thing on thin inputs — but the boundary sits higher
here: the usable range starts at 24, not 18.

Second, **there is a cliff between 28 and 30.** 30 and 32 fail on quality,
and they fail on the *first attempt*. That phrase matters, because the
engine retries a failed quality attempt at `global_quality - 2` — lower
ICQ, higher quality — and logs the retry. A line reading "ok at quality+2"
is a **first-attempt failure**, not a pass. I read that line backwards in my
own notes the first time; the rule is to read the first attempt, not the
recovery.

Third, the floor is where the choice gets made: with a **0.995 mean-SSIM
floor**, q28 passes with 0.0001 of margin and q26 with 0.0010.

## Full files confirm the ladder — and one shortcut survives

Excerpts (180-second lossless slices) are how you rank ten candidates
without ten full encodes. They are only legitimate if they agree with
full-file runs, so I ran the real files end to end:

| source | ICQ | preset | video = source | file | SSIM min / mean | encode |
|---|---|---|---|---|---|---|
| 254 MB, 2023 s, 872 kbps | 26 | medium | **61.0%** | 168,435,266 B (66%) | 0.9938 / **0.9960** | 360 s ≈ 5.6× realtime |
| the same file | 28 | medium | 46.3% | 54% | 0.9927 / 0.9952 | 330 s |
| 315 MB, 3280 s, 546 kbps | 26 | medium | 50.4% | 60% | 0.9965 / **0.9977** | 600 s ≈ 5.5× realtime |

The 180-second excerpt of the first file measured mean 0.9959; the full file
measured 0.9960. That is close enough to trust the shortcut **for quality
ranking** — and I now have the evidence to say so rather than hope so.

## The preset that does nothing

The excerpt runs said preset only changes speed: veryfast 30 s / 61.6%, fast
30 s / 61.0%, medium 60 s / 61.0%, slow and veryslow 60 s. Clean story: pick
veryfast, save 30 seconds per file.

Then the full files disagreed with it. The same source at ICQ 26, one run
with `fast` and one with `medium`:

```text
fast:    168,435,266 B   enc 360 s   mean SSIM 0.9960
medium:  168,435,266 B   enc 360 s   mean SSIM 0.9960
```

Byte-for-byte identical, same encode time, same quality. The 30-vs-60-second
difference in the excerpt layer was per-run fixed overhead, not encoding
speed — and it is the kind of number that would have gone into a config file
and a blog post if I had stopped one step early.

So: **excerpts rank quality, never speed.** Speed has to be measured on full
files, where the fixed overhead disappears.

## What I shipped

`qsv_quality = 26`, `qsv_preset = medium` — 61.0% of the source's video
bitrate at mean SSIM 0.9960 (min 0.9938) at roughly 5.6× realtime, and 50.4%
at mean 0.9977 on the thinnest file in the library. q28's margin over the
floor is 0.0002, and the cost of a quality failure is two full encodes plus
a park, so the extra 15 percentage points of savings are not worth the tail
risk.

Two supporting measurements worth recording, because they answer "is it QSV
or is it the file?":

- **Cross-encoder control.** One 720p source with non-uniform cadence scored
  QSV q24 mean 0.9154, QSV q28 mean 0.9146, and x265 crf26 mean 0.9164 — and
  the engine's alignment search reported
  `no alignment explains the mismatch`. The rejection belongs to the source,
  not to Quick Sync.
- **Concurrency.** The engine refuses a second encoder on a shared work
  root:
  `another transcode is already running (pid 7, /work/run-qsv.lock, 3 s old)`.
  With separate work roots, two jobs run — and take 201 s where one takes
  107 s. Twice the wall clock, zero throughput gain. One QSV job at a time.

## Four traps, each of which cost a cycle

1. **Never mount a shared library at `/lib`.** I mounted the SMB share into
   the container at `/lib` — which is where the container's own dynamic
   loader lives. All eight jobs died in under a second with
   `exec /bin/bash: no such file or directory`, which reads like a broken
   image, not a bad mount. `/library` works.
2. **Synthetic sources lie about ICQ.** `hevc_qsv`'s rate control is
   rejected by the driver for `rgb24` input
   (`Selected ratecontrol mode is unsupported`) — so a
   `testsrc`/`mandelbrot` self-test concludes the opposite of the truth. Use
   a real file, or add `format=nv12`.
3. **Already-HEVC sources cannot be calibrated at all.** The gate answers
   `SKIP already HEVC hvc1` and `--overwrite` does not lift it. The fat
   1080p H.264 originals in this library had all been converted long ago;
   the remaining fat files are HEVC — so that tier has nothing left to
   measure on. That is a missing input, not a skipped task.
4. **Read the first attempt, not the retry** (see the cliff above).

## What I would do differently

1. **Measure speed on the artifact you will actually run**, not on a slice
   of it. My one wrong number in this whole exercise came from trusting an
   excerpt's wall clock.
2. **Make the machine prove it has the hardware before you plan the test.**
   Six minutes of `lspci` and `virsh dumpxml` retired the "why does QSV fail
   in the VM" question that had been open for weeks.
3. **Keep the gate in the loop.** Because the engine's own verification
   judged every run, the tables above are decisions, not opinions — the same
   gate that will reject a bad file in production rejected my ICQ 30.
4. **Write the honest scope line into the notes.** These numbers are Linux
   iHD 26.3.5 + oneVPL 2.17 on a UHD 770; the production target is the
   Windows MSDK path on the same silicon, and I cannot claim they agree.
   What this establishes is that the command runs on real hardware, where
   the quality↔size curve sits, and how the lock behaves — leaving a
   re-measurement on Windows+Intel as one table, instead of another
   afternoon.

## The result

QSV went from "present in the code and never selectable" to a calibrated
tier with published numbers, and the safety rule changed with it: the test
that used to pin *shipping defaults must mean uncalibrated* now pins
*shipping defaults must be measured values*, while keeping every assertion
that an empty value is a hard exclusion. The doctor output on a machine with
no Intel GPU moved from an informational "waiting for hardware" line to a
warning — which is its own small lesson about what a calibrated encoder
failing self-check should look like, and why hardware the machine does not
have belongs in a summary row rather than a warning.

文章 16 / 16：转成 HEVC 到底会不会让文件变小？我实测了

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "转成 HEVC 到底会不会让文件变小？我实测了"
description: "在真实文件上对 HEVC 编码器做的实测对比：NVENC cq27 让三个测试文件中的两个变得更大；同画质下软件 x265 比 NVENC 小 10–22%；同样是 1080p，一个源缩了 49%，另一个只缩了 17%。"
pubDate: 2026-10-08
category: tutorials
tags: [ffmpeg, hevc, x265, nvenc, video]
ogImage: /og/zh-does-hevc-actually-shrink-your-files.jpg
banner: /banners/does-hevc-actually-shrink-your-files.jpg
draft: false
---

## 一个值得验证的假设

"转成 HEVC，文件就变小了。"这句话被重复得太多，以至于没人去实测它。我不得不测：我手上有一个 582 个课程视频的库，之前那轮重编码在 54 个文件里有 **50 个把文件变大了**，平均产出是原来的 111%。

所以我测了。真实文件、一次只改一个变量、画质对着源来量而不是靠假设。

简短版本：

- **H.264** 源转 HEVC、同画质下能省 **33–66%**——但跨度极大，取决于源，而不只是取决于码率。
- 一个很常见的"高质量"硬件设置（`hevc_nvenc` cq27）让**三个测试文件里的两个变得比源更大**。
- 同画质下，**软件 x265 比 NVENC 小 10–22%**。硬件编码买的是时间，不是体积。
- 对已经很高效率的源（YouTube 的 VP9/AV1）重编码一无所获，只会变大。

## 方法：要么在同画质下比，要么别比

只比"输出码率"是没有意义的——任何编码器都可以靠丢画质把码率压下来。测量必须把画质固定住。

我把每个源的 60 秒窗口用多种设置编码，然后对每一份输出：

1. 量**输出的视频码率**占**源视频码率**的百分比（不是文件体积——把音频排除掉，两者才可比）；
2. 量**输出对源的 SSIM**，逐帧、缩小到 480×270 以获得稳定的度量，报告最小值和均值。

有一个前提：这套度量工具必须自洽。把一个文件和**它自己**走同一条代码路径比对，必须得到 SSIM 1.0000。当我的工具做不到时——它在一个和自己的比对中给出了 0.87，因为其中一侧是从 open GOP 源的中间 seek 解码的——我本可以"测出"任何结论并信以为真。

```bash
# 自检：这里必须打印 1.000000，否则你的比对工具在骗你
ffmpeg -i any.mp4 -f rawvideo -pix_fmt yuv420p a.yuv
ffmpeg -i any.mp4 -f rawvideo -pix_fmt yuv420p b.yuv
ffmpeg -i a.yuv -i b.yuv -lavfi ssim -f null -
```

## 结果

三个来自我在处理的素材库的真实源，都是 25–30 fps 的 H.264；表中数字为**输出视频码率占源视频码率的百分比**：

| 设置 | 1080p 音乐（2720 kbps，bpp 0.0437） | 720p（1208 kbps，bpp 0.0437） | 1080×1920 竖屏（4490 kbps，bpp 0.0722） |
|---|---|---|---|
| `hevc_nvenc` cq27 | 72.4%（min SSIM 0.9963） | **115.2%**（0.9935） | **119.3%**（0.9930） |
| `hevc_nvenc` cq30 | 51.0%（0.9945） | 82.6%（0.9907） | 83.8%（0.9901） |
| `hevc_nvenc` cq31 | 43.0%（0.9941） | 72.9%（0.9894） | 74.3%（0.9890） |
| `hevc_nvenc` cq30 + AQ/lookahead | 55.8%（0.9958） | 88.1%（0.9914） | 84.4%（0.9935） |
| `libx265` crf26 | **33.7%**（0.9927） | **56.7%**（0.9879） | **67.0%**（0.9913） |
| `libx265` crf28 | 26.4%（0.9910） | 45.0%（0.9852） | 53.2%（0.9888） |

这张表里掉出来四件事。

**1. 那个"著名默认值"是错的默认值。** 网上说"HEVC，高质量"时，很多人会落到 cq27 —— 而在三个文件里的两个上，它产出的视频流**比它要替换掉的源还大**。这并不是说 HEVC 比 H.264 差，而是说：一个在你自己的测试片段上校准出来的固定质量旋钮，对别人编好的文件毫无意义。

**2. 缩小的幅度取决于源的效率，而不是码率。** 样本 1 和样本 2 的 bpp（每像素每秒位数）完全相同——都是 0.0437——一个缩了 49%，另一个只缩了 17%。720p 那个本身已经是不错的编码，1080p 那个是"肥"的。同样的 bpp，不同的余量。bpp 仍是最便宜的第一道筛子（它告诉了我该重编码**哪些**文件），但它预测不了比例。

**3. 同画质下软件 x265 比 NVENC 更小。** 比较 SSIM 接近的那两行：cq31 vs crf26（在 720p 文件上 min SSIM 0.9894 vs 0.9927）——72.9% vs 56.7%，也就是说**文件小了 22%，而量出来的画质更高**。整体看，多出来的节省大约是 **10–22%**。这不是说 NVENC 编得差，而是它为吞吐量做了优化。约束是磁盘就花 CPU，约束是时间就花 GPU。

**4. 音频会成为地板。** 样本 1 的视频流降到了源视频码率的 33.7%——但在全片测试里，**整个文件**只降到原体积的 53.1%，因为在视频降下来之后，AAC 音轨（跟随源的音频码率，因为 iOS 要求 MP4 里是 AAC）大约占了输出总码率的 43%。视频压到这个程度之后，音轨就是下一个杠杆——而且是个小得多的杠杆。

## 这对 VP9/AV1 源意味着什么

上面讲的都是 **H.264** 源。现代 YouTube 格式是另一回事，我也测了：抽样 14 个 AVC1 源，bpp 中位数是 **0.053**（YouTube 给不支持 VP9/AV1 的设备发的那条"兼容梯队"，码率给得很足），而同一批素材的 VP9/AV1 版本要低得多。可压缩的余量：H.264 上很多，VP9/AV1 上没有。

在一个由 YouTube VP9/AV1 组成的素材库上，用硬件编码器以 cq27 重编码，产出是原文件的 **105–112%**——明显不如不动它。这种情况下正确的动作是 `-c copy`：让视频流逐位不变，只重编音频容器（MP4 里的 MP3 在 iOS 上放不了，AAC 可以）。

## 我现在实际在用的判断规则

1. **动手之前先按 bpp 分类。** 每像素每秒位数（`视频码率 / (宽 × 高 × 帧率)`）。一次 ffprobe 调用，它区分"有余量"和"别碰它"的能力，比分辨率或文件大小都强。
2. **已经很高效率的源绝不重编码。** 复制视频流、需要的话修音频，然后走开。重编码变不出本来不存在的余量。
3. **设质量，不设码率——然后对着源验证。** 逐帧比对的度量，加上一个证明工具本身可用的自检。
4. **按瓶颈选编码器。** 磁盘 → x265（同画质小 10–22%）。时间 → NVENC。
5. **给输出加上限。** 无论怎么做，都要断言结果不比源更大。就这一条断言，本可以在那 54 个文件被碰之前就抓住整个 111% 的问题。
6. **别忘了音频。** 视频被压狠之后，音轨可能是文件里剩下的最大部分。

## 实测结果

在那个 582 个文件的库上，用上面的分档和我另文写的回读校验：

- **copy 档**（高效率源，视频逐位不变）：约原体积的 **100%**——有意为之，画质精确保留。
- **中间档**（NVENC，cq31）：落在 **72%** 左右。
- **"肥"档**（x265，crf26）：落在 **33–44%**，中位数 **39%**。

这就是"HEVC 会不会让我的文件变小"的诚实答案：取决于你在重编码什么，而唯一知道你那个数字的办法，就是在你自己的文件上、把画质固定住去测一遍。

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "Does Re-encoding to HEVC Actually Shrink Your Files? I Measured It."
description: "A measured comparison of HEVC encoders on real files: NVENC at cq27 made two of three files bigger, software x265 landed 10–22% smaller at matched quality, and one 1080p source shrank 49% while another shrank 17%."
pubDate: 2026-10-08
category: tutorials
tags: [ffmpeg, hevc, x265, nvenc, video]
ogImage: /og/does-hevc-actually-shrink-your-files.jpg
banner: /banners/does-hevc-actually-shrink-your-files.jpg
draft: false
---

## The assumption worth testing

"Convert it to HEVC, the files get smaller." It is repeated often enough
that nobody measures it. I had a reason to: a library of 582 course videos
where re-encoding had made files **bigger** in 50 of 54 cases, with the
average output at 111% of the original.

So I measured. Real files, one variable at a time, quality checked against
the source rather than assumed.

The short version:

- Re-encoding a **H.264** source to HEVC at matched quality shrank files by
**33–66%** — but the spread is huge and depends on the source, not just its
bitrate.
- A common "good quality" hardware setting (`hevc_nvenc` at cq27) made **two
of three** test files *larger* than the source.
- At matched quality, **software x265 is 10–22% smaller than NVENC**. Hardware
encoding buys time, not size.
- Re-encoding an already-efficient source (VP9/AV1 from YouTube) buys
nothing. It grows.

## Method: measure at equal quality, or don't bother

Comparing "output bitrate" alone is meaningless — any encoder can hit a low
bitrate by throwing quality away. The measurement has to hold quality fixed.

I encode a 60-second window of each source at several settings, and for each
output:

1. measure the output's **video bitrate** as a percentage of the source's
video bitrate (not the file size — audio is excluded so the two are
comparable), and 2. measure **SSIM against the source**, frame by frame,
downscaled to 480×270 for a stable metric, reporting min and mean.

One prerequisite: the harness must be self-consistent. Comparing a file with
*itself* through the same code path must return SSIM 1.0000. When mine did not
— it returned 0.87 on a file compared with itself, because one side was
decoded from a mid-file seek on an open-GOP source — I could have "measured"
anything and believed it.

```bash
# self-test: this must print 1.000000, or your comparison is lying to you
ffmpeg -i any.mp4 -f rawvideo -pix_fmt yuv420p a.yuv
ffmpeg -i any.mp4 -f rawvideo -pix_fmt yuv420p b.yuv
ffmpeg -i a.yuv -i b.yuv -lavfi ssim -f null -
```

## Results

Three real sources from the library I was working on, all 25–30 fps H.264,
video bitrate as a percentage of the source's video bitrate:

| Setting | 1080p music (2720 kbps, bpp 0.0437) | 720p (1208 kbps, bpp 0.0437) | 1080×1920 vertical (4490 kbps, bpp 0.0722) |
|---|---|---|---|
| `hevc_nvenc` cq27 | 72.4% (min SSIM 0.9963) | **115.2%** (0.9935) | **119.3%** (0.9930) |
| `hevc_nvenc` cq30 | 51.0% (0.9945) | 82.6% (0.9907) | 83.8% (0.9901) |
| `hevc_nvenc` cq31 | 43.0% (0.9941) | 72.9% (0.9894) | 74.3% (0.9890) |
| `hevc_nvenc` cq30 + AQ/lookahead | 55.8% (0.9958) | 88.1% (0.9914) | 84.4% (0.9935) |
| `libx265` crf26 | **33.7%** (0.9927) | **56.7%** (0.9879) | **67.0%** (0.9913) |
| `libx265` crf28 | 26.4% (0.9910) | 45.0% (0.9852) | 53.2% (0.9888) |

Four things fall out of that table.

**1. The famous default is the wrong default.** cq27 is where a lot of the
internet lands when it says "HEVC, high quality" — and on two of three files
it produced a *bigger* video stream than the source it was replacing. That
is not an argument that HEVC is worse than H.264. It is an argument that a
fixed quality knob calibrated on your own test clip means nothing on someone
else's encode.

**2. The shrink depends on the source's efficiency, not its bitrate.** Samples
1 and 2 have identical bpp (bits per pixel per second) — 0.0437 — and one
shrank 49% where the other shrank 17%. The 720p file was already a decent
encode; the 1080p one was a fat one. Same bpp, different headroom. bpp is
still the cheapest first filter (it told me which *files* to re-encode), but
it does not predict the ratio.

**3. Software x265 beats NVENC on size, at equal quality.** Compare the rows
at similar SSIM: cq31 vs crf26 (min SSIM 0.9894 vs 0.9927 on the 720p file)
— 72.9% vs 56.7%. That is a 22% smaller file at a *higher* measured quality.
Across the set, analysis gives roughly **10–22%** extra saving. NVENC is not
worse at encoding; it is optimized for throughput. If your constraint is
disk, spend CPU; if it is time, spend GPU.

**4. Audio becomes the floor.** Sample 1's video stream dropped to 33.7% of
the source's video bitrate — but the *file* only reached 53.1% of the
original size on the full-length test, because after that drop the AAC audio
(matched to the source's audio bitrate, as iOS requires AAC in MP4) was
about 43% of the output's total bitrate. Once the video is this compressed,
the audio track is the next lever — and it is a much smaller one.

## What this means for VP9/AV1 sources

Everything above is about **H.264** sources. The modern YouTube formats are
a different story, and I measured that too: 14 sampled AVC1 sources had a
median bpp of 0.053 (the "compatibility ladder" YouTube serves to devices
that cannot do VP9/AV1 is generous), while the VP9/AV1 renditions of the
same material sit far lower. Re-encodable headroom: plenty on H.264, none on
VP9/AV1.

On a library of YouTube-sourced VP9/AV1 files, re-encoding with a hardware
encoder at cq27 produced files at **105–112%** of the originals — measurably
worse than leaving them alone. The correct action there is `-c copy`: keep
the video stream bit-identical and re-encode only the audio container (MP3
in MP4 is not playable on iOS; AAC is).

## The decision rules I actually use now

1. **Classify by bpp before encoding anything.** Bits per pixel per second
(`video_bitrate / (width × height × fps)`). It is one ffprobe call and it
separates "there is headroom here" from "leave it alone" better than
resolution or filesize does. 2. **Never re-encode a source that is already
efficient.** Copy the video stream, fix the audio if it needs it, and move
on. A re-encode cannot create headroom that is not there. 3. **Set quality,
not bitrate — then verify against the source.** A metric compared frame by
frame, with a self-test proving your harness works. 4. **Choose the encoder
by your bottleneck.** Disk → x265 (10–22% smaller at equal quality). Time →
NVENC. 5. **Cap the output.** Whatever you do, assert the result is not
bigger than the source. That single assertion would have caught the entire
111% problem before it touched 54 files. 6. **Remember the audio.** After
aggressive video compression, the audio track can be the largest remaining
component of the file.

## The measured outcome

On the 582-file library, with the tiering above and the read-back
verification I described elsewhere:

- **copy tier** (efficient sources, video bit-identical): ~100% of the
original — by design, quality preserved exactly.
- **mid tier** (NVENC, cq31): landed around **72%**.
- **fat tier** (x265, crf26): landed **33–44%**, median **39%**.

That is the honest answer to "does HEVC shrink my files": it depends on what
you are re-encoding, and the only way to know your number is to measure it
on your own files with quality held fixed.
Reference in new issue View Git Blame Copy Permalink