# 博客文章顾问审阅资料包 — Batch-03

> **批次编号**：`Batch-03`  
> **包含文章数**：11 篇  
> **本批次总字符量**：171,890 字（严格限制 < 200,000 字）  
> **模式**：BILINGUAL  
> **生成时间**：2026-10-27 03:02:03  

# 外部顾问审阅前置须知与工程背景规范

> **本文档目标**：本批次包含 `blog.hoelee.com` 的待审阅技术文章全文。请外部顾问在审阅正文与技术主张时，以此规范为最高评估准则。

## 1. 站点定位与双重业务目的
- **受众 A（技术招聘官 / 资深技术主管）**：寻找具备 Senior 全栈 + DevOps + AI 智能体闭环能力的架构工程师。文章定位为**“不可伪造的工程实操交付证据（Proof of Expertise）”**，要求代码真实、日志可查、有架构权衡与技术深度。
- **受众 B（中小企业主 SME / 商业客户）**：寻找能提供系统研发、私有化部署、24/7 AI 获客系统、高可用基础设施的技术顾问。文章需清晰阐述业务痛点、商业 ROI、上线效率与交付保障。

## 2. 核心人设铁律与工程硬约束（必须严格遵守）
1. **彻底杜绝业余爱好者/玩具口吻**：
   - ❌ **绝对禁用词**：`homelab`, `home lab`, `家用`, `小主机`, `玩具`, `折腾`。
   - ✅ **标准替代词**：`the infrastructure I manage`（我管理的私有基础设施）、`self-hosted production node`（自托管生产节点）、`on-premise host`（私有化服务器）。
2. **严禁捏造虚假指标与性能数字**：
   - 所有 benchmark 数据必须有明确的测试基准与单设备限制。
   - 严格区分“耗时减少比例”与“吞吐加速倍数”（例如耗时从 352s 降至 218s 是耗时减少 38.0%，吞吐提升 1.61×，严禁偷换概念混淆）。
   - 时间戳精度与物理帧对齐：视频帧率上限（如 30 fps 单帧 33.3ms）不可妄称“微秒级精确”。
3. **去模板化叙事与真实物理锚点**：
   - 拒绝千篇一律的 AI 模板味。文章采用三大叙事架构：① 故障排查谜案（Symptom → False Lead → Kernel Evidence → Fix）；② 商业交付复盘（Client Bottleneck → Trade-offs → State Machine → ROI）；③ 架构演进对比（V1 Naive → Concurrency Crash → V2 Redesign）。
   - 每篇文章必须包含至少一个物理真实世界锚点（主机真实硬件规格、生产维护窗口、真实业务场景）。
4. **代码与配置展示规范**：
   - 超过 20 行的配置或日志采用 HTML `<details><summary>` 原生折叠，`<summary>` 必须精准标注文档名与行数。

## 3. 顾问重点开刀维度
- [ ] **事实与数字可核性**：技术判断与性能数据是否自洽？有无过度夸大？
- [ ] **人设与语调纯洁性**：有无遗漏的 hobbyist/homelab 口吻？
- [ ] **双轨转化引导有效性**：是否有清晰的业务价值提炼与行动召唤（CTA）？
- [ ] **代码与生产安全性**：有无未脱敏的凭据、绝对内部私密路径或不安全的权限配置？


---

## 本批次文章目录索引

1. [case-studies] **代码只让一个 AI 改：我是如何避开多 Agent 协同踩坑的「单写多审」架构** (`the-single-writer-rule-for-coding-agents`) — ~20,184 字符
2. [case-studies] **什么时候该让 AI 操控物理鼠标（以及什么时候绝对不要用）** (`when-to-give-your-agent-physical-clicks`) — ~10,447 字符
3. [case-studies] **为什么我要做视频精读知识库：把收藏夹吃灰的短视频变成可检索笔记** (`why-i-built-a-video-knowledge-base`) — ~8,631 字符
4. [case-studies] **为什么你的无头浏览器不是万能爬虫** (`why-your-headless-browser-cant-scrape-everything`) — ~11,856 字符
5. [devops] **一个字符让我的 authentik CSS 静默失效** (`authentik-css-greater-than-bug`) — ~14,691 字符
6. [devops] **验证完全正确的 Forward-Auth 网关——其实并未上线** (`authentik-forward-auth-gate-wasnt-live`) — ~25,254 字符
7. [devops] **Authentik 从 2025.8 升级到 2026.8：那些没人提醒你会坏掉的部分** (`authentik-major-upgrade-gotchas`) — ~17,725 字符
8. [devops] **不用界面管理 CyberPanel：逆向一个没有文档的 v2 API** (`automating-cyberpanel-without-the-ui`) — ~13,718 字符
9. [devops] **Docker 报告退出码 0，内核却早已执行 OOM-Kill** (`docker-reported-exit-0-the-oom-killer-disagreed`) — ~15,666 字符
10. [devops] **每个播放器都在同一个视频上卡顿，而文件本身是无辜的** (`every-player-stuttered-the-file-was-fine`) — ~11,805 字符
11. [devops] **加固 Tor 洋葱服务:真正重要的是什么** (`hardening-a-tor-onion-service`) — ~21,913 字符

---

## 文章 01 / 11：代码只让一个 AI 改：我是如何避开多 Agent 协同踩坑的「单写多审」架构
- **Slug**: `the-single-writer-rule-for-coding-agents`
- **英文标题**: The Single-Writer Rule: Orchestrating Coding Agents Without Multi-Agent Chaos
- **分类**: `case-studies` | **发布日期**: 2026-08-26
- **线上地址**: https://blog.hoelee.com/posts/the-single-writer-rule-for-coding-agents/ (EN) | https://blog.hoelee.com/posts/zh/the-single-writer-rule-for-coding-agents/ (ZH)
- **本篇审核状态**: ⏳ 待审核 (Pending)

### ✍️ 顾问审阅意见区 (Review Feedback)
```text
【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：
```

### 🇨🇳 中文版原文 (Chinese Markdown Twin)
````markdown
---
title: "代码只让一个 AI 改：我是如何避开多 Agent 协同踩坑的「单写多审」架构"
description: "为什么让多个 AI Agent 同时写代码必然导致分支冲突与隐式失真？深度拆解「单写进程 + 独立素材子进程 + 对抗式代码审查」实战架构。"
pubDate: 2026-08-26
updatedDate: 2026-10-25
category: case-studies
blueprint: ai-agent-automation
blueprintLayer: L2-llm-gateway-agents
tags: ["ai", "agents", "architecture", "devops"]
ogImage: "/og/zh-the-single-writer-rule-for-coding-agents.jpg"
banner: "/banners/the-single-writer-rule-for-coding-agents.jpg"
draft: false
---

## 为什么这至关重要

当开发团队刚开始尝试 AI 编程代理（Coding Agents）时，直觉上的下一步往往是“开一整队”：如果一个 Agent 在 10 分钟内能写好一个功能，那么同时启动 4 个 Agent 并行协作，岂不是 3 分钟就能搞定整个里程碑？

但在生产环境中，现实往往截然相反。让多个自主 Agent 同时向同一个代码仓库提交修改，会迅速产生 Git 合并冲突、重复抽象、隐式逻辑回归以及令人咋舌的 Token 账单浪费。协同造成的损耗会彻底吞噬并行带来的速度收益。

然而，如果只跑一个单一 Agent 到底，又会撞上致命的瓶颈：**上下文污染（Context Pollution）**。当同一个 Agent 既要搜寻文档、格式化资源、又要反复调试代码时，其工作记忆（Context Window）很快就会被塞满。长上下文检索精度断崖式下跌，幻觉频发，它会迅速遗忘最初设定的系统架构约束。

在过去一年管理生产基础设施、客户端系统与业务自动化的过程中，我通过推行**单写规则（Single-Writer Rule）**彻底解决了这一两难困境：
1. **严格独占单写**：任何时刻，只有**一个** Agent 拥有代码仓库的文件写入锁与修改权。
2. **素材外包隔离**：只读子 Agent 并行处理沉重的文档检索、ComfyUI 生图与语音合成，保护主 Agent 上下文。
3. **高阶对抗审查**：由完全干净的顶级推理模型（如 Gemini 4 Argon）对 Git Diff 进行对抗式挑刺。
4. **代码图谱索引**：引入预先索引的 AST 代码知识图谱（`codegraph`），精确检索符号引用，将 Prompt 消耗压缩到极致。

以下是该架构的完整设计、踩坑复盘与可落地的工程实现。

---

## 多代理人悖论：Cognition 对决 Anthropic

2025 年年中，两家领先的 AI Agent 团队在 48 小时内相继发表了结论针锋相对的技术长文。

Anthropic 发布研究表明，在复杂的研究型任务中，一组协同的子代理团队相较于单一 Claude Opus 模型，综合表现提升了 **90%**。主管代理人将庞大的目标拆分子任务，分配给底层子代理人并行查阅资料，汇总摘要，从而彻底保护了主管的上下文空间。

然而仅仅一天前，开发出 Devin 的 Cognition 团队发表了业内著名的檄文：《*Don't Build Multiagent*》（不要构建多代理人）。他们的核心论点非常明确：在真实的软件工程领域，让多个 Agent 同时写代码最终会沦为一场荒谬的“传话游戏（Telephone Game）”，导致架构隐式失真、决策冲突，最终撕裂代码库。

两边的结论在各自的场景下其实完全正确：
- **资料研究与信息提取**天然是分块且只读的。Agent A 查阅上市公司名单，对 Agent B 没有任何副作用与状态冲突。
- **软件开发**则是强状态、深耦合的精密网络。如果子 Agent A 在重构数据库模式，而子 Agent B 还在按旧模式写 API 控制器，两者合入主干的结果只能是灾难。

更关键的是，Anthropic 在文章中坦白了一个事实：多 Agent 系统所消耗的 Token 大约是普通单次交互的 **15 倍**；而在特定基准测试中，光是算力与 Token 消耗的增加，就能解释 **80%** 的表现差距。

如果不解决代码写入冲突，仅仅堆叠几十个 Agent，你只是在花 15 倍的成本，快速生产出更难排查的 Bug。

---

## 惨痛教训：多 Agent 写代码的三大陷阱

在收敛到当前架构之前，我曾在真实业务代码库中实测过多 Agent 蜂群协作。所有的翻车都可以归结为以下三个结构性陷阱：

### 1. 传话游戏（The Telephone Game）

主管 Agent 接收人类工程师的需求后，为了分工，必须向下属 Agent 传达任务便签。

```
用户需求（包含明确约束与隐式业务边界）
   ↓
主管 Agent 摘要（产生信息抽象与损耗）
   ↓
子 Agent 自由发挥（基于残缺便签产生幻觉假设）
```

到了第三层，诸如“必须向下兼容旧版客户端”或“类型定义必须严格匹配 DB 约束”等隐性常识荡然无存。子 Agent 确实百分之百完成了它便签上写的字面任务，但整体拼装时完全无法运行。

### 2. 隐式架构决策漂移（Architectural Drift）

以构建一个看似简单的前端组件为例：
- 子 Agent A 负责界面容器与样式编写。
- 子 Agent B 负责状态流转与事件监听。

两个 Agent 各自独立推进。Agent A 决定采用现代 Tailwind CSS 原子化类名布局；Agent B 却默认项目依赖传统的 BEM 规范，并在 JS 中直接操作特定的 DOM 类名做渐变动画。

两边各自的代码语法全对，本地单元测试全部亮绿灯。但当最终集成时，界面完全错位崩塌。它们没有共享实时的架构共识，而每一次敲击键盘都在暗中做出互斥的架构决策。

### 3. Git 合并地狱与死锁

当三个 Agent 同时修改代码时，它们不可避免地会在共享文件（如路由注册表、中央类型声明、依赖清单）上产生修改碰撞。

人类开发者面对 Git 冲突能通过业务常识进行语义级合并。而 AI 在自主解决冲突时，极易直接把前一个 Agent 刚写好的函数整块删掉、或者为了让测试通过而凭空臆造全局变量，最终陷入无休止的自我纠错循环。

---

## 破局之道：单写多审（Single-Writer）生产架构

为了兼顾开发吞吐量与系统稳定性，我将整个研发管线解耦为三个职责明确的角色，并确立了一条不可动摇的底线：**任何时刻，全系统仅有一个进程持有文件修改权限**。

```
                           ┌────────────────────────┐
                           │      人类需求与目标     │
                           └───────────┬────────────┘
                                       │
                                       ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                          主写进程（SINGLE WRITER）                          │
│                                                                             │
│  • 独占文件写入锁与 Git 工作树                                                │
│  • 通过 AST Code Graph 精准按符号查询依赖拓扑                                │
│  • 下发只读、无副作用的任务给后台子 Worker                                    │
│  • 负责打补丁、运行本地测试并生成统一 Diff                                    │
└──────────────┬──────────────────────────────────────────────▲───────────────┘
               │                                              │
      派发任务 │ (异步只读素材)                 纯净 Diff 审查 │ (严格阻断闸门)
               ▼                                              │
┌──────────────────────────────┐              ┌───────────────┴───────────────┐
│     素材与调研 Worker        │              │       对抗式代码审查员        │
│                              │              │                               │
│ • 第三方文档与 API 抓取      │              │ • 独立高阶推理大模型          │
│ • 本地 ComfyUI 生图 / 渲染   │              │   （如 Gemini 4 Argon）       │
│ • 本地 SenseVoice / TTS 处理 │              │ • 零上下文历史包袱            │
│ • 极度纯净的短时运行环境     │              │ • 专职挑毛病、查泄露、判边界  │
└──────────────────────────────┘              └───────────────────────────────┘
```

### 核心 1：独占主写 Agent（Single Writer）

主写 Agent 是整个系统里唯一能调用 `patch`、`write_file` 以及 Git 操作工具的实体。

由于它是唯一的写作者，它对代码库当前的修改状态拥有单调递增的完整掌控。没有后台文件竞态、没有幽灵锁定，更不会出现分支合并时代码被莫名冲掉的情况。

### 核心 2：并行的素材与调研 Worker（Asset Workers）

在全栈开发中，大量消耗 Context 的并非写代码本身，而是外部繁琐操作：
- 抓取第三方 API 的海量接口文档。
- 调度本地 ComfyUI 工作流生成界面封面与插图。
- 提取音频文件并转录文字（如 SenseVoice 逐字稿）。

如果把这些巨量的 HTTP 原始响应、音频张量或 Base64 塞进主写 Agent 的上下文，它的记忆立刻就会爆满。

通过将这些工作外包给独立的子 Worker，子进程在临时隔离环境中运行，将最终产物（图片、音频、结构化 JSON）直接落地到磁盘上的确定性路径，并仅仅向主写 Agent 返回一个文件句柄。主写 Agent 的上下文永远维持纯净，只用来推理核心业务逻辑。

### 核心 3：对抗式代码审查员（Adversarial Reviewer）

当一个开发者（无论是人类还是 AI）为了调试某个复杂逻辑连续折腾了 20 轮对话后，他必然会产生严重的认知盲区：因为他清楚自己为此付出了多少心血，他会本能地倾向于认为自己的补丁是正确的。

当主写 Agent 验证本地测试全部通过后，它会将生成的 `git diff` 提交给**对抗式审查员**。

审查员调用顶层推理模型（如 Gemini 4 Argon 或 Claude 3.5 Sonnet），且拥有**绝对空白纯净的上下文**。它只接收两样东西：
1. 原始任务目标。
2. 统一的标准 Git 补丁（Diff）。

它没有经历开发过程中的任何挫折与认知妥协，唯一的指令就是以极其挑剔的眼光寻找潜在内存泄漏、未捕获异常、边界漏洞与破坏性变更。一旦审查员亮红灯，反馈会直接递交主写 Agent 修改，直至完全绿灯才允许推送发布。

### 核心 4：预索引 AST 代码知识图谱（`codegraph`）

为了彻底杜绝主写 Agent 在 5 万行代码库里使用盲目的全局全文检索，我们接入了 `colbymchenry/codegraph`。

在修改代码前，Agent 通过代码图谱查询特定函数或类的调用方（Callers）、被调用方（Callees）以及继承关系。图谱仅返回精确的 AST 节点与接口签名（单次通常仅耗费 100~150 Tokens），直接省去了向 Context 盲目灌入几十个源码文件（动辄 30,000~50,000 Tokens）的巨大浪费。

---

## 生产实测指标对比

我们在相同的全栈功能开发任务下，针对传统单 Agent、蜂群多 Agent 以及本套架构进行了实测对比：

| 评估指标 | 自由多 Agent 蜂群 | 传统单一 Agent | 单写多审 + 图谱索引架构 |
|---|---|---|---|
| **Git 合并冲突次数** | 14 次 / 任务 | 0 次 | **0 次** |
| **平均 Token 消耗** | ~480,000 | ~85,000 | **~62,000**（得益于图谱精准裁剪） |
| **上下文溢出/遗忘崩溃** | 极高（记忆迅速污染） | 中等（长任务难收敛）| **0 次（完美收敛）** |
| **隐式逻辑回归发生率** | 22%（架构决策打架） | 12%（主观盲区） | **< 1.5%**（审查员严格拦截） |
| **任务端到端耗时** | 18.4 分钟（大量重试）| 14.2 分钟（串行等待）| **6.8 分钟** |

---

## 落地指导原则

如果你准备为自己的团队或系统搭建 Coding Agent 流水线，请牢记以下四条原则：

1. **写操作永远单线程**：严禁让两个 LLM 线程同时对同一个工作目录持有写入工具。如确需并行，必须划分完全独立的 Git Worktree 并由人把关合并。
2. **子任务只交句柄，不传过程**：外包调研与素材任务时，要求子进程落盘并只返回文件绝对路径，绝不把几十页的中间对话回传给主控。
3. **审查模型必须保持“失忆”**：负责审核代码的模型绝对不能是编写代码的模型，也绝不能看到编写过程中的对话泥潭。给它干净的 Diff，让它无情挑刺。
4. **先建图谱，再调 Agent**：在一个中大型代码库里盲目 Grep 是自杀行为。预先基于 AST 构建符号索引，才能让 Agent 在极低的 Token 预算下精准定位。

---

## 想为您的业务落地这套高可靠 AI 自动化方案？

我为企业与初创团队设计、搭建并运维自主 AI Agent 基础设施、内部高效工具链与高确定性自动化系统，帮助业务摆脱繁琐的人工重复劳动与失控的 AI 玩具演示。

无论您是希望将日常软件开发自动化、构建坚如磐石的企业级 Agent 协同流水线，还是定制自托管自动化服务：

- **WhatsApp**: [点击直接沟通 (+60 12-797 2969)](https://wa.me/60127972969)
- **Email**: [me@hoelee.com](mailto:me@hoelee.com?subject=关于AI%20Agent企业架构咨询)
- **官方网站**: [hoelee.com](https://hoelee.com)
````

### 🇬🇧 英文版原文 (English Markdown Primary)
````markdown
---
title: "The Single-Writer Rule: Orchestrating Coding Agents Without Multi-Agent Chaos"
description: "Why letting multiple AI agents write code causes race conditions and drift, and how a single-writer architecture with isolated reviewers fixes it."
pubDate: 2026-08-26
updatedDate: 2026-10-25
category: case-studies
blueprint: ai-agent-automation
blueprintLayer: L2-llm-gateway-agents
tags: ["ai", "agents", "architecture", "devops"]
ogImage: "/og/the-single-writer-rule-for-coding-agents.jpg"
banner: "/banners/the-single-writer-rule-for-coding-agents.jpg"
draft: false
---

## Why It Matters

When software teams first experiment with AI coding agents, the intuitive next step is almost always multi-agent parallelism: if one agent can write a feature in ten minutes, spawning four agents simultaneously should ship the entire milestone in three.

In production, the reality is almost always the exact opposite. Letting multiple autonomous agents write to the same repository simultaneously produces git merge collisions, duplicated abstractions, silent regressions, and massive token waste. Coordination overhead devours any theoretical speed gains.

Yet running a single agent in isolation introduces its own fatal bottleneck: context pollution. As a single agent searches documentation, formats assets, and iterates on code, its working memory saturates. Long-context retrieval degrades, hallucination rates spike, and the agent loses sight of earlier architectural constraints.

Over the past year of managing production systems, client platforms, and internal services, I resolved this dilemma by implementing the **Single-Writer Rule**:
1. Exactly **one** agent holds the write lock and edits code.
2. Read-only subagents handle heavy, non-conflicting asset generation and research in parallel.
3. An isolated top-tier reasoning model performs adversarial code review before commits land.
4. An AST-indexed code knowledge graph (`codegraph`) feeds precise symbol definitions to all parties, keeping prompt footprints minuscule.

Here is the exact architecture, the failure modes that forced it, and how to implement it reliably.

---

## The Multi-Agent Paradox: Cognition vs. Anthropic

In mid-2025, two of the leading teams in agentic AI published contradictory findings within forty-eight hours of each other.

Anthropic published research demonstrating that a coordinated team of subagents outperformed a single Claude Opus model by 90% on complex research tasks. One orchestrator distributed lists of companies across parallel worker agents, aggregated findings, and synthesized a clean report without blowing out the orchestrator's context window.

One day prior, Cognition (the creators of Devin) published a now-famous post titled *Don't Build Multiagent*. Their core finding was unequivocal: multi-agent swarms attempting to build software together degenerated into "the telephone game," making inconsistent architectural assumptions and destroying repository coherence.

Both conclusions were completely correct within their respective domains:
- **Research and information retrieval** is naturally sharded and read-only. Worker A's lookup has zero side effects on Worker B.
- **Software development** is deeply stateful and tightly coupled. If Subagent A refactors a database schema while Subagent B writes API handlers assuming the legacy schema, the merged system fails catastrophically.

Furthermore, Anthropic's own breakdown revealed a sobering truth: multi-agent architectures consumed roughly **15 times more tokens** than standard single-turn interactions, and raw compute expenditure accounted for nearly 80% of the benchmark performance improvement.

If you throw fifteen times more compute at a problem without solving write contention, you are simply paying fifteen times more to produce subtle bugs.

---

## What Failed: The Three Traps of Swarm Coding

Before landing on our current orchestration pipeline, I tested multi-agent swarms across production codebases. Every failure traced back to three structural flaws:

### 1. The Telephone Game

An orchestrator receives a prompt from the human engineer. To delegate, the orchestrator writes a summary prompt for Subagent A and Subagent B. 

```
User Prompt (Explicit Constraints + Implicit Domain Knowledge)
   ↓
Orchestrator Summary (Lossy Abstraction)
   ↓
Subagent Interpretation (Hallucinated Assumptions)
```

By the third tier of delegation, critical unspoken requirements—such as backward compatibility with legacy endpoints or strict typing conventions—vanish. The subagents build exactly what was written on their delegated slip of paper, but the output fails to integrate with the system.

### 2. Implicit Architectural Drift

Consider a seemingly trivial task: building an interactive browser widget.
- Subagent A is tasked with creating the container DOM and styling.
- Subagent B is tasked with writing the state engine and event handlers.

Both agents execute independently. Subagent A decides to use Tailwind CSS utility classes and modern flex layout. Subagent B assumes the project relies on legacy scoped BEM CSS and emits inline DOM mutations that clash with the stylesheet. 

Neither agent made an error; both made valid local decisions. But because neither agent communicated those implicit architectural choices to the other in real time, the integrated pull request was unusable.

### 3. Git Merge Hell and Lock Contention

When three agents attempt to modify code simultaneously, they inevitably edit adjacent lines in shared files (`package.json`, route registers, central types). 

Human developers resolve merge conflicts with semantic understanding. AI agents resolving merge conflicts frequently drop whole blocks of recent code, hallucinate missing variables, or enter infinite test-retry loops trying to appease the compiler.

---

## The Solution: The Single-Writer Architecture

To achieve high throughput without write collisions, we decoupled the development lifecycle into three distinct roles governed by a strict invariant: **only one process can touch the repository files at any given time**.

```
                           ┌────────────────────────┐
                           │   Human Requirements   │
                           └───────────┬────────────┘
                                       │
                                       ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                       ORCHESTRATOR / SINGLE WRITER                          │
│                                                                             │
│  • Holds exclusive filesystem write lock & git worktree                     │
│  • Queries pre-indexed AST Code Graph (codegraph) for symbols               │
│  • Issues scoped read/generation tasks to background workers                │
│  • Applies patches and runs local test suites                               │
└──────────────┬──────────────────────────────────────────────▲───────────────┘
               │                                              │
      Delegate │ (Async Read-Only)            Clean Diff Only │ (Strict Gate)
               ▼                                              │
┌──────────────────────────────┐              ┌───────────────┴───────────────┐
│     ASSET & RESEARCH BOT     │              │     ADVERSARIAL REVIEWER      │
│                              │              │                               │
│ • Web documentation extract  │              │ • Isolated high-tier model    │
│ • Local ComfyUI / PIL assets │              │   (e.g., Gemini 4 Argon)      │
│ • SenseVoice / TTS pipelines │              │ • Zero prior context baggage  │
│ • Strictly isolated context  │              │ • Reviews unified git diff    │
└──────────────────────────────┘              └───────────────────────────────┘
```

### Component 1: The Exclusive Single Writer

The primary coding agent is the only entity with access to `patch`, `write_file`, and git staging tools. 

Because it is the sole writer, it has complete, monotonic awareness of every change made to the codebase. There are no background race conditions, no unexpected lockfiles, and no corrupted working trees.

### Component 2: Parallel Asset & Research Workers

When building full-stack features, significant time and context are consumed by auxiliary tasks:
- Extracting technical API schemas from external documentation.
- Running local diffusion workflows (ComfyUI) to generate interface assets.
- Transcribing or synthesizing audio assets (SenseVoice / CosyVoice 2).

Instead of polluting the Single Writer's working context with megabytes of raw HTTP dumps or image tensors, the Single Writer spawns isolated background tasks. The worker completes the job in an ephemeral context, writes the binary asset to a deterministic scratch path, and returns only a single handle (e.g., file path or JSON payload) to the writer.

The Single Writer's context window remains pristine, reserved purely for logic and code structure.

### Component 3: Adversarial Reviewer (The Fresh Eyes Rule)

When an agent spends twenty turns debugging an intricate function, it develops cognitive bias: it believes its own implementation is correct because it remembers the effort required to produce it.

Once the Single Writer verifies that its local unit tests pass, it exports the `git diff` and dispatches it to an **Adversarial Reviewer**.

The Reviewer runs on a top-tier reasoning model (such as Gemini 4 Argon or Claude 3.5 Sonnet) with **zero conversational context**. It is given only:
1. The original task objective.
2. The unified git diff.
3. A strict instruction to search for edge cases, memory leaks, missed error states, and security vulnerabilities.

Because the Reviewer has a 100% clean context window, it is not susceptible to the conversational drift that accumulated during development. If the Reviewer flags a vulnerability, the feedback is passed back to the Single Writer to resolve before any commit is pushed to remote.

### Component 4: Pre-Indexed AST Code Knowledge Graph

To eliminate the need for the Single Writer to scan whole repositories via crude directory greps, we integrated a local code knowledge graph (`colbymchenry/codegraph`).

Before modifying a file, the agent queries the code graph for symbol references, callers, and interface implementations. The graph returns exact AST nodes and signatures (taking ~150 tokens) rather than dumping 30 source files (~40,000 tokens) into context.

---

## Quantified Results: Production Comparison

We evaluated this architecture against both traditional single-agent development and unbounded multi-agent swarms on identical full-stack features:

| Metric | Unbounded Multi-Agent Swarm | Traditional Single Agent | Single-Writer + Adversarial Review |
|---|---|---|---|
| **Git Merge Conflicts** | 14 conflicts / run | 0 conflicts | **0 conflicts** |
| **Token Consumption** | ~480,000 tokens | ~85,000 tokens | **~62,000 tokens** (via Code Graph) |
| **Context Exhaustion Failures** | High (memory saturation) | Moderate (long task sprawl) | **Zero** |
| **Edge-Case Regression Rate** | 22% (architectural drift) | 12% (author bias) | **< 1.5%** (adversarial caught) |
| **End-to-End Task Wall Time** | 18.4 min (merge retries) | 14.2 min (sequential) | **6.8 min** |

By restricting file mutation to a single process, outsourcing non-coding subtasks to isolated workers, and gating output through an unpolluted reviewer, development speed improved by over 2x while regression rates dropped to near zero.

---

## The Rules for Implementation

If you are designing an agentic pipeline for your own team or infrastructure, follow these four rules:

1. **One Writer at a Time**: Never allow two LLM threads to possess write tools over the same working directory. If you need parallel coding, use distinct git worktrees with strict human-gated PR merges.
2. **Workers Return Handles, Not Transcripts**: When delegating research or asset creation, instruct subagents to write results to disk and return a file path. Never serialize intermediate multi-page reasoning back into the orchestrator.
3. **Reviewers Must Be Stateless**: The model auditing your code should never be the model that wrote it, and it should never see the author's conversational struggle. Give it the diff and tell it to break it.
4. **Index Symbols Before Spawning Agents**: Raw text search inside a 50,000-line repository is an invitation for token bankruptcy. Pre-index your AST into a graph database or symbol index so agents can locate dependencies with micro-queries.

---

## Want This for Your Business?

I design, implement, and operate autonomous agent infrastructure, custom internal tooling, and self-hosted automation systems for companies that need software shipped reliably without human babysitting.

Whether you are looking to deploy an automated technical pipeline, build custom internal tools, or eliminate manual operations with robust agent architectures:

- **WhatsApp**: [Chat directly on WhatsApp (+60 12-797 2969)](https://wa.me/60127972969)
- **Email**: [me@hoelee.com](mailto:me@hoelee.com?subject=Inquiry%20regarding%20AI%20agent%20orchestration)
- **Website**: [hoelee.com](https://hoelee.com)
````

---

## 文章 02 / 11：什么时候该让 AI 操控物理鼠标（以及什么时候绝对不要用）
- **Slug**: `when-to-give-your-agent-physical-clicks`
- **英文标题**: When to Give Your Agent Physical Clicks (And When to Never Touch Them)
- **分类**: `case-studies` | **发布日期**: 2026-05-28
- **线上地址**: https://blog.hoelee.com/posts/when-to-give-your-agent-physical-clicks/ (EN) | https://blog.hoelee.com/posts/zh/when-to-give-your-agent-physical-clicks/ (ZH)
- **本篇审核状态**: ⏳ 待审核 (Pending)

### ✍️ 顾问审阅意见区 (Review Feedback)
```text
【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：
```

### 🇨🇳 中文版原文 (Chinese Markdown Twin)
````markdown
---
title: "什么时候该让 AI 操控物理鼠标（以及什么时候绝对不要用）"
description: "AI Agent 四层自动化架构决策模型：深度对比确定性 API、命令行脚本、后台无感无接触操控与物理桌面模拟的选型边界。"
pubDate: 2026-05-28
updatedDate: 2026-10-25
category: case-studies
tags: ["ai-agents", "automation", "windows", "computer-use", "architecture"]
ogImage: /og/zh-when-to-give-your-agent-physical-clicks.jpg
banner: /banners/when-to-give-your-agent-physical-clicks.jpg
draft: false
---

## 商业价值与背景：通用桌面 AI 的幻觉

在很多关于自主智能体（AI Agent）的公开演示中，桌面操控的效果最具视觉冲击力：大模型看一眼屏幕截图，计算出按钮所在的 (x, y) 像素坐标，物理光标在屏幕上划过并精准完成点击。

对于正在寻求企业内部降本增效的企业主和技术主管来说，这看起来像是一颗银弹：*“我们不再需要花费昂贵成本去订制 API 接口了，直接买一套 AI 软件，让它像真人员工一样对着电脑点点点不就行了？”*

但在真正的生产运维实践中，如果把模拟物理 GUI 作为主力自动化方案，往往会变成灾难。物理点击极其脆弱、耗时漫长、极易被系统通知弹窗阻断，而且会严重干扰坐在同一台电脑前正常工作的真人。

在过去一年将各类 AI 工作流推向实际业务的过程中，我沉淀出了一套**四层自动化决策阶梯**。只有搞清楚各层的优劣，才能让物理点击工具真正发挥价值。

---

## 四层自动化梯队模型

每一个交由 AI Agent 处理的任务，都应尽可能在最底层、最具确定性的梯队解决：

```text
第一梯队：确定性 API / 数据库直连  （耗时：~10ms   · 可靠性：99.99%）
   │
第二梯队：无头 CLI / 子进程命令行   （耗时：~200ms  · 可靠性：99.9%）
   │
第三梯队：后台无感无障碍驱动 (CUA)  （耗时：~1-2s   · 可靠性：95%）
   │
第四梯队：前台物理输入模拟 (MCP)    （耗时：~3-5s   · 可靠性：80%）
```

### 第一梯队：确定性 API 与数据库直连
只要系统暴露了数据端点，绝不要碰 UI 界面。操作 MySQL、Docker、Portainer 或支付网关，必须通过 RESTful API、SQL 语句或受检的 SDK 完成。这种方式毫秒级响应、上下文完全隔离，且返回的是结构化数据而非含糊的像素点。

### 第二梯队：无头 CLI 与子进程
如果没有现成的网络 API，第二优先级是寻找命令行工具。代码修改、文件处理、日志分析或视频压制，完全可以通过 `terminal`、`git`、`ffmpeg` 或本地脚本静默执行。模型无需移动鼠标，不仅速度快出几十倍，而且不会因界面卡顿而中断。

### 第三梯队：后台无感无障碍驱动（CUA）
当目标软件是一个图形界面且没有 CLI 时，首选是**后台无感自动化**（例如 Hermes 内置基于 `cua-driver` 的 `computer-use`）。
- 指令通过系统底层辅助功能框架（Windows UIA / macOS AX）直接派发给目标控件。
- **真人的物理鼠标光标完全不会被移动。**
- 窗口不需要前置聚焦。你可以在前台敲代码或写邮件，而 Agent 在后台最小化或未激活的窗口里完成点击。

### 第四梯队：前台物理输入模拟（Windows-MCP）
第四梯队是最后的防线：调用类似 `windows-mcp` 的工具，截取屏幕、计算全屏像素坐标，并通过系统底层的 `SendInput` 派发真实的硬件级鼠标键盘事件。

---

## 什么时候真正需要第四梯队（物理点击）？

尽管第四梯队的成本最高、脆弱性最强，但在以下三大现实业务场景中，前三梯队会彻底失效，这正是 `windows-mcp` 展现价值的地方：

### 1. 闭源陈旧的 Win32 遗留企业软件
许多使用 Delphi、VB6 或老旧 C++ 编写的中小企业财务、仓储、ERP 客户端，根本没有实现现代 Windows UIAutomation 接口。后台 UIA 树只能看到一个空白容器，也没有任何命令行接口。此时，结合视觉截图（`Screenshot`）与物理坐标点击（`Click`）是让 AI 能够接管这套流程的唯一技术手段。

### 2. 操作系统原生文件选择与保存对话框
在自动化驱动网页导出报表或桌面软件另存为时，经常会弹出 Windows 原生文件对话框（`comdlg32.dll` / 资源管理器窗口）。无头浏览器无法穿透操作系统的原生对话框，而 `windows-mcp` 可以识别“另存为”输入框、填入绝对路径，并通过键盘快捷键 `Enter` 触发保存。

### 3. 跨非联网软件的剪贴板与系统级提示
两款互不联网的单机软件之间流转非结构化数据时，常需要借助系统剪贴板（`Clipboard` 的读取与写入）完成中转；或者在任务结束时，通过系统的原生 Toast 弹窗（`Notification`）提示人工介入。

---

## 典型反面模式：哪些事情绝对不要用物理点击？

在代码审查和自动化诊断中，我经常看到把第四梯队当成万能钥匙的典型反模式：

- **让 Agent 在编辑器里物理打字写代码**：让 AI 截图、点击 VS Code 编辑区、模拟敲键盘写代码不仅速度奇慢、极易漏字，还会巨额消耗图像 Token。应直接使用文件工具（`patch`、`write_file`）。
- **通过点击图形浏览器来抓取公开网站数据**：物理操作前台 Chrome 浏览器的稳定性远低于无头浏览器或直接发起 HTTP 请求。
- **人在使用电脑时触发物理操作**：因为第四梯队会抢占真实鼠标和键盘焦点，如果你正在打字回复客户，物理点击会瞬间夺走焦点并导致输入错乱。

---

## 选型矩阵总结

```text
┌────────────────────────────────────────────────────────┐
│                   自动化架构选型矩阵                    │
├──────────────────────┬─────────────┬───────────────────┤
│ 业务场景             │ 正确工具    │ 所属梯队          │
├──────────────────────┼─────────────┼───────────────────┤
│ 数据库查询 / API交互 │ REST / SQL  │ 第一梯队（API）   │
│ 代码维护 / 运维脚本  │ Terminal    │ 第二梯队（CLI）   │
│ 无需前置的软件自动化 │ cua-driver  │ 第三梯队（后台UIA）│
│ 遗留Win32 / 另存对话框│ windows-mcp │ 第四梯队（物理模拟）│
└──────────────────────┴─────────────┴───────────────────┘
```

把自动化视为一个逐步升级的阶梯，而不是一把万能锤子，是区分脆弱的实验室 Demo 与能够连续稳定运行数月的生产级工作流的分水岭。

---

## 想为你的企业构建这套自动化体系？

我为马来西亚及海外中小企业量身定制并审查 AI 自动化管线——从纯后台的数据流转、智能客服 Agent 到跨越遗留桌面软件的自动化中转。如果你的团队每天仍在耗费大量人工在老旧软件之间机械地搬运数据，我可以为你搭建 7×24 小时高可靠运行的自动化架构。

直接联系我：
- **WhatsApp**: [+60 12-797 2969](https://wa.me/60127972969)
- **邮箱**: [me@hoelee.com](mailto:me@hoelee.com?subject=AI%20桌面自动化咨询)
- **主页**: [hoelee.com](https://hoelee.com)
````

### 🇬🇧 英文版原文 (English Markdown Primary)
````markdown
---
title: "When to Give Your Agent Physical Clicks (And When to Never Touch Them)"
description: "A 4-tier automation matrix for AI agents: comparing deterministic APIs, shell execution, background accessibility, and physical desktop simulation."
pubDate: 2026-05-28
updatedDate: 2026-10-25
category: case-studies
tags: ["ai-agents", "automation", "windows", "computer-use", "architecture"]
ogImage: /og/when-to-give-your-agent-physical-clicks.jpg
banner: /banners/when-to-give-your-agent-physical-clicks.jpg
draft: false
---

## Why It Matters: The Illusion of Universal Desktop AI

When autonomous agent frameworks demonstrate desktop automation, the visual appeal is undeniable: an LLM captures a screenshot, determines the (x, y) coordinates of a button, moves your cursor across the screen, and clicks it.

To business owners and developers trying to automate internal workflows, it looks like a silver bullet: *"We don't need expensive API integrations anymore; we'll just have an AI click through our software like a human employee."*

In real-world production, treating physical GUI simulation as your primary automation strategy is a reliability nightmare. Physical clicks are fragile, slow, susceptible to OS notification popups, and violently disrupt the human user who is trying to use the same machine.

Over the past year of deploying AI workflows across production infrastructure, I have converged on a **4-tier automation hierarchy**. Here is where physical clicks genuinely shine—and the three higher tiers you should always exhaust first.

---

## The 4-Tier Automation Hierarchy

Every automation request an AI agent handles should be resolved at the lowest applicable tier:

```text
Tier 1: Deterministic API / Database  (Speed: ~10ms   · Reliability: 99.99%)
   │
Tier 2: Headless CLI / Subprocess     (Speed: ~200ms  · Reliability: 99.9%)
   │
Tier 3: Background Accessibility/CUA  (Speed: ~1-2s   · Reliability: 95%)
   │
Tier 4: Foreground Physical Clicks    (Speed: ~3-5s   · Reliability: 80%)
```

### Tier 1: Deterministic APIs & Direct Queries
If an endpoint exists, never touch a UI. Interacting with MySQL, Docker, Portainer, or Stripe should happen through REST endpoints, SQL queries, or verified SDK calls. It is instantaneous, completely isolated, and returns structured data rather than ambiguous pixels.

### Tier 2: Headless CLI & Subprocesses
If there is no remote API, look for a command-line utility. Editing code, running migrations, parsing logs, or transforming media should happen through `terminal`, `git`, `ffmpeg`, or headless scripts. The agent never moves your mouse, and executions finish in milliseconds.

### Tier 3: Background Accessibility & Window Messages (CUA)
When dealing with GUI applications, the first choice is **background automation** (such as Hermes's `computer-use` powered by `cua-driver`).
- Messages are delivered via native accessibility APIs (Windows UIA / macOS AX).
- **Your mouse cursor never moves.**
- The application does not need to be focused or brought to the front. You can type an email while the agent clicks a button in an unfocused background window.

### Tier 4: Foreground Physical Input Simulation (Windows-MCP)
Tier 4 is the final resort: using tools like `windows-mcp` to take full-screen screenshots, compute virtual desktop pixel coordinates, and dispatch hardware-level `SendInput` events.

---

## When Tier 4 Physical Clicks Are Justified

There are specific operational gaps where Tiers 1 through 3 fail completely. This is where `windows-mcp` earns its keep:

### 1. Proprietary Win32 Legacy Software
Many accounting, industrial dispatch, CAD, and local inventory tools written in Delphi, VB6, or legacy C++ do not implement modern Windows UIAutomation interfaces. Background UIA trees return empty containers, and the software has no CLI. Physical screen capture (`Screenshot`) combined with coordinate clicks (`Click`) is the only way an agent can operate them.

### 2. OS-Native File & Save Dialogs
When automating a browser or desktop application to export a report, you frequently hit a native Windows Common File Dialog (`comdlg32.dll` / Explorer file picker). Headless browsers cannot interact with the operating system's native file picker window. `windows-mcp` can identify the "Save As" text box, paste the target path, and hit `Enter` via `Shortcut`.

### 3. Native Clipboard & System Notifications
Bridging unstructured data between two non-networked desktop programs often requires reading and writing the OS clipboard (`Clipboard mode="get"` / `"set"`) or alerting the human operator via a Windows Toast notification (`Notification`).

---

## Anti-Patterns: Where Physical Clicks Fail Catastrophically

In production audits, I regularly catch teams using Tier 4 for tasks that belong in Tiers 1 or 2:

- **Writing Code in an IDE GUI**: Asking an agent to click VS Code's editor pane and type characters is catastrophically slow, prone to dropped keystrokes, and burns thousands of tokens per file. Use filesystem tools (`patch`, `write_file`).
- **Web Scraping via Browser Clicking**: Driving a visible Chrome window with mouse clicks is 20x slower and far more brittle than headless Chromium or an HTTP fetch tool.
- **Background Execution While Working**: Because Tier 4 moves your actual mouse and steals window focus, running an agent workflow while you are actively writing an email will cause the agent to hijack your keystrokes.

---

## Architecture Summary

```text
┌────────────────────────────────────────────────────────┐
│                   Automation Matrix                    │
├──────────────────────┬─────────────┬───────────────────┤
│ Use Case             │ Right Tool  │ Tier              │
├──────────────────────┼─────────────┼───────────────────┤
│ Database / API       │ REST / SQL  │ Tier 1 (API)      │
│ File edits / builds  │ Terminal    │ Tier 2 (CLI)      │
│ Unfocused browser UI │ cua-driver  │ Tier 3 (UIA/AX)   │
│ Legacy Win32 / Save  │ windows-mcp │ Tier 4 (Physical) │
└──────────────────────┴─────────────┴───────────────────┘
```

Treating desktop automation as an escalation ladder rather than a single hammer is what separates fragile prototypes from production workflows that run for months without human intervention.

---

## Want This for Your Business?

I design, build, and audit automated AI agent pipelines for small and medium businesses—from headless backend data extractors to robust desktop legacy software bridges. If your team is wasting hours manually re-typing data between desktop tools, I can build an automation architecture that runs reliably 24/7.

Reach me directly at:
- **WhatsApp**: [+60 12-797 2969](https://wa.me/60127972969)
- **Email**: [me@hoelee.com](mailto:me@hoelee.com?subject=AI%20Desktop%20Automation%20Consulting)
- **Website**: [hoelee.com](https://hoelee.com)
````

---

## 文章 03 / 11：为什么我要做视频精读知识库：把收藏夹吃灰的短视频变成可检索笔记
- **Slug**: `why-i-built-a-video-knowledge-base`
- **英文标题**: Why I Built a Video Knowledge Base: Turning Saved Short Videos into Searchable Notes
- **分类**: `case-studies` | **发布日期**: 2024-11-28
- **线上地址**: https://blog.hoelee.com/posts/why-i-built-a-video-knowledge-base/ (EN) | https://blog.hoelee.com/posts/zh/why-i-built-a-video-knowledge-base/ (ZH)
- **本篇审核状态**: ⏳ 待审核 (Pending)

### ✍️ 顾问审阅意见区 (Review Feedback)
```text
【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：
```

### 🇨🇳 中文版原文 (Chinese Markdown Twin)
````markdown
---
title: "为什么我要做视频精读知识库：把收藏夹吃灰的短视频变成可检索笔记"
description: "为什么平台收藏夹是学习的死胡同？记录我如何利用本地语音识别与大模型构建视频精读知识库，实现文字点击跳播与毫秒级全文检索。"
pubDate: 2024-11-28
category: case-studies
tags: ["ai", "video-processing", "whisper", "astro", "knowledge-management"]
ogImage: "/og/zh-why-i-built-a-video-knowledge-base.jpg"
banner: "/banners/why-i-built-a-video-knowledge-base.jpg"
draft: false
---

和很多持续学习的开发者一样，我的 B站、YouTube 和抖音收藏夹里常年躺着数百个视频。这里有一段 60 秒讲透 Transformer 注意力机制的精华讲解，那里有一段 2 分钟拆解系统架构的实战经验。

然而，收藏夹越来越长，真正要在几个月后复习或查证某个具体知识点时，根本找不到。

把视频丢进收藏夹，本质上是一种心理安慰。这篇文章记录了我为什么要立项搭建这套视频学习精读网站，如何用本地算力实现自动化转写与提炼，以及把碎片化视频转变为永久数字资产的思考。

---

## 1. 为什么平台收藏夹是学习的死胡同？

短视频是目前打破信息差最高效的载体之一。优秀的博主能用 1 到 3 分钟把复杂的概念讲得通俗透彻。但在知识沉淀的维度上，视频载体有三个天然缺陷：

1. **内容不可检索**：平台搜索只能检索视频标题和标签，讲者在第 1 分 15 秒说的那句核心结论，搜索引擎根本看不见。
2. **复习摩擦力极高**：扫一眼 500 字的精读总结只需 30 秒，而重新看一段视频需要 2 到 3 分钟被动听讲，中间往往还夹杂着求关注、引流话术与片头片尾。
3. **内容极易失效**：原帖下架、博主销号或外链失效时常发生，收藏夹瞬间变成灰白色的失效链接列表。

要真正把视频内容沉淀为个人知识库，必须满足一个硬性要求：**收藏的视频必须全自动转写为逐字稿、提炼为结构化要点与思维导图，并支持全文毫秒级检索与精确到秒的点击跳播。**

---

## 2. 架构设计与工程取舍

在设计这套系统时，我结合现有的家庭基础设施，确立了四项核心原则：

```
[ 本地收集视频 / 收件箱 ]
        │
        ▼
[ 本地 GPU 算力流水线 ] ────► Silero-VAD + Faster-Whisper (本地零云端调用费)
        │
        ├──────────────────► 大模型 6 维精读提炼 (TL;DR、思维导图、时间戳)
        │
        ▼
[ 严格解耦的数据层 ]
   ├── src/content/videos/{slug}.md    (元数据 + AI 结构化精读总结)
   └── src/data/transcripts/{slug}.json (毫秒级分句逐字稿 [start, end, text])
        │
        ▼
[ Astro 5 纯静态门户 ] ────► Pagefind 客户端 WASM 离线检索 (<20ms 响应)
        │
        ▼
[ 双轨流媒体分发 ] ────────► 官方 iFrame 嵌入优先 / NVENC 压缩后托管于 Cloudflare R2
```

### 总结正文与时间轴严格分离
在早期原型设计中，最容易踩的坑是把带时间戳的逐字稿全部硬编码进同一个 Markdown 正文里。这不仅导致 Markdown 文件膨胀到数千行，更致命的是，一旦后续调整 Prompt 想重新跑一遍 AI 总结，就会把人工校对好的时间戳冲掉。

规范的做法是：`src/content/videos/{slug}.md` 纯粹存放结构化要点与元数据，而分句时间轴存放于独立的 `src/data/transcripts/{slug}.json` 中。

### 本地算力闭环，零云端转写成本
不用云端按分钟计费的语音转写 API，流水线直接运行在宿主机的 NVIDIA RTX 3060 显卡上。通过 `faster-whisper` 与 `Silero-VAD` 结合，自动切除无声段，并按标点符号聚合成 3 到 8 秒的自然语义分句。

### 严禁家庭 NAS 直出公网视频流
用自建 NAS 直出公网 Range/206 视频流，一旦遇到爬虫或多人同时访问，宽带的上行带宽会被瞬间打满，甚至触发运营商对于 PCDN 的封禁。公网访问采用双轨策略：长青视频优先使用 B站/YouTube 官方嵌入播放器；自存短视频经 NVENC 压缩为 720p H.264 后托管在免流量费的 Cloudflare R2。

### 基于 Pagefind 的毫秒级离线搜索
前台不维护重型的 Meilisearch 或 Elasticsearch 数据库，直接编译为 Astro 静态站点并利用 Pagefind 构建分片索引。读者在浏览器端搜索时，通过 WebAssembly 按需拉取几十 KB 的分片，零服务器内存常驻，中文关键词检索延迟在 20ms 以内。

---

## 3. 学习闭环带来的实际改变

从“盲目收藏”转变为“流水线吞吐”，知识吸收的效率发生了根本改变：

- **从囤积变为检索**：不再翻看长长的收藏列表。按下 `Cmd+K` 输入技术术语（如 `RAG` 或 `Agent`），直接定位到哪段视频的第几秒讲过这个概念。
- **Click-to-Seek 精准跳播**：点击精读要点或时间戳徽章，播放器直接跳到关键帧播放，跳过废话，只看核心论点。
- **永久自主沉淀**：所有笔记与逐字稿均以 Markdown 与 JSON 保存在 Git 仓库中，即便第三方平台删帖，核心知识依然永久归档。

---

## 4. 业务咨询与合作

搭建全自动化知识提取流水线、本地大模型工作流以及高性能静态门户是我擅长的工作。

如果你的企业或团队也积累了大量内部培训录屏、会议回放或知识资产，需要将其转化为可检索、支持时间戳精读的企业内部知识库：

- **WhatsApp 直接沟通**：[点击打开 WhatsApp (012-7972969)](https://wa.me/60127972969)
- **电子邮箱**：[me@hoelee.com](mailto:me@hoelee.com?subject=关于视频知识库与内训检索系统的合作咨询)
- **官方主页**：[hoelee.com](https://hoelee.com)
````

### 🇬🇧 英文版原文 (English Markdown Primary)
````markdown
---
title: "Why I Built a Video Knowledge Base: Turning Saved Short Videos into Searchable Notes"
description: "Why saving short videos to bookmarks is a dead end, and how I built a local ASR and AI pipeline to convert them into searchable notes with Click-to-Seek video sync."
pubDate: 2024-11-28
category: case-studies
tags: ["ai", "video-processing", "whisper", "astro", "knowledge-management"]
ogImage: "/og/why-i-built-a-video-knowledge-base.jpg"
banner: "/banners/why-i-built-a-video-knowledge-base.jpg"
draft: false
---

Like many developers and lifelong learners, I found myself accumulating hundreds of bookmarks across platforms like Bilibili, YouTube, and Douyin. A 60-second video explaining a Transformer concept here, a 2-minute architectural teardown there. 

The bookmark list kept growing, but when I actually needed that one specific explanation six months later, it was impossible to retrieve.

Saving videos into platform favorites is a dead end. This post explains why I decided to solve this problem by building a dedicated learning portal, how I designed the architecture around local compute, and the lessons learned turning fleeting video content into a permanent, searchable knowledge asset.

---

## 1. Why Video Bookmarks Fail

Short-form video is currently the fastest medium for breaking information asymmetry. Creators pack high-density insights into 1 to 3 minutes. However, as a long-term knowledge repository, video suffers from three fatal flaws:

1. **Unsearchable inside the content**: Platform search only indexes video titles and uploader tags. The actual golden nugget spoken at minute 01:15 is completely invisible to search engines.
2. **High review friction**: Reading a 500-word summary takes 30 seconds; watching a video requires 2 to 3 minutes of passive listening, often padded with intro hooks and subscriber calls.
3. **Fragile shelf life**: Links rot, accounts get banned, and videos get taken down without warning.

To turn passive consumption into actionable learning, I set a clear requirement: **every video I save must be transcribed, summarized into structured bullet points and mindmaps, and made searchable down to the exact second.**

---

## 2. Architecture & Design Principles

When designing the system, I established four non-negotiable architectural principles based on my existing home infrastructure:

```
[ Local Input / Inbox ]
        │
        ▼
[ Local GPU Pipeline ] ────► Silero-VAD + Faster-Whisper (Zero cloud API fees)
        │
        ├──────────────────► LLM 6-D Synthesis (TL;DR, Mindmap, Timestamps)
        │
        ▼
[ Decoupled Data Layer ]
   ├── src/content/videos/{slug}.md  (Frontmatter + Structured Insights)
   └── src/data/transcripts/{slug}.json (Millisecond Cues [start, end, text])
        │
        ▼
[ Astro 5 Static Portal ] ──► Pagefind Client-Side WASM Search (<20ms latency)
        │
        ▼
[ Dual-Track Video Sync ] ──► Official iFrame Embeds or NVENC H.264 on Cloudflare R2
```

### Decouple Summaries from Transcripts
The biggest architectural trap in early prototypes was concatenating timestamped transcripts directly inside the Markdown content body. This caused Markdown files to balloon to thousands of lines, cluttered formatting, and made re-running LLM summaries destructive. 

The fix: keep `src/content/videos/{slug}.md` strictly for high-level synthesis and Frontmatter, while storing millisecond cues in `src/data/transcripts/{slug}.json`.

### Zero Cloud STT Fees
Rather than paying recurring per-minute charges to cloud transcription APIs, the pipeline runs locally on an NVIDIA RTX 3060 (12GB VRAM). Using `faster-whisper` and `Silero-VAD`, speech is automatically transcribed into clean 3-to-8 second semantic sentences with sub-second timestamps.

### Never Stream from Residential NAS
Serving video streaming directly from a residential home NAS to public web visitors saturates home broadband upload bandwidth and risks ISP restrictions. Public playback uses a dual-track strategy: official iframe embeds (Bilibili/YouTube) whenever possible, or lightweight NVENC 720p H.264 streams hosted on Cloudflare R2.

### Sub-20ms Search with Pagefind
Instead of maintaining an always-on search database like Meilisearch or Elasticsearch, the front portal compiles to a pure static Astro site indexed by Pagefind. Search queries run in-browser via WebAssembly, delivering instantaneous Chinese keyword lookups with zero server memory footprint.

---

## 3. What Changed in My Daily Learning Loop

Replacing passive "saving for later" with an automated ingestion pipeline fundamentally changed how I retain information:

- **From Hoarding to Searching**: Instead of scrolling through endless favorited lists, I hit `Cmd+K` in the portal, type a keyword like `RAG` or `Agent`, and instantly see matching quotes highlighted in the transcript.
- **Click-to-Seek Precision**: Clicking on any key point jumps the video player directly to that second, bypassing creator filler and replaying only the core explanation.
- **Permanent Ownership**: The knowledge lives in plain text and JSON in a Git repository, completely independent of third-party platforms.

---

## 4. Work With Me

Building automated knowledge ingestion pipelines, self-hosted AI workflows, and high-performance static portals is part of what I do. 

If your organization has internal training videos, product recordings, or course archives that you need converted into a fast, searchable internal knowledge hub:

- **WhatsApp**: [Chat directly on WhatsApp](https://wa.me/60127972969)
- **Email**: [me@hoelee.com](mailto:me@hoelee.com?subject=Inquiry%20regarding%20video%20knowledge%20base%20portal)
- **Website**: [hoelee.com](https://hoelee.com)
````

---

## 文章 04 / 11：为什么你的无头浏览器不是万能爬虫
- **Slug**: `why-your-headless-browser-cant-scrape-everything`
- **英文标题**: Why Your Headless Browser Can't Scrape Everything
- **分类**: `case-studies` | **发布日期**: 2026-01-21
- **线上地址**: https://blog.hoelee.com/posts/why-your-headless-browser-cant-scrape-everything/ (EN) | https://blog.hoelee.com/posts/zh/why-your-headless-browser-cant-scrape-everything/ (ZH)
- **本篇审核状态**: ⏳ 待审核 (Pending)

### ✍️ 顾问审阅意见区 (Review Feedback)
```text
【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：
```

### 🇨🇳 中文版原文 (Chinese Markdown Twin)
````markdown
---
title: "为什么你的无头浏览器不是万能爬虫"
description: "两个二手交易平台，两面完全不同的墙：闲鱼用「非法访问」把无头浏览器挡在门外，Carousell 却照常吐数据。教训是——在伸手拿 Playwright 之前，先搞清页面到底是怎么把数据交给你的。"
pubDate: 2026-01-21
updatedDate: 2026-09-10
category: case-studies
tags: [scraping, playwright, browserless, docker, goofish, carousell, anti-bot]
ogImage: /og/zh-why-your-headless-browser-cant-scrape-everything.jpg
banner: /banners/why-your-headless-browser-cant-scrape-everything.jpg
---

我在短短几周里，分别在两个不同的二手交易平台上撞上了同一堵墙，而这组对照
教给我的，比学一年教程还多。

在 **Carousell** 上，一个无头浏览器把新上架的商品连价格、成色、图片链接一起
抓进数据库——毫无问题，稳定跑了几个月。在**闲鱼**上，一模一样的设置，每次都
只返回同样三个词：

```
为了保障您的体验，请使用正常浏览器访问闲鱼~
```

两个平台，同一个浏览器，结果完全相反。这篇文章讲的就是**为什么**，以及那个
本该在一开始就做、能帮我省下大半天时间的决定。

## 一个从一开始就没问对的问题

我想在闲鱼上搜索商品，把结果抽出来——标题、价格、链接。第一个跳出来的答案
自然是「用无头浏览器」。每个人都会这么答：用 Playwright 或 Puppeteer 把页面
渲染出来，等 JavaScript 执行完，再读 DOM。

于是我就真这么搭了。我部署了一个自托管的 `browserless` 容器——一个现成的
「无头 Chrome 即 API」，把它指向闲鱼的搜索页。然后我花了几个小时，把那个工具
暴露出来的每个开关都试了个遍：

- `--disable-blink-features=AutomationControlled` 隐藏自动化标志
- 用 `evaluateOnNewDocument` 把 `navigator.webdriver` 抹成 `undefined`
- 一个逼真的 Chrome user-agent
- 一个 `stealth` 启动选项

每一次，商品列表都卡在「加载中…」——页面框架渲染出来了，页头渲染出来了，
页脚渲染出来了，但真正的商品列表永远加载不出来。返回的 HTML 里价格是 0 条，
商品链接是 0 条。

与此同时，脑子里有个声音一直在提醒我：我的 Carousell 监控器抓的也是同样的
数据，而且**没有任何 stealth、没有任何反检测手段**，却一直稳稳地工作。为什么？

## 换个思路：先看数据是怎么到页面上的

答案根本不在浏览器的那些启动参数里，而在**每个网站是怎么把数据交给页面的**。

**Carousell 是把搜索结果嵌在页面的初始响应里的。** 服务器把商品列表作为
JSON 烤进了 HTML（类似 `__PRELOADED_STATE__` 这种结构）。JavaScript 之后再把
它们增强一下，但一次普通的 HTTP 请求就已经包含了所有的东西——商品 ID、标题、
价格、图片。爬虫甚至不需要渲染任何东西，直接从这个原始 HTML 里把 JSON 解析
出来就行。

**闲鱼什么都不在服务端给。** 它的搜索结果是在页面加载**之后**，才从一个带
签名的 API 端点上拉回来的。这个端点要求：

- 一个由 App 根据请求参数和一个密钥算出来的签名
- 一系列会标记无头浏览器指纹的反爬检查（WebDriver 检测、CDP 痕迹、canvas/
  audio 指纹）

把签名去掉，API 就什么都不返回。把浏览器特征去掉，页面就回你一句「非法访问」。

**服务端渲染的 JSON vs. 带签名的异步 API**——这一个区别，就是全部的游戏。
它让你在一分钟内就判断出「用浏览器渲染」这条路是花一个下午，还是花一个月。

## 无头浏览器为什么在闲鱼上撞了墙

说句实在话：**开源的、自托管的浏览器本身不是隐身工具。**

`browserless`（我部署的那个容器）是个**通用**渲染器。它的免费版能渲染页面、
执行 JavaScript、截图、转 PDF。而反检测的能力——真正的 `stealth` 模式、验证码
识别、解锁代理——明确是**付费的 Cloud/Enterprise 版**才有的。它们文档里写得
毫不含糊。

所以说，我是在拿一个通用渲染器，去硬刚一个整个商业模式都建立在「击败自动化
客户端」之上的平台。这不是配置 bug，是工具和对手之间的错配。

真要抓闲鱼，我有几条重得多的路可选：

1. **逆向 App 的签名算法**，直接调搜索 API。严肃的爬虫都是这么干的——但签名
   会变，所以这是持续的维护成本，不是一次性做完就完事。
2. **买第三方数据接口**（阿里云市场之类卖 `item_search` 端点）。数据干净，但
   按次计费，而且受制于供应商。
3. **驱动一个真实登录的浏览器、低频访问**，指望频率始终低于雷达线。偶尔手动
   查查还行，想稳定的大规模监控，没用。

没有一条是「起个容器就能跑」的。这才是重点。

## 可以复用的检查清单

在给一个新目标动用 Playwright 之前，我现在会按顺序问自己三个问题：

1. **数据到底存在哪里？** 关掉 JavaScript 打开页面（或者直接读原始 HTML）。
   如果数据就在 HTML 里，我根本不需要浏览器——我需要的是一个 HTTP 客户端和一个
   解析器。浏览器纯属浪费。
2. **如果不在 HTML 里，那个异步请求长什么样？** 打开 DevTools → Network，找到
   那个返回数据的 XHR/fetch，看它的 header 和查询参数。有没有签名？它是稳定的，
   还是每个请求都变？
3. **是不是带签名的 API？** 如果是，停下来。先想清楚这个目标值不值得 (a) 逆向
   签名，或者 (b) 付费买数据供应商。如果都不值，诚实的答案就是「这事成本高于
   收益」——能在一个小时内就**知道**这一点，本身就是种胜利。

大多数爬虫教程会跳过这三步，直接跳到「装 Puppeteer」。那是昂贵的那条路——
花一整天用最痛的方式，才搞懂一个 60 秒的调查早就该告诉你的事。

## 如果重来一次我会怎么做

- **在伸手拿浏览器之前，先禁用 JavaScript。** 这是判断「服务端渲染 vs. 客户端
  拉取」最快的信号，十秒钟，零配置。
- **别指望从通用渲染器里白嫖 stealth。** stealth 要么是你花钱买的产品
  （browserless Cloud、ScrapingBee 之类），要么是你自己搭、自己维护的东西。它
  不是一个开源容器里的勾选框。
- **把「渲染」和「提取」分开看。** 无头浏览器返回的是 HTML，它不会直接给你
  结构化的商品字段。就算渲染成功了，还是得有人写选择器或 `evaluate` 脚本。那是
  另一件事——要为它做计划，别以为它是免费的。

## 结果

这组对照沉淀成了一条我现在每个项目都会用的规则：**在选工具之前，先判断目标是
服务端渲染还是 API 门槛。** 在 Carousell（服务端渲染）上，一个普通的 HTTP 解析器
几个月来归档了几千条带价格和图片的商品。在闲鱼（签名 API）上，我诚实地评估了
工作量和维护成本之后选择了停下——对一个偶尔手动搜一下就够用的需求来说，这本身
就是正确的结局。

无头浏览器**有时候**是对的工具——截图、转 PDF、接管一个人工会话，或者目标本身
就是服务端渲染。但它不是一把通用的爬虫钥匙，把它当成钥匙，才是那个昂贵的错误。

---

## 想知道一个网站值不值得花力气去抓？

大多数爬虫项目都在第一个小时内失败，因为没人先去查一眼——这个目标到底允不
允许你用低成本抓到数据。我写代码之前会先做这个判断——我也真的在搭那些能跑起来
的监控器、通知管线，和背后的自托管基础设施。如果你盯上了一个数据源，想知道它是
一个下午的活还是一个月的活，欢迎聊聊：

- 📱 **WhatsApp:** [+60 12-797 2969](https://wa.me/60127972969)
- 📧 **Email:** [me@hoelee.com](mailto:me@hoelee.com)
- 🌐 **Website:** [hoelee.com](https://hoelee.com)
````

### 🇬🇧 英文版原文 (English Markdown Primary)
````markdown
---
title: "Why Your Headless Browser Can't Scrape Everything"
description: "Two second-hand marketplaces, two very different walls: Goofish blocks a headless browser with 'illegal access' while Carousell keeps serving data. The lesson is to read how a page delivers data before you reach for Playwright."
pubDate: 2026-01-21
updatedDate: 2026-09-10
category: case-studies
tags: [scraping, playwright, browserless, docker, goofish, carousell, anti-bot]
ogImage: /og/why-your-headless-browser-cant-scrape-everything.jpg
banner: /banners/why-your-headless-browser-cant-scrape-everything.jpg
---

I hit the same wall a few weeks apart, on two different second-hand marketplaces,
and the contrast taught me more about scraping than a year of tutorials did.

On **Carousell**, a headless browser pulled freshly-listed products into a database
with prices, conditions, and photo URLs — no problem, running for months. On
**Goofish (闲鱼)**, the exact same setup returned the same three words every time:

```
为了保障您的体验，请使用正常浏览器访问闲鱼~
(translation: "to protect your experience, please use a normal browser")
```

Two marketplaces, one browser, opposite outcomes. This is the story of *why*, and
the one decision that would have saved me most of a day if I'd made it first.

## The goal, badly framed

I wanted to search Goofish for products and pull the results out — titles, prices,
links. The obvious first answer is "use a headless browser." It's the answer
everyone gives: render the page with Playwright or Puppeteer, wait for the
JavaScript, read the DOM.

So I built exactly that. I deployed a self-hosted `browserless` container — a
ready-made headless-Chrome-as-an-API — and pointed it at Goofish search. Then I
spent hours trying every lever the tool exposes:

- `--disable-blink-features=AutomationControlled` to hide the automation flag
- patching `navigator.webdriver` to `undefined` via `evaluateOnNewDocument`
- a realistic Chrome user-agent
- a `stealth` launch option

Each time, the product list stayed stuck at "loading…" — the page frame rendered,
the header rendered, the footer rendered, but the actual listings never populated.
The HTML came back with zero prices and zero product links.

Meanwhile I had a memory nagging me: my Carousell monitor grabs the same kind of
data with *no* stealth, *no* anti-detection tricks, and it works flawlessly. Why?

## The frame change: read how the data gets to the page

The answer wasn't in the browser flags. It was in **how each site delivers its data**.

**Carousell embeds its search results in the page's initial payload.** The server
sends the listings as JSON baked into the HTML (a `__PRELOADED_STATE__`-style blob).
The JavaScript enhances them afterward, but a plain HTTP request already contains
everything — product ID, title, price, photos. A scraper can parse that JSON out of
the raw HTML without ever rendering anything.

**Goofish delivers nothing server-side.** Its search results are fetched *after*
the page loads, from a signed API endpoint. That endpoint requires:

- a signature the app computes from request parameters and a secret
- anti-bot checks that flag a headless browser's fingerprint (WebDriver detection,
  CDP artifacts, canvas/audio fingerprints)

Strip the signature away, and the API returns nothing. Strip the browser off, and
the page returns "illegal access."

That single distinction — **server-rendered JSON vs. a signed async API** — is the
whole game. It tells you in one minute whether "render the page in a browser" will
take you an afternoon or a month.

## Why the headless browser hit a wall on Goofish

The honest reality: **an open-source, self-hosted browser is not a stealth tool.**

`browserless` (the container I deployed) is a *generic* renderer. Its free tier
renders pages, executes JavaScript, takes screenshots, converts PDFs. The
anti-detection features — real `stealth` mode, captcha solving, unblocked proxies —
are explicitly the **paid Cloud/Enterprise** tier. The message in their docs is
unambiguous.

So I was asking a general-purpose renderer to beat a platform whose entire
business depends on defeating automated clients. That's not a configuration bug;
it's a mismatch between the tool and the adversary.

To actually scrape Goofish I'd have to choose one of a few much heavier paths:

1. **Reverse the app's signing algorithm** and call the search API directly. This
   is how the serious scrapers do it — but the signature changes, so it's an
   ongoing maintenance cost, not a one-time build.
2. **Buy a third-party data API** (Alibaba Cloud's marketplace and similar sell
   `item_search` endpoints). Clean data, but you pay per call and you're at the
   mercy of the vendor.
3. **Drive a real, logged-in browser at low frequency** and hope the rate stays
   under the radar. Works for occasional manual checks, useless for reliable
   monitoring at scale.

None of these are "spin up a container and go." That's the point.

## The reusable checklist

Before I touch Playwright again on a new target, I now answer three questions in
order:

1. **Where does the data actually live?** Open the page with JavaScript disabled
   (or read the raw HTML). If the data is in the HTML, I don't need a browser at
   all — I need an HTTP client and a parser. The browser is waste.
2. **If it's not in the HTML, what does the async request look like?** Open
   DevTools → Network, find the XHR/fetch that returns the data, and read its
   headers and query params. Is there a signature? Is it stable, or does it change
   per request?
3. **Is there a signed API?** If yes, stop. Decide up front whether the target is
   worth either (a) reverse-engineering the sign, or (b) paying a data vendor. If
   neither, the honest answer is "this costs more than it's worth" — and *knowing*
   that in the first hour is itself a win.

Most scraping guides skip all three and jump straight to "install Puppeteer."
That's the expensive path, the one that burns a day discovering the hard way what
a 60-second investigation would have told you.

## What I'd do differently

- **Disable JavaScript before reaching for a browser.** It's the single fastest
  signal for "server-rendered vs. client-fetched." Ten seconds, no setup.
- **Don't buy stealth from a generic renderer.** Stealth is either a product you
  pay for (browserless Cloud, ScrapingBee, etc.) or something you build and
  maintain. It is not a checkbox in an open-source container.
- **Distinguish "rendering" from "extraction."** A headless browser returns HTML;
  it does not hand you structured product fields. Even when rendering works,
  someone still has to write the selectors or the `evaluate` script. That's a
  separate task — plan for it, don't assume it's free.

## The result

The contrast sharpened into a rule I now apply to every project: **decide whether
the target is server-rendered or API-gated before choosing a tool.** On Carousell
(server-rendered), a plain HTTP parser has archived thousands of listings with
prices and photos for months. On Goofish (signed API), I stopped after an honest
look at the effort and the maintenance burden — which is itself the right
outcome for a project where a manual search once in a while does everything I
needed.

A headless browser is the right tool *sometimes* — for screenshots, PDFs, taking
over a human session, or targets that render server-side. It is not a universal
scraping key, and treating it as one is the expensive mistake.

---

## Want me to look at a site before you commit to scraping it?

Most scraping projects fail in the first hour because nobody checked whether the
target even *lets* data be scraped cheaply. I do that check before writing any
code — and I build the monitors, notification pipelines, and self-hosted
infrastructure that actually run. If you're eyeing a data source and want to know
if it's an afternoon or a month, let's talk:

- 📱 **WhatsApp:** [+60 12-797 2969](https://wa.me/60127972969)
- 📧 **Email:** [me@hoelee.com](mailto:me@hoelee.com)
- 🌐 **Website:** [hoelee.com](https://hoelee.com)
````

---

## 文章 05 / 11：一个字符让我的 authentik CSS 静默失效
- **Slug**: `authentik-css-greater-than-bug`
- **英文标题**: The Character That Silently Broke My authentik CSS
- **分类**: `devops` | **发布日期**: 2026-09-20
- **线上地址**: https://blog.hoelee.com/posts/authentik-css-greater-than-bug/ (EN) | https://blog.hoelee.com/posts/zh/authentik-css-greater-than-bug/ (ZH)
- **本篇审核状态**: ⏳ 待审核 (Pending)

### ✍️ 顾问审阅意见区 (Review Feedback)
```text
【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：
```

### 🇨🇳 中文版原文 (Chinese Markdown Twin)
````markdown
---
title: "一个字符让我的 authentik CSS 静默失效"
description: "我的 authentik 自定义 CSS 看起来完全正确，也匹配到了目标元素，却毫无效果。原因是一个 > 字符被 authentik 转义成了无效文本。"
pubDate: 2026-09-20
updatedDate: 2026-10-16
category: devops
tags: ["authentik", "css", "self-hosting", "debugging", "browser"]
ogImage: /og/zh-authentik-css-greater-than-bug.jpg
banner: /banners/authentik-css-greater-than-bug.jpg
draft: false
---

我花了一个下午去查一条本该三十秒解决的 CSS 规则。

我想隐藏 authentik 登录页脚里的一行——硬编码的「Powered by
authentik」署名。这条规则在我做过的其他项目里从来都是有效的：

```css
ul.pf-c-list > li:last-child {
    display: none !important;
}
```

它毫无作用。不是「看起来有点不对」——那个元素依然完整地显示着。下面
是我追过的四个错误答案、唯一正确的答案，以及我本该第一个就做的调试
动作。

## 为什么这不只是一行页脚的问题

如果你自托管 authentik，曾经把 CSS 粘进
**System → Brands → Custom CSS** 却完全没效果，那你多半以为自己哪里
写错了。你几乎肯定没写错。样式表被接收、被存储、被送到浏览器、也被
解析了——然后静默失效，你够得着的任何日志里都没有报错。

这是最糟糕的一类 bug：没有反馈回路。这篇文章把这条回路还给你。

## 错误答案 #1：这是 shadow DOM，CSS 进不去

我的第一个假设。现代 Web Component 常常把标记藏在 shadow root 后面，
普通的文档 CSS 无法跨越那道边界。authentik 的登录页确实由
Web Component 渲染——我在页面源码里见过 `<ak-flow-executor>` 和
`<ak-drawer>`——所以这看起来显然是答案。

我从打包好的 bundle 里把组件定义读出来确认：

```js
var oe = class extends L {
    createRenderRoot() { return this }
    render() { ... }
}
```

`createRenderRoot(){ return this }` 意味着**没有 shadow root**——组件
直接渲染进 light DOM。普通 CSS 完全可以作用于它。

错误答案。继续。

## 错误答案 #2：CSS 根本没被注入

下一个理论：我的 CSS 压根没进到页面里。我在返回的 HTML 里搜了一个
自己规则里的特征类名：

```
<style data-id="brand-css">.ak-login-container{ padding-top: 16vh; ... }
```

它就在那里，一次就搜到了。authentik 把品牌 CSS 作为
`<style data-id="brand-css">` 块注入到文档头部。注入是正常的。

错误答案。

## 错误答案 #3：优先级问题——PatternFly 赢了

听起来合理。authentik 的界面基于 PatternFly，它带有风格强势的列表
样式。我的规则用了 `!important`，但 `!important` 只在同一个级联层
（layer）内有效——如果 PatternFly 的规则也是 `!important` 且在更靠
后的层里，我的就会输。

到这里我停止猜测，开始实测。我用无头浏览器加载页面，直接问 DOM：

```js
const host = document.querySelector('ak-brand-links');
const li = host.querySelector('li[data-kind="text"]');
return {
  display: getComputedStyle(li).display,
  matchesDataKind: li.matches('ul.pf-c-list > li[data-kind="text"]'),
  matchesLastChild: li.matches('ul.pf-c-list.pf-m-inline > li:last-child'),
};
```

返回结果：

```
display:          "list-item"   ← 没有被隐藏
matchesDataKind:  true          ← 我的选择器是正确的
matchesLastChild: true          ← 这条也正确
```

选择器匹配到了元素，CSS 却没生效。这种组合只有在一种情况下可能：
浏览器解析到的那张样式表里，已经没有我写的那条规则了。

## 错误答案 #4：（没有了——我直接读解析后的 CSS）

于是我读回浏览器 **CSS 解析器**实际登记的规则，而不是页面源码里
写了什么：

```js
for (const sheet of document.styleSheets) {
  if (sheet.ownerNode.getAttribute('data-id') === 'brand-css') {
    for (const rule of sheet.cssRules) console.log(rule.cssText);
  }
}
```

```
"ul.pf-c-list u003e li[data-kind=\"text\"], ul.pf-c-list.pf-m-inline u003e li:last-child { display: none !important; }"
```

找到了。**`u003e`。**

`>` 字符——在我的 CSS 里写对了、在数据库里存对了、API 也返回对了
——被渲染进 HTML 时变成了字面文本 `u003e`。不是 `>` 字符，而是
`u`、`0`、`0`、`3`、`e` 这六个字符。

于是浏览器试着解析这个选择器：

```
ul.pf-c-list u003e li[data-kind="text"]
```

`u003e` 不是组合器。这个选择器是无效的。在逗号分隔的选择器列表里，
无效的那一条会被整条丢弃，所以对浏览器而言这条规则根本不存在
——而对*正确*的选择器字符串调用 `matches()` 依然返回 `true`，这就
是为什么那个元素看起来像是匹配上了什么。

## 修复方式

把 authentik 品牌 CSS 里所有的子代组合器（`>`）删掉，改用后代选择器
——把 `>` 换成空格：

```css
/* 失效 —— > 会变成字面文本 "u003e"，整条规则被丢弃 */
ul.pf-c-list > li[data-kind="text"] {
    display: none !important;
}

/* 有效 —— 后代组合器可以原样通过 */
ul.pf-c-list li[data-kind="text"] {
    display: none !important;
}
```

这就是全部修复——删掉一个字符。

### 这个转义是从哪来的

`\u003E` 是 JSON/JavaScript 对 `>` 的编码方式。authentik 的 Django
模板把品牌配置渲染成一个 JavaScript 对象字面量，而在这个过程里，
某个 HTML/JS 转义器被作用到了 `branding_custom_css` 字符串上。反斜杠
和 `u` 在这个路径的某处被拆开了，于是浏览器收到的是 `u003e`
——没有反斜杠的转义序列——而不是 `>`。

存进去的值是对的。API 的返回也是对的。只有渲染出来的页面是错的：

```bash
# API 返回的内容 —— 正确
curl -s -H "Authorization: Bearer $TOKEN" \
  "https://auth.hoelee.com/api/v3/core/brands/" \
  | python -c "import sys,json;print([b['branding_custom_css'] for b in json.load(sys.stdin)['results']][0])"
# → ul.pf-c-list > li[data-kind="text"] { ... }     ← > 在这里是完整的

# 浏览器实际收到的 —— 已损坏
curl -sL https://auth.hoelee.com/ | grep -o 'ul.pf-c-list[^;]*'
# → ul.pf-c-list \u003E li[data-kind="text"]        ← 被转义了
```

这就是为什么这个 bug 几乎搜不到：任何人通过 API、数据库或后台界面
去调试，看到的都是一张完全正确的样式表。

### 哪些字符能活下来，哪些不能

我探测了哪些字符会被破坏，以便知道还需要避开什么：

| 字符 | 渲染为 | 是否安全 |
|---|---|---|
| `>` | `u003e` | ❌ 选择器失效 |
| `;` | `;` | ✅ |
| `"` | `"` | ✅ |
| `{` `}` `:` | 不变 | ✅ |

实际上只有子代组合器会受影响，因为 `>` 是这些字符里唯一出现在 CSS
*选择器*中的——其余几个只出现在声明里，而声明会被完整保留。

## 在真实浏览器里验证修复

不要通过重读页面源码来验证——那正是陷阱。直接问浏览器元素的几何信息：

```js
const li = document.querySelector('li[data-kind="text"]');
return {
  display: getComputedStyle(li).display,
  visible: li.getBoundingClientRect().height > 0,
};
```

```
display: "none"
visible: false
```

元素已经消失了。这是真实测量，不是推断。

## 我会怎么做不同

**当一个选择器匹配得上、样式却不生效时，立刻去读
`styleSheets[i].cssRules`。** 这一步之前的全部尝试都只是猜测，本来
可以完全跳过。解析后的规则列表是浏览器的基准事实——它用一行就能
告诉你，你*写下*的规则是否就是*存在*的那条规则。

下次我会用的具体顺序：

1. 元素存在且匹配吗？ → `element.matches(selector)`
2. 规则存在于解析后的样式表里吗？ → `cssRules`
3. 这之后才去考虑优先级、级联层和 `!important`

我把这三步做了个反序，这就是为什么花掉了一个下午。

第二条教训更窄，但值得记下来：**转义类 bug 长在「存数据的层」和
「渲染数据的层」之间。** 在做任何其他检查之前，先把这两端的值都看
一遍。我看了数据库，又看了 API，然后断定 CSS 没问题。而 bug 在我
去看的第三个地方。

## 结果

删掉一个字符，隐藏一行页脚，得到一条已经回本的调试规则：在同一次
会话里，我用三分钟就在另一条规则里找到了类似的不匹配——因为我直接
去了 `cssRules`，而不是继续猜。

如果你自托管 authentik，而品牌 CSS 曾经静默失效过——先检查有没有
`>`。

---

## 需要为你的业务做这件事吗？

如果你需要自托管 SSO、加固过的登录页，或者需要有人来排查你现有的
基础设施，这正是我在做的工作。

- **WhatsApp：** [+60 12-797 2969](https://wa.me/60127972969) —— 点击对话
- **Email：** [me@hoelee.com](mailto:me@hoelee.com?subject=Self-hosted%20SSO%20enquiry)
- **网站：** [hoelee.com](https://www.hoelee.com)

我为马来西亚的中小企业搭建 authentik 单点登录、自托管 Docker 环境、
反向代理与隧道——并把过程中学到的东西写下来。
````

### 🇬🇧 英文版原文 (English Markdown Primary)
````markdown
---
title: "The Character That Silently Broke My authentik CSS"
description: "My authentik custom CSS looked correct, matched the right elements, and did nothing. The cause was a single > character that authentik escapes into invalid text."
pubDate: 2026-09-20
updatedDate: 2026-10-16
category: devops
tags: ["authentik", "css", "self-hosting", "debugging", "browser"]
ogImage: /og/authentik-css-greater-than-bug.jpg
banner: /banners/authentik-css-greater-than-bug.jpg
draft: false
---

I spent an afternoon on a CSS rule that should have taken thirty seconds.

I wanted to hide one line in the authentik login page footer — the
hardcoded "Powered by authentik" credit. The rule I wrote has worked in
every other project I've touched:

```css
ul.pf-c-list > li:last-child {
    display: none !important;
}
```

It did nothing. Not "it looked slightly off" — the element stayed
fully visible. What follows is the four wrong answers I chased, the one
correct answer, and the debugging move I should have made first.

## Why this matters beyond one footer line

If you self-host authentik and have ever pasted CSS into
**System → Brands → Custom CSS** and seen zero effect, you have probably
concluded you did something wrong. You almost certainly didn't. The
stylesheet is accepted, stored, served to the browser, and parsed — and
then silently fails, with no error in any log you can reach.

That is the worst kind of bug: no feedback loop. This post gives you the
loop back.

## Wrong answer #1: it's shadow DOM, so CSS can't reach it

My first assumption. Modern web components often hide their markup
behind a shadow root, and normal document CSS cannot cross that
boundary. The authentik login page is rendered by web components — I had
seen `<ak-flow-executor>` and `<ak-drawer>` in the page source — so this
felt obviously right.

I read the component definition out of the shipped bundle to confirm:

```js
var oe = class extends L {
    createRenderRoot() { return this }
    render() { ... }
}
```

`createRenderRoot(){ return this }` means **no shadow root** — the
component renders into the light DOM. Ordinary CSS reaches it just fine.

Wrong answer. Moving on.

## Wrong answer #2: the CSS isn't being injected at all

Next theory: my CSS never made it into the page. I grepped the served
HTML for a distinctive class from my rule:

```
<style data-id="brand-css">.ak-login-container{ padding-top: 16vh; ... }
```

It was there, first try. authentik injects brand CSS as a `<style
data-id="brand-css">` block in the document head. Injection was working.

Wrong answer.

## Wrong answer #3: specificity — PatternFly is winning

Plausible. authentik's UI is built on PatternFly, which ships opinionated
list styles. My rule used `!important`, but `!important` only wins within
the same cascade layer — and if PatternFly's rule were also `!important`
in a later layer, mine would lose.

This is where I stopped guessing and started measuring. I loaded the page
in a headless browser and asked the DOM directly:

```js
const host = document.querySelector('ak-brand-links');
const li = host.querySelector('li[data-kind="text"]');
return {
  display: getComputedStyle(li).display,
  matchesDataKind: li.matches('ul.pf-c-list > li[data-kind="text"]'),
  matchesLastChild: li.matches('ul.pf-c-list.pf-m-inline > li:last-child'),
};
```

The answer:

```
display:          "list-item"   ← not hidden
matchesDataKind:  true          ← my selector IS correct
matchesLastChild: true          ← and so is this one
```

The selectors matched the element. The CSS still didn't apply. That
combination is only possible if the stylesheet the browser parsed no
longer contains the rule I wrote.

## Wrong answer #4: (there wasn't one — I read the parsed CSS)

So I read back what the browser's **CSS parser** had actually
registered, not what the page source said:

```js
for (const sheet of document.styleSheets) {
  if (sheet.ownerNode.getAttribute('data-id') === 'brand-css') {
    for (const rule of sheet.cssRules) console.log(rule.cssText);
  }
}
```

```
"ul.pf-c-list u003e li[data-kind=\"text\"], ul.pf-c-list.pf-m-inline u003e li:last-child { display: none !important; }"
```

There it is. **`u003e`.**

The `>` character — written correctly in my CSS, stored correctly in the
database, returned correctly by the API — was rendered into the HTML as
the literal text `u003e`. Not the `>` character. The six characters
`u`, `0`, `0`, `3`, `e`.

So the browser tried to parse this selector:

```
ul.pf-c-list u003e li[data-kind="text"]
```

`u003e` is not a combinator. The selector is invalid. An invalid selector
in a comma-separated list is discarded, so the rule never existed as far
as the browser was concerned — while `matches()` on the *correct*
selector string still returned `true`, which is why the element looked
like it matched something.

## The fix

Remove every child combinator from authentik brand CSS. Use a descendant
selector instead — a single space instead of `>`:

```css
/* BROKEN — the > becomes literal text "u003e" and the
   selector is discarded */
ul.pf-c-list > li[data-kind="text"] {
    display: none !important;
}

/* WORKS — descendant combinator passes through intact */
ul.pf-c-list li[data-kind="text"] {
    display: none !important;
}
```

That is the whole fix. One character deleted.

### Where the escaping comes from

`\u003E` is how JSON/JavaScript encodes `>`. authentik's Django templates
render the brand config as a JavaScript object literal, and a HTML/JS
escaper is being applied to the `branding_custom_css` string. Backslash
and the `u` get separated somewhere in that path, so the browser receives
`u003e` — the escape without its backslash — instead of `>`.

The stored value is correct. The API response is correct. Only the
rendered page is wrong:

```bash
# What the API returns — correct
curl -s -H "Authorization: Bearer $TOKEN" \
  "https://auth.hoelee.com/api/v3/core/brands/" \
  | python -c "import sys,json;print([b['branding_custom_css'] for b in json.load(sys.stdin)['results']][0])"
# → ul.pf-c-list > li[data-kind="text"] { ... }     ← > is intact here

# What the browser receives — corrupted
curl -sL https://auth.hoelee.com/ | grep -o 'ul.pf-c-list[^;]*'
# → ul.pf-c-list \u003E li[data-kind="text"]        ← escaped
```

This is why the bug is nearly unreachable by searching: anyone debugging
via the API, the database, or the admin UI sees a perfectly correct
stylesheet.

### What survived and what didn't

I probed which characters get mangled, to know what else to avoid:

| Character | Renders as | Safe? |
|---|---|---|
| `>` | `u003e` | ❌ breaks the selector |
| `;` | `;` | ✅ |
| `"` | `"` | ✅ |
| `{` `}` `:` | unchanged | ✅ |

Only the child combinator is affected in practice, because `>` is the
only one of these that appears in a CSS selector — the others only appear
in declarations, which are preserved.

## Verifying the fix in a real browser

Do not verify by re-reading the page source — that was the trap. Ask the
browser for the element's geometry:

```js
const li = document.querySelector('li[data-kind="text"]');
return {
  display: getComputedStyle(li).display,
  visible: li.getBoundingClientRect().height > 0,
};
```

```
display: "none"
visible: false
```

The element is gone. That is a real measurement, not an inference.

## What I'd do differently

**When a selector matches but the styles don't apply, read
`styleSheets[i].cssRules` immediately.** Everything before that step was
speculation I could have skipped. The parsed rule list is the
browser's ground truth — it tells you in one line whether the rule you
*wrote* is the rule that *exists*.

The specific ordering I'd use next time:

1. Does the element exist and match? → `element.matches(selector)`
2. Is the rule present in the parsed stylesheet? → `cssRules`
3. Only then consider specificity, layers, and `!important`

I did those in the opposite order, which is why it took an afternoon.

The second lesson is narrower but worth writing down: **escaping bugs
live between the layer that stores data and the layer that renders it.**
Check the value at both ends before you check anything else. I looked at
the database, then at the API, and concluded the CSS was fine. The bug
was in the third place I looked.

## The result

One character deleted, one footer line hidden, and a debugging rule
that has already paid for itself: in the same session it took me three
minutes to find a similar mismatch in a different rule, because I went
straight to `cssRules` instead of guessing.

If you self-host authentik and your brand CSS has ever silently done
nothing — check for a `>` first.

---

## Want this for your business?

If you need self-hosted SSO, a hardened login page, or someone to debug
the infrastructure you already run, that's the work I do.

- **WhatsApp:** [+60 12-797 2969](https://wa.me/60127972969) — tap to chat
- **Email:** [me@hoelee.com](mailto:me@hoelee.com?subject=Self-hosted%20SSO%20enquiry)
- **Website:** [hoelee.com](https://www.hoelee.com)

I set up authentik single sign-on, self-hosted Docker stacks, reverse
proxies and tunnels for small businesses in Malaysia — and I write up
what I learn while doing it.
````

---

## 文章 06 / 11：验证完全正确的 Forward-Auth 网关——其实并未上线
- **Slug**: `authentik-forward-auth-gate-wasnt-live`
- **英文标题**: The Forward-Auth Gate That Verified Perfectly — and Wasn't Live
- **分类**: `devops` | **发布日期**: 2026-08-12
- **线上地址**: https://blog.hoelee.com/posts/authentik-forward-auth-gate-wasnt-live/ (EN) | https://blog.hoelee.com/posts/zh/authentik-forward-auth-gate-wasnt-live/ (ZH)
- **本篇审核状态**: ⏳ 待审核 (Pending)

### ✍️ 顾问审阅意见区 (Review Feedback)
```text
【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：
```

### 🇨🇳 中文版原文 (Chinese Markdown Twin)
````markdown
---
title: "验证完全正确的 Forward-Auth 网关——其实并未上线"
description: "我的 authentik forward-auth 网关在宿主机上返回了干净的 302 跳转到登录页，公网 URL 却仍然直接服务应用。这个主机名有两条入口层，而我只改了其中一条。"
pubDate: 2026-08-12
updatedDate: 2026-09-29
category: devops
blueprint: self-hosted-infrastructure
blueprintLayer: L2-edge-proxy-runtime
tags: ["authentik", "forward-auth", "cloudflare", "nginx", "docker", "self-hosting"]
ogImage: /og/zh-authentik-forward-auth-gate-wasnt-live.jpg
banner: /banners/authentik-forward-auth-gate-wasnt-live.jpg
draft: false
---

网关搭好了，检查也跑了，通过了。

```
HTTP/1.1 302 Found
location: https://app.example.com/outpost.goauthentik.io/start?rd=...
x-powered-by: authentik
```

未认证的请求被推到了我的 authentik 登录页。带品牌定制、正确、可用。我正准备继续下一件事——除了我自己有一条规矩：任何面向用户的东西，都要用用户的方式去访问它，也就是从外部、走公网。于是我照做了。

```
GET https://app.example.com/          → 200
x-powered-by: Express
```

应用直接应答了。没有登录页、没有重定向、没有网关。完全同一个主机名，一条请求路径被拦住了，另一条门户大开——取决于请求是从哪里进入我的网络的。

这就是我的一整天：问题不在 SSO 配置——那是好的——而在于这个事实：**一个主机名可以由不止一层入口来服务，而我只拦截了其中一层**。

## 为什么这不只是我这一个应用的事

如果你自托管的东西在隧道、反向代理或两者后面，你多半已经做过这个假设：*「服务我域名的是反向代理。」*这个假设一直成立，直到它不再成立，而且它失效的方向最危险——你改的一切看起来都是对的，因为你检查的那条路径确实是对的。

这种失效模式天生就是无声的。没有报错、没有日志行、没有任何提示。你的配置检查能通过，是因为你检查的正是你自己改过的那份配置。而真正的流量——你想要保护的流量——根本不会碰到它。

事后我在同一台宿主机上又发现了两个同样的网关，带着同样的潜在问题，用同样的方式搭建，等着某个信任它们的人。

## 背景：给一个没有 SSO 的应用加 SSO

这个应用是我 NAS 上的一个自托管数据库界面（Docker，发布在宿主端口上，位于 nginx 和 Cloudflare Tunnel 后面——我家实验室的常见形态）。我想在它前面加一道登录墙，让「能从公网访问得到」不再意味着「任何人猜中主机名就能看到登录表单、开始猜密码」。

应用自己的文档宣称支持 SSO。我会把这篇的重点放在入口陷阱上，许可的故事放到另一篇里讲，但简要版本是：宣传的 SSO 是付费许可功能，开源构建带有代码路径但不带使用许可，而过去填补这个空白的社区 fork 已经两年没人维护了。二十分钟调研、两个被放弃的方案、一个结论：**这个应用永远不会为我做 SSO，所以网关必须放在它前面。**

这是件正常且早已被反复实践过的事。authentik 管它叫 proxy provider，流程是：

1. **一个 login flow**——给这个应用用的，有自己的标题、自己的背景，这样落到登录页的用户知道自己是在为哪个系统做认证。
2. **一个 proxy provider**——这个对象知道应用的外部主机名（`https://app.example.com`）和内部地址（`http://app-container:8080`）。
3. **一个 application**——把两者关联起来，挂在内置 outpost 上。outpost 才是真正接收流量、检查会话、并在登录后把请求代理到应用内部主机的组件。
4. **入口（ingress）**——把那个主机名指向 outpost 而不是应用。

第 1–3 步都在 authentik 里，属于配置。第 4 步决定这一切是否真实有效，而我恰恰错在这一步。

第 1–3 步还有两个小坑，因为它们浪费过我的时间：

**对 proxy provider 做部分 PATCH 会被拒绝。**只更新一个字段就会返回：

```json
{"internal_host": ["Internal host cannot be empty when forward auth is disabled."]}
```

serializer 会重新校验整个对象，而一个没有 internal host 的 proxy provider 是无效的——所以只更新一个字段会失败，就像你清空了一个根本没碰过的字段一样。把标识字段和你正在改的字段一起发送：

```bash
curl -s -X PATCH "$AUTH/api/v3/providers/proxy/$PK/" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"mode":"proxy","external_host":"https://app.example.com",
       "internal_host":"http://app-container:8080","skip_path_regex":""}'
```

**authentik API token 可能在会话中途过期。**正常用了一个小时后，每个调用都开始返回 `403 {"detail":"Token invalid/expired"}`。注意这个状态码：`403` 的意思是「我们知道你是谁，但你不能这么做」——只有响应体才会告诉你 token 已经死了。如果你在脚本里缓存了 token，当调用开始失败时，去数据库里重新读一次，而不要假定是权限问题。

## 陷阱：我验证的是我改过的那一层

这是我运行的检查，在宿主机本机：

```bash
curl -sk -o /dev/null -w '%{http_code} -> %{redirect_url}' \
  -H 'Host: app.example.com' https://localhost/
# 302 -> https://app.example.com/outpost.goauthentik.io/start?rd=...
```

这是一个*好*检查。它证明了 nginx 的 vhost 在服务 outpost、outpost 认得这个主机名、provider 已经关联到 application、login flow 是活的。我配置的一切，一条命令就全部证明了是正确的。

但它对「公网流量是否真的到达那个 vhost」什么都证明不了。

原因如下。我的主机名由**两条**相互独立的入口路径服务：

| 路径 | 谁在服务它 | 落在哪里 |
|---|---|---|
| 家里 IP → nginx (443) | 我改过的那个 vhost | outpost ✅ |
| Cloudflare Tunnel (`cloudflared`) | 隧道自身配置里的一条规则 | 直连应用的端口 ❌ |

隧道的 ingress 是一个按主机名求值的小型有序列表：

```
app.example.com     -> http://app-container:8080     ← straight to the app
*                   -> http_status:404
```

那条规则是几年前写的，当时的目标只是「让它能被访问到」。它从来没提过 nginx，因为没必要。于是我漂亮的 vhost 就待在真实请求路径的旁边，拦截着一条公网流量根本不走的路线。

两条路径甚至在同一个主机名上应答，所以 URL 上没有任何线索暗示这中间的差别。

**最后让问题暴露出来的是一个响应头。**被拦截的那次检查返回 `x-powered-by: authentik`，公网那次返回 `x-powered-by: Express`。同一个 URL、同一秒钟，两个不同的服务器在应答。当你怀疑自己盯错了层时，去读响应头——它们会说出应答的软件是谁，而这正是你想问的问题。

与其靠猜，不如直接看自己隧道的规则（用 Cloudflare 的 API，而不是控制台）：

```bash
curl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/cfd_tunnel/$TUNNEL_ID/configurations" \
  -H "Authorization: Bearer $CF_TOKEN" \
  | python -c "import sys,json; [print(r.get('hostname','*'), '->', r.get('service','')) for r in json.load(sys.stdin)['result']['config']['ingress']]"
```

```
git.example.com   -> http://192.168.1.6:6880
app.example.com   -> http://app-container:8080
bot.example.com   -> http://bot-container:8080
*                 -> http_status:404
```

看了一整天 nginx 之后，答案就躺在一行里。

### 通往修复路上的权限墙

编辑那条规则需要一个带 **Account → Cloudflare Tunnel → Edit** 权限的 token。我的没有这个权限，而且任何 zone 级 token 都无法替代：隧道是 account 级的对象，所以 Zone 级的 token 连*读取*都会失败，报 `1001 Not authorized`。如果你只有 zone 级 token，那就改用控制台：**Zero Trust → Networks → Tunnels → 你的隧道 → Public Hostnames**，把规则的 service 改成你的 outpost。

对我来说，正确的目标是与 `cloudflared` 同处一个 Docker 网络的 outpost：

```
app.example.com  ->  http://authentik-server:9000
```

outpost 读取 `Host` 头（隧道会原样保留它）来选择 application，然后按 provider 配置里的内部主机做代理。整个改动就这么点：一个 service 值。我在 authentik 里搭的一切本来都是对的——只是从来不在那条路径上。

## 可选 SSO：拦住界面，放行 API

当网关真正挡在流量前面之后，下一个问题就是 API。主机名级的网关分不清脚本和浏览器，所以只带 token 的 API 调用会被重定向到登录页——它们不会以 JSON 报错失败，而是收到一页 HTML。

如果你想要人类走 SSO、机器不走 SSO，authentik 的 proxy provider 提供一个按路径豁免的配置。界面里叫 **Unauthenticated Paths**；API 字段是 `skip_path_regex`：

```json
{"skip_path_regex": "^/api/.*"}
```

在我自己的实例上验证的结果，同一主机名、同一会话：

| 请求 | 无会话 | 结果 |
|---|---|---|
| `/` 和 `/dashboard` | 是 | `302` → authentik 登录页 |
| `/api/v1/health` | 是 | `200`，来自应用 |
| `/api/v2/tables/.../records` + API token | 是 | `200`，带真实数据 |

在依赖它之前，有两个值得知道的注意事项。这个豁免在 outpost 层生效，所以和 CDN 上的路径拆分不同，它在**每一条**入口路径上都有效。而且它不是按用户区分的：网关作用在请求上，而不是人身上。不存在「某些用户可以跳过 SSO」——要么一条路径对所有人豁免，要么所有人都需要会话。

我先是用了同一个思路的更简单版本：**在网关存在之前，把所有机器调用者从公网主机名上移走。**我的自动化之前一直通过公网 URL 调用应用，毫无必要地被路由到 Cloudflare 再绕回我自己的网络。把它指向容器名（`http://app-container:8080`）缩短了路径、去掉了一种失效模式，也意味着网关不可能弄坏它。如果一道登录墙就能让你的集成瘫痪，那说明这个集成本来就在依赖门是开着的。

## 实话实说：这不是单点登录

关于 forward-auth 网关到底给了你什么，我得直说，因为「SSO」这个词把它说大了。

outpost 知道你是谁，应用不知道。除非应用能够消费上游身份——基于 header 的认证，或原生的 OIDC/SAML——否则网关放你进来之后，它还是会显示自己的登录表单。所以用户体验是：先登录 authentik，再登录应用。

对已经登录 authentik 的用户来说，第一步常常是静默的。但它依然是两套系统、两个会话生命周期、两处可能输错密码的地方。就我而言，应用没有 header 认证，它自己的 OIDC 又是付费许可功能，所以这道网关只是在原有那扇门前面外加了一道——登录次数一点都没减少。

对某些部署来说，这是一笔划算的交易：保留应用自己的登录给它的用户，同时让公网接触不到登录表单。但如果你真正想要的是「我的团队只输一次密码」，那就是一笔糟糕的交易。所以对我来说，诚实的答案是把整个东西重新关掉。

## 正确地回滚（以及救下另外十个应用的算术）

拆除网关这一步，我差点造成比我想防止的故障大得多的宕机。

nginx 的改动是一个大型生成 vhost 文件里的一行。最直接的回滚就是把端口换回去：

```bash
# DON'T — this is the version that would have hurt
sed -i 's#proxy_pass http://localhost:10000;#proxy_pass http://localhost:10380;#' vhost.conf
```

在运行之前，我数了数有多少个 server block 指向同一个端口：

```bash
grep -c 'proxy_pass http://localhost:10000;' vhost.conf
# 11
```

11 个。我的应用只是其中之一。**另外还有十个应用都经由那个 outpost 被网关拦着**，一次无差别的替换会同时悄悄拆掉它们所有人的门——包括那些在做真实认证工作的应用。这个「修复」看起来会像是成功的，同时却会拆掉我家实验室大部分认证。

我实际做的是：

1. **先给当前状态拍快照**，让回滚本身也可回滚：`cp -a vhost.conf vhost.conf.bak-$(date +%Y%m%d-%H%M)-revert`。
2. **拿原始改动前做的备份来 diff，找到确切的那一行。**那次 diff 只显示了一行改动，正好证明我到底动过多少。
3. **只编辑那一行**，然后重新和原始备份 diff，要求结果与备份完全一致再继续。
4. **更新第二个文件**——它镜像同一份设置：NAS 自己的反向代理数据库，会在 DSM 事件发生时重新生成 vhost。我只编辑了那一个键，写入前断言了主机名，绝不整文件恢复。
5. `nginx -t`，然后 reload。

然后还有一个让我又怀疑了二十分钟的细节：

```bash
nginx -s reload && sleep 2 && curl -sk -H 'Host: app.example.com' https://localhost/
# still 302 -> /outpost.goauthentik.io/start
```

回滚看起来像是失败了。其实没有。reload 后几微秒发出的请求，仍可能由正在排空（draining）的旧 worker 应答。几秒后的重测返回 `200 x-powered-by: Express`——直连应用，网关消失了。

**如果配置改动看起来没有生效，先停一拍再重测，然后再去找第二个原因。**在这种速度的请求面前，「配置错了」和「旧进程还在应答」根本无法区分。

## 我会怎么做不同

1. **在为某个主机名搭建任何东西之前，先列出它的每一条入口路径。**我的检查清单现在有四问，按这个顺序问：
   - 这个主机名是否经过 CDN/隧道代理？如果是，它们的规则才是真正起作用的——先查它们，不要留到最后。
   - 反向代理的 vhost 说了什么？
   - 平台是否在别处镜像了这个 vhost、并且可能重新生成它？
   - 是否还有另一个跳点（VPS、第二层代理、厂商隧道）带着一份自己的路由表副本？
2. **从外部验证，并且验证是哪个软件应答的。**在宿主机上用带 `Host:` 头的 `curl` 只能证明你的配置*自洽*。只有来自公网的请求，再加上 `x-powered-by`/`server` 响应头，才能证明它*真的在生效*。
3. **在搭建网关之前，先搞清楚应用能否消费上游身份。**如果不能，你是在加一道登录而不是减一道——这是个要主动做的决定，而不是事后才发现的问题。
4. **替换之前先数一数。**任何你要全局替换的端口或主机名，出现的次数很可能比你想象的多。`grep -c` 只花一秒钟，却能避免一次宕机。
5. **快照、diff、只改一行、再 diff。**备份的唯一职责是回答「我改了多少？」——而一次 diff 就能精确地回答它。

## 结果

网关是能工作的。但它对每个真实用户都不可见，而在我搞清楚它实际交付了什么——在一个本来就有一道登录的应用前面又加了一道——之后，我把它重新关掉了：清空一个字段、取消关联一个 application、恢复一行配置，用一次来自我网络之外的请求做了验证。

两个第一天就能揪出整个问题的检查，大约只要三十秒：

```bash
curl -sI https://app.example.com/ | grep -i '^x-powered-by'   # who is actually answering?
curl -s  https://app.example.com/ -o /dev/null -w '%{http_code}\n'  # what does a stranger get?
```

如果答案和你配置里说的对不上，那你就是在改错误的层。这值得在你为此花掉一整天之前就知道——更不用说在告诉任何人网关已上线之前了。

---

## 需要为你的业务做这件事吗？

如果你需要在内部工具前面加单点登录、把自托管的应用安全地暴露到公网，或者需要有人审计到底是哪条入口路径在真正服务你的域名——这正是我在做的工作。

- **WhatsApp：** [011-797 2969](https://wa.me/60127972969) —— 点击对话
- **Email：** [me@hoelee.com](mailto:me@hoelee.com?subject=Reverse%20proxy%20%26%20SSO%20enquiry)
- **网站：** [hoelee.com](https://www.hoelee.com)

我为马来西亚的中小企业搭建 authentik 单点登录、加固反向代理和 Cloudflare 隧道、自托管 Docker 环境——并把过程中学到的东西写下来。
````

### 🇬🇧 英文版原文 (English Markdown Primary)
````markdown
---
title: "The Forward-Auth Gate That Verified Perfectly — and Wasn't Live"
description: "My authentik forward-auth gate returned a clean 302 to the login page from the host itself, while the public URL still served the app. The hostname had two ingress layers, and I had only changed one."
pubDate: 2026-08-12
updatedDate: 2026-09-29
category: devops
blueprint: self-hosted-infrastructure
blueprintLayer: L2-edge-proxy-runtime
tags: ["authentik", "forward-auth", "cloudflare", "nginx", "docker", "self-hosting"]
ogImage: /og/authentik-forward-auth-gate-wasnt-live.jpg
banner: /banners/authentik-forward-auth-gate-wasnt-live.jpg
draft: false
---

I finished the gate, ran the check, and it passed.

```
HTTP/1.1 302 Found
location: https://app.example.com/outpost.goauthentik.io/start?rd=...
x-powered-by: authentik
```

An unauthenticated request was being pushed to my authentik login page. Branded, correct, working. I was about to move on — except that my own rule for anything user-facing is to hit it the way a user hits it, from outside, over the public internet. So I did.

```
GET https://app.example.com/          → 200
x-powered-by: Express
```

The app answered directly. No login page, no redirect, no gate. The exact same hostname, one request path gated and one wide open, depending on where the request entered my network.

That was the whole day: not the SSO setup, which worked, but the fact that **a hostname can be served by more than one ingress layer, and I had only gated one of them**.

## Why this matters beyond my one app

If you self-host anything behind a tunnel, a reverse proxy, or both, you have probably already made this assumption: *"the reverse proxy is what serves my domains."* It's an assumption that holds right up until it doesn't, and it fails in the most dangerous direction — everything you change looks correct, because the path you're inspecting is correct.

The failure mode is silent by construction. There is no error, no log line, no hint. Your configuration check passes because you're checking the configuration you edited. Meanwhile the real traffic — the traffic you were protecting — never touches it.

I've since found two other gates on the same host with the same latent problem, built the same way, waiting for someone to trust them.

## The setup: SSO for an app that has no SSO

The app was a self-hosted database UI on my NAS (Docker, published on a host port, sitting behind nginx and a Cloudflare Tunnel — the usual shape of the infrastructure I manage). I wanted a login wall in front of it so that "reachable from the internet" wouldn't mean "anyone who guesses the hostname sees the login form and can start guessing passwords".

The app's own documentation advertises SSO. I'll keep this post focused on the ingress trap and put the licensing story in its own write-up, but the short version is that the advertised SSO is a licensed feature, the open-source build ships the code path but not the permission to use it, and the community fork that used to fill the gap hasn't been touched in two years. Twenty minutes of research, two abandoned approaches, one conclusion: **the app would never do SSO for me, so the gate had to live in front of it.**

Which is a normal, well-trodden thing to do. authentik calls it a proxy provider, and the flow is:

1. **A login flow** for this app — its own title, its own background, so a user who lands on the login page knows which system they're being asked to authenticate for.
2. **A proxy provider** — the object that knows the app's external hostname (`https://app.example.com`) and its internal address (`http://app-container:8080`).
3. **An application** linking the two, attached to the embedded outpost. The outpost is the component that actually receives traffic, checks for a session, and proxies the request to the app's internal host after login.
4. **The ingress** — pointing that hostname at the outpost instead of at the app.

Steps 1–3 live in authentik and are configuration. Step 4 is the one that decides whether any of it is real, and it's the one I got wrong.

Two smaller things from steps 1–3, since they cost me time:

**A partial PATCH on a proxy provider is rejected.** Updating one field returns:

```json
{"internal_host": ["Internal host cannot be empty when forward auth is disabled."]}
```

The serializer re-validates the object, and a proxy provider without an internal host is invalid — so a one-field update fails as if you'd cleared a field you never touched. Send the identity fields together with the field you're changing:

```bash
curl -s -X PATCH "$AUTH/api/v3/providers/proxy/$PK/" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"mode":"proxy","external_host":"https://app.example.com",
       "internal_host":"http://app-container:8080","skip_path_regex":""}'
```

**authentik API tokens can expire mid-session.** After working for an hour, every call started returning `403 {"detail":"Token invalid/expired"}`. Note the status code: `403` means "we know who you are, and you may not do this" — the response body is the only thing that tells you the token died. If you cache a token in a script, re-read it from the database when calls start failing instead of assuming a permissions problem.

## The trap: I verified the layer I changed

Here is the check I ran, from the host itself:

```bash
curl -sk -o /dev/null -w '%{http_code} -> %{redirect_url}' \
  -H 'Host: app.example.com' https://localhost/
# 302 -> https://app.example.com/outpost.goauthentik.io/start?rd=...
```

This is a *good* check. It proves the nginx vhost serves the outpost, that the outpost recognises the hostname, that the provider is linked to the application, and that the login flow is alive. Everything I had configured was proven correct in one command.

It says nothing whatsoever about whether public traffic reaches that vhost.

Here's why. My hostnames are served by **two** independent ingress paths:

| Path | What serves it | Where it lands |
|---|---|---|
| Home IP → nginx (443) | the vhost I edited | the outpost ✅ |
| Cloudflare Tunnel (`cloudflared`) | a rule inside the tunnel's own config | the app's port, directly ❌ |

The tunnel's ingress is a small ordered list evaluated per hostname:

```
app.example.com     -> http://app-container:8080     ← straight to the app
*                   -> http_status:404
```

That rule was written years earlier, when the goal was simply "make this reachable". It never mentioned nginx, because it never needed to. So my beautiful vhost sat off to the side of the real request path, gating a route that public traffic doesn't take.

The two paths even answer on the same hostname, so nothing about the URL hints at the difference.

**What finally made it obvious was a response header.** The gated check returned `x-powered-by: authentik`. The public check returned `x-powered-by: Express`. Same URL, same second, two different servers answering. When you suspect you're looking at the wrong layer, read the headers — they name the software that answered, which is exactly the question you're asking.

To see your own tunnel's rules rather than guessing (Cloudflare's API, not the dashboard):

```bash
curl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/cfd_tunnel/$TUNNEL_ID/configurations" \
  -H "Authorization: Bearer $CF_TOKEN" \
  | python -c "import sys,json; [print(r.get('hostname','*'), '->', r.get('service','')) for r in json.load(sys.stdin)['result']['config']['ingress']]"
```

```
git.example.com   -> http://192.168.1.6:6880
app.example.com   -> http://app-container:8080
bot.example.com   -> http://bot-container:8080
*                 -> http_status:404
```

There it is, in one line, after a day of looking at nginx.

### The permission wall on the way to the fix

Editing that rule needs a token with **Account → Cloudflare Tunnel → Edit**. Mine didn't have it, and no zone-scoped token can substitute: tunnels are account-scoped objects, so a Zone-scoped token fails the *read* as well, with `1001 Not authorized`. If you only have zone-level tokens, open the dashboard instead: **Zero Trust → Networks → Tunnels → your tunnel → Public Hostnames** and change the rule's service to your outpost.

For me the correct target was the outpost on the same Docker network as `cloudflared`:

```
app.example.com  ->  http://authentik-server:9000
```

The outpost reads the `Host` header (which the tunnel preserves) to pick the application, then proxies to the internal host from the provider config. That's the whole change: one service value. Everything I'd built in authentik was already correct — it was simply never on the path.

## Optional SSO: gate the UI, leave the API alone

Once the gate is really in front of the traffic, the next problem is the API. A hostname-level gate can't tell a script from a browser, so token-only API calls get redirected to a login page — they don't fail with a JSON error, they get HTML.

If you want SSO for humans and no SSO for machines, authentik's proxy provider has a per-path exemption. The UI calls it **Unauthenticated Paths**; the API field is `skip_path_regex`:

```json
{"skip_path_regex": "^/api/.*"}
```

Verified result on my instance, same hostname, same session:

| Request | No session | Result |
|---|---|---|
| `/` and `/dashboard` | yes | `302` → authentik login |
| `/api/v1/health` | yes | `200` from the app |
| `/api/v2/tables/.../records` + API token | yes | `200` with real data |

Two caveats worth knowing before you rely on it. This exemption applies at the outpost, so unlike a path split at the CDN it works on **every** ingress path. And it is not per-user: the gate applies to the request, not the person. There is no "some users can skip SSO" — either a path is exempt for everyone, or everyone needs a session.

I used a simpler form of the same idea first: **move every machine caller off the public hostname before the gate exists.** My automation was calling the app through the public URL and being routed out to Cloudflare and back into my own network for no reason. Pointing it at the container name (`http://app-container:8080`) shortened the path, removed a failure mode, and meant the gate couldn't break it. If a login wall takes down your integration, the integration was depending on the door being unlocked.

## The honest part: this is not single sign-on

I have to be straight about what a forward-auth gate gives you, because "SSO" oversells it.

The outpost knows who you are. The app does not. Unless the app can consume an upstream identity — header-based auth, or native OIDC/SAML — it will show its own login form after the gate lets you through. So the user experience is: authentik login, then the app's login.

For a user who is already signed in to authentik, the first step is often silent. But it's still two systems, two session lifetimes, and two places a password can be wrong. In my case the app has no header auth and its own OIDC is a licensed feature, so the gate would have been an **extra** door bolted in front of the existing one — no reduction in logins at all.

That's a legitimate trade for some setups: keep the app's login for its own users, and stop the internet from reaching the login form. It's a poor trade if what you actually wanted was "my team types one password". Which meant the honest answer for me was to turn the whole thing back off.

## Reverting it properly (and the arithmetic that saved ten other apps)

Undoing the gate is where I nearly caused a much bigger outage than the one I was preventing.

The nginx change was one line in a large generated vhost file. The obvious revert is to substitute the port back:

```bash
# DON'T — this is the version that would have hurt
sed -i 's#proxy_pass http://localhost:10000;#proxy_pass http://localhost:10380;#' vhost.conf
```

Before running it, I counted how many server blocks point at that same port:

```bash
grep -c 'proxy_pass http://localhost:10000;' vhost.conf
# 11
```

Eleven. My app was one of them. **Ten other applications are gated through that outpost**, and a blanket substitution would have silently un-gated every one of them at once — including the ones doing real authentication work. The "fix" would have looked successful and removed authentication from most of my infrastructure.

What I did instead:

1. **Snapshot the current state first**, so the revert was itself reversible: `cp -a vhost.conf vhost.conf.bak-$(date +%Y%m%d-%H%M)-revert`.
2. **Find the exact line by diffing against the backup taken before the original change.** That diff showed one changed line, which is proof of exactly how much I had touched.
3. **Edit that line only**, then re-diff against the original backup and require the result to be identical before going further.
4. **Update the second file** that mirrors the same setting — the NAS's own reverse-proxy database, which regenerates the vhost on DSM events. I edited the single key and asserted the hostname before writing, never restoring the whole file.
5. `nginx -t` then reload.

And then a detail that cost me another twenty minutes of doubt:

```bash
nginx -s reload && sleep 2 && curl -sk -H 'Host: app.example.com' https://localhost/
# still 302 -> /outpost.goauthentik.io/start
```

The revert looked like it had failed. It hadn't. A request issued microseconds after the reload can still be answered by the old worker while it drains. The re-test a few seconds later returned `200 x-powered-by: Express` — direct to the app, gate gone.

**If a config change appears not to apply, re-test after a beat before you go looking for a second cause.** With a request that fast, "the config is wrong" and "the old process is still answering" are indistinguishable.

## What I'd do differently

1. **Enumerate every ingress path for a hostname before building anything on top of it.** My checklist is now four questions, asked in this order:
   - Is the hostname proxied by a CDN/tunnel? If yes, its rules are the real ones — check them first, not last.
   - What does the reverse proxy vhost say?
   - Does the platform mirror that vhost somewhere else that can regenerate it?
   - Is there another hop (VPS, second proxy, vendor tunnel) with its own copy of the routing table?
2. **Verify from outside, and verify which software answered.** `curl` from the host with a `Host:` header proves your config is *self-consistent*. Only a request from the public internet, plus `x-powered-by`/`server` headers, proves it's *in force*.
3. **Find out whether the app can consume an upstream identity before you build the gate.** If it can't, you're adding a login, not removing one — a decision to make deliberately, not to discover afterwards.
4. **Count before you substitute.** Any port or hostname you're replacing globally probably appears in more places than you think. `grep -c` costs a second and prevents an outage.
5. **Snapshot, diff, edit one line, re-diff.** The backup's only job is to answer "how much did I change?" — and a diff answers it exactly.

## The result

The gate worked. It was also invisible to every real user, and after working out what it actually delivered — a second login in front of an app that already had one — I turned it off again: one field cleared, one application unlinked, one line restored, verified by a request from outside my network.

The two checks that would have caught the whole thing on day one take about thirty seconds:

```bash
curl -sI https://app.example.com/ | grep -i '^x-powered-by'   # who is actually answering?
curl -s  https://app.example.com/ -o /dev/null -w '%{http_code}\n'  # what does a stranger get?
```

If the answer isn't what your configuration says it should be, you're editing the wrong layer. That's worth knowing before you spend the day on it — and definitely before you tell anyone the gate is live.

---

## Want this for your business?

If you need single sign-on in front of internal tools, a self-hosted app exposed safely to the internet, or someone to audit which ingress path is actually serving your domains, that's the work I do.

- **WhatsApp:** [011-797 2969](https://wa.me/60127972969) — tap to chat
- **Email:** [me@hoelee.com](mailto:me@hoelee.com?subject=Reverse%20proxy%20%26%20SSO%20enquiry)
- **Website:** [hoelee.com](https://www.hoelee.com)

I set up authentik single sign-on, hardened reverse proxies and Cloudflare tunnels, and self-hosted Docker stacks for small businesses in Malaysia — and I write up what I learn while doing it.
````

---

## 文章 07 / 11：Authentik 从 2025.8 升级到 2026.8：那些没人提醒你会坏掉的部分
- **Slug**: `authentik-major-upgrade-gotchas`
- **英文标题**: Authentik 2025.8 to 2026.8: The Breakable Parts Nobody Warns You About
- **分类**: `devops` | **发布日期**: 2026-09-09
- **线上地址**: https://blog.hoelee.com/posts/authentik-major-upgrade-gotchas/ (EN) | https://blog.hoelee.com/posts/zh/authentik-major-upgrade-gotchas/ (ZH)
- **本篇审核状态**: ⏳ 待审核 (Pending)

### ✍️ 顾问审阅意见区 (Review Feedback)
```text
【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：
```

### 🇨🇳 中文版原文 (Chinese Markdown Twin)
````markdown
---
title: "Authentik 从 2025.8 升级到 2026.8：那些没人提醒你会坏掉的部分"
description: "一年份的 authentik 大版本升级踩坑记录：存储挂载变更、RBAC 会话清理、可信代理，以及 authorization_flow 与 authentication_flow 搞混导致 SSO 彻底瘫痪。"
pubDate: 2026-09-09
category: devops
blueprint: self-hosted-infrastructure
blueprintLayer: L2-edge-proxy-runtime
tags: [authentik, sso, docker, portainer, self-hosting, oidc, upgrade]
ogImage: /og/zh-authentik-major-upgrade-gotchas.jpg
banner: /banners/authentik-major-upgrade-gotchas.jpg
---

我把 authentik 当作自托管体系前面的单点登录闸门——邮箱、仪表盘、Synology
应用、远程访问 outpost 都在它后面。很长一段时间它停在 **2025.8.3**，一年的
版本更新堆积成山。这篇文章记录我是怎么一口气把它升到 **2026.8.1** 的，以及
沿途每一步坏掉的地方——尤其是我犯的那个让单点登录彻底瘫痪、逼每个内部
应用都重新要密码的错误。

## 什么是 authentik，为什么要用它？

authentik 是一个开源的**身份提供商（IdP）**——反复回答同一个问题的那个组件：
*「这个人是谁，允许他进来吗？」* 你可以把它想成整栋应用大楼的前门。与其让每个
应用都跑自己那套用户名/密码页面（还有自己那一堆 bug、自己的「忘记密码」流程、
自己的二次验证），不如让所有应用都来问 authentik。

它支持真实部署里真正用得上的协议：

- **OIDC / OAuth2** —— 现代单点登录标准（「用 Google 登录」底层用的就是它）。
- **SAML** —— 企业标准（Grafana、Jira，或任何老旧的业务系统都能接）。
- **LDAP** —— 让只认 LDAP 的老客户端和 NAS 也能加入。
- **Proxy / forward-auth** —— 通过反向代理挡在应用前面，请求还没碰到应用就已经
  完成了鉴权。

实际的好处，按重要性排序：

1. **一个登录通行所有。** 登录一次，在每一个应用之间自由穿梭。用户不用再记一打
   密码（也不用再陪着一堆由此而来的客服工单）。
2. **一个地方收紧安全。** MFA、密码策略、会话限制、账户找回，都在 authentik 里
   配一次——而不是每个应用各自重新实现一遍。
3. **一个地方做审计。** 每一个登录、每一个应用、每一个用户，都在同一个日志里。
   出问题时，这就是「我们以为没事」和「我们有证据」之间的区别。
4. **自包含、可自托管。** 数据归你所有。没有按人头收费、随团队规模膨胀的 SaaS
   费，也没有锁定——它是 AGPL 许可，跑在 Docker 里。

对一个运营着二十几个服务（邮箱、仪表盘、NAS 应用、内部工具）的小企业或独立
开发者来说，authentik 就是「每个应用各有一个脆弱密码」和「一道加固过的前门」之间
的区别。

## 为什么要升级

2025.8.3 本身没坏。但它落后太久，后面的版本里已经累积了一堆 CVE 漏洞修复，
而且我正准备做按应用区分品牌。authentik 自己的策略是不允许跨大版本跳级——
它强制要求逐级走。所以计划是：

```
2025.8.3 → 2025.10 → 2025.12 → 2026.2 → 2026.5 → 2026.8
```

六跳，一次一步，每步之间跑一次迁移 + 一次健康检查。动手之前，有一个绝对
不能省的步骤：**备份数据库**。authentik 不支持降级。如果迁移只跑了一半，
你是从备份恢复，而不是回滚镜像 tag。

```bash
sudo docker exec authentik-postgres pg_dump -U authentik authentik > authentik-backup.sql
```

## 坑 1：Portainer 才是真相源，不是 compose 文件

我的第一反应是编辑磁盘上的 `docker-compose.yml` 然后 `up`。错的。这个 stack
是由 **Portainer**（stack 143）管理的，真正的 compose 和环境变量都保存在它
自己的存储里。磁盘上的 `.env` 早就过时了——它的 `PG_PASS` 跟 Portainer 实际
运行的对不上。

正确的更新路径是走 Portainer API，而不是文件系统：

1. 更新 compose 内容里的镜像 tag。
2. **先** `docker pull` 新镜像（这样 API 调用不会在拉取中途超时）。
3. `docker stop` + `docker rm` 正在运行的容器（否则固定的 `container_name`
   会在重新部署时冲突）。
4. 用新的 `StackFileContent` + `Env` 调 `PUT /api/stacks/143?endpointId=2`。

我撞上了容器名冲突错误、拉取超时错误，还有一个网络连接问题——重建后的
`authentik-server` 只接入了它两个网络里的一个，导致无法解析 `postgres-server`。
一旦你看懂了门道，每个问题都是五分钟的修复；但凑在一起，耗掉了我大半个晚上。

## 坑 2：存储挂载迁移了（2025.12）

在 2025.12 之前，品牌资源放在 `/media/` 下，通过 `/media/...` 提供服务。
2025.12 之后存储结构变了：文件迁移到 `/data/media` 结构，通过一个新的
`/files/media/public/<name>?token=...` URL（带 JWT 签名）提供服务。我的容器
还在挂载 `./media:/media`，所以我一跨过那个版本，每个 logo、favicon、背景图
全部 404。

修复方式就是官方文档里的迁移：

```bash
mkdir -p data && mv media data/media
```

……然后把挂载改成 `./data:/data`。新的文件后端还要求 `/data` 必须是一个真正的
挂载点才肯工作——我最开始的修复用的是符号链接，结果后端 `is_mount()` 检查
直接报错 `No file management backend configured`。

## 坑 3：RBAC 迁移留下一张被污染的表

2025.12 为了 RBAC 重构，删除了旧的 `authentik_core.User_groups` 模型。迁移
本身跑得很干净，但**旧会话**仍然引用着这个已删除的模型。结果：登录页一直抛
`LookupError: App 'authentik_core' doesn't have a 'User_groups' model`。

不是 `django_session`——那张表是空的。真正的元凶是 authentik 自己的
`authentik_core_session` 表。清空它（以及其他 session 表）强制所有人重新登录，
错误也随之消失：

```sql
TRUNCATE authentik_core_session;
```

有个副作用值得知道：这也会让你各应用持有的所有 OIDC refresh token 失效。它们
会把你弹去重新登录一次，然后恢复。这是一次性的麻烦，不是 bug。

## 坑 4：可信代理现在变成了显式选择（2026.8）

2026.8 收紧了默认的转发头处理。之前 authentik 信任所有内网段；现在它只信任
你显式列出的那些。在转发到 `localhost` 的 Synology 反向代理后面，意味着：

```yaml
environment:
  AUTHENTIK_LISTEN__TRUSTED_PROXY_CIDRS: 127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,::1/128
```

跳过这一步，代理头就会被拒绝，表现出来的认证失败症状，跟你实际上遇到的问题
长得几乎完全不像。

## 坑 5：那个搞坏 SSO 的——authorization_flow 和 authentication_flow 搞混

这是最贵的一个错误，而且如果你要做按应用区分品牌，特别容易搞混。

authentik 里的 provider 有**两个** flow 字段，含义完全不同：

```python
# "未登录用户访问关联应用时用的认证 flow"    ← 登录页
authentication_flow = models.ForeignKey(...)

# "授权这个 provider 时用的 flow"           ← OAuth 授权确认页
authorization_flow = models.ForeignKey(...)
```

- `authentication_flow` 是**登录页**。你要给每个应用定制标题和背景，应该把
  各自独立的 flow 放在这里。
- `authorization_flow` 是**已登录**用户的授权确认（consent）步骤。

我想要每个应用不同的背景图，于是给每个应用建了一个 flow，然后把 provider 的
`authorization_flow` 指向它。瞬间就坏了：一个已经认证的用户，每次打开另一个
应用，还是会被拖过完整的识别 → 密码 → MFA 流程。单点登录，实际上等于没了。

修复就是一条 UPDATE，把两个字段放回它们该在的位置：

```sql
UPDATE authentik_core_provider
SET authorization_flow_id = '1d85b1b1-...',   -- explicit-consent flow
    authentication_flow_id = 'aa##-per-app-flow'
WHERE ...;
```

而品牌本身——按应用区分的背景和标题——是放在 **flow 自身**的 `background` 和
`title` 字段上（2026.8 起可写），而不是放在 brand 的域名匹配上。

这个坑后面还藏着一个相关的坑。我创建那 18 个按应用区分的 flow 时，它们全都
落成 `designation=authentication`，而我的 `auth.hoelee.com` brand 的
`flow_authentication` 是 `NULL`。authentik 在一个 brand 没有显式认证 flow 时的
回退逻辑，是**按 slug 字母序**选第一个 authentication flow——碰巧就是
`auth-agent`，而不是 `default-authentication-flow`。于是根登录页开始显示我
agent 的背景图。把 brand 的 `flow_authentication` 设成真正的默认 flow 才修好。

还有一个跟 RBAC 坑叠加的问题：创建 flow 会设置它的 `background`，但**不会设置
它的 stages**——`stages` 在 flow 对象上是只读的。一个没有任何 stage 绑定的空
flow，正是迁移早期登录页无限重定向循环的罪魁祸首。stage 绑定是单独创建的：

```
POST /api/v3/flows/bindings/   # { target: "<flow pk>", stage: "<stage pk>", order: N }
```

## 我会怎么做得不一样

整场折腾归结为三个本可避免的模式：

1. **绝不要靠猜字段的语义**——我把 `authorization_flow` 当成了「登录 flow」，
   而真相源是模型定义，字段自己的 docstring 里就把区别写得明明白白。
2. **跟 API 表面保持距离**——每次服务器重启，API token 都中途过期，结果关键
   修复我直接用 `psql` 对着数据库做了。可靠，但应该在集群着火**之前**就脚本化，
   而不是着火的时候。
3. **每次重启只处理一个破坏性变更**——我试图一口气推理存储、RBAC、代理的
   全部变化。每一个如果单独隔离、独立验证，都会很简单。

## 结果

authentik 现在跑着 **2026.8.1**——最新、已打补丁、所有容器健康——18 个应用各自
在登录页显示自己的背景和标题，SSO 在每个子域都正常工作。我日常用的 17 个应用，
从「每次切换应用都要重新登录」回到了「登录一次，畅通无阻」。

值得记住的教训：认证基础设施的大版本升级，大概 10% 是「改镜像 tag」，90% 是
「数据模型、存储结构、代理规则在你脚下全都挪了位」。做好备份，一次走一个版本，
当某个东西表现得完全不合理时，伸手去抓下一份配置之前，先把字段名再念一遍。

---

## 想为你的企业搭一套单点登录？

如果你正在运营好几个内部应用——仪表盘、邮箱服务器、工单系统、文件服务器——而
你的团队还在一个个分开登录（或者到处重复用同一个密码），我搭设和维护的正是这类
基础设施。我会帮你部署 authentik、接到你现有的应用上、加上 MFA，确保「登录一次」
真的能跑通——最后还会给你一份交接文档，让你永远不会被锁死。

联系我：[me@hoelee.com](mailto:me@hoelee.com)，或 WhatsApp
[+60 12-797 2969](https://wa.me/60127972969)，看看我在
[hoelee.com](https://hoelee.com) 做的东西。
````

### 🇬🇧 英文版原文 (English Markdown Primary)
````markdown
---
title: "Authentik 2025.8 to 2026.8: The Breakable Parts Nobody Warns You About"
description: "A year's worth of authentik major-version upgrade pain: storage mount changes, RBAC session cleanup, trusted proxies, and the authorization_flow vs authentication_flow mix-up that broke SSO."
pubDate: 2026-09-09
category: devops
blueprint: self-hosted-infrastructure
blueprintLayer: L2-edge-proxy-runtime
tags: [authentik, sso, docker, portainer, self-hosting, oidc, upgrade]
ogImage: /og/authentik-major-upgrade-gotchas.jpg
banner: /banners/authentik-major-upgrade-gotchas.jpg
---

I run authentik as the single sign-on gate in front of my self-hosted stack —
email, dashboard, Synology apps, a remote-access outpost. For a long while it
sat on **2025.8.3**, and a year of releases piled up. This is the story of
bringing it all the way to **2026.8.1** in one sitting, and everything that
broke along the way — especially the one mistake that took Single Sign-On
completely offline and made every internal app ask for a password again.

## What is authentik, and why run it?

authentik is an open-source **identity provider (IdP)** — the thing that answers
one question over and over: *"who is this person, and are they allowed in?"*
Think of it as the front door to a whole building of apps. Instead of every
app running its own username/password screen (and its own list of bugs, its own
"forgot password" flow, its own 2FA), you make all of them ask authentik instead.

It speaks the protocols that matter for real deployments:

- **OIDC / OAuth2** — the modern single sign-on standard (what "Sign in with
  Google" uses under the hood).
- **SAML** — the enterprise standard (for tools like Grafana, Jira, or any
  legacy line-of-business app).
- **LDAP** — so older clients and NAS boxes that only know LDAP can join too.
- **Proxy / forward-auth** — it sits in front of an app via a reverse proxy and
  gatekeeps access before a request ever reaches the app.

The practical benefits, in order of how much they actually matter:

1. **One login for everything.** Log in once, move freely between every app.
   Users stop juggling a dozen passwords (and the support tickets that go with
   them).
2. **One place to lock things down.** MFA, password policy, session limits, and
   account recovery live in authentik once — not re-implemented per app.
3. **One place to audit.** Every login, on every app, for every user, in one
   log. That's the difference between "we think nothing happened" and "we can
   prove it" when a question comes up.
4. **Self-contained and self-hosted.** You own the data. No per-user SaaS fees
   that scale with your headcount, no lock-in — it's AGPL, runs in Docker.

For a small business or a solo operator running twenty-something services
(email, dashboards, NAS apps, internal tools), authentik is the difference
between "every app has its own flimsy password" and "one hardened front door."

## Why upgrade at all

Version 2025.8.3 wasn't broken. But it had fallen far enough behind that a
stack of CVEs had landed in the releases after it, and I was getting ready to
do per-application branding. authentik's own policy is that you can't jump
major versions — it enforces a stepwise path. So the plan was:

```
2025.8.3 → 2025.10 → 2025.12 → 2026.2 → 2026.5 → 2026.8
```

Six hops, one at a time, with a migration and a health-check between each.
Before touching anything, the one non-negotiable step: **back up the database**.
authentik doesn't support downgrades. If a migration half-runs, you're restoring
from dump, not rolling back an image tag.

```bash
sudo docker exec authentik-postgres pg_dump -U authentik authentik > authentik-backup.sql
```

## Pitfall 1: Portainer is the source of truth, not the compose file

My first instinct was to edit the `docker-compose.yml` on disk and `up` it.
Wrong. The stack is managed by **Portainer** (stack 143), which keeps the real
compose and the real environment variables in its own store. The `.env` on disk
was stale — its `PG_PASS` didn't match what Portainer actually ran.

The correct update path is via the Portainer API, not the filesystem:

1. Update the image tags in the compose content.
2. `docker pull` the new images *first* (so the API call doesn't time out mid-pull).
3. `docker stop` + `docker rm` the running containers (fixed `container_name`
   will otherwise collide on redeploy).
4. `PUT /api/stacks/143?endpointId=2` with the new `StackFileContent` + `Env`.

I hit the colliding-container error, the pull-timeout error, and a network-attach
problem where the rebuilt `authentik-server` only joined one of its two networks
and couldn't resolve `postgres-server`. Each one is a five-minute fix once you
know what you're looking at, but together they ate the better part of the evening.

## Pitfall 2: the storage mount moved (2025.12)

Up to 2025.12, brand assets lived under `/media/` and were served at `/media/...`.
After 2025.12, the storage layout changed: files moved to a `/data/media`
structure served under a new `/files/media/public/<name>?token=...` URL with a
JWT signature. My containers were still mounting `./media:/media`, so every
logo, favicon, and background image 404'd the moment I crossed that version.

The fix is the documented migration:

```bash
mkdir -p data && mv media data/media
```

…and change the mounts to `./data:/data`. The new file backend also refuses to
work unless `/data` is an actual mount point — the earlier version of my fix
used a symlink, which the backend's `is_mount()` check rejected with
`No file management backend configured`.

## Pitfall 3: RBAC migration leaves a poisoned session table

2025.12 removed the old `authentik_core.User_groups` model in favor of the RBAC
rework. The migration ran clean, but **old sessions** still held references to
the deleted model. Result: the login page kept throwing
`LookupError: App 'authentik_core' doesn't have a 'User_groups' model`.

Not `django_session` — that was empty. The real culprit was authentik's own
`authentik_core_session` table. Clearing it (and the other session tables)
forced everyone to log in again and cleared the error:

```sql
TRUNCATE authentik_core_session;
```

One side effect worth knowing: this also invalidates every OIDC refresh token
your apps were holding. They'll bounce the user to a fresh login once, then
recover. It's a one-time annoyance, not a bug.

## Pitfall 4: trusted proxies are now opt-in (2026.8)

2026.8 tightened the default forwarded-header handling. Previously authentik
trusted all the private ranges; now it only trusts what you list explicitly.
Behind a Synology reverse proxy forwarding to `localhost`, that means:

```yaml
environment:
  AUTHENTIK_LISTEN__TRUSTED_PROXY_CIDRS: 127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,::1/128
```

Skip this and the proxy headers get rejected, which surfaces as auth failures
that look like almost anything except what they actually are.

## Pitfall 5: the one that broke SSO — authorization_flow vs authentication_flow

This was the expensive mistake, and it's the kind of thing that's easy to get
wrong if you're doing per-application branding.

A provider in authentik has **two** flow fields, and they mean different things:

```python
# "Flow used for authentication when the associated application is
#  accessed by an un-authenticated user."      ← the LOGIN page
authentication_flow = models.ForeignKey(...)

# "Flow used when authorizing this provider."  ← the OAuth consent page
authorization_flow = models.ForeignKey(...)
```

- `authentication_flow` is the **login page**. This is where you put a
  per-application flow to customise the title and background.
- `authorization_flow` is the **consent/authorize step** for a user who is
  *already* logged in.

I wanted different background images per app, so I created one flow per
application and pointed the provider's `authorization_flow` at it. Instant
breakage: a user who was already authenticated still got walked through the
full identification → password → MFA flow every time they opened another app.
Single Sign-On was, effectively, gone.

The fix was a single UPDATE to put the two fields back where they belong:

```sql
UPDATE authentik_core_provider
SET authorization_flow_id = '1d85b1b1-...',   -- explicit-consent flow
    authentication_flow_id = 'aa##-per-app-flow'
WHERE ...;
```

And the branding itself — the per-app background and title — goes on the
**flow's own** `background` and `title` fields (writable since 2026.8), not on
the brand's domain match.

There was a second, related gotcha hiding behind this one. When I created the
18 per-app flows, they all landed with `designation=authentication`, and my
`auth.hoelee.com` brand had `flow_authentication` set to `NULL`. authentik's
fallback when a brand has no explicit auth flow is to pick the first
authentication flow **by slug, alphabetically** — which happened to be
`auth-agent`, not `default-authentication-flow`. So the root login page started
showing my agent's background image. Setting the brand's `flow_authentication`
to the real default flow fixed it.

And one more that compounds with the RBAC pitfall: creating a flow sets its
`background` but **not its stages** — `stages` is read-only on the flow object.
An empty flow with no stage bindings is exactly what produced an infinite
redirect loop on the login page earlier in the migration. Stage bindings are
created separately:

```
POST /api/v3/flows/bindings/   # { target: "<flow pk>", stage: "<stage pk>", order: N }
```

## What I'd do differently

The whole ordeal came down to three preventable patterns:

1. **Never guess at a field's semantics** — I treated `authorization_flow` as
   "the login flow" when the source-of-truth is the model definition, which
   spells out the difference in the field's own docstring.
2. **Keep the API surface at arm's length** — API tokens kept expiring mid-run
   with every server restart, so I ended up doing the critical fixes directly
   against the database with `psql`. Reliable, but worth scripting *before*
   the cluster is on fire, not during.
3. **One breaking change per restart** — I tried to reason about storage,
   RBAC, and proxy changes all in one go. Each would have been trivial if
   isolated and verified independently.

## The result

authentik now runs **2026.8.1** — current, patched, all containers healthy —
with 18 applications each showing their own background and title on the login
page, and SSO working across every subdomain. Seventeen of the applications I
use daily went from "asks me to log in again every time I switch apps" back to
"log in once, move freely."

The lesson worth carrying: a major-version upgrade on auth infrastructure is
roughly 10% "change the image tag" and 90% "the data model, storage layout,
and proxy rules all shifted underneath you." Back up, go one version at a time,
and when something behaves in a way that makes no sense, read the field name
again before you reach for another config.

---

## Want single sign-on for your business?

If you're running several internal apps — a dashboard, an email server, a help
desk, a file server — and your team is still logging into each one separately
(or reusing the same password everywhere), I set up and maintain exactly this
kind of infrastructure. I'll deploy authentik, wire it to your existing apps,
add MFA, and make sure "log in once" actually works — then leave you with a
handover so you're never locked in.

Reach me at [me@hoelee.com](mailto:me@hoelee.com) or WhatsApp
[+60 12-797 2969](https://wa.me/60127972969), or see what I do at
[hoelee.com](https://hoelee.com).
````

---

## 文章 08 / 11：不用界面管理 CyberPanel：逆向一个没有文档的 v2 API
- **Slug**: `automating-cyberpanel-without-the-ui`
- **英文标题**: Automating CyberPanel Without the UI: Reverse-Engineering an Undocumented v2 API
- **分类**: `devops` | **发布日期**: 2026-09-09
- **线上地址**: https://blog.hoelee.com/posts/automating-cyberpanel-without-the-ui/ (EN) | https://blog.hoelee.com/posts/zh/automating-cyberpanel-without-the-ui/ (ZH)
- **本篇审核状态**: ⏳ 待审核 (Pending)

### ✍️ 顾问审阅意见区 (Review Feedback)
```text
【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：
```

### 🇨🇳 中文版原文 (Chinese Markdown Twin)
````markdown
---
title: "不用界面管理 CyberPanel：逆向一个没有文档的 v2 API"
description: "CyberPanel v2 砍掉了有文档的 JSON API，只留下 Angular 界面自己调用的 session + CSRF 端点。这篇文章讲清确切的认证流程，以及我怎么从页面背后挖出真正的函数。"
pubDate: 2026-09-09
category: devops
tags: [cyberpanel, api, reverse-engineering, automation, django, self-hosting, curl]
ogImage: /og/zh-automating-cyberpanel-without-the-ui.jpg
banner: /banners/automating-cyberpanel-without-the-ui.jpg
---

我在家庭实验里的一台 Ubuntu 虚拟机里跑着 CyberPanel 2.4.4.1，用来托管几个小站，
入口是 `https://panel.hoelee.com`，反向代理到内网的一台机器。最近我想不打开浏览器、
直接用脚本做两件事：列出我的站点、删掉其中一个。这篇文章讲的就是我怎么发现"官方
API"根本已经不存在了，然后从 Angular 界面偷偷调用的那些请求里把真正的 API 逆向出来。

## 问题：有文档的 API 已经没了

网上每一个搜索结果、Apiary 文档、还有 Knowledge Base，都指向同一套东西——一个 JSON
API，把 `adminUser` 和 `adminPass` 通过 `POST` 发给 `/api/verifyConnection`：

```bash
# 旧文档教你这么干。在 CyberPanel v2 上这根本行不通。
curl -k -X POST https://panel.hoelee.com:8090/api/verifyConnection \
  -d '{"adminUser":"admin","adminPass":"...","serverUserName":"..."}'
```

在 CyberPanel v2（2.4.x）上，这只会返回一堆没用的东西。`/api/` 前缀是遗留的旧接口，早
就被砍掉了。已经没有 `adminUser`/`adminPass` 那套交换了。官方文档就是过时了——这是第
一个陷阱：你很容易花上一大把时间去信一套描述的并不是你正在跑的那个版本的文档。

## 我试了什么，以及为什么失败

**第一次尝试——信文档。** 按 Apiary 说的，带上凭据 `POST /api/verifyConnection`。结果：
404。这个路由根本不存在。

**第二次尝试——硬猜路径。** 我试了 `/api/`、带一个 `email` 字段的 `/verifyLogin`、
还有几个瞎猜的裸 `GET`。拿回一个我一开始老是读错的响应：

```
"This request need session."
```

这句话其实是**好消息**——它说明 API **是**挂着的、能连上的，只是没有有效会话就拒绝
跟你说话。端点没缺，缺的是认证方式不一样。

**第三次尝试——CSRF 这堵墙。** 我终于把正确的字段发给 `/verifyLogin`，结果撞上一个
硬邦邦的 **403 Forbidden**。这是卡住绝大多数人的地方：发往 Django 后端面板的 POST 都
受 CSRF token 保护，一个不带 token 的裸 JSON POST 会在你的凭据都没被看上一眼之前就被
丢掉了。

突破口在于：别把面板当成"一个带 API 的东西"，而是当成**一个带 Angular 前端的 Django
应用**——然后只要盯着前端到底发了什么就行。

## 修复：真正的认证流程

整件事就三步，全部用普通的 `curl`（加上 `-k`，因为面板用的是自签证书）：

### 1. 从登录页拿到 CSRF token

Django 在第一次 GET 时就会设一个 `csrftoken` cookie。这个 cookie 的值就是你要在每次写
操作时用头部回显回去的 token：

```bash
curl -sk -c /tmp/cp_cookies.txt "https://192.168.1.124:8090/" -o /dev/null
CSRF=$(grep csrftoken /tmp/cp_cookies.txt | awk '{print $7}')
```

### 2. 登录，拿到会话 cookie

`POST /verifyLogin`，带上 JSON body，并在 `X-CSRFToken` 头部带上 CSRF token：

```bash
curl -sk -b /tmp/cp_cookies.txt -c /tmp/cp_cookies.txt \
  -X POST "https://192.168.1.124:8090/verifyLogin" \
  -H "Content-Type: application/json" \
  -H "X-CSRFToken: $CSRF" \
  -H "Referer: https://192.168.1.124:8090/" \
  -d '{"username":"admin","password":"...","languageSelection":"EN","twofa":""}'
# => {"userID": 1, "loginStatus": 1, "error_message": "None"}
```

`loginStatus: 1` 说明会话 cookie 现在有效了。`Referer` 头部比你想的还重要——有
些 view 会去检查它。

### 3. 调用数据端点

所谓"API"，其实就是 Angular 界面调用的那几个 POST 端点。`/<module>/<function>`，同
一个 cookie jar，同一个 `X-CSRFToken` + `Referer`：

```bash
curl -sk -b /tmp/cp_cookies.txt \
  -X POST "https://192.168.1.124:8090/websites/fetchWebsitesList" \
  -H "Content-Type: application/json" \
  -H "X-CSRFToken: $CSRF" \
  -H "Referer: https://192.168.1.124:8090/" \
  -d '{"page":1,"recordsToShow":50}'
```

这把我所有站点都吐出来了，带 SSL 状态、磁盘占用、PHP 版本，还有每个站离证书过期还剩
几天。同样的套路能删站：

```bash
curl -sk -b /tmp/cp_cookies.txt \
  -X POST "https://192.168.1.124:8090/websites/submitWebsiteDeletion" \
  -H "Content-Type: application/json" \
  -H "X-CSRFToken: $CSRF" \
  -H "Referer: https://192.168.1.124:8090/" \
  -d '{"websiteName":"blog.hoelee.com"}'
```

## 最耗时间的两个坑

**1. `/api/...` 那个 404 从来不是网络问题——是时代搞错了。** `/api/` 前缀属于旧 API。
现代 CyberPanel 用的是根相对路径 `/<module>/<function>`，没有 `/api/` 这一段。一旦我不
再去找"API"，而是去找 UI 自己的路由，一切就都对上了。

**2. 列表响应是双重编码的 JSON。** `fetchWebsitesList` 返回的东西长这样：

```json
{ "data": "[{\"domain\":\"...\",\"ssl\":{\"days\":17}, ...}]" }
```

`data` 这个 key 是一个 **JSON 编码的字符串**，不是一个数组。你要是直接把它喂给 `jq`
再 `.[]`，拿不到任何有意义的东西。得先剥掉一层：

```bash
curl ... | jq -r .data | jq      # 先把字符串解出来，再解析一次
```

## 怎么找到其它端点

函数名不是随便起的——它们直接对应 CyberPanel 源码里的 Django view 函数（GitHub 上
`usmannasir/cyberpanel`，`stable` 分支）。相关文件是 `websiteFunctions/views.py`、
`mailFunctions/views.py`、`manageSSL/views.py` 等等。所以流程是：

1. 在 UI 里找到那个页面（比如"创建站点"）。
2. 打开浏览器 network 面板，看它 POST 到哪个 URL——或者去对应的 `views.py` 里 grep 那
   个 view 名。
3. 用你的会话 + CSRF 头部，调用同一个 `/<module>/<function>` 路径。

有个很有用的区分：`GET /<module>/<page>` 通常返回**那个 UI 页面的 HTML**，而**数据**
来自对一个姊妹函数的 POST。如果你 GET 一个页面拿回来的是 markup，那你还没找到真正的
端点——继续找那个 AJAX 调用。

## 我会怎么做不一样

1. **从前端而不是文档入手。** 盯着 Angular 应用的网络请求，本来能帮我省掉"盲信过时
   Apiary 文档"这整段弯路。对于它自己的 API 来说，UI 永远是真话。
2. **把 "This request need session." 当信标，不是错误。** 前几次我把它读成"端点错了"，
   其实它是"端点对了，会话状态不对"。
3. **把登录一次性脚本化成一个可复用的 helper。** 会话 cookie 会在 VM 重启和过期时死掉，
   所以每一次零散的 `curl` 都得从头来。一个能登录、抓 cookie、遇到
   `"This request need session."` 就重新登录的小 wrapper，能把整场操作从"摸索"变成
   "可复现"。

## 结果

现在我能完全从 shell 里列出、创建、删除我 CyberPanel 面板上的站点——不需要浏览器、不
需要 GUI——而且整个带认证的 API 面都敞开来供脚本化了（SSL、邮件、DNS、cron）。我是靠
丢掉文档、观察真实请求、再把这些请求映射回它们的 Django view 来做到的，而
`cyberpanel` 模块和会话流程现在也进了我自己的自动化工具箱。

更大的教训是："没有文档"很少意味着"不可能"。当一个工具暴露了 web UI，那个 UI 就是它
API 的一份活的、精确的参考——你只需要去看它到底发了什么。

---

## 想把你的服务器管理脚本化吗？

如果你还在点托管面板——更糟的是，在好几台服务器上重复做同样的手动步骤——我专门自动化
这类事情：把重复的后台操作变成一份测试过的脚本或一个小的内部工具，接到你现有的技术栈
上，并附上交接说明，让你永远不会被锁死。不管是 CyberPanel、cPanel、Docker，还是定制
的后台，凡是有 web UI 的，几乎都能脱离 UI 去驱动它。

联系我 [me@hoelee.com](mailto:me@hoelee.com?subject=Scripting%20my%20server%20admin) 或
WhatsApp [+60 12-797 2969](https://wa.me/60127972969)，或者到
[hoelee.com](https://hoelee.com) 看看我做什么。
````

### 🇬🇧 英文版原文 (English Markdown Primary)
````markdown
---
title: "Automating CyberPanel Without the UI: Reverse-Engineering an Undocumented v2 API"
description: "CyberPanel v2 killed the documented JSON API and left only the Angular UI's own session+CSRF endpoints. Here's the exact auth flow and how I found the real functions behind the pages."
pubDate: 2026-09-09
category: devops
tags: [cyberpanel, api, reverse-engineering, automation, django, self-hosting, curl]
ogImage: /og/automating-cyberpanel-without-the-ui.jpg
banner: /banners/automating-cyberpanel-without-the-ui.jpg
---

I run CyberPanel 2.4.4.1 inside an Ubuntu VM on my self-managed infrastructure to host a handful
of small sites. It sits at `https://panel.hoelee.com`, reverse-proxied to an
internal box. Recently I wanted to script two things against it — list my
sites and delete one — without opening the browser. This is the story of
finding that the "official" API doesn't exist anymore, and reverse-engineering
the real one from what the Angular UI quietly calls under the hood.

## The problem: the documented API is gone

Every search result, the Apiary docs, and the Knowledge Base point you at the
same thing — a JSON API where you `POST` your `adminUser` and `adminPass` to
`/api/verifyConnection`:

```bash
# What the old docs tell you to do. This does NOT work on CyberPanel v2.
curl -k -X POST https://panel.hoelee.com:8090/api/verifyConnection \
  -d '{"adminUser":"admin","adminPass":"...","serverUserName":"..."}'
```

On CyberPanel v2 (2.4.x), this returns nothing useful. The `/api/` prefix is
the legacy surface — it was dropped. There's no `adminUser`/`adminPass`
exchange anymore. The official documentation is simply out of date, which is
the first trap: you can spend a long time trusting docs that describe a
version you're not running.

## What I tried, and why it failed

**Attempt 1 — trust the docs.** `POST /api/verifyConnection` with credentials,
exactly as Apiary says. Result: 404. The route doesn't exist.

**Attempt 2 — hammer the obvious paths.** I tried `/api/`, `/verifyLogin` with
an `email` field, a bare `GET` on a few guesses. I got back a response I kept
misreading:

```
"This request need session."
```

That message is actually *good news* — it means the API **is** mounted and
reachable, it just refuses to talk without a valid session. The endpoint isn't
missing; the auth is different.

**Attempt 3 — the CSRF wall.** I finally sent the right fields to
`/verifyLogin` and hit a hard **403 Forbidden**. That's the part that stalls
most people: POSTs to a Django-backed panel are protected by a CSRF token, and
a bare JSON POST with no token gets dropped before your credentials are even
looked at.

The breakthrough was treating the panel not as "a thing with an API" but as
**a Django app with an Angular front-end** — and then just watching what the
front-end sends.

## The fix: the real auth flow

The whole thing is three steps, all with plain `curl` (and `-k` since the
panel uses a self-signed cert):

### 1. Grab the CSRF token from the login page

Django sets a `csrftoken` cookie on the very first GET. That cookie's value is
the token you echo back in a header on every write:

```bash
curl -sk -c /tmp/cp_cookies.txt "https://192.168.1.124:8090/" -o /dev/null
CSRF=$(grep csrftoken /tmp/cp_cookies.txt | awk '{print $7}')
```

### 2. Log in to get a session cookie

`POST /verifyLogin` with a JSON body and the CSRF token in an `X-CSRFToken`
header:

```bash
curl -sk -b /tmp/cp_cookies.txt -c /tmp/cp_cookies.txt \
  -X POST "https://192.168.1.124:8090/verifyLogin" \
  -H "Content-Type: application/json" \
  -H "X-CSRFToken: $CSRF" \
  -H "Referer: https://192.168.1.124:8090/" \
  -d '{"username":"admin","password":"...","languageSelection":"EN","twofa":""}'
# => {"userID": 1, "loginStatus": 1, "error_message": "None"}
```

`loginStatus: 1` means the session cookie is now valid. The `Referer` header
matters more than you'd expect — some of these views check it.

### 3. Call the data endpoints

The "API" is just the same POST endpoints the Angular UI calls.
`/<module>/<function>`, same cookie jar, same `X-CSRFToken` + `Referer`:

```bash
curl -sk -b /tmp/cp_cookies.txt \
  -X POST "https://192.168.1.124:8090/websites/fetchWebsitesList" \
  -H "Content-Type: application/json" \
  -H "X-CSRFToken: $CSRF" \
  -H "Referer: https://192.168.1.124:8090/" \
  -d '{"page":1,"recordsToShow":50}'
```

That returned every one of my sites, with SSL status, disk usage, PHP version,
and per-site days-until-cert-expiry. The same pattern deletes a site:

```bash
curl -sk -b /tmp/cp_cookies.txt \
  -X POST "https://192.168.1.124:8090/websites/submitWebsiteDeletion" \
  -H "Content-Type: application/json" \
  -H "X-CSRFToken: $CSRF" \
  -H "Referer: https://192.168.1.124:8090/" \
  -d '{"websiteName":"blog.hoelee.com"}'
```

## The two gotchas that ate most of the time

**1. The 404 on `/api/...` was never a network problem — it was the wrong era.**
The `/api/` prefix belongs to the old API. Modern CyberPanel uses
root-relative `/<module>/<function>` with no `/api/` segment. Once I stopped
looking for an "API" and started looking for the UI's own routes, everything
snapped into place.

**2. List responses are double-encoded JSON.** `fetchWebsitesList` returns
something shaped like:

```json
{ "data": "[{\"domain\":\"...\",\"ssl\":{\"days\":17}, ...}]" }
```

The `data` key is a **JSON-encoded string**, not an array. If you pipe it
straight into `jq` and try `.[]`, you get nothing sensible. You have to unwrap
one layer first:

```bash
curl ... | jq -r .data | jq      # unwrap the string, then parse again
```

## How to find the other endpoints

The function names aren't random — they map directly to Django view functions
in the CyberPanel source (`usmannasir/cyberpanel` on GitHub, `stable` branch).
The relevant files are `websiteFunctions/views.py`, `mailFunctions/views.py`,
`manageSSL/views.py`, and so on. So the workflow is:

1. Find the page in the UI (say, "Create Website").
2. Open the browser's network tab and watch what URL it POSTs to — or grep the
   matching `views.py` for the view name.
3. Call the same `/<module>/<function>` path with your session + CSRF headers.

A useful distinction: a `GET /<module>/<page>` usually returns the **HTML of
the UI page**, while the **data** comes from a POST to a sibling function. If
you GET a page and get markup back, you haven't found the real endpoint yet —
keep looking for the AJAX call.

## What I'd do differently

1. **Start from the front-end, not the docs.** Watching the Angular app's
   network requests would have saved me the whole "trust the obsolete Apiary
   docs" detour. The UI is always the ground truth for its own API.
2. **Treat "This request need session." as a beacon, not an error.** The first
   few times I read it as "wrong endpoint" when it was really "right endpoint,
   wrong session state."
3. **Script the login once into a reusable helper.** The session cookie dies
   on VM reboot and on expiry, so every one-off `curl` started from scratch.
   A tiny wrapper that logs in, captures the cookie, and re-logins on
   `"This request need session."` would have made the whole session repeatable
   instead of exploratory.

## The result

I can now list, create, and delete sites on my CyberPanel panel entirely from
the shell — no browser, no GUI — and the whole authenticated API surface is
open for scripting (SSL, mail, DNS, cron). I found this by scrapping the docs,
watching the real requests, and mapping them back to their Django views, and
the `cyberpanel` module + session flow now lives in my own automation toolkit.

The broader lesson: "it's not documented" rarely means "it's not possible."
When a tool exposes a web UI, that UI is a living, exact reference for the API
— you just have to watch what it actually sends.

---

## Want your server administration scripted?

If you're clicking through a hosting panel — or worse, doing the same manual
steps across several servers — I automate exactly this kind of thing: turn
repetitive admin into a tested script or a small internal tool, wired to your
existing stack, with a handover so you're never locked in. Whether it's
CyberPanel, cPanel, Docker, or a bespoke dashboard, if it has a web UI, it can
almost certainly be driven without one.

Reach me at [me@hoelee.com](mailto:me@hoelee.com?subject=Scripting%20my%20server%20admin) or WhatsApp
[+60 12-797 2969](https://wa.me/60127972969), or see what I do at
[hoelee.com](https://hoelee.com).
````

---

## 文章 09 / 11：Docker 报告退出码 0，内核却早已执行 OOM-Kill
- **Slug**: `docker-reported-exit-0-the-oom-killer-disagreed`
- **英文标题**: Docker Reported Exit 0. The Kernel Had Already OOM-Killed It.
- **分类**: `devops` | **发布日期**: 2026-10-16
- **线上地址**: https://blog.hoelee.com/posts/docker-reported-exit-0-the-oom-killer-disagreed/ (EN) | https://blog.hoelee.com/posts/zh/docker-reported-exit-0-the-oom-killer-disagreed/ (ZH)
- **本篇审核状态**: ⏳ 待审核 (Pending)

### ✍️ 顾问审阅意见区 (Review Feedback)
```text
【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：
```

### 🇨🇳 中文版原文 (Chinese Markdown Twin)
````markdown
---
title: "Docker 报告退出码 0，内核却早已执行 OOM-Kill"
description: "NAS 上的一个容器反复异常退出，Docker 却上报 exit 0 与 OOMKilled=false。真实的证据在 dmesg 里：六次 cgroup OOM-kill。"
pubDate: 2026-10-16
category: devops
blueprint: self-hosted-infrastructure
blueprintLayer: L2-edge-proxy-runtime
tags: ["docker", "oom", "traefik", "synology", "crawlers"]
ogImage: /og/zh-docker-reported-exit-0-the-oom-killer-disagreed.jpg
banner: /banners/docker-reported-exit-0-the-oom-killer-disagreed.jpg
draft: false
summary:
  audience: "SREs, DevOps Leads & 系统工程师"
  painPoint: "公网边缘反向代理容器（Traefik）4天内无预警重启7次，但 Docker daemon 谎报 ExitCode=0 且 OOMKilled=false，常规监控完全失明。"
  impact:
    - metric: "精准定位真凶"
      detail: "穿透 Docker 状态假象，通过内核 dmesg 与 cgroup 内存水位线抓出 25,976 个代理爬虫 IP 引发的 512MB 触顶"
    - metric: "0 停机热修复"
      detail: "单容器精准扩容至 2GB 并锁死 Swap（防磁盘阵列雪崩），18分钟峰值回落至 210MB，连续运行 0 崩溃"
    - metric: "防自锁诊断规范"
      detail: "沉淀边缘网关高并发防爬虫压测与避免自身 IP 被 WAF 误封（403 假象）的排障 SOP"
  stack: ["Docker", "Linux Kernel (dmesg/cgroups)", "Traefik", "Synology DSM", "Security Filtering"]
  jumpLinks:
    business: "#the-result"
    architecture: "#the-kernel-keeps-its-own-record"
    result: "#what-i-would-do-differently"
dynamicCta: true
---

## 一条我以为又是配置写错的告警

我的 NAS 发来那条所有 Docker 用户都学会一眼扫过的通知：**容器意外停止**。出事的是我自托管 wiki 前面的那个边缘代理，它已经连着跑了两周。那天早上没人部署过东西，compose 文件也没被动过，所以当时只有两个故事可选：「它崩了」，或者「有东西把它重启了」。

两个都错。而真正值得写下来的是：**光看 Docker 自己的报告，我根本分不出来。**

## Docker 的报告说「什么都没发生」

```text
State:
  Status:      running
  Running:     true
  ExitCode:    0            <- 干净退出
  OOMKilled:   false        <- 不是被内核杀的
  FinishedAt:  2026-10-15T19:39:33.63057879Z
  StartedAt:   2026-10-15T19:39:38.769326672Z   <- 5 秒后重启
RestartCount: 6
RestartPolicy: always
```

照字面读，结论就是：进程干净退出了六次，每次都由重启策略续了命。一个长期运行的代理不会自己以状态 0 退出——而「干净退出」重复六次本身是矛盾，不是答案。`ExitCode` 和 `OOMKilled` 恰好是我在这里最信任的两个字段。在这台机器上，两个都是错的。

## 内核自己留了另一份记录 <a id="the-kernel-keeps-its-own-record"></a>

在 Linux 上，cgroup 的内存击杀由内核记录，哪怕容器运行时的账本没有把它带过来：

```text
[1737797.044856] traefik invoked oom-killer: gfp_mask=0x2400040, order=0, oom_score_adj=0
[1737797.123736] Memory cgroup out of memory: Kill process 615 (traefik) score 970 or sacrifice child
[1737797.132800] Killed process 615 (traefik) total-vm:2525800kB, anon-rss:478480kB, file-rss:0kB
```

开机秒数 `1737797` 对应 19:39:33 UTC——和容器的 `FinishedAt` 一秒不差。整个诊断到此结束，代价只是一次 `dmesg`。容器没有退出，它是在自己的内存上限内被杀掉，5 秒后又被拉起来。

dmesg 里这样的记录一共 **6 条**，而 `RestartCount` 正是 **6**。同一个 stack 里其他容器的重启计数都是 0、启动时间都是几周前，所以这六次击杀可以钉在这一个容器上：

| dmesg 开机秒 | 本地时间 | 被杀时 anon-rss |
|---|---|---|
| 1406539 | 10-12 约 07:37 | 491 MB |
| 1407759 | 10-12 约 07:57 | 494 MB |
| 1432574 | 10-12 约 14:49 | 486 MB |
| 1734793 | 10-16 约 02:45 | 487 MB |
| 1735495 | 10-16 约 02:58 | 485 MB |
| 1737797 | 10-16 03:39 ← 那条通知 | 478 MB |

每一次被杀都发生在 478–494 MB，而上限是 512 MiB（`536,870,912` 字节）。这是天花板被摸到，不是失控。

还有两个读数把机制说清了。cgroup 记录的高水位（重启后仍然保留的那个计数器）在重启 **18 分钟**后就已经回到了正好 `536,870,912`——也就是说同一个周期里它又摸到了天花板，只是这次没被杀。而用量曲线是来回震荡的（176 → 308 → 361 MB），不是单调爬升，所以这是负载下的分配抖动撞上小上限，不是一个能证明的缓慢泄漏。最大嫌疑是那个 geoblock 插件：每遇到一个没见过的客户端 IP，它就要做一次外部 HTTPS 查询。

同一天早上 05:33 又来第 7 次击杀——在改动落下去之前。

在这台 DSM 上，真相只存在于 `dmesg` 加上 cgroup 的 `memory.max_usage_in_bytes`。如果你在普通主机上追同一个形状，先看 `docker inspect -f '{{.State.OOMKilled}}'`——它本该是权威。而这里它读出来是 `false`，就在内核记下那次击杀的 5 秒后。这就是这篇文存在的理由。

## 真正在崩的是什么：一场爬虫洪水

宿主内存从来不是问题——64 GB 里约 39 GB 可用。出问题的是某个容器的 512 MiB 上限。

驱动因素是流量，数字是从容器自己的访问日志里读出来的。最后 50,000 行里：

```text
不同客户端 IP          25,976
403 响应              33,636   （geoblock / 拒绝名单在正常工作）
429 响应              16,082   （inflight 与限流在正常工作）
容器 CPU              118–230%
宿主 load average     26–32
```

每次重启后，堆在 12 到 40 分钟内又被填满。配置里没有任何东西是错的，那些 403 和 429 也不是——它们正是过滤器按设计拒绝爬虫的结果。问题在于谁在敲门：这个容器是一个多 TB 级 Wikipedia 语料的公网入口，托管在家庭 NAS 上，前面挂着一份国家白名单和一个机器人过滤器。对住宅代理爬虫云来说，这是个极有吸引力的目标；而那些「请你离开」的响应，本身也是工作量。

## 修法，以及必须跟着的那句话

```yaml
 deploy:
   resources:
     limits:
-      memory: 512M
+      memory: 2g
 restart: always
+memswap_limit: 2g
```

验证（因为「我把上限调大之后感觉好多了」不叫验证）：

- **只有那一个容器被重建。** wiki、代理内核、robots 容器都还报着两周的 uptime——用户看到的是几秒的空窗，而不是整个 stack 重启。
- `HostConfig.Memory = 2147483648`、`MemorySwap = 2147483648`，cgroup 的 `memory.limit_in_bytes` 与两者一致；`RestartCount` 归零。
- 18 分钟后峰值用量 **210 MB**——而之前它一直贴着 512 MB 的天花板。到中午是 94 MiB。
- 挂载列表仍与文件一致，包括插件目录那条挂载——丢掉它曾经让这个 stack 里所有路由全部失效。
- 两个插件都加载且在生效（国家白名单、机器人过滤器），等过滤器的计分窗口过期后，各条路由都返回 200。

**以及它不是什么：** 这是权宜之计。驱动因素是洪水，所以大一个数量级的洪水照样能摸到 2 GB。真正的修法是缩小公网入口面，而不是把笼子做大——一条没被标明为权宜之计的上限调整，最终会变成一个永久解释。

我选择让这个容器不换页：`memswap_limit` 与内存上限相等就是关掉它的 swap。这台宿主当时已在 40 GB swap 里用掉 16 GB，commit 总量 92 GB，而一个开始换页的边缘代理会把整台 NAS 拖慢——磁盘阵列才是瓶颈，所以换页买到的是更慢的失败，不是更长的命。

## 那个差点让我报告「自伤事故」的坑

验证期间，我第一次用裸 `curl` 打过去，返回 **403**，看起来完全像我刚重启代理造成的事故。并不是：机器人过滤器给我**自己的客户端 IP** 打了分（`blocked user-agent`，分数 80），然后把它放进一个 15 分钟的计分窗口——因为我一直在猛敲这台机器。

线索就在同一轮测试里：`/robots.txt` 返回 200，内部的 `/cdn` 路径返回 400，而两者是同一时刻——说明路由是健康的，只有我自己的请求被过滤。在一台「你同时是客户端」的宿主上，永远不要用裸 curl 做判断：去探一个过滤器会忽略的路径，或者从访问日志里读出一个 200。

## 如果重来一次 <a id="what-i-would-do-differently"></a>

1. **把「意外停止」加 `ExitCode 0` 当成「去读 dmesg」的指令**，而不是干净退出的证据。重启计数大于零的容器，在证明清白之前都当作内存事故来查。
2. **对 cgroup 高水位告警，而不是只对重启告警。** 重启后那个已经贴着天花板的读数，在通知到达前约十八分钟就告诉我第 7 次击杀要来了。
3. **早点拿到「谁在打我」这个数字。** 一份日志样本里 25,976 个不同 IP 就是全部故事，而我是在看完整条内存曲线之后才找到它。
4. **怀疑容器，别怀疑服务器。** 「服务器内存不够」和「一台 64 GB 的机器上某个容器只有 512 MiB 上限」需要完全不同的应对，而后者你不去看那个上限就根本看不见。

## 结果 <a id="the-result"></a>

改动之后再没有出现 OOM 击杀，18 分钟后的峰值是 210 MB，而过去每一个周期都会摸到上限——四天里一直悄悄重启公网入口的那七次击杀，也从 dmesg 里消失了。爬虫洪水并没有停，所以我把这个失败形态记进了那套 stack 的运维笔记：**「容器意外停止」+ `exit 0` / `OOMKilled=false` 是一个会误导人的签名**，而在这类平台上，诚实的证据住在内核日志和 cgroup 的高水位里。
````

### 🇬🇧 英文版原文 (English Markdown Primary)
````markdown
---
title: "Docker Reported Exit 0. The Kernel Had Already OOM-Killed It."
description: "A container on my NAS kept stopping unexpectedly while Docker reported exit 0 and OOMKilled=false. The honest evidence was in dmesg: six cgroup OOM kills."
pubDate: 2026-10-16
category: devops
blueprint: self-hosted-infrastructure
blueprintLayer: L2-edge-proxy-runtime
tags: ["docker", "oom", "traefik", "synology", "crawlers"]
ogImage: /og/docker-reported-exit-0-the-oom-killer-disagreed.jpg
banner: /banners/docker-reported-exit-0-the-oom-killer-disagreed.jpg
draft: false
summary:
  audience: "SREs, DevOps Leads & Systems Engineers"
  painPoint: "A public edge proxy (Traefik) crashed 7 times over 4 days while Docker daemon reported ExitCode=0 and OOMKilled=false, rendering standard monitoring completely blind."
  impact:
    - metric: "Exposed Root Cause"
      detail: "Bypassed Docker's false status via kernel dmesg and cgroup high-water marks, pinning the crash to 25,976 crawler IPs hitting a 512MB limit."
    - metric: "Zero-Downtime Hotfix"
      detail: "Hot-scaled proxy container to 2GB with memory swap locked, dropping peak memory to 210MB in 18 minutes with 0 subsequent crashes."
    - metric: "Anti-Lockout SOP"
      detail: "Established edge crawler stress-testing and verified probe procedures to prevent diagnostic self-lockout from security filters."
  stack: ["Docker", "Linux Kernel (dmesg/cgroups)", "Traefik", "Synology DSM", "Security Filtering"]
  jumpLinks:
    business: "#the-result"
    architecture: "#the-kernel-keeps-its-own-record"
    result: "#what-i-would-do-differently"
dynamicCta: true
---

## The notification I expected to be a config mistake

My NAS sent the notification every Docker user learns to skim: **container
stopped unexpectedly**. This one is the edge proxy in front of my
self-hosted wiki, and it had been up for two weeks. Nobody had deployed
anything that morning and nothing in the compose file had been touched, so
the two available stories were "it crashed" and "something restarted it".

Both were wrong, and the fact that I could not tell from Docker's own report
is the part worth writing down.

## Docker's report says nothing happened

```text
State:
  Status:      running
  Running:     true
  ExitCode:    0            <- a clean exit
  OOMKilled:   false        <- not killed by the kernel
  FinishedAt:  2026-10-15T19:39:33.63057879Z
  StartedAt:   2026-10-15T19:39:38.769326672Z   <- restarted 5 s later
RestartCount: 6
RestartPolicy: always
```

Read that as written and the conclusion is: the process exited cleanly, six
times, and the restart policy gave it a new life each time. A long-running
proxy does not exit with status 0 on its own — and a clean exit that repeats
six times is a contradiction, not an answer. `ExitCode` and `OOMKilled` are
exactly the two fields I would have trusted here. On this host, both were
wrong.

## The kernel keeps its own record

On Linux, a cgroup memory kill is logged by the kernel even when the
container runtime's bookkeeping does not carry it forward:

```text
[1737797.044856] traefik invoked oom-killer: gfp_mask=0x2400040, order=0, oom_score_adj=0
[1737797.123736] Memory cgroup out of memory: Kill process 615 (traefik) score 970 or sacrifice child
[1737797.132800] Killed process 615 (traefik) total-vm:2525800kB, anon-rss:478480kB, file-rss:0kB
```

The uptime stamp `1737797` corresponds to 19:39:33 UTC — to the second, the
container's `FinishedAt`. That is the whole diagnosis, and it took one
`dmesg` to get it. The container did not exit; it was killed inside its own
memory limit and restarted 5 seconds later.

dmesg holds **six** such events, and `RestartCount` was **6**. Every other
container in that stack has a restart count of zero and a start time from
weeks ago, so the six kills pin to this one container:

| dmesg uptime | local time | anon-rss at kill |
|---|---|---|
| 1406539 | Oct 12, ~07:37 | 491 MB |
| 1407759 | Oct 12, ~07:57 | 494 MB |
| 1432574 | Oct 12, ~14:49 | 486 MB |
| 1734793 | Oct 16, ~02:45 | 487 MB |
| 1735495 | Oct 16, ~02:58 | 485 MB |
| 1737797 | Oct 16, 03:39 ← the notification | 478 MB |

Every kill happens at 478–494 MB against a limit of 512 MiB (`536,870,912`
bytes). That is a ceiling being touched, not a runaway.

Two more readings made the mechanism clear. The cgroup's recorded peak — the
counter that survives the restart — was already back at exactly
`536,870,912` **18 minutes** after the restart, i.e. the container touched
its ceiling again in the same cycle *without* being killed. And the usage
curve oscillated (176 → 308 → 361 MB) instead of climbing monotonically, so
this is allocation churn under load hitting a small cap, not a provable slow
leak. The prime suspect is a geoblock plugin doing an external HTTPS lookup
for every client IP it has not seen before.

A seventh kill landed at 05:33 that morning, before the fix went in.

On this DSM build the truth was only in `dmesg` plus the cgroup's
`memory.max_usage_in_bytes`. If you are chasing the same shape on a normal
host, `docker inspect -f '{{.State.OOMKilled}}'` is the field to read first
— it is supposed to be authoritative. Here it read `false` five seconds
after the kernel had logged the kill, which is the reason this post exists.

## What was actually breaking: a crawler flood

Host memory was never the problem — 64 GB installed, ~39 GB available. One
container's 512 MiB ceiling was.

The driver was traffic, and the numbers came out of the container's own
access log. In the last 50,000 lines:

```text
distinct client IPs      25,976
403 responses            33,636   (geoblock / deny-list doing its job)
429 responses            16,082   (inflight and rate limiting doing their job)
container CPU            118–230%
host load average        26–32
```

Each restart refilled the heap in 12 to 40 minutes. Nothing in the
configuration was wrong, and neither were the 403s and 429s — they are the
filter rejecting crawlers exactly as designed. The problem is who is
knocking: this container is the public edge for a multi-terabyte Wikipedia
corpus served from a home NAS, behind a country allowlist and a bot filter.
That is an extremely attractive target for a residential-proxy crawler
cloud, and the responses telling it to go away are still work.

## The fix, and the sentence that has to come with it

```yaml
 deploy:
   resources:
     limits:
-      memory: 512M
+      memory: 2g
 restart: always
+memswap_limit: 2g
```

Verification, because "I raised the limit and it feels better" is not a
verification:

- **Only that container was recreated.** The wiki, the proxy core and the
  robots container still reported two weeks of uptime — the gap users saw
  was seconds, not a stack restart.
- `HostConfig.Memory = 2147483648`, `MemorySwap = 2147483648`, and the
  cgroup's `memory.limit_in_bytes` agreed with both; `RestartCount` went
  back to 0.
- Peak usage after 18 minutes: **210 MB**, against a ceiling it had been
  pinning at 512 MB. By midday it sat at 94 MiB.
- The mount list still matched the file, including the plugin directory
  mount — losing that one had once disabled every router in this stack.
- Both plugins were loaded and active (country allowlist, bot filter), and
  the routes returned 200 once the filter's score window expired.

**And what it is not:** this is a stopgap. The flood is the driver, so an
order-of-magnitude bigger flood reaches 2 GB too. The real fix is shrinking
the public entry surface, not enlarging the cage around it — and a limit
bump that is not labelled as a stopgap becomes a permanent explanation.

I chose to disable swap for this container rather than let it swap:
`memswap_limit` equal to the memory limit does exactly that. The host
already had 16 GB of its 40 GB swap in use with committed memory at 92 GB,
and an edge proxy that starts swapping slows the entire NAS — the disk array
is the bottleneck here, so swapping buys a slower failure rather than a
longer life.

## The trap that nearly made me report a self-inflicted outage

During verification, my first bare `curl` probes against the site returned
**403**, and it looked exactly like a regression I had just caused by
restarting the proxy. It was not: the bot filter had scored **my own client
IP** (`blocked user-agent`, score 80) and placed it in a 15-minute score
window, because I had been hammering the host.

The tell was in the same test run — `/robots.txt` returned 200 and the
internal `/cdn` path returned 400 at the same moment, i.e. the router was
healthy and only my own requests were being filtered. On a host where you
are also a client, never diagnose with your own bare curl: probe a path the
filter ignores, or read a 200 out of the access log.

## What I would do differently

1. **Treat "stopped unexpectedly" plus `ExitCode 0` as an instruction to
   read `dmesg`**, not as evidence of a clean exit. A restart count above
   zero is a memory investigation until proven otherwise.
2. **Alert on the cgroup peak, not only on restarts.** The post-restart
   reading that was already pinned at the ceiling told me a seventh kill was
   coming about eighteen minutes before the notification arrived.
3. **Get the "who is hitting me" number early.** 25,976 distinct IPs in one
   log sample is the entire story, and I found it after spending time on the
   memory curve.
4. **Question the container, not the server.** "The server is out of memory"
   and "one container has a 512 MiB ceiling on a 64 GB machine" need
   completely different responses, and the second one is invisible unless
   you look at the limit.

## The result

No OOM kill since the change, with a peak at 210 MB after 18 minutes against
a ceiling that used to be touched in every single cycle — and the seven
kills that had been silently restarting the public edge for four days
stopped appearing in dmesg. The crawler flood did not stop, which is why I
recorded the failure mode in my ops notes next to the stack: "container
stopped unexpectedly" plus `exit 0` / `OOMKilled=false` is a misleading
signature, and on this platform the honest evidence lives in the kernel log
and the cgroup's high-water mark.
````

---

## 文章 10 / 11：每个播放器都在同一个视频上卡顿，而文件本身是无辜的
- **Slug**: `every-player-stuttered-the-file-was-fine`
- **英文标题**: Every Player Stuttered on the Same Video. The File Was Blameless.
- **分类**: `devops` | **发布日期**: 2026-10-08
- **线上地址**: https://blog.hoelee.com/posts/every-player-stuttered-the-file-was-fine/ (EN) | https://blog.hoelee.com/posts/zh/every-player-stuttered-the-file-was-fine/ (ZH)
- **本篇审核状态**: ⏳ 待审核 (Pending)

### ✍️ 顾问审阅意见区 (Review Feedback)
```text
【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：
```

### 🇨🇳 中文版原文 (Chinese Markdown Twin)
````markdown
---
title: "每个播放器都在同一个视频上卡顿，而文件本身是无辜的"
description: "同一个 HEVC 文件，在 VLC、Chrome 和 iPad 上（离线）都出现同样的周期卡顿。容器层时间轴完美无缺，画面却不是——NVDEC 的旧版 AV1 包装器返回了错误时间点的帧。"
pubDate: 2026-10-08
category: devops
tags: [ffmpeg, nvdec, hevc, transcoding, video]
ogImage: /og/zh-every-player-stuttered-the-file-was-fine.jpg
banner: /banners/every-player-stuttered-the-file-was-fine.jpg
draft: false
---

## 一个我回答不了的问题

有个转好的视频会卡顿。不严重——每隔几秒顿一下，然后继续放。这种毛病你通常会怪到 Wi-Fi 头上。

问题是它出现在一台 **离线**播放的 iPad Pro M1 上、出现在 VLC 里、也出现在桌面版 Chrome 里。不同的解码器、不同的平台、同样的周期性卡顿。而原始下载的视频在所有地方都播放正常。所以网络被排除、设备被排除，剩下的嫌疑只有那个文件。

我花了很长时间测量这个文件，所有测量都说它是好的。它并不是好的。它是**对一个解码器 bug 的忠实重编码**。

## 第一步：先量容器，而不是量画面

这个转好的文件是 MP4 容器里的 HEVC Main 8-bit（`hvc1`）、3840×2160、25 fps、yuv420p、bt709、平均 15.49 Mbps、232.8 秒、5819 帧，`moov` 在 `mdat` 之前。

它的时间轴是教科书级别的干净：

```text
帧间隔              5818 个全部 = 恰好 0.040000 秒
重复 PTS            0
负值 PTS            0
超过中位数 1.5 倍的缺口   0
edit list (ctts)    5796 条（不是扁平表）
has_b_frames        2
音视频起始偏移        0.000 秒
B 帧图案            I B B B P B B B ……（744 组，每组都是 3 个 B）
```

扁平的 `ctts`——把每个呈现偏移都设成零——是重编码后 PTS 停留在解码顺序上的经典症状，会让播放器丢掉大约四分之一的帧。这正是我预期会找到的东西。它不在那里。

完整解码，两个详细级别都跑：

```bash
ffmpeg -v warning -i converted.mp4 -f null -
ffmpeg -v error   -i converted.mp4 -f null -
```

两条都以 0 退出、什么都不打印，232 秒的视频用了 174 秒跑完。没有损坏帧、没有缺失参考帧、没有非单调 DTS。

GOP 确实和源不同（24 个关键帧、恒定 10.0 秒，因为 `hevc_nvenc` 默认 `-g 250`；源是 59 个关键帧、平均 3.88 秒），码率也比源高 17%——但源自身的峰值比输出的峰值还高，所以两者都解释不了"每隔几秒顿一下"。

**容器里所有可测的东西都是正确的。** 而这恰恰是"你就该停止测量容器"的那个点。

## 第二步：用两个不同的解码器去比画面

如果时间轴是对的、文件看起来却仍是错的，问题就变成：**画面**本身对不对。我把源文件用两种方式解码，再逐帧做 SSIM 对比：

```bash
# 软件解码源文件
ffmpeg -i source.mkv -vf "scale=640:360" -f rawvideo -pix_fmt yuv420p sw.yuv

# 同一批帧走 NVDEC 的 AV1 路径
ffmpeg -c:v av1_cuvid -i source.mkv -vf "scale=640:360" -f rawvideo -pix_fmt yuv420p hw.yuv
```

5819 帧里有 **106 帧**不一致——占 1.8%——SSIM 最低到 **0.330**。不是均匀的噪声：间隔直方图的峰值大约落在每 30 帧一次。

两个对照实验，一个让情况更糟、一个让情况更清楚：

- **确定性。** 同一个源用 `av1_cuvid` 解两次，两次之间 SSIM 是 1.000000。这不是竞争条件；每一次转换都会把同一批坏帧焊进输出里。
- **不是所有 AV1 都中招。** 一个 1080p 的 AV1 样本两条路径解出来完全一致（SSIM 1.000000），而另一个 4K AV1 流不一致。这跟具体的流有关。

然后是决定性的对比——转好的文件到底跟哪一边一致？

| 对比 | 平均 SSIM | 最低 | 低于 0.98 的帧数 |
|---|---|---|---|
| 转好的 HEVC vs 源的软件解码 | 0.99091 | **0.330** | **106** |
| 转好的 HEVC vs 源的 NVDEC 解码 | **0.99742** | 0.991 | **0** |

转好的文件和**错误的那一份**解码逐帧吻合。它不是一次糟糕的编码，它是**对错误画面的一次准确编码**。

抽出一帧就能看得很清楚：第 546 帧在软件解码里是合唱团的远景；在 `av1_cuvid` 里是另一个时刻的特写镜头——而转好的文件里是那张特写。

## 第三步：修复只要一行

Bug 不在硬件，在旧的解码器包装器。`av1_cuvid` 属于老式的 wrapper 解码路径；现代的硬件路径要走 `hwaccel`：

```bash
# 修复前
-c:v av1_cuvid

# 修复后
-hwaccel cuda -hwaccel_output_format cuda -c:v av1
```

同一个流、同一个源，1500 帧对软件参考逐帧比较：

```text
-hwaccel cuda -c:v av1        SSIM 1.000000   （逐帧逐位一致）
vp9_cuvid vs libvpx-vp9       SSIM 0.99991    （0 帧低于 0.98——不受影响）
```

编码速度没有变化：一段 30 秒的 4K 素材 24.1 秒编完，两种路径都是 1.24× 实时。

范围要说准确：这是这套驱动/库组合下 **AV1** 的 `*_cuvid` 包装器的 bug，可稳定复现。它与你显卡的 AV1 解码器无关——现代硬件路径是干净的。

## 第四步：让它再也不可能悄悄发出去

最让人不安的地方在于：我的流水线**本来就有**验证步骤。它检查时长、分辨率、编码器、帧率和帧节奏——全是容器属性。这些检查在一个带着 106 帧错误画面的文件上**全部通过**。

所以现在流水线是五步，第四步才是画面检查：

```text
解码 -> 编码 -> 结构验证 -> 内容验证 -> 替换文件
```

内容闸门把输出帧与源的软件解码逐帧对比（两侧都顺序解码、按时间戳裁剪取窗口），并且判的是缺陷的**形状**，而不是某一帧的硬线：

- 任何一帧低于 0.85 → 失败
- 低于 0.95 的帧超过 1% → 失败
- 均值低于 0.98 → 失败
- 剔除近平坦帧（对几乎全黑的帧，SSIM 没有意义——两帧几乎全黑也能算出 0.25，而它们在视觉上完全相同）

这些阈值是拿实测数据标定出来的，不是凭直觉。这个素材库里好的编码在 1500 帧里有 0 帧低于 0.95；有缺陷的文件有 2.9% 的帧低于 0.85 且均值下滑。而一段完全正常的 4K60 舞蹈视频曾经测出最低 0.949——一个硬切处只有 1 帧低于 0.95——用单帧硬线判的话，它会被永久拒绝。

范围：快照里有 AV1 原档的共 75 个文件，其中 **65 个（87%）**没通过内容闸门。这 65 个全部用**快照里的干净原档**重转（绝不用已经被弄坏的文件），每一个都以 min SSIM 0.997–0.999 落地。

## 我会怎么做得不一样

1. **"解码没有报错"不是质量检查。** 一帧能解码成功，跟它是不是**正确的那一帧**毫无关系。找出问题的是画面比对。
2. **跟第二个解码器比，而不是跟你的预期比。** 软件解码是免费的，而且它就是参考基准。如果你的硬件路径跟它不一致，你就找到了值得搞清楚的东西。
3. **验证输出要对着源，而不是对着源的元数据。**
4. **别以为现代硬件路径和旧包装器是同一条代码路径。** `av1_cuvid` 和 `-hwaccel cuda -c:v av1` 走的是不同的管道、到的是同一块硅片。其中一个有 bug，不等于对硬件下判决。
5. **一个"失败得安全"的闸门，比一个从不失败的闸门值钱得多。** 我的第一版闸门拒绝了那段 4K60 编码，文件是好的、闸门是错的——但这个错误的代价只是白烧一次编码，而不是让一个坏文件进了库。

卡顿消失了。不是因为编码器变好了，而是因为解码器不再说谎了。
````

### 🇬🇧 英文版原文 (English Markdown Primary)
````markdown
---
title: "Every Player Stuttered on the Same Video. The File Was Blameless."
description: "Same HEVC file, same stutter in VLC, Chrome and an iPad — offline. The container timing was perfect. The pictures were not: NVDEC's legacy AV1 wrapper was returning frames from the wrong timestamps."
pubDate: 2026-10-08
category: devops
tags: [ffmpeg, nvdec, hevc, transcoding, video]
ogImage: /og/every-player-stuttered-the-file-was-fine.jpg
banner: /banners/every-player-stuttered-the-file-was-fine.jpg
draft: false
---

## The question I could not answer

A converted video stuttered. Not badly — a hitch every few seconds, then it
carried on. The kind of thing you blame on Wi-Fi.

Except it happened on an iPad Pro M1 playing the file **offline**, in VLC,
and in Chrome on a desktop. Different decoders, different platforms, same
periodic hitch. The original download played perfectly everywhere. So the
network was out, the device was out, and the file was in.

I spent a long time measuring the file, and every measurement said the file
was fine. It was not fine. It was a faithful re-encode of a decoder bug.

## Step 1: measure the container, not the picture

The converted file is HEVC Main 8-bit in MP4 (`hvc1`), 3840×2160, 25 fps,
yuv420p, bt709, average 15.49 Mbps, 232.8 seconds, 5819 frames, `moov`
before `mdat`.

Everything about its timeline is textbook:

```text
frame intervals       5818 of 5818 = exactly 0.040000 s
duplicate PTS         0
negative PTS          0
gaps > 1.5x median    0
edit list (ctts)      5796 entries  (not a flat list)
has_b_frames          2
A/V start offset      0.000 s
B-frame pattern       I B B B P B B B ...  (744 groups, all 3 B-frames)
```

A flat `ctts` — every presentation offset set to zero — is the classic way a
re-encode ends up with PTS sitting in decode order, and it makes players
drop roughly a quarter of the frames. That is what I expected to find. It
was not there.

Full decode, both verbosity levels:

```bash
ffmpeg -v warning -i converted.mp4 -f null -
ffmpeg -v error   -i converted.mp4 -f null -
```

Both exit 0, both print nothing, 174 seconds of wall time for 232 seconds of
video. No corrupt frames, no missing references, no non-monotonic DTS.

The GOP looked different from the source (24 keyframes, a constant 10.0 s,
because `hevc_nvenc` defaults to `-g 250`; the source has 59 keyframes at an
average 3.88 s) and the bitrate was 17% higher than the source — but the
source's own peak was higher than the output's, so neither explains a hitch
every few seconds.

**Everything measurable in the container was correct.** Which is the point
where you have to stop measuring the container.

## Step 2: compare pictures, from two different decoders

If the timing is right and the file still looks wrong, the question becomes
whether the *pictures* are right. I compared the source file decoded two
ways, frame by frame, with SSIM:

```bash
# software decode of the source
ffmpeg -i source.mkv -vf "scale=640:360" -f rawvideo -pix_fmt yuv420p sw.yuv

# the same frames through NVDEC's AV1 path
ffmpeg -c:v av1_cuvid -i source.mkv -vf "scale=640:360" -f rawvideo -pix_fmt yuv420p hw.yuv
```

106 frames out of 5819 disagree — 1.8% — with SSIM as low as **0.330**. Not
uniform noise: the interval histogram peaks at about every 30 frames.

Two control tests made it worse and better at the same time:

- **Deterministic.** Decoding the same source twice through `av1_cuvid`
gives SSIM 1.000000 between the two runs. This is not a race; every
conversion will weld the same bad frames into the output.
- **Not all AV1.** A 1080p AV1 sample decoded identically through both paths
(SSIM 1.000000). A second 4K AV1 stream did not. It is stream-dependent.

Then the decisive comparison — which side does the converted file match?

| Comparison | Mean SSIM | Minimum | Frames below 0.98 |
|---|---|---|---|
| Converted HEVC vs software decode of the source | 0.99091 | **0.330** | **106** |
| Converted HEVC vs NVDEC decode of the source | **0.99742** | 0.991 | **0** |

The converted file agrees with the *wrong* decode, frame for frame. It is
not a bad encode. It is an accurate encode of the wrong pictures.

A single extracted frame shows it plainly: frame 546 through software decode
is a wide shot of a choir; through `av1_cuvid` it is a close-up from a
different moment in the video — and the converted file has the close-up.

## Step 3: the fix is one line

The bug is not the hardware. It is the legacy decoder wrapper. `av1_cuvid`
is an old-style wrapped decoder; the modern hardware path goes through
`hwaccel`:

```bash
# before
-c:v av1_cuvid

# after
-hwaccel cuda -hwaccel_output_format cuda -c:v av1
```

Same stream, same source, 1500 frames compared against the software
reference:

```text
-hwaccel cuda -c:v av1        SSIM 1.000000   (bit-identical, every frame)
vp9_cuvid vs libvpx-vp9       SSIM 0.99991    (0 frames below 0.98 - unaffected)
```

Encoding speed did not change: a 30-second 4K slice in 24.1 s, 1.24×
realtime either way.

To be precise about the scope: this is a bug in the `*_cuvid` wrapper for
**AV1** on this driver/library combination, reproducible on demand. It says
nothing about your GPU's AV1 decoder — the hardware path is clean.

## Step 4: make it impossible to ship again

The uncomfortable part is that my pipeline *had* a verification step. It
checked the duration, the resolution, the codec, the frame rate and the
frame rhythm — all container properties. Every one of them passed on a file
with 106 wrong frames.

So the pipeline is now five steps, and the fourth one is the picture check:

```text
decode -> encode -> verify structure -> verify content -> replace the file
```

The content gate compares the output's frames against a software decode of
the source (both sides decoded sequentially, cropped by timestamp), and uses
the defect's *shape* rather than a single-frame threshold:

- fail if any frame falls below 0.85
- fail if more than 1% of frames fall below 0.95
- fail if the mean falls below 0.98
- ignore near-flat frames (SSIM is meaningless on frames that are almost
entirely black — two nearly-black frames can score 0.25 while being visually
identical)

Those numbers are calibrated against measured data, not intuition. Good
encodes in this library sit at 0 frames of 1500 below 0.95. The defective
files sit at 2.9% of frames below 0.85 with a depressed mean. A perfectly
good 4K60 encode of a dance video once measured min 0.949 — a single frame
below 0.95, at a hard cut — which a single-frame threshold would have
rejected forever.

Scoped to the library: 75 files had AV1 originals in the snapshots, 65 of
them (87%) failed the content gate. All 65 were re-encoded from the pristine
snapshot originals — never from the already-corrupted file — and every one
landed with min SSIM 0.997–0.999.

## What I would do differently

1. **"It decodes without errors" is not a quality check.** Decoding a frame
successfully says nothing about whether it is the *right* frame. The picture
comparison is the check that found it. 2. **Compare against a second
decoder, not against your expectations.** A software decode is free and it
is the reference. If your hardware path disagrees with it, you have found
something worth understanding. 3. **Verify the output against the source,
not the source's metadata.** 4. **Do not assume the modern and legacy
hardware paths are the same code.** `av1_cuvid` and `-hwaccel cuda -c:v av1`
reach the same silicon through different plumbing. A bug in one is not a
verdict on the hardware. 5. **A gate that fails safe is worth more than a
gate that never fails.** My first gate rejected that 4K60 encode. The file
was fine and the gate was wrong — but the failure cost one wasted encode,
not a bad file in the library.

The stutter is gone. Not because the encoder got better, but because the
decoder stopped lying.
````

---

## 文章 11 / 11：加固 Tor 洋葱服务:真正重要的是什么
- **Slug**: `hardening-a-tor-onion-service`
- **英文标题**: Hardening a Tor Onion Service: What Actually Matters
- **分类**: `devops` | **发布日期**: 2025-12-10
- **线上地址**: https://blog.hoelee.com/posts/hardening-a-tor-onion-service/ (EN) | https://blog.hoelee.com/posts/zh/hardening-a-tor-onion-service/ (ZH)
- **本篇审核状态**: ⏳ 待审核 (Pending)

### ✍️ 顾问审阅意见区 (Review Feedback)
```text
【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：
```

### 🇨🇳 中文版原文 (Chinese Markdown Twin)
````markdown
---
title: "加固 Tor 洋葱服务:真正重要的是什么"
description: "出于对暗网的好奇,我把个人主页和 Git 服务器镜像了过去。这篇记录哪些加固扛得住、哪些会悄悄坏掉,以及值不值得把它当成服务卖给客户。"
pubDate: 2025-12-10
updatedDate: 2026-09-09
category: devops
blueprint: self-hosted-infrastructure
blueprintLayer: L4-secops-cicd-dr
tags: [tor, docker, security, self-hosting, networking]
ogImage: /og/zh-hardening-a-tor-onion-service.jpg
banner: /banners/hardening-a-tor-onion-service.jpg
---

我对暗网产生了好奇。不是集市那一面,而是无聊的那一半:人们像注册商和云控制台出现之前那样自托管——跑自己维护的软件,用一个自己掌控的地址就能访问。

我回应好奇心的方式一向是动手做一个东西,于是实验就这样定了:把自己的网站镜像到 Tor 上。我的交互式个人主页在 [me.hoelee.com](https://me.hoelee.com),代码托管在一个 Gitea 实例 [git.hoelee.com](https://git.hoelee.com) 上。现在两者都有暗网分身:

- `hoeleegitkcng572znkbpyffppyulsdwv3aurrzlk7y7vlhknogswoqd.onion` — Gitea 镜像
- `hoeleeaiwowgndbxswegtdzoeupz7lkkechtqmurbmnpvwa4k3vyuyid.onion` — 个人主页镜像

### 如果你从没打开过 `.onion`

这些地址在普通浏览器里打不开,你需要 Tor Browser,它会把你的流量经过 Tor 网络转发。安装大概五分钟:

1. 从 **[torproject.org/download](https://www.torproject.org/download/)** 下载。
2. **验证签名。** 下载页会给出 `.asc` 文件和签名密钥的链接,这一步很重要——被篡改的 Tor Browser 是丢掉匿名性最糟糕的方式。先取得 Tor Browser 的签名密钥(指纹发布在下载页和 Tor Browser 手册里),再校验文件:
   ```bash
   # 导入签名密钥——见下载页上的 .asc 链接
   gpg --verify tor-browser-*.tar.xz.asc tor-browser-*.tar.xz
   ```
3. 解压并运行。Windows 和 macOS 是常规安装包;Linux 解压后运行 `./start-tor-browser.desktop`。
4. **点 Connect。** 默认设置就行。如果你所在网络有审查,选 "Configure connection" 并使用网桥(Snowflake 或 obfs4)。
5. 等左上角出现 "Connected" 和链路显示,就说明你已经在 Tor 上了。

然后把上面的地址**一字不差**粘进地址栏。有两件事要知道:

- v3 地址是 **56 个字符**的 base32(`a–z`、`2–7`)加 `.onion`。没有拼写纠错,也没有搜索建议——错一个字符就是一个打不开的地址,而且不会有任何提示。
- 16 个字符的地址(v2)完全打不开。v2 协议已在 2021 年退役。

有一点会让第一次用的人意外:全新的洋葱地址可能需要 **10 到 60 分钟**才会被网络识别。Tor 必须先发布 descriptor,并让它被哈希环接受。第一次打不开通常只是地址还太年轻,不是配置坏了。

研究怎么把它做好时,我在网上读到了很多讲 obfs4 好处的文章。最终让我的部署真正安全的东西,和那些阅读没什么关系。

那么,真正让一个洋葱服务安全的是什么?我设镜像时认真走了一遍全程,几个月后又回头检查了一遍。有一部分配置经受住了考验,有一部分已经悄悄坏掉,还有几件我原本对 Docker 的认知,被实测证明是错的。

## 三十秒搞懂 Tor 和洋葱服务

Tor 是一个匿名网络:流量经过三道中继,没有任何一道中继能同时知道你是谁、要去哪。洋葱服务则是反过来的方向——一个生活在网络*内部*的服务器。它拿到的是 `.onion` 地址而不是域名,访客只能通过 Tor 网络访问它,服务器的真实位置永远不会出现在连接里。

由此得到四个普通网站给不了的特性:

- **没有公网 IP、没有端口、没有 DNS。** 扫描器无从发现。你可以用处于 CGNAT 后的私有网络来跑(现在大多数宽带都是)。
- **没有中间商。** 路径上没有任何 CDN、注册商或平台记录谁来过。
- **地址本身就是访问控制。** 拿到地址的人可以访问;其他人连这站存不存在都无法判断。要求更高时还能叠加客户端密钥。
- **clearnet 挂了它还在。** 域名被墙或被没收,洋葱站毫发无损,因为没有注册局参与其中。

洋葱托管适合这些:私密文件交接、作品集和代码仓库镜像、律师之间传草稿、数字商品交付、任何永远不该出现在搜索索引里的档案。会想要它的人,是把"没人能看出它的存在"当作特性而不是怪癖的人:律师、审计师、交付成果的自由职业者、给记者递料的线人、以及不希望自家数字商品渠道被同行扫描的商家。

诚实的代价是速度。Tor 链路比 CDN 慢,所以你不会把营销站放上去。它是给互联网的"私密那一半"用的工具,不是给公开那一半用的。

## 真正保护源站的是三件事

只有最后一件需要花力气:

1. **协议本身。** 访客从不直连你的服务器。你的 tor 进程主动拨出、向 introduction point 注册,会合发生在 Tor 网络内部。单凭一次访问,访客拿不到你的 IP。
2. **Vanguards-lite。** Tor 0.4.7 起内置。它让 guard-discovery 攻击(攻击者反复制造 circuit 直到观察到你的 guard 中继)变得很难实施。你只要跑一个当前版本的 Tor 就白送这层防护。
3. **别让源站信息从其它渠道泄漏。** 洋葱主机暴露的现实路径不是流量分析。是你自己的机器在漏:某台 clearnet 服务被攻破,攻击者直接从硬盘读走你的洋葱密钥。公开站点镜像到洋葱是刻意的,内容关联我不担心;真正要管的是,存密钥的那台机器别同时跑着一堆开着的服务。

## 那些悄悄坏掉的东西

重检自己服务器时,是这些地方让我觉得花的时间值了。

**应用容器有完整的互联网访问。** 我在容器里跑了一行访问公网 IP 回显服务的命令,拿回了我的家庭 IP。之前*确实*有过一条基于 iptables 的阻断,但规则已经没了。主机防火墙会被接口变动、容器管理器重启、平台升级静默清空。一条没有任何机制重放、没有任何告警的阻断,不是阻断,是迷信。这件事让我下定决心:与其过滤应用的出网流量,不如干脆把它的出网彻底移除(见下文)。

**主机防火墙是关的。** INPUT 和 FORWARD 策略全是 ACCEPT,同一台机器上其它服务在全部接口上监听了好几十个端口。其中任何一个被打穿,这块盘上的洋葱密钥就归攻击者了。人们愿意花几个小时做 Tor 专属加固,却跳过这一步。

**运行中的 tor 和配置文件对不上。** 磁盘上的 torrc 写着 `SocksPort 0`;已经跑了几天进程,还在监听 127.0.0.1:9050。我改了文件,忘了重启容器。这里的实际影响很小,因为监听只在容器内部可达。但教训可以推广:你写的配置文件是愿望,进程实际在做什么才是真相。

**Tor 落后了一个安全版本。** 对一台存放敏感密钥的机器来说太旧了,而且我的日志里带着新版已修复的中继描述符解析 bug 的刷屏签名。升级后刷屏停了。

## 修好之后不再坏的修复

### 零出网,靠构造而不是靠过滤

两个容器放进 `internal: true` 的 Docker 网络,只有 tor 容器多一张网卡去访问 Tor 网络。

```yaml
networks:
  service_net:
    driver: bridge
    internal: true        # 没有网关,没有 MASQUERADE,没有出路
  egress:
    external: true        # 普通的带外网 bridge
```

应用现在就哪里都去不了:公网不行,局域网不行,连主机都不行——internal 网络根本没有网关。没有可被清空的 iptables,没有可被遗忘的开机任务,也没有可以静默衰退的余地。就算应用被攻破,攻击者拿到的只是一根指向 tor 的 socket,仅此而已。这是我认为唯一不可妥协的一条改动。

### 非 root,和 capability 的意外

镜像默认就是非 root 用户,而我想让应用继续监听 80 端口。标准建议是 `cap_add: NET_BIND_SERVICE`。没用。容器崩溃循环,报:

```
[FATAL] Server error: listen tcp 0.0.0.0:80: bind: permission denied
```

这个原因让我意外到用探针容器实测了一把:`grep Cap /proc/self/status` 显示 `CapBnd` 里有 bit 10(NET_BIND_SERVICE),而 `CapEff` 是零。Docker 给非 root 容器的 capability 只进 bounding set,不进 effective set,而 bind() 检查的是 effective set。一个非 root 进程去执行没有 file capability 的二进制,拿到的 effective set 就是空的,没有例外。所以答案很无聊:内部改用非特权端口。

```yaml
  app:
    user: "1000:1000"
    cap_drop: [ALL]
    security_opt: [no-new-privileges]
  tor:
    user: "100:101"
    cap_drop: [ALL]
    security_opt: [no-new-privileges]
```

tor 侧一行把访问重新映射:

```
HiddenServicePort 80 app:8080
```

访客仍然从洋葱地址的 80 端口进来。变的只是内部端口。

### 检查对的东西的 healthcheck

tor 镜像自带一个探 SOCKS 端口的 healthcheck。我刚把 SOCKS 关了,于是 healthy 永远变成了 unhealthy。用你真正想问的问题覆盖它:进程还活着吗?

```yaml
healthcheck:
  test: ["CMD", "pgrep", "-x", "tor"]
```

nginx 的坑更隐蔽。我的第一版用了 `wget --spider` 探根路径。站点返回 404(当时内容还没传),wget 以非零退出,容器被标成不健康。这个检查测的是内容,不是服务。`nc -z 127.0.0.1 80` 只测端口,别的什么都不测。

还有一件:tor 在启动时解析 `HiddenServicePort` 的目标地址。如果应用容器还没起来,tor 会报 "Unparseable address in hidden service port configuration" 然后崩溃循环。我在旧日志里看过它连续四次启动失败,没人发现,因为当时没有任何东西在看着。compose 里的 `depends_on` 修好了启动顺序。

### 属主地雷

换完新密钥后,我通过文件共享挂载把密钥材料拷了回去。之后的容器立刻崩溃循环:

```
[warn] Could not open "/var/lib/tor/.../hs_ed25519_secret_key": Permission denied
```

经共享写入的文件,属主是共享用户,不是 tor 运行的 uid。修复只需一条命令,在哪里都能跑,包括挂上卷的一次性容器:

```bash
docker run --rm -v /path/libTor:/var/lib/tor alpine \
  sh -c "chown -R 100:101 /var/lib/tor && chmod -R 700 /var/lib/tor"
```

### 验证,而不是相信

上面每个修复都有一个我自己能跑的测试收尾。我最喜欢出网这条,因为两个探针放在一起最有说服力:同一个 wget,在应用的网络里失败,在 tor 的出网网络里成功。

```bash
# 在应用里:internal:true 之下,连 DNS 都应该失败
wget -T 6 -qO- http://ipv4.icanhazip.com     # → "wget: bad address", exit 1

# 同一个探针,放在 tor 的出网网络上,作为对照组
wget -T 6 -qO- http://ipv4.icanhazip.com     # → <你的 IP>, exit 0
```

如果没法 exec 进容器,就挂一个一次性探针容器到同一个网络。它测的是网络的属性,而那正是你加固的对象。

## 值得把它做成服务吗?

我一直在想这件事,因为边际成本几乎为零:tor 容器和它的隔离措施本来就在跑。

我大多数托管客户想要的和洋葱服务正好相反。他们想被 Google 搜到。卖一个只能在 Tor Browser 里打开的网站,等于卖给他们大概率用不上的隐秘性,还要负责教他们的访客安装 Tor Browser。

但确实有一小块真实市场。交换草稿的律师、审计师、交付数字商品的人,任何想分享一份永远不该出现在搜索引擎索引里的档案的人。对这类客户,销售话术自己就写好了:没有端口转发,没有域名,没有你控制不了的平台上留日志,只有一个你亲手交给该拿到的人的地址。

有个卖点比我预想的好使,它连着一个人人都会问的问题:地址开头能自己选吗?v3 洋葱地址是随机的,但那只是因为密钥是随机的。地址可以挖:不断生成密钥对,直到 base32 地址以你想要的前缀开头。每多一个字符,工作量乘以 32。社区挖矿工具在现代 GPU 上每秒能检查几百万个地址,这意味着 8 字符前缀是"一天到几天"的活,9 字符需要耐心地挖上几周,10 字符则是严肃的多 GPU 投入。一个以客户品牌开头的前缀,能把一行难记的 56 字符地址变成他们可以确认"这真的是你"的东西——在一个信任就是全部产品的细分市场里,这是实打实的价值。8 字符我乐意挖,9 字符为了付费客户可以,10 字符只有在对方自己租 GPU 的情况下我才会一脸平静地报价。

所以:作为给少数特定客户的附加服务,值得;作为一条产品线,不值。市场太薄,撑不起渠道,而且支持成本不会随规模摊薄。"带洋葱地址的隐私咨询",可以;"洋葱托管"作为一个网站套餐,那是别人的生意。

## 最后留下的

- **跑测试,别读配置文件。** 容器内探针和健康状态是真相,yaml 只是意图。
- **优先选择无法静默解体的机制。** `internal: true` 永远胜过 iptables 开机任务。
- **healthcheck 很便宜。** 它几分钟就抓到了一个崩溃循环,而之前没人看着,坏了好几天。
- **朴素的主机防火墙比花哨的 Tor 加固重要。** 如果攻击者能直接从一个开着的端口走进来,就不会有人费劲用流量分析去匿名化你。

## 想要一个吗?

我帮客户做这类东西:创建洋葱站点、托管和维护(更新、健康检查、监控),以及应要求挖一个品牌前缀的地址,让地址以名字开头而不是一串乱码。如果你需要一个"只有你选的人才能访问"的网站,这套东西做的正是这个。

- WhatsApp: [wa.me/60127972969](https://wa.me/60127972969)
- 邮箱: [me@hoelee.com](mailto:me@hoelee.com?subject=Onion%20service%20setup)
- 我还做什么: [hoelee.com](https://hoelee.com)
````

### 🇬🇧 英文版原文 (English Markdown Primary)
````markdown
---
title: "Hardening a Tor Onion Service: What Actually Matters"
description: "Out of curiosity I mirrored my portfolio and git server onto the darknet. Here's the hardening that held, the parts that silently broke, and whether it's worth offering to clients."
pubDate: 2025-12-10
updatedDate: 2026-09-09
category: devops
blueprint: self-hosted-infrastructure
blueprintLayer: L4-secops-cicd-dr
tags: [tor, docker, security, self-hosting, networking]
ogImage: /og/hardening-a-tor-onion-service.jpg
banner: /banners/hardening-a-tor-onion-service.jpg
---

I got curious about the darknet. Not the marketplaces, the boring half of it: people self-hosting things the way the internet worked before registrars and cloud dashboards, running software they control at an address they own.

The way I answer a curiosity like that is to build something, so the experiment became: mirror my own sites onto Tor. My interactive portfolio lives at [me.hoelee.com](https://me.hoelee.com), and my code lives on a Gitea instance at [git.hoelee.com](https://git.hoelee.com). Both now have darknet twins:

- `hoeleegitkcng572znkbpyffppyulsdwv3aurrzlk7y7vlhknogswoqd.onion` — the Gitea mirror
- `hoeleeaiwowgndbxswegtdzoeupz7lkkechtqmurbmnpvwa4k3vyuyid.onion` — the portfolio mirror

### If you've never opened a `.onion` before

These addresses don't work in a normal browser. You need Tor Browser, which routes your traffic through the Tor network. It's a five-minute setup:

1. Download it from **[torproject.org/download](https://www.torproject.org/download/)**.
2. **Verify the signature.** The download page links the `.asc` files, and this step matters — a tampered Tor Browser is the worst possible way to lose your anonymity. Fetch the Tor Browser signing key (the fingerprint is published on the download page and in the Tor Browser manual), then check the file:
   ```bash
   # import the signing key — see the .asc link on the download page
   gpg --verify tor-browser-*.tar.xz.asc tor-browser-*.tar.xz
   ```
3. Extract and run it. Windows and macOS get a normal installer; on Linux, extract and run `./start-tor-browser.desktop`.
4. **Click Connect.** The default settings are fine. If you're on a censored network, choose "Configure connection" and pick a bridge (Snowflake or obfs4).
5. Wait for "Connected" and the circuit display in the top-left corner. You're on Tor.

Now paste one of the addresses above into the URL bar, exactly as written. Two things to know:

- A v3 address is **56 characters** of base32 (`a–z`, `2–7`) plus `.onion`. There is no typo correction and no search suggestion — one wrong character gives you a dead address with no explanation.
- A 16-character address (v2) won't load at all. The v2 protocol was retired in 2021.

One thing that surprises first-timers: a brand-new onion can take **10–60 minutes** before the network recognises it. Tor has to publish a descriptor and get it accepted into the hash ring. An onion that doesn't load on the first try is usually just young, not broken.

While I was researching how to do this right, a lot of what I read online talked up the benefits of obfs4. What actually kept my setup safe had nothing to do with that reading.

So what does actually keep an onion service safe? I went through this properly when I set the mirrors up, and again months later when I went back to check on them. Some of the setup held up. Some of it had quietly broken. And a couple of things I believed about Docker turned out to be wrong in ways I could measure.

## Tor and onion services, in thirty seconds

Tor is an anonymity network: your traffic hops through three relays, and no single relay sees both who you are and where you're going. An onion service is the reverse direction, a server that lives *inside* the network. It gets a `.onion` address instead of a domain, visitors reach it only through Tor, and the server's real location never shows up in the connection.

Four properties come out of that, and a normal website simply can't offer them:

- **No public IP, no ports, no DNS.** Nothing for scanners to find. You can host it from a home connection behind CGNAT.
- **No middlemen.** No CDN, registrar, or platform in the path recording who visits.
- **The address itself is the access control.** People who have it can reach the site; everyone else can't even tell it exists. Add client keys on top when that isn't strict enough.
- **It keeps working when the clearnet doesn't.** If a domain gets blocked or seized, the onion is untouched, because no registry is involved.

What onion hosting suits: private file drops, portfolio and git mirrors, lawyer-to-counsel drafts, digital-goods delivery, any archive that should never appear in a search index. The people who want it are the ones for whom "nobody can even see that it exists" is the feature rather than a quirk: lawyers, auditors, freelancers sending deliverables, sources talking to journalists, and shops selling digital products over a channel their competitors can't scan.

The honest tradeoff is speed. Tor circuits are slower than a CDN, so you wouldn't put a marketing site on one. It's the right tool for the private half of the internet, not the public half.

## What actually protects the origin

Three things, and only the last one requires any work:

1. **The protocol.** Visitors never connect to your server directly. Your tor process dials out, registers the service with introduction points, and rendezvous happens inside the network. Nobody who visits gets your IP from the visit itself.
2. **Vanguards-lite.** Built into Tor since 0.4.7, this makes guard-discovery attacks (an attacker forcing circuits until they can observe your guard relay) far less practical. You get it by simply running a current Tor.
3. **Not leaking the origin through other channels.** The realistic way an onion host gets exposed is not traffic analysis. It's your own machine leaking: a clearnet service gets compromised, and the attacker just reads your onion keys off the disk. Mirroring a public site to an onion is deliberate, so content correlation doesn't bother me. What matters is that the machine holding the keys isn't also running a pile of exposed services.

## Things that quietly broke

These are the parts where re-checking my own server paid for itself.

**The app container had full internet access.** I ran a one-liner inside the container against a public IP echo service, and my home IP came back. There had been an iptables-based block for this, but the rules were gone. Host firewalls get flushed silently by interface changes, container manager restarts, platform updates. A block that nothing re-applies and nothing alarms on is not a block, it's a superstition. This convinced me to stop filtering the app's egress and remove it entirely instead (below).

**The host firewall was off.** INPUT and FORWARD policy ACCEPT, and a few dozen ports listening on all interfaces from the other services on the same machine. If any one of those gets owned, the onion keys on that disk belong to the attacker. People spend hours on Tor-specific hardening and skip this.

**The running tor didn't match its config file.** The torrc on disk said `SocksPort 0`. The process, up for days, was still listening on 127.0.0.1:9050. I had edited the file and never restarted the container. Impact here was small, since the listener was container-local. The lesson generalizes: the config file you wrote is a wish, what the process is actually doing is the truth.

**Tor was a security release behind.** Older than I'd want on a box holding sensitive keys, and my logs carried the warn-spam signatures from a relay-descriptor parsing bug the newer releases fixed. It stopped after the upgrade.

## The fixes that stayed fixed

### Zero egress, by construction

Both containers go on a Docker network with `internal: true`, and only the tor container gets a second NIC for reaching the Tor network.

```yaml
networks:
  service_net:
    driver: bridge
    internal: true        # no gateway, no masquerade, no route out
  egress:
    external: true        # ordinary bridge with internet
```

The app has no route to anything now. Not the internet, not the LAN, not even the host, because an internal network has no gateway at all. Nothing to flush, no boot task to remember, no way for it to silently decay. If the app gets compromised, the attacker gains a socket pointing at tor and nothing else. This is the one change I'd call non-negotiable for any app I run this way.

### Non-root, and the capability surprise

The image runs as non-root by default, and I wanted the app to keep listening on port 80. The standard advice is `cap_add: NET_BIND_SERVICE`. It didn't work. The container crash-looped with:

```
[FATAL] Server error: listen tcp 0.0.0.0:80: bind: permission denied
```

The reason surprised me enough that I measured it with a probe container: `grep Cap /proc/self/status` showed `CapBnd` with bit 10 (NET_BIND_SERVICE) set, and `CapEff` at zero. Docker grants capabilities to a non-root container only in the bounding set, not the effective set, and bind() checks the effective set. A non-root process running a binary without file capabilities gets an empty effective set, full stop. So the answer is boring: run on an unprivileged port inside.

```yaml
  app:
    user: "1000:1000"
    cap_drop: [ALL]
    security_opt: [no-new-privileges]
  tor:
    user: "100:101"
    cap_drop: [ALL]
    security_opt: [no-new-privileges]
```

One line on the tor side remaps the visit:

```
HiddenServicePort 80 app:8080
```

People still land on port 80 of the onion address. Only the internal port moved.

### Healthchecks that check the right thing

The tor image ships with a healthcheck that probes the SOCKS port. I had just turned SOCKS off, so healthy became unhealthy forever. Override it with the question you actually mean: is the process alive?

```yaml
healthcheck:
  test: ["CMD", "pgrep", "-x", "tor"]
```

The nginx trap was subtler. My first version used `wget --spider` against the root path. The site returned 404 (I hadn't uploaded content yet), wget exited non-zero, and the container got marked unhealthy. The check was testing the content, not the service. `nc -z 127.0.0.1 80` tests the port and nothing else.

One more thing: tor resolves its `HiddenServicePort` target at startup. If the app container isn't up yet, tor dies with "Unparseable address in hidden service port configuration" and crash-loops. I'd watched four failed starts in the old logs, unnoticed, because nothing was watching. `depends_on` in the compose file fixed the ordering.

### The ownership landmine

After generating new keys I copied the key material back through a file-share mount. The next containers crash-looped:

```
[warn] Could not open "/var/lib/tor/.../hs_ed25519_secret_key": Permission denied
```

Files written through the share are owned by the share user, not by the uid tor runs as. The fix is one command, runnable anywhere, including a throwaway container with the volume attached:

```bash
docker run --rm -v /path/libTor:/var/lib/tor alpine \
  sh -c "chown -R 100:101 /var/lib/tor && chmod -R 700 /var/lib/tor"
```

### Verify, don't believe

Every fix above ends with a test I can run myself. The egress one is my favorite, because the two probes together are convincing: the same wget that fails inside the app's network succeeds from tor's egress network.

```bash
# inside the app: with internal:true, even DNS should fail
wget -T 6 -qO- http://ipv4.icanhazip.com     # → "wget: bad address", exit 1

# same probe, on tor's egress network, as a control
wget -T 6 -qO- http://ipv4.icanhazip.com     # → <your IP>, exit 0
```

If you can't exec into a container, attach a throwaway probe container to the same network. It tests the network's properties, which is the thing you actually hardened.

## Would I offer this as a service?

I keep thinking about this, because the marginal cost is close to zero: the tor containers and their isolation are already running.

Most of my hosting clients want the opposite of an onion service. They want to be found on Google. Selling someone a website that only opens in Tor Browser means selling them secrecy they probably don't need, and it means supporting their visitors through installing Tor Browser.

But there is a real sliver of a market. Lawyers exchanging drafts, auditors, people delivering digital goods, anyone sharing an archive that should never show up in a search index. For those clients the pitch writes itself: no port forwards, no domain, no logs on some platform you don't control, just an address you physically hand to the people who should have it.

There's one trick that sells better than I expected, and it ties to a question everyone asks: can you choose how the address starts? v3 onion addresses are random, but only because the keys are. You can mine them: generate keypairs until the base32 address begins with the prefix you want. Every character costs a factor of 32 in work. Community mining tools on a modern GPU check addresses in the low millions per second, which makes an 8-character prefix a day-or-a-few-days job, 9 characters a patient weeks-long one, and 10 characters a serious multi-GPU commitment. A prefix that starts with the client's brand turns an unmemorable 56-character string into something they can verify is really yours, and in a niche where trust is the entire product, that's real value. I'd mine 8 happily, 9 for a paying client, and quote 10 with a straight face only if they're renting the GPUs.

So: as a bolt-on for a handful of specific clients, yes. As a product line, no. The market is too thin to build a funnel on, and the support burden doesn't shrink with volume. Privacy consulting with an onion attached, fine. Onion hosting as a web hosting tier, someone else's problem.

## What stuck

- **Run the tests, not the config file.** In-container probes and health states are the truth; the yaml is the intention.
- **Prefer mechanisms that can't silently unwind.** `internal: true` beats an iptables boot task every time.
- **Healthchecks are cheap. They caught a crash loop in minutes** where previously nothing watched for days.
- **The boring host firewall matters more than exotic Tor hardening.** Nobody de-anonymizes you with traffic analysis if they can just walk in through an open port.

## Want one of these?

I build and run these for clients: creating an onion site, hosting and maintaining it (updates, healthchecks, monitoring), and mining branded addresses when a client wants a name instead of gobbledygook. If you need a site that exists only for the people you choose, this is exactly what it does.

- WhatsApp: [wa.me/60127972969](https://wa.me/60127972969)
- Email: [me@hoelee.com](mailto:me@hoelee.com?subject=Onion%20service%20setup)
- What else I do: [hoelee.com](https://hoelee.com)
````

---