hoelee/hoelee-blog
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
050d73305d65dc0766f66b31a68a56adbf75144a
hoelee-blog/docs/consultant-review/batches/batch-02.md
T

242 KiB
Raw Blame History

博客文章顾问审阅资料包 — Batch-02

批次编号：Batch-02
包含文章数：11 篇
本批次总字符量：179,502 字（严格限制 < 200,000 字）
模式：BILINGUAL
生成时间：2026-10-27 04:23:31

外部顾问审阅前置须知与工程背景规范

本文档目标：本批次包含 blog.hoelee.com 的待审阅技术文章全文。请外部顾问在审阅正文与技术主张时，以此规范为最高评估准则。

1. 站点定位与双重业务目的

  • 受众 A（技术招聘官 / 资深技术主管）：寻找具备 Senior 全栈 + DevOps + AI 智能体闭环能力的架构工程师。文章定位为**“不可伪造的工程实操交付证据（Proof of Expertise）”**，要求代码真实、日志可查、有架构权衡与技术深度。
  • 受众 B（中小企业主 SME / 商业客户）：寻找能提供系统研发、私有化部署、24/7 AI 获客系统、高可用基础设施的技术顾问。文章需清晰阐述业务痛点、商业 ROI、上线效率与交付保障。

2. 核心人设铁律与工程硬约束（必须严格遵守）

  1. 彻底杜绝业余爱好者/玩具口吻：
    • ❌ 绝对禁用词：homelab, home lab, 家用, 小主机, 玩具, 折腾。
    • ✅ 标准替代词：the infrastructure I manage（我管理的私有基础设施）、self-hosted production node（自托管生产节点）、on-premise host（私有化服务器）。
  2. 严禁捏造虚假指标与性能数字：
    • 所有 benchmark 数据必须有明确的测试基准与单设备限制。
    • 严格区分“耗时减少比例”与“吞吐加速倍数”（例如耗时从 352s 降至 218s 是耗时减少 38.0%，吞吐提升 1.61×，严禁偷换概念混淆）。
    • 时间戳精度与物理帧对齐：视频帧率上限（如 30 fps 单帧 33.3ms）不可妄称“微秒级精确”。
  3. 去模板化叙事与真实物理锚点：
    • 拒绝千篇一律的 AI 模板味。文章采用三大叙事架构：① 故障排查谜案（Symptom → False Lead → Kernel Evidence → Fix）；② 商业交付复盘（Client Bottleneck → Trade-offs → State Machine → ROI）；③ 架构演进对比（V1 Naive → Concurrency Crash → V2 Redesign）。
    • 每篇文章必须包含至少一个物理真实世界锚点（主机真实硬件规格、生产维护窗口、真实业务场景）。
  4. 代码与配置展示规范：
    • 超过 20 行的配置或日志采用 HTML <details><summary> 原生折叠，<summary> 必须精准标注文档名与行数。

3. 顾问重点开刀维度

  • 事实与数字可核性：技术判断与性能数据是否自洽？有无过度夸大？
  • 人设与语调纯洁性：有无遗漏的 hobbyist/homelab 口吻？
  • 双轨转化引导有效性：是否有清晰的业务价值提炼与行动召唤（CTA）？
  • 代码与生产安全性：有无未脱敏的凭据、绝对内部私密路径或不安全的权限配置？

本批次文章目录索引

  1. [devops] 空间回收没卡住，它只是在等明天早上 09:29 (space-reclamation-wasnt-stuck-it-was-waiting) — ~8,795 字符
  2. [devops] Synology 电子表格 API 是一个容器，不是一个 API 端点 (synology-spreadsheet-api-is-a-container) — ~25,049 字符
  3. [devops] 我差点把这颗 SSD 送修——真正坏的是 SATA 数据线 (that-dying-ssd-was-just-a-bad-sata-cable) — ~9,910 字符
  4. [devops] 换了不同的硬盘，同样的损坏又来了——真凶是 TRIM，不是 SSD (the-cause-was-trim-not-the-ssds) — ~17,933 字符
  5. [devops] 上游被删了，然后带着一个重写版回来 (the-upstream-was-deleted-then-came-back-rewritten) — ~16,678 字符
  6. [devops] Unraid 停止阵列时卡死的真相：Swapfile 锁住挂载点 — 附修复方案 (unraid-stop-array-hangs-on-swapfile) — ~16,184 字符
  7. [devops] 用 CDP 自动化上传百兆视频到 YouTube：跨越 DevTools 内存墙的 Docker 归档法 (uploading-large-videos-to-youtube-via-cdp) — ~16,824 字符
  8. [devops] 如何在押注前验证一个开源依赖：我用的六项检查 (vetting-an-open-source-dependency-before-you-bet-on-it) — ~12,623 字符
  9. [devops] SMART 显示健康，RAID 却在悄悄损坏数据 (when-smart-says-healthy-but-your-raid-is-corrupting-data) — ~17,708 字符
  10. [devops] 为什么容器里的 Chrome 会忘记标签页 —— 以及我是怎么修好的 (why-chrome-forgets-its-tabs-in-a-container) — ~23,650 字符
  11. [devops] Telegram 机器人通知为什么发不出去：IPv6、DNS 和一个折腾了一整天的 400 (why-telegram-bot-notifications-die) — ~14,148 字符

文章 01 / 11：空间回收没卡住，它只是在等明天早上 09:29

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "空间回收没卡住，它只是在等明天早上 09:29"
description: "存储管理员那张卡整天写着 running，可用空间一动不动。其实 121 GiB 已经分两步回来了——然后下午又被写掉了 47 GiB。"
pubDate: 2026-10-14
updatedDate: 2026-10-16
category: devops
tags: ["synology", "btrfs", "snapshots", "storage", "nas"]
ogImage: /og/zh-space-reclamation-wasnt-stuck-it-was-waiting.jpg
banner: /banners/space-reclamation-wasnt-stuck-it-was-waiting.jpg
draft: false
---

## 我问错了问题

我清掉几个文件夹，在 NAS 上点了空间回收，然后等。几个小时后，存储管理员那张卡还写着 **running**，可用空间看起来跟早上一样。我的问题是「它为什么什么都没回收」。

**它回收过了——两次**，而且其中一次比我删掉的东西大得多。测量回答的是另一个问题，而这才是这件事里最有意思的地方。

## 在相信任何东西之前，先给可用空间上定时采样

我没有去看那张进度卡，而是给 `volume1` 的可用空间做了当天每五分钟一次的采样：

| 时间 | 可用 | 事件 |
|---|---|---|
| 08:59 → 09:24 | 1134.1 GiB | 平——什么都没在回收 |
| **09:29** | **+69.0 GiB** | Class 快照 `…10.14-09.29.02` 轮转 |
| **09:34** | **+46.8 GiB** | Video 快照 `…10.14-09.30.35` 轮转 |
| **15:28:45** | — | DSM 日志：`Start to empty all recycle bins.` |
| **15:34** | **+5.0 GiB** | 清回收站结束——里面只有 5 GiB |
| 16:35 → 17:39 | **−39 GiB** | 新数据写入（一阵爆发） |
| 15:00 → 20:44 | **−47.2 GiB** | 净值：这个卷那天下午**丢**了空间 |

两次回收，合计约 121 GiB；然后 47 GiB 的新写入把下午吃掉了。「什么都没回来」是我这边的读法错误：我拿早上和晚上比，而晚上里面多了一整天的写入。

我特意手动清空的回收站是 **5 GiB**；我压根没想到的快照是 **115.8 GiB**。这个比例就是全部故事。

## 为什么那张卡写着 running，却什么都不代表

```text
$ synostgreclaim --get-all-status --verbose
{"state":"running","num_subvolume_remain":1,"now_runnable":true}
   schedule_table: 168 × 1
   run_time_option: run_all_time
```

`state: running` 加上 168 个 1 的调度表和 `run_all_time`，意思是这个任务被配置成**按设计 24×7 都处于 running**。它是调度状态，不是进度指示器——所以在某天完全无事可排的日子，这张卡照样这么写。而当时那台机器上**没有** `synostgreclaim` 进程、没有 btrfs balance、md2/dm-3 的 I/O 基本为零。那一刻没有任何东西在被回收，也没有任何东西卡住。

然后是日志，而 DSM 上它们大多是红鲱鱼：

- 回收任务本身什么都不记；
- `/var/log/space_operation.log` 是 LUKS 和卷列表的噪音；
- `/var/log/synorecycle.log` 是 `synounlinkd` 的路径噪音；
- 当天的 Log Center 里只有我自己那条清回收站的记录，外加几条 UPS 断电/来电的提示。

一个永远为真的 UI 状态字段，加上三个永远不会给出答案的日志文件：唯一剩下的仪器，就是**随时间变化的可用空间**。

## 没人告诉你的那个机制

空间回收释放的是**已删除的 btrfs 子卷**。它释放不了仍然被快照钉住的数据。那两个共享文件夹的保留策略是 **retention 1**（各留一代），所以**我删掉的任何东西，都要等到第二天的轮转、大约 09:29 / 09:30 才会变成可用空间**，一次跳几十 GB，然后安静 24 小时。

那两次跳变正是这个。它也就意味着，对于「我删了 50 GB 却什么都没发生」这个问题，在这台机器上的诚实答案是：「等明天早上再看」——而如果删掉的数据本来就在某个快照里面，答案是「等那个快照过期，之前不会释放」。

## 如果重来一次

1. **采样那个数字，不要采样状态。** 每五分钟量一次可用空间，比任何任务卡都更能回答「现在有没有事情在发生」，而且它同时让「回长」变得可见。
2. **读调度字段，不读状态字段。** `run_all_time` 解释了那个永久 running，而且不需要任何 bug 存在。
3. **搞清楚每个共享文件夹的保留策略。** 它是决定「删除何时变成可用空间」的那口钟；不知道它，每一次删除都会看起来坏掉一整天。
4. **把两本账分开。** 回收站（5 GiB）和快照（115.8 GiB）行为完全不同，也是两套心理模型。
5. **重新检查分母。** 这个卷平时就在以约 1.5 GiB/时 的速度被正常写入吃掉；回收回来 121 GiB、当天又花掉 47 GiB，只要你只比同一个数字的两个快照，这两件事看起来完全一样。

## 测量真正告诉我的，是这台机器的趋势

我本来是去修一个卡住的任务，结果带回来的是一条存储趋势：

```text
volume1   42 TB   已用 98%   剩 1.2 TB   近来约 -1.5 GiB/时（8 天 -344 GiB）
volume2   16 TB   已用 100%  剩 70 GB    约 -19 GiB / 2 天
健康      两个卷 btrfs corruption_errs 均为 0
          所有 md 阵列 [UUUU] / [UU]，没有在跑 balance
          read/write_io_errs 与九月基线一致
```

所以有用的结论不是「回收是好的」。而是：**相对于写入速率，回收只是一个舍入误差**；在保留策略为 1 的前提下，删除并不是我以为的那根杠杆；而该拿来做告警的是趋势，不是那张任务卡。这跟[《你的磁盘快满告警在撒谎》](/posts/your-disk-full-alert-is-lying/)是同一课，只是从另一个方向来的：对**后果**告警（随时间变化的可用字节），因为机制自己的状态不是证据。

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "Space Reclamation Wasn't Stuck. It Was Waiting for Tomorrow at 09:29."
description: "The Storage Manager card said 'running' all day and free space hadn't moved. 121 GiB had come back in two steps — and the afternoon ate 47 GiB of it."
pubDate: 2026-10-14
updatedDate: 2026-10-16
category: devops
tags: ["synology", "btrfs", "snapshots", "storage", "nas"]
ogImage: /og/space-reclamation-wasnt-stuck-it-was-waiting.jpg
banner: /banners/space-reclamation-wasnt-stuck-it-was-waiting.jpg
draft: false
---

## The question I asked was the wrong question

I emptied a few folders, kicked off Space Reclamation on the NAS, and
waited. Hours later the Storage Manager card still read **running**, and
free space looked the way it had looked in the morning. My question was "why
didn't it reclaim anything".

It had reclaimed — twice — and one of those reclaims was larger than
anything I had deleted. The measurement answered a different question than
the one I asked, which is the interesting part.

## Sample free space on a timer before believing anything

I sampled `volume1` free space every five minutes for the day instead of
looking at a progress card:

| time | free | event |
|---|---|---|
| 08:59 → 09:24 | 1134.1 GiB | flat — nothing reclaiming |
| **09:29** | **+69.0 GiB** | Class snapshot `…10.14-09.29.02` rotated |
| **09:34** | **+46.8 GiB** | Video snapshot `…10.14-09.30.35` rotated |
| **15:28:45** | — | DSM log: `Start to empty all recycle bins.` |
| **15:34** | **+5.0 GiB** | recycle-bin empty finished — only 5 GiB was in there |
| 16:35 → 17:39 | **−39 GiB** | new data written (a burst) |
| 15:00 → 20:44 | **−47.2 GiB** | net: the volume *lost* space that afternoon |

Two reclaims, ~121 GiB in total, and then 47 GiB of new writes swallowed the
afternoon. "Nothing came back" was a reading error on my side: I compared
the morning to the evening, and the evening had a day of writing in it.

The recycle bin — the thing I had bothered to empty by hand — was **5 GiB**.
The snapshots, which I had not thought about at all, were **115.8 GiB**.
That ratio is the whole story.

## Why the card said "running" and meant nothing

```text
$ synostgreclaim --get-all-status --verbose
{"state":"running","num_subvolume_remain":1,"now_runnable":true}
   schedule_table: 168 × 1
   run_time_option: run_all_time
```

`state: running` with a schedule table of 168 ones and `run_all_time` means
the task is configured to be "running" **24×7, by design**. It is a
schedule, not a progress indicator — so the card will read the same thing on
a day when nothing is scheduled to happen at all. Meanwhile the machine had
no `synostgreclaim` process, no btrfs balance, and md2/dm-3 I/O of roughly
zero. Nothing was being reclaimed at that moment, and nothing was stuck
either.

Then the logs, which are mostly red herrings on DSM:

- the reclaim task itself logs nothing;
- `/var/log/space_operation.log` is LUKS and volume-list spam;
- `/var/log/synorecycle.log` is `synounlinkd` path noise;
- Log Center for the day contained only my own recycle-bin line and a few
  UPS on-battery/back-online blips.

A UI state field that is always true and three log files that never contain
the answer: the only instrument left is free space over time.

## The mechanism nobody tells you about

Space Reclamation frees **deleted btrfs subvolumes**. It cannot free data
that a snapshot is still pinning. Those two shares keep **retention 1** —
one generation each — so anything I delete becomes free space only at the
**next day's rotation, around 09:29 / 09:30**, in a step of tens of
gigabytes, and then it is silent for 24 hours.

That is exactly what the two jumps were. It also means the honest answer to
"I deleted 50 GB and nothing happened" on this box is: "wait for tomorrow
morning, then check again" — and if the deleted data lived inside a
snapshot, the answer is "it frees when that snapshot expires, and not
before".

## What I would do differently

1. **Sample the number, not the status.** A five-minute timer on free space
   answers "is anything happening" in a way no task card can, and it made
   the regrowth visible at the same time.
2. **Read the schedule field, not the state field.** `run_all_time` explains
   a permanent "running" without any bug existing.
3. **Know each share's retention.** It is the clock that decides when a
   delete becomes free space; without it, every deletion looks broken for a
   day.
4. **Separate the two ledgers.** Recycle bin (5 GiB) and snapshots (115.8
   GiB) behave nothing alike and are two different mental models.
5. **Re-check the denominator.** The volume was losing ~1.5 GiB/h to normal
   writes; a reclaim that returns 121 GiB and a day that consumes 47 GiB
   look identical if you only compare two snapshots of the same number.

## What the measurement actually says about the machine

I went in expecting to fix a stuck job and came out with a storage trend
instead:

```text
volume1   42 TB   98% used   1.2 TB free   ~-1.5 GiB/h lately  (-344 GiB over 8 days)
volume2   16 TB  100% used    70 GB free   ~-19 GiB / 2 days
health    btrfs corruption_errs 0 on both volumes
          all md arrays [UUUU] / [UU], no balance running
          read/write_io_errs identical to the September baseline
```

So the useful conclusion is not "the reclaim works". It is that reclaim is a
rounding error next to the write rate, deletion is not the lever I thought
it was while retention is 1, and the trend — not the task card — is what I
should be alerting on. That is the same lesson as
[Your Disk-Full Alert Is Lying](/posts/your-disk-full-alert-is-lying/),
arriving from the other direction: alert on the consequence (bytes free over
time), because the mechanism's own status is not evidence.

文章 02 / 11：Synology 电子表格 API 是一个容器，不是一个 API 端点

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "Synology 电子表格 API 是一个容器，不是一个 API 端点"
description: "如何在 Synology Office 电子表格上做自动化：官方的 spreadsheet-api 容器，以及四个陷阱——一个假的\"未安装\"、凭据正确却报 401、永远无法通过认证的 2FA，还有一堵权限墙。"
pubDate: 2025-08-20
updatedDate: 2026-09-26
category: devops
blueprint: sme-digital-suite
blueprintLayer: L3-entitlement-backoffice
tags: [synology, dsm, docker, portainer, rest-api, spreadsheet, debugging, self-hosting]
ogImage: /og/zh-synology-spreadsheet-api-is-a-container.jpg
banner: /banners/synology-spreadsheet-api-is-a-container.jpg
draft: false
---

我想要的东西很普通：从我 NAS 上的电子表格读一个单元格，把一个单元格写回去，然后从另一端得到一张图表——不用打开 Excel，也不用把公司的数据送到某个云 API。

我跑着一台 Synology DS1821+，装好了 Synology Office（`Spreadsheet` 套件），所以这看起来是个已解决的问题。它确实是个已解决的问题。只是要穿过四个相互矛盾的假信号才能找到真正的答案，而且其中有一个我中途得出的结论错得相当彻底，值得写下来。

## 为什么这件事值得写

如果你自托管，总有一天你需要一个服务去和同一台机器上的另一个服务对话。这篇帖子里的故障模式并不只属于 Synology——它们就是"厂商把东西寄来了，但不在你会去找的地方"这个问题的通用形态：

1. **API 清单里缺一条，不等于这个 API 不存在。** 我枚举了 1,515 个 API，得出结论说这个功能不存在。其实它是一个容器。
2. **我信任的两个诊断工具在撒谎。** `synopkg` 把一个正在运行的套件报成"未开启"，`ps` 给我看了一台空荡荡、却正在服务生产流量的机器。
3. **范围最精准的凭据失败，看起来却像是你的错。** 密码验证过正确还返回 `401`，几乎总是关于你把请求发到了*哪里*，而不是你发了*什么*。
4. **权限模型在认证之后才咬人。** 你可以完美登录，却仍然对每个文件都收到 `403`，原因在于文件所在的位置。

如果以上任何一条听起来耳熟，这篇帖子的后半部分就是能跑通的完整配置。

## 文档说了什么，以及它为什么把我带偏

Synology 宣传"Office Suite APIs"——面向 Drive、Spreadsheet、MailPlus 和 Calendar 的 REST API。营销页面上逐字列出了我想要的正是这些：

- "读取并写入某个范围内的单元格样式。"
- "新增、重命名、删除表格，并把工作表导出为 CSV 文件。"
- "创建、读取、导出和删除电子表格。执行批量更新……"

文档本身藏在 Synology 账户登录后面——这是很常见的事，不是什么丑闻——但它意味着搜索引擎先找到的是承诺，而不是契约。

于是我自己在 NAS 上找。DSM 暴露了一个网关 API 目录，你可以直接问它存在什么：

```bash
curl -s "http://192.168.1.1:8081/webapi/query.cgi?api=SYNO.API.Info&query=all" \
  | python -c "import json,sys; d=json.load(sys.stdin)['data']; print(len(d), 'APIs')"
```

```
1515 APIs
```

一千五百一十五个。其中出现了 `SYNO.Office.*`——但只有快照和一个"最近使用的公式"列表：

```
SYNO.Office.Sheet.Snapshot
SYNO.Office.Sheet.Snapshot.History
SYNO.Office.Sheet.MruFc
```

没有 values 端点。没有 styles 端点。没有写入端点。我把整个目录过滤了一遍，找任何提到 sheet 或 cell 的东西，结果一无所获，然后我得出结论——并且告诉了托我建这个东西的人——**这台 NAS 没有单元格级的电子表格 API。**

那个结论是错的。我想把错在哪说清楚，因为那个推理错误才是可以复用的部分：**我把一个发现面当成了全部。** 网关目录列出的是 DSM 自己的 Web 网关服务的 API。Office Suite API 不由那个网关提供。它是一个独立服务，作为一个独立产物发布，因此对我跑的那次查询不可见。

## 陷阱一：把正在运行的服务报成未安装的工具

在找到真正的答案之前，我花了不少时间确信这个软件根本没在跑。两条命令让这个说法看起来很成立。

第一条是 Synology 自己的套件状态查询：

```bash
sudo synopkg is_onoff SynologyDrive
sudo synopkg is_onoff Spreadsheet
```

```
SynologyDrive isn't turned on, [262]
Spreadsheet isn't turned on, [262]
```

与此同时两个套件都明摆着在服务请求。`SynologyDrive` 自己的状态文件里是 `status=enabled`，Office 的守护进程也在进程表里——一个任务守护进程、一个跑在 `office` 用户账户下的连接池，还有网关 worker。在这套 DSM 版本上，`is_onoff` 根本不是一个可靠的答案。

第二条命令更糟，因为那是我自己的失误：

```bash
ps w | grep -iE "office|drive|pgbouncer"
```

```
(no output)
```

那看起来像一台死机。它不是。**在 DSM 上，不带 `sudo` 的 `ps` 只会显示你自己的进程。** 用 root 重跑一遍，画面就反转了：

```bash
sudo ps aux | grep -E "office|pgbouncer|synoscgi" | grep -v grep
```

它们全都在。我用两条根本不可能让我看到真相的命令，"证明"了两次这套栈是挂的：

> 如果要从这篇帖子里带走一个习惯：当一个诊断工具说"什么都没有在跑"时，先检查这个诊断工具到底有没有权限看到任何东西。

## 陷阱二：没有启动错误信息的容器

真正的答案，在我停止相信自己亲手得出的证据之后，出现在 Docker Hub 上：`synology/spreadsheet-api`。它不是 Office 套件的一部分，也不是网关端点。它自己的描述说得很清楚：

> "Spreadsheet API 不是 Office 套件的一部分。这个镜像在客户端和 DSM 之间提供代理服务。每个 worker 专属于单个用户和单个电子表格。"

兼容性表格把镜像标签对到 Office 版本上，这是第一件要查的事，因为配对不是可选的：

| Docker 镜像标签 | 所需的 Synology Office 版本 |
|---|---|
| `3.4.1` | 3.7.0 或更高 |
| `3.3.2` | 更老的版本 |

Synology 还警告不要把镜像和 Office 放在同一台机器上，因为每个 worker 会把整个电子表格加载进内存。我仍然把它跑在同一个 DSM 上，并加了内存上限——这是有意的取舍，不是推荐做法。

然后我跑了它，它立刻死掉，吐出一段这个：

```
/app/dist/index.js:424
}`;var ot=UD(function(){return Ht($,qe+"return "+Ee).apply(e,H)});...
```

一段压缩过的 JavaScript——最没用的一类错误信息。真相一旦找到，其实很平常：**`AUTH_SECRET` 是必填项，而且没有默认值。** 它给会话令牌签名。没有它，服务就起不来，而且没法告诉你为什么。

要可靠地读出容器的真实要求，而不是听它的启动噪音，直接向打包产物要答案：

```bash
docker run --rm --entrypoint grep synology/spreadsheet-api:3.4.1 \
  -ohE '.{0,80}process\.env\.PORT.{0,80}' /app/dist/index.js
```

```js
serverPort: parseInt(process.env.PORT) || 3e3,
serverHost: process.env.HOST || "0.0.0.0",
workerTimeout: parseInt(process.env.WORKER_TIMEOUT) || 600*1e3,
```

一行就给出配置契约：`AUTH_SECRET` 给令牌签名，`PORT` 默认 `3000`，`HOST` 默认 `0.0.0.0`，worker 十分钟后回收。它还自带一份 OpenAPI 规范和 Swagger UI，这个我后面会再说到。

## 陷阱三：密码被证明正确，却收到 401

容器跑起来之后，认证成了下一堵墙。这是请求，这是它返回的东西：

```bash
curl -s -X POST http://192.168.1.1:8791/spreadsheets/authorize \
  -H 'Content-Type: application/json' \
  -d '{"username":"<service-account>","password":"<password>","host":"192.168.1.1:5001","protocol":"https"}'
```

```json
{"error":"Unauthorized"}
```

`401 Unauthorized`，而我用的正是刚刚登录过 DSM 的密码。最自然的解读——密码错了、账户错了——是一条死路，而我真走了进去：重新敲一遍密码，在 DSM 登录页试那个账户（能登录），把 HTTPS 换成 HTTP 端口（同样 401）。

真正的问题出在 `host` 字段。这个服务反过来问*你*该对着哪个 DSM 做认证，然后自己跟那个 host 做一次 TLS 握手。我给它的是一个裸 IP 地址，而用那个名字出示的证书和地址不匹配，于是握手失败——代理把一个失败的握手报成 `Unauthorized`，跟密码错误一模一样。

换成一个持有有效证书的主机名，立刻就修好了：

```bash
curl -s -X POST http://192.168.1.1:8791/spreadsheets/authorize \
  -H 'Content-Type: application/json' \
  -d '{"username":"<service-account>","password":"<password>","host":"cloud.example.com","protocol":"https"}'
```

```json
{"token":"eyJhbGciOi...","host":"cloud.example.com"}
```

> **如果你的 Synology API 登录返回 401 而你又确定密码是对的，去查 `host`：它必须是一个容器能接受其证书的名字。一个裸的局域网 IP 不行。**

它返回的令牌是 JWT，有效期 28 天，绑定一个 DSM 会话——所以 DSM 重启或账户被强制登出，它就会失效。集成中途突然冒出来的 `401`，当作"重新授权"来对待，而不是"凭据变了"。

## 陷阱四：2FA 永远无法认证

这个是如果你事先不知道就会浪费最多时间的一个，所以直说：**登录 schema 里没有一次性验证码字段。**

```yaml
AuthorizationBody:
  properties:
    username: {type: string}
    password: {type: string}
    host:     {type: string}
    protocol: {type: string}
```

没有 OTP，没有应用密码，没有令牌交换。如果账户开了两步验证，这个 API 会永远返回 `401`——密码正确也一样，换哪个 host 都一样。唯一可行的安排是开一个关掉 2FA 的专用服务账户，权限只限定在它需要的文件夹——这本来就是更好的做法，也是我一开始就该做的，而不是先用我自己的管理员账户去试。

## 进得来之后：权限陷阱

认证不等于授权。等我的服务账户能登录了，针对我真正关心的那个电子表格的每次调用都返回：

```json
{"statusCode":403,"code":"403","error":"Forbidden","message":"Permission denied"}
```

仔细读一下，它信息量很大：`403 Permission denied` 意味着**文件存在，但你无权拿到。** 一个错误的 ID 会返回别的东西：

```json
{"statusCode":404,"code":"404","error":"Not Found","message":"Spreadsheet not found"}
```

两种不同的失败，两种不同的修法，把两者混为一谈就要白花一个小时。我的问题是位置问题：那个电子表格建在 Synology Drive 的个人 **My Drive** 里，在磁盘上是 `/volume1/homes/<user>/Drive/Document/…`。个人主目录别的 DSM 账户够不着——权限设置不行，共享文件夹的技巧也不行，因为它根本不在任何共享文件夹里。修法是把这个文件移进一个共享文件夹并在那里授予读写权限，或者用 Drive 把那个特定文件分享给服务账户并授予编辑权。

## 修复：一整套能跑通的配置

容器，配上一个真正的密钥和内存上限：

```yaml
services:
  spreadsheet-api:
    image: synology/spreadsheet-api:3.4.1
    container_name: spreadsheet-api
    restart: always
    environment:
      AUTH_SECRET: "<long-random-string>"
      PORT: "3000"
      HOST: "0.0.0.0"
    ports:
      - "8791:3000"
    mem_limit: 2g
```

一条只属于 DSM 的注意事项：DSM 上任何 compose 文件都别写 `cpus:`。它的内核没有 CPU CFS 调度器，这个键会被静默忽略——你得到的是有 CPU 限制的错觉，其实没有任何限制。

然后 API 就能用了，而且形态很讨喜。登录，创建或指定一个电子表格，读写范围：

```bash
TOKEN=$(curl -s -X POST http://192.168.1.1:8791/spreadsheets/authorize \
  -H 'Content-Type: application/json' \
  -d '{"username":"<acct>","password":"<pw>","host":"cloud.example.com","protocol":"https"}' \
  | python -c "import json,sys; print(json.load(sys.stdin)['token'])")

# read a range (sheet-qualified A1 notation)
curl -s -H "Authorization: Bearer ***" \
  "http://192.168.1.1:8791/spreadsheets/<id>/values/Sheet1!A1:C4"

# write a range, formulas included
curl -s -X PUT -H "Authorization: Bearer ***" -H 'Content-Type: application/json' \
  -d '{"values": [["Item","Qty"],["Widget",12],["Gadget",7],["Total","=SUM(B2:B3)"]]}' \
  "http://192.168.1.1:8791/spreadsheets/<id>/values/Sheet1!A1:B4"
```

两个我没想到的细节，都是对我有利的：

- **电子表格 ID 来自 Office 的 URL。** 一个位于 `…/oo/r/<spreadsheetId>` 的电子表格——包括分享链接转到的也正是这个路径——把它那一段放进 API 调用就能寻址。
- **公式是 Office 算的，不是你算的。** 把表格导出成 CSV 返回的是 `=SUM(B2:B3)` 和 `=SUMPRODUCT(B2:B3,C2:C3)` 的*求值后*结果——这些公式我从没在客户端算过。这正是用这个 API 而不是自己解析文件的全部理由：你继承了 Synology 自己的计算引擎。

有一个值得说出来的真实缺口：**没有 list 端点。** 规范声明了一个 "Statistics" 标签，却没有为它提供任何路由（`/statistics` → `404 Route not found`），也没有别的东西能枚举电子表格。你必须已经从文档的 URL 知道那个 ID。对你有意搭好的自动化来说没问题；如果你想浏览，那就没用。

## 从另一端得到一张图表

读单元格只是半件事；最初的要求里还有图表。同一个 API 能直接给你一份真正的 workbook：

```bash
curl -s -H "Authorization: Bearer ***" \
  "http://192.168.1.1:8791/spreadsheets/<id>/xlsx" -o export.xlsx
```

那会返回一个合法的 `.xlsx`——`PK` 魔数，什么都能打开。接着我自己的服务（一个小的 FastAPI 容器，里面有 LibreOffice 负责公式求值，matplotlib 负责渲染）读取范围并生成 PNG 图表和仪表盘，整条管线端到端跑通，不需要 Excel，也不需要任何云依赖：

```
Synology Office  →  spreadsheet-api  →  .xlsx export  →  chart renderer  →  PNG
```

## 重来一次我会怎么做

1. **在断定某个 Synology 功能不存在之前，先去查 Docker Hub。** 我花了不少真实时间，从一个根本不可能包含答案的 API 清单去证明一个反面结论。`synology/*` 在 Docker Hub 上，一次搜索就到。
2. **一开始就用一个关掉 2FA 的专用服务账户**，权限限定在单个共享文件夹。我先用管理员账户试，所以同一小时内同时撞上了 2FA 墙和 `403` 墙。
3. **当一个诊断工具说"什么都没有在跑"，检查它的权限。** DSM 上不带 root 的 `ps` 不是进程列表，它是你自己的进程列表，它会高高兴兴地让你得出机器已死的结论。
4. **读容器本身，别只读它的文档。** 厂商自己的运行命令在镜像页面上，但精确的默认值（`PORT` 3000，worker 超时 600 秒）和必填的 `AUTH_SECRET` 来自 grep 打包产物——一条命令，不用猜。

## 结果

在 NAS 上端到端验证：登录、创建、写单元格、读回单元格、CSV 导出和 `.xlsx` 导出全部返回 `200`，CSV 导出证明了 Office 引擎在服务端求值了 `=SUM`/`=SUMPRODUCT`。导出的 workbook 喂给图表服务，把同一份数据渲染成 PNG。四个陷阱记录在案，一个错误结论得到纠正，大约一天的活——现在我自己的硬件上有一份脚本可以驱动的电子表格。

---

*我在马来西亚为小企业做这类自托管集成——电子表格和 NAS 自动化、仪表盘，以及把你的数据留在你自己硬件上的内部工具。如果你需要这类东西，[WhatsApp 联系我](https://wa.me/60127972969) 或 [给我发邮件](mailto:[email protected]?subject=Self-hosted%20spreadsheet%20automation)——也可以看看我在 [hoelee.com](https://hoelee.com) 还做些什么。*

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "The Synology Spreadsheet API Is a Container, Not an API Endpoint"
description: "How to automate Synology Office spreadsheets: the official spreadsheet-api container, and the four traps — a fake 'not installed', a 401 with correct credentials, 2FA that can never authenticate, and a permission wall."
pubDate: 2025-08-20
updatedDate: 2026-09-26
category: devops
blueprint: sme-digital-suite
blueprintLayer: L3-entitlement-backoffice
tags: [synology, dsm, docker, portainer, rest-api, spreadsheet, debugging, self-hosting]
ogImage: /og/synology-spreadsheet-api-is-a-container.jpg
banner: /banners/synology-spreadsheet-api-is-a-container.jpg
draft: false
---

I wanted something ordinary: read a cell from a spreadsheet on my NAS, write a cell back, and have a chart come out the other end — without opening Excel, and without shipping my company's numbers to a cloud API.

I run a Synology DS1821+ with Synology Office (the `Spreadsheet` package) installed, so this looked like a solved problem. It is a solved problem. It just took four separate false signals to find the actual solution, and one of my conclusions along the way was flatly wrong in a way worth writing down.

## Why this matters

If you self-host, you will eventually need a service to talk to another service on the same box. The failure modes in this post are not Synology-specific — they are the general shape of "the vendor ships the thing, but not where you'd look for it":

1. **A missing entry in an API list is not proof the API doesn't exist.** I enumerated 1,515 APIs and concluded the feature was absent. It was a container.
2. **Two of the diagnostics I trusted were lying.** `synopkg` reported a running package as "not turned on", and `ps` showed me an empty machine that was serving production traffic.
3. **The best-scoped credential failure looks like your fault.** A `401` with a verified-correct password is almost always about *where* you sent the request, not *what* you sent.
4. **The permission model bites after authentication.** You can log in perfectly and still be told `403` on every file, because of where the file lives.

If any of that sounds familiar, the second half of this post is the working setup.

## What the docs say, and why that sent me the wrong way

Synology advertises "Office Suite APIs" — REST APIs for Drive, Spreadsheet, MailPlus and Calendar. The marketing page lists exactly what I wanted, verbatim:

- "Read and write cell styles within a range."
- "Add, rename, delete, and export sheets as CSV files."
- "Create, retrieve, export, and delete spreadsheet. Perform batch updates…"

The documentation itself sits behind a Synology Account login, which is a normal thing to hit and not a scandal — but it means the first thing a search engine finds is the promise, not the contract.

So I went looking on my own NAS. DSM exposes a gateway API catalogue, and you can just ask it what exists:

```bash
curl -s "http://192.168.1.1:8081/webapi/query.cgi?api=SYNO.API.Info&query=all" \
  | python -c "import json,sys; d=json.load(sys.stdin)['data']; print(len(d), 'APIs')"
```

```
1515 APIs
```

Fifteen hundred and fifteen. Among them, `SYNO.Office.*` appears — but only for snapshots and a "recently used formulas" list:

```
SYNO.Office.Sheet.Snapshot
SYNO.Office.Sheet.Snapshot.History
SYNO.Office.Sheet.MruFc
```

No values endpoint. No styles endpoint. No write endpoint. I filtered the whole catalogue for anything mentioning sheets or cells and came up empty, and I concluded — and told the person I was building this for — that **the NAS had no cell-level spreadsheet API.**

That conclusion was wrong. I want to be precise about why it was wrong, because the reasoning error is the reusable part: **I treated one discovery surface as exhaustive.** The gateway catalogue lists APIs served by DSM's own web gateway. The Office Suite API is not served by that gateway. It is a separate service, published as a separate artifact, and therefore invisible to the query I ran.

## Trap 1: the tool that says a running service isn't installed

Before finding the real answer, I spent a while convinced the software wasn't even running. Two commands made that look true.

The first was Synology's own package-state query:

```bash
sudo synopkg is_onoff SynologyDrive
sudo synopkg is_onoff Spreadsheet
```

```
SynologyDrive isn't turned on, [262]
Spreadsheet isn't turned on, [262]
```

Meanwhile both packages were plainly serving requests. `SynologyDrive` had `status=enabled` in its own state file, and the Office daemons were in the process table — a task daemon, a connection pooler running under an `office` user account, and gateway workers. `is_onoff` was simply not a reliable answer on this DSM build.

The second command was worse because it was my own mistake:

```bash
ps w | grep -iE "office|drive|pgbouncer"
```

```
(no output)
```

That looks like a dead system. It isn't. **On DSM, `ps` without `sudo` shows you only your own processes.** Re-run it as root and the picture inverts:

```bash
sudo ps aux | grep -E "office|pgbouncer|synoscgi" | grep -v grep
```

There they all are. I had "proven" the stack was down twice, using two commands that couldn't have shown me otherwise:

> If you take one habit from this post: when a diagnostic says "nothing is running", check whether the diagnostic had the privileges to see anything at all.

## Trap 2: the container with no startup error message

The actual answer, once I stopped believing my own evidence, was on Docker Hub: `synology/spreadsheet-api`. Not part of the Office package, not a gateway endpoint. Its own description is unambiguous:

> "Spreadsheet API is not part of the Office package. This image provides a proxy service between clients and DSM. Each worker is dedicated to a single user and a single spreadsheet."

The compatibility table matches image tags to Office versions, which is the first thing to check because the pairing is not optional:

| Docker image tag | Required Synology Office |
|---|---|
| `3.4.1` | 3.7.0 or higher |
| `3.3.2` | older |

Synology also warns against co-locating it with Office, since each worker loads whole spreadsheets into memory. I ran it on the same DSM anyway and capped it, which is a deliberate trade rather than a recommendation.

Then I ran it and it died immediately with this:

```
/app/dist/index.js:424
}`;var ot=UD(function(){return Ht($,qe+"return "+Ee).apply(e,H)});...
```

A minified JavaScript blob — the least useful class of error message there is. The cause, once found, was mundane: **`AUTH_SECRET` is required and has no default.** It signs the session tokens. Without it the service cannot start and cannot tell you why.

The reliable way to read a container's actual requirements, rather than its startup noise, is to ask the bundle directly:

```bash
docker run --rm --entrypoint grep synology/spreadsheet-api:3.4.1 \
  -ohE '.{0,80}process\.env\.PORT.{0,80}' /app/dist/index.js
```

```js
serverPort: parseInt(process.env.PORT) || 3e3,
serverHost: process.env.HOST || "0.0.0.0",
workerTimeout: parseInt(process.env.WORKER_TIMEOUT) || 600*1e3,
```

That gives you the config contract in one line: `AUTH_SECRET` to sign tokens, `PORT` defaulting to `3000`, `HOST` to `0.0.0.0`, workers recycled after ten minutes. It also ships its own OpenAPI spec and a Swagger UI, which I'll come back to.

## Trap 3: a 401 while the password is provably correct

With the container running, authentication became the next wall. Here is the request, and here is what it returned:

```bash
curl -s -X POST http://192.168.1.1:8791/spreadsheets/authorize \
  -H 'Content-Type: application/json' \
  -d '{"username":"<service-account>","password":"<password>","host":"192.168.1.1:5001","protocol":"https"}'
```

```json
{"error":"Unauthorized"}
```

`401 Unauthorized` with a password I had just used to sign into DSM. The natural reading — wrong password, wrong account — is a dead end, and I went down it: retyped the password, tried the account on the DSM login page (it worked), tried the HTTP port instead of HTTPS (same 401).

The actual problem was the `host` field. The service asks *you* which DSM it should authenticate against, then does its own TLS handshake with that host. I had given it a bare IP address, and the certificate presented on that name doesn't match the address, so the handshake failed — and the proxy reports a failed handshake as `Unauthorized`, exactly like a bad password would.

Switching to a hostname with a valid certificate fixed it instantly:

```bash
curl -s -X POST http://192.168.1.1:8791/spreadsheets/authorize \
  -H 'Content-Type: application/json' \
  -d '{"username":"<service-account>","password":"<password>","host":"cloud.example.com","protocol":"https"}'
```

```json
{"token":"eyJhbGciOi...","host":"cloud.example.com"}
```

> **If your Synology API login returns 401 and you're sure the password is right, check the `host`: it must be a name whose certificate the container accepts. A bare LAN IP is not that.**

The token it returns is a JWT, valid for 28 days, tied to a DSM session — so it dies if DSM restarts or the account is forcibly logged out. Treat a sudden `401` mid-integration as "re-authorize", not "credentials changed".

## Trap 4: 2FA cannot ever authenticate

This is the one that would waste the most time if you didn't know it in advance, so here it is plainly: **the sign-in schema has no one-time-code field.**

```yaml
AuthorizationBody:
  properties:
    username: {type: string}
    password: {type: string}
    host:     {type: string}
    protocol: {type: string}
```

No OTP, no app-password, no token exchange. If the account has two-factor authentication enabled, this API will return `401` forever, with the correct password, on every host. The only workable arrangement is a dedicated service account with 2FA off, scoped to the folders it needs — which is better practice anyway, and is what I should have started with instead of testing against my own admin account first.

## The permissions trap, after you're in

Authentication is not authorization. Once my service account could log in, every call against the spreadsheet I actually cared about returned:

```json
{"statusCode":403,"code":"403","error":"Forbidden","message":"Permission denied"}
```

Read that carefully, because it is informative: `403 Permission denied` means **the file exists and you may not have it.** A bad ID returns something different:

```json
{"statusCode":404,"code":"404","error":"Not Found","message":"Spreadsheet not found"}
```

Two different failures, two different fixes, and conflating them costs an hour. Mine was a location problem: the spreadsheet had been created in Synology Drive's personal **My Drive**, which on disk is `/volume1/homes/<user>/Drive/Document/…`. A personal home directory is not reachable by another DSM account — not by permission settings, not by shared-folder tricks, because it isn't in a shared folder at all. The fix is to move the file into a shared folder and grant Read/Write there, or to Drive-share the specific file to the service account with edit rights.

## The fix: the whole working setup

The container, configured with a real secret and a memory cap:

```yaml
services:
  spreadsheet-api:
    image: synology/spreadsheet-api:3.4.1
    container_name: spreadsheet-api
    restart: always
    environment:
      AUTH_SECRET: "<long-random-string>"
      PORT: "3000"
      HOST: "0.0.0.0"
    ports:
      - "8791:3000"
    mem_limit: 2g
```

One DSM-specific note: omit `cpus:` from any compose file on DSM. Its kernel has no CPU CFS scheduler, so the key is silently ignored — you get the illusion of a CPU limit and no limit.

Then the API works, and the shape is pleasant. Sign in, create or address a spreadsheet, read and write ranges:

```bash
TOKEN=$(curl -s -X POST http://192.168.1.1:8791/spreadsheets/authorize \
  -H 'Content-Type: application/json' \
  -d '{"username":"<acct>","password":"<pw>","host":"cloud.example.com","protocol":"https"}' \
  | python -c "import json,sys; print(json.load(sys.stdin)['token'])")

# read a range (sheet-qualified A1 notation)
curl -s -H "Authorization: Bearer $TOKEN" \
  "http://192.168.1.1:8791/spreadsheets/<id>/values/Sheet1!A1:C4"

# write a range, formulas included
curl -s -X PUT -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"values": [["Item","Qty"],["Widget",12],["Gadget",7],["Total","=SUM(B2:B3)"]]}' \
  "http://192.168.1.1:8791/spreadsheets/<id>/values/Sheet1!A1:B4"
```

Two details I did not expect, both in my favour:

- **The spreadsheet ID comes from the Office URL.** A spreadsheet at `…/oo/r/<spreadsheetId>` — including one behind a share link, which redirects into exactly that path — is addressable by putting that segment in the API call.
- **Office computes the formulas, not you.** Exporting a sheet as CSV returned the *evaluated* results of `=SUM(B2:B3)` and `=SUMPRODUCT(B2:B3,C2:C3)` — formulas I never evaluated client-side. That's the whole reason to use this API instead of parsing a file yourself: you inherit Synology's own calculation engine.

There is one real gap worth stating: **there is no list endpoint.** The spec declares a "Statistics" tag but serves no route for it (`/statistics` → `404 Route not found`), and nothing else enumerates spreadsheets. You must already know the ID from the document's URL. Fine for automation you set up on purpose; useless if you wanted to browse.

## Getting a chart out the other end

Reading cells was half the job; the original ask included charts. The same API can hand you a real workbook:

```bash
curl -s -H "Authorization: Bearer $TOKEN" \
  "http://192.168.1.1:8791/spreadsheets/<id>/xlsx" -o export.xlsx
```

That returns a valid `.xlsx` — `PK` magic, opens in anything. From there my own service (a small FastAPI container with LibreOffice inside it for formula evaluation and matplotlib for rendering) reads the ranges and produces PNG charts and dashboards, so the pipeline runs end to end without Excel and without any cloud dependency:

```
Synology Office  →  spreadsheet-api  →  .xlsx export  →  chart renderer  →  PNG
```

## What I'd do differently

1. **Check Docker Hub before concluding a Synology feature doesn't exist.** I burned real time proving a negative from an API list that could not have contained the answer. `synology/*` on Docker Hub was one search away.
2. **Start with a dedicated, non-2FA service account** scoped to one shared folder. I tested with an admin account first, which is how I met the 2FA wall and the `403` wall in the same hour.
3. **When a diagnostic says "nothing is running", check its privileges.** `ps` without root on DSM is not a process list, it's a list of your own processes, and it will happily let you conclude the machine is dead.
4. **Read the container, not just its docs.** The vendor's own run command is on the image page, but the exact defaults (`PORT` 3000, worker timeout 600 s) and the required `AUTH_SECRET` came from grepping the bundle — one command, no guessing.

## The result

Verified end to end on the NAS: sign-in, create, cell write, cell read-back, CSV export and `.xlsx` export all returned `200`, and the CSV export proved the Office engine evaluated `=SUM`/`=SUMPRODUCT` server-side. The exported workbook fed a chart service that rendered the same data to PNG. Four traps documented, one wrong conclusion corrected, roughly a day of work — and a spreadsheet on my own hardware that a script can now drive.

---

*I build self-hosted integrations like this for small businesses in Malaysia — spreadsheet and NAS automation, dashboards, and internal tools that keep your data on your own hardware. If that's the kind of thing you need, [WhatsApp me](https://wa.me/60127972969) or [email me](mailto:[email protected]?subject=Self-hosted%20spreadsheet%20automation) — or see what else I do at [hoelee.com](https://hoelee.com).*

文章 03 / 11：我差点把这颗 SSD 送修——真正坏的是 SATA 数据线

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "我差点把这颗 SSD 送修——真正坏的是 SATA 数据线"
description: "一颗 Seagate IronWolf 110 SSD 疯狂报 WRITE FPDMA 超时、mkfs 报 'superblock magic doesn't match'、然后读写静默——教科书般的坏盘。我只换了一根 SATA 线，它就干净格式化、零报错。动手判死刑前，先这样排查。"
pubDate: 2026-09-16
category: devops
blueprint: self-hosted-infrastructure
blueprintLayer: L1-compute-storage-network
tags: [unraid, sata, ssd, storage, troubleshooting, smart, fio, cable]
draft: false
ogImage: /og/zh-that-dying-ssd-was-just-a-bad-sata-cable.jpg
banner: /banners/that-dying-ssd-was-just-a-bad-sata-cable.jpg
---

所有存储指南都告诉你：先看 SMART，再信事件日志。我也照做了，两者都指向同一个结论：这颗 SSD 要死了。而我距离把一颗健康硬盘送去 RMA 只差一步。

2026 年 9 月，一颗 Seagate IronWolf 110 960 GB SATA SSD——SMART 全绿、寿命约 99%、重映射扇区为零——反复在 btrfs `mkfs` 时失败，后来干脆连读都不回。内核日志里堆满了 `WRITE FPDMA QUEUED` 超时、`NCQ disabled due to excessive errors`、`lost async page write`。最后一次 `mkfs` 打印出 superblock 后，回读失败了。

只有一件事救了这颗盘：在判它死刑之前，我换了根 SATA 数据线。

## 为什么这很重要

一根有问题的 SATA 口或线材，会产生**和坏盘一模一样的症状**，而且 SMART 全程保持漂亮。线材故障发生在链路层——传输出错超时，内存在你的文件系统或 SMART 看到任何错误字节之前，就把这次 I/O 丢了。只靠 SMART 判断，你会冤枉健康硬件、发起一趟无意义的 RMA，而换上的新盘在同一根线上照样坏。

## 那些"盘要死了"的症状

这是一颗独立的 Seagate IronWolf 110（ZA960NM10001，企业级 SATA SSD，约 894 GiB），我准备把它加进 Unraid 缓存池。三件事陆续发生，全都很像坏盘：

1. 内核日志每约 30 秒就循环一轮 libata 错误处理：
   ```
   ata7.00: exception Emask 0x4 (timeout), WRITE FPDMA QUEUED, status { DRDY }
   ata7.00: exception ... action 0x6 frozen
   attribute NCQ disabled due to excessive errors
   Buffer I/O error on dev sdd1 ... lost async page write
   ```
   硬盘无法完成排队的写入，内核硬重置链路；降级之后它**丢掉了真实数据**——`/dev/sdd` 有 28 个 LBA 报 lost async page write。

2. `mkfs.btrfs -K -f /dev/sdd1` 打印出完整的文件系统头，然后校验失败：
   ```
   Filesystem size: 894.25GiB
   ERROR: superblock magic doesn't match
   ```
   btrfs 写入了 superblock，回读时拿到的是**不一样的字节**。这是最可怕的静默写路径损坏。

3. 之后 `smartctl -H /dev/sdd` **超时**。这颗盘连健康查询都不回应了。结合上面的写损坏内核标志，"送 RMA"是合理的结论。

## 改变结论的那个细节

排查到一半我必须物理移动这颗盘——它被接到另一个 SATA 口、换了根线。我在动任何其他操作之前，重跑了**完全相同**的 `mkfs.btrfs` 命令。

它通过了。零报错。`btrfs device stats` 全部清零。一次加了 `verify=crc32c`（写入带校验和的数据并**回读核对**）的 8 GiB `fio` 写入以 err=0 干净跑完，回读一致。`smartctl` 超时再也没有出现。`dmesg` 自那次换线以来对这颗盘**零**报错。

除了链路，什么都没变。那些"静默写损坏"，是 SATA 传输在线上损坏了帧，不是闪存的问题。

## 能抓到它 vs 抓不到它的测试

这一趟得出两个教训：

**普通的 `fio randrw` 对这个故障是瞎的。** 标准的 60 秒 `randrw`（70/30、4k、iodepth 32）写读随机扇区——但它从不*核对*读回来的字节是否和你写的一致。对一颗在线上损坏数据的盘，它会报 err=0、"干净通过"。我第一轮测试就是这样：*干净*，却在一颗随后 mkfs 失败的盘上。

**你需要 `verify`，而且要点对区域。** fio 的 `verify=crc32c` 写入校验和并回读核对，这正是能揪出回读不一致的开关。而且故障是**局部**的：坏掉的 LBA（约 234,422,526–869，距设备起始约 111.8 GiB）落在我第一轮测试跑过位置再往后 100 GiB 处。在错误的偏移量上测试完全测不到。证据就是：指向健康区域的 verify 任务通过了；同一任务指向已知坏区间时，硬盘像先前缓存池故障时那样直接卡住。

```
# 判别性测试：写入带校验和，回读，不一致就失败
# 要么跑全表面，要么跑 dmesg 标记的精确 LBA 区间
fio --name=verify --filename=/dev/sdd \
    --offset=111G --size=2G \
    --rw=write --verify=crc32c --do_verify=1 \
    --bs=4k --ioengine=io_uring --direct=1 --iodepth=32
```

同一任务跑在*健康*盘上，`verify` 会通过；链路坏的话，fio 会报校验和不符或卡死——远比普通的 randrw 更早暴露问题。

## 我会改的做法

- **先换线。** 在看到超时或写损坏时，在给盘判死刑之前，重新插拔/更换线材，或换到另一个口，重跑*相同*的失败命令。问题跟着**端口**走就是链路故障；跟着**盘**走（换了好口还坏）才是盘的问题。这是最便宜、最能一锤定音的测试，而我花了好几小时才做。
- **别信一个干净的 `randrw`。** 它看不见静默损坏。凡是盘出现过任何写症状，都用 `verify=crc32c` 和 `do_verify=1`。
- **对准日志里的 LBA 区间。** 内核或文件系统已经标记了具体扇区的话，就在*那里*测，而不是从偏移 0 开始。
- **SMART 全绿 ≠ 链路健康。** UDMA_CRC 为零只说明*当前*链路干净，它看不到过去那段不稳定的链路；而传输超时的页面在算出 CRC 之前就已经被丢了。

## 结果

这颗几分钟前就要被我送去 RMA 的盘，现在已经是一颗格式化好、挂载上、verify 全过的缓存 SSD，`btrfs device stats` 全部清零。修好它的成本：一根线。相信原有诊断的成本：一趟没必要的保修，还有一台迟早会在那根坏线上损坏数据的机器。

如果你在 NAS 或自管服务器上遇到一颗盘报 FPDMA 超时、但 SMART 看起来没问题——换线，别急着换盘。

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "I Was About to RMA This SSD — the Fault Was a SATA Cable"
description: "A Seagate IronWolf 110 SSD threw WRITE FPDMA timeouts, failed mkfs with 'superblock magic doesn't match', and went silent — a textbook dying drive. I swapped one SATA cable and it formatted clean with zero errors. How to test before condemning a drive."
pubDate: 2026-09-16
category: devops
blueprint: self-hosted-infrastructure
blueprintLayer: L1-compute-storage-network
tags: [unraid, sata, ssd, storage, troubleshooting, smart, fio, cable]
draft: false
ogImage: /og/that-dying-ssd-was-just-a-bad-sata-cable.jpg
banner: /banners/that-dying-ssd-was-just-a-bad-sata-cable.jpg
---

Every storage guide tells you to start with SMART and trust the event log. I did. Both pointed at one verdict: this SSD is dying. I was one config change away from RMA-ing a healthy drive.

In September 2026 a Seagate IronWolf 110 960 GB SATA SSD — SMART-clean, ~99% life, zero reallocated sectors — kept failing a btrfs `mkfs` and then stopped answering reads. The kernel log was full of `WRITE FPDMA QUEUED` timeouts, `NCQ disabled due to excessive errors`, and `lost async page write`. The final `mkfs` printed its superblock, then failed to read it back.

Exactly one detail saved the drive: I swapped the SATA cable before I condemned it.

## Why this matters

A bad SATA port or cable produces **the same symptoms as a dead drive**, and it produces them while SMART stays immaculate. Cable faults happen at the link layer — the transport times out and the kernel drops the I/O before your filesystem or SMART ever sees a wrong byte. If you diagnose by SMART alone, you'll condemn healthy hardware, raise a pointless RMA, and the replacement will fail the same way on the same cable.

## The symptoms that screamed "dead drive"

This was a standalone Seagate IronWolf 110 (ZA960NM10001, an enterprise SATA SSD, ~894 GiB) I was preparing for an Unraid cache pool. Three things happened, all consistent with a failing SSD:

1. The kernel log cycled through libata error handling every ~30 s:
   ```
   ata7.00: exception Emask 0x4 (timeout), WRITE FPDMA QUEUED, status { DRDY }
   ata7.00: exception ... action 0x6 frozen
   attribute NCQ disabled due to excessive errors
   Buffer I/O error on dev sdd1 ... lost async page write
   ```
   The drive stopped completing queued writes, the kernel hard-reset the link, and once it degraded it **dropped real pages** — 28 `/dev/sdd` LBAs lost async page writes.

2. `mkfs.btrfs -K -f /dev/sdd1` printed a full filesystem header, then failed verification:
   ```
   Filesystem size: 894.25GiB
   ERROR: superblock magic doesn't match
   ```
   btrfs wrote its superblock, reread it, and got different bytes back. That's silent write-path corruption — the worst kind.

3. After that, `smartctl -H /dev/sdd` **timed out**. The drive wasn't answering even a health poll. Combined with the write-corruption kernel flags, "RMA it" was the reasonable conclusion.

## The detail that changed the verdict

Mid-diagnosis I had to move the drive physically — I landed it on a different SATA port with a different cable. I reran the *identical* `mkfs.btrfs` command before doing anything else.

It passed. Zero errors. `btrfs device stats` came back all-zero. An 8 GiB `fio` write with `verify=crc32c` (which writes patterned data and **rereads it to check bytes**) completed at err=0 with a clean read-back. The `smartctl` timeout never returned. `dmesg` shows **zero** error lines for the drive since that swap.

Nothing changed except the link. The "silent write corruption" was the SATA transport corrupting frames on the wire, not the NAND.

## The test that would have caught it (and the one that won't)

Two lessons came out of this:

**A plain `fio randrw` is blind to this fault.** A standard 60 s `randrw` test (70/30, 4k, iodepth 32) writes and reads random sectors — but it never *checks* that the bytes you read back match what you wrote. On a drive that corrupts on the wire, it reports err=0 and a clean pass. That's exactly what my first test did: *clean*, on a drive that then failed mkfs.

**You need `verify`, and you need the right region.** fio's `verify=crc32c` writes checksums and rereads to confirm them — that's what trips read-back mismatches. And the fault was **localized**: the corrupt LBAs (~234,422,526–869, around 111.8 GiB into the device) sat 100 GiB past where my first test ran. A test at the wrong offset misses it entirely. The proof: a verify job pointed at the healthy region passed; the same job pointed at the known-bad range stalled the drive exactly like the pool failure had.

```
# The discriminating test: write w/ checksum, reread, fail on mismatch
# Run against the ENTIRE surface or the exact LBA range dmesg flagged
fio --name=verify --filename=/dev/sdd \
    --offset=111G --size=2G \
    --rw=write --verify=crc32c --do_verify=1 \
    --bs=4k --ioengine=io_uring --direct=1 --iodepth=32
```

If you run the same job on a *healthy* drive, `verify` succeeds. If the link is bad, fio reports checksum mismatches or hangs — long before a plain run would tell you anything.

## What I'd do differently

- **Swap the cable first.** Before condemning a drive for timeout or write-corruption, reseat/replace the cable or move to a different port and rerun the *same* failing command. If the problem follows the **port**, it's a link fault; if it follows the **drive** across good ports, it's the drive. This is the single cheapest, most decisive test and I skipped it for hours.
- **Don't trust a clean `randrw`.** It can't see silent corruption. Use `verify=crc32c` and `do_verify=1` whenever a drive has shown any write symptom.
- **Target the logged LBA range.** If the kernel or filesystem already flagged specific sectors, test *there*, not at offset 0.
- **SMART-clean ≠ healthy link.** Zero UDMA_CRC just means the *current* link is clean; it can't see the past flaky one, and transport-timeout pages are dropped before a CRC is even computed.

## The result

The drive I was minutes away from RMA-ing is now a formatted, mounted, verified-working cache SSD with `btrfs device stats` reading all zeros. The cost of the fix: one cable. The cost of trusting the diagnosis I had: a needless warranty claim and a rig kept on a cable that was going to corrupt data sooner or later anyway.

If you're on a NAS or a home server and a drive "fails" with FPDMA timeouts but SMART looks fine — change the cable before you change the drive.

文章 04 / 11：换了不同的硬盘，同样的损坏又来了——真凶是 TRIM，不是 SSD

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "换了不同的硬盘，同样的损坏又来了——真凶是 TRIM，不是 SSD"
description: "两周前 btrfs RAID1 阵列在看似健康的 NVMe 盘上静默损坏数据，两周后同一个全零校验和出现在完全不同的硬盘上。真凶是 Seagate IronWolf 110 上的队列化 TRIM（NCQ）——FreeBSD 早就拉黑了这个固件缺陷，Linux 却一直没有。我是怎么抓到它、以及修复方法的全过程。"
pubDate: 2026-09-18
category: devops
tags: [unraid, btrfs, trim, ssd, raid, data-loss, smart, troubleshooting, seagate]
ogImage: /og/zh-the-cause-was-trim-not-the-ssds.jpg
banner: /banners/the-cause-was-trim-not-the-ssds.jpg
---

两周前我写了一篇文章，讲一个 btrfs RAID1 阵列如何在两块看起来完全健康的三星 PM9A3 NVMe 盘上静默损坏 Windows 虚拟机——读出来的全是**全零**数据块，单靠一块坏盘根本无法解释。那次调查以一句诚实的「不确定」收尾：两块盘测试都干净，最大嫌疑是它们共用的 M.2 转接卡。

然后它又发生了。在**不同的硬盘**上，走**不同的总线**。校验和指纹一模一样。这一次我找到了根因——而且跟 SSD 本身无关。

## 为什么这很重要

如果你在 Unraid（或任何 Linux 机器）上跑 btrfs RAID1 + SATA SSD + 自动 TRIM，这篇是给你的提醒：**队列化 TRIM 正在同时、在同一位置、静默地摧毁我阵列的两份镜像。** RAID 冗余在这种损坏面前毫无保护力。而全程被 SMART 标记为「failing」的那块盘，跟真正的损坏一点关系都没有。

---

## 环境

- **unRaid 7.x**，Intel i9-13900K。
- **`ssd` 池**：两块 **Seagate IronWolf 110 960 GB** SATA SSD
  （`ZA960NM10001`，序列号 `HKR02TFK` + `HKR02L9P`）组 **btrfs RAID1**，
  挂载在 `/mnt/ssd`，上面跑着 Docker 数据根目录（`/var/lib/docker`）和
  Windows 11 虚拟机的虚拟磁盘（`domains/Win11Enterprise/vdisk1.img`）。
- 虚拟机虚拟磁盘**没有**开 `NOCOW`，所以 btrfs 每次读取都会校验——和上次
  事故能被发现靠的是同一个设置。

顺带一提：其中一块盘（`sdd`）正是 *[《我当时差点就去 RMA 这块 SSD——其实是 SATA 线坏了》](/posts/zh/that-dying-ssd-was-just-a-bad-sata-cable/)*
里那块盘——换线之后格式化干净，就进了这个池。

## 第一步——一个「假」的 SMART 故障

凌晨 02:53，Unraid 的通知铃响了：

```
Unraid Ssd disk SMART health [1] — Warning [UNRAID] -
raw read error rate (failing now) is 19665   (sdd = ZA960NM10001_HKR02TFK)
```

「failing now」是 SMART 里最强的措辞——盘自报该属性**跌破阈值**。但在这些
Seagate 盘上，raw 值是障眼法：IronWolf 110 的属性 1 raw 字段是厂商编码的
复合值（高半部分是错误数，低半部分是操作计数）。`19665 = 0x00004CD1` →
高半部分 = **0 个错误**。真正触发的是*归一化*值：`093`，**最差值 088**，
低于 `090` 阈值，于是盘在 SMART 返回状态里自报 `FAILING_NOW`/`IN_THE_PAST`。
另一块盘（`sdb`）形状相同，raw 只有 `341`。

所有真实的故障计数器都是零：重映射 0、坏块增长 0、编程/擦除失败 0、
不可纠正 ECC 0、UDMA CRC 0、寿命剩余 99 %、SMART 总体 **PASSED**，刚跑完的
扩展自检也干净。这是 IronWolf 110 有据可查的行为——多年前 Synology 用户就
在 100 % 寿命时撞上过同样的「Failing」旗标，社区共识是：只要真实计数器不动，
这个型号的属性 1 触发就当固件噪音处理。

所以：盘是误报，但这次报警让我去看了——然后才找到真正的损坏。

## 第二步——一个不该存在的校验和

`btrfs device stats /mnt/ssd`：

```
[/dev/sdd1].corruption_errs  27     [/dev/sdb1].corruption_errs  31
```

全部集中在同一个文件：**ino 261 = Windows 虚拟机的虚拟磁盘**。
`dmesg` 里是上一个 NVMe 事故里我盯了一周的同一个签名：

```
btrfs: checksum verify failed on logical 1505629372416 mirror 1 wanted 0x8941f998
```

**`0x8941f998` 是 4 KiB 全零块的 CRC32C。** 池在读出空白块——而且关键的，是
在**两块盘的相同逻辑偏移上同时读出**。我翻了 PM9A3 阵列那次的笔记：值相同，
「确定性全零」行为相同，让 RAID1 失效的关联方式也相同。

这个匹配是指纹级的。无论这是什么，它都不是硬盘的问题，也不是 M.2 转接卡的
问题——IronWolf 这对盘走的是 **SATA 口**，完全不同的控制器路径。问题出在
**比硬盘高一层**：有一个机制决定同时擦掉两块盘上相同的数据。Linux 里恰好
只有一个机制干这事：**discard / TRIM**。

## 第三步——根因：在 Linux 从未拉黑的硬盘上跑队列化 TRIM

Unraid 的池配置开了自动 TRIM，btrfs 以 `discard=async` 挂载——内核批量收集
空闲块发给硬盘。对 SATA SSD 来说，那就是**队列化 TRIM**（`DATA SET
MANAGEMENT` 命令以 NCQ 命令形式延迟执行，即 `SEND FPDMA QUEUED`）。

IronWolf 110 的队列化 TRIM **不稳定的问题早就被记录了**。FreeBSD 内核有一个
文档化的报告——[bug 264139](https://bugs.freebsd.org/264139)，
*「ata: NCQ_DSM_TRIM trim method for Seagate IronWolf 110 SATA SSD hangs drives」*，
2024 年合并的修复提交写得毫不客气：[commit `a6cef617660a`](https://lists.freebsd.org/archives/freebsd-fs/2024-March/003273.html)：
「Seagate IronWolf 110 SATA SSD 已被报告在启用 NCQ trim 时不稳定。」

而 Linux 自己的怪癖清单（`drivers/ata/libata-core.c`）——内核会在这里主动把
有问题的盘降级为非队列化或禁用 TRIM——收录了 Micron M500/M550/1100、
Crucial M500/M550/MX100、三星 840/850/860/870 等，但**没有一条 Seagate
条目**。我自己机器上的 `dmesg` 恰好证明这个机制存在而且在选择性生效：

```
ata6.00: Model 'Samsung SSD 840 PRO Series', rev 'DXM06B0Q', applying quirks: noncqtrim zeroaftertrim
```

这条怪癖给了机箱里*另一块* SSD，而两块 IronWolf 110 什么怪癖都没有，
拿到了完整的队列化 TRIM。

为什么这会毁掉 RAID1？因为 btrfs RAID1 在两块成员盘上存的是*同一份*空闲
映射——会在同一轮里对两块盘丢弃相同的逻辑区间。如果这个固件上的 TRIM 路径
会多失效（或失效错）LBA，就会**同时清掉两块镜像上的同一批活动块**。你的
镜像变成同一个洞的两份拷贝：两块盘都校验失败，自愈没有任何可抄的源，数据
彻底没了。

> 诚实的保留意见：FreeBSD 记录的是盘侧的*不稳定/卡死*，而「队列化 TRIM 抹掉
> 了活动数据」是我根据这个记录加指纹（双镜像全零、SMART 干净、CRC 错误计数
> 不涨）推断的机制。它符合每一条观察。证明靠实验——见下面的结果。

## 第四步——修复（持久化，一次只改一个变量）

只改一处，双管齐下——持久配置 + 热挂载：

```bash
# /boot/config/pools/ssd.cfg   （备份：/root/ssd.cfg.bak.20260918-0329）
diskAutotrim="off"                       # 原来是 "on"

# 不停阵列、不动 Docker（36 个容器全程在线）直接生效
mount -o remount,nodiscard /mnt/ssd
findmnt /mnt/ssd   # → rw,noatime,ssd,space_cache=v2   （不再有 discard=async）
```

然后做一次完整读写 scrub，搞清楚到底坏了多少：

```
scrub started 03:29:53, finished 03:53:07 (23 min)
csum_errors: 18    corrected_errors: 16    uncorrectable_errors: 2
read_errors: 0     verify_errors: 0        super_errors: 0
```

`read/verify/super = 0` 是指纹的另一半：硬件读盘*完全正常*——没有任何部件在
故障，只是盘上存的字节和校验和不符。18 个坏块里，**16 块只有一块镜像坏，
已从完好副本自愈；2 个 uncorrectable 其实是同一个 4 KiB 块两块镜像都坏**——
永久丢失，位于虚拟机虚拟磁盘约 45.2 GiB 处（`logical 1505629372416`，文件
偏移 `48560156672`）。我把这一个块填零，至少虚拟机读到那里不再报错；原始
字节无论如何都找不回来了。

还有一个值得知道的坑：**scrub 跑了 18 个错误，`btrfs device stats` 计数器
却纹丝不动**——scrub 发现的错误和读取时发现的错误是分开统计的。你没法只盯
`device stats` 一个数；dmesg 和 scrub 输出也得一起盯。（我的看门狗三样都查，
细节见文末。）

## 换成我会怎么做

1. **任何 btrfs 池开自动 TRIM 之前，先查这块盘的队列化 TRIM 记录。**
   FreeBSD 维护一份「别给这盘发 TRIM」清单不是没有原因的。Linux 在
   `libata-core.c` 里有同样的机制——花一个下午 grep + 首次开机后看一眼
   `dmesg`，好过一次从备份里恢复虚拟机。
2. **TRIM 可以有，但不能队列化。** SATA SSD 想跑 fstrim 就强制非队列化
   （`libata.force=…:noncqtrim`），并在第一次 trim 后做一次 scrub 验证。
   `discard=sync` **没用**——只要盘支持，sync 同样走队列化路径。
3. **三个损坏信号一起盯**，别只看一个：`btrfs device stats`、
   `dmesg | grep 'csum failed'`、周期性 scrub。三者各有各的计数器（见上文）。
4. **并入池之前先交叉验证。** 同批次、同固件的盘是关联故障风险；两块盘共一个
   固件 bug 就是最坏的那种关联故障。镜像两次都没救得了我。
5. **买企业级 SATA SSD 时**：Linux 怪癖清单上的盘（Micron M500/M550/1100、
   Crucial M500/M550/MX100、三星 840/850/860/870）以及一切有记录在案的
   NCQ-TRIM bug（IronWolf 110）直接排除。三星 PM893/PM897、Micron 5400
   PRO/MAX、Solidigm D3-S4610/S4620、WD Ultrastar DC SA620/SA630、Kingston
   DC600M 都有断电保护（PLP）且不在 Linux 怪癖清单上；而且 Micron、Solidigm、
   WD、Kingston 会公开发布固件更新说明，真出了问题还能给你推送修复。NVMe
   从根上绕开了 SATA 队列化 TRIM 这一类问题——我那两块 PM9A3 也不在 NVMe
   怪癖表里。

## 结果

只改了一个变量（关自动 TRIM），其他原封不动。紧接着：

- 第二次 scrub（04:10–04:39）：**`corrected_errors: 0`** ——16 块自愈块保持
  住了，整盘重读没有出现任何新错误。
- `corruption_errs` 维稳在 sdd1=27 / sdb1=31，持续数小时的 Docker + 无虚拟机
  运行（此前计数器是每天在涨的）。
- dmesg 里没有新增 `csum failed` 行。

确认窗口还有几天：Win11 虚拟机重新上线（它产生的是能复现问题的写入模式）、
对照基线文件每日检查、2026-09-22 再做一次复核 scrub。如果关了 TRIM 损坏还在，
下一个嫌疑就不是硬盘，而是内存（这台机器没有 ECC）——用 memtest 测。如果
一直平静，那 IronWolf 110 队列化 TRIM 固件 bug 就算坐实了——我敢说论坛里
「健康硬盘毁掉我的 RAID」的帖子，相当一部分也是它干的。

一个「正在故障」的 SMART 告警、一块零错误的硬盘、一个全新阵列上双镜像同时
损坏——每个迹象都指向别处，而真正的凶手是一个我们大多数人不假思索就开启的
存储功能。

---

*我的存储故障系列之一：[NVMe 事故](/posts/when-smart-says-healthy-but-your-raid-is-corrupting-data/) 和 [SATA 线警醒](/posts/that-dying-ssd-was-just-a-bad-sata-cable/)。*

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "The Corruption Came Back on Different Drives — the Cause Was TRIM, Not the SSDs"
description: "Two weeks after a btrfs RAID1 pool corrupted on healthy-looking NVMe drives, the same all-zero checksum reappeared on a different drive stack. The culprit was queued TRIM (NCQ) on Seagate IronWolf 110 SSDs — a firmware bug FreeBSD blacklisted but Linux never did. How I caught it and the fix."
pubDate: 2026-09-18
category: devops
tags: [unraid, btrfs, trim, ssd, raid, data-loss, smart, troubleshooting, seagate]
ogImage: /og/the-cause-was-trim-not-the-ssds.jpg
banner: /banners/the-cause-was-trim-not-the-ssds.jpg
---

Two weeks ago I published how a btrfs RAID1 pool on two healthy-looking Samsung
PM9A3 NVMe drives silently corrupted a Windows VM — reads returning
**all-zero** blocks that no single bad drive could explain. That investigation
ended with an honest shrug: the drives tested clean, the strongest suspect was
the shared M.2 riser.

Then it happened again. On *different* drives, on a *different* bus. The exact
same checksum fingerprint. This time I found the root cause — and it wasn't
the SSDs.

## Why this matters

If you run Unraid (or any Linux box) with a btrfs RAID1 pool on SATA SSDs and
auto-TRIM enabled, this post is a heads-up: **queued TRIM was silently
destroying both mirrors of my pool, in the same places, simultaneously.** RAID
redundancy gives you zero protection against it. And SMART — which flagged a
drive as *failing* through all of this — had nothing to do with the actual
corruption.

---

## The setup

- **unRaid 7.x** on an Intel i9-13900K.
- **Pool `ssd`**: two **Seagate IronWolf 110 960 GB** SATA SSDs
  (`ZA960NM10001`, serials `HKR02TFK` + `HKR02L9P`) in **btrfs RAID1**,
  mounted at `/mnt/ssd`, holding Docker's data root (`/var/lib/docker`) and
  the Windows 11 VM's virtual disk (`domains/Win11Enterprise/vdisk1.img`).
- The VM's vdisk is **not** `NOCOW`, so btrfs validates every read against a
  checksum — the same setting that caught the previous incident.

Trivia: one of these drives (`sdd`) is the very drive from *["I Was About to
RMA This SSD — the Fault Was a SATA Cable"](/posts/that-dying-ssd-was-just-a-bad-sata-cable/)*
— after the cable swap it formatted clean and joined this pool.

## Step 1 — a SMART "failure" that wasn't

At 02:53 the Unraid notification bell went off:

```
Unraid Ssd disk SMART health [1] — Warning [UNRAID] -
raw read error rate (failing now) is 19665   (sdd = ZA960NM10001_HKR02TFK)
```

"Failing now" is the drive declaring the attribute **worse than its
threshold** — the strongest wording SMART has. But the raw value is a red
herring on these Seagates: on IronWolf 110s, attribute 1's raw field is a
vendor-encoded composite (errors in the high half, an operation counter in the
low half). `19665 = 0x00004CD1` → high half = **0 errors**. The *normalized*
value is what tripped: `093` with a **worst of 088**, below the `090`
threshold, so the drive self-reports `FAILING_NOW` / `IN_THE_PAST` on its
SMART return status. The twin (`sdb`) showed the same shape with a raw of
only `341`.

Every real failure counter was zero: reallocated 0, grown bad block 0,
program/erase fails 0, uncorrectable ECC 0, UDMA CRC 0, SSD life left 99 %,
SMART overall **PASSED**, and a just-completed extended self-test was clean.
This is a documented IronWolf 110 behaviour — the same "Failing" flag at 100 %
lifespan hit Synology users years ago, and it's why the community treats that
model's attribute-1 trip as firmware noise unless the real counters move.

So: false alarm on the *drive*, but the alarm made me go look — and that's
when I found the real damage.

## Step 2 — the checksum that had no right to exist

`btrfs device stats /mnt/ssd`:

```
[/dev/sdd1].corruption_errs  27     [/dev/sdb1].corruption_errs  31
```

All of it confined to one file: **ino 261 = the Windows VM's vdisk**.
`dmesg` showed the same signature I'd spent a week staring at in the NVMe
incident:

```
btrfs: checksum verify failed on logical 1505629372416 mirror 1 wanted 0x8941f998
```

**`0x8941f998` is the CRC32C of 4 KiB of zeros.** The pool was reading back
empty blocks — and crucially, on **both mirrors at the same logical offsets**.
I verified it against my notes from the PM9A3 pool: identical value, identical
"deterministic zeros" behaviour, identical RAID1-defeating correlation.

That match is a fingerprint. Whatever this is, it isn't the drives and it
isn't the M.2 riser — the IronWolf pair lives on **SATA ports**, a
completely different controller path. It's the *layer above* the drives
deciding to erase the same data on both members at once. There is exactly one
mechanism in Linux that does that: **discard / TRIM**.

## Step 3 — the root cause: queued TRIM on a drive Linux never blacklisted

Unraid's pool config had auto-TRIM on, which mounts btrfs with
`discard=async` — the kernel batches frees and sends them to the drive. On
SATA SSDs that means **queued TRIM** (the `DATA SET MANAGEMENT` command
deferred as an NCQ command, `SEND FPDMA QUEUED`).

The IronWolf 110 is *known to be unstable with queued TRIM*. FreeBSD's
kernel has a documented report — [bug 264139](https://bugs.freebsd.org/264139),
*"ata: NCQ_DSM_TRIM trim method for Seagate IronWolf 110 SATA SSD hangs
drives"* — and the fix, merged in 2024, is blunt:
[commit `a6cef617660a`](https://lists.freebsd.org/archives/freebsd-fs/2024-March/003273.html):
"The Seagate IronWolf 110 SATA SSD drive has been reported to be unstable with
NCQ trim enabled."

Meanwhile Linux's own quirk list (`drivers/ata/libata-core.c`) — where the
kernel deliberately downgrades broken drives to non-queued or disabled TRIM —
contains Micron M500/M550/1100, Crucial M500/M550/MX100, Samsung 840/850/860/870
and a few others, but **no Seagate entry at all**. My own `dmesg` proved the
mechanism exists and is selective:

```
ata6.00: Model 'Samsung SSD 840 PRO Series', rev 'DXM06B0Q', applying quirks: noncqtrim zeroaftertrim
```

…applied to the *other* SSD in the box, while the two IronWolf 110s got
no quirk and full queued TRIM.

Why does that corrupt a RAID1? Because btrfs RAID1 stores the *same* free-space
map on both members — it discards the same logical extents on both drives in
the same pass. A TRIM path that invalidates more (or the wrong) LBAs on this
firmware zeroes the **same live blocks on both mirrors at once**. Your mirror
becomes two copies of the same hole: the checksum fails on both, self-heal has
nothing to copy from, and the data is gone.

> The honest caveat: FreeBSD documents *instability/hangs* from the drive
> side, and "queued TRIM erased live data" is the mechanism I infer from that
> plus the fingerprint (zeros on both mirrors, SMART-clean, CRC error counter
> untouched). It fits every observation. The proof is the test — see the
> result below.

## Step 4 — the fix (persistent, one variable at a time)

One change, applied twice — persistent config and live mount:

```bash
# /boot/config/pools/ssd.cfg   (backup: /root/ssd.cfg.bak.20260918-0329)
diskAutotrim="off"                       # was "on"

# apply live without stopping the array or Docker (36 containers stayed up)
mount -o remount,nodiscard /mnt/ssd
findmnt /mnt/ssd   # → rw,noatime,ssd,space_cache=v2   (no more discard=async)
```

Then a full read-write scrub to find out what was actually damaged:

```
scrub started 03:29:53, finished 03:53:07 (23 min)
csum_errors: 18    corrected_errors: 16    uncorrectable_errors: 2
read_errors: 0     verify_errors: 0        super_errors: 0
```

`read/verify/super = 0` is the second half of the fingerprint: the hardware
reads *fine* — nothing is failing, the stored bytes just don't match their
checksums. Of the 18 bad blocks, **16 were bad on one mirror and were healed
from the clean copy. Two uncorrectable entries were one single 4 KiB block
that was bad on both mirrors** — permanent loss, ~45.2 GiB into the VM's
vdisk (`logical 1505629372416`, file offset `48560156672`). I zero-filled
that one block so the VM's reads at least don't error; the original bytes are
unrecoverable either way.

And a wrinkle worth knowing: **`btrfs device stats` counters did not move
during the scrub** even though it found 18 errors — scrub-detected errors and
read-detected errors are tracked separately. You cannot watch only `device
stats`; you have to watch dmesg + scrub output too. (My watchdog checks all
three; the details are at the end.)

## What I'd do differently

1. **Never enable auto-TRIM on a btrfs pool without checking the drive's
   queued-TRIM record.** FreeBSD keeps a "don't TRIM this drive" list for a
   reason. Linux has the same mechanism in `libata-core.c` — an afternoon of
   grep + a `dmesg` check after the first boot beats a restored-from-backup VM.
2. **Keep TRIM on the menu, just not queued.** If you want fstrim on SATA
   SSDs, force non-queued TRIM (`libata.force=…:noncqtrim`) and verify with a
   scrub after the first trim pass. `discard=sync` does **not** help — it
   still uses the queued path on drives that advertise it.
3. **Watch all three corruption signals**, not one: `btrfs device stats`,
   `dmesg | grep 'csum failed'`, and periodic scrubs. Each can move
   independently (see above).
4. **Corroborate before joining pairs.** Same-batch, same-firmware drives are
   a correlated-failure risk; a pair sharing one firmware bug is the
   correlated-failure worst case. The mirror didn't save me either time.
5. **When buying enterprise SATA SSDs**: the drives on Linux's quirk list
   (Micron M500/M550/1100, Crucial M500/M550/MX100, Samsung 840/850/860/870)
   and anything with a documented NCQ-TRIM bug (IronWolf 110) are out.
   Samsung PM893/PM897, Micron 5400 PRO/MAX, Solidigm D3-S4610/S4620,
   WD Ultrastar DC SA620/SA630, and Kingston DC600M all carry PLP plus no
   Linux-quirk entries, and vendors that publish firmware release notes
   (Micron, Solidigm, WD, Kingston) can actually ship you a fix. NVMe
   sidesteps the SATA queued-TRIM class entirely — my PM9A3s aren't in the
   NVMe quirk table either.

## The result

One variable changed (auto-TRIM off), everything else identical. Immediately
after:

- scrub #2 (04:10–04:39): **`corrected_errors: 0`** — the 16 heal-blocks
  stayed healed, nothing new appeared during the full re-read.
- `corruption_errs` flat at sdd1=27 / sdb1=31 for hours of normal Docker +
  VM-less operation (counters had been climbing daily before).
- No new `csum failed` lines in dmesg.

The confirmation window runs a few days: Win11 VM back on (it produces the
write pattern that reproduced this), daily checks against a baseline file,
and a re-scrub on 2026-09-22. If corruption reappears with TRIM off, the next
suspect isn't the drives — it's the RAM (no ECC on this box), and memtest is
the test. If it stays quiet, the IronWolf 110 queued-TRIM firmware bug is
confirmed as the cause of both this incident and — I'd bet — a good share of
the "healthy drives corrupting my RAID" reports the forums keep collecting.

A "failing" SMART warning, a drive with zero errors, and corruption on
both mirrors of a fresh pool — every sign pointed somewhere else, and the
actual culprit was a storage feature most of us enable without a second
thought.

---

*Part of my storage-failure series: [the NVMe incident](/posts/when-smart-says-healthy-but-your-raid-is-corrupting-data/) and the [SATA cable wake-up call](/posts/that-dying-ssd-was-just-a-bad-sata-cable/).*

文章 05 / 11：上游被删了，然后带着一个重写版回来

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "上游被删了，然后带着一个重写版回来"
description: "我线上跑的阅读服务，依赖的镜像官方仓库已经 404。几个月后上游回来了——233 个提交、换了语言、零 release。怎么分辨「项目死了」和「项目在重写」。"
pubDate: 2026-10-06
category: devops
tags: ["docker", "self-hosting", "upstream", "maintenance", "registry"]
ogImage: "/og/zh-the-upstream-was-deleted-then-came-back-rewritten.jpg"
banner: "/banners/the-upstream-was-deleted-then-came-back-rewritten.jpg"
draft: false
---

`docker pull hectorqin/reader` 返回 **404**。而这正是我线上阅读服务所依赖的项目：一个只发邀请码的小书库，给我家人和几位付费读者用，每天在跑，上面还搭了一条我自己做的语音朗读管线，让「朗读」按钮念出来的是正经的神经网络音色。

404 还不是最有意思的部分。**2026-09-16**，同一个仓库被从零重新初始化——提交信息是 `chore: 初始化仓库`——十六天后，它已经有了 **233 个提交**、另一门语言、另一个端口、另一种数据库、另一个镜像仓库。我依赖的项目没有死，它是被**一个重写版替换了**，名字没变、地址没变、那 11,036 颗星也没变。

所以问题是：**怎么判断一个上游项目是死了，还是正在被重写？而当它以另一个应用的样子回来时，你该怎么做？**

## 先说我这边的情况，因为细节才是这件事的关键

这个应用是一个自托管阅读服务器。浏览器端是 hash 路由的 Vue SPA，服务端是 Kotlin + Spring Boot，配置按用户存成磁盘上的 JSON。而我专门为它的一条能力搭了一整套管线：**HTTP 语音合成**。阅读器会把朗读请求代理到你配置的 URL，也就是说音色不是应用自带的——我的指向 n8n 的两个 webhook，由它们去换取短期 Azure / Google 凭据并合成音频。语音选择器里那 9 个引擎是这么来的，不是应用自带的。

此外，应用的镜像外面还有我自己的一层 nginx 网关容器：对外落地页、社交分享卡片、中英切换、以及一层英文界面补丁，全在那里，不在应用镜像里。当初做这个决定是为了品牌，而正是它让整件事从头到尾都很平静——后面我会回到这点。

## 我去找升级路径的时候，发现了什么

三个选项，没有一个叫「升级」：

| 选项 | 它实际是什么 | 我为什么没选 |
|---|---|---|
| 官方镜像 | Docker Hub 上的 `hectorqin/reader` | 没了。仓库返回 404，`docker pull` 直接失败 |
| 还在维护的 fork | `changshengyu/reader`，版本 `2.5.4` | 基于最后一个完全开源的快照。它的朗读只用浏览器自带的 `speechSynthesis`，没有 HTTP 语音。它的整个历史里没有任何 `httpTTS` 字样，`/reader3/httpTts` 也是 404。我那 9 个引擎会全部消失 |
| 别人重新打包的镜像 | `liangnianzhi/reader.hectorqin:latest-20250525` | **这就是我现在跑的。** 唯一还能拉取、并且保留了 HTTP 语音的 3.x 构建。最后更新 2025-09-05，一共三个标签，发布者不是原作者 |

我把那个标签钉死，把数据目录打包备份，把源码镜像到自己的 Git 服务器上——至少我依赖的代码得存在于我能控制的地方。然后把那句决定未来任何镜像是否有资格的话写了下来：**它是否还讲我这条管线所实现的 TTS 协议？**

这就是当时的处境：一个跑在「官方仓库已被删除」的镜像上的服务，靠着某个陌生人重新打包的构建活着。

## 然后上游回来了

以下数据于 2026-10-06 直接取自 GitHub API 与镜像仓库：

| 指标 | 数值 |
|---|---|
| 仓库 | `hectorqin/reader` —— 活着，AGPL-3.0，默认分支 `main` |
| Star / Fork | 11,036 / 5,471 |
| 最后推送 | 2026-10-02 |
| 提交数 | **233** —— 而最早的一条是 `chore: 初始化仓库`，日期 **2026-09-16** |
| release / tag | **0 / 0** |
| Docker Hub | 仍然 **404** |

2021 年那段历史还在，但已经断开了。`main` 上是 233 个提交，最早一条的日期是 2026-09-16；旧分支仍然存在，但只剩**一个提交，日期 2026-06-22**，提交信息就是 `commit`，而它的 README 全文只有一个词——`deleted`。两个分支**没有共同祖先**——这就是「先清空、再重来」在 git 里的样子。

而这 233 个提交里的代码，不是我原来那个代码库的延续。它是另一个应用，只是恰好沿用了同一个名字：

| | 我现在跑的（3.x） | 上游现在的样子 |
|---|---|---|
| 运行时 | Kotlin，Spring Boot + Vert.x | TypeScript，Node |
| 端口 | 8080 | 5888 |
| 数据 | `/storage/data`，按用户存 JSON | `/data`，SQLite（`reader.db`） |
| 配置 | 环境变量 + 每用户 JSON 文件 | 管理后台界面，存进数据库 |
| 镜像 | Docker Hub，命名空间已删除 | `cnb.cool/hectorqin/reader:main` |
| 范围 | 只有书 | 书 **和** 电影、剧集、音乐、有声书、OPDS、OpenList、PWA、Android 客户端 |

表里有三个细节值得单独拎出来，因为它们各自都有一个后果：

**镜像仓库搬去了一个大多数人从没拉过东西的平台。** 官方 compose 里写的是 `image: ${READER_IMAGE:-cnb.cool/hectorqin/reader:main}`，旁边一行注释解释了原因：CI 只发布**分支和提交标签，刻意不发布 `:latest`**。也就是说默认配置跟的是一个会动的分支，唯一稳定的引用是一个 commit SHA。

**配置离开了环境变量。** 业务设置——HTTP 语音接口、扫描行为、登录有效期、对外 URL、允许的浏览器来源、WebDAV——现在都在管理后台里，持久化进数据库。文档说明：首次升级时「旧版业务环境变量会被导入一次」，并且已存在的数据库值永远优先。这是一次性的导入加一条优先级规则，正是你希望在副本上验证、而不是在线上实例上发现的东西。

**HTTP 语音回来了，只是换了个地方。** 新版本重新提供了 HTTP 语音上游的配置：合成 URL、token、voices URL、超时、缓存上限，还有连接测试和试听，全部在管理后台里配。而我的 n8n webhook 实现的协议，是我照着旧版构建自己设计的。这两个形状是否匹配，现在是一个需要用实验回答的问题，不是查文档就能回答的问题。

## 没人提醒你的部分：重写版是一个新应用

本能反应是把「上游复活了，还多了影音支持和 Android 客户端」读成「我等的升级到了」。并不是。一次重写会同时改变数据格式、部署面和配置模型——这意味着要做的是**一场带 schema 边界的迁移**，而不是 `docker pull` 加一次重启。

让我决定慢下来的，是项目自己的进度记录。不要拿厂商的营销页当状态报告，去看它自己写的东西，你会看到作者在列**没做完**的事情：

- 旧数据**自动迁移和首次接入向导都还没实现**；
- **镜像拉取与真实容器升级演练从未执行过**，因为构建机上没有跑 Docker 引擎。

这是一份诚实得罕见的文档，也是整个仓库里最有用的一个文件。重写的作者在告诉我：升级路径写好了，但没人走过。如果连他们都没走过，我当然不会在工作日下午、在我读者正在使用的实例上、在没有数据副本的情况下走去试。

## 我现在会把一个依赖交给线上服务之前做的事

1. **仓库健康不等于产物健康。** `pushed_at`、star、fork 说明**源码**还活着；对你要跑的那个确切标签执行 `docker pull`，才能说明**你部署的东西**还在。两个都要查，并读一遍 compose 看它到底指向哪个镜像仓库——我这个现在指向一个我本周之前从没打开过的域名。
2. **「没有 release、没有 tag」是一条部署事实。** 零 tag 意味着每次升级要么写死一个提交哈希，要么跟一个会动的分支。把那个 SHA 记进自己的笔记，上游不会为你记。
3. **你输不起的东西，就镜像一份——源码和镜像都要。** 一个钉在别人命名空间里的标签，是一种「那个陌生人账号别出事」的依赖。我活下来的 3.x 源码现在在我自己的 Git 服务器上，重写版也是。
4. **先比形状，再比功能。** 端口、挂载路径、变量名、数据库引擎、配置存放位置。如果这些大多变了，就把它当成一个新产品，按迁移来规划。
5. **你重建不了的部件，放在你控制不了的镜像之外。** 我的落地页、分享卡片、语言切换和翻译层，是同一条 stack 里的另一个容器。正因为如此，上游这次重写没有打扰到任何一个读者——应用镜像怎么换，都碰不到前门。
6. **读维护者的「还没做」清单，而不是功能清单。** 功能清单是销售页，未完成清单才是风险登记表。
7. **如果你把它当成对外服务来跑，读一下许可证。** 这个是 AGPL-3.0，如果你拿它给付费用户提供服务、而不是自己用，值得认真读一遍。
8. **「仓库被归档或被清空」不等于「项目被放弃」。** 有好几个月，「这个项目死了吗」的诚实答案是「是」——README 就写着 `deleted`，而且它说的是实话。然后有人重新开始了，换了门语言，而诚实的答案在名字不变的情况下变了。

## 结果

书库在跑，跑在我钉住的那个构建上，读者完全无感——这本来就是目标，而且它一直成立。变的是：「我们被困在一个没人管的镜像上」从一句含糊的担忧，变成了一个有方案的具体问题：

- 数据目录已复制，新镜像自带的 `backup` / `verify` / `restore` 命令先**对着副本**跑，确认没问题才碰线上卷；
- TTS 协议要拿我现有的 webhook 实测，因为一旦对不上，我丢掉的就是朗读这个功能；
- 旧镜像标签留着做回滚，而且无论走哪条路，源码现在都在我自己的服务器上；
- 前门不在讨论范围内，因为它在另一个容器里，换镜像够不着它。

真正有用的教训不是「要钉版本」——那个我早就做了。而是：**「这个项目还活着吗」有三个答案，而它们可以互相矛盾**。仓库可以活着，而镜像已被删除；README 可以写着 `deleted`，而重写版已经开工四周；版本号可以纹丝不动，而运行时、端口、数据库和配置模型在底下全换了一遍。这三件事要分别去查，否则你迟早会为一个「已经不再以你测试过的形态存在」的应用规划升级。

---

**你在替别人跑自托管服务，但不确定底下的东西还有没有人管？** 这正是我在做的审计：依赖与镜像健康度、升级可行性，以及随之而来的迁移方案。

[WhatsApp +60 12-797 2969](https://wa.me/60127972969) ·
[[email protected]](mailto:[email protected]?subject=Self-hosted%20dependency%20audit) ·
[hoelee.com](https://hoelee.com)

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "The Upstream Was Deleted, Then Came Back as a Rewrite"
description: "My production reading service runs a build whose official Docker Hub repo returns 404. Then upstream came back — 233 commits, a new language, no releases. How to tell dead from rewriting."
pubDate: 2026-10-06
category: devops
tags: ["docker", "self-hosting", "upstream", "maintenance", "registry"]
ogImage: "/og/the-upstream-was-deleted-then-came-back-rewritten.jpg"
banner: "/banners/the-upstream-was-deleted-then-came-back-rewritten.jpg"
draft: false
---

`docker pull hectorqin/reader` returns **404**. That is the project my
production reading service runs on: a small invite-only library I host for
my family and a handful of paying readers, serving books every day, with a
text-to-speech pipeline I built on top of it so the read-aloud button speaks
in proper neural voices.

The 404 is not the interesting part. On **2026-09-16** that same repository
was re-initialised from scratch — the commit message is `chore: 初始化仓库`,
"initialise repository" — and sixteen days later it carried **233 commits**,
a different language, a different port, a different database engine and a
different container registry. The project I depend on did not die. It was
**replaced by a rewrite** under the same name, the same URL and the same
11,036 stars.

So: **how do you tell whether an upstream project is dead — and what do you
do when it comes back as a different application?**

## The setup, because the details are why this mattered

The app is a self-hosted reading server. Hash-routed Vue SPA in the browser,
Kotlin and Spring Boot on the server, per-user JSON config on disk, and one
feature I had deliberately built a whole pipeline around: **HTTP
text-to-speech**. The reader proxies read-aloud requests to a URL you
configure, so the voices are not baked into the app — mine point at two n8n
webhooks that mint short-lived Azure and Google credentials and synthesise
the audio. Nine engines show up in the voice picker because of that, not
because the app shipped them.

I also run my own nginx gateway container in front of the app — the public
landing page, the social card, the bilingual switch and an English-UI layer
all live there rather than inside the app's image. That decision, made for
branding reasons, is the reason this whole incident stayed boring. I'll come
back to it.

## What I found when I went looking for an upgrade

Three options, and none of them was "upgrade":

| Option | What it actually is | Why I didn't take it |
|---|---|---|
| The official image | `hectorqin/reader` on Docker Hub | Gone. The repository returns 404; `docker pull` fails |
| The maintained fork | `changshengyu/reader`, pinned at `2.5.4` | Built on the last fully open snapshot. Its read-aloud uses the browser's own `speechSynthesis` only — no HTTP speech. No `httpTTS` string anywhere in its history, and `/reader3/httpTts` 404s on it. My nine engines would vanish |
| A rebuild published by someone else | `liangnianzhi/reader.hectorqin:latest-20250525` | **This is what I run.** The only still-pullable 3.x build that kept HTTP speech. Last updated 2025-09-05, three tags total, published by a user who is not the original author |

I pinned that tag, tarred up the data directory, and mirrored the source
into my own Git server so the code I depend on existed somewhere I control.
Then I wrote down the one line that decides whether any future image is even
a candidate: **does it still speak the TTS protocol my pipeline
implements?**

That was the state of things: a running service on an image whose official
home had been deleted, surviving as somebody's rebuild.

## Then upstream came back

Verified on 2026-10-06, straight from the GitHub API and the registry:

| Signal | Value |
|---|---|
| Repo | `hectorqin/reader` — alive, AGPL-3.0, default branch `main` |
| Stars / forks | 11,036 / 5,471 |
| Last push | 2026-10-02 |
| Commits | **233** — and the oldest is `chore: 初始化仓库`, dated **2026-09-16** |
| Releases / tags | **0 / 0** |
| Docker Hub | still **404** |

The 2021 history is there but disconnected. `main` carries 233 commits and
its oldest is dated 2026-09-16; the old branch survives as a **single commit
dated 2026-06-22** whose message is just `commit`, and whose entire README is
the one word `deleted`. The two branches share **no common ancestor** — which
is what a wipe followed by a fresh start looks like in git. (`created_at`
still reads 2021-08-13, because GitHub keeps that field when a repository is
re-populated; the history is what tells you the truth.)

And the code behind those 233 commits is not my app's codebase continued. It
is a different application that happens to keep the name:

| | What I run (3.x) | What upstream is now |
|---|---|---|
| Runtime | Kotlin, Spring Boot + Vert.x | TypeScript, Node |
| Port | 8080 | 5888 |
| Data | `/storage/data`, per-user JSON | `/data`, SQLite (`reader.db`) |
| Configuration | environment variables + per-user JSON files | the admin UI, persisted in the database |
| Container image | Docker Hub, namespace deleted | `cnb.cool/hectorqin/reader:main` |
| Scope | books | books **and** movies, series, music, audiobooks, OPDS, OpenList, PWA, an Android client |

Three details are worth pulling out of that table, because each one has a
consequence:

**The registry moved to a platform most of us have never pulled from.** The shipped compose file uses `image: ${READER_IMAGE:-cnb.cool/hectorqin/reader:main}`, and a comment explains that CI publishes **branch and commit tags and deliberately does not publish `:latest`**. So the default configuration tracks a moving branch, and the only stable reference is a commit SHA.

**Configuration left the environment.** Business settings — the HTTP speech endpoint, scan behaviour, login lifetimes, the public URL, allowed browser origins, WebDAV — now live in the admin UI and persist in the database. The docs note that on first upgrade "legacy business environment variables are imported once", and that existing database values always win. That is a one-shot import with a precedence rule, which is exactly the kind of thing you want to test on a copy instead of discovering on your live instance.

**HTTP speech came back — in a different place.** The new version documents an HTTP speech upstream again, with a synthesis URL, a token, a voices URL, a timeout and a cache limit, plus a connection test and audio preview, all configured in the admin UI. My n8n webhooks implement a protocol I designed against the old build's expectations. Whether those two shapes match is now an empirical question, not a documentation question.

## The part nobody warns you about: a rewrite is a new application

The instinct is to read "upstream is alive again, with media support and an
Android client" as *my upgrade arrived*. It isn't. A rewrite changes the
data format, the deployment surface and the configuration model at the same
time — which means the work is a migration with a schema boundary in it, not
a `docker pull` and a restart.

What convinced me to slow down was the project's own progress note. Read it
for yourself rather than taking a vendor's marketing page as the status
report, and you find the authors listing what is **not** done:

- legacy data **auto-migration and a first-run setup wizard are not implemented**;
- the **image pull and the real container upgrade drill were never executed**, because the Docker engine was not running on the build machine.

That is unusually honest documentation, and it is the single most useful
file in the repository. The people writing the rewrite are telling me the
upgrade path has been written but not walked. If they haven't walked it, I'm
certainly not walking it on the instance my readers are using, on a Tuesday,
without a copy of the data.

## What I do now, before trusting any dependency with a running service

1. **Repository health is not artifact health.** `pushed_at`, stars and
forks tell you the *source* is alive. `docker pull` on the exact tag you run
tells you the *thing you deploy* still exists. Check both, and read the
compose file to see which registry it names — mine now points at a domain I
had never opened before this week. 2. **"No releases, no tags" is a
deployment fact.** Zero tags means every upgrade is a commit hash or a
moving branch. Write the SHA in your own notes, because the upstream will
not. 3. **Mirror anything you cannot afford to lose — the source *and* the
image.** A pinned tag in someone else's namespace is a dependency on that
stranger's account staying alive. My surviving 3.x source sits in my own Git
server now, and so does the rewrite. 4. **Compare the shape before the
features.** Port, volume paths, variable names, database engine, where
configuration lives. If most of those moved, treat the new version as a new
product and plan a migration. 5. **Keep what you can't rebuild outside the
image you don't control.** My landing page, social card, language switch and
translation layer are a separate container in the same stack. That is why a
rewrite landing upstream did not disturb a single reader — the front door
cannot be broken by an app image swap. 6. **Read the maintainer's "not done
yet" list, not the feature list.** The feature list is a sales page; the
unfinished list is the risk register. 7. **Check the licence if you run it
as a service.** This one is AGPL-3.0, which is worth reading properly if you
host it for paying users rather than just yourself. 8. **An archived or
wiped repository is not the same as an abandoned project.** For months the
honest answer to "is this project dead?" was yes — the README said `deleted`
and meant it. Then somebody started again, in another language, and the
honest answer changed without the name changing.

## The result

The library is up, on the build I pinned, with readers unaffected — that was
always the goal, and it never stopped being true. What changed is that
"we're stranded on an orphaned image" became a known quantity with a plan:

- the data directory is copied, and the new image's own `backup` / `verify` / `restore` commands get run **against the copy** before anything touches the live volume;
- the TTS protocol gets tested against my existing webhooks, because if it doesn't match, the read-aloud feature is the thing I lose;
- the old image tag stays present for rollback, and I now have the source on my own server either way;
- the front door stays out of scope, because it lives in a different container and an image swap cannot reach it.

And the useful lesson is not "pin your versions", which I already did. It is
that **the question "is this project alive?" has three answers, and they can
disagree**: the repository can be alive while the image is deleted; the
README can say `deleted` while a rewrite is four weeks into existence; and
the version number can stay the same while the runtime, the port, the
database and the configuration model all change underneath it. Check each
one separately, or you will eventually plan an upgrade for an application
that no longer exists in the shape you tested.

---

**Running a self-hosted app for other people, and not sure whether the thing underneath it is still supported?** That's the kind of audit I do — dependency and image health, upgrade feasibility, and the migration plan that comes with it.

[WhatsApp +60 12-797 2969](https://wa.me/60127972969) ·
[[email protected]](mailto:[email protected]?subject=Self-hosted%20dependency%20audit)
· [hoelee.com](https://hoelee.com)

文章 06 / 11：Unraid 停止阵列时卡死的真相：Swapfile 锁住挂载点 — 附修复方案

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "Unraid 停止阵列时卡死的真相：Swapfile 锁住挂载点 — 附修复方案"
description: "Unraid 停止阵列时一直卡在卸载磁盘：btrfs RAID1 上的 8GB swapfile 通过 loop 设备锁住 /mnt/cache。根因分析、/boot/config/stop 陷阱、两个 User Scripts 彻底修复。"
pubDate: 2026-09-15
category: devops
blueprint: self-hosted-infrastructure
blueprintLayer: L1-compute-storage-network
tags: [unraid, swap, btrfs, zram, docker, user-scripts]
ogImage: /og/zh-unraid-stop-array-hangs-on-swapfile.jpg
banner: /banners/unraid-stop-array-hangs-on-swapfile.jpg
draft: false
---

我给 Unraid 服务器加了 8 GB swapfile。缓存池是 btrfs RAID1，而内核不允许
swapfile 直接建在多设备 btrfs 文件系统上，所以走了 loop 设备这个标准变通方
案。Swap 生效了，内存压力降了，一切正常——直到有一天我点了「停止阵列」。

WebUI 永远停在 **"Retry unmounting user shares…"**。缓存池就是卸不掉。
Docker 已停、虚拟机已关、共享已卸载——`umount /mnt/cache` 依然每 5 秒报一
次 *target is busy*，无限循环。唯一出路是重启。

这篇文章讲清楚三件事：为什么会这样；我中途踩的坑（`/boot/config/stop` 在
你点「停止」时**根本不会运行**）；以及两个脚本的修复方案——之后每次停止/启
动都干净利落。

## 问题本身（按你会搜索的方式来描述）

如果你是搜索进来的，你大概率在搜：

- *"unraid stop array stuck retry unmounting user shares"*
- *"umount /mnt/cache target is busy unraid"*
- *"unraid swapfile stop array hang"*
- *"unraid 停止阵列 卡住"*

同一个 bug。一句话版本：**swapfile（或它的 loop 设备）被内核 swap 子系统
持有打开，而 Unraid 的停止阵列流程里没有任何一步会先关掉 swap 再卸载磁
盘。** 挂载点忙，是因为内核自己在用它——除非有人执行 `swapoff`，否则它会
一直用下去。而没有人执行。

## 我试过的方案，以及为什么都失败

### 尝试 1：重启后再停阵列——照样卡死

意料之中。swapfile 配置在 `/boot/config/go` 里，每次开机都会重建 loop 设
备并启用 swap，每次停止都撞同一堵墙。至少这证明了问题是确定性的，不是偶
发进程卡住。

### 尝试 2：在 `/boot/config/stop` 里写 `swapoff`

这是最经典的陷阱，Unraid 官方文档让它看起来就是正确答案。`stop` 脚本的存
在意义就是「磁盘卸载前跑清理」——它确实会运行，**但只在完整关机或重启
时**。

我在自己的机器上验证过（Unraid 7.1.4）：`/boot/config/stop` 由
`rc.local_shutdown` 调用，而后者在 `rc.6` 里。在 WebUI 点**停止**走的是完
全不同的路径——`emhttpd` 触发一系列 *event*——`rc.6` 根本不参与。没有关
机，就没有 `stop` 脚本，就没有 `swapoff`。阵列停止照样卡死。

### 尝试 3：搞清楚 GUI「停止」到底做了什么

真正的答案在这里。Unraid 的事件调度器（`emhttp_event`）在阵列停止时按文
档化顺序执行钩子脚本：

```text
stopping → stopping_libvirt → stopping_docker → stopping_svcs
        → unmounting_disks → stopping_array → stopped
```

两个关键事实：

1. 卸载发生在 `unmounting_disks`。**任何 swapoff 必须在 `stopping_svcs`
   或更早触发**——比卸载早一个事件。
2. 这些事件正是 **User Scripts** 插件暴露出来的调度选项（「At Stopping of
   Array」= `stopping_svcs`，「At First Array Start only」=
   `disks_mounted`）。不需要写新代码——自动化接口本来就在那里。

### 隐蔽的部分：`/proc/swaps` 对 loop swap 说谎

这是让「看起来正确」的清理脚本也会失败的坑。我的 `/boot/config/go` 原来
是这样配置 swap 的（btrfs RAID1 标准写法）：

```bash
# /boot/config/go — 原始（停止时会卡的）配置
mkdir -p /mnt/cache
MOUNTPOINT=/mnt/cache
SWAPFILE=${MOUNTPOINT}/swapfile

# 等待缓存池挂载
while [ ! -f "${SWAPFILE}" ]; do
  sleep 1
done

# btrfs RAID1 拒绝直接 swapfile：loop 设备 + NOCOW
chattr +C "${SWAPFILE}"
LOOPDEV=$(losetup -f)
losetup -p 100 "${LOOPDEV}" "${SWAPFILE}"
swapon -p 1 "${LOOPDEV}"
```

现在看内核在活动状态下报告的内容：

```console
$ cat /proc/swaps
Filename                Type        Size      Used    Priority
/dev/loop0              partition   8388604   0       -2

$ swapon --show
NAME       TYPE SIZE USED PRIO
/dev/loop0 partition  8G   0B   -2
```

**文件名是 `/dev/loop0`，不是 `/mnt/cache/swapfile`。** 所以这样的清理脚
本：

```bash
# 错误 — 对 loop swap 静默失效
grep -q '/mnt/cache/swapfile' /proc/swaps && swapoff /mnt/cache/swapfile
```

……什么都匹配不到，什么都关不掉，然后你又一次卡在停止阵列。必须*先*解析
loop 设备——`losetup -j <文件>` 就是干这个的——再 `swapoff` 那个 loop 设
备，最后卸载它。ZRAM Compressed Memory 插件的源码注释几乎一字不差地记录
了同一个 bug：*"for loop-backed swap, /proc/swaps lists /dev/loopN, not
the image path — a plain grep never matches, so swapoff never ran and the
loop device kept the pool busy forever."*（loop 型 swap 在 /proc/swaps 里显
示为 /dev/loopN 而不是镜像路径——普通 grep 永远匹配不到，swapoff 从未执
行，loop 设备让缓存池永远忙碌。）

## 修复方案：两个 User Scripts

大多数 Unraid 机器已经装了 [User
Scripts](https://forums.unraid.net/topic/48286-plugin-ca-user-scripts/)
插件。创建两个脚本：

**脚本 1 —「Array Stop Swapoff」**——调度选 **At Stopping of Array**（在
`stopping_svcs` 触发，比卸载早一个事件）：

```bash
#!/bin/bash
# 先释放所有 swap，再只卸载【我们的】swapfile 背后的 loop 设备。
swapoff -a 2>/dev/null
for l in $(losetup -j /mnt/cache/swapfile 2>/dev/null | awk -F: '{print $1}'); do
  losetup -d "$l" 2>/dev/null
done
```

`losetup -j`（「哪个 loop 设备挂在这个文件上？」）是关键一步——它不受
`/proc/swaps` 命名谎言的影响。而且只卸载*我们自己的* loop 设备很重要：机
器上可能还有别的 loop 设备（挂载的 ISO、其他镜像），它们必须在停止阵列时
存活下来。

**脚本 2 —「Array Start Swapon」**——调度选 **At First Array Start
only**（在 `disks_mounted` 触发）：

```bash
#!/bin/bash
# 缓存池挂载后重建 loop swap。
[ -f /mnt/cache/swapfile ] || exit 0
LOOP=$(losetup -f)
losetup -p 100 "$LOOP" /mnt/cache/swapfile && swapon -p 1 "$LOOP"
```

为什么选「First Array Start only」而不是普通的「At Startup of the
Array」？因为开机时 `/boot/config/go` 也会运行并配置一次 swap。开机后的
*第一次*阵列启动由 `go` 负责；这个脚本真正管的是**不重启、纯 GUI 停止 →
启动的循环**——而 `go` 永远看不到这种场景。（在我的配置里，我把 swap 那段
从 `go` 里整个移除了，让 User Script 成为唯一事实来源——一处配置，两条路
径都覆盖。）

之后：停止 → 缓存池第一次尝试就干净卸载。启动 → swap 恢复。多次停止/启动
循环和重启后都验证通过。

## 我会做得不一样的地方

**1. 不要把 swap 放在已经快死的盘上。** 我的缓存池是一对后来确认在静默损
坏数据的 NVMe（SMART 全绿，读取却返回全零块）。8 GB swapfile 放在那里意
味着对我即将更换的硬件持续写入磨损——而且它正是在迁移期间卡住我阵列停止
的元凶。swap 的位置要匹配磁盘健康状况，而不是图方便。

**2. CPU 够用时，压缩内存 swap 优于磁盘 swap。** 这台机器的最终方案是
[ZRAM Compressed
Memory](https://forums.unraid.net/topic/196763-new-plugin-created-zram/)
插件：Tier-1 swap 放在 zstd 压缩的 RAM 块设备里。两个好处：

- **卡死在结构上变得不可能。** zram 设备不是挂载点，`umount /mnt/cache`
  永远看不见它，停止时无需释放任何东西。（如果你启用它的 Tier-2 磁盘
  swapfile，插件也自带正确的 `event/stopping` 钩子，用的正是
  `losetup -j` 这个修复。）
- **swap 换入换出是微秒级 CPU 操作而不是磁盘 I/O**——在现代多核 CPU 上，
  压缩相对 NVMe 延迟几乎免费，而且被换出的页面在 RAM 里省约 3:1 空间。

一个我用惨痛教训学到的注意事项：插件的自动大小（RAM 的 50%）在我 62 GB
内存、已用 90% 的机器上给了一个 **31.3 GB** 的 zram 设备。那个数字是*未压
缩上限*——实际 RAM 消耗 = 容量 ÷ 压缩比，所以装满的 31.3 GB zram 可能吃掉
10–30 GB 我本来就稀缺的空闲内存。在接近满载的机器上，zram 要固定且保守地
设置大小（我改成 8 GB——原来 swapfile 的大小——swappiness 150），让它缓冲
OOM 而不是和 OOM 抢内存。

**3. swap 只能推迟 OOM，治不了 OOM。** 我内存压力的诚实根因是虚拟机内存
分配，不是缺 swapfile。swap 买来了余量、让宿主机在峰值时活了下来——但真正
的修复是给虚拟机合理分配内存。

## 结果

停止阵列从「永远卡死、强制重启」变成「第一次尝试就干净卸载」，swap 在停止/
启动循环之间自动存活，无需人工干预。修复总量：两个 shell 脚本，跑在一个早
就装好的插件里。贵的部分是知道该把它们放在事件序列的**哪个位置**——以及
`/proc/swaps` 不会告诉你 loop swap 的真相。

如果你的 Unraid 阵列拒绝停止，而你配置过任何形式的 swap——文件、loop 或其
他——在 root shell 里执行 `swapoff -a` 会立刻释放它，诊断就此确认。然后在
下次重启抹掉你的记忆之前，把它自动化。

---

*你也有需要这种深度调试的自托管环境吗——Unraid、Docker、反向代理、NAS 迁
移？这是我的本行：[网站设计与开发](https://hoelee.com)是主业，自托管基础
设施是我钻研最深的领域。欢迎通过
[WhatsApp](https://wa.me/60127972969) 或
[邮件](mailto:[email protected]?subject=Unraid%20infrastructure%20help)联系
我。*

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "Why Unraid Hangs When You Stop the Array With a Swapfile — and the Fix"
description: "My Unraid array stop hung forever at unmounting disks because an 8 GB btrfs swapfile pinned /mnt/cache. Root cause, the /boot/config/stop trap, and two User Scripts that fix it."
pubDate: 2026-09-15
category: devops
blueprint: self-hosted-infrastructure
blueprintLayer: L1-compute-storage-network
tags: [unraid, swap, btrfs, zram, docker, user-scripts]
ogImage: /og/unraid-stop-array-hangs-on-swapfile.jpg
banner: /banners/unraid-stop-array-hangs-on-swapfile.jpg
draft: false
---

I added an 8 GB swapfile to my Unraid server — my cache pool is a btrfs RAID1,
and the kernel refuses a swapfile that lives directly on a multi-device btrfs
filesystem, so it goes through a loop device, the standard workaround. Swap
worked. Memory pressure dropped. Everything was fine… until the day I stopped
the array.

The WebUI sat on **"Retry unmounting user shares…"** forever. The cache pool
would not release. Docker was stopped, VMs were shut down, shares were
unmounted — and still, `umount /mnt/cache` failed with *target is busy*,
every five seconds, indefinitely. The only way out was a reboot.

This is the story of finding out why, the trap I fell into on the way
(`/boot/config/stop` does **not** run when you click Stop), and the two-script
fix that survives every stop/start cycle since.

## The problem, phrased the way you'd Google it

If you landed here from a search, you're probably typing one of these:

- *"unraid stop array stuck retry unmounting user shares"*
- *"umount /mnt/cache target is busy unraid"*
- *"unraid swapfile stop array hang"*
- *"unraid won't stop array after adding swap"*

Same bug. The short version: **a swapfile — or its loop device — is held open
by the kernel's swap subsystem, and Unraid's array-stop sequence has no step
that turns swap off before it unmounts disks.** The mount is busy because the
kernel itself is using it, and it will keep using it until someone runs
`swapoff`. Nobody does.

## What I tried, and why it failed

### Attempt 1: reboot, stop array again — hung again

Obviously. The swapfile was still configured in `/boot/config/go`, so every
boot recreated the loop device and enabled swap on it. Every stop hit the same
wall. At least this confirmed it was deterministic, not a one-off stuck
process.

### Attempt 2: put a `swapoff` in `/boot/config/stop`

This is the classic trap, and the Unraid docs make it look like the right
answer. The `stop` script exists precisely for "run cleanup before disks go
away" — and it does run, **but only during a full shutdown or reboot**.

I checked on my own box (Unraid 7.1.4): `/boot/config/stop` is invoked by
`rc.local_shutdown`, which is called from `rc.6`. Clicking **Stop** in the
WebUI goes through a completely different path — `emhttpd` fires a series of
*events* — and `rc.6` is never involved. No shutdown, no `stop` script, no
`swapoff`. The array stop still hung.

### Attempt 3: read what the GUI Stop actually does

This is where the real answer lives. Unraid's event dispatcher (`emhttp_event`)
runs hook scripts in a documented sequence when the array stops:

```text
stopping → stopping_libvirt → stopping_docker → stopping_svcs
        → unmounting_disks → stopping_array → stopped
```

Two facts matter:

1. The unmount happens at `unmounting_disks`. **Any swapoff must fire at
   `stopping_svcs` or earlier** — one event before the unmount.
2. These events are exactly what the **User Scripts** plugin exposes as
   schedule options ("At Stopping of Array" = `stopping_svcs`, "At Startup of
   the array" / "At First Array Start only" = `array_started` /
   `disks_mounted`). No new code needed — the automation surface already
   exists.

### The subtle part: `/proc/swaps` lies about loop-backed swap

Here's the gotcha that makes even a "correct-looking" cleanup script fail.
My `/boot/config/go` set swap up like this (the standard btrfs-RAID1 recipe):

```bash
# /boot/config/go — the original (broken-on-stop) setup
mkdir -p /mnt/cache
MOUNTPOINT=/mnt/cache
SWAPFILE=${MOUNTPOINT}/swapfile

# Wait for the cache pool to be mounted
while [ ! -f "${SWAPFILE}" ]; do
  sleep 1
done

# Btrfs RAID1 rejects direct swapfiles: loop device + NOCOW
chattr +C "${SWAPFILE}"
LOOPDEV=$(losetup -f)
losetup -p 100 "${LOOPDEV}" "${SWAPFILE}"
swapon -p 1 "${LOOPDEV}"
```

Now look at what the kernel reports while that's active:

```console
$ cat /proc/swaps
Filename                Type        Size      Used    Priority
/dev/loop0              partition   8388604   0       -2

$ swapon --show
NAME       TYPE SIZE USED PRIO
/dev/loop0 partition  8G   0B   -2
```

**The filename is `/dev/loop0`, not `/mnt/cache/swapfile`.** So a cleanup
script like this one:

```bash
# WRONG — silently does nothing for loop-backed swap
grep -q '/mnt/cache/swapfile' /proc/swaps && swapoff /mnt/cache/swapfile
```

…matches nothing, swaps nothing off, and you're back to the hang. You have to
resolve the loop device *first* — `losetup -j <file>` does exactly that — then
`swapoff` the loop device, then detach it. The ZRAM Compressed Memory plugin
documents this same bug in its source comment, almost word for word: *"for
loop-backed swap, /proc/swaps lists /dev/loopN, not the image path — a plain
grep never matches, so swapoff never ran and the loop device kept the pool
busy forever."*

## The fix: two User Scripts

[User Scripts](https://forums.unraid.net/topic/48286-plugin-ca-user-scripts/)
is already installed on most Unraid boxes. Create two scripts:

**Script 1 — "Array Stop Swapoff"** — schedule: **At Stopping of Array**
(fires at `stopping_svcs`, one event before the unmount):

```bash
#!/bin/bash
# Release ALL swap, then detach only the loop device backing OUR swapfile.
swapoff -a 2>/dev/null
for l in $(losetup -j /mnt/cache/swapfile 2>/dev/null | awk -F: '{print $1}'); do
  losetup -d "$l" 2>/dev/null
done
```

`losetup -j` ("which loop device is attached to this file?") is the key move —
it's immune to the `/proc/swaps` naming lie. And detaching *only* our loop
device matters: a box can have other loop devices (mounted ISOs, other images)
that must survive an array stop.

**Script 2 — "Array Start Swapon"** — schedule: **At First Array Start only**
(fires at `disks_mounted`):

```bash
#!/bin/bash
# Re-create the loop-backed swap after the cache pool mounts.
[ -f /mnt/cache/swapfile ] || exit 0
LOOP=$(losetup -f)
losetup -p 100 "$LOOP" /mnt/cache/swapfile && swapon -p 1 "$LOOP"
```

Why "First Array Start only" and not plain "At Startup of the Array"? Because
`/boot/config/go` also runs at boot and would set swap up a second time. The
*first* array start after boot is covered by `go`; this script matters for the
**GUI stop → start cycle with no reboot in between** — which `go` never sees.
(In my setup I moved the swap block out of `go` entirely so the User Script is
the single source of truth — one place, both paths.)

After this: Stop → cache unmounts cleanly on the first try. Start → swap is
back. Verified across multiple stop/start cycles and reboots.

## What I'd do differently

**1. Don't put swap on a disk that's already dying.** My cache pool is a pair
of NVMe drives I've since confirmed are silently corrupting data (reads
returning all-zero blocks despite clean SMART). An 8 GB swapfile there meant
constant write wear on hardware I was about to replace — and it was the very
thing pinching my array stops during the migration. Match swap location to
disk health, not convenience.

**2. Prefer compressed RAM over disk swap when you have the CPU for it.** The
endgame on this box was the [ZRAM Compressed
Memory](https://forums.unraid.net/topic/196763-new-plugin-created-zram/)
plugin: Tier-1 swap lives in a zstd-compressed RAM-backed block device. Two
wins:

- **The hang becomes structurally impossible.** A zram device is not a mount.
  `umount /mnt/cache` can never see it, so there's nothing to release at stop
  time. (The plugin *also* ships a proper `event/stopping` hook with the
  `losetup -j` fix, if you enable its Tier-2 disk swapfile.)
- **Swap-in/out is microseconds of CPU work instead of disk I/O** — on a
  modern many-core CPU, compression is nearly free relative to even NVMe
  latency, and you save ~3:1 on the RAM the swapped pages occupy.

One caveat I learned the hard way: the plugin's auto-size (50% of RAM) gave me
a **31.3 GB** zram device on a 62 GB host that was already 90% full. That
number is an *uncompressed ceiling* — real RAM cost is capacity ÷ compression
ratio, so a full 31.3 GB zram could eat 10–30 GB of the very RAM I was short
of. On a nearly-full host, size zram fixed and modest (I went to 8 GB, the old
swapfile size, swappiness 150) so it buffers OOM instead of competing with it.

**3. Swap delays OOM; it doesn't cure it.** The honest root cause of my memory
pressure was VM RAM budgets, not a missing swapfile. Swap bought headroom and
kept the host alive during spikes — but the real fix was right-sizing guests.

## The result

Array stop went from *"hangs forever, forced reboot"* to *clean unmount on the
first attempt*, and swap now survives stop/start cycles without manual
intervention. Total fix: two shell scripts in a plugin that was already
installed. The expensive part was knowing **where** in the event sequence to
put them — and that `/proc/swaps` won't tell you the truth about loop-backed
swap.

If your Unraid array refuses to stop and you have any swap configured — file,
loop, or otherwise — `swapoff -a` from a root shell will release it
immediately, and you've confirmed the diagnosis. Then automate it before the
next reboot wipes your memory of how.

---

*Run a self-hosted stack that needs this kind of debugging — Unraid, Docker,
reverse proxies, NAS migrations? I do this for a living: [website design &
development](https://hoelee.com) is my main work, and self-hosted
infrastructure is where I go deep. Reach me on
[WhatsApp](https://wa.me/60127972969) or
[email](mailto:[email protected]?subject=Unraid%20infrastructure%20help).*

文章 07 / 11：用 CDP 自动化上传百兆视频到 YouTube：跨越 DevTools 内存墙的 Docker 归档法

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "用 CDP 自动化上传百兆视频到 YouTube：跨越 DevTools 内存墙的 Docker 归档法"
description: "了解如何使用 Docker Archive API 将大型视频直接送入容器文件系统，再通过 CDP 秒级挂载，彻底避免将百兆文件经由 WebSocket 慢速传输挤爆浏览器。"
pubDate: 2025-08-22
updatedDate: 2026-10-22
category: devops
tags: ["docker", "cdp", "playwright", "youtube", "portainer", "automation"]
ogImage: "/og/zh-uploading-large-videos-to-youtube-via-cdp.jpg"
banner: "/banners/uploading-large-videos-to-youtube-via-cdp.jpg"
draft: false
---

在前端自动化与爬虫开发中，通过无头浏览器或 Chrome DevTools Protocol（CDP）来提交表单、抓取数据或上传几百 KB 的图片通常十分顺畅。然而，当你试图在自动化流程中将一个 100MB 以上的高清 1080P 视频通过远程 CDP 自动上传到 YouTube 时，常规的浏览器自动化方案几乎一定会崩溃。

本文记录这个让远程自动化频繁掉线的内存陷阱，以及如何通过轻量优雅的 Docker 归档模式（Docker Archive Pattern）实现百兆大视频的稳定秒级注入。

> **TL;DR** —— 不要把上百兆字节塞进 CDP 的 WebSocket。Docker Engine API 的 `PUT /containers/{id}/archive?path=…`（这里走 Portainer）把文件打成 tar 直接解进 Chrome 容器的文件系统，再用 `DOM.setFileInputFiles` 把**容器内路径**交给网页。千兆局域网下 130 MB 落盘 1.89 秒。这一篇是 [全自动 MV 生成流水线](/posts/zh/how-i-built-an-automated-music-video-pipeline/) 的落盘那一半；渲染层相关的转场与低频抽动排查见 [排查 Remotion 视频抽搐](/posts/zh/debugging-remotion-video-jitter/)。

---

## 1. 什么时候该用哪种文件上传方案

在动手写代码前，先理清四种常见方案的适用边界：

| 方案 | 适合场景 | 权衡与取舍 |
|---|---|---|
| **Playwright `setInputFiles()`** | 同一台机器上的本地 Chrome 实例 | 挂接系统原生文件选择器。跨远程 WebSocket 传输大文件极易内存溢出或超时。 |
| **CDP `DOM.setFileInputFiles`** | 文件已经存在于 Chrome 宿主机/容器内部 | 耗时仅 2 毫秒，网络零内存开销，但本身不具备跨机器传文件的能力。 |
| **Docker Archive API (`/archive`)** | 把文件注入已有容器，免配 SSH/NFS/卷挂载 | 原生 tar 流式导入，快速自包含，适合带外落盘。 |
| **YouTube Data API v3** | 后台纯无头服务，拥有 Google Cloud 服务凭据 | 支持官方断点续传，但每日免费配额极紧（每次上传消耗 1600 单位，每天最多传 6 个视频），且很多 YouTube Studio 独占设置（如特定多语种 CC 字幕、音频轨道）API 无法完全覆盖。 |

### 为什么这里选择 CDP 而不是官方 Data API？
YouTube Data API 默认每天只有 10,000 单位配额，上传一个视频扣除 1,600 单位，一天上传 6 个视频就会触发限流配额耗尽。此外，官方 API 对外挂多语种字幕的授权和上传要求频繁更新的 OAuth2 授权圈子。在已有登录态的自托管 Chrome 容器中通过 CDP 驱动，不仅零配额限制，还能 100% 保持与人工在 YouTube Studio 点击完全一致的行为体验。

---

## 2. 50MB 远程浏览器协议陷阱

在控制本地 Chrome 时，Playwright 或 Puppeteer 使用 `page.setInputFiles("input[type=file]", local_path)`。浏览器直接从当前机器的物理磁盘中读取文件句柄，操作极其平稳。

但在自托管私有基础设施或生产部署中，为了让 YouTube、电商等带有登录态的账号 7x24 小时保持在线且不污染日常桌面，我们通常将 Chrome 运行在 Synology NAS 或 Linux 服务器的 Docker 容器中。

当我们在外部工作站或 CI 流水线上运行脚本，试图跨网络通过远程 CDP 将视频推给容器里的浏览器时，问题接踵而至：
1. Playwright 会尝试通过 WebSocket 协议将巨大的二进制文件转为 Base64 或分片推流；
2. 庞大的数据帧直接挤爆 WebSocket 协议栈缓冲区，两端内存激增；
3. 远程浏览器随之报出 `Target closed` 崩溃退出，或者由于帧超时直接断开连接。

即使你费尽心思调大 WebSocket 参数，跨协议在网络上传输上百兆视频依然极其脆弱。

---

## 3. 破局思路：本地文件路径直接注入

核心思路的转变其实非常直观：**如果视频文件本身就已经保存在容器内部的磁盘里，Chrome 根本不需要走 WebSocket 网络把数据传过来。**

当 `/config/Downloads/video.mp4` 已经落在容器文件系统内时，CDP 提供了一个原生且完全不消耗协议内存的底层指令：

```python
cdp = context.new_cdp_session(page)
doc = cdp.send("DOM.getDocument")
file_node = cdp.send("DOM.querySelector", {
    "nodeId": doc["root"]["nodeId"],
    "selector": "input[type='file']",
})

# 耗时仅需 2 毫秒，且在浏览器 WebSocket 上传输 0 字节：
cdp.send("DOM.setFileInputFiles", {
    "files": ["/config/Downloads/video.mp4"],
    "nodeId": file_node["nodeId"],
})
```

于是，真正的工程问题转变为：**如何无需繁琐的 SSH 密钥或挂载庞大卷，就能将一个 130MB 的文件在 2 秒内送入容器内部？**

---

## 4. Portainer Docker Archive 归档注入法（生产强化版实现）

每个 Docker 守护进程原生暴露了一个容器归档接口：
`PUT /containers/{id}/archive?path=/path/in/container`

在 Synology NAS 或 unRaid 环境下，Portainer 会将此 Docker socket API 封装在统一的 API Key 鉴权之后：

`PUT http://<host>:9000/api/endpoints/2/docker/containers/{container_id}/archive?path=/config/Downloads`

以下是带有完整异常捕获、入参校验与安全权限的生产级 Python 脚本：

```python
import io
import os
import tarfile
import urllib.error
import urllib.request

PORTAINER_KEY = os.environ.get("PORTAINER_API_KEY", "")
PORTAINER_BASE = "http://192.168.1.1:9000/api/endpoints/2/docker"
CONTAINER_ID = "2dc0fb6a11c7"  # 目标 Chrome 容器 ID

def stage_file_to_container(local_path: str, dest_filename: str) -> None:
    if not PORTAINER_KEY:
        raise ValueError("未检测到 PORTAINER_API_KEY 环境变量")
    if not os.path.exists(local_path):
        raise FileNotFoundError(f"本地文件不存在: {local_path}")

    size = os.path.getsize(local_path)
    tar_stream = io.BytesIO()

    # 在内存中打包 tar（对于拥有 16GB+ RAM 的工作站，100-200MB 媒体文件完全安全可控）
    with tarfile.open(fileobj=tar_stream, mode="w") as tar:
        with open(local_path, "rb") as f:
            ti = tarfile.TarInfo(name=dest_filename)
            ti.size = size
            # 0o644：文件属主可读写，其他组只读（严禁赋予全局可写 0o666/0o777）
            ti.mode = 0o644
            tar.addfile(ti, f)

    tar_stream.seek(0)
    data = tar_stream.read()

    endpoint = f"{PORTAINER_BASE}/containers/{CONTAINER_ID}/archive?path=/config/Downloads"
    req = urllib.request.Request(
        endpoint,
        data=data,
        headers={
            "X-Api-Key": PORTAINER_KEY,
            "Content-Type": "application/x-tar",
        },
        method="PUT",
    )

    try:
        with urllib.request.urlopen(req, timeout=120) as resp:
            if resp.status not in (200, 204):
                raise RuntimeError(f"Archive PUT 请求失败，HTTP 状态码: {resp.status}")
        print(f"成功将 {dest_filename} ({size / (1024 * 1024):.2f} MB) 注入容器！")
    except urllib.error.HTTPError as e:
        body = e.read().decode("utf-8", errors="replace")
        raise RuntimeError(f"Portainer 接口返回错误 {e.code}: {body}") from e
```

### 运维注意事项与安全清单
1. **客户端 RAM 内存开销**：虽然 Chrome 端彻底消除了内存尖刺，但 `io.BytesIO()` 会在当前 Python 脚本进程中驻留整份 tar 镜像。如果要处理数 GB 的大型视频归档，应先将 `.tar` 写入临时文件，并通过 `requests` 或 `urllib3` 以流式分块形式上传。
2. **目标目录必须预先存在**：Docker Archive API 是直接解包到 `?path=` 指向的目录中。如果目标容器内部缺失 `/config/Downloads` 目录，接口会直接返回 404/500，而不会自动创建父级目录。
3. **API Key 与 CDP 端口暴露风险**：Portainer API Key 与未鉴权的 Chrome DevTools 端口（默认 `9222` 或 `9242`）都拥有宿主或容器等效的最高执行权限。切勿将这两个端口直接映射到公网 IP（`0.0.0.0`），务必严格限制在内网隔离桥或套在双向 TLS / Authentik SSO 之后。

### 实测性能读数
在千兆局域网环境下实测：
- **128.85 MB 视频成片（.mp4）**：从发起请求到容器解包就绪，耗时 **1.89 秒**（固态硬盘存储）；
- **Chrome 容器内存占用**：所有浏览器标签页均观察不到额外内存峰值；
- **WebSocket 流量**：DevTools 协议通道 0 字节负载。

---

## 5. YouTube Studio 上传流程自动化

当视频和自定义封面都已安静地躺在容器内部的 `/config/Downloads/` 下，Playwright 只需要通过 CDP 驱动页面向导：

1. **挂载主视频**：获取主上传框 `input[type='file']`，调用 `DOM.setFileInputFiles` 传入容器视频路径；
2. **填充标题与描述**：定位 contenteditable 输入区域，自动输入预生成的 SEO 标题与详细简介；
3. **挂载封面缩略图**：通过 `DOM.getDocument` 重新拉取动态注入的 DOM 树，获取 `input[type='file']#file-loader` 节点，挂载 `/config/Downloads/cover.jpg`；
4. **受众选择**：勾选 `VIDEO_MADE_FOR_KIDS_NOT_MFK`；
5. **跳过向导检查**：连续点击 `#next-button` 进入发布选项；
6. **公开可见性与发布**：选中 `PUBLIC`，点击 `#done-button` 即可完成发布。

因为文件在容器磁盘上本地可读，YouTube Studio 以最饱满的网络带宽直推谷歌服务器，全程无一次断流或卡顿。

---

## 6. 核心总结

- **严禁在 CDP 的 WebSocket 通道传输大型文件**：远程浏览器不是为了在内存中缓冲几百兆媒体流而设计的；
- **带外落盘先行**：使用 Docker Archive PUT 接口，利用 Docker 原生机制以极高速度直接将文件推入容器文件系统；
- **容器内部路径秒挂载**：通过 CDP 的 `DOM.setFileInputFiles` 直接指定容器路径，实现零内存损耗、零失败率的文件绑定。

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "Uploading 100MB+ Videos to YouTube via CDP: The Docker Archive Pattern"
description: "Learn how to stage large video files inside a Dockerized Chrome container using the Docker Archive API, then attach them through CDP without sending the entire file over the browser WebSocket."
pubDate: 2025-08-22
updatedDate: 2026-10-22
category: devops
tags: ["docker", "cdp", "playwright", "youtube", "portainer", "automation"]
ogImage: "/og/uploading-large-videos-to-youtube-via-cdp.jpg"
banner: "/banners/uploading-large-videos-to-youtube-via-cdp.jpg"
draft: false
---

Automating web uploads via headless browser or Chrome DevTools Protocol (CDP) works seamlessly for small JSON payloads, forms, and occasional screenshots. But when you attempt to upload a 100MB+ 1080p video file to YouTube through a remote browser session, standard browser automation approaches quickly collapse.

Here is the exact trap that breaks remote CDP uploads, why browser memory limits fight you, and the lightweight Docker Archive pattern that makes large-file uploads rock-solid.

> **TL;DR** — Do not push megabytes through the CDP WebSocket. `PUT /containers/{id}/archive?path=…` on the Docker Engine API (here via Portainer) tars the file straight into the Chrome container's filesystem, then `DOM.setFileInputFiles` hands the *container path* to the page input. 130 MB staged in 1.89 s on a gigabit LAN. This is the staging half of our [music-video pipeline](/posts/how-i-built-an-automated-music-video-pipeline/); the rendering and visual transition tuning is covered in [debugging Remotion video jitter](/posts/debugging-remotion-video-jitter/).

---

## 1. When to Use Which File-Upload Strategy

Before jumping into implementation, it helps to understand where each upload technique belongs:

| Method | Best For | Trade-offs |
|---|---|---|
| **Playwright `setInputFiles()`** | Local Chrome instances on the same host | Native OS picker hook. Fails or spikes memory over remote WebSockets. |
| **CDP `DOM.setFileInputFiles`** | Files already on the Chrome host's filesystem | Instantaneous (2ms), zero network memory overhead, but cannot move files across machines. |
| **Docker Archive API (`/archive`)** | Staging files into an existing remote container without SSH/NFS | Direct tar stream to Docker daemon; fast and self-contained. |
| **YouTube Data API v3** | Headless backend services with Google Cloud credentials | Direct resumable uploads, but daily quota limits are tight (1600 units per upload = 6 videos/day max on free tier) and cannot access newer YouTube Studio features (custom audio tracks, specific CC localization workflows). |

### Why CDP Over YouTube Data API?
YouTube's official Data API imposes a 10,000-unit daily default quota limit. A single video upload consumes 1,600 units, meaning you hit quota exhaustion after just 6 videos a day. Furthermore, uploading multi-language `.srt` tracks and custom video localization through the Data API requires complex OAuth2 consent scopes that expire frequently. 

By driving an already-authenticated Chrome instance in Docker via CDP, we operate under standard browser sessions with zero quota walls.

---

## 2. The 50MB Remote Browser Pitfall

When controlling a local Chrome instance, Playwright or Puppeteer attaches files using `page.setInputFiles("input[type=file]", local_path)`. The browser reads the file from your local disk and passes the handle directly to the OS file picker.

However, in a self-hosted on-premise or production setup, you frequently run Chrome inside an isolated Docker container on a NAS or server to keep your authenticated session alive 24/7 without exposing your daily desktop.

When your automation script runs on a different workstation or runner and tries to push a video over a remote CDP connection:
1. Playwright attempts to base64-encode or stream the file buffer over the WebSocket protocol.
2. The WebSocket frame exceeds protocol buffers or consumes massive memory on both ends.
3. The remote browser crashes with `Target closed`, or CDP drops the socket due to frame size timeouts.

Even if you increase the WebSocket chunk size, pushing a 130MB file over CDP is excruciatingly slow and fragile.

---

## 3. The Reframe: Local File Injection

The key realization is simple: **Chrome inside the container doesn't need the file pushed over the wire if the file already exists on its internal filesystem.**

If the file `/config/Downloads/video.mp4` already sits inside the container, CDP provides a native, zero-memory command that requires no file transfer at all:

```python
cdp = context.new_cdp_session(page)
doc = cdp.send("DOM.getDocument")
file_node = cdp.send("DOM.querySelector", {
    "nodeId": doc["root"]["nodeId"],
    "selector": "input[type='file']",
})

# Takes 2 milliseconds and transfers ZERO bytes over the browser socket:
cdp.send("DOM.setFileInputFiles", {
    "files": ["/config/Downloads/video.mp4"],
    "nodeId": file_node["nodeId"],
})
```

The real question becomes: **How do you get a 130MB file into the container filesystem without SSH keys, volume mounts, or slow SFTP?**

---

## 4. The Portainer Docker Archive API (Hardened Implementation)

Every Docker daemon exposes a container archive endpoint:
`PUT /containers/{id}/archive?path=/path/in/container`

If you manage your Docker host with Portainer (standard in Synology and unRaid homelabs), Portainer proxies this Docker socket endpoint behind an API key:

`PUT http://<host>:9000/api/endpoints/2/docker/containers/{container_id}/archive?path=/config/Downloads`

Here is a tested, production-ready staging script with proper error handling, parameter validation, and secure file modes:

```python
import io
import os
import tarfile
import urllib.error
import urllib.request

PORTAINER_KEY = os.environ.get("PORTAINER_API_KEY", "")
PORTAINER_BASE = "http://192.168.1.1:9000/api/endpoints/2/docker"
CONTAINER_ID = "2dc0fb6a11c7"  # Target Chrome container ID

def stage_file_to_container(local_path: str, dest_filename: str) -> None:
    if not PORTAINER_KEY:
        raise ValueError("PORTAINER_API_KEY environment variable is not set")
    if not os.path.exists(local_path):
        raise FileNotFoundError(f"Local source file not found: {local_path}")

    size = os.path.getsize(local_path)
    tar_stream = io.BytesIO()

    # Build tar in memory (safe for 100-200MB payloads on workstations with 16GB+ RAM)
    with tarfile.open(fileobj=tar_stream, mode="w") as tar:
        with open(local_path, "rb") as f:
            ti = tarfile.TarInfo(name=dest_filename)
            ti.size = size
            # 0o644: owner read/write, group/others read-only (avoid world-writable 0o666/0o777)
            ti.mode = 0o644
            tar.addfile(ti, f)

    tar_stream.seek(0)
    data = tar_stream.read()

    endpoint = f"{PORTAINER_BASE}/containers/{CONTAINER_ID}/archive?path=/config/Downloads"
    req = urllib.request.Request(
        endpoint,
        data=data,
        headers={
            "X-Api-Key": PORTAINER_KEY,
            "Content-Type": "application/x-tar",
        },
        method="PUT",
    )

    try:
        with urllib.request.urlopen(req, timeout=120) as resp:
            if resp.status not in (200, 204):
                raise RuntimeError(f"Archive PUT failed with HTTP {resp.status}")
        print(f"Staged {dest_filename} ({size / (1024 * 1024):.2f} MB) in container successfully.")
    except urllib.error.HTTPError as e:
        body = e.read().decode("utf-8", errors="replace")
        raise RuntimeError(f"Portainer Archive API error {e.code}: {body}") from e
```

### Operational Caveats & Security Checklist
1. **Client RAM Footprint**: While Chrome experiences zero memory pressure, `io.BytesIO()` buffers the 130MB tar archive in the Python process memory. For multi-gigabyte video dumps, write the `.tar` to a temporary file and stream it using `requests` or `urllib3` chunked uploads.
2. **Pre-existing Destination Directory**: The Docker Archive API extracts *into* the directory specified by `?path=`. If `/config/Downloads` does not exist in the container image, the API returns HTTP 404/500 rather than creating the parent folders.
3. **API Key & CDP Exposure**: Both the Portainer API key and the unauthenticated Chrome DevTools port (default `9222` or `9242`) grant root-level execution capability. Never bind these ports to public IPs (`0.0.0.0`). Keep them strictly on an isolated internal bridge or behind mutual-TLS / Authentik SSO.

### Measured Performance
On a local 1 Gbps LAN:
- **128.85 MB video file (`.mp4`)**: Transferred and extracted inside the container in **1.89 seconds** (single run, warm container, SSD target on Synology NAS).
- **Chrome container memory consumption**: 0 MB RAM spike across all browser tabs.
- **WebSocket overhead**: 0 bytes transferred over DevTools Protocol.

---

## 5. YouTube Studio Upload Flow Automation

Once the video and custom thumbnail are staged in `/config/Downloads/`, the Playwright script navigates to `https://studio.youtube.com/`, triggers the upload modal, and executes the wizard:

1. **Attach Video**: Call `DOM.setFileInputFiles` on the primary `input[type='file']`.
2. **Title & Description**: Click `div[contenteditable="true"][aria-label*="title"]` and type the title; use `.fill()` on the description area.
3. **Attach Thumbnail**: Refresh the DOM with `DOM.getDocument` to discover the newly mounted `input[type='file']#file-loader` and inject `/config/Downloads/cover.jpg`.
4. **Audience Setting**: Check `VIDEO_MADE_FOR_KIDS_NOT_MFK`.
5. **Wizard Steps**: Click `#next-button` through Elements and Checks.
6. **Visibility**: Select `PUBLIC` and click `#done-button`.

Because the file was already on disk inside the container, YouTube Studio reads the stream locally, achieving maximum upload bandwidth without dropping a single frame.

---

## 6. Summary

- **Never stream large binaries over WebSocket CDP**. Remote browsers are not designed to buffer multi-hundred-megabyte payloads in memory.
- **Stage files out-of-band**: Use Docker Archive PUT to drop files onto the container filesystem in seconds.
- **Inject via filesystem path**: Call `DOM.setFileInputFiles` with the container path for instantaneous, zero-overhead file attachments.

文章 08 / 11：如何在押注前验证一个开源依赖：我用的六项检查

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "如何在押注前验证一个开源依赖：我用的六项检查"
description: "六项检查，专门抓出「文档说得漂亮、代码里其实没有」的仓库：用 grep 找功能、读文档头部标注的目标分支、看清付费档位门槛，以及收款通道在不在你的国家能用。"
pubDate: 2026-09-29
category: devops
blueprint: sme-digital-suite
blueprintLayer: L4-managed-ops-handover
tags: ["open-source", "due-diligence", "oidc", "self-hosting", "docker"]
ogImage: "/og/zh-vetting-an-open-source-dependency-before-you-bet-on-it.jpg"
banner: "/banners/vetting-an-open-source-dependency-before-you-bet-on-it.jpg"
draft: false
---

有人给我一份 1306 行的伙伴系统架构规格书。读起来像一份计划：前后一致、交叉引用、语气笃定。它指定一个自托管的开源引擎作为归因与佣金层，并且假定了那个引擎的三件事——它支持 OIDC，所以我现有的身份提供方能登录伙伴；它的佣金规则覆盖业务需要的五种形态；它的白标功能已经上线。

十分钟的 `curl` 加 `grep` 之后，三件里有两件是假的，第三件只对了一半。

这没什么稀奇。README 描述的是**产品**，仓库描述的是**软件**。这两者之间的落差，正是集成项目死掉的地方——而且在写任何代码之前，测量这个落差很便宜。

## 问题，写成我真的会去搜的那句话

**我怎么知道一个开源项目真的做了它文档宣称的事，然后才在它上面开始建东西？**

规格书是**关于别的软件**的一种主张。规格书越漂亮，它的假设就越像事实——这一份引用了来源、定义了稳定 ID、还画了一张「真源矩阵」。它唯一没做的事，就是核对链条下游那个引擎到底有没有实现被指派给它的功能。所以在安装任何东西之前，我去找那几个具体功能。

## 我一开始试的，以及为什么不够

**一、把 README 从头读到尾。** 它把流水线卖得很好：`click → identity → event → attribution → commission → payout`。但它一次都没说这个应用讲哪种认证协议。对一个你**必须要有**的功能来说，沉默不是中立——只是当页面其他部分都很有说服力时，一个「没被提到」很容易被眼睛滑过去。

**二、上网搜。** 好几条查询都没返回能用的结果。这个项目才几个月大，SEO 面很薄。对软件来说这没关系——源码**就是**文档。只是意味着阅读必须在仓库里进行，而不是在博客文章里。

**三、看到一份 78 KB 的功能设计文档，就假定功能已上线。** 它标着「FINAL, post-review」，列了三位评审，还带着日期。我把它当成了证据。这是三个推断里最糟的一个，也是最容易核对的一个。

## 真正定案的六项检查

### 1. 先看仓库的生命体征，再看任何功能宣称

```bash
curl -s -H "User-Agent: hermes" https://api.github.com/repos/<owner>/<repo> \
  | grep -E '"created_at"|"pushed_at"|"stargazers_count"|"forks_count"|"open_issues_count"|"archived"|"spdx_id"'
```

返回的是：MIT、创建于 2026-04-23、最后推送 2026-09-10、8 个 star、3 个 fork、15 个 open issue、未归档。翻译过来就是：**真实、还在动、但非常小**——单一厂商，而且它同时卖托管档。现在我在开始评它的功能之前，先知道了我愿意把多少生意压在它身上。

### 2. 用 grep 去找功能，而不是用读的

```bash
grep -in 'oidc\|sso\|saml' README.md ARCHITECTURE.md docs/*.md | wc -l
```

零命中。这个应用用的是它自己的 magic-link 会话和按身份签发的 API key；它根本不讲 OIDC。就这一行输出，砍掉了计划里的一个阶段——「把门户挂到我们 SSO 后面」——否则这一步会在几周后才浮现，而且是在集成到一半、伙伴已经被邀请进来的时候。**用读的方式去找一个找不到的功能，就是你失去一个下午的方式；grep 让「不存在」这件事变得响亮。**

### 3. 读功能文档最上方那行「目标分支」

```bash
head -5 docs/white-label-custom-domains.md
```

`Target branch: multi-tenant`。这份文档描述的是**另一个分支**上的设计，不是你默认会安装的那个分支——它写的是在途工作，所以功能不在你会装下来的代码里。一份有评审、有日期、看起来已经完成的设计文档，完全可能描述着尚未上线的东西。这不是不诚实，设计文档本来就是这个样子。

### 4. 读数据模型，不要读功能清单

佣金那一节只列了两种规则类型——`percent`（带续期标记）与 `fixed`。业务规格书列了五种。这个落差不是 bug，而是我必须在**配置层**做的建模工作，不能指望产品自带：

| 规格书假设 | 引擎实际有的 |
|---|---|
| 首单奖励 | 对首个事件用一条 `fixed` 规则 |
| 按事件区分（`lead` vs `invoice_paid`） | 一个 campaign 一条规则——所以：两个 campaign |
| 续期佣金 | `percent` + `recurring: true` |
| 百分比 | `percent` ✅ |
| 固定金额 | `fixed` ✅ |

两种规则原语，五种行为——能做，但前提是你**一开始**就照这个设计，而不是在实现途中才发现。

### 5. 看清你需要的那项功能被锁在哪一档

同一个技术栈里还有个数据库工具，它的版本对照表很直白：

| 我需要的功能 | Community（免费） | Enterprise（付费） |
|---|---|---|
| API token，好让我自动化 | ✅ | ✅ |
| SSO / OIDC | ❌ | ✅ |
| 页面设计器、隐藏品牌、双因素 | ❌ | ✅ |

我想给「伙伴看到的那个门」用的东西（SSO）在商业门后面；而我想用来做自动化的东西（API token）是免费的。在围绕它做设计**之前**，先弄清每项需求落在墙的哪一侧。

### 6. 核对收款通道在你的国家能不能用

它的打款模型是 `stripe_connect | manual`，而自托管模式被写成「operator manages out-of-band」——也就是钱由你自己搬。Stripe Connect 不是马来西亚企业付钱给伙伴的方式，所以就是这一行决定了整个部署形态：自托管模式、手工打款、事件由我自己的订单系统推送过去，而不是依赖它内置的支付集成。这一行从来不会出现在营销页上，而它比任何功能都重要。

### 然后：在你需要之前，先问「兜底方案」

「如果我需要的功能不在，怎么办？」针对缺失的 SSO，答案是：不要把登录做进应用里，把登录放到应用**前面**。authentik 的 proxy provider 存在的意义就是保护「不支持 OIDC、SAML 或 LDAP 等原生认证协议的应用」，并且向上游注入 `X-authentik-username` 和 `X-authentik-groups` 头，于是反向代理就能承担准入判断。这就把「不支持 SSO」从一条否决理由，变成了一个下午的配置工作。

## 如果重来，我会怎么做

- **先写需求清单，再去找它。** 五行——**我**需要的功能——然后就搜这几个。从 README 的功能列表出发，等于用厂商的考卷给软件打分。
- **把「文档存在」和「功能上线」当成两件不同的事。** 去读那行目标分支。
- **用 grep，不要用读的。** 证明「有」很便宜；你要猎的是「没有」。
- **在决定之前先定兜底**，这样一个缺失的功能就只是一次设计调整，而不是一场意外。
- **把 commit 钉住。** 这个项目的 README 自己写着 API「stable but unversioned」，而且仓库里没有任何 OpenAPI 描述。没有可以编译期对齐的契约，那么 commit hash **就是**契约——我用到的端点要记进我自己的文档，升级前跑一遍回归。

## 结果

六个假设在大概十分钟的 `curl` 和 `grep` 里被修正，而当时一行集成代码都还没写。项目依然可行——只是换了形状：没有 OIDC、手工打款、钉住 commit，以及一份不再依赖某个未合并分支上功能的计划。

那份规格书乐观并没有错。它错在**没被验证过**——而这是整个项目里最便宜的修补项。

---

**你正在自托管技术栈上建东西，想在集成之前找人看一眼那些功能到底在不在？** 这正是我在做的复核工作。

[WhatsApp +60 12-797 2969](https://wa.me/60127972969) ·
[[email protected]](mailto:[email protected]?subject=Open-source%20dependency%20review) ·
[hoelee.com](https://hoelee.com)

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "How I Vet an Open-Source Dependency Before Betting On It"
description: "Six checks that catch a repo which doesn't do what its docs claim — grep for the feature, read the doc's target branch, check the tier gate and the payment rail."
pubDate: 2026-09-29
category: devops
blueprint: sme-digital-suite
blueprintLayer: L4-managed-ops-handover
tags: ["open-source", "due-diligence", "oidc", "self-hosting", "docker"]
ogImage: "/og/vetting-an-open-source-dependency-before-you-bet-on-it.jpg"
banner: "/banners/vetting-an-open-source-dependency-before-you-bet-on-it.jpg"
draft: false
---

I was handed a 1,300-line architecture spec for a partner program. It read like a
plan: consistent, cross-referenced, confident. It named a self-hosted
open-source engine as the attribution and commission layer, and it assumed three
things about that engine — that it spoke OIDC so my existing identity provider
could log partners in, that its commission rules covered the five shapes the
business needed, and that its white-label feature was shipped.

Ten minutes of `curl` and `grep` later, two of those were false and the third was
half-true. None of that is unusual. The README describes the *product*; the
repository describes the *software*. The gap between those two is exactly where
integration projects die — and it is cheap to measure before you write any code.

## The problem, phrased as the question I'd actually search for

**How do I know an open-source project does what its documentation claims before
I build on it?**

A spec is a claim *about other software*. The more polished the spec, the more its
assumptions read like facts — this one cited sources, defined stable IDs, and had
a source-of-truth matrix. What it never did was check whether the engine down the
chain implemented the features being assigned to it. So before installing
anything, I went looking for those specific features.

## What I tried first, and why it wasn't enough

**1. Read the README end to end.** It sells the pipeline well:
`click → identity → event → attribution → commission → payout`. It never once
says which authentication protocol the app speaks. For a feature you *need*,
silence isn't neutral — but an absent mention is easy to skim past when the rest
of the page is convincing.

**2. Searched the web.** Several of my queries returned nothing usable. The
project is a few months old and its SEO surface is thin. For software that's
fine — the source *is* the documentation. It just means the reading has to happen
in the repo, not in a blog post.

**3. Found a 78 KB feature design doc and assumed the feature shipped.** It was
labelled "FINAL, post-review", listed three reviewers, and carried a date. I took
that as proof. That was the worst inference of the three, and the easiest to check.

## The six checks that actually settled it

### 1. Read the repo's vital signs before any feature claim

```bash
curl -s -H "User-Agent: hermes" https://api.github.com/repos/<owner>/<repo> \
  | grep -E '"created_at"|"pushed_at"|"stargazers_count"|"forks_count"|"open_issues_count"|"archived"|"spdx_id"'
```

What came back: MIT, created 2026-04-23, last push 2026-09-10, 8 stars, 3 forks,
15 open issues, not archived. Read the same as: **real, still moving, and very
small** — one vendor, which also sells a hosted tier. Now I know how much of my
business I want resting on it *before* I start grading its features.

### 2. Grep for the feature instead of reading for it

```bash
grep -in 'oidc\|sso\|saml' README.md ARCHITECTURE.md docs/*.md | wc -l
```

Zero hits. The app authenticates with its own magic-link sessions and
per-persona API keys; it does not speak OIDC at all. That one line of output
killed a phase of the plan — "put the portal behind our SSO provider" — which
would otherwise have surfaced weeks later, mid-integration, with a partner
already invited. Reading for a feature you can't find is how you lose an
afternoon; grepping makes the absence loud.

### 3. Read the target branch line at the top of a feature doc

```bash
head -5 docs/white-label-custom-domains.md
```

`Target branch: multi-tenant`. The doc is a design for a branch that is not the
default branch — it describes work in flight, so the feature is not in the code
you would install. A finished-looking document with reviewers and dates can
document something unshipped. That isn't dishonesty; it's what design docs are.

### 4. Read the data model, not the feature list

The commission section named exactly two rule types — `percent` (with a recurring
flag) and `fixed`. The business spec had enumerated five. That gap isn't a bug;
it's a modelling job I'd have to do in configuration instead of expecting from
the product:

| Spec assumed | What the engine actually has |
|---|---|
| first-order bonus | a `fixed` rule on the first event |
| event-specific rules (`lead` vs `invoice_paid`) | one rule per campaign — so: two campaigns |
| recurring commission | `percent` + `recurring: true` |
| percentage | `percent` ✅ |
| fixed | `fixed` ✅ |

Two rule primitives, five behaviours — workable, but only if you design for it up
front instead of discovering it during implementation.

### 5. Check which paid tier gates the feature you need

For the database tool in the same stack, the edition table was blunt:

| Feature I needed | Community (free) | Enterprise (paid) |
|---|---|---|
| API tokens, so I can automate it | ✅ | ✅ |
| SSO / OIDC | ❌ | ✅ |
| Page designer, hide branding, 2FA | ❌ | ✅ |

The thing I wanted for the partner-facing door (SSO) sits behind the commercial
gate, while the thing I wanted for automation (an API token) is free. Knowing
which side of the wall each requirement falls on changes the design — before
you've designed around it.

### 6. Check that the money rail works in your country

The payout model was `stripe_connect | manual`, and self-hosted mode was
documented as "operator manages out-of-band" — i.e. you move the money yourself.
Stripe Connect is not how Malaysian businesses pay partners, so that one line
decided the entire deployment shape: self-host mode, manual payouts, and event
ingestion pushed from my own order system rather than leaning on the provider's
payment integration. That line is never on a marketing page, and it mattered more
than any feature.

### Then: ask the fallback question before you need it

"What if the feature I need isn't there?" For the missing SSO the answer was:
don't put login in the app, put it in front of the app. authentik's proxy
provider exists to protect "applications that do not support native
authentication protocols such as OIDC, SAML, or LDAP", and it injects
`X-authentik-username` and `X-authentik-groups` headers upstream, so the reverse
proxy can do the gating. That turned "no SSO support" from a disqualification
into an afternoon of config.

## What I'd do differently

- **Write the requirement list first, then go look for it.** Five lines — the
  features *I* need — and search for those. Starting from the README's feature
  list means grading the software on the vendor's exam.
- **Treat "a doc exists" as different from "a feature ships".** Read the branch
  line.
- **Grep, don't read.** Presence is cheap to prove; absence is the thing you're
  hunting.
- **Decide the fallback before the decision**, so a missing feature becomes a
  design change rather than a surprise.
- **Pin the commit.** This project's README says its API is "stable but
  unversioned" and there is no OpenAPI description anywhere in the tree. There's
  no contract to compile against, so the commit hash *is* the contract — the
  endpoints I use go into my own docs, and upgrades get a regression pass.

## The result

Six assumptions corrected in about ten minutes of `curl` and `grep`, before a
single line of integration code existed. The project stayed viable — but
reshaped: no OIDC, manual payouts, a pinned commit, and a plan that no longer
depends on a feature sitting on an unmerged branch.

The spec wasn't wrong to be optimistic. It was wrong to be unverified, and that's
the cheapest thing on the whole project to fix.

---

**Building on a self-hosted stack, and want a second pair of eyes on whether
those features will actually be there when you integrate?** That's the kind of
review I do.

[WhatsApp +60 12-797 2969](https://wa.me/60127972969) ·
[[email protected]](mailto:[email protected]?subject=Open-source%20dependency%20review) ·
[hoelee.com](https://hoelee.com)

文章 09 / 11：SMART 显示健康，RAID 却在悄悄损坏数据

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "SMART 显示健康，RAID 却在悄悄损坏数据"
description: "两块看起来完全健康的 NVMe 组成的 btrfs RAID1 阵列，静默地把 Windows 虚拟机喂成了全零字节。我靠校验和取证抓住它——但根因（M.2 转接卡链路故障 vs 批次缺陷）至今仍未定论，现在两块盘测试都干净。"
pubDate: 2026-09-16
category: devops
blueprint: self-hosted-infrastructure
blueprintLayer: L1-compute-storage-network
tags: [unraid, btrfs, nvme, raid, storage, data-loss, troubleshooting]
ogImage: /og/zh-when-smart-says-healthy-but-your-raid-is-corrupting-data.jpg
banner: /banners/when-smart-says-healthy-but-your-raid-is-corrupting-data.jpg
---

所有存储指南都告诉你同一件事：相信 SMART、检查事件日志、相信你的文件系统。我的硬件不这么想。2026 年 9 月，一个 btrfs RAID1 阵列——两块企业级 NVMe 盘镜像，我用每个能想到的工具查它都说**「健康」**——却在悄悄地把我的 Windows 11 虚拟机喂成全零字节。不是看得见的故障，是静默损坏。

下面是完整故事：损坏是怎么浮现的、哪个校验和出卖了它——以及为什么最终定论如今变得不确定。

> **更新（2026-09-16）：根因至今未定论。** 下面的「两块盘都换」是我当时的判断，但后来一次不相关的故障让我不得不重新审视。现在两块 NVMe 测试都干净，而最强的头号嫌疑是两块盘共用的那块 **M.2 转接卡**——转接卡上的链路层故障，恰好会复现我当初归咎于硬盘的那种「两块盘关联地吐出全零」的损坏。SMART 对链路故障完全无感，所以无法排除它。我在一篇配套文章里写了那次警醒（一颗「死掉的」SATA SSD 其实只是坏线）。先读下面的原始分析，再看更正——然后还有**续集**：两周后同一个全零指纹出现在**完全不同的硬盘堆栈**上，这次终于挖出了根因。见[《换了不同的硬盘，同样的损坏又来了——真凶是 TRIM，不是 SSD》](/posts/zh/the-cause-was-trim-not-the-ssds/)。

## 更新：为什么我不再怪两块盘

这篇文章发出后，有两件事改变了我的想法。

**1. 现在两块盘测试都是干净的。** 在当前条件下，两块 PM9A3 读写都正常。如果它们带着 9 月还在活跃损坏数据的批次级缺陷，应该会挂掉一轮持续的写入+校验（`verify=crc32c`）泡测——但它们没有。

**2. 链路故障才是真正的教训。** 就在这次事故之后，我在另一颗盘上撞见了**完全相同**的故障签名（一颗 Seagate IronWolf 110 狂报 `WRITE FPDMA QUEUED` 超时、`mkfs` 失败、然后静默——教科书般的「死盘」）。结果是坏在 **SATA 数据线**上。一条不稳定的链路会复现下面我归咎于硬件的每一个症状：在算出 CRC 之前就已丢掉的超时 I/O、真的丢掉的页面，以及——如果两个 RAID1 成员共用同一块有问题的 **M.2 转接卡**或插槽——两块盘关联地损坏。这是唯一一个不需要任何硬盘缺陷、却能解释「两个成员用同一种方式撒谎」的答案。

所以我没法再诚实地断言「盘有问题」。它们可能一直好好的，损坏可能来自它们共同坐着的 **M.2 转接卡**。注意：**这并不代表这次事故是虚惊一场**——真实数据确实读不回来过，无论根因是什么，我都把它移走、迁移数据、并逐字节验证过。但正确的教训是「先排除链路，再给盘判死刑」，而不是「同批次盘不可信」。

完整的链路对盘排查，请看 **["我差点把这颗 SSD 送修——真正坏的是 SATA 数据线"](/posts/zh/that-dying-ssd-was-just-a-bad-sata-cable/)**。

## 原始分析（保留供透明考量）

下面是我*当时*得出的结论。这是当时证据下最站得住的解读，而且可能仍然是对的——只是我无法排除别的可能——但请把它当作事故报告来读，而不是最终定论。

## 为什么这件事重要

损坏状态的磁盘是最糟的存储故障，因为它最容易漏掉：

- **SMART 看起来完美。** 零媒体错误、几乎满余量，而且是能扛企业级写入负载的盘。
- **阵列有冗余。** btrfs RAID1 把每个块镜像到两块盘上。当**两个成员**用同一种方式撒谎时，冗余毫无意义。
- **故障是静默的。** 数组状态没有红色警告、没有硬盘报错、没有硬件事件——只有虚拟机读回垃圾，直到备份恢复成了唯一的选择。

如果你的数据在一切看起来都正常的时候悄悄坏掉，「备份」就不是偏执——它是全部。这次事故让我明白：一条**可测试、可验证的恢复**胜过任何阵列状态屏幕。

## 环境

- **unRaid 7.x**，Intel i9-13900K，Windows 11 虚拟机是主力工作站（GPU 直通）。
- **缓存池：** 两块 **Samsung PM9A3 3.84TB** NVMe 组 **btrfs RAID1**。而且两块是**同一生产批次**（同一个控制器/固件版本 `GDC7702Q`）。
- Windows 虚拟机的虚拟磁盘（`vdisk1.img`、`vdisk2.img`）就在这个池的 `domains/` 共享里，另外还有 **Docker 数据根目录**。
- 虚拟磁盘**没有**设 `NOCOW`——所以每次读取都会被 btrfs 校验和验证。也正因为这个，我最后才抓得住它。

## 症状

我开始发现缓存读取慢得离谱，主机命令也卡住：`cat`、读 `/proc/loadavg`、甚至 `docker image ls` 都在超时。那块坏 NVMe 正以 **45–75 MB/s** 被狂读，每次读都塞满整台主机的 I/O。本该几秒跑完的 `docker system prune` 直接挂死。

然后真正的麻烦来了：Windows 虚拟机开始狂报读错误，回来的是**错的**数据——满文件的全零字节、guest 里的校验和失败，最后主力 VM 直接引导不了。

## 取证：`csum 0x8941f998`

btrfs 在读校验失败时会往内核日志记一条校验和不匹配。同一个值反复出现：

```text
btrfs: checksum verify failed on <device> wanted 0x8941f998 found <other>
```

这个特定值是最重要的线索。**`0x8941f998` 就是一整块零字节的 CRC32C。** 它每次都出现、跨越互不相关的文件和 inode，绝不是巧合。正在发生的不是随机比特腐烂，而是盘**稳定地返回全零块**，而这些零填充的读取正是 btrfs 校验失败的对象：

- 验证一下：零填充块的 `crc32c` 就等于 `0x8941f998`。
- 同一个值反复出现 → 读取是**确定性**地吐出空数据，不是偶尔翻几个比特。

两块虚拟磁盘都中招（`vdisk1.img` inode 2358574、`vdisk2.img` inode 2362817），正好对上「存储层在把零当成被请求的数据返回」的图景。

## 破绽：永远不生效的「自愈」

btrfs RAID1 本应自愈：一个成员坏块、从镜像重建、写回好数据。但我一遍遍 scrub、重读同一块（仍然全零的）数据后发现，**重写根本不落地**。每次重建「成功」后，立刻重读同一 extent 又吐出零。

这是关键区别：

- 一块**只读不出来**的盘是读问题——自愈应该能从镜像救回来。
- 一块**连回调也无脑吞垃圾存进去**的盘是**写或控制器**问题——镜像救不了你，因为两条路径都对同一个错误答案达成一致。

当自愈重写一直不生效，你就不在「坏一个扇区」的范畴了，而是「这块盘在写入上撒谎」——RAID 冗余救不了。

## 为什么两块盘都得换（我当时的推理）

> 这是当初「都换掉」决定的依据。动手前先读上面更新里的更正——一块共用的 M.2 转接卡链路故障，在无需任何硬盘缺陷的情况下也能对上这份证据。

数据指向的是**同批次盘的硬件缺陷**，而不是磨损或一次性的偶发事件：

- SMART 干净（0 媒体错误、几乎满余量，`nvme0` 大约写了 586 TiB——对能扛企业级负载的盘来说算重负载但仍正常）。
- 两块盘**同批次、同控制器/固件**（`GDC7702Q`）。
- 损坏以**相关的方式**同时命中两个 RAID1 成员——镜像「死一块也能幸存」的整个意义，因为两块盘正以同一种方式一起死掉而完全失效。

面对批次级的关联缺陷，留下那块「看起来健康」的盘当幸存成员是假省心。**我从备份恢复了 Windows 虚拟机，然后把两块盘都换掉。**

## 我同时做的事：把数据迁出那个池

在换盘之前，任何还值得读的数据都得搬出这个坏池——而且要**验证**，不能想当然：

- **Docker 数据根目录**迁出缓存：`docker` 挪到 `/mnt/user/docker`（在 `disk1` 上，overlay2）。整棵树通过**路径、大小、mtime** 逐文件比对、并与字节对比交叉验证，共 **3,997,569** 个文件。
- 缓存的 `isos/`、`system/`、`swapfile` 搬到 `disk1`；`isos/`（约 13 GB）用 `diff -rq` 检查 → **EXIT=0**，无差异。
- 所有搬走的文件都在**删除缓存副本之前**先跟源比对过——没验证过的副本绝不删。

因为存在读侧损坏（全零块）和写侧损坏（自愈不落地），那个池里的任何东西，没有镜像、没有校验和、没有新鲜备份可对照，我就一律不信任。

## 如果重来我会怎么做

1. **在需要它之前就先测试恢复虚拟机备份。** 我当时确认了备份存在，但损坏事故不是该第一次学恢复流程的时候。定时做的测试恢复既能发现备份问题，也能发现静默的存储损坏。
2. **把同批次的两块盘当成一块盘。** 两块 PM9A3 同批次同固件。对需要长期搭档的缓存池（任何 RAID1）这是关联失效风险——优先选不同批次的盘，或者至少在信任镜像前把这对盘狠狠测一遍。
3. **主动检测，而不是被动祈祷。** 正是 btrfs 校验和让它在噪声里浮出来。不开 `NOCOW` 意味着文件系统在验证每一次读取；一个带校验的数据池，就是「当场抓住并恢复」和「把损坏文件带出去几个月」之间的分水岭。
4. **别用 SMART 判断闪存。** 一块看起来健康却照样损坏的 NVMe，事件计数什么也说明不了——固件级的缺陷可以躲过所有 SMART 属性。

## 结果

一台 Windows 虚拟机从备份恢复、数据安全迁出那个坏池、Docker 数据根目录也搬了家——**外加一条事后再推翻的教训。** 我当时换掉了两块盘，但现在我没法确定它们就是元凶：证据同样支持一块共用的 **M.2 转接卡**链路故障，而且现在两块盘测试都干净。（完整排查见 **["我差点把这颗 SSD 送修——真正坏的是 SATA 数据线"](/posts/zh/that-dying-ssd-was-just-a-bad-sata-cable/)**。）

事故被控制住，是因为损坏被校验和抓住了、而不是在里面泡了几个星期——这一点无论根因是什么都成立。但更诚实的教训，比「把盘换掉」这句要更广、也更不戏剧化：

1. **一块「正在损坏」的盘，并不等于盘本身坏了。** 在给硬件判死刑前，先排除链路：重新插拔/更换线材、换到别的口/转接卡/插槽，重跑*相同*的失败操作。如果故障跟着**口/转接卡**走，那就是连接的问题。
2. **SMART 看不见链路故障。** 零媒体错误和几乎满余量，对一条不稳定的线材或转接卡毫无意义——超时在算出 CRC 之前就把 I/O 丢了。
3. **同批次盘共用一条通道，无论谁坏都是共享失效风险。** 不管损坏来自盘还是转接卡，一对盘坐同一条故障路径上，就是穿着两个序列号的单点故障。
4. **给你的数据加校验、并提前测试恢复备份。** 把数星期静默损坏变成一次被控制住的事故的，是这一点——不是我那部分诊断。

---

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "When SMART Says Healthy and Your RAID Is Silently Corrupting Data"
description: "A btrfs RAID1 pool on two healthy-looking NVMe drives quietly corrupted a Windows VM. How I caught it with a checksum forensic — and why the root cause (M.2 riser link fault vs batch defect) is still uncertain, with both drives now testing clean."
pubDate: 2026-09-16
category: devops
blueprint: self-hosted-infrastructure
blueprintLayer: L1-compute-storage-network
tags: [unraid, btrfs, nvme, raid, storage, data-loss, troubleshooting]
ogImage: /og/when-smart-says-healthy-but-your-raid-is-corrupting-data.jpg
banner: /banners/when-smart-says-healthy-but-your-raid-is-corrupting-data.jpg
---

Every storage guide tells you the same thing: trust SMART, check the event
log, and believe your filesystem. My hardware disagreed. In September 2026 a
btrfs RAID1 pool — two mirrored enterprise NVMe drives that said **"healthy"**
in every tool I could run — was quietly feeding my Windows 11 VM all-zero
bytes. Not visibly failing. Silently corrupting.

This is the full story: how the corruption surfaced, and the checksum that
gave it away.

> **Update (2026-09-16): the root cause turned out to be uncertain.** The
> "replace both drives" verdict below was my best diagnosis at the time, but a
> later, unrelated failure taught me to question it. Both NVMe drives now test
> clean, and the strongest remaining suspect is the **M.2 riser** both drives
> were plugged into — a link-layer fault on the riser reproduces exactly the
> correlated, all-zero corruption I blamed on the drives. SMART is silent to
> link faults, so it can't rule one out. I describe that wake-up call (a "dead"
> SATA SSD that was just a bad cable) in a companion post. Read the original
> analysis, then the correction — and then the *sequel*: two weeks later the
> same all-zero fingerprint appeared on a **completely different drive stack**,
> and this time the root cause surfaced. See [The Corruption Came Back on
> Different Drives — the Cause Was TRIM, Not the SSDs](/posts/the-cause-was-trim-not-the-ssds/).

## Update: why I no longer blame the drives

Two things changed my mind after this post went up.

**1. The drives test clean now.** Under current conditions both PM9A3s read and
write correctly. If they carried a batch-level defect that was actively
corrupting in September, I'd expect them to fail a sustained write+verify
(`verify=crc32c`) soak — they don't.

**2. A link fault was the real lesson all along.** Right after this incident I
hit the *same* failure signature on a different drive (a Seagate IronWolf 110 that threw
`WRITE FPDMA QUEUED` timeouts, failed `mkfs`, and went silent — textbook
"dead SSD"). It turned out to be the **SATA cable**. A flaky link reproduces
every symptom I blamed on hardware below: timeouts dropped before a CRC is
computed, real pages lost, and — if both RAID1 members share the same
faulty **M.2 riser** or slot — *correlated* corruption across both drives.
That's the one explanation that fits "both members lied the same way," and it
doesn't require any drive defect at all.

So I can't honestly say "the drives were bad." They may have been perfectly
fine, and the corruption may have come from the **M.2 riser** they both sat
on. Note that **this does not make the incident a false alarm**: real data
*could not be read back*, and whatever the cause, I removed it, migrated the
data, and verified every byte. But the correct lesson is "rule out the link
before condemning the drive," not "same-batch drives are untrustworthy."

The full link-vs-drive detective work is in **["I Was About to RMA This SSD —
the Fault Was a SATA Cable"](/posts/that-dying-ssd-was-just-a-bad-sata-cable/)**.

## The original analysis (kept for transparency)

What follows is what I concluded *at the time*. It was the most defensible
reading of the evidence available, and it may still be right for reasons I
can't rule out — but read it as the incident report it is, not the final
verdict.

## Why this matters

A corrupting disk is the worst kind of storage failure because it's the
easiest to miss:

- **SMART looked perfect.** Zero media errors, ~100% spare, on drives rated
  for heavy enterprise write loads.
- **The pool had redundancy.** btrfs RAID1 mirrored every block across two
  drives. Redundancy means nothing when *both* members lie the same way.
- **The failure was silent.** No red warning in the Unraid array status, no
  failing drive, no hardware event — just VM reads slowly returning garbage
  until a backup restore became the only option.

If your data only gets corrupted when nothing looks wrong, "backup" is not
paranoia — it's the entire ballgame. This incident is why a testable,
verified restore beats any array status screen.

## The setup

- **unRaid 7.x** on an Intel i9-13900K, with a Windows 11 VM as the primary
  workstation (GPU passed through).
- **Cache pool:** two **Samsung PM9A3 3.84 TB** NVMe drives in **btrfs RAID1**.
  Both from the **same production batch** (same controller/firmware
  revision, `GDC7702Q`).
- The Windows VM's virtual disks (`vdisk1.img`, `vdisk2.img`) lived on that
  pool, in the `domains/` share — plus **Docker's data root**.
- The VM's vdisks were **not** `NOCOW` — so every read was validated against
  a btrfs checksum. That setting ended up being the reason I caught it at all.

## The symptom

I started noticing cache reads were crawling, and host commands were stalling:
`cat`, reading `/proc/loadavg`, even `docker image ls` were timing out. The
corrupt NVMe was being hammered at **45–75 MB/s of reads**, and every read
saturating the drive stalled the whole host's I/O. A `docker system prune`
that should have finished in seconds just hung.

Then the real problem surfaced: the Windows VM began throwing read errors and
data that came back was **wrong** — files that were full of zero bytes,
checksum failures in the guest, and ultimately an unbootable primary VM.

## The forensic: `csum 0x8941f998`

On btrfs, when a read fails verification the kernel logs a checksum mismatch
for the affected extent. The same value kept recurring:

```text
btrfs: checksum verify failed on <device> wanted 0x8941f998 found <other>
```

That specific value was the first real clue. **`0x8941f998` is the CRC32C of
a block of zeros.** It has to be — it came up every time, across unrelated
files and inodes. What was happening wasn't random bit rot; the drive was
returning **all-zero blocks**, consistently, and those zero-filled reads were
exactly what btrfs was failing to verify:

- Check the value: `crc32c` of a zero-filled extent matches `0x8941f998`.
- It's the same value again and again → reads were **deterministically**
  producing empty data, not occasionally flipping bits.

Two VM vdisks were affected (`vdisk1.img` inode 2358574, `vdisk2.img` inode
2362817), which fit the picture: a storage layer was returning zeros instead
of the data it was asked to read.

## The tell: self-heal "fixes" that never stuck

btrfs RAID1 is supposed to be self-healing: read a bad block on one member,
reconstruct it from the mirror, and rewrite the good data back. But as I
scrubbed and re-read the same (still all-zero) data, it became clear the
**rewrites were not sticking**. Each rebuild "succeeded" and the very next
read of the same extent produced zeros again.

That's the critical distinction:

- A drive that only **fails to read** is a read problem — self-heal should
  recover it from the mirror.
- **A drive that also accepts and silently stores garbage** is a *write* or
  *controller* problem — the mirror can't save you, because both paths agree
  on the wrong answer.

When self-heal rewrites keep not sticking, you're no longer in "one bad
sector" territory. You're in "the device is lying about writes" territory,
and RAID redundancy won't help.

## Why both drives had to go (my reasoning at the time)

> This was the basis for the original "replace both" decision. Read the
> correction in the Update above before acting on it — a shared M.2 riser
> link fault also fits this evidence without any drive being defective.

The data pointed at a **hardware defect in the same-batch drives**, not a
wear or one-off event:

- SMART was clean (0 media errors, ~100% spare, and `nvme0` had written
  roughly 586 TiB — heavy, but on a drive rated for enterprise workloads).
- The two drives were **same-batch, same controller/firmware** (`GDC7702Q`).
- Corruption hit **both** RAID1 members in correlated fashion — the whole
  point of mirroring (survive one drive dying) failed because both were dying
  the same way, in lockstep.

With a correlated, batch-level defect, keeping one "healthy-looking" drive as
the surviving member is a false economy. **I restored the Windows VM from
backup and replaced both drives.**

## What I did in parallel: move the data off that pool

Before the drives were replaced, anything still worth reading had to come off
the failing pool — and it had to be **verified**, not assumed:

- **Docker's data root** moved off the cache: `docker` was relocated to
  `/mnt/user/docker` (on `disk1`, overlay2). The whole tree was verified by
  matching **path, size, and mtime** for all **3,997,569 files** against a
  byte-comparison.
- The cache's `isos/`, `system/`, and `swapfile` moved to `disk1`; the
  `isos/` set (~13 GB) was checked with `diff -rq` → **EXIT=0**, no
  differences.
- Everything moved was compared against the source **before** the cache copy
  was deleted — never delete a copy you haven't verified.

The read-side corruption (all-zero blocks) and the write-side corruption
(self-heal not sticking) meant I trusted nothing from that pool without a
mirror, a checksum, or a fresh backup to compare against.

## What I'd do differently

1. **Test-restore the VM backup before needing it.** I'd verified backups
   existed, but a corruption incident is not the time to first learn your
   restore procedure. A scheduled test-restore catches both backup failure
   *and* silent storage corruption.
2. **Treat same-batch drives as one drive.** Both PM9A3s came from the same
   batch with the same firmware. For a married cache pool (or any RAID1)
   that's a correlated-failure risk — prefer drives of different production
   batches or at least hammer-test the pair before trusting a mirror.
3. **Detect, don't just hope.** btrfs checksums are what surfaced this
   through the noise. The `NOCOW`-less vdisk config meant the filesystem was
   validating every read; a pool of checksummed data is the difference
   between "caught it and restored" and "shipped corrupted files for months".
4. **Don't trust SMART for flash.** For a healthy-looking NVMe that still
   corrupts, the event count says nothing — the firmware-level defect can be
   invisible to every SMART attribute.

## The result

One Windows VM restored from backup, the data safely off the failing pool, and
Docker's data root relocated — **plus a lesson I had to un-learn after the
fact.** I replaced both drives, but I can no longer be certain they were the
cause: the evidence fits a shared **M.2 riser** link fault just as well, and
both drives test clean now. (Full detective work in ["I Was About to RMA This
SSD — the Fault Was a SATA Cable"](/posts/that-dying-ssd-was-just-a-bad-sata-cable/).)

The incident was contained because corruption was caught by checksums, not
wallowed in for weeks — that part stands regardless of cause. But the honest
lesson is broader and less dramatic than "replace the drives":

1. **A "corrupting" drive isn't proof the drive is bad.** Before condemning
   hardware, rule out the link: reseat/swap the cable, move to a different
   port/riser/slot, and rerun the *same* failing operation. If the fault
   follows the **port/riser**, it's the connection.
2. **SMART can't see link faults.** Zero media errors and ~100% spare say
   nothing about a flaky cable or riser — timeouts drop I/O before a CRC is
   even computed.
3. **Same-batch drives sharing a channel is a shared-failure risk either
   way.** Whether the corruption was the drives or the riser, a married pair
   on one faulty path is a single point of failure wearing two serial numbers.
4. **Checksum your data and test-restore your backups.** That — not the part
   diagnosis — is what turned a weeks-long silent corruption into a contained
   incident.

---

文章 10 / 11：为什么容器里的 Chrome 会忘记标签页 —— 以及我是怎么修好的

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "为什么容器里的 Chrome 会忘记标签页 —— 以及我是怎么修好的"
description: "Chrome 的「继续上次浏览」在 Docker 容器里根本不会生效：三次失败的尝试、真正的原因，以及最后管用的 70 行守护脚本。"
pubDate: 2026-09-29
category: devops
tags: [docker, chrome, cdp, persistence, synology, self-hosting]
ogImage: /og/zh-why-chrome-forgets-its-tabs-in-a-container.jpg
banner: /banners/why-chrome-forgets-its-tabs-in-a-container.jpg
draft: false
---

我在 NAS 上的 Docker 容器里跑着一个真正的 Google Chrome。它不是爬虫实例，也不是截图服务 —— 是我真的在用的浏览器：登录着那些绝不该交给无头机器人的账号，可以从任何设备通过网页桌面看着它操作，需要自动化的时候还能用 Chrome DevTools Protocol (CDP) 驱动它。

它一直很好用，只有一个例外：**每次容器重启，浏览器回来时只剩一个空白标签页。**

Chrome 明明有专门对付这件事的设置 ——「继续浏览上次打开的网页」。它一直是开着的，却毫无作用。下面就是三次失败的尝试、真正的原因、路上踩到的两颗地雷，以及最后真正让重启变得可用的那七十来行守护脚本。

## 为什么这件事值得认真对待

当容器里的浏览器就是你的工作台时，标签页就是状态本身：填了一半的表单、正在编辑的商品上架页、筛选到指定日期的仪表盘、客户基础设施的三个页面 —— 那些就是工作。浏览器一启动停在空白页，意味着每次重启后的头十分钟都在重建你本来就有的上下文。

这个问题我查过不止一次，网上能搜到的只有「把 `restore_on_startup` 设上」这类建议，而那套建议一进容器就失效。如果你也撞上了同一堵墙，这篇文章就是这堵墙的解释。

## 我的环境

Synology NAS 上的一个 Docker stack，里面两个浏览器（真 Chrome 加 Brave），各自挂在自己的 SSO 网关后面：

| 组件 | 取值 |
|---|---|
| 镜像 | `lscr.io/linuxserver/chrome` —— 真正的 Google Chrome 154 |
| 桌面 | Selkies 网页桌面，容器 `3000`/`3001` → 宿主 `3030`/`3031` |
| 自动化 | CDP 暴露在 `http://<nas>:9232/json/version` |
| Profile | bind mount，容器内路径 `/config/profile` |
| 启动参数 | `--remote-debugging-port=9222 --user-data-dir=/config/profile --restore-last-session` |

Profile 是 bind mount，所以登录态、cookie、历史记录在容器重建后确实都还在 —— 那部分从来没坏过。丢掉的只有**会话**（当时开着哪些标签页）。

## 尝试一：那个启动参数

Chromium 有一个听起来正合适的开关：`--restore-last-session`，文档说法是「在**意外退出**后恢复上次的会话」。

我通过镜像的 CLI 环境变量加上它，并且确认它确实传到了运行中的进程上 —— 因为这类镜像的命令行是包装脚本拼出来的：

```bash
docker exec chrome sh -c 'tr "\0" " " < /proc/$(pgrep -f "google-chrome --" | head -1)/cmdline' | tr ' ' '\n' | grep -E 'restore|user-data-dir|remote-debug'
# --remote-debugging-port=9222
# --user-data-dir=/config/profile
# --restore-last-session
```

重启。空白页。参数在，只是它没有任何「愿意恢复」的东西。

## 尝试二：直接改 profile 里的配置

Chrome 把「继续浏览上次打开的网页」存在 profile 的 `Preferences` JSON 里，字段是 `session.restore_on_startup: 1`。于是我停掉容器，直接把这两个值改了：

```python
import json
p = '/config/profile/Default/Preferences'
d = json.load(open(p))
d.setdefault('session', {})['restore_on_startup'] = 1   # 1 = restore last session
d.setdefault('profile', {})['exit_type'] = 'Normal'
json.dump(d, open(p, 'w'))
```

这就是网上最流行的那套做法，而它「半成功」了：下次启动后，Chrome **保留**了文件里的 `restore_on_startup: 1` —— 然后照样打开空白页。它没有保留的是我写的 `exit_type`：启动时它已经把那个值改回了 `Crashed`。

这次改写反而是第一条真正的线索。Chrome 把启动状态当成安全边界 —— 劫持者最想改的那些设置（主页、搜索引擎、启动页面）会跟另一个文件 `Secure Preferences` 做 MAC 校验。所以，磁盘上被改过的值，并不自动等于 Chrome 会遵守的值。

## 尝试三：托管策略

让浏览器行为具备权威性的正式办法是**托管策略**（managed policy），它不受上面那套 MAC 校验约束。在 Linux 上，Chrome 读取 `/etc/opt/chrome/policies/managed/*.json`：

```json
{ "RestoreOnStartup": 1 }
```

动手之前有两个细节：Chrome **会忽略**运行浏览器的那个用户可写的策略文件，所以必须是 `root:root` 加 `644`；而在 Synology 的共享目录上，某些目录即使 `sudo` 也会拒绝执行 `chmod`，所以别假设你改成功了，去确认当前的实际权限。

挂进容器，启动后确认文件在位：

```bash
docker exec chrome sh -c 'ls -l /etc/opt/chrome/policies/managed/; cat /etc/opt/chrome/policies/managed/hoeleepolicy.json'
# -rw-r--r-- 1 root root 28 hoeleepolicy.json
# { "RestoreOnStartup": 1 }
```

重启。还是空白页。

## 尝试四：先证明「退出是不干净的」

到这一步，问题已经不是「我该用哪个设置」，而是「容器停掉的时候，这个浏览器到底看到了什么」。

Chrome 的会话恢复机制需要一个它认为**被中断、或可续接**的会话：只有在正常退出时，它才会把 `Current Session` 落成 `Last Session`。于是我改用最温和的方式收掉浏览器 —— 通过 CDP 让它自己优雅退出，而不是把容器从它脚下抽走：

```python
# browser-level target from /json/version
ws.send(json.dumps({"id": 1, "method": "Browser.close", "params": {}}))
```

Chrome 确实退出了 —— 然后桌面会话在几秒内又把它拉了起来，`exit_type` 立刻回到 `Crashed`。

这就是全部的机关，它一次性解释了前面三次失败：**这个浏览器从来没有干净退出过。** 你停容器时，桌面会话被一起拆掉，浏览器跟着被杀。没有干净退出，就没有 `Last Session` 的落盘，也就没有任何东西可供「继续上次浏览」或 `RestoreOnStartup` 策略去恢复。那个参数、那个首选项、那条策略其实都在正常工作 —— 只是它们手上空无一物。

数据其实一直都在。那段时间 profile 的会话目录每隔几秒就在写：

```
-rw------- 1 hoelee users 40722 Session_13434833255312492
-rw------- 1 hoelee users 58879 Tabs_13434833757632480
-rw------- 1 hoelee users 79361 Tabs_13434833833646133
```

## 修法：别再去配置它，开始替它记住

如果浏览器记不住自己的会话，那就让容器替它记住。两件事，都很小：

1. **启动时**：等浏览器起来，如果它一个真正的页面都没打开，就把上次的网址重新打开。
2. **之后一直**：每 60 秒把当前打开的页面地址快照一次。

这就是一段从镜像的容器初始化钩子拉起的后台脚本。我实际在跑的版本，去掉无关部分后是这样：

```python
#!/usr/bin/env python3
"""Restore last session's tabs on start, then snapshot open tabs every 60s."""
import json, os, time, urllib.parse, urllib.request

CDP   = "http://127.0.0.1:9222"          # loopback inside the container
STORE = "/config/tabs-last.txt"          # lives in the persistent profile mount

def open_tabs():
    """http(s) page URLs, or None when the browser isn't answering yet."""
    try:
        with urllib.request.urlopen(CDP + "/json/list", timeout=5) as r:
            tabs = json.load(r)
    except Exception:
        return None
    return [t.get("url", "") for t in tabs
            if t.get("type") == "page" and t.get("url", "").startswith("http")]

saved = [u.strip() for u in open(STORE)] if os.path.exists(STORE) else []

# 1) restore — wait up to 3 minutes for the browser to come up
if saved:
    for _ in range(60):
        if open_tabs() is not None:
            break
        time.sleep(3)
    if open_tabs() == []:                      # nothing but a blank new tab
        for url in saved[:20]:
            req = urllib.request.Request(
                CDP + "/json/new?" + urllib.parse.quote(url, safe=""), method="PUT")
            urllib.request.urlopen(req, timeout=10).read()
            time.sleep(0.7)

# 2) snapshot — write even an empty list, so "I closed everything" is respected
while True:
    time.sleep(60)
    tabs = open_tabs()
    if tabs is None:
        continue
    tmp = STORE + ".tmp"
    open(tmp, "w").write("\n".join(tabs) + ("\n" if tabs else ""))
    os.replace(tmp, STORE)                     # atomic; a kill can't truncate it
```

两个设计选择值得抄走：

- **只在浏览器确实空空如也时才恢复。** 万一将来某个 Chrome 版本自己会恢复标签页，这段脚本就会让路，而不是把每个标签页再复制一遍。
- **快照是原子的，而且允许为空。** 先写临时文件再 `os.replace`，意味着写到一半被杀也不会留下半截列表；而空列表是有意义的数据 —— 它表示你主动关掉了全部标签页，所以下次启动就该保持空白。

拉起它的初始化钩子：

```sh
#!/bin/sh
# clear stale singleton state (see below)
rm -f /config/profile/Singleton* 2>/dev/null
# long-running, so background it or you block container init
/usr/bin/env python3 /custom-cont-init.d/tabs_keeper.py >> /config/tabs-keeper.log 2>&1 &
exit 0
```

### 地雷一：残留的 `Singleton*` 会让 Chrome 干脆不启动

测试重启的过程中，我撞上了一个比空白页更糟的故障：容器**健康地**起来了，里面却没有浏览器。没有 `google-chrome` 进程、CDP 不通、中继容器的健康检查报 `unhealthy`，日志里除了桌面启动之外什么都没有。

原因在 profile 里的 `SingletonLock`/`SingletonSocket`/`SingletonCookie` 这几个符号链接 —— 它们记录了「这个 profile 属于主机 *h* 上的进程 *n*」。而容器里的 PID 会跨重启复用，于是下一次启动可能发现自己这个 PID 已经被「占用」，于是立刻退出。在容器启动时、浏览器拉起之前把它们删掉就是解法，这也是上面钩子里那句 `rm -f` 存在的原因。

### 地雷二：`custom-cont-init.d` 只执行**可执行**文件，而且用 bash 执行

LinuxServer 的镜像会执行 `/custom-cont-init.d` 下的每个文件，但前提是它可执行，并且是用 `/bin/bash` 执行。也就是说，你放进去的 Python 文件**绝不能**带执行位，否则会被喂给 bash 然后以 2 退出：

```
[custom-init] tabs_keeper.py: executing...
[custom-init] tabs_keeper.py: exited 2
```

可行的分工是：`.sh` 钩子设 `755`，把常驻任务放到后台；同目录的 `.py` 设 `644`。另外，任何不会立刻返回的东西都必须后台化，否则容器初始化会一直等它。

## 顺带带出来的管道陷阱

守护脚本是走 loopback 上的 CDP 跟 Chrome 说话的，所以它不需要任何端口。但如果你还想从另一台机器驱动这个浏览器，就会多出一个容器 —— 因为 Chrome ≥ 136 把 devtools 端点钉在 `127.0.0.1:9222`，并且忽略 `--remote-debugging-address`。标准做法是让一个 `socat` 边车共用浏览器的网络命名空间：

```yaml
chrome-relay:
  image: alpine/socat
  network_mode: service:chrome            # must share the browser's netns
  command: socat TCP-LISTEN:9223,fork,reuseaddr TCP:127.0.0.1:9222
  healthcheck:
    test: ["CMD-SHELL", "nc -z 127.0.0.1 9222 || exit 1"]
```

端口映射**不**写在中继上（它自己没有独立命名空间），而是写在浏览器服务上：`9232:9223`。

这个边车有一个你必须记住的行为：**重启浏览器，中继会跟着死。** 它当初所在的命名空间被重建了，socat 进程还活着、套接字却已经失效，于是 CDP 报 `connection reset`，而容器看起来一切正常。按这个顺序收拾：

```bash
docker restart chrome && docker restart chrome-relay   # order matters
```

上面那条健康检查的作用，就是把这种静默故障变成一个看得见的 `unhealthy`。如果你打算自己驱动容器里的浏览器，客户端那一侧我单独写过一篇：[Scraping a Bot-Walled Marketplace With a Warm Browser Session](/posts/scraping-bot-walled-marketplace-warm-browser-session/)。

## 如果重来一次

- **先确认进程能不能干净退出，再考虑相信任何「启动时恢复」的机制。** 我在一个前提条件（浏览器能察觉到的退出）根本不存在的情况下，连试了三种配置。一次 `Browser.close` 测试就能让整个排查过程坍缩。
- **不要手改浏览器首选项。** 它们会跟 `Secure Preferences` 做 MAC 校验；磁盘上一个看起来合理的值，并不等于 Chrome 会遵守的值。需要权威性就用托管策略 —— 并且去验证策略真的加载了，而不是假设文件放进去就够了。
- **确认谁有权限写那条策略。** `root:root`、`644`，放在 `/etc/opt/chrome/policies/managed` 下才是对的形状；用户可写的策略文件会被静默忽略。
- **把「浏览器在跑」和「容器在跑」当成两个不同的问题。** 残留 singleton 那次故障，从外面看就是一个健康的容器。
- **与其让软件记住状态，不如自己存快照。** 任何关闭路径上带强杀的软件 —— 容器、kiosk、桌面会话 —— 都不是依赖「干净退出」记账的好地方。一个 60 秒一次、原子写入的快照很无聊，但它管用。

## 结果

- 真实的容器重启之后，两个浏览器各自 **2/2 个标签页**都恢复了（守护日志里是 `restored 2/2 tabs`），重启后再用 CDP 复核过
- 同一批重启里，登录态、cookie 和 `localStorage` 都确认存活 —— 持久化 profile 从来不是问题所在
- 三个 stack（两个浏览器加一个试验品）合并成一个，并且靠中继的健康检查，把唯一那个「重启顺序」陷阱由静默变成看得见

## 你的业务需要这个吗？

如果你想要一个始终登录着的浏览器 —— 跑在你自己的硬件上、任何地方都能访问、挂在 SSO 后面、标签页状态能扛住重启，还带一个能给你脚本用的 CDP 端点 —— 那正是我在跑、也在维护的这套东西。

**WhatsApp: [+60 12-797 2969](https://wa.me/60127972969)** · **Email: [[email protected]](mailto:[email protected]?subject=Self-hosted%20container%20browser)** · **[hoelee.com](https://hoelee.com)**

网站设计与开发是我的主业；自托管基础设施与自动化是另外一半。

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "Why Chrome Forgets Its Tabs in a Container — And How I Fixed It"
description: "Chrome's \"continue where you left off\" never fires inside a Docker container. The three fixes that failed, the real root cause, and the 70-line keeper that works."
pubDate: 2026-09-29
category: devops
tags: [docker, chrome, cdp, persistence, synology, self-hosting]
ogImage: /og/why-chrome-forgets-its-tabs-in-a-container.jpg
banner: /banners/why-chrome-forgets-its-tabs-in-a-container.jpg
draft: false
---

I run a real Google Chrome inside a Docker container on my NAS. It is not a scraping instance and not a screenshot service — it is a browser I actually use: signed into the accounts a headless bot should never touch, watchable through a web desktop from any device, and drivable over the Chrome DevTools Protocol when I want to automate part of a job.

It works well, with one exception. **Every time the container restarts, the browser comes back with a single blank tab.**

Chrome has a setting for exactly this — "Continue where you left off". It was already enabled. It did nothing. What follows is three fixes that failed, the actual root cause, the two landmines I hit on the way, and the ~70-line keeper that finally made restarts survivable.

## Why this matters

When a container browser is your workspace, the tabs *are* the state. Half-finished forms, a marketplace listing in progress, a dashboard filtered to the right date range, three tabs open on a customer's infrastructure — that is the work. A browser that opens on `about:blank` means the first ten minutes after every restart are spent rebuilding context you already had.

I looked for this answer more than once and found nothing but "set `restore_on_startup`" advice that does not survive contact with a container. If you have hit the same wall, this is the wall explained.

## The setup

A single Docker stack on a Synology NAS, two browsers in it (real Chrome and Brave), each behind my own SSO gate:

| Piece | Value |
|---|---|
| Image | `lscr.io/linuxserver/chrome` — real Google Chrome 154 |
| Desktop | Selkies web desktop, container `3000`/`3001` → host `3030`/`3031` |
| Automation | CDP published at `http://<nas>:9232/json/version` |
| Profile | a bind mount, `/config/profile` inside the container |
| Launch flags | `--remote-debugging-port=9222 --user-data-dir=/config/profile --restore-last-session` |

The profile is a bind mount, so the logins, cookies and history genuinely do persist across a container rebuild — that part was never broken. Only the *session* (which tabs were open) was lost.

## Attempt 1: the flag

Chromium ships a switch that sounds like the answer: `--restore-last-session`. It is documented as "restore the last session *after an unexpected exit*".

I added it via the image's CLI env var and confirmed it was actually applied to the running process, because these images build the command line from a wrapper script:

```bash
docker exec chrome sh -c 'tr "\0" " " < /proc/$(pgrep -f "google-chrome --" | head -1)/cmdline' | tr ' ' '\n' | grep -E 'restore|user-data-dir|remote-debug'
# --remote-debugging-port=9222
# --user-data-dir=/config/profile
# --restore-last-session
```

Restart. Blank tab. The flag was there; it just had nothing it was willing to restore.

## Attempt 2: edit the profile's preferences

Chrome stores "Continue where you left off" in the profile's `Preferences` JSON as `session.restore_on_startup: 1`. So I stopped the container and edited it directly, along with the `exit_type` marker:

```python
import json
p = '/config/profile/Default/Preferences'
d = json.load(open(p))
d.setdefault('session', {})['restore_on_startup'] = 1   # 1 = restore last session
d.setdefault('profile', {})['exit_type'] = 'Normal'
json.dump(d, open(p, 'w'))
```

This is the classic advice you will find online, and it half-worked: after the next boot, Chrome had **kept** `restore_on_startup: 1` in the file — and still opened a blank tab. What it did not keep was my `exit_type`: it had already rewritten that to `Crashed` at startup.

That rewrite turned out to be the first real clue. Chrome treats startup state as a security boundary — the settings a hijacker would most like to change (homepage, search provider, startup pages) are MAC-checked against a separate `Secure Preferences` file. A value edited on disk does not automatically become a value Chrome obeys.

## Attempt 3: a managed policy

The supported way to force browser behaviour is a **managed policy**, which is authoritative and not subject to that MAC check. On Linux, Chrome reads `/etc/opt/chrome/policies/managed/*.json`:

```json
{ "RestoreOnStartup": 1 }
```

Two details before you bother: Chrome ignores a policy file that is writable by the user running the browser, so it needs `root:root` and mode `644` — and on a Synology share, `chmod` refuses to run even under `sudo` for some directories, so verify the mode you already have instead of assuming you changed it.

Mounted into the container, verified present after boot:

```bash
docker exec chrome sh -c 'ls -l /etc/opt/chrome/policies/managed/; cat /etc/opt/chrome/policies/managed/hoeleepolicy.json'
# -rw-r--r-- 1 root root 28 hoeleepolicy.json
# { "RestoreOnStartup": 1 }
```

Restart. Blank tab. Still.

## Attempt 4: prove the exit is ungraceful

At this point the question stopped being "which setting do I need" and became "what does this browser actually see when the container stops".

Chrome's restore machinery needs a session it considers *interrupted or resumable*. It rolls `Current Session` into `Last Session` on a clean exit. I asked the CDP endpoint to close the browser gracefully instead of pulling the container out from under it:

```python
# browser-level target from /json/version
ws.send(json.dumps({"id": 1, "method": "Browser.close", "params": {}}))
```

Chrome exited — and the desktop session relaunched it within seconds. `exit_type` was back to `Crashed` immediately.

That is the whole trick, and it explains all three failures: **the browser never exits cleanly.** When you stop the container, the desktop session is torn down and the browser is killed with it. No clean exit means no `Last Session` rollover, which means there is nothing for "continue where you left off" — or a `RestoreOnStartup` policy — to restore. The flag, the preference and the policy were all working correctly and all had nothing to work with.

The data was never missing. The profile's session directory was being written continuously, seconds apart, the whole time:

```
-rw------- 1 hoelee users 40722 Session_13434833255312492
-rw------- 1 hoelee users 58879 Tabs_13434833757632480
-rw------- 1 hoelee users 79361 Tabs_13434833833646133
```

## The fix: stop configuring, start keeping

If the browser cannot be trusted to remember its own session, the container can remember it instead. Two jobs, both tiny:

1. **On start**: wait for the browser, and if it came up with no real page open, re-open the URLs it had last time.
2. **Forever after**: every 60 seconds, snapshot the open page URLs.

That is a background script launched from the image's container-init hook. The version I run, trimmed to the useful parts:

```python
#!/usr/bin/env python3
"""Restore last session's tabs on start, then snapshot open tabs every 60s."""
import json, os, time, urllib.parse, urllib.request

CDP   = "http://127.0.0.1:9222"          # loopback inside the container
STORE = "/config/tabs-last.txt"          # lives in the persistent profile mount

def open_tabs():
    """http(s) page URLs, or None when the browser isn't answering yet."""
    try:
        with urllib.request.urlopen(CDP + "/json/list", timeout=5) as r:
            tabs = json.load(r)
    except Exception:
        return None
    return [t.get("url", "") for t in tabs
            if t.get("type") == "page" and t.get("url", "").startswith("http")]

saved = [u.strip() for u in open(STORE)] if os.path.exists(STORE) else []

# 1) restore — wait up to 3 minutes for the browser to come up
if saved:
    for _ in range(60):
        if open_tabs() is not None:
            break
        time.sleep(3)
    if open_tabs() == []:                      # nothing but a blank new tab
        for url in saved[:20]:
            req = urllib.request.Request(
                CDP + "/json/new?" + urllib.parse.quote(url, safe=""), method="PUT")
            urllib.request.urlopen(req, timeout=10).read()
            time.sleep(0.7)

# 2) snapshot — write even an empty list, so "I closed everything" is respected
while True:
    time.sleep(60)
    tabs = open_tabs()
    if tabs is None:
        continue
    tmp = STORE + ".tmp"
    open(tmp, "w").write("\n".join(tabs) + ("\n" if tabs else ""))
    os.replace(tmp, STORE)                     # atomic; a kill can't truncate it
```

Two design choices worth copying:

- **Restore only when the browser came up empty.** If a future Chrome version *does* restore its own tabs, the keeper stays out of the way instead of duplicating every tab.
- **The snapshot is atomic and may be empty.** Writing through a temp file plus `os.replace` means a container killed mid-write cannot leave a truncated list, and an empty snapshot is meaningful data: it means you closed everything on purpose, so the next start should stay empty.

The container-init hook that runs it:

```sh
#!/bin/sh
# clear stale singleton state (see below)
rm -f /config/profile/Singleton* 2>/dev/null
# long-running, so background it or you block container init
/usr/bin/env python3 /custom-cont-init.d/tabs_keeper.py >> /config/tabs-keeper.log 2>&1 &
exit 0
```

### Landmine 1: a stale `Singleton*` stops Chrome from starting at all

While testing restarts I hit a worse failure than a blank tab: the container came up **healthy with no browser in it**. No `google-chrome` process, CDP dead, the relay's healthcheck reporting `unhealthy`, nothing in the logs beyond the desktop starting.

The cause is the profile's `SingletonLock`/`SingletonSocket`/`SingletonCookie` symlinks. They encode "this profile is owned by PID *n* on host *h*" — and inside a container, PIDs are reused across restarts, so the next boot can find its own PID already claiming the profile and exit immediately. Deleting them at container start, before the browser launches, is the fix. It is also why that `rm -f` sits in the hook above.

### Landmine 2: `custom-cont-init.d` only runs *executable* files — and runs them with bash

The LinuxServer images execute every file in `/custom-cont-init.d`, but only if it is executable, and each one through `/bin/bash`. Which means a Python file you drop in there must **not** be executable, or it gets fed to bash and exits 2:

```
[custom-init] tabs_keeper.py: executing...
[custom-init] tabs_keeper.py: exited 2
```

The split that works: a `.sh` hook at `755` that backgrounds the long-running work, and the `.py` next to it at `644`. And anything that does not return immediately has to be backgrounded, or container init waits on it forever.

## The plumbing trap that came with it

The keeper talks to Chrome over CDP on loopback, which is why it needs no ports. If you also want to drive the browser from another machine, there is one more container in the picture, because Chrome ≥ 136 pins its devtools endpoint to `127.0.0.1:9222` and ignores `--remote-debugging-address`. The standard workaround is a `socat` sidecar sharing the browser's network namespace:

```yaml
chrome-relay:
  image: alpine/socat
  network_mode: service:chrome            # must share the browser's netns
  command: socat TCP-LISTEN:9223,fork,reuseaddr TCP:127.0.0.1:9222
  healthcheck:
    test: ["CMD-SHELL", "nc -z 127.0.0.1 9222 || exit 1"]
```

The port mapping does **not** go on the relay (it has no namespace of its own) — it goes on the browser service: `9232:9223`.

That sidecar has one behaviour you have to internalise: **restart the browser and the relay dies with it.** The namespace it was living in is rebuilt underneath it, socat keeps running and its sockets stop working, so CDP answers with `connection reset` while the container still looks fine. Fix the symptoms in this order:

```bash
docker restart chrome && docker restart chrome-relay   # order matters
```

The healthcheck above is what turns a silent failure into a visible `unhealthy` flag. If you want to drive a container browser yourself, I wrote the client side up separately in [Scraping a Bot-Walled Marketplace With a Warm Browser Session](/posts/scraping-bot-walled-marketplace-warm-browser-session/).

## What I'd do differently

- **Determine whether the process can exit cleanly before trusting any "restore on startup" mechanism.** I spent three attempts configuring a feature whose precondition — an exit the browser notices — was never met. One `Browser.close` test would have collapsed the whole search.
- **Don't hand-edit browser preferences.** They are MAC-checked against `Secure Preferences`; a plausible-looking value on disk is not a value Chrome obeys. If you need a switch to be authoritative, use a managed policy — and verify the policy is loaded instead of assuming the file is enough.
- **Check who is allowed to write the policy.** A root-owned `644` file in `/etc/opt/chrome/policies/managed` is the shape; a user-writable one is ignored silently.
- **Treat "the browser is running" and "the container is running" as different questions.** The stale-singleton failure looked like a healthy container from the outside.
- **Snapshot state instead of asking software to remember it.** Anything with a hard kill in its shutdown path — containers, kiosks, desktop sessions — is a poor place to rely on clean-exit bookkeeping. A 60-second atomic snapshot is boring, and it works.

## The result

- Tabs restored **2/2 on both browsers** across a real container restart (`restored 2/2 tabs` in the keeper log), verified over CDP afterwards
- Logins, cookies and `localStorage` confirmed surviving the same restarts — the persistent profile was never the problem
- Three stacks (two browsers plus a throwaway experiment) collapsed into one, with the relay's healthcheck making the one restart-order trap visible instead of silent

## Want this for your business?

If you want a browser that stays logged in — running on your own hardware, reachable from anywhere, behind SSO, with tab state that survives restarts and a CDP endpoint your scripts can drive — that is the setup I run and maintain.

**WhatsApp: [+60 12-797 2969](https://wa.me/60127972969)** · **Email: [[email protected]](mailto:[email protected]?subject=Self-hosted%20container%20browser)** · **[hoelee.com](https://hoelee.com)**

Website design and development is my main line of work; self-hosted infrastructure and automation is the other half of it.

文章 11 / 11：Telegram 机器人通知为什么发不出去：IPv6、DNS 和一个折腾了一整天的 400

✍️ 顾问审阅意见区 (Review Feedback)

【审阅结论】：[ ] 通过  [ ] 需微调  [ ] 需重构
【事实与数字核对】：
【人设与用词卫生】：
【CRO与转化建议】：

🇨🇳 中文版原文 (Chinese Markdown Twin)

---
title: "Telegram 机器人通知为什么发不出去：IPv6、DNS 和一个折腾了一整天的 400"
description: "一次排障记录：一个跑在 Docker 里的 Telegram 监控机器人，sendPhoto 一直返回 nginx/1.30.1 的 400，但 getMe 却正常——根因是只返回 IPv6 的 DNS 应答、一个写错的 extra_hosts IP，以及一个用 join 拼接破坏了 JPEG 二进制数据的 multipart body。"
pubDate: 2026-03-25
updatedDate: 2026-09-09
category: devops
blueprint: ai-agent-automation
blueprintLayer: L1-deterministic-orchestration
tags: [docker, telegram, python, dns, ipv6, debugging, portainer]
ogImage: /og/zh-why-telegram-bot-notifications-die.jpg
banner: /banners/why-telegram-bot-notifications-die.jpg
---

我跑着一个小监控机器人，专门盯着 Carousell 上新上架的商品，一旦有新品就通过
Telegram 通知我。有一天它突然不再通知了。容器本身是健康的，爬虫也照常往
NocoDB 里归档商品——但每一条通知都以日志里那三个字符宣告失败：

```
telegram sendPhoto failed: 400 ... nginx/1.30.1
```

`nginx/1.30.1` 这个字符串，成了我接下来一整天排障的"反派"。这是一个看似
微不足道的 bug，实际却是由三个问题层层堆叠而成的故事——以及下一次我会
如何更快地逐个定位它们。

## 问题本身

这个机器人有两件事要做：**采集**商品，以及**通知**我。采集正常，通知挂了。
日志显示 `400 Bad Request`，并指向一个自称 `nginx/1.30.1` 的服务器。

我的第一反应很教科书：**网络里有什么东西在拦截请求。** 因为
`nginx/1.30.1` 这个 header 跟我印象里的 Telegram API 对不上——而且一个
带 HTML 响应体的 400（Telegram 就算出错也返回 JSON）看起来就像本地反向
代理或 VPN 网关在请求还没离开机器之前就把它拒掉了。

这种假设很省力，但也常常是错的。下面说说我到底发现了什么。

## 我尝试了什么，以及为什么每个都失败了

### 第一轮：一定是 DNS / IPv6

我先在容器里跑了 `getMe`——它能通。然后又跑了 `sendMessage`——它挂了。
同一个 token、同一个容器，一个方法通、一个方法挂。

线索藏在两个端点的解析方式里：

```
# 容器内部
$ getent hosts api.telegram.org
2001:67c:4e8:f004::9   api.telegram.org
```

容器把 `api.telegram.org` 解析成了**只有 IPv6**——一条 `AAAA` 记录，
没有 `A` 记录。而容器跑在一个 **没有 IPv6 连通性的** Docker bridge
网络上。于是每个需要访问 `api.telegram.org` 的请求，都在试图连接一个
它根本走不到的 IPv6 地址。

修复看起来很简单：用 `extra_hosts` 把 IPv4 地址钉死。我照做了，DNS
应答变得完全符合预期：

```
149.154.167.220   api.telegram.org
```

结果通知**还是失败**，同一个 400。我那得意的修复啥也没改变。

### 第二轮：我选错了 IP

这里有一个值得记住的教训：我是**凭记忆**硬编码了 `149.154.167.220`。
等我终于去查公共解析器时，`api.telegram.org` 真正的 `A` 记录并不是它：

```
149.154.166.110   api.telegram.org   # DNS 实际返回的
149.154.167.220   api.telegram.org   # 我硬编码的
```

`167.220` 确实**在** Telegram 的 IP 段内（`149.154.160.0/20`），所以它
并不是"别人的服务器"这种错——但它不是当前活跃的端点，bot-API 的流量打到
它上面的行为是不可预测的。我把它换成 `166.110`，仍然失败。所以 IP 也
不是全部原因。

### 第三轮：同样的命令，两个相反的结果

这是整个事情开始变得诡异的一刻。在同一个容器里，几乎同一秒：

- **常驻的监控进程**（`PID 1`）→ `sendPhoto` 失败，400
- 一个**新起的 `docker exec` 进程**跑着完全相同的代码 → `sendPhoto`
  成功，200

同一个容器、同样的代码、同样的 payload、同一秒。

我逐一排除了 token、chat_id、caption 里的 emoji、Unicode 引号、图片格式、
环境变量、以及图片 URL 本身——我把那张图下载下来测过，是一张完全合法的
JPEG。两条路径的代码没有任何差别。

当你撞上这种"输入相同、输出相反"的矛盾时，答案通常是：**输入其实并不
相同**。但找到那个差异，还需要我再换一次思路。

### 第四轮：`nginx/1.30.1` 就是 Telegram

我重新读了原始的 400 响应，而不是去猜。响应是一张 HTML 的
"400 Bad Request" 页面，`Server` 头是 `nginx/1.30.1`。

**那个 nginx 就是 Telegram 自家的边缘服务器。** Telegram 的 API 背后就是
nginx（版本 1.30.1），当一个请求在到达它们应用层**之前**就已经畸形时，
nginx 会自己回一张 400 页面——没有 JSON，也没有友好的错误码。

这一下彻底改变了整个视角。请求并没有被拦截，而是**确实到达了 Telegram，
然后被 Telegram 的 nginx 以"畸形"为由拒绝了**。

而那个畸形的东西，正是我一直没仔细看的部分：`sendPhoto` 的 multipart
请求体。

## 修复

我最初的代码是这样拼 multipart 请求体的——正是那种"看着显然没问题"、
实则埋下微妙 bug 的一行：

```python
body = b"\r\n".join(body_lines)
```

这个 `join` 看起来很省事，但 JPEG 是**二进制数据**。它的字节里本身就包含
`\r\n` 这样的字节序列，出现在图片数据中间。用 `\r\n` 作为分隔符去 join，
就把本该是图片 payload 的字节给破坏了。Telegram 收到的是一个 multipart
边界被破坏的请求体，于是它的 nginx 在 bot API 还没来得及解析之前就回了个
400。

正确的做法是逐字段拼装 body，让二进制数据保持原样：

```python
boundary = "----tg" + token_hex(8)

parts = []
parts.append(f"--{boundary}\r\n".encode())
parts.append(b'Content-Disposition: form-data; name="chat_id"\r\n\r\n')
parts.append(f"{chat_id}\r\n".encode())

parts.append(f"--{boundary}\r\n".encode())
parts.append(b'Content-Disposition: form-data; name="caption"\r\n\r\n')
parts.append(f"{caption}\r\n".encode())

parts.append(f"--{boundary}\r\n".encode())
parts.append(
    b'Content-Disposition: form-data; name="photo"; '
    b'filename="image.jpg"\r\nContent-Type: image/jpeg\r\n\r\n'
)
parts.append(image_bytes)          # ← 二进制保持二进制
parts.append(b"\r\n")

parts.append(f"--{boundary}--\r\n".encode())

body = b"".join(parts)             # join 的是字节块，不是行
```

区别在于用 `b"".join(parts)` 而不是 `b"\r\n".join(...)`：每一块本身就是
一个带完整边框的片段，我们只是把它们的字节**原样拼接**，而不是在
**每个元素之间**都插入一个分隔符。

还有一个更高层的修复值得点名：**别再让 Telegram 替你去下载图片。** 我
早期的代码是把图片当作 *URL* 传过去，让 Telegram 自己去抓：

```python
tg("sendPhoto", {"photo": image_url, "caption": caption})
```

这意味着 Telegram 的服务器得主动去访问源 CDN——如果那个 CDN 不稳定、
或者对 Telegram 的爬虫做了地域封锁，你就会得到本地根本无法复现的间歇性
失败。自己先把图片下载下来、再上传字节，就消除了一整类的不稳定因素，
还给了你在发送前校验字节的机会。

## 如果重来一次我会怎么做

1. **读原始响应，而不是读假设。** `nginx/1.30.1` 在日志里躺了好几个小时，
   我却在追一个根本不存在的中间人。在凭空捏造拦截者之前，先查清楚那个
   header 到底属于谁。
2. **对硬编码的 IP 用实时解析器验证。** 记忆不是 DNS。`dig +short
   api.telegram.org` 只要三秒钟，就能省掉我整整一轮。
3. **把"输入相同、输出相反"当成谎言。** 输入从来就不相同——两个进程在某
   个我没看到的地方有差异。诚实的做法是去 diff 字节，而不是 diff 代码。
4. **永远不要用文本分隔符去手工拼接二进制数据。** 把文本和 JPEG 混在一起
   做 `b"\r\n".join()`，迟早会出破坏 bug。

## 结果

multipart 修复之后，监控机器人在下一个 tick 就真正把通知发出去了——
图片和文字都在。我还一并加了一个重试队列：一条失败的通知会保持"未发送"
状态，30 秒后自动重试，而不是悄无声息地消失。整件事耗掉了大半天的功夫，
但它变成了那种我这辈子不会再踩第二次的坑。

---

*这个机器人是我个人的项目，但教训可以推广到任何依赖外发通知的服务。如果你
有一个监控、抓取，或者需要可靠地触达用户的告警管线，我们可以聊聊。*

**我搭建可靠的监控与通知管线——Telegram/WhatsApp 告警、抓取器，以及背后的
基础设施。** 告诉我你要监控什么、通知要发到哪里：

- [WhatsApp 联系 →](https://wa.me/60127972969?text=Hi%20Hoelee%2C%20I%20need%20a%20monitoring%2Fnotification%20pipeline)
- [Email 联系 →](mailto:[email protected]?subject=Monitoring%20%2F%20notification%20pipeline)
- [hoelee.com](https://www.hoelee.com)

🇬🇧 英文版原文 (English Markdown Primary)

---
title: "Why Telegram Bot Notifications Die: IPv6, DNS, and a 400 That Took a Day"
description: "A debugging story: a Dockerized telegram monitor whose sendPhoto calls returned 400 from nginx/1.30.1 while getMe worked fine — an IPv6-only DNS answer, a wrong extra_hosts IP, and a multipart join that corrupted JPEG bytes."
pubDate: 2026-03-25
updatedDate: 2026-09-09
category: devops
blueprint: ai-agent-automation
blueprintLayer: L1-deterministic-orchestration
tags: [docker, telegram, python, dns, ipv6, debugging, portainer]
ogImage: /og/why-telegram-bot-notifications-die.jpg
banner: /banners/why-telegram-bot-notifications-die.jpg
---

I run a small monitoring bot that watches Carousell for newly-listed
listings and pings me on Telegram when something fresh appears. One day it
stopped pinging. The container was healthy, the scraper was still archiving
listings into NocoDB — but every notification died with the same three
characters in the log:

```
telegram sendPhoto failed: 400 ... nginx/1.30.1
```

That string, `nginx/1.30.1`, became the villain of a day-long debugging
session. This is the story of how a deceptively small bug turned out to be
three separate problems stacked on top of each other — and how I'd find each
one faster next time.

## The problem

The bot had two jobs: **collect** listings, and **notify** me. Collection
worked. Notification didn't. The log said `400 Bad Request` and pointed at a
server identifying itself as `nginx/1.30.1`.

My first assumption was textbook: **something in the network is intercepting
the request.** `nginx/1.30.1` is not a header I associated with Telegram's
API. A 400 with an HTML body (Telegram returns JSON even on errors) smelled
like a local reverse proxy or a VPN gateway rejecting traffic before it ever
left the machine.

Assumptions like that are cheap. They're also wrong a lot. Let me walk
through what I actually found.

## What I tried, and why each thing failed

### Round 1: it must be DNS / IPv6

I checked `getMe` from inside the container. It worked. Then I checked
`sendMessage`. It failed. Same token, same container, one method up, one down.

The giveaway was in how the two endpoints resolved:

```
# inside the container
$ getent hosts api.telegram.org
2001:67c:4e8:f004::9   api.telegram.org
```

The container resolved `api.telegram.org` to **IPv6 only** — a single `AAAA`
record and no `A` record. The container ran on a Docker bridge network with
**no IPv6 connectivity**. So every request that needed `api.telegram.org` was
trying to reach an IPv6 address it had no route to.

The fix looked obvious: pin the IPv4 address with `extra_hosts`. I did, and
the DNS answer became exactly what I expected:

```
149.154.167.220   api.telegram.org
```

And the notifications **still failed.** Same 400. My proud fix did nothing.

### Round 2: I picked the wrong IP

Here's a mistake worth remembering: I hard-coded `149.154.167.220` from
memory. When I finally queried a public resolver, the real `A` record for
`api.telegram.org` was different:

```
149.154.166.110   api.telegram.org   # what DNS actually says
149.154.167.220   api.telegram.org   # what I hard-coded
```

`167.220` is *inside* Telegram's IP range (`149.154.160.0/20`), so it's not
"wrong" in the sense of being somebody else's server — but it's not the
active endpoint, and bot-API traffic to it behaved unpredictably. I switched
to `166.110`. It still failed. So the IP wasn't the whole story either.

### Round 3: the same command, two different outcomes

This was the moment the whole thing got weird. Inside one container, at
almost the same second:

- the **long-running monitor process** (`PID 1`) → `sendPhoto` failed, 400
- a **fresh `docker exec` process** running identical code → `sendPhoto`
  succeeded, 200

Same container. Same code. Same payload. Same second.

I ruled out token, chat_id, emoji in the caption, Unicode quotes, image
format, environment variables, and the image URL itself — I download-tested
the photo and it was a perfectly valid JPEG. Nothing in the code differed
between the two paths.

When you hit a contradiction like that — "identical inputs, opposite
outputs" — the answer is usually that the inputs *aren't* identical. But
finding the difference took one more frame change.

### Round 4: `nginx/1.30.1` is Telegram

I re-read the raw 400 response instead of assuming. The response was an HTML
"400 Bad Request" page, and the `Server` header was `nginx/1.30.1`.

**That nginx is Telegram's own edge.** Telegram's API sits behind nginx
(version 1.30.1), and when a request is malformed *before* it reaches their
application layer, nginx itself answers with a plain 400 page — no JSON, no
friendly error code.

That reframed everything. The request wasn't being intercepted. It was
**reaching Telegram, and Telegram's nginx was rejecting it as malformed.**

And the malformed thing was the one part I hadn't scrutinized: the
`sendPhoto` multipart body.

## The fix

My original code built the multipart body like this — the kind of
"obviously fine" line that ships a subtle bug:

```python
body = b"\r\n".join(body_lines)
```

That `join` looks convenient, but a JPEG is **binary data**. Its bytes include
`\r\n` byte sequences in the middle of the image data. Joining with `\r\n`
as the delimiter corrupts the exact bytes that are supposed to be the image
payload. Telegram receives a body whose multipart boundaries are broken, so
its nginx rejects the request with a 400 before the bot API ever parses it.

The correct approach is to assemble the body field by field, keeping binary
data untouched:

```python
boundary = "----tg" + token_hex(8)

parts = []
parts.append(f"--{boundary}\r\n".encode())
parts.append(b'Content-Disposition: form-data; name="chat_id"\r\n\r\n')
parts.append(f"{chat_id}\r\n".encode())

parts.append(f"--{boundary}\r\n".encode())
parts.append(b'Content-Disposition: form-data; name="caption"\r\n\r\n')
parts.append(f"{caption}\r\n".encode())

parts.append(f"--{boundary}\r\n".encode())
parts.append(
    b'Content-Disposition: form-data; name="photo"; '
    b'filename="image.jpg"\r\nContent-Type: image/jpeg\r\n\r\n'
)
parts.append(image_bytes)          # ← binary stays binary
parts.append(b"\r\n")

parts.append(f"--{boundary}--\r\n".encode())

body = b"".join(parts)             # join bytes, not lines
```

The difference is `b"".join(parts)` instead of `b"\r\n".join(...)`: each part
is already a complete chunk with its own framing, and we concatenate them
verbatim rather than inserting a delimiter *between every element*.

There's a second, higher-level fix worth naming: **stop asking Telegram to
download the image for you.** My earlier code passed the image as a *URL* and
let Telegram fetch it:

```python
tg("sendPhoto", {"photo": image_url, "caption": caption})
```

That means Telegram's servers have to reach out to the source CDN — and if
that CDN is flaky or geo-blocks Telegram's crawlers, you get intermittent
failures that are impossible to reproduce locally. Downloading the image
yourself, then uploading the bytes, removes an entire class of flakiness and
gives you a chance to validate the bytes before you send them.

## What I'd do differently

1. **Read the raw response, not the assumption.** `nginx/1.30.1` sat in logs
   for hours while I chased phantom proxies. Look up who owns the header
   before inventing a man-in-the-middle.
2. **Verify a hard-coded IP against a live resolver.** Memory is not DNS.
   `dig +short api.telegram.org` takes three seconds and would have saved me
   a whole round.
3. **Treat "identical inputs, opposite outputs" as a lie.** The inputs were
   never identical — the two processes differed somewhere I wasn't looking.
   The honest move is to diff the bytes, not the code.
4. **Never hand-join binary data with a text delimiter.** `b"\r\n".join()` on
   a mix of text and JPEG is a corruption bug waiting to happen.

## The result

After the multipart fix, the monitor pushed a real notification through on
the next tick — image and all. I also folded in a retry queue so a failed
notification stays marked un-sent and is retried 30 seconds later instead of
silently vanishing. The whole episode cost the better part of a day, but it
turned into exactly the kind of bug I'll never trip over twice.

---

*The bot is a personal project, but the lesson generalizes to any service
that relies on outbound notifications. If you've got a monitor, a scraper, or
an alert pipeline that needs to reach people reliably, let's talk.*

**I build reliable notification and monitoring pipelines — Telegram/WhatsApp
alerts, scrapers, and the infrastructure around them.** Tell me what you need
to watch and where you need it delivered:

- [WhatsApp me →](https://wa.me/60127972969?text=Hi%20Hoelee%2C%20I%20need%20a%20monitoring%2Fnotification%20pipeline)
- [Email me →](mailto:[email protected]?subject=Monitoring%20%2F%20notification%20pipeline)
- [hoelee.com](https://www.hoelee.com)
Reference in new issue View Git Blame Copy Permalink