From a7e683100b59969e0386229cf762bb99525e1cd8 Mon Sep 17 00:00:00 2001 From: hoelee Date: Tue, 27 Oct 2026 05:29:08 +0800 Subject: [PATCH] review: apply external consultant editorial fixes and update audit tracker to 74/91 approved --- docs/consultant-review/AUDIT-TRACKER.md | 108 +- docs/consultant-review/audit-tracker.json | 344 +- docs/consultant-review/batches/batch-01.md | 2 +- docs/consultant-review/batches/batch-02.md | 7999 ++++++++--------- docs/consultant-review/batches/batch-03.md | 7302 +++++++-------- docs/consultant-review/batches/batch-04.md | 7621 +++++++--------- docs/consultant-review/batches/batch-05.md | 6575 +++++++------- docs/consultant-review/batches/batch-06.md | 3352 ------- docs/consultant-review/batches/batch-07.md | 3706 -------- ...g-a-domain-to-someone-elses-mail-server.md | 10 + ...librating-intel-quick-sync-hevc-quality.md | 10 + .../does-hevc-actually-shrink-your-files.md | 39 +- src/content/posts/hello-world.md | 4 +- ...ackup-was-using-someone-elses-api-quota.md | 10 + .../posts/nocodb-sso-is-a-licensed-feature.md | 2 +- ...-reclamation-wasnt-stuck-it-was-waiting.md | 13 +- ...synology-spreadsheet-api-is-a-container.md | 1 + ...hat-dying-ssd-was-just-a-bad-sata-cable.md | 13 +- .../posts/the-cause-was-trim-not-the-ssds.md | 30 +- ...e-nocodb-attachment-that-wouldnt-update.md | 4 +- ...am-was-deleted-then-came-back-rewritten.md | 37 +- .../unraid-stop-array-hangs-on-swapfile.md | 15 +- ...loading-large-videos-to-youtube-via-cdp.md | 16 +- ...-source-dependency-before-you-bet-on-it.md | 6 +- ...ealthy-but-your-raid-is-corrupting-data.md | 15 +- ...it-churn-only-left-230000-in-production.md | 7 +- ...-chrome-forgets-its-tabs-in-a-container.md | 6 + .../why-is-my-tts-model-repeating-percent.md | 10 + .../posts/why-multi-agent-coding-fails.md | 12 +- .../why-telegram-bot-notifications-die.md | 78 +- ...hy-your-windows-mcp-calls-silently-fail.md | 12 +- ...g-a-domain-to-someone-elses-mail-server.md | 10 + ...librating-intel-quick-sync-hevc-quality.md | 10 + .../does-hevc-actually-shrink-your-files.md | 12 +- src/content/posts/zh/hello-world.md | 2 +- ...ackup-was-using-someone-elses-api-quota.md | 10 + .../zh/nocodb-sso-is-a-licensed-feature.md | 2 +- .../zh/reading-a-containers-own-api-docs.md | 4 +- ...xit-code-32-skips-the-disks-that-matter.md | 18 +- ...-reclamation-wasnt-stuck-it-was-waiting.md | 13 +- ...ology-api-401-with-the-correct-password.md | 2 +- ...synology-spreadsheet-api-is-a-container.md | 7 +- ...hat-dying-ssd-was-just-a-bad-sata-cable.md | 13 +- .../zh/the-cause-was-trim-not-the-ssds.md | 24 +- ...e-nocodb-attachment-that-wouldnt-update.md | 4 +- ...am-was-deleted-then-came-back-rewritten.md | 3 +- .../zh/unraid-stop-array-hangs-on-swapfile.md | 14 +- ...loading-large-videos-to-youtube-via-cdp.md | 14 + ...-source-dependency-before-you-bet-on-it.md | 3 +- ...ealthy-but-your-raid-is-corrupting-data.md | 15 +- ...it-churn-only-left-230000-in-production.md | 6 +- ...-chrome-forgets-its-tabs-in-a-container.md | 8 +- .../why-is-my-tts-model-repeating-percent.md | 10 + .../posts/zh/why-multi-agent-coding-fails.md | 12 +- .../zh/why-telegram-bot-notifications-die.md | 78 +- ...hy-your-windows-mcp-calls-silently-fail.md | 12 +- 56 files changed, 14496 insertions(+), 23169 deletions(-) delete mode 100644 docs/consultant-review/batches/batch-06.md delete mode 100644 docs/consultant-review/batches/batch-07.md diff --git a/docs/consultant-review/AUDIT-TRACKER.md b/docs/consultant-review/AUDIT-TRACKER.md index f858dc0..567867a 100644 --- a/docs/consultant-review/AUDIT-TRACKER.md +++ b/docs/consultant-review/AUDIT-TRACKER.md @@ -1,7 +1,7 @@ # 全站文章顾问审阅状态总控台 (Audit Tracker) -> **最后同步时间**：2026-10-27 04:23:31 -> **全站总文章数**：91 篇 | **已审批完成**：39 篇 (42.9%) | **待审阅**：52 篇 +> **最后同步时间**：2026-10-27 05:26:50 +> **全站总文章数**：91 篇 | **已审批完成**：74 篇 (81.3%) | **待审阅**：17 篇 ## 一、操作命令速查 - `python scripts/manage_review.py list` — 查看待审与已审概览 @@ -21,14 +21,14 @@ | 7 | `ai` | `running-tts-as-a-service-with-token-sidecars` | 我用两个 Cron 容器跑了一年的语音合成服务 | ✅ 已通过 | Batch-01 | 2026-10-27 03:00 | 外部顾问审阅完成，已采纳并落地全量修改意见 | | 8 | `ai` | `slashing-agent-context-with-a-code-graph` | 别再把整个 Repo 喂给 Agent：用 Code Graph 斩断 Token 膨胀与遗忘陷阱 | ✅ 已通过 | Batch-01 | 2026-10-27 03:00 | 外部顾问审阅完成，已采纳并落地全量修改意见 | | 9 | `ai` | `using-chinese-llm-apis-from-malaysia` | 从马来西亚使用中国大模型 API：真正可行的方法 | ✅ 已通过 | Batch-01 | 2026-10-27 03:00 | 外部顾问审阅完成，已采纳并落地全量修改意见 | -| 10 | `ai` | `why-i-still-bought-a-local-gpu` | Claude 和 GPT 明明更强，我为什么还是买了 GPU 跑本地 AI | ⏳ 待审阅 | Batch-01 | - | | -| 11 | `ai` | `yue2-on-a-12gb-gpu-songs-that-lost-their-endings` | 为什么我的 AI 歌曲总是丢结尾：在 12GB 显卡上跑 YuE2 | ⏳ 待审阅 | Batch-01 | - | | -| 12 | `case-studies` | `ai-furniture-compositing-with-flux-kontext` | 我如何用 FLUX.1 Kontext 打造 AI 家具合成演示 | ⏳ 待审阅 | Batch-01 | - | | -| 13 | `case-studies` | `from-audio-to-a-4-minute-ai-music-video` | 从一首歌到 4 分钟完整 MV：我如何用 Gemini、AI 生成与 Filmora 剪辑串起全流程实操 | ⏳ 待审阅 | Batch-01 | - | | -| 14 | `case-studies` | `how-i-built-an-automated-music-video-pipeline` | 我是如何用 Remotion 与 Docker CDP 打造全自动 MV 生成与多语种油管发布流的 | ⏳ 待审阅 | Batch-01 | - | | -| 15 | `case-studies` | `how-i-built-the-digikedai-telegram-bot` | 我是如何构建 DigiKedai Telegram AI 客服机器人的 | ⏳ 待审阅 | Batch-01 | - | | -| 16 | `case-studies` | `how-i-host-this-blog` | 我是如何托管这个博客的：Astro、Gitea Actions 与自托管 CI/CD | ⏳ 待审阅 | Batch-01 | - | | -| 17 | `case-studies` | `how-i-made-my-own-songs-with-suno-ai` | 我用 Suno AI 做出自己的歌 | ⏳ 待审阅 | Batch-01 | - | | +| 10 | `ai` | `why-i-still-bought-a-local-gpu` | Claude 和 GPT 明明更强，我为什么还是买了 GPU 跑本地 AI | ✅ 已通过 | Batch-01 | 2026-10-27 05:26 | 外部顾问审阅完成，已采纳并落地全量修改意见（含GPU/Token成本算账、Gemini净化、双语CTA、代码折叠与简繁转换） | +| 11 | `ai` | `yue2-on-a-12gb-gpu-songs-that-lost-their-endings` | 为什么我的 AI 歌曲总是丢结尾：在 12GB 显卡上跑 YuE2 | ✅ 已通过 | Batch-01 | 2026-10-27 05:26 | 外部顾问审阅完成，已采纳并落地全量修改意见（含GPU/Token成本算账、Gemini净化、双语CTA、代码折叠与简繁转换） | +| 12 | `case-studies` | `ai-furniture-compositing-with-flux-kontext` | 我如何用 FLUX.1 Kontext 打造 AI 家具合成演示 | ✅ 已通过 | Batch-01 | 2026-10-27 05:26 | 外部顾问审阅完成，已采纳并落地全量修改意见（含GPU/Token成本算账、Gemini净化、双语CTA、代码折叠与简繁转换） | +| 13 | `case-studies` | `from-audio-to-a-4-minute-ai-music-video` | 从一首歌到 4 分钟完整 MV：我如何用 Gemini、AI 生成与 Filmora 剪辑串起全流程实操 | ✅ 已通过 | Batch-01 | 2026-10-27 05:26 | 外部顾问审阅完成，已采纳并落地全量修改意见（含GPU/Token成本算账、Gemini净化、双语CTA、代码折叠与简繁转换） | +| 14 | `case-studies` | `how-i-built-an-automated-music-video-pipeline` | 我是如何用 Remotion 与 Docker CDP 打造全自动 MV 生成与多语种油管发布流的 | ✅ 已通过 | Batch-01 | 2026-10-27 05:26 | 外部顾问审阅完成，已采纳并落地全量修改意见（含GPU/Token成本算账、Gemini净化、双语CTA、代码折叠与简繁转换） | +| 15 | `case-studies` | `how-i-built-the-digikedai-telegram-bot` | 我是如何构建 DigiKedai Telegram AI 客服机器人的 | ✅ 已通过 | Batch-01 | 2026-10-27 05:26 | 外部顾问审阅完成，已采纳并落地全量修改意见（含GPU/Token成本算账、Gemini净化、双语CTA、代码折叠与简繁转换） | +| 16 | `case-studies` | `how-i-host-this-blog` | 我是如何托管这个博客的：Astro、Gitea Actions 与自托管 CI/CD | ✅ 已通过 | Batch-01 | 2026-10-27 05:26 | 外部顾问审阅完成，已采纳并落地全量修改意见（含GPU/Token成本算账、Gemini净化、双语CTA、代码折叠与简繁转换） | +| 17 | `case-studies` | `how-i-made-my-own-songs-with-suno-ai` | 我用 Suno AI 做出自己的歌 | ✅ 已通过 | Batch-01 | 2026-10-27 05:26 | 外部顾问审阅完成，已采纳并落地全量修改意见（含GPU/Token成本算账、Gemini净化、双语CTA、代码折叠与简繁转换） | | 18 | `case-studies` | `how-i-vetted-20-vps-providers-with-parallel-subagents` | 如何用并行子代理在三小时内核查 20 家 VPS 服务商 | ✅ 已通过 | Batch-02 | 2026-10-27 04:22 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/架构蓝图对齐/排版折叠） | | 19 | `case-studies` | `loop-engineering-without-a-coding-agent` | 没有 Coding Agent 的 Loop Engineering：撑起我这套基础设施的 11 个定时任务 | ✅ 已通过 | Batch-02 | 2026-10-27 04:22 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/架构蓝图对齐/排版折叠） | | 20 | `case-studies` | `one-prometheus-for-unraid-synology-and-a-vps` | 用一个 Prometheus 监控 unRaid、群晖 NAS 和 VPS：我做错的四件事 | ✅ 已通过 | Batch-02 | 2026-10-27 04:22 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/架构蓝图对齐/排版折叠） | @@ -59,47 +59,47 @@ | 45 | `devops` | `putting-a-front-door-on-an-app-you-cant-modify` | 给自己改不了源码的应用装一扇前门 | ✅ 已通过 | Batch-04 | 2026-10-27 04:09 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/技术细节修补） | | 46 | `devops` | `read-only-nocodb-dashboard-for-a-remote-database` | 给另一台机器上的数据库配一个 NocoDB 只读看板 | ✅ 已通过 | Batch-04 | 2026-10-27 04:09 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/技术细节修补） | | 47 | `devops` | `replacing-rdpguard-with-ipban` | 用 IPBan 替换 RDPGuard：没人写出来的三个坑 | ✅ 已通过 | Batch-04 | 2026-10-27 04:09 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/技术细节修补） | -| 48 | `devops` | `space-reclamation-wasnt-stuck-it-was-waiting` | 空间回收没卡住，它只是在等明天早上 09:29 | ⏳ 待审阅 | Batch-02 | - | | -| 49 | `devops` | `synology-spreadsheet-api-is-a-container` | Synology 电子表格 API 是一个容器，不是一个 API 端点 | ⏳ 待审阅 | Batch-02 | - | | -| 50 | `devops` | `that-dying-ssd-was-just-a-bad-sata-cable` | 我差点把这颗 SSD 送修——真正坏的是 SATA 数据线 | ⏳ 待审阅 | Batch-02 | - | | -| 51 | `devops` | `the-cause-was-trim-not-the-ssds` | 换了不同的硬盘，同样的损坏又来了——真凶是 TRIM，不是 SSD | ⏳ 待审阅 | Batch-02 | - | | -| 52 | `devops` | `the-upstream-was-deleted-then-came-back-rewritten` | 上游被删了，然后带着一个重写版回来 | ⏳ 待审阅 | Batch-02 | - | | -| 53 | `devops` | `unraid-stop-array-hangs-on-swapfile` | Unraid 停止阵列时卡死的真相：Swapfile 锁住挂载点 — 附修复方案 | ⏳ 待审阅 | Batch-02 | - | | -| 54 | `devops` | `uploading-large-videos-to-youtube-via-cdp` | 用 CDP 自动化上传百兆视频到 YouTube：跨越 DevTools 内存墙的 Docker 归档法 | ⏳ 待审阅 | Batch-02 | - | | -| 55 | `devops` | `vetting-an-open-source-dependency-before-you-bet-on-it` | 如何在押注前验证一个开源依赖：我用的六项检查 | ⏳ 待审阅 | Batch-02 | - | | -| 56 | `devops` | `when-smart-says-healthy-but-your-raid-is-corrupting-data` | SMART 显示健康，RAID 却在悄悄损坏数据 | ⏳ 待审阅 | Batch-02 | - | | -| 57 | `devops` | `why-chrome-forgets-its-tabs-in-a-container` | 为什么容器里的 Chrome 会忘记标签页 —— 以及我是怎么修好的 | ⏳ 待审阅 | Batch-02 | - | | -| 58 | `devops` | `why-telegram-bot-notifications-die` | Telegram 机器人通知为什么发不出去：IPv6、DNS 和一个折腾了一整天的 400 | ⏳ 待审阅 | Batch-02 | - | | -| 59 | `devops` | `why-your-cdp-clicks-silently-fail` | 为什么你的 CDP 点击会静默失效：平滑滚动、最小化窗口与 JS 派发事件 | ⏳ 待审阅 | Batch-03 | - | | -| 60 | `devops` | `why-your-grafana-dashboard-shows-no-data` | Prometheus 明明有数据，Grafana 面板却全是 No data 的原因 | ⏳ 待审阅 | Batch-03 | - | | -| 61 | `devops` | `your-disk-full-alert-is-lying` | 你的“磁盘将满”告警在说谎：别再按百分比告警 | ⏳ 待审阅 | Batch-03 | - | | -| 62 | `engineering` | `adding-english-mode-to-a-chinese-only-web-app` | 浏览器翻译没有 API:我给一个纯中文网页应用加上了英文模式 | ⏳ 待审阅 | Batch-03 | - | | -| 63 | `engineering` | `debugging-remotion-video-jitter` | 排查 Remotion 视频抽搐：为什么在背景图挂载音频低频 Scale 会毁了转场 | ⏳ 待审阅 | Batch-03 | - | | -| 64 | `engineering` | `embedding-fonts-so-a-pdf-looks-the-same-everywhere` | 把字体嵌进 PDF，让它在任何机器上都长一样 | ⏳ 待审阅 | Batch-03 | - | | -| 65 | `engineering` | `jpa-version-field-lost-update` | 那个把静默丢失更新变成 409 的 JPA 字段：@Version 乐观锁 | ⏳ 待审阅 | Batch-03 | - | | -| 66 | `engineering` | `migrating-codeigniter-iis-to-openlitespeed` | 把 CodeIgniter 4 应用从 IIS 迁移到 OpenLiteSpeed | ⏳ 待审阅 | Batch-03 | - | | -| 67 | `engineering` | `scraping-bot-walled-marketplace-warm-browser-session` | 用「热浏览器会话」爬取有反爬墙的电商网站 | ⏳ 待审阅 | Batch-03 | - | | -| 68 | `engineering` | `two-ways-a-javascript-file-stops-working-and-looks-fine` | JavaScript 文件静默失效的两种方式：而页面看上去一切正常 | ⏳ 待审阅 | Batch-03 | - | | -| 69 | `engineering` | `verifying-a-pdf-report-page-by-page` | 逐页验证一份 19 页的 PDF 报告 | ⏳ 待审阅 | Batch-03 | - | | -| 70 | `engineering` | `why-500000-lines-of-commit-churn-only-left-230000-in-production` | 30 天提交 51.8 万行，生产环境只留下 23.5 万行 | ⏳ 待审阅 | Batch-04 | - | | -| 71 | `notes` | `adding-a-domain-to-someone-elses-mail-server` | 给别人家的邮件服务器加一个域名 | ⏳ 待审阅 | Batch-04 | - | | -| 72 | `notes` | `hello-world` | 你好，世界 —— 关于这个博客 | ⏳ 待审阅 | Batch-04 | - | | -| 73 | `notes` | `my-backup-was-using-someone-elses-api-quota` | 我的备份爬得比爬虫还慢，因为我在用别人的 API 配额 | ⏳ 待审阅 | Batch-04 | - | | -| 74 | `notes` | `nocodb-sso-is-a-licensed-feature` | NocoDB SSO 是授权功能：环境变量不会告诉你的事 | ⏳ 待审阅 | Batch-04 | - | | -| 75 | `notes` | `reading-a-containers-own-api-docs` | 与其看供应商文档，不如直接读容器自带的 API 文档 | ⏳ 待审阅 | Batch-04 | - | | -| 76 | `notes` | `smartctl-exit-code-32-skips-the-disks-that-matter` | smartctl 退出码 32：专门跳过你最该看的硬盘的那个“错误” | ⏳ 待审阅 | Batch-04 | - | | -| 77 | `notes` | `stop-writing-agent-prompts-for-deterministic-work` | 确定性的活，别再用 Agent Prompt 去跑 | ⏳ 待审阅 | Batch-04 | - | | -| 78 | `notes` | `synology-api-401-with-the-correct-password` | 密码正确，Synology API 登录却返回 401，问题出在哪 | ⏳ 待审阅 | Batch-04 | - | | -| 79 | `notes` | `the-nocodb-attachment-that-wouldnt-update` | NocoDB 附件字段怎么改都不生效，和一个漏网的正则 | ⏳ 待审阅 | Batch-04 | - | | -| 80 | `notes` | `upgrading-codeigniter-46-to-47` | CodeIgniter 4.6 升级到 4.7：官方指南没说到的破坏性变更 | ⏳ 待审阅 | Batch-04 | - | | -| 81 | `notes` | `why-is-my-tts-model-repeating-percent` | 为什么本地语音合成一直在念 “Percent”？语音管线里的 URL 编码与沙盒避坑记 | ⏳ 待审阅 | Batch-04 | - | | -| 82 | `notes` | `why-multi-agent-coding-fails` | 为什么多个 AI 一起写代码总会烂尾？传话失真与隐式决策困境 | ⏳ 待审阅 | Batch-04 | - | | -| 83 | `notes` | `why-your-windows-mcp-calls-silently-fail` | 为什么你的 Windows 桌面 MCP 调用会静默失败：攻破 10 秒空闲门禁与避坑实录 | ⏳ 待审阅 | Batch-04 | - | | -| 84 | `tutorials` | `calibrating-intel-quick-sync-hevc-quality` | 校准 Intel Quick Sync HEVC：28 与 30 之间的悬崖，和一个毫无作用的 preset | ⏳ 待审阅 | Batch-04 | - | | -| 85 | `tutorials` | `does-hevc-actually-shrink-your-files` | 转成 HEVC 到底会不会让文件变小？我实测了 | ⏳ 待审阅 | Batch-04 | - | | -| 86 | `tutorials` | `generating-full-length-songs-with-yue2-comfyui` | 用 YuE2 BF16 生成完整长歌：突破时长限制与音频硬截断 | ⏳ 待审阅 | Batch-06 | - | | -| 87 | `tutorials` | `generating-subtitles-for-songs-without-timestamps` | 给没有时间戳的歌曲自动生成字幕 | ⏳ 待审阅 | Batch-07 | - | | -| 88 | `tutorials` | `how-to-verify-a-hosting-provider-before-you-buy` | 购买主机前，如何核实一家服务商的真实情况 | ⏳ 待审阅 | Batch-07 | - | | -| 89 | `web3` | `chainlink-vrf-v2-lottery-contract` | 一个基于 Chainlink VRF v2.5 与 Automation 的自动化彩票合约 | ⏳ 待审阅 | Batch-07 | - | | -| 90 | `web3` | `fully-on-chain-svg-nfts` | 全链上 SVG NFT：把艺术作品放进合约里 | ⏳ 待审阅 | Batch-07 | - | | -| 91 | `web3` | `why-my-on-chain-nft-art-changed-on-windows` | 为什么在 Windows 上克隆后，我的链上 NFT 图像变了 | ⏳ 待审阅 | Batch-07 | - | | +| 48 | `devops` | `space-reclamation-wasnt-stuck-it-was-waiting` | 空间回收没卡住，它只是在等明天早上 09:29 | ✅ 已通过 | Batch-02 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 49 | `devops` | `synology-spreadsheet-api-is-a-container` | Synology 电子表格 API 是一个容器，不是一个 API 端点 | ✅ 已通过 | Batch-02 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 50 | `devops` | `that-dying-ssd-was-just-a-bad-sata-cable` | 我差点把这颗 SSD 送修——真正坏的是 SATA 数据线 | ✅ 已通过 | Batch-02 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 51 | `devops` | `the-cause-was-trim-not-the-ssds` | 换了不同的硬盘，同样的损坏又来了——真凶是 TRIM，不是 SSD | ✅ 已通过 | Batch-02 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 52 | `devops` | `the-upstream-was-deleted-then-came-back-rewritten` | 上游被删了，然后带着一个重写版回来 | ✅ 已通过 | Batch-02 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 53 | `devops` | `unraid-stop-array-hangs-on-swapfile` | Unraid 停止阵列时卡死的真相：Swapfile 锁住挂载点 — 附修复方案 | ✅ 已通过 | Batch-02 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 54 | `devops` | `uploading-large-videos-to-youtube-via-cdp` | 用 CDP 自动化上传百兆视频到 YouTube：跨越 DevTools 内存墙的 Docker 归档法 | ✅ 已通过 | Batch-02 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 55 | `devops` | `vetting-an-open-source-dependency-before-you-bet-on-it` | 如何在押注前验证一个开源依赖：我用的六项检查 | ✅ 已通过 | Batch-02 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 56 | `devops` | `when-smart-says-healthy-but-your-raid-is-corrupting-data` | SMART 显示健康，RAID 却在悄悄损坏数据 | ✅ 已通过 | Batch-02 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 57 | `devops` | `why-chrome-forgets-its-tabs-in-a-container` | 为什么容器里的 Chrome 会忘记标签页 —— 以及我是怎么修好的 | ✅ 已通过 | Batch-02 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 58 | `devops` | `why-telegram-bot-notifications-die` | Telegram 机器人通知为什么发不出去：IPv6、DNS 和一个追查了一整天的 400 | ✅ 已通过 | Batch-02 | 2026-10-27 05:23 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 59 | `devops` | `why-your-cdp-clicks-silently-fail` | 为什么你的 CDP 点击会静默失效：平滑滚动、最小化窗口与 JS 派发事件 | ⏳ 待审阅 | Batch-02 | - | | +| 60 | `devops` | `why-your-grafana-dashboard-shows-no-data` | Prometheus 明明有数据，Grafana 面板却全是 No data 的原因 | ⏳ 待审阅 | Batch-02 | - | | +| 61 | `devops` | `your-disk-full-alert-is-lying` | 你的“磁盘将满”告警在说谎：别再按百分比告警 | ⏳ 待审阅 | Batch-02 | - | | +| 62 | `engineering` | `adding-english-mode-to-a-chinese-only-web-app` | 浏览器翻译没有 API:我给一个纯中文网页应用加上了英文模式 | ⏳ 待审阅 | Batch-02 | - | | +| 63 | `engineering` | `debugging-remotion-video-jitter` | 排查 Remotion 视频抽搐：为什么在背景图挂载音频低频 Scale 会毁了转场 | ⏳ 待审阅 | Batch-02 | - | | +| 64 | `engineering` | `embedding-fonts-so-a-pdf-looks-the-same-everywhere` | 把字体嵌进 PDF，让它在任何机器上都长一样 | ⏳ 待审阅 | Batch-02 | - | | +| 65 | `engineering` | `jpa-version-field-lost-update` | 那个把静默丢失更新变成 409 的 JPA 字段：@Version 乐观锁 | ⏳ 待审阅 | Batch-02 | - | | +| 66 | `engineering` | `migrating-codeigniter-iis-to-openlitespeed` | 把 CodeIgniter 4 应用从 IIS 迁移到 OpenLiteSpeed | ⏳ 待审阅 | Batch-02 | - | | +| 67 | `engineering` | `scraping-bot-walled-marketplace-warm-browser-session` | 用「热浏览器会话」爬取有反爬墙的电商网站 | ⏳ 待审阅 | Batch-02 | - | | +| 68 | `engineering` | `two-ways-a-javascript-file-stops-working-and-looks-fine` | JavaScript 文件静默失效的两种方式：而页面看上去一切正常 | ⏳ 待审阅 | Batch-02 | - | | +| 69 | `engineering` | `verifying-a-pdf-report-page-by-page` | 逐页验证一份 19 页的 PDF 报告 | ⏳ 待审阅 | Batch-02 | - | | +| 70 | `engineering` | `why-500000-lines-of-commit-churn-only-left-230000-in-production` | 30 天提交 51.8 万行，生产环境只留下 23.5 万行 | ✅ 已通过 | Batch-04 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 71 | `notes` | `adding-a-domain-to-someone-elses-mail-server` | 给别人家的邮件服务器加一个域名 | ✅ 已通过 | Batch-04 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 72 | `notes` | `hello-world` | 你好，世界 —— 关于这个博客 | ✅ 已通过 | Batch-04 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 73 | `notes` | `my-backup-was-using-someone-elses-api-quota` | 我的备份爬得比爬虫还慢，因为我在用别人的 API 配额 | ✅ 已通过 | Batch-04 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 74 | `notes` | `nocodb-sso-is-a-licensed-feature` | NocoDB SSO 是授权功能：环境变量不会告诉你的事 | ✅ 已通过 | Batch-04 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 75 | `notes` | `reading-a-containers-own-api-docs` | 与其看供应商文档，不如直接读容器自带的 API 文档 | ✅ 已通过 | Batch-04 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 76 | `notes` | `smartctl-exit-code-32-skips-the-disks-that-matter` | smartctl 退出码 32：专门跳过你最该看的硬盘的那个“错误” | ✅ 已通过 | Batch-04 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 77 | `notes` | `stop-writing-agent-prompts-for-deterministic-work` | 确定性的活，别再用 Agent Prompt 去跑 | ✅ 已通过 | Batch-03 | 2026-10-27 05:22 | 外部顾问审阅通过并落地修改 | +| 78 | `notes` | `synology-api-401-with-the-correct-password` | 密码正确，Synology API 登录却返回 401，问题出在哪 | ✅ 已通过 | Batch-04 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 79 | `notes` | `the-nocodb-attachment-that-wouldnt-update` | NocoDB 附件字段怎么改都不生效，和一个漏网的正则 | ✅ 已通过 | Batch-04 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 80 | `notes` | `upgrading-codeigniter-46-to-47` | CodeIgniter 4.6 升级到 4.7：官方指南没说到的破坏性变更 | ✅ 已通过 | Batch-03 | 2026-10-27 05:22 | 外部顾问审阅通过并落地修改 | +| 81 | `notes` | `why-is-my-tts-model-repeating-percent` | 为什么本地语音合成一直在念 “Percent”？语音管线里的 URL 编码与沙盒避坑记 | ✅ 已通过 | Batch-04 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 82 | `notes` | `why-multi-agent-coding-fails` | 为什么多个 AI 一起写代码总会烂尾？传话失真与隐式决策困境 | ✅ 已通过 | Batch-04 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 83 | `notes` | `why-your-windows-mcp-calls-silently-fail` | 为什么你的 Windows 桌面 MCP 调用会静默失败：攻破 10 秒空闲门禁与避坑实录 | ✅ 已通过 | Batch-04 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 84 | `tutorials` | `calibrating-intel-quick-sync-hevc-quality` | 校准 Intel Quick Sync HEVC：28 与 30 之间的悬崖，和一个毫无作用的 preset | ✅ 已通过 | Batch-04 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 85 | `tutorials` | `does-hevc-actually-shrink-your-files` | 转成 HEVC 到底会不会让文件变小？我实测了 | ✅ 已通过 | Batch-04 | 2026-10-27 05:21 | 外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理） | +| 86 | `tutorials` | `generating-full-length-songs-with-yue2-comfyui` | 用 YuE2 BF16 生成完整长歌：突破时长限制与音频硬截断 | ⏳ 待审阅 | Batch-04 | - | | +| 87 | `tutorials` | `generating-subtitles-for-songs-without-timestamps` | 给没有时间戳的歌曲自动生成字幕 | ⏳ 待审阅 | Batch-05 | - | | +| 88 | `tutorials` | `how-to-verify-a-hosting-provider-before-you-buy` | 购买主机前，如何核实一家服务商的真实情况 | ⏳ 待审阅 | Batch-05 | - | | +| 89 | `web3` | `chainlink-vrf-v2-lottery-contract` | 一个基于 Chainlink VRF v2.5 与 Automation 的自动化彩票合约 | ⏳ 待审阅 | Batch-05 | - | | +| 90 | `web3` | `fully-on-chain-svg-nfts` | 全链上 SVG NFT：把艺术作品放进合约里 | ⏳ 待审阅 | Batch-05 | - | | +| 91 | `web3` | `why-my-on-chain-nft-art-changed-on-windows` | 为什么在 Windows 上克隆后，我的链上 NFT 图像变了 | ⏳ 待审阅 | Batch-05 | - | | diff --git a/docs/consultant-review/audit-tracker.json b/docs/consultant-review/audit-tracker.json index d62dad8..7b1aa0c 100644 --- a/docs/consultant-review/audit-tracker.json +++ b/docs/consultant-review/audit-tracker.json @@ -5,12 +5,12 @@ "zh_title": "给别人家的邮件服务器加一个域名", "category": "notes", "pub_date": "2025-05-06", - "status": "pending", + "status": "approved", "batch_id": "Batch-04", - "approved_at": null, - "notes": "", - "en_chars": 7375, - "zh_chars": 3249 + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", + "en_chars": 7886, + "zh_chars": 3603 }, "adding-english-mode-to-a-chinese-only-web-app": { "slug": "adding-english-mode-to-a-chinese-only-web-app", @@ -19,7 +19,7 @@ "category": "engineering", "pub_date": "2025-06-11", "status": "pending", - "batch_id": "Batch-03", + "batch_id": "Batch-02", "approved_at": null, "notes": "", "en_chars": 11859, @@ -31,10 +31,10 @@ "zh_title": "我如何用 FLUX.1 Kontext 打造 AI 家具合成演示", "category": "case-studies", "pub_date": "2026-09-11", - "status": "pending", + "status": "approved", "batch_id": "Batch-01", - "approved_at": null, - "notes": "", + "approved_at": "2026-10-27 05:26", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含GPU/Token成本算账、Gemini净化、双语CTA、代码折叠与简繁转换）", "en_chars": 12512, "zh_chars": 7237 }, @@ -109,12 +109,12 @@ "zh_title": "校准 Intel Quick Sync HEVC：28 与 30 之间的悬崖，和一个毫无作用的 preset", "category": "tutorials", "pub_date": "2026-10-14", - "status": "pending", + "status": "approved", "batch_id": "Batch-04", - "approved_at": null, - "notes": "", - "en_chars": 10490, - "zh_chars": 5631 + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", + "en_chars": 10968, + "zh_chars": 5968 }, "chainlink-vrf-v2-lottery-contract": { "slug": "chainlink-vrf-v2-lottery-contract", @@ -123,7 +123,7 @@ "category": "web3", "pub_date": "2024-12-10", "status": "pending", - "batch_id": "Batch-07", + "batch_id": "Batch-05", "approved_at": null, "notes": "", "en_chars": 15801, @@ -136,7 +136,7 @@ "category": "engineering", "pub_date": "2025-11-26", "status": "pending", - "batch_id": "Batch-03", + "batch_id": "Batch-02", "approved_at": null, "notes": "", "en_chars": 9584, @@ -161,12 +161,12 @@ "zh_title": "转成 HEVC 到底会不会让文件变小？我实测了", "category": "tutorials", "pub_date": "2026-10-08", - "status": "pending", + "status": "approved", "batch_id": "Batch-04", - "approved_at": null, - "notes": "", - "en_chars": 7417, - "zh_chars": 3695 + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", + "en_chars": 8042, + "zh_chars": 4159 }, "embedding-fonts-so-a-pdf-looks-the-same-everywhere": { "slug": "embedding-fonts-so-a-pdf-looks-the-same-everywhere", @@ -175,7 +175,7 @@ "category": "engineering", "pub_date": "2025-01-22", "status": "pending", - "batch_id": "Batch-03", + "batch_id": "Batch-02", "approved_at": null, "notes": "", "en_chars": 8033, @@ -213,10 +213,10 @@ "zh_title": "从一首歌到 4 分钟完整 MV：我如何用 Gemini、AI 生成与 Filmora 剪辑串起全流程实操", "category": "case-studies", "pub_date": "2026-10-25", - "status": "pending", + "status": "approved", "batch_id": "Batch-01", - "approved_at": null, - "notes": "", + "approved_at": "2026-10-27 05:26", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含GPU/Token成本算账、Gemini净化、双语CTA、代码折叠与简繁转换）", "en_chars": 11027, "zh_chars": 5168 }, @@ -227,7 +227,7 @@ "category": "web3", "pub_date": "2024-10-08", "status": "pending", - "batch_id": "Batch-07", + "batch_id": "Batch-05", "approved_at": null, "notes": "", "en_chars": 14598, @@ -240,7 +240,7 @@ "category": "tutorials", "pub_date": "2026-10-26", "status": "pending", - "batch_id": "Batch-06", + "batch_id": "Batch-04", "approved_at": null, "notes": "", "en_chars": 131667, @@ -253,7 +253,7 @@ "category": "tutorials", "pub_date": "2026-10-20", "status": "pending", - "batch_id": "Batch-07", + "batch_id": "Batch-05", "approved_at": null, "notes": "", "en_chars": 46399, @@ -291,12 +291,12 @@ "zh_title": "你好，世界 —— 关于这个博客", "category": "notes", "pub_date": "2024-09-24", - "status": "pending", + "status": "approved", "batch_id": "Batch-04", - "approved_at": null, - "notes": "", - "en_chars": 1794, - "zh_chars": 955 + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", + "en_chars": 1938, + "zh_chars": 1074 }, "how-i-built-an-automated-music-video-pipeline": { "slug": "how-i-built-an-automated-music-video-pipeline", @@ -304,10 +304,10 @@ "zh_title": "我是如何用 Remotion 与 Docker CDP 打造全自动 MV 生成与多语种油管发布流的", "category": "case-studies", "pub_date": "2025-04-18", - "status": "pending", + "status": "approved", "batch_id": "Batch-01", - "approved_at": null, - "notes": "", + "approved_at": "2026-10-27 05:26", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含GPU/Token成本算账、Gemini净化、双语CTA、代码折叠与简繁转换）", "en_chars": 9265, "zh_chars": 4931 }, @@ -317,10 +317,10 @@ "zh_title": "我是如何构建 DigiKedai Telegram AI 客服机器人的", "category": "case-studies", "pub_date": "2026-09-06", - "status": "pending", + "status": "approved", "batch_id": "Batch-01", - "approved_at": null, - "notes": "", + "approved_at": "2026-10-27 05:26", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含GPU/Token成本算账、Gemini净化、双语CTA、代码折叠与简繁转换）", "en_chars": 10719, "zh_chars": 5271 }, @@ -330,10 +330,10 @@ "zh_title": "我是如何托管这个博客的：Astro、Gitea Actions 与自托管 CI/CD", "category": "case-studies", "pub_date": "2026-09-04", - "status": "pending", + "status": "approved", "batch_id": "Batch-01", - "approved_at": null, - "notes": "", + "approved_at": "2026-10-27 05:26", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含GPU/Token成本算账、Gemini净化、双语CTA、代码折叠与简繁转换）", "en_chars": 5908, "zh_chars": 4145 }, @@ -343,10 +343,10 @@ "zh_title": "我用 Suno AI 做出自己的歌", "category": "case-studies", "pub_date": "2026-09-11", - "status": "pending", + "status": "approved", "batch_id": "Batch-01", - "approved_at": null, - "notes": "", + "approved_at": "2026-10-27 05:26", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含GPU/Token成本算账、Gemini净化、双语CTA、代码折叠与简繁转换）", "en_chars": 16935, "zh_chars": 11737 }, @@ -370,7 +370,7 @@ "category": "tutorials", "pub_date": "2025-11-26", "status": "pending", - "batch_id": "Batch-07", + "batch_id": "Batch-05", "approved_at": null, "notes": "", "en_chars": 6743, @@ -409,7 +409,7 @@ "category": "engineering", "pub_date": "2026-08-19", "status": "pending", - "batch_id": "Batch-03", + "batch_id": "Batch-02", "approved_at": null, "notes": "", "en_chars": 12217, @@ -435,7 +435,7 @@ "category": "engineering", "pub_date": "2026-09-27", "status": "pending", - "batch_id": "Batch-03", + "batch_id": "Batch-02", "approved_at": null, "notes": "", "en_chars": 14943, @@ -447,12 +447,12 @@ "zh_title": "我的备份爬得比爬虫还慢，因为我在用别人的 API 配额", "category": "notes", "pub_date": "2026-10-15", - "status": "pending", + "status": "approved", "batch_id": "Batch-04", - "approved_at": null, - "notes": "", - "en_chars": 8078, - "zh_chars": 4209 + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", + "en_chars": 8572, + "zh_chars": 4527 }, "n8n-v1-to-v2-upgrade-gotchas": { "slug": "n8n-v1-to-v2-upgrade-gotchas", @@ -473,12 +473,12 @@ "zh_title": "NocoDB SSO 是授权功能：环境变量不会告诉你的事", "category": "notes", "pub_date": "2026-09-29", - "status": "pending", + "status": "approved", "batch_id": "Batch-04", - "approved_at": null, - "notes": "", - "en_chars": 9934, - "zh_chars": 5418 + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", + "en_chars": 9937, + "zh_chars": 5421 }, "one-hostname-public-tracker-sso-dashboard": { "slug": "one-hostname-public-tracker-sso-dashboard", @@ -590,12 +590,12 @@ "zh_title": "与其看供应商文档，不如直接读容器自带的 API 文档", "category": "notes", "pub_date": "2025-10-01", - "status": "pending", + "status": "approved", "batch_id": "Batch-04", - "approved_at": null, - "notes": "", + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", "en_chars": 8382, - "zh_chars": 5483 + "zh_chars": 5489 }, "replacing-rdpguard-with-ipban": { "slug": "replacing-rdpguard-with-ipban", @@ -656,7 +656,7 @@ "category": "engineering", "pub_date": "2026-09-13", "status": "pending", - "batch_id": "Batch-03", + "batch_id": "Batch-02", "approved_at": null, "notes": "", "en_chars": 11245, @@ -746,12 +746,12 @@ "zh_title": "smartctl 退出码 32：专门跳过你最该看的硬盘的那个“错误”", "category": "notes", "pub_date": "2026-04-14", - "status": "pending", + "status": "approved", "batch_id": "Batch-04", - "approved_at": null, - "notes": "", + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", "en_chars": 5475, - "zh_chars": 2907 + "zh_chars": 2881 }, "space-reclamation-wasnt-stuck-it-was-waiting": { "slug": "space-reclamation-wasnt-stuck-it-was-waiting", @@ -759,12 +759,12 @@ "zh_title": "空间回收没卡住，它只是在等明天早上 09:29", "category": "devops", "pub_date": "2026-10-14", - "status": "pending", + "status": "approved", "batch_id": "Batch-02", - "approved_at": null, - "notes": "", - "en_chars": 5763, - "zh_chars": 3032 + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", + "en_chars": 6475, + "zh_chars": 3424 }, "stop-writing-agent-prompts-for-deterministic-work": { "slug": "stop-writing-agent-prompts-for-deterministic-work", @@ -772,10 +772,10 @@ "zh_title": "确定性的活，别再用 Agent Prompt 去跑", "category": "notes", "pub_date": "2026-10-01", - "status": "pending", - "batch_id": "Batch-04", - "approved_at": null, - "notes": "", + "status": "approved", + "batch_id": "Batch-03", + "approved_at": "2026-10-27 05:22", + "notes": "外部顾问审阅通过并落地修改", "en_chars": 4951, "zh_chars": 2216 }, @@ -798,12 +798,12 @@ "zh_title": "密码正确，Synology API 登录却返回 401，问题出在哪", "category": "notes", "pub_date": "2025-09-10", - "status": "pending", + "status": "approved", "batch_id": "Batch-04", - "approved_at": null, - "notes": "", + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", "en_chars": 5053, - "zh_chars": 3053 + "zh_chars": 3063 }, "synology-spreadsheet-api-is-a-container": { "slug": "synology-spreadsheet-api-is-a-container", @@ -811,12 +811,12 @@ "zh_title": "Synology 电子表格 API 是一个容器，不是一个 API 端点", "category": "devops", "pub_date": "2025-08-20", - "status": "pending", + "status": "approved", "batch_id": "Batch-02", - "approved_at": null, - "notes": "", - "en_chars": 15944, - "zh_chars": 9105 + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", + "en_chars": 15961, + "zh_chars": 9131 }, "that-dying-ssd-was-just-a-bad-sata-cable": { "slug": "that-dying-ssd-was-just-a-bad-sata-cable", @@ -824,12 +824,12 @@ "zh_title": "我差点把这颗 SSD 送修——真正坏的是 SATA 数据线", "category": "devops", "pub_date": "2026-09-16", - "status": "pending", + "status": "approved", "batch_id": "Batch-02", - "approved_at": null, - "notes": "", - "en_chars": 6484, - "zh_chars": 3426 + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", + "en_chars": 7128, + "zh_chars": 3785 }, "the-cause-was-trim-not-the-ssds": { "slug": "the-cause-was-trim-not-the-ssds", @@ -837,12 +837,12 @@ "zh_title": "换了不同的硬盘，同样的损坏又来了——真凶是 TRIM，不是 SSD", "category": "devops", "pub_date": "2026-09-18", - "status": "pending", + "status": "approved", "batch_id": "Batch-02", - "approved_at": null, - "notes": "", - "en_chars": 11452, - "zh_chars": 6481 + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", + "en_chars": 12427, + "zh_chars": 7072 }, "the-nocodb-attachment-that-wouldnt-update": { "slug": "the-nocodb-attachment-that-wouldnt-update", @@ -850,12 +850,12 @@ "zh_title": "NocoDB 附件字段怎么改都不生效，和一个漏网的正则", "category": "notes", "pub_date": "2026-02-04", - "status": "pending", + "status": "approved", "batch_id": "Batch-04", - "approved_at": null, - "notes": "", - "en_chars": 3570, - "zh_chars": 2156 + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", + "en_chars": 3798, + "zh_chars": 2312 }, "the-single-writer-rule-for-coding-agents": { "slug": "the-single-writer-rule-for-coding-agents", @@ -876,12 +876,12 @@ "zh_title": "上游被删了，然后带着一个重写版回来", "category": "devops", "pub_date": "2026-10-06", - "status": "pending", + "status": "approved", "batch_id": "Batch-02", - "approved_at": null, - "notes": "", - "en_chars": 11536, - "zh_chars": 5142 + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", + "en_chars": 11554, + "zh_chars": 5159 }, "two-ways-a-javascript-file-stops-working-and-looks-fine": { "slug": "two-ways-a-javascript-file-stops-working-and-looks-fine", @@ -890,7 +890,7 @@ "category": "engineering", "pub_date": "2025-02-20", "status": "pending", - "batch_id": "Batch-03", + "batch_id": "Batch-02", "approved_at": null, "notes": "", "en_chars": 7750, @@ -902,12 +902,12 @@ "zh_title": "Unraid 停止阵列时卡死的真相：Swapfile 锁住挂载点 — 附修复方案", "category": "devops", "pub_date": "2026-09-15", - "status": "pending", + "status": "approved", "batch_id": "Batch-02", - "approved_at": null, - "notes": "", - "en_chars": 10067, - "zh_chars": 6117 + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", + "en_chars": 10151, + "zh_chars": 6226 }, "upgrading-codeigniter-46-to-47": { "slug": "upgrading-codeigniter-46-to-47", @@ -915,10 +915,10 @@ "zh_title": "CodeIgniter 4.6 升级到 4.7：官方指南没说到的破坏性变更", "category": "notes", "pub_date": "2026-09-27", - "status": "pending", - "batch_id": "Batch-04", - "approved_at": null, - "notes": "", + "status": "approved", + "batch_id": "Batch-03", + "approved_at": "2026-10-27 05:22", + "notes": "外部顾问审阅通过并落地修改", "en_chars": 9469, "zh_chars": 5411 }, @@ -928,12 +928,12 @@ "zh_title": "用 CDP 自动化上传百兆视频到 YouTube：跨越 DevTools 内存墙的 Docker 归档法", "category": "devops", "pub_date": "2025-08-22", - "status": "pending", + "status": "approved", "batch_id": "Batch-02", - "approved_at": null, - "notes": "", - "en_chars": 10255, - "zh_chars": 6569 + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", + "en_chars": 11122, + "zh_chars": 7070 }, "using-chinese-llm-apis-from-malaysia": { "slug": "using-chinese-llm-apis-from-malaysia", @@ -955,7 +955,7 @@ "category": "engineering", "pub_date": "2026-09-28", "status": "pending", - "batch_id": "Batch-03", + "batch_id": "Batch-02", "approved_at": null, "notes": "", "en_chars": 12525, @@ -967,12 +967,12 @@ "zh_title": "如何在押注前验证一个开源依赖：我用的六项检查", "category": "devops", "pub_date": "2026-09-29", - "status": "pending", + "status": "approved", "batch_id": "Batch-02", - "approved_at": null, - "notes": "", - "en_chars": 8459, - "zh_chars": 4164 + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", + "en_chars": 8512, + "zh_chars": 4201 }, "when-smart-says-healthy-but-your-raid-is-corrupting-data": { "slug": "when-smart-says-healthy-but-your-raid-is-corrupting-data", @@ -980,12 +980,12 @@ "zh_title": "SMART 显示健康，RAID 却在悄悄损坏数据", "category": "devops", "pub_date": "2026-09-16", - "status": "pending", + "status": "approved", "batch_id": "Batch-02", - "approved_at": null, - "notes": "", - "en_chars": 12365, - "zh_chars": 5343 + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", + "en_chars": 13283, + "zh_chars": 5858 }, "when-to-give-your-agent-physical-clicks": { "slug": "when-to-give-your-agent-physical-clicks", @@ -1006,12 +1006,12 @@ "zh_title": "30 天提交 51.8 万行，生产环境只留下 23.5 万行", "category": "engineering", "pub_date": "2026-10-18", - "status": "pending", + "status": "approved", "batch_id": "Batch-04", - "approved_at": null, - "notes": "", - "en_chars": 8794, - "zh_chars": 4125 + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", + "en_chars": 8949, + "zh_chars": 4197 }, "why-chrome-forgets-its-tabs-in-a-container": { "slug": "why-chrome-forgets-its-tabs-in-a-container", @@ -1019,12 +1019,12 @@ "zh_title": "为什么容器里的 Chrome 会忘记标签页 —— 以及我是怎么修好的", "category": "devops", "pub_date": "2026-09-29", - "status": "pending", + "status": "approved", "batch_id": "Batch-02", - "approved_at": null, - "notes": "", - "en_chars": 14761, - "zh_chars": 8889 + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", + "en_chars": 14876, + "zh_chars": 8930 }, "why-i-built-a-video-knowledge-base": { "slug": "why-i-built-a-video-knowledge-base", @@ -1045,10 +1045,10 @@ "zh_title": "Claude 和 GPT 明明更强，我为什么还是买了 GPU 跑本地 AI", "category": "ai", "pub_date": "2026-09-21", - "status": "pending", + "status": "approved", "batch_id": "Batch-01", - "approved_at": null, - "notes": "", + "approved_at": "2026-10-27 05:26", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含GPU/Token成本算账、Gemini净化、双语CTA、代码折叠与简繁转换）", "en_chars": 11811, "zh_chars": 4924 }, @@ -1058,12 +1058,12 @@ "zh_title": "为什么本地语音合成一直在念 “Percent”？语音管线里的 URL 编码与沙盒避坑记", "category": "notes", "pub_date": "2026-10-22", - "status": "pending", + "status": "approved", "batch_id": "Batch-04", - "approved_at": null, - "notes": "", - "en_chars": 7263, - "zh_chars": 4652 + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", + "en_chars": 7728, + "zh_chars": 4968 }, "why-multi-agent-coding-fails": { "slug": "why-multi-agent-coding-fails", @@ -1071,12 +1071,12 @@ "zh_title": "为什么多个 AI 一起写代码总会烂尾？传话失真与隐式决策困境", "category": "notes", "pub_date": "2025-07-29", - "status": "pending", + "status": "approved", "batch_id": "Batch-04", - "approved_at": null, - "notes": "", - "en_chars": 5704, - "zh_chars": 2936 + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", + "en_chars": 6364, + "zh_chars": 3353 }, "why-my-on-chain-nft-art-changed-on-windows": { "slug": "why-my-on-chain-nft-art-changed-on-windows", @@ -1085,7 +1085,7 @@ "category": "web3", "pub_date": "2026-08-19", "status": "pending", - "batch_id": "Batch-07", + "batch_id": "Batch-05", "approved_at": null, "notes": "", "en_chars": 7725, @@ -1094,15 +1094,15 @@ "why-telegram-bot-notifications-die": { "slug": "why-telegram-bot-notifications-die", "title": "Why Telegram Bot Notifications Die: IPv6, DNS, and a 400 That Took a Day", - "zh_title": "Telegram 机器人通知为什么发不出去：IPv6、DNS 和一个折腾了一整天的 400", + "zh_title": "Telegram 机器人通知为什么发不出去：IPv6、DNS 和一个追查了一整天的 400", "category": "devops", "pub_date": "2026-03-25", - "status": "pending", + "status": "approved", "batch_id": "Batch-02", - "approved_at": null, - "notes": "", - "en_chars": 8963, - "zh_chars": 5185 + "approved_at": "2026-10-27 05:23", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", + "en_chars": 9327, + "zh_chars": 5334 }, "why-your-cdp-clicks-silently-fail": { "slug": "why-your-cdp-clicks-silently-fail", @@ -1111,7 +1111,7 @@ "category": "devops", "pub_date": "2025-03-19", "status": "pending", - "batch_id": "Batch-03", + "batch_id": "Batch-02", "approved_at": null, "notes": "", "en_chars": 8010, @@ -1124,7 +1124,7 @@ "category": "devops", "pub_date": "2026-05-17", "status": "pending", - "batch_id": "Batch-03", + "batch_id": "Batch-02", "approved_at": null, "notes": "", "en_chars": 7347, @@ -1149,12 +1149,12 @@ "zh_title": "为什么你的 Windows 桌面 MCP 调用会静默失败：攻破 10 秒空闲门禁与避坑实录", "category": "notes", "pub_date": "2026-03-12", - "status": "pending", + "status": "approved", "batch_id": "Batch-04", - "approved_at": null, - "notes": "", - "en_chars": 5024, - "zh_chars": 3435 + "approved_at": "2026-10-27 05:21", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理）", + "en_chars": 5663, + "zh_chars": 3857 }, "your-disk-full-alert-is-lying": { "slug": "your-disk-full-alert-is-lying", @@ -1163,7 +1163,7 @@ "category": "devops", "pub_date": "2026-06-24", "status": "pending", - "batch_id": "Batch-03", + "batch_id": "Batch-02", "approved_at": null, "notes": "", "en_chars": 5930, @@ -1175,10 +1175,10 @@ "zh_title": "为什么我的 AI 歌曲总是丢结尾：在 12GB 显卡上跑 YuE2", "category": "ai", "pub_date": "2026-10-20", - "status": "pending", + "status": "approved", "batch_id": "Batch-01", - "approved_at": null, - "notes": "", + "approved_at": "2026-10-27 05:26", + "notes": "外部顾问审阅完成，已采纳并落地全量修改意见（含GPU/Token成本算账、Gemini净化、双语CTA、代码折叠与简繁转换）", "en_chars": 28955, "zh_chars": 24414 } diff --git a/docs/consultant-review/batches/batch-01.md b/docs/consultant-review/batches/batch-01.md index 587282a..d76d8c5 100644 --- a/docs/consultant-review/batches/batch-01.md +++ b/docs/consultant-review/batches/batch-01.md @@ -4,7 +4,7 @@ > **包含文章数**：8 篇 > **本批次总字符量**：174,959 字（严格限制 < 200,000 字） > **模式**：BILINGUAL -> **生成时间**：2026-10-27 04:23:31 +> **生成时间**：2026-10-27 05:23:38 # 外部顾问审阅前置须知与工程背景规范 diff --git a/docs/consultant-review/batches/batch-02.md b/docs/consultant-review/batches/batch-02.md index 1ffce93..5126112 100644 --- a/docs/consultant-review/batches/batch-02.md +++ b/docs/consultant-review/batches/batch-02.md @@ -2,9 +2,9 @@ > **批次编号**：`Batch-02` > **包含文章数**：11 篇 -> **本批次总字符量**：179,502 字（严格限制 < 200,000 字） +> **本批次总字符量**：167,652 字（严格限制 < 200,000 字） > **模式**：BILINGUAL -> **生成时间**：2026-10-27 04:23:31 +> **生成时间**：2026-10-27 05:23:38 # 外部顾问审阅前置须知与工程背景规范 @@ -39,25 +39,25 @@ ## 本批次文章目录索引 -1. [devops] **空间回收没卡住，它只是在等明天早上 09:29** (`space-reclamation-wasnt-stuck-it-was-waiting`) — ~8,795 字符 -2. [devops] **Synology 电子表格 API 是一个容器，不是一个 API 端点** (`synology-spreadsheet-api-is-a-container`) — ~25,049 字符 -3. [devops] **我差点把这颗 SSD 送修——真正坏的是 SATA 数据线** (`that-dying-ssd-was-just-a-bad-sata-cable`) — ~9,910 字符 -4. [devops] **换了不同的硬盘，同样的损坏又来了——真凶是 TRIM，不是 SSD** (`the-cause-was-trim-not-the-ssds`) — ~17,933 字符 -5. [devops] **上游被删了，然后带着一个重写版回来** (`the-upstream-was-deleted-then-came-back-rewritten`) — ~16,678 字符 -6. [devops] **Unraid 停止阵列时卡死的真相：Swapfile 锁住挂载点 — 附修复方案** (`unraid-stop-array-hangs-on-swapfile`) — ~16,184 字符 -7. [devops] **用 CDP 自动化上传百兆视频到 YouTube：跨越 DevTools 内存墙的 Docker 归档法** (`uploading-large-videos-to-youtube-via-cdp`) — ~16,824 字符 -8. [devops] **如何在押注前验证一个开源依赖：我用的六项检查** (`vetting-an-open-source-dependency-before-you-bet-on-it`) — ~12,623 字符 -9. [devops] **SMART 显示健康，RAID 却在悄悄损坏数据** (`when-smart-says-healthy-but-your-raid-is-corrupting-data`) — ~17,708 字符 -10. [devops] **为什么容器里的 Chrome 会忘记标签页 —— 以及我是怎么修好的** (`why-chrome-forgets-its-tabs-in-a-container`) — ~23,650 字符 -11. [devops] **Telegram 机器人通知为什么发不出去：IPv6、DNS 和一个折腾了一整天的 400** (`why-telegram-bot-notifications-die`) — ~14,148 字符 +1. [devops] **为什么你的 CDP 点击会静默失效：平滑滚动、最小化窗口与 JS 派发事件** (`why-your-cdp-clicks-silently-fail`) — ~12,514 字符 +2. [devops] **Prometheus 明明有数据，Grafana 面板却全是 No data 的原因** (`why-your-grafana-dashboard-shows-no-data`) — ~11,212 字符 +3. [devops] **你的“磁盘将满”告警在说谎：别再按百分比告警** (`your-disk-full-alert-is-lying`) — ~8,912 字符 +4. [engineering] **浏览器翻译没有 API:我给一个纯中文网页应用加上了英文模式** (`adding-english-mode-to-a-chinese-only-web-app`) — ~18,070 字符 +5. [engineering] **排查 Remotion 视频抽搐：为什么在背景图挂载音频低频 Scale 会毁了转场** (`debugging-remotion-video-jitter`) — ~15,111 字符 +6. [engineering] **把字体嵌进 PDF，让它在任何机器上都长一样** (`embedding-fonts-so-a-pdf-looks-the-same-everywhere`) — ~11,465 字符 +7. [engineering] **那个把静默丢失更新变成 409 的 JPA 字段：@Version 乐观锁** (`jpa-version-field-lost-update`) — ~19,575 字符 +8. [engineering] **把 CodeIgniter 4 应用从 IIS 迁移到 OpenLiteSpeed** (`migrating-codeigniter-iis-to-openlitespeed`) — ~23,361 字符 +9. [engineering] **用「热浏览器会话」爬取有反爬墙的电商网站** (`scraping-bot-walled-marketplace-warm-browser-session`) — ~16,928 字符 +10. [engineering] **JavaScript 文件静默失效的两种方式：而页面看上去一切正常** (`two-ways-a-javascript-file-stops-working-and-looks-fine`) — ~11,294 字符 +11. [engineering] **逐页验证一份 19 页的 PDF 报告** (`verifying-a-pdf-report-page-by-page`) — ~19,210 字符 --- -## 文章 01 / 11：空间回收没卡住，它只是在等明天早上 09:29 -- **Slug**: `space-reclamation-wasnt-stuck-it-was-waiting` -- **英文标题**: Space Reclamation Wasn't Stuck. It Was Waiting for Tomorrow at 09:29. -- **分类**: `devops` | **发布日期**: 2026-10-14 -- **线上地址**: https://blog.hoelee.com/posts/space-reclamation-wasnt-stuck-it-was-waiting/ (EN) | https://blog.hoelee.com/posts/zh/space-reclamation-wasnt-stuck-it-was-waiting/ (ZH) +## 文章 01 / 11：为什么你的 CDP 点击会静默失效：平滑滚动、最小化窗口与 JS 派发事件 +- **Slug**: `why-your-cdp-clicks-silently-fail` +- **英文标题**: Why Your CDP Clicks Silently Fail: Smooth Scroll, Minimised Windows, and JS-Dispatched Events +- **分类**: `devops` | **发布日期**: 2025-03-19 +- **线上地址**: https://blog.hoelee.com/posts/why-your-cdp-clicks-silently-fail/ (EN) | https://blog.hoelee.com/posts/zh/why-your-cdp-clicks-silently-fail/ (ZH) - **本篇审核状态**: ⏳ 待审核 (Pending) ### ✍️ 顾问审阅意见区 (Review Feedback) @@ -71,808 +71,262 @@ ### 🇨🇳 中文版原文 (Chinese Markdown Twin) ````markdown --- -title: "空间回收没卡住，它只是在等明天早上 09:29" -description: "存储管理员那张卡整天写着 running，可用空间一动不动。其实 121 GiB 已经分两步回来了——然后下午又被写掉了 47 GiB。" -pubDate: 2026-10-14 -updatedDate: 2026-10-16 +title: "为什么你的 CDP 点击会静默失效：平滑滚动、最小化窗口与 JS 派发事件" +description: "Chrome DevTools Protocol 点击报告成功却毫无反应的三个原因：平滑滚动导致的过期坐标、最小化窗口被丢弃的坐标输入、以及页面收到了点击但拒绝了它。" +pubDate: 2025-03-19 +updatedDate: 2026-09-29 category: devops -tags: ["synology", "btrfs", "snapshots", "storage", "nas"] -ogImage: /og/zh-space-reclamation-wasnt-stuck-it-was-waiting.jpg -banner: /banners/space-reclamation-wasnt-stuck-it-was-waiting.jpg +tags: [cdp, chrome, react, browser-automation, debugging, python] +ogImage: /og/zh-why-your-cdp-clicks-silently-fail.jpg +banner: /banners/why-your-cdp-clicks-silently-fail.jpg draft: false --- -## 我问错了问题 +如果你用 Chrome DevTools Protocol 驱动网页应用，你一定见过这个失败：脚本报告 `clicked @591,361`，元素明明在，点击也确实发出去了 —— 然后什么都没发生。没有异常，没有控制台报错，页面毫无变化。 -我清掉几个文件夹，在 NAS 上点了空间回收，然后等。几个小时后，存储管理员那张卡还写着 **running**，可用空间看起来跟早上一样。我的问题是「它为什么什么都没回收」。 +我在自动化某平台卖家后台时整整耗了一个晚上在这上面。找到的三个原因值得写下来，因为**它们对「报告点击成功」的那段代码全都是隐形的**。 -**它回收过了——两次**，而且其中一次比我删掉的东西大得多。测量回答的是另一个问题，而这才是这件事里最有意思的地方。 +## 原因一：坐标过期，因为页面是平滑滚动的 -## 在相信任何东西之前，先给可用空间上定时采样 +最直观的点击辅助函数大概长这样： -我没有去看那张进度卡，而是给 `volume1` 的可用空间做了当天每五分钟一次的采样： - -| 时间 | 可用 | 事件 | -|---|---|---| -| 08:59 → 09:24 | 1134.1 GiB | 平——什么都没在回收 | -| **09:29** | **+69.0 GiB** | Class 快照 `…10.14-09.29.02` 轮转 | -| **09:34** | **+46.8 GiB** | Video 快照 `…10.14-09.30.35` 轮转 | -| **15:28:45** | — | DSM 日志：`Start to empty all recycle bins.` | -| **15:34** | **+5.0 GiB** | 清回收站结束——里面只有 5 GiB | -| 16:35 → 17:39 | **−39 GiB** | 新数据写入（一阵爆发） | -| 15:00 → 20:44 | **−47.2 GiB** | 净值：这个卷那天下午**丢**了空间 | - -两次回收，合计约 121 GiB；然后 47 GiB 的新写入把下午吃掉了。「什么都没回来」是我这边的读法错误：我拿早上和晚上比，而晚上里面多了一整天的写入。 - -我特意手动清空的回收站是 **5 GiB**；我压根没想到的快照是 **115.8 GiB**。这个比例就是全部故事。 - -## 为什么那张卡写着 running，却什么都不代表 - -```text -$ synostgreclaim --get-all-status --verbose -{"state":"running","num_subvolume_remain":1,"now_runnable":true} - schedule_table: 168 × 1 - run_time_option: run_all_time +```python +r = self.ev("""(()=>{const e=%s; e.scrollIntoView({block:'center'}); + const b=e.getBoundingClientRect(); + return JSON.stringify({x:b.left+b.width/2, y:b.top+b.height/2});})()""") +self.click_xy(json.loads(r)["x"], json.loads(r)["y"]) ``` -`state: running` 加上 168 个 1 的调度表和 `run_all_time`，意思是这个任务被配置成**按设计 24×7 都处于 running**。它是调度状态，不是进度指示器——所以在某天完全无事可排的日子，这张卡照样这么写。而当时那台机器上**没有** `synostgreclaim` 进程、没有 btrfs balance、md2/dm-3 的 I/O 基本为零。那一刻没有任何东西在被回收，也没有任何东西卡住。 +滚动到可视区、读取元素矩形、点它的中心。逻辑没错 —— 除非页面有 `scroll-behavior: smooth`（现代 UI 组件库里非常常见）。那时 `scrollIntoView()` 启动的是**动画**，而同一个 tick 里的 `getBoundingClientRect()` 返回的是元素**即将离开**的位置。你的点击落在元素**原来**的地方，落在当时在那儿的别的东西上，而日志忠实地告诉你：它点击了你要求的坐标。 -然后是日志，而 DSM 上它们大多是红鲱鱼： +修法是两行 —— 关掉平滑滚动，然后重新读矩形： -- 回收任务本身什么都不记； -- `/var/log/space_operation.log` 是 LUKS 和卷列表的噪音； -- `/var/log/synorecycle.log` 是 `synounlinkd` 的路径噪音； -- 当天的 Log Center 里只有我自己那条清回收站的记录，外加几条 UPS 断电/来电的提示。 - -一个永远为真的 UI 状态字段，加上三个永远不会给出答案的日志文件：唯一剩下的仪器，就是**随时间变化的可用空间**。 - -## 没人告诉你的那个机制 - -空间回收释放的是**已删除的 btrfs 子卷**。它释放不了仍然被快照钉住的数据。那两个共享文件夹的保留策略是 **retention 1**（各留一代），所以**我删掉的任何东西，都要等到第二天的轮转、大约 09:29 / 09:30 才会变成可用空间**，一次跳几十 GB，然后安静 24 小时。 - -那两次跳变正是这个。它也就意味着，对于「我删了 50 GB 却什么都没发生」这个问题，在这台机器上的诚实答案是：「等明天早上再看」——而如果删掉的数据本来就在某个快照里面，答案是「等那个快照过期，之前不会释放」。 - -## 如果重来一次 - -1. **采样那个数字，不要采样状态。** 每五分钟量一次可用空间，比任何任务卡都更能回答「现在有没有事情在发生」，而且它同时让「回长」变得可见。 -2. **读调度字段，不读状态字段。** `run_all_time` 解释了那个永久 running，而且不需要任何 bug 存在。 -3. **搞清楚每个共享文件夹的保留策略。** 它是决定「删除何时变成可用空间」的那口钟；不知道它，每一次删除都会看起来坏掉一整天。 -4. **把两本账分开。** 回收站（5 GiB）和快照（115.8 GiB）行为完全不同，也是两套心理模型。 -5. **重新检查分母。** 这个卷平时就在以约 1.5 GiB/时 的速度被正常写入吃掉；回收回来 121 GiB、当天又花掉 47 GiB，只要你只比同一个数字的两个快照，这两件事看起来完全一样。 - -## 测量真正告诉我的，是这台机器的趋势 - -我本来是去修一个卡住的任务，结果带回来的是一条存储趋势： - -```text -volume1 42 TB 已用 98% 剩 1.2 TB 近来约 -1.5 GiB/时（8 天 -344 GiB） -volume2 16 TB 已用 100% 剩 70 GB 约 -19 GiB / 2 天 -健康 两个卷 btrfs corruption_errs 均为 0 - 所有 md 阵列 [UUUU] / [UU]，没有在跑 balance - read/write_io_errs 与九月基线一致 +```python +self.ev("""(()=>{document.documentElement.style.scrollBehavior='auto'; + if(document.body) document.body.style.scrollBehavior='auto'; return 'ok';})()""") +moved = self.ev("""(()=>{const e=%s; if(!e) return 'no'; + e.scrollIntoView({block:'center', behavior:'instant'}); return 'ok';})()""" % sel) +time.sleep(0.35) # 等布局稳定 +# 然后再读矩形（元素晚一帧才渲染的话，外面再套一层重试） ``` -所以有用的结论不是「回收是好的」。而是：**相对于写入速率，回收只是一个舍入误差**；在保留策略为 1 的前提下，删除并不是我以为的那根杠杆；而该拿来做告警的是趋势，不是那张任务卡。这跟[《你的磁盘快满告警在撒谎》](/posts/your-disk-full-alert-is-lying/)是同一课，只是从另一个方向来的：对**后果**告警（随时间变化的可用字节），因为机制自己的状态不是证据。 -```` +两个细节很关键：`scrollIntoView` 上的 `behavior:'instant'` 会覆盖 CSS；而读取前那一小段等待，才是让坐标可信的原因。没有重试循环的话，这个辅助函数在晚一帧渲染的元素上仍会偶发失败 —— **偶发失败是最糟的一类，因为它会让你怀疑自己的判断。** -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Space Reclamation Wasn't Stuck. It Was Waiting for Tomorrow at 09:29." -description: "The Storage Manager card said 'running' all day and free space hadn't moved. 121 GiB had come back in two steps — and the afternoon ate 47 GiB of it." -pubDate: 2026-10-14 -updatedDate: 2026-10-16 -category: devops -tags: ["synology", "btrfs", "snapshots", "storage", "nas"] -ogImage: /og/space-reclamation-wasnt-stuck-it-was-waiting.jpg -banner: /banners/space-reclamation-wasnt-stuck-it-was-waiting.jpg -draft: false ---- +## 原因二：窗口最小化时，坐标点击被丢弃 -## The question I asked was the wrong question - -I emptied a few folders, kicked off Space Reclamation on the NAS, and -waited. Hours later the Storage Manager card still read **running**, and -free space looked the way it had looked in the morning. My question was "why -didn't it reclaim anything". - -It had reclaimed — twice — and one of those reclaims was larger than -anything I had deleted. The measurement answered a different question than -the one I asked, which is the interesting part. - -## Sample free space on a timer before believing anything - -I sampled `volume1` free space every five minutes for the day instead of -looking at a progress card: - -| time | free | event | -|---|---|---| -| 08:59 → 09:24 | 1134.1 GiB | flat — nothing reclaiming | -| **09:29** | **+69.0 GiB** | Class snapshot `…10.14-09.29.02` rotated | -| **09:34** | **+46.8 GiB** | Video snapshot `…10.14-09.30.35` rotated | -| **15:28:45** | — | DSM log: `Start to empty all recycle bins.` | -| **15:34** | **+5.0 GiB** | recycle-bin empty finished — only 5 GiB was in there | -| 16:35 → 17:39 | **−39 GiB** | new data written (a burst) | -| 15:00 → 20:44 | **−47.2 GiB** | net: the volume *lost* space that afternoon | - -Two reclaims, ~121 GiB in total, and then 47 GiB of new writes swallowed the -afternoon. "Nothing came back" was a reading error on my side: I compared -the morning to the evening, and the evening had a day of writing in it. - -The recycle bin — the thing I had bothered to empty by hand — was **5 GiB**. -The snapshots, which I had not thought about at all, were **115.8 GiB**. -That ratio is the whole story. - -## Why the card said "running" and meant nothing - -```text -$ synostgreclaim --get-all-status --verbose -{"state":"running","num_subvolume_remain":1,"now_runnable":true} - schedule_table: 168 × 1 - run_time_option: run_all_time -``` - -`state: running` with a schedule table of 168 ones and `run_all_time` means -the task is configured to be "running" **24×7, by design**. It is a -schedule, not a progress indicator — so the card will read the same thing on -a day when nothing is scheduled to happen at all. Meanwhile the machine had -no `synostgreclaim` process, no btrfs balance, and md2/dm-3 I/O of roughly -zero. Nothing was being reclaimed at that moment, and nothing was stuck -either. - -Then the logs, which are mostly red herrings on DSM: - -- the reclaim task itself logs nothing; -- `/var/log/space_operation.log` is LUKS and volume-list spam; -- `/var/log/synorecycle.log` is `synounlinkd` path noise; -- Log Center for the day contained only my own recycle-bin line and a few - UPS on-battery/back-online blips. - -A UI state field that is always true and three log files that never contain -the answer: the only instrument left is free space over time. - -## The mechanism nobody tells you about - -Space Reclamation frees **deleted btrfs subvolumes**. It cannot free data -that a snapshot is still pinning. Those two shares keep **retention 1** — -one generation each — so anything I delete becomes free space only at the -**next day's rotation, around 09:29 / 09:30**, in a step of tens of -gigabytes, and then it is silent for 24 hours. - -That is exactly what the two jumps were. It also means the honest answer to -"I deleted 50 GB and nothing happened" on this box is: "wait for tomorrow -morning, then check again" — and if the deleted data lived inside a -snapshot, the answer is "it frees when that snapshot expires, and not -before". - -## What I would do differently - -1. **Sample the number, not the status.** A five-minute timer on free space - answers "is anything happening" in a way no task card can, and it made - the regrowth visible at the same time. -2. **Read the schedule field, not the state field.** `run_all_time` explains - a permanent "running" without any bug existing. -3. **Know each share's retention.** It is the clock that decides when a - delete becomes free space; without it, every deletion looks broken for a - day. -4. **Separate the two ledgers.** Recycle bin (5 GiB) and snapshots (115.8 - GiB) behave nothing alike and are two different mental models. -5. **Re-check the denominator.** The volume was losing ~1.5 GiB/h to normal - writes; a reclaim that returns 121 GiB and a day that consumes 47 GiB - look identical if you only compare two snapshots of the same number. - -## What the measurement actually says about the machine - -I went in expecting to fix a stuck job and came out with a storage trend -instead: - -```text -volume1 42 TB 98% used 1.2 TB free ~-1.5 GiB/h lately (-344 GiB over 8 days) -volume2 16 TB 100% used 70 GB free ~-19 GiB / 2 days -health btrfs corruption_errs 0 on both volumes - all md arrays [UUUU] / [UU], no balance running - read/write_io_errs identical to the September baseline -``` - -So the useful conclusion is not "the reclaim works". It is that reclaim is a -rounding error next to the write rate, deletion is not the lever I thought -it was while retention is 1, and the trend — not the task card — is what I -should be alerting on. That is the same lesson as -[Your Disk-Full Alert Is Lying](/posts/your-disk-full-alert-is-lying/), -arriving from the other direction: alert on the consequence (bytes free over -time), because the mechanism's own status is not evidence. -```` - ---- - -## 文章 02 / 11：Synology 电子表格 API 是一个容器，不是一个 API 端点 -- **Slug**: `synology-spreadsheet-api-is-a-container` -- **英文标题**: The Synology Spreadsheet API Is a Container, Not an API Endpoint -- **分类**: `devops` | **发布日期**: 2025-08-20 -- **线上地址**: https://blog.hoelee.com/posts/synology-spreadsheet-api-is-a-container/ (EN) | https://blog.hoelee.com/posts/zh/synology-spreadsheet-api-is-a-container/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "Synology 电子表格 API 是一个容器，不是一个 API 端点" -description: "如何在 Synology Office 电子表格上做自动化：官方的 spreadsheet-api 容器，以及四个陷阱——一个假的\"未安装\"、凭据正确却报 401、永远无法通过认证的 2FA，还有一堵权限墙。" -pubDate: 2025-08-20 -updatedDate: 2026-09-26 -category: devops -blueprint: sme-digital-suite -blueprintLayer: L3-entitlement-backoffice -tags: [synology, dsm, docker, portainer, rest-api, spreadsheet, debugging, self-hosting] -ogImage: /og/zh-synology-spreadsheet-api-is-a-container.jpg -banner: /banners/synology-spreadsheet-api-is-a-container.jpg -draft: false ---- - -我想要的东西很普通：从我 NAS 上的电子表格读一个单元格，把一个单元格写回去，然后从另一端得到一张图表——不用打开 Excel，也不用把公司的数据送到某个云 API。 - -我跑着一台 Synology DS1821+，装好了 Synology Office（`Spreadsheet` 套件），所以这看起来是个已解决的问题。它确实是个已解决的问题。只是要穿过四个相互矛盾的假信号才能找到真正的答案，而且其中有一个我中途得出的结论错得相当彻底，值得写下来。 - -## 为什么这件事值得写 - -如果你自托管，总有一天你需要一个服务去和同一台机器上的另一个服务对话。这篇帖子里的故障模式并不只属于 Synology——它们就是"厂商把东西寄来了，但不在你会去找的地方"这个问题的通用形态： - -1. **API 清单里缺一条，不等于这个 API 不存在。** 我枚举了 1,515 个 API，得出结论说这个功能不存在。其实它是一个容器。 -2. **我信任的两个诊断工具在撒谎。** `synopkg` 把一个正在运行的套件报成"未开启"，`ps` 给我看了一台空荡荡、却正在服务生产流量的机器。 -3. **范围最精准的凭据失败，看起来却像是你的错。** 密码验证过正确还返回 `401`，几乎总是关于你把请求发到了*哪里*，而不是你发了*什么*。 -4. **权限模型在认证之后才咬人。** 你可以完美登录，却仍然对每个文件都收到 `403`，原因在于文件所在的位置。 - -如果以上任何一条听起来耳熟，这篇帖子的后半部分就是能跑通的完整配置。 - -## 文档说了什么，以及它为什么把我带偏 - -Synology 宣传"Office Suite APIs"——面向 Drive、Spreadsheet、MailPlus 和 Calendar 的 REST API。营销页面上逐字列出了我想要的正是这些： - -- "读取并写入某个范围内的单元格样式。" -- "新增、重命名、删除表格，并把工作表导出为 CSV 文件。" -- "创建、读取、导出和删除电子表格。执行批量更新……" - -文档本身藏在 Synology 账户登录后面——这是很常见的事，不是什么丑闻——但它意味着搜索引擎先找到的是承诺，而不是契约。 - -于是我自己在 NAS 上找。DSM 暴露了一个网关 API 目录，你可以直接问它存在什么： - -```bash -curl -s "http://192.168.1.1:8081/webapi/query.cgi?api=SYNO.API.Info&query=all" \ - | python -c "import json,sys; d=json.load(sys.stdin)['data']; print(len(d), 'APIs')" -``` +修好之后，我的点击辅助函数在一次性探针里能用，一放进真正的脚本就再也没成功过。同样的选择器、同样的坐标、同样的页面。差别最后落在可见性上： ``` -1515 APIs +document.visibilityState → "hidden" +document.hasFocus() → true ``` -一千五百一十五个。其中出现了 `SYNO.Office.*`——但只有快照和一个"最近使用的公式"列表： +当目标标签页不是可见窗口的活动标签页时，CDP 合成的坐标级鼠标事件是不可靠的 —— 浏览器可能接受、可能限流、也可能直接不往任何地方送。注意 `hasFocus()` **在文档隐藏时依然返回 `true`**，所以它不是有效的判断依据；要看的是 `document.hidden` / `visibilityState`。 -``` -SYNO.Office.Sheet.Snapshot -SYNO.Office.Sheet.Snapshot.History -SYNO.Office.Sheet.MruFc +两个修法，我都用： + +```python +def activate(self): # 把标签页提到窗口最前 + self.cdp("Page.bringToFront") + time.sleep(0.2) + +def click_js(self, sel): # 在页面内部派发事件来点击 + return self.ev("""(()=>{const e=%s; if(!e) return 'no'; + ['mousedown','mouseup','click'].forEach(k=> + e.dispatchEvent(new MouseEvent(k,{bubbles:true,cancelable:true,view:window}))); + return 'js-clicked';})()""" % sel) ``` -没有 values 端点。没有 styles 端点。没有写入端点。我把整个目录过滤了一遍，找任何提到 sheet 或 cell 的东西，结果一无所获，然后我得出结论——并且告诉了托我建这个东西的人——**这台 NAS 没有单元格级的电子表格 API。** +`Page.bringToFront` 解决了常见情况。但真正从不出错的是 JS 派发的点击：它不依赖窗口是否可见，而且 React 挂在根节点上的事件监听器会收到它，和收到真实点击的方式完全一样 —— 因为事件冒泡到的是同一个根。一个用坐标点击五次才能打开一次的日期选择器，用这种方式**每次都开**。 -那个结论是错的。我想把错在哪说清楚，因为那个推理错误才是可以复用的部分：**我把一个发现面当成了全部。** 网关目录列出的是 DSM 自己的 Web 网关服务的 API。Office Suite API 不由那个网关提供。它是一个独立服务，作为一个独立产物发布，因此对我跑的那次查询不可见。 +需要知道的取舍：派发事件会跳过 hit-testing，所以它也会「点到」用户根本够不到的元素（被遮住、滚出屏幕、透明度为零）。在这里这是优点 —— 如果你用它去掩盖布局 bug，那就是坑 —— 所以凡是可视化的东西我都用坐标点击，而遇到自定义控件时才伸手去拿 `click_js`。 -## 陷阱一：把正在运行的服务报成未安装的工具 +## 原因三：页面收到了点击，然后拒绝了它 -在找到真正的答案之前，我花了不少时间确信这个软件根本没在跑。两条命令让这个说法看起来很成立。 +这个最浪费时间，因为它的症状和前两个一模一样：点击发出去了，什么都没变。 -第一条是 Synology 自己的套件状态查询： - -```bash -sudo synopkg is_onoff SynologyDrive -sudo synopkg is_onoff Spreadsheet -``` - -``` -SynologyDrive isn't turned on, [262] -Spreadsheet isn't turned on, [262] -``` - -与此同时两个套件都明摆着在服务请求。`SynologyDrive` 自己的状态文件里是 `status=enabled`，Office 的守护进程也在进程表里——一个任务守护进程、一个跑在 `office` 用户账户下的连接池，还有网关 worker。在这套 DSM 版本上，`is_onoff` 根本不是一个可靠的答案。 - -第二条命令更糟，因为那是我自己的失误： - -```bash -ps w | grep -iE "office|drive|pgbouncer" -``` - -``` -(no output) -``` - -那看起来像一台死机。它不是。**在 DSM 上，不带 `sudo` 的 `ps` 只会显示你自己的进程。** 用 root 重跑一遍，画面就反转了： - -```bash -sudo ps aux | grep -E "office|pgbouncer|synoscgi" | grep -v grep -``` - -它们全都在。我用两条根本不可能让我看到真相的命令，"证明"了两次这套栈是挂的： - -> 如果要从这篇帖子里带走一个习惯：当一个诊断工具说"什么都没有在跑"时，先检查这个诊断工具到底有没有权限看到任何东西。 - -## 陷阱二：没有启动错误信息的容器 - -真正的答案，在我停止相信自己亲手得出的证据之后，出现在 Docker Hub 上：`synology/spreadsheet-api`。它不是 Office 套件的一部分，也不是网关端点。它自己的描述说得很清楚： - -> "Spreadsheet API 不是 Office 套件的一部分。这个镜像在客户端和 DSM 之间提供代理服务。每个 worker 专属于单个用户和单个电子表格。" - -兼容性表格把镜像标签对到 Office 版本上，这是第一件要查的事，因为配对不是可选的： - -| Docker 镜像标签 | 所需的 Synology Office 版本 | -|---|---| -| `3.4.1` | 3.7.0 或更高 | -| `3.3.2` | 更老的版本 | - -Synology 还警告不要把镜像和 Office 放在同一台机器上，因为每个 worker 会把整个电子表格加载进内存。我仍然把它跑在同一个 DSM 上，并加了内存上限——这是有意的取舍，不是推荐做法。 - -然后我跑了它，它立刻死掉，吐出一段这个： - -``` -/app/dist/index.js:424 -}`;var ot=UD(function(){return Ht($,qe+"return "+Ee).apply(e,H)});... -``` - -一段压缩过的 JavaScript——最没用的一类错误信息。真相一旦找到，其实很平常：**`AUTH_SECRET` 是必填项，而且没有默认值。** 它给会话令牌签名。没有它，服务就起不来，而且没法告诉你为什么。 - -要可靠地读出容器的真实要求，而不是听它的启动噪音，直接向打包产物要答案： - -```bash -docker run --rm --entrypoint grep synology/spreadsheet-api:3.4.1 \ - -ohE '.{0,80}process\.env\.PORT.{0,80}' /app/dist/index.js -``` +区分它们只需要注入一个监听器： ```js -serverPort: parseInt(process.env.PORT) || 3e3, -serverHost: process.env.HOST || "0.0.0.0", -workerTimeout: parseInt(process.env.WORKER_TIMEOUT) || 600*1e3, +window.__ev = []; +['mousedown','mouseup','click'].forEach(k => + document.addEventListener(k, e => window.__ev.push(k + ':' + e.target.tagName), true)); ``` -一行就给出配置契约：`AUTH_SECRET` 给令牌签名，`PORT` 默认 `3000`，`HOST` 默认 `0.0.0.0`，worker 十分钟后回收。它还自带一份 OpenAPI 规范和 Swagger UI，这个我后面会再说到。 +- 有事件记录 → 输入链路是通的，问题在应用侧（原因三），或者在你读取结果的方式上（见下）。 +- 一条都没有 → 是原因一或原因二。 -## 陷阱三：密码被证明正确，却收到 401 +挂上监听器后我点了一次 `Confirm`，缓冲区满了 —— 说明点击没问题，是应用在说「不」。理由在一段我从没想过要看的文字里： -容器跑起来之后，认证成了下一堵墙。这是请求，这是它返回的东西： +> “Please create a new shop welcome voucher after the existing one is expired.” -```bash -curl -s -X POST http://192.168.1.1:8791/spreadsheets/authorize \ - -H 'Content-Type: application/json' \ - -d '{"username":"","password":"","host":"192.168.1.1:5001","protocol":"https"}' -``` +表单拒绝创建同类型的第二张券。没有字段级红字、头 500 毫秒里没有提示条、URL 也没变。**拒绝是真的，点击是无辜的。** -```json -{"error":"Unauthorized"} -``` - -`401 Unauthorized`，而我用的正是刚刚登录过 DSM 的密码。最自然的解读——密码错了、账户错了——是一条死路，而我真走了进去：重新敲一遍密码，在 DSM 登录页试那个账户（能登录），把 HTTPS 换成 HTTP 端口（同样 401）。 - -真正的问题出在 `host` 字段。这个服务反过来问*你*该对着哪个 DSM 做认证，然后自己跟那个 host 做一次 TLS 握手。我给它的是一个裸 IP 地址，而用那个名字出示的证书和地址不匹配，于是握手失败——代理把一个失败的握手报成 `Unauthorized`，跟密码错误一模一样。 - -换成一个持有有效证书的主机名，立刻就修好了： - -```bash -curl -s -X POST http://192.168.1.1:8791/spreadsheets/authorize \ - -H 'Content-Type: application/json' \ - -d '{"username":"","password":"","host":"cloud.example.com","protocol":"https"}' -``` - -```json -{"token":"eyJhbGciOi...","host":"cloud.example.com"} -``` - -> **如果你的 Synology API 登录返回 401 而你又确定密码是对的，去查 `host`：它必须是一个容器能接受其证书的名字。一个裸的局域网 IP 不行。** - -它返回的令牌是 JWT，有效期 28 天，绑定一个 DSM 会话——所以 DSM 重启或账户被强制登出，它就会失效。集成中途突然冒出来的 `401`，当作"重新授权"来对待，而不是"凭据变了"。 - -## 陷阱四：2FA 永远无法认证 - -这个是如果你事先不知道就会浪费最多时间的一个，所以直说：**登录 schema 里没有一次性验证码字段。** - -```yaml -AuthorizationBody: - properties: - username: {type: string} - password: {type: string} - host: {type: string} - protocol: {type: string} -``` - -没有 OTP，没有应用密码，没有令牌交换。如果账户开了两步验证，这个 API 会永远返回 `401`——密码正确也一样，换哪个 host 都一样。唯一可行的安排是开一个关掉 2FA 的专用服务账户，权限只限定在它需要的文件夹——这本来就是更好的做法，也是我一开始就该做的，而不是先用我自己的管理员账户去试。 - -## 进得来之后：权限陷阱 - -认证不等于授权。等我的服务账户能登录了，针对我真正关心的那个电子表格的每次调用都返回： - -```json -{"statusCode":403,"code":"403","error":"Forbidden","message":"Permission denied"} -``` - -仔细读一下，它信息量很大：`403 Permission denied` 意味着**文件存在，但你无权拿到。** 一个错误的 ID 会返回别的东西： - -```json -{"statusCode":404,"code":"404","error":"Not Found","message":"Spreadsheet not found"} -``` - -两种不同的失败，两种不同的修法，把两者混为一谈就要白花一个小时。我的问题是位置问题：那个电子表格建在 Synology Drive 的个人 **My Drive** 里，在磁盘上是 `/volume1/homes//Drive/Document/…`。个人主目录别的 DSM 账户够不着——权限设置不行，共享文件夹的技巧也不行，因为它根本不在任何共享文件夹里。修法是把这个文件移进一个共享文件夹并在那里授予读写权限，或者用 Drive 把那个特定文件分享给服务账户并授予编辑权。 - -## 修复：一整套能跑通的配置 - -容器，配上一个真正的密钥和内存上限： - -```yaml -services: - spreadsheet-api: - image: synology/spreadsheet-api:3.4.1 - container_name: spreadsheet-api - restart: always - environment: - AUTH_SECRET: "" - PORT: "3000" - HOST: "0.0.0.0" - ports: - - "8791:3000" - mem_limit: 2g -``` - -一条只属于 DSM 的注意事项：DSM 上任何 compose 文件都别写 `cpus:`。它的内核没有 CPU CFS 调度器，这个键会被静默忽略——你得到的是有 CPU 限制的错觉，其实没有任何限制。 - -然后 API 就能用了，而且形态很讨喜。登录，创建或指定一个电子表格，读写范围： - -```bash -TOKEN=$(curl -s -X POST http://192.168.1.1:8791/spreadsheets/authorize \ - -H 'Content-Type: application/json' \ - -d '{"username":"","password":"","host":"cloud.example.com","protocol":"https"}' \ - | python -c "import json,sys; print(json.load(sys.stdin)['token'])") - -# read a range (sheet-qualified A1 notation) -curl -s -H "Authorization: Bearer ***" \ - "http://192.168.1.1:8791/spreadsheets//values/Sheet1!A1:C4" - -# write a range, formulas included -curl -s -X PUT -H "Authorization: Bearer ***" -H 'Content-Type: application/json' \ - -d '{"values": [["Item","Qty"],["Widget",12],["Gadget",7],["Total","=SUM(B2:B3)"]]}' \ - "http://192.168.1.1:8791/spreadsheets//values/Sheet1!A1:B4" -``` - -两个我没想到的细节，都是对我有利的： - -- **电子表格 ID 来自 Office 的 URL。** 一个位于 `…/oo/r/` 的电子表格——包括分享链接转到的也正是这个路径——把它那一段放进 API 调用就能寻址。 -- **公式是 Office 算的，不是你算的。** 把表格导出成 CSV 返回的是 `=SUM(B2:B3)` 和 `=SUMPRODUCT(B2:B3,C2:C3)` 的*求值后*结果——这些公式我从没在客户端算过。这正是用这个 API 而不是自己解析文件的全部理由：你继承了 Synology 自己的计算引擎。 - -有一个值得说出来的真实缺口：**没有 list 端点。** 规范声明了一个 "Statistics" 标签，却没有为它提供任何路由（`/statistics` → `404 Route not found`），也没有别的东西能枚举电子表格。你必须已经从文档的 URL 知道那个 ID。对你有意搭好的自动化来说没问题；如果你想浏览，那就没用。 - -## 从另一端得到一张图表 - -读单元格只是半件事；最初的要求里还有图表。同一个 API 能直接给你一份真正的 workbook： - -```bash -curl -s -H "Authorization: Bearer ***" \ - "http://192.168.1.1:8791/spreadsheets//xlsx" -o export.xlsx -``` - -那会返回一个合法的 `.xlsx`——`PK` 魔数，什么都能打开。接着我自己的服务（一个小的 FastAPI 容器，里面有 LibreOffice 负责公式求值，matplotlib 负责渲染）读取范围并生成 PNG 图表和仪表盘，整条管线端到端跑通，不需要 Excel，也不需要任何云依赖： - -``` -Synology Office → spreadsheet-api → .xlsx export → chart renderer → PNG -``` - -## 重来一次我会怎么做 - -1. **在断定某个 Synology 功能不存在之前，先去查 Docker Hub。** 我花了不少真实时间，从一个根本不可能包含答案的 API 清单去证明一个反面结论。`synology/*` 在 Docker Hub 上，一次搜索就到。 -2. **一开始就用一个关掉 2FA 的专用服务账户**，权限限定在单个共享文件夹。我先用管理员账户试，所以同一小时内同时撞上了 2FA 墙和 `403` 墙。 -3. **当一个诊断工具说"什么都没有在跑"，检查它的权限。** DSM 上不带 root 的 `ps` 不是进程列表，它是你自己的进程列表，它会高高兴兴地让你得出机器已死的结论。 -4. **读容器本身，别只读它的文档。** 厂商自己的运行命令在镜像页面上，但精确的默认值（`PORT` 3000，worker 超时 600 秒）和必填的 `AUTH_SECRET` 来自 grep 打包产物——一条命令，不用猜。 - -## 结果 - -在 NAS 上端到端验证：登录、创建、写单元格、读回单元格、CSV 导出和 `.xlsx` 导出全部返回 `200`，CSV 导出证明了 Office 引擎在服务端求值了 `=SUM`/`=SUMPRODUCT`。导出的 workbook 喂给图表服务，把同一份数据渲染成 PNG。四个陷阱记录在案，一个错误结论得到纠正，大约一天的活——现在我自己的硬件上有一份脚本可以驱动的电子表格。 - ---- - -*我在马来西亚为小企业做这类自托管集成——电子表格和 NAS 自动化、仪表盘，以及把你的数据留在你自己硬件上的内部工具。如果你需要这类东西，[WhatsApp 联系我](https://wa.me/60127972969) 或 [给我发邮件](mailto:me@hoelee.com?subject=Self-hosted%20spreadsheet%20automation)——也可以看看我在 [hoelee.com](https://hoelee.com) 还做些什么。* -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "The Synology Spreadsheet API Is a Container, Not an API Endpoint" -description: "How to automate Synology Office spreadsheets: the official spreadsheet-api container, and the four traps — a fake 'not installed', a 401 with correct credentials, 2FA that can never authenticate, and a permission wall." -pubDate: 2025-08-20 -updatedDate: 2026-09-26 -category: devops -blueprint: sme-digital-suite -blueprintLayer: L3-entitlement-backoffice -tags: [synology, dsm, docker, portainer, rest-api, spreadsheet, debugging, self-hosting] -ogImage: /og/synology-spreadsheet-api-is-a-container.jpg -banner: /banners/synology-spreadsheet-api-is-a-container.jpg -draft: false ---- - -I wanted something ordinary: read a cell from a spreadsheet on my NAS, write a cell back, and have a chart come out the other end — without opening Excel, and without shipping my company's numbers to a cloud API. - -I run a Synology DS1821+ with Synology Office (the `Spreadsheet` package) installed, so this looked like a solved problem. It is a solved problem. It just took four separate false signals to find the actual solution, and one of my conclusions along the way was flatly wrong in a way worth writing down. - -## Why this matters - -If you self-host, you will eventually need a service to talk to another service on the same box. The failure modes in this post are not Synology-specific — they are the general shape of "the vendor ships the thing, but not where you'd look for it": - -1. **A missing entry in an API list is not proof the API doesn't exist.** I enumerated 1,515 APIs and concluded the feature was absent. It was a container. -2. **Two of the diagnostics I trusted were lying.** `synopkg` reported a running package as "not turned on", and `ps` showed me an empty machine that was serving production traffic. -3. **The best-scoped credential failure looks like your fault.** A `401` with a verified-correct password is almost always about *where* you sent the request, not *what* you sent. -4. **The permission model bites after authentication.** You can log in perfectly and still be told `403` on every file, because of where the file lives. - -If any of that sounds familiar, the second half of this post is the working setup. - -## What the docs say, and why that sent me the wrong way - -Synology advertises "Office Suite APIs" — REST APIs for Drive, Spreadsheet, MailPlus and Calendar. The marketing page lists exactly what I wanted, verbatim: - -- "Read and write cell styles within a range." -- "Add, rename, delete, and export sheets as CSV files." -- "Create, retrieve, export, and delete spreadsheet. Perform batch updates…" - -The documentation itself sits behind a Synology Account login, which is a normal thing to hit and not a scandal — but it means the first thing a search engine finds is the promise, not the contract. - -So I went looking on my own NAS. DSM exposes a gateway API catalogue, and you can just ask it what exists: - -```bash -curl -s "http://192.168.1.1:8081/webapi/query.cgi?api=SYNO.API.Info&query=all" \ - | python -c "import json,sys; d=json.load(sys.stdin)['data']; print(len(d), 'APIs')" -``` - -``` -1515 APIs -``` - -Fifteen hundred and fifteen. Among them, `SYNO.Office.*` appears — but only for snapshots and a "recently used formulas" list: - -``` -SYNO.Office.Sheet.Snapshot -SYNO.Office.Sheet.Snapshot.History -SYNO.Office.Sheet.MruFc -``` - -No values endpoint. No styles endpoint. No write endpoint. I filtered the whole catalogue for anything mentioning sheets or cells and came up empty, and I concluded — and told the person I was building this for — that **the NAS had no cell-level spreadsheet API.** - -That conclusion was wrong. I want to be precise about why it was wrong, because the reasoning error is the reusable part: **I treated one discovery surface as exhaustive.** The gateway catalogue lists APIs served by DSM's own web gateway. The Office Suite API is not served by that gateway. It is a separate service, published as a separate artifact, and therefore invisible to the query I ran. - -## Trap 1: the tool that says a running service isn't installed - -Before finding the real answer, I spent a while convinced the software wasn't even running. Two commands made that look true. - -The first was Synology's own package-state query: - -```bash -sudo synopkg is_onoff SynologyDrive -sudo synopkg is_onoff Spreadsheet -``` - -``` -SynologyDrive isn't turned on, [262] -Spreadsheet isn't turned on, [262] -``` - -Meanwhile both packages were plainly serving requests. `SynologyDrive` had `status=enabled` in its own state file, and the Office daemons were in the process table — a task daemon, a connection pooler running under an `office` user account, and gateway workers. `is_onoff` was simply not a reliable answer on this DSM build. - -The second command was worse because it was my own mistake: - -```bash -ps w | grep -iE "office|drive|pgbouncer" -``` - -``` -(no output) -``` - -That looks like a dead system. It isn't. **On DSM, `ps` without `sudo` shows you only your own processes.** Re-run it as root and the picture inverts: - -```bash -sudo ps aux | grep -E "office|pgbouncer|synoscgi" | grep -v grep -``` - -There they all are. I had "proven" the stack was down twice, using two commands that couldn't have shown me otherwise: - -> If you take one habit from this post: when a diagnostic says "nothing is running", check whether the diagnostic had the privileges to see anything at all. - -## Trap 2: the container with no startup error message - -The actual answer, once I stopped believing my own evidence, was on Docker Hub: `synology/spreadsheet-api`. Not part of the Office package, not a gateway endpoint. Its own description is unambiguous: - -> "Spreadsheet API is not part of the Office package. This image provides a proxy service between clients and DSM. Each worker is dedicated to a single user and a single spreadsheet." - -The compatibility table matches image tags to Office versions, which is the first thing to check because the pairing is not optional: - -| Docker image tag | Required Synology Office | -|---|---| -| `3.4.1` | 3.7.0 or higher | -| `3.3.2` | older | - -Synology also warns against co-locating it with Office, since each worker loads whole spreadsheets into memory. I ran it on the same DSM anyway and capped it, which is a deliberate trade rather than a recommendation. - -Then I ran it and it died immediately with this: - -``` -/app/dist/index.js:424 -}`;var ot=UD(function(){return Ht($,qe+"return "+Ee).apply(e,H)});... -``` - -A minified JavaScript blob — the least useful class of error message there is. The cause, once found, was mundane: **`AUTH_SECRET` is required and has no default.** It signs the session tokens. Without it the service cannot start and cannot tell you why. - -The reliable way to read a container's actual requirements, rather than its startup noise, is to ask the bundle directly: - -```bash -docker run --rm --entrypoint grep synology/spreadsheet-api:3.4.1 \ - -ohE '.{0,80}process\.env\.PORT.{0,80}' /app/dist/index.js -``` +这一类 bug 还有两个工具： ```js -serverPort: parseInt(process.env.PORT) || 3e3, -serverHost: process.env.HOST || "0.0.0.0", -workerTimeout: parseInt(process.env.WORKER_TIMEOUT) || 600*1e3, +document.elementFromPoint(x, y) // 那个坐标上到底是什么？ +document.querySelectorAll('button').length // 你的选择器到底命中了几个？ ``` -That gives you the config contract in one line: `AUTH_SECRET` to sign tokens, `PORT` defaulting to `3000`, `HOST` to `0.0.0.0`, workers recycled after ten minutes. It also ships its own OpenAPI spec and a Swagger UI, which I'll come back to. +第二个抓到过一处很细的问题：门户页面在 DOM 里留了一个**隐藏的** `Confirm` 按钮（来自收起状态的日期面板），于是 `.pop()` 返回的是那个看不见的副本，点击自然落到了无人的地方。按可见性和尺寸一起过滤： -## Trap 3: a 401 while the password is provably correct - -With the container running, authentication became the next wall. Here is the request, and here is what it returned: - -```bash -curl -s -X POST http://192.168.1.1:8791/spreadsheets/authorize \ - -H 'Content-Type: application/json' \ - -d '{"username":"","password":"","host":"192.168.1.1:5001","protocol":"https"}' +```python +"[...document.querySelectorAll('button')].filter(b=>/^Confirm$/.test(b.innerText.trim())" +" && b.offsetParent!==null && b.getBoundingClientRect().width>0).pop()" ``` -```json -{"error":"Unauthorized"} -``` +## 第四种「失败」，只有回读才发现 -`401 Unauthorized` with a password I had just used to sign into DSM. The natural reading — wrong password, wrong account — is a dead end, and I went down it: retyped the password, tried the account on the DSM login page (it worked), tried the HTTP port instead of HTTPS (same 401). +还有一类「失败」其实不是失败：**写入成功了，而你的读取是旧的。** 我因为列表页没有立刻显示，就断定创建被静默拒绝，又点了一次；后来才发现那张券一直都存在 —— 列表只是延迟了大约十分钟。这个教训可以推广：一次写入之后，唯一可信的结论是在**等待足够时间之后**回读得到的，而「看不到变化」意味着**未知**，不是失败。 -The actual problem was the `host` field. The service asks *you* which DSM it should authenticate against, then does its own TLS handshake with that host. I had given it a bare IP address, and the certificate presented on that name doesn't match the address, so the handshake failed — and the proxy reports a failed handshake as `Unauthorized`, exactly like a bad password would. +## 检查清单 -Switching to a hostname with a valid certificate fixed it instantly: +1. 关掉平滑滚动；用 `behavior:'instant'` 滚动；等一会儿再读矩形。 +2. 点击前调用 `Page.bringToFront`；判断依据用 `document.hidden`，不是 `hasFocus()`。 +3. 自定义控件用 JS 派发 `mousedown`/`mouseup`/`click`，别点坐标。 +4. 先挂事件监听器，证明输入是否到达，再去怪应用。 +5. 数一数选择器命中几个 —— 组件库里隐藏的重复元素很常见。 +6. 写入之后先等待再回读；永远不要重复点击一个还没验证过的提交按钮。 -```bash -curl -s -X POST http://192.168.1.1:8791/spreadsheets/authorize \ - -H 'Content-Type: application/json' \ - -d '{"username":"","password":"","host":"cloud.example.com","protocol":"https"}' -``` +这些都不算冷门知识，但它们中的每一条，在一行 `clicked @591,361` 面前都是隐形的。如果你正在对着一个自己无法控制的后台做自动化：点击是整条链路里最不可靠的一环 —— 也是最容易被最后才怀疑的一环。 -```json -{"token":"eyJhbGciOi...","host":"cloud.example.com"} -``` - -> **If your Synology API login returns 401 and you're sure the password is right, check the `host`: it must be a name whose certificate the container accepts. A bare LAN IP is not that.** - -The token it returns is a JWT, valid for 28 days, tied to a DSM session — so it dies if DSM restarts or the account is forcibly logged out. Treat a sudden `401` mid-integration as "re-authorize", not "credentials changed". - -## Trap 4: 2FA cannot ever authenticate - -This is the one that would waste the most time if you didn't know it in advance, so here it is plainly: **the sign-in schema has no one-time-code field.** - -```yaml -AuthorizationBody: - properties: - username: {type: string} - password: {type: string} - host: {type: string} - protocol: {type: string} -``` - -No OTP, no app-password, no token exchange. If the account has two-factor authentication enabled, this API will return `401` forever, with the correct password, on every host. The only workable arrangement is a dedicated service account with 2FA off, scoped to the folders it needs — which is better practice anyway, and is what I should have started with instead of testing against my own admin account first. - -## The permissions trap, after you're in - -Authentication is not authorization. Once my service account could log in, every call against the spreadsheet I actually cared about returned: - -```json -{"statusCode":403,"code":"403","error":"Forbidden","message":"Permission denied"} -``` - -Read that carefully, because it is informative: `403 Permission denied` means **the file exists and you may not have it.** A bad ID returns something different: - -```json -{"statusCode":404,"code":"404","error":"Not Found","message":"Spreadsheet not found"} -``` - -Two different failures, two different fixes, and conflating them costs an hour. Mine was a location problem: the spreadsheet had been created in Synology Drive's personal **My Drive**, which on disk is `/volume1/homes//Drive/Document/…`. A personal home directory is not reachable by another DSM account — not by permission settings, not by shared-folder tricks, because it isn't in a shared folder at all. The fix is to move the file into a shared folder and grant Read/Write there, or to Drive-share the specific file to the service account with edit rights. - -## The fix: the whole working setup - -The container, configured with a real secret and a memory cap: - -```yaml -services: - spreadsheet-api: - image: synology/spreadsheet-api:3.4.1 - container_name: spreadsheet-api - restart: always - environment: - AUTH_SECRET: "" - PORT: "3000" - HOST: "0.0.0.0" - ports: - - "8791:3000" - mem_limit: 2g -``` - -One DSM-specific note: omit `cpus:` from any compose file on DSM. Its kernel has no CPU CFS scheduler, so the key is silently ignored — you get the illusion of a CPU limit and no limit. - -Then the API works, and the shape is pleasant. Sign in, create or address a spreadsheet, read and write ranges: - -```bash -TOKEN=$(curl -s -X POST http://192.168.1.1:8791/spreadsheets/authorize \ - -H 'Content-Type: application/json' \ - -d '{"username":"","password":"","host":"cloud.example.com","protocol":"https"}' \ - | python -c "import json,sys; print(json.load(sys.stdin)['token'])") - -# read a range (sheet-qualified A1 notation) -curl -s -H "Authorization: Bearer $TOKEN" \ - "http://192.168.1.1:8791/spreadsheets//values/Sheet1!A1:C4" - -# write a range, formulas included -curl -s -X PUT -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \ - -d '{"values": [["Item","Qty"],["Widget",12],["Gadget",7],["Total","=SUM(B2:B3)"]]}' \ - "http://192.168.1.1:8791/spreadsheets//values/Sheet1!A1:B4" -``` - -Two details I did not expect, both in my favour: - -- **The spreadsheet ID comes from the Office URL.** A spreadsheet at `…/oo/r/` — including one behind a share link, which redirects into exactly that path — is addressable by putting that segment in the API call. -- **Office computes the formulas, not you.** Exporting a sheet as CSV returned the *evaluated* results of `=SUM(B2:B3)` and `=SUMPRODUCT(B2:B3,C2:C3)` — formulas I never evaluated client-side. That's the whole reason to use this API instead of parsing a file yourself: you inherit Synology's own calculation engine. - -There is one real gap worth stating: **there is no list endpoint.** The spec declares a "Statistics" tag but serves no route for it (`/statistics` → `404 Route not found`), and nothing else enumerates spreadsheets. You must already know the ID from the document's URL. Fine for automation you set up on purpose; useless if you wanted to browse. - -## Getting a chart out the other end - -Reading cells was half the job; the original ask included charts. The same API can hand you a real workbook: - -```bash -curl -s -H "Authorization: Bearer $TOKEN" \ - "http://192.168.1.1:8791/spreadsheets//xlsx" -o export.xlsx -``` - -That returns a valid `.xlsx` — `PK` magic, opens in anything. From there my own service (a small FastAPI container with LibreOffice inside it for formula evaluation and matplotlib for rendering) reads the ranges and produces PNG charts and dashboards, so the pipeline runs end to end without Excel and without any cloud dependency: - -``` -Synology Office → spreadsheet-api → .xlsx export → chart renderer → PNG -``` - -## What I'd do differently - -1. **Check Docker Hub before concluding a Synology feature doesn't exist.** I burned real time proving a negative from an API list that could not have contained the answer. `synology/*` on Docker Hub was one search away. -2. **Start with a dedicated, non-2FA service account** scoped to one shared folder. I tested with an admin account first, which is how I met the 2FA wall and the `403` wall in the same hour. -3. **When a diagnostic says "nothing is running", check its privileges.** `ps` without root on DSM is not a process list, it's a list of your own processes, and it will happily let you conclude the machine is dead. -4. **Read the container, not just its docs.** The vendor's own run command is on the image page, but the exact defaults (`PORT` 3000, worker timeout 600 s) and the required `AUTH_SECRET` came from grepping the bundle — one command, no guessing. - -## The result - -Verified end to end on the NAS: sign-in, create, cell write, cell read-back, CSV export and `.xlsx` export all returned `200`, and the CSV export proved the Office engine evaluated `=SUM`/`=SUMPRODUCT` server-side. The exported workbook fed a chart service that rendered the same data to PNG. Four traps documented, one wrong conclusion corrected, roughly a day of work — and a spreadsheet on my own hardware that a script can now drive. +我平时就在做这类浏览器自动化与排障 —— 自托管服务、Docker 与 Traefik 栈，偶尔还有被 bot 墙挡住的市场页面。如果你有条工作流总因为「UI 有自己的想法」而断掉，跟我说说：[WhatsApp](https://wa.me/60127972969) · [me@hoelee.com](mailto:me@hoelee.com?subject=Browser%20automation%20debugging) · [hoelee.com](https://hoelee.com)。 +```` +### 🇬🇧 英文版原文 (English Markdown Primary) +````markdown +--- +title: "Why Your CDP Clicks Silently Fail: Smooth Scroll, Minimised Windows, and JS-Dispatched Events" +description: "Three reasons a Chrome DevTools Protocol click reports success and does nothing: a stale rect from smooth scrolling, a minimised window that drops coordinate input, and a page that received the click but rejected it." +pubDate: 2025-03-19 +updatedDate: 2026-09-29 +category: devops +tags: [cdp, chrome, react, browser-automation, debugging, python] +ogImage: /og/why-your-cdp-clicks-silently-fail.jpg +banner: /banners/why-your-cdp-clicks-silently-fail.jpg +draft: false --- -*I build self-hosted integrations like this for small businesses in Malaysia — spreadsheet and NAS automation, dashboards, and internal tools that keep your data on your own hardware. If that's the kind of thing you need, [WhatsApp me](https://wa.me/60127972969) or [email me](mailto:me@hoelee.com?subject=Self-hosted%20spreadsheet%20automation) — or see what else I do at [hoelee.com](https://hoelee.com).* +If you drive a web app with the Chrome DevTools Protocol, you have met this failure: your script reports `clicked @591,361`, the element is there, the click landed — and nothing happens. No exception, no console error, no change on the page. + +I spent an evening on exactly this while automating a marketplace's seller admin panel, and the three causes I found are worth writing down, because they are all invisible to the code that reports the click. + +## Cause 1: a stale rect, because the page scrolls smoothly + +The obvious click helper looks like this: + +```python +r = self.ev("""(()=>{const e=%s; e.scrollIntoView({block:'center'}); + const b=e.getBoundingClientRect(); + return JSON.stringify({x:b.left+b.width/2, y:b.top+b.height/2});})()""") +self.click_xy(json.loads(r)["x"], json.loads(r)["y"]) +``` + +Scroll into view, read the element's rectangle, click its centre. Correct — unless the page has `scroll-behavior: smooth` (very common in modern UI kits). Then `scrollIntoView()` starts an **animation**, and `getBoundingClientRect()` in the same tick returns the position the element is *about to leave*. Your click goes to where the element was, lands on whatever is there, and the log tells you it clicked the coordinates you asked for. + +The fix is two lines — turn smooth scrolling off, then re-read the rect: + +```python +self.ev("""(()=>{document.documentElement.style.scrollBehavior='auto'; + if(document.body) document.body.style.scrollBehavior='auto'; return 'ok';})()""") +moved = self.ev("""(()=>{const e=%s; if(!e) return 'no'; + e.scrollIntoView({block:'center', behavior:'instant'}); return 'ok';})()""" % sel) +time.sleep(0.35) # let layout settle +# then read the rect (and retry a couple of times if the element is not there yet) +``` + +Two details matter. `behavior:'instant'` on `scrollIntoView` overrides the CSS; and the small wait before reading is what makes the rect trustworthy. Without the retry loop the helper still fails intermittently on elements that render a frame late — intermittent failures are the worst kind, because they make you doubt your own diagnosis. + +## Cause 2: the window is minimised, and coordinate clicks are dropped + +After the fix, my click helper worked in a one-off probe and then never again inside the real script. Same selector, same coordinates, same page. The difference turned out to be visibility: + +``` +document.visibilityState → "hidden" +document.hasFocus() → true +``` + +CDP-synthesised mouse events at coordinates are unreliable when the target tab is not the active tab of a visible window — the browser may accept them, throttle them, or route them nowhere. Note that `hasFocus()` returns `true` even when the document is hidden, so it is not a useful check; `document.hidden`/`visibilityState` is. + +Two fixes, and I use both: + +```python +def activate(self): # bring the tab to the front of its window + self.cdp("Page.bringToFront") + time.sleep(0.2) + +def click_js(self, sel): # click by dispatching events from inside the page + return self.ev("""(()=>{const e=%s; if(!e) return 'no'; + ['mousedown','mouseup','click'].forEach(k=> + e.dispatchEvent(new MouseEvent(k,{bubbles:true,cancelable:true,view:window}))); + return 'js-clicked';})()""" % sel) +``` + +`Page.bringToFront` fixes the common case. But the JS-dispatched click is the one that never fails: it does not depend on the window being visible, and React's root-level event listener receives it exactly as it receives a real click, because the events bubble to the same root. A date-picker input that opened maybe one time in five with coordinate clicks opens **every** time this way. + +The trade-off to know: dispatching events skips hit-testing, so it will also “click” an element a user could not reach (covered, scrolled off, zero-opacity). That is a feature here — and a trap if you use it to paper over a layout bug — so I keep coordinate clicking for anything visual and reach for `click_js` when the widget is a custom control. + +## Cause 3: the page received the click and rejected it + +This is the one that wasted the most time, because the symptom is identical to causes 1 and 2: click fires, nothing changes. + +Distinguishing them takes one injected listener: + +```js +window.__ev = []; +['mousedown','mouseup','click'].forEach(k => + document.addEventListener(k, e => window.__ev.push(k + ':' + e.target.tagName), true)); +``` + +- Events recorded → the input path works. The problem is in the app (cause 3), or in how you read the result (see below). +- Nothing recorded → causes 1 or 2. + +With the listener armed I clicked `Confirm` and the buffer filled up — so the click was fine, and the app was saying no. The reason was in text I had not thought to look at: + +> “Please create a new shop welcome voucher after the existing one is expired.” + +The form refuses a second voucher of that type. No field-level red text, no toast in the first 500 ms, the URL unchanged. **The rejection was real and the click was innocent.** + +Two more tools for this class of bug: + +```js +document.elementFromPoint(x, y) // what is actually on top at that coordinate? +document.querySelectorAll('button').length // how many match your selector? +``` + +The second one caught a subtle one: the portal leaves a **hidden** `Confirm` button in the DOM from the collapsed date-panel, so `.pop()` returned the invisible duplicate and the click went nowhere visible. Filter by visibility *and* size: + +```python +"[...document.querySelectorAll('button')].filter(b=>/^Confirm$/.test(b.innerText.trim())" +" && b.offsetParent!==null && b.getBoundingClientRect().width>0).pop()" +``` + +## And the fourth failure I only found by re-reading + +There is a class of “failure” that is not one: **the write succeeded and your read is stale.** I concluded a create had been silently refused because the list page did not show it immediately, clicked again, and later discovered the voucher had existed the whole time — the list just lagged about ten minutes. The lesson generalises: after a write, the only trustworthy verdict is a read-back *after* a real waiting period, and “no visible change” means *unknown*, not *failed*. + +## The checklist + +1. Turn off smooth scrolling; scroll with `behavior:'instant'`; wait, then read the rect. +2. Call `Page.bringToFront` before clicking; treat `document.hidden` as the signal, not `hasFocus()`. +3. For custom widgets, dispatch `mousedown`/`mouseup`/`click` from JS rather than clicking coordinates. +4. Arm an event listener to prove whether input arrived before you blame the app. +5. Count your selectors — hidden duplicates are common in component libraries. +6. After a write, wait and re-read; never re-click a submit you have not verified. + +None of this is exotic, but every item is invisible in a log line that says `clicked @591,361`. If you are building automation against an admin panel you do not control, the click is the least reliable part of the pipeline — and the one most likely to be blamed last. + +I build and debug this kind of browser automation for a living — self-hosted services, Docker and Traefik stacks, and the odd scraped-but-bot-walled marketplace. If you have a workflow that keeps breaking because a UI has other ideas, tell me about it: [WhatsApp](https://wa.me/60127972969) · [me@hoelee.com](mailto:me@hoelee.com?subject=Browser%20automation%20debugging) · [hoelee.com](https://hoelee.com). ```` --- -## 文章 03 / 11：我差点把这颗 SSD 送修——真正坏的是 SATA 数据线 -- **Slug**: `that-dying-ssd-was-just-a-bad-sata-cable` -- **英文标题**: I Was About to RMA This SSD — the Fault Was a SATA Cable -- **分类**: `devops` | **发布日期**: 2026-09-16 -- **线上地址**: https://blog.hoelee.com/posts/that-dying-ssd-was-just-a-bad-sata-cable/ (EN) | https://blog.hoelee.com/posts/zh/that-dying-ssd-was-just-a-bad-sata-cable/ (ZH) +## 文章 02 / 11：Prometheus 明明有数据，Grafana 面板却全是 No data 的原因 +- **Slug**: `why-your-grafana-dashboard-shows-no-data` +- **英文标题**: Why Your Grafana Dashboard Shows No Data (When Prometheus Is Fine) +- **分类**: `devops` | **发布日期**: 2026-05-17 +- **线上地址**: https://blog.hoelee.com/posts/why-your-grafana-dashboard-shows-no-data/ (EN) | https://blog.hoelee.com/posts/zh/why-your-grafana-dashboard-shows-no-data/ (ZH) - **本篇审核状态**: ⏳ 待审核 (Pending) ### ✍️ 顾问审阅意见区 (Review Feedback) @@ -886,3023 +340,290 @@ Verified end to end on the NAS: sign-in, create, cell write, cell read-back, CSV ### 🇨🇳 中文版原文 (Chinese Markdown Twin) ````markdown --- -title: "我差点把这颗 SSD 送修——真正坏的是 SATA 数据线" -description: "一颗 Seagate IronWolf 110 SSD 疯狂报 WRITE FPDMA 超时、mkfs 报 'superblock magic doesn't match'、然后读写静默——教科书般的坏盘。我只换了一根 SATA 线，它就干净格式化、零报错。动手判死刑前，先这样排查。" -pubDate: 2026-09-16 +title: "Prometheus 明明有数据，Grafana 面板却全是 No data 的原因" +description: "所有 target 都是 up、数据就在 Prometheus 里、面板表达式也没错——41 个面板却全部 No data。原因是 Grafana 变量过滤了自己，而我的验证方式恰好掩盖了它。" +pubDate: 2026-05-17 +updatedDate: 2026-09-29 category: devops blueprint: self-hosted-infrastructure -blueprintLayer: L1-compute-storage-network -tags: [unraid, sata, ssd, storage, troubleshooting, smart, fio, cable] +blueprintLayer: L3-observability-self-healing +tags: [grafana, prometheus, dashboards, monitoring, promql, observability] +ogImage: /og/zh-why-your-grafana-dashboard-shows-no-data.jpg +banner: /banners/why-your-grafana-dashboard-shows-no-data.jpg draft: false -ogImage: /og/zh-that-dying-ssd-was-just-a-bad-sata-cable.jpg -banner: /banners/that-dying-ssd-was-just-a-bad-sata-cable.jpg --- -所有存储指南都告诉你：先看 SMART，再信事件日志。我也照做了，两者都指向同一个结论：这颗 SSD 要死了。而我距离把一颗健康硬盘送去 RMA 只差一步。 +打开每天在看的面板，结果是空的。不是\"某一格坏了\"，而是**从上到下每一格**都写着 *No data*。去查 Prometheus，一切健康。这篇讲我踩到的那个具体且不直观的原因，以及为什么我自己的验证会同时告诉我\"面板没问题\"。 -2026 年 9 月，一颗 Seagate IronWolf 110 960 GB SATA SSD——SMART 全绿、寿命约 99%、重映射扇区为零——反复在 btrfs `mkfs` 时失败，后来干脆连读都不回。内核日志里堆满了 `WRITE FPDMA QUEUED` 超时、`NCQ disabled due to excessive errors`、`lost async page write`。最后一次 `mkfs` 打印出 superblock 后，回读失败了。 +## 最省时间的排查顺序 -只有一件事救了这颗盘：在判它死刑之前，我换了根 SATA 数据线。 +常见原因（数据源选错、时间范围不对、`rate()` 用在会重置的计数器上、target 没被抓取）都不适用。真正有效的检查顺序是： -## 为什么这很重要 +```bash +# 1. target 真被抓取了吗？ +curl -s http://prometheus:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job, health}' -一根有问题的 SATA 口或线材，会产生**和坏盘一模一样的症状**，而且 SMART 全程保持漂亮。线材故障发生在链路层——传输出错超时，内存在你的文件系统或 SMART 看到任何错误字节之前，就把这次 I/O 丢了。只靠 SMART 判断，你会冤枉健康硬件、发起一趟无意义的 RMA，而换上的新盘在同一根线上照样坏。 +# 2. 指标现在存在吗？（完全绕过 Grafana） +curl -s -G http://prometheus:9090/api/v1/query \ + --data-urlencode 'query=node_uname_info{job="unraid-host"}' | jq '.data.result[0].metric' -## 那些"盘要死了"的症状 - -这是一颗独立的 Seagate IronWolf 110（ZA960NM10001，企业级 SATA SSD，约 894 GiB），我准备把它加进 Unraid 缓存池。三件事陆续发生，全都很像坏盘： - -1. 内核日志每约 30 秒就循环一轮 libata 错误处理： - ``` - ata7.00: exception Emask 0x4 (timeout), WRITE FPDMA QUEUED, status { DRDY } - ata7.00: exception ... action 0x6 frozen - attribute NCQ disabled due to excessive errors - Buffer I/O error on dev sdd1 ... lost async page write - ``` - 硬盘无法完成排队的写入，内核硬重置链路；降级之后它**丢掉了真实数据**——`/dev/sdd` 有 28 个 LBA 报 lost async page write。 - -2. `mkfs.btrfs -K -f /dev/sdd1` 打印出完整的文件系统头，然后校验失败： - ``` - Filesystem size: 894.25GiB - ERROR: superblock magic doesn't match - ``` - btrfs 写入了 superblock，回读时拿到的是**不一样的字节**。这是最可怕的静默写路径损坏。 - -3. 之后 `smartctl -H /dev/sdd` **超时**。这颗盘连健康查询都不回应了。结合上面的写损坏内核标志，"送 RMA"是合理的结论。 - -## 改变结论的那个细节 - -排查到一半我必须物理移动这颗盘——它被接到另一个 SATA 口、换了根线。我在动任何其他操作之前，重跑了**完全相同**的 `mkfs.btrfs` 命令。 - -它通过了。零报错。`btrfs device stats` 全部清零。一次加了 `verify=crc32c`（写入带校验和的数据并**回读核对**）的 8 GiB `fio` 写入以 err=0 干净跑完，回读一致。`smartctl` 超时再也没有出现。`dmesg` 自那次换线以来对这颗盘**零**报错。 - -除了链路，什么都没变。那些"静默写损坏"，是 SATA 传输在线上损坏了帧，不是闪存的问题。 - -## 能抓到它 vs 抓不到它的测试 - -这一趟得出两个教训： - -**普通的 `fio randrw` 对这个故障是瞎的。** 标准的 60 秒 `randrw`（70/30、4k、iodepth 32）写读随机扇区——但它从不*核对*读回来的字节是否和你写的一致。对一颗在线上损坏数据的盘，它会报 err=0、"干净通过"。我第一轮测试就是这样：*干净*，却在一颗随后 mkfs 失败的盘上。 - -**你需要 `verify`，而且要点对区域。** fio 的 `verify=crc32c` 写入校验和并回读核对，这正是能揪出回读不一致的开关。而且故障是**局部**的：坏掉的 LBA（约 234,422,526–869，距设备起始约 111.8 GiB）落在我第一轮测试跑过位置再往后 100 GiB 处。在错误的偏移量上测试完全测不到。证据就是：指向健康区域的 verify 任务通过了；同一任务指向已知坏区间时，硬盘像先前缓存池故障时那样直接卡住。 - -``` -# 判别性测试：写入带校验和，回读，不一致就失败 -# 要么跑全表面，要么跑 dmesg 标记的精确 LBA 区间 -fio --name=verify --filename=/dev/sdd \ - --offset=111G --size=2G \ - --rw=write --verify=crc32c --do_verify=1 \ - --bs=4k --ioengine=io_uring --direct=1 --iodepth=32 +# 3. 面板表达式本身对吗？ +curl -s -G http://prometheus:9090/api/v1/query \ + --data-urlencode 'query=count(node_cpu_seconds_total{mode="idle",job="unraid-host"})' ``` -同一任务跑在*健康*盘上，`verify` 会通过；链路坏的话，fio 会报校验和不符或卡死——远比普通的 randrw 更早暴露问题。 +三项都过了：7 个 target `up`、2.5 万条序列在采集、手动跑面板表达式也有数据。所以问题不在指标、不在抓取、也不在 PromQL——而在**它们之间的变量层**。 -## 我会改的做法 +## 真正的原因：变量过滤了自己 -- **先换线。** 在看到超时或写损坏时，在给盘判死刑之前，重新插拔/更换线材，或换到另一个口，重跑*相同*的失败命令。问题跟着**端口**走就是链路故障；跟着**盘**走（换了好口还坏）才是盘的问题。这是最便宜、最能一锤定音的测试，而我花了好几小时才做。 -- **别信一个干净的 `randrw`。** 它看不见静默损坏。凡是盘出现过任何写症状，都用 `verify=crc32c` 和 `do_verify=1`。 -- **对准日志里的 LBA 区间。** 内核或文件系统已经标记了具体扇区的话，就在*那里*测，而不是从偏移 0 开始。 -- **SMART 全绿 ≠ 链路健康。** UDMA_CRC 为零只说明*当前*链路干净，它看不到过去那段不稳定的链路；而传输超时的页面在算出 CRC 之前就已经被丢了。 +面板上有一个主机选择器，两个变量串联：`$nodename`（哪台机器）和 `$node`（它的 `instance` 标签）。第一个的定义是： + +```promql +# BROKEN — 变量用自身的值来过滤自己 +label_values(node_uname_info{job="unraid-host", nodename=~"$nodename"}, nodename) +``` + +首次加载时 `$nodename` 还没有值，选择器就变成 `nodename=~""`——**什么都匹配不到**。查询返回零行的变量没有任何选项，于是它保持空；而 `$node` 又是基于 `$nodename` 定义的： + +```promql +label_values(node_uname_info{job="unraid-host", nodename="$nodename"}, instance) +``` + +于是 `$node` 也是空，所有以 `$node` 为过滤条件的面板自然匹配不到序列。Grafana 说 *No data* 并没有骗人：面板查询真的没有数据，因为给它限定范围的变量解析成了空。 + +修法是去掉自引用，并给每个变量存一个明确的默认值： + +```promql +# FIXED — 不自引用，每个变量只跳一层 +# $nodename +label_values(node_uname_info{job="unraid-host"}, nodename) +# $node +label_values(node_uname_info{job="unraid-host", nodename="$nodename"}, instance) +``` + +## 为什么我的验证没抓到 + +这才是值得抄的部分。我\"验证\"面板的方式是：**手动**把每个面板表达式里的变量替换成我知道正确的值（`$nodename=unRaid`、`$node=host.docker.internal:9100`），然后断言查询有数据点。13 项检查全部通过——面板依然是空的，因为 bug 在被替换的那个值**上游**：Grafana 自己对变量的解析是空的。 + +**一个替换掉\"可疑值\"的验证，永远发现不了那个值解析失败。** 要抓它，得读 Grafana 真实存下来的值： + +```bash +curl -s -u admin:"$PW" http://grafana:4010/api/dashboards/uid/rYdddlPWk \ + | jq '.dashboard.templating.list[] | {name, current: .current.value}' +``` + +更直接的检查对象是变量查询本身——跑 Grafana 会跑的那条： + +```bash +# $nodename 的 label_values() 到底返回什么？ +curl -s -G http://prometheus:9090/api/v1/query \ + --data-urlencode 'query=count by (nodename) (node_uname_info{job="unraid-host"})' | jq '.data.result' +``` + +那里是空，面板就不可能渲染出来，无论数据多健康。 + +## 第二个坑：`$__all` 不是 `.*` + +做自动化面板检查时，值为\"All\"的变量，其 current 是哨兵字符串 `$__all`——不是正则。把 `name=~"$__all"` 原样展开会匹配不到任何东西，于是完全健康的容器面板\"全军覆没\"。展开前要把 `$__all` 解析成该变量的 `allValue`（通常是 `.*`），否则你追的是一个只存在于自己检查脚本里的 bug。 + +## 不是每个空面板都是 bug + +修完之后 25 个面板有 21 个出数。剩下 4 个是**设计上就不可能**有数据的，最好在面板说明里写清楚： + +- **PSI 面板**需要 `/proc/pressure`，而这台 NAS 的内核不暴露它——指标在这台机器上无法存在；同样的面板在会导出 `node_pressure_*` 的主机上正常显示。 +- **根文件系统面板**的表达式带 `fstype!="rootfs"`，而某台主机的 `/` 是**内存盘**（unRaid 就是），所以永远匹配不到。 + +区分\"因为 bug 空\"和\"因为这台机器不可能有这个指标而空\"，是真修好和瞎忙一下午的分界线。 + +## 我下次会怎么做 + +- **永远不要定义过滤自己的模板变量。** 一个变量只跳一层。 +- **给面板依赖的变量存明确的 `current` 值**，别让全新加载依赖下拉框被选中。 +- **用 Grafana 存下来的值做验证**，而不是我以为的值。 +- **故意留空的缺口写进面板说明**，否则未来的你会花一个下午去\"修\"一个本来就没打算工作的面板。 ## 结果 -这颗几分钟前就要被我送去 RMA 的盘，现在已经是一颗格式化好、挂载上、verify 全过的缓存 SSD，`btrfs device stats` 全部清零。修好它的成本：一根线。相信原有诊断的成本：一趟没必要的保修，还有一台迟早会在那根坏线上损坏数据的机器。 +三台主机现在跑同一套布局——每台一个 41 面板的主机面板 + 一个 10 面板的容器面板，分别有 21/23/25 个面板出数，两类结构性空缺写在面板里，而不是留给下一个打开它的人去猜。 -如果你在 NAS 或自管服务器上遇到一颗盘报 FPDMA 超时、但 SMART 看起来没问题——换线，别急着换盘。 +## 需要为你的业务做这个吗？ + +如果你有一堆没人信任的 Grafana 面板，或者服务器和 NAS 完全没有监控，我可以帮你搭自托管的 Prometheus + Grafana（主机与容器指标、合理的告警规则、通知推到 Telegram 和邮件），也会把你那些悄悄\"不出数\"的面板修好。 + +**WhatsApp：[+60 12-797 2969](https://wa.me/60127972969)** · **邮箱：[me@hoelee.com](mailto:me@hoelee.com?subject=Grafana%20monitoring)** · **[hoelee.com](https://hoelee.com)** + +网站设计与开发是我的主业；服务器加固与自托管基础设施是它的另一半。 ```` ### 🇬🇧 英文版原文 (English Markdown Primary) ````markdown --- -title: "I Was About to RMA This SSD — the Fault Was a SATA Cable" -description: "A Seagate IronWolf 110 SSD threw WRITE FPDMA timeouts, failed mkfs with 'superblock magic doesn't match', and went silent — a textbook dying drive. I swapped one SATA cable and it formatted clean with zero errors. How to test before condemning a drive." -pubDate: 2026-09-16 +title: "Why Your Grafana Dashboard Shows No Data (When Prometheus Is Fine)" +description: "Every target up, data in Prometheus, panel expressions correct — and all 41 panels reading No data. The culprit was a Grafana variable filtering on itself, plus the verification habit that hid it." +pubDate: 2026-05-17 +updatedDate: 2026-09-29 category: devops blueprint: self-hosted-infrastructure -blueprintLayer: L1-compute-storage-network -tags: [unraid, sata, ssd, storage, troubleshooting, smart, fio, cable] +blueprintLayer: L3-observability-self-healing +tags: [grafana, prometheus, dashboards, monitoring, promql, observability] +ogImage: /og/why-your-grafana-dashboard-shows-no-data.jpg +banner: /banners/why-your-grafana-dashboard-shows-no-data.jpg draft: false -ogImage: /og/that-dying-ssd-was-just-a-bad-sata-cable.jpg -banner: /banners/that-dying-ssd-was-just-a-bad-sata-cable.jpg --- -Every storage guide tells you to start with SMART and trust the event log. I did. Both pointed at one verdict: this SSD is dying. I was one config change away from RMA-ing a healthy drive. +You open the dashboard you use every day and it's empty. Not "one panel broke" empty — +**every panel**, top to bottom, says *No data*. You check Prometheus and everything is +healthy. This post is the specific, non-obvious cause I hit, and the way my own +verification convinced me the dashboard was fine while it wasn't. -In September 2026 a Seagate IronWolf 110 960 GB SATA SSD — SMART-clean, ~99% life, zero reallocated sectors — kept failing a btrfs `mkfs` and then stopped answering reads. The kernel log was full of `WRITE FPDMA QUEUED` timeouts, `NCQ disabled due to excessive errors`, and `lost async page write`. The final `mkfs` printed its superblock, then failed to read it back. +## The searchable question -Exactly one detail saved the drive: I swapped the SATA cable before I condemned it. +*My Grafana panels show No data but the data is in Prometheus — why?* -## Why this matters +The usual answers (wrong datasource, wrong time range, `rate()` on a counter that +resets, missing scrape target) did not apply. Here is the checklist that narrowed it +down, in the order that costs the least time: -A bad SATA port or cable produces **the same symptoms as a dead drive**, and it produces them while SMART stays immaculate. Cable faults happen at the link layer — the transport times out and the kernel drops the I/O before your filesystem or SMART ever sees a wrong byte. If you diagnose by SMART alone, you'll condemn healthy hardware, raise a pointless RMA, and the replacement will fail the same way on the same cable. +```bash +# 1. is the target actually being scraped? +curl -s http://prometheus:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job, health}' -## The symptoms that screamed "dead drive" +# 2. does the metric exist right now? (bypasses Grafana entirely) +curl -s -G http://prometheus:9090/api/v1/query \ + --data-urlencode 'query=node_uname_info{job="unraid-host"}' | jq '.data.result[0].metric' -This was a standalone Seagate IronWolf 110 (ZA960NM10001, an enterprise SATA SSD, ~894 GiB) I was preparing for an Unraid cache pool. Three things happened, all consistent with a failing SSD: - -1. The kernel log cycled through libata error handling every ~30 s: - ``` - ata7.00: exception Emask 0x4 (timeout), WRITE FPDMA QUEUED, status { DRDY } - ata7.00: exception ... action 0x6 frozen - attribute NCQ disabled due to excessive errors - Buffer I/O error on dev sdd1 ... lost async page write - ``` - The drive stopped completing queued writes, the kernel hard-reset the link, and once it degraded it **dropped real pages** — 28 `/dev/sdd` LBAs lost async page writes. - -2. `mkfs.btrfs -K -f /dev/sdd1` printed a full filesystem header, then failed verification: - ``` - Filesystem size: 894.25GiB - ERROR: superblock magic doesn't match - ``` - btrfs wrote its superblock, reread it, and got different bytes back. That's silent write-path corruption — the worst kind. - -3. After that, `smartctl -H /dev/sdd` **timed out**. The drive wasn't answering even a health poll. Combined with the write-corruption kernel flags, "RMA it" was the reasonable conclusion. - -## The detail that changed the verdict - -Mid-diagnosis I had to move the drive physically — I landed it on a different SATA port with a different cable. I reran the *identical* `mkfs.btrfs` command before doing anything else. - -It passed. Zero errors. `btrfs device stats` came back all-zero. An 8 GiB `fio` write with `verify=crc32c` (which writes patterned data and **rereads it to check bytes**) completed at err=0 with a clean read-back. The `smartctl` timeout never returned. `dmesg` shows **zero** error lines for the drive since that swap. - -Nothing changed except the link. The "silent write corruption" was the SATA transport corrupting frames on the wire, not the NAND. - -## The test that would have caught it (and the one that won't) - -Two lessons came out of this: - -**A plain `fio randrw` is blind to this fault.** A standard 60 s `randrw` test (70/30, 4k, iodepth 32) writes and reads random sectors — but it never *checks* that the bytes you read back match what you wrote. On a drive that corrupts on the wire, it reports err=0 and a clean pass. That's exactly what my first test did: *clean*, on a drive that then failed mkfs. - -**You need `verify`, and you need the right region.** fio's `verify=crc32c` writes checksums and rereads to confirm them — that's what trips read-back mismatches. And the fault was **localized**: the corrupt LBAs (~234,422,526–869, around 111.8 GiB into the device) sat 100 GiB past where my first test ran. A test at the wrong offset misses it entirely. The proof: a verify job pointed at the healthy region passed; the same job pointed at the known-bad range stalled the drive exactly like the pool failure had. - -``` -# The discriminating test: write w/ checksum, reread, fail on mismatch -# Run against the ENTIRE surface or the exact LBA range dmesg flagged -fio --name=verify --filename=/dev/sdd \ - --offset=111G --size=2G \ - --rw=write --verify=crc32c --do_verify=1 \ - --bs=4k --ioengine=io_uring --direct=1 --iodepth=32 +# 3. is the panel's expression sound? +curl -s -G http://prometheus:9090/api/v1/query \ + --data-urlencode 'query=count(node_cpu_seconds_total{mode="idle",job="unraid-host"})' ``` -If you run the same job on a *healthy* drive, `verify` succeeds. If the link is bad, fio reports checksum mismatches or hangs — long before a plain run would tell you anything. +All three were green: 7 targets `up`, 25k series ingested, and the panel's raw +expression returned data when I ran it by hand. So the problem was not the metrics, +the scrape, or the PromQL — it was **the variable layer between them**. + +## The actual cause: a variable that filtered on itself + +The dashboard had a host picker. Two variables chained: `$nodename` (which machine) +and `$node` (its `instance` label). The definition of the first one was: + +```promql +# BROKEN — the variable filters on its own value +label_values(node_uname_info{job="unraid-host", nodename=~"$nodename"}, nodename) +``` + +On first load `$nodename` has no value, so the selector becomes +`nodename=~""` — which matches **nothing**. A variable whose query returns no rows has +no options, so it stays empty. `$node` is then defined against `$nodename`: + +```promql +label_values(node_uname_info{job="unraid-host", nodename="$nodename"}, instance) +``` + +…which is also empty, so every panel filtering on `$node` matches no series. Grafana +isn't lying when it says *No data*: the panel query genuinely has no data, because the +variable that scopes it resolved to nothing. + +The fix is to stop the self-reference and give each variable an explicit saved default: + +```promql +# FIXED — no self-reference, one hop per variable +# $nodename +label_values(node_uname_info{job="unraid-host"}, nodename) +# $node +label_values(node_uname_info{job="unraid-host", nodename="$nodename"}, instance) +``` + +Then save `current` values for both so the dashboard opens populated instead of +depending on a click. (This is also why the same 41-panel layout can be cloned to +three hosts and just work.) + +## Why my verification missed it + +This is the part worth stealing. I "verified" the dashboard by expanding each panel's +expression **by hand**, substituting the variable values I knew were correct +(`$nodename=unRaid`, `$node=host.docker.internal:9100`), and asserting the query +returned points. Thirteen of those checks passed. The dashboard was still blank, +because the bug was upstream of the substitution: Grafana's *own* resolution of the +variables was empty. + +**A verification that substitutes the very value under suspicion cannot detect that +value failing to resolve.** To catch it, read the values Grafana actually saved and +expand with those: + +```bash +curl -s -u admin:"$PW" http://grafana:4010/api/dashboards/uid/rYdddlPWk \ + | jq '.dashboard.templating.list[] | {name, current: .current.value}' +``` + +The correct check is on the variable query itself — run what Grafana runs: + +```bash +# what does $nodename's label_values() actually return? +curl -s -G http://prometheus:9090/api/v1/query \ + --data-urlencode 'query=count by (nodename) (node_uname_info{job="unraid-host"})' | jq '.data.result' +``` + +Empty there means the dashboard cannot possibly render, no matter how healthy the data is. + +## The second trap: `$__all` is not `.*` + +While automating panel checks, a variable set to *All* reports its current value as the +sentinel string `$__all` — not a regex. Expanding panel expressions naively turns +`name=~"$__all"` into a literal that matches nothing, so a perfectly healthy +container dashboard "fails" every panel. Resolve `$__all` to the variable's `allValue` +(usually `.*`) before expanding, or you'll chase a bug that only exists in your checker. + +## Not every empty panel is a bug + +After the fix, 21 of 25 panels had data. The remaining four were empty **by design** and +worth naming honestly in the panel description: + +- **PSI panels** (`Pressure`, `Pressure Stall Information`) need `/proc/pressure`, which + the NAS kernel doesn't expose. The metric cannot exist there — the same panels on a + host that *does* export `node_pressure_*` render fine. +- **Root filesystem panels** whose expression carries `fstype!="rootfs"` can never match + on a host whose `/` is a **ramdisk**, which is what unRaid's root is. + +Distinguishing "broken because of a bug" from "empty because the metric cannot exist on +this host" is the difference between a real fix and a wild goose chase. ## What I'd do differently -- **Swap the cable first.** Before condemning a drive for timeout or write-corruption, reseat/replace the cable or move to a different port and rerun the *same* failing command. If the problem follows the **port**, it's a link fault; if it follows the **drive** across good ports, it's the drive. This is the single cheapest, most decisive test and I skipped it for hours. -- **Don't trust a clean `randrw`.** It can't see silent corruption. Use `verify=crc32c` and `do_verify=1` whenever a drive has shown any write symptom. -- **Target the logged LBA range.** If the kernel or filesystem already flagged specific sectors, test *there*, not at offset 0. -- **SMART-clean ≠ healthy link.** Zero UDMA_CRC just means the *current* link is clean; it can't see the past flaky one, and transport-timeout pages are dropped before a CRC is even computed. +- **Never define a template variable that filters on itself.** One hop per variable. +- **Always save explicit `current` values** for variables a dashboard depends on, so a + fresh load doesn't rely on a populated dropdown. +- **Verify with Grafana's saved values, not my idea of them.** The masking is invisible + in the test result and obvious in the UI. +- **Describe intentional gaps in the panel description.** Future-you will otherwise + spend an afternoon "fixing" a panel that was never meant to work there. ## The result -The drive I was minutes away from RMA-ing is now a formatted, mounted, verified-working cache SSD with `btrfs device stats` reading all zeros. The cost of the fix: one cable. The cost of trusting the diagnosis I had: a needless warranty claim and a rig kept on a cable that was going to corrupt data sooner or later anyway. - -If you're on a NAS or a home server and a drive "fails" with FPDMA timeouts but SMART looks fine — change the cable before you change the drive. -```` - ---- - -## 文章 04 / 11：换了不同的硬盘，同样的损坏又来了——真凶是 TRIM，不是 SSD -- **Slug**: `the-cause-was-trim-not-the-ssds` -- **英文标题**: The Corruption Came Back on Different Drives — the Cause Was TRIM, Not the SSDs -- **分类**: `devops` | **发布日期**: 2026-09-18 -- **线上地址**: https://blog.hoelee.com/posts/the-cause-was-trim-not-the-ssds/ (EN) | https://blog.hoelee.com/posts/zh/the-cause-was-trim-not-the-ssds/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "换了不同的硬盘，同样的损坏又来了——真凶是 TRIM，不是 SSD" -description: "两周前 btrfs RAID1 阵列在看似健康的 NVMe 盘上静默损坏数据，两周后同一个全零校验和出现在完全不同的硬盘上。真凶是 Seagate IronWolf 110 上的队列化 TRIM（NCQ）——FreeBSD 早就拉黑了这个固件缺陷，Linux 却一直没有。我是怎么抓到它、以及修复方法的全过程。" -pubDate: 2026-09-18 -category: devops -tags: [unraid, btrfs, trim, ssd, raid, data-loss, smart, troubleshooting, seagate] -ogImage: /og/zh-the-cause-was-trim-not-the-ssds.jpg -banner: /banners/the-cause-was-trim-not-the-ssds.jpg ---- - -两周前我写了一篇文章，讲一个 btrfs RAID1 阵列如何在两块看起来完全健康的三星 PM9A3 NVMe 盘上静默损坏 Windows 虚拟机——读出来的全是**全零**数据块，单靠一块坏盘根本无法解释。那次调查以一句诚实的「不确定」收尾：两块盘测试都干净，最大嫌疑是它们共用的 M.2 转接卡。 - -然后它又发生了。在**不同的硬盘**上，走**不同的总线**。校验和指纹一模一样。这一次我找到了根因——而且跟 SSD 本身无关。 - -## 为什么这很重要 - -如果你在 Unraid（或任何 Linux 机器）上跑 btrfs RAID1 + SATA SSD + 自动 TRIM，这篇是给你的提醒：**队列化 TRIM 正在同时、在同一位置、静默地摧毁我阵列的两份镜像。** RAID 冗余在这种损坏面前毫无保护力。而全程被 SMART 标记为「failing」的那块盘，跟真正的损坏一点关系都没有。 - ---- - -## 环境 - -- **unRaid 7.x**，Intel i9-13900K。 -- **`ssd` 池**：两块 **Seagate IronWolf 110 960 GB** SATA SSD - （`ZA960NM10001`，序列号 `HKR02TFK` + `HKR02L9P`）组 **btrfs RAID1**， - 挂载在 `/mnt/ssd`，上面跑着 Docker 数据根目录（`/var/lib/docker`）和 - Windows 11 虚拟机的虚拟磁盘（`domains/Win11Enterprise/vdisk1.img`）。 -- 虚拟机虚拟磁盘**没有**开 `NOCOW`，所以 btrfs 每次读取都会校验——和上次 - 事故能被发现靠的是同一个设置。 - -顺带一提：其中一块盘（`sdd`）正是 *[《我当时差点就去 RMA 这块 SSD——其实是 SATA 线坏了》](/posts/zh/that-dying-ssd-was-just-a-bad-sata-cable/)* -里那块盘——换线之后格式化干净，就进了这个池。 - -## 第一步——一个「假」的 SMART 故障 - -凌晨 02:53，Unraid 的通知铃响了： - -``` -Unraid Ssd disk SMART health [1] — Warning [UNRAID] - -raw read error rate (failing now) is 19665 (sdd = ZA960NM10001_HKR02TFK) -``` - -「failing now」是 SMART 里最强的措辞——盘自报该属性**跌破阈值**。但在这些 -Seagate 盘上，raw 值是障眼法：IronWolf 110 的属性 1 raw 字段是厂商编码的 -复合值（高半部分是错误数，低半部分是操作计数）。`19665 = 0x00004CD1` → -高半部分 = **0 个错误**。真正触发的是*归一化*值：`093`，**最差值 088**， -低于 `090` 阈值，于是盘在 SMART 返回状态里自报 `FAILING_NOW`/`IN_THE_PAST`。 -另一块盘（`sdb`）形状相同，raw 只有 `341`。 - -所有真实的故障计数器都是零：重映射 0、坏块增长 0、编程/擦除失败 0、 -不可纠正 ECC 0、UDMA CRC 0、寿命剩余 99 %、SMART 总体 **PASSED**，刚跑完的 -扩展自检也干净。这是 IronWolf 110 有据可查的行为——多年前 Synology 用户就 -在 100 % 寿命时撞上过同样的「Failing」旗标，社区共识是：只要真实计数器不动， -这个型号的属性 1 触发就当固件噪音处理。 - -所以：盘是误报，但这次报警让我去看了——然后才找到真正的损坏。 - -## 第二步——一个不该存在的校验和 - -`btrfs device stats /mnt/ssd`： - -``` -[/dev/sdd1].corruption_errs 27 [/dev/sdb1].corruption_errs 31 -``` - -全部集中在同一个文件：**ino 261 = Windows 虚拟机的虚拟磁盘**。 -`dmesg` 里是上一个 NVMe 事故里我盯了一周的同一个签名： - -``` -btrfs: checksum verify failed on logical 1505629372416 mirror 1 wanted 0x8941f998 -``` - -**`0x8941f998` 是 4 KiB 全零块的 CRC32C。** 池在读出空白块——而且关键的，是 -在**两块盘的相同逻辑偏移上同时读出**。我翻了 PM9A3 阵列那次的笔记：值相同， -「确定性全零」行为相同，让 RAID1 失效的关联方式也相同。 - -这个匹配是指纹级的。无论这是什么，它都不是硬盘的问题，也不是 M.2 转接卡的 -问题——IronWolf 这对盘走的是 **SATA 口**，完全不同的控制器路径。问题出在 -**比硬盘高一层**：有一个机制决定同时擦掉两块盘上相同的数据。Linux 里恰好 -只有一个机制干这事：**discard / TRIM**。 - -## 第三步——根因：在 Linux 从未拉黑的硬盘上跑队列化 TRIM - -Unraid 的池配置开了自动 TRIM，btrfs 以 `discard=async` 挂载——内核批量收集 -空闲块发给硬盘。对 SATA SSD 来说，那就是**队列化 TRIM**（`DATA SET -MANAGEMENT` 命令以 NCQ 命令形式延迟执行，即 `SEND FPDMA QUEUED`）。 - -IronWolf 110 的队列化 TRIM **不稳定的问题早就被记录了**。FreeBSD 内核有一个 -文档化的报告——[bug 264139](https://bugs.freebsd.org/264139)， -*「ata: NCQ_DSM_TRIM trim method for Seagate IronWolf 110 SATA SSD hangs drives」*， -2024 年合并的修复提交写得毫不客气：[commit `a6cef617660a`](https://lists.freebsd.org/archives/freebsd-fs/2024-March/003273.html)： -「Seagate IronWolf 110 SATA SSD 已被报告在启用 NCQ trim 时不稳定。」 - -而 Linux 自己的怪癖清单（`drivers/ata/libata-core.c`）——内核会在这里主动把 -有问题的盘降级为非队列化或禁用 TRIM——收录了 Micron M500/M550/1100、 -Crucial M500/M550/MX100、三星 840/850/860/870 等，但**没有一条 Seagate -条目**。我自己机器上的 `dmesg` 恰好证明这个机制存在而且在选择性生效： - -``` -ata6.00: Model 'Samsung SSD 840 PRO Series', rev 'DXM06B0Q', applying quirks: noncqtrim zeroaftertrim -``` - -这条怪癖给了机箱里*另一块* SSD，而两块 IronWolf 110 什么怪癖都没有， -拿到了完整的队列化 TRIM。 - -为什么这会毁掉 RAID1？因为 btrfs RAID1 在两块成员盘上存的是*同一份*空闲 -映射——会在同一轮里对两块盘丢弃相同的逻辑区间。如果这个固件上的 TRIM 路径 -会多失效（或失效错）LBA，就会**同时清掉两块镜像上的同一批活动块**。你的 -镜像变成同一个洞的两份拷贝：两块盘都校验失败，自愈没有任何可抄的源，数据 -彻底没了。 - -> 诚实的保留意见：FreeBSD 记录的是盘侧的*不稳定/卡死*，而「队列化 TRIM 抹掉 -> 了活动数据」是我根据这个记录加指纹（双镜像全零、SMART 干净、CRC 错误计数 -> 不涨）推断的机制。它符合每一条观察。证明靠实验——见下面的结果。 - -## 第四步——修复（持久化，一次只改一个变量） - -只改一处，双管齐下——持久配置 + 热挂载： - -```bash -# /boot/config/pools/ssd.cfg （备份：/root/ssd.cfg.bak.20260918-0329） -diskAutotrim="off" # 原来是 "on" - -# 不停阵列、不动 Docker（36 个容器全程在线）直接生效 -mount -o remount,nodiscard /mnt/ssd -findmnt /mnt/ssd # → rw,noatime,ssd,space_cache=v2 （不再有 discard=async） -``` - -然后做一次完整读写 scrub，搞清楚到底坏了多少： - -``` -scrub started 03:29:53, finished 03:53:07 (23 min) -csum_errors: 18 corrected_errors: 16 uncorrectable_errors: 2 -read_errors: 0 verify_errors: 0 super_errors: 0 -``` - -`read/verify/super = 0` 是指纹的另一半：硬件读盘*完全正常*——没有任何部件在 -故障，只是盘上存的字节和校验和不符。18 个坏块里，**16 块只有一块镜像坏， -已从完好副本自愈；2 个 uncorrectable 其实是同一个 4 KiB 块两块镜像都坏**—— -永久丢失，位于虚拟机虚拟磁盘约 45.2 GiB 处（`logical 1505629372416`，文件 -偏移 `48560156672`）。我把这一个块填零，至少虚拟机读到那里不再报错；原始 -字节无论如何都找不回来了。 - -还有一个值得知道的坑：**scrub 跑了 18 个错误，`btrfs device stats` 计数器 -却纹丝不动**——scrub 发现的错误和读取时发现的错误是分开统计的。你没法只盯 -`device stats` 一个数；dmesg 和 scrub 输出也得一起盯。（我的看门狗三样都查， -细节见文末。） - -## 换成我会怎么做 - -1. **任何 btrfs 池开自动 TRIM 之前，先查这块盘的队列化 TRIM 记录。** - FreeBSD 维护一份「别给这盘发 TRIM」清单不是没有原因的。Linux 在 - `libata-core.c` 里有同样的机制——花一个下午 grep + 首次开机后看一眼 - `dmesg`，好过一次从备份里恢复虚拟机。 -2. **TRIM 可以有，但不能队列化。** SATA SSD 想跑 fstrim 就强制非队列化 - （`libata.force=…:noncqtrim`），并在第一次 trim 后做一次 scrub 验证。 - `discard=sync` **没用**——只要盘支持，sync 同样走队列化路径。 -3. **三个损坏信号一起盯**，别只看一个：`btrfs device stats`、 - `dmesg | grep 'csum failed'`、周期性 scrub。三者各有各的计数器（见上文）。 -4. **并入池之前先交叉验证。** 同批次、同固件的盘是关联故障风险；两块盘共一个 - 固件 bug 就是最坏的那种关联故障。镜像两次都没救得了我。 -5. **买企业级 SATA SSD 时**：Linux 怪癖清单上的盘（Micron M500/M550/1100、 - Crucial M500/M550/MX100、三星 840/850/860/870）以及一切有记录在案的 - NCQ-TRIM bug（IronWolf 110）直接排除。三星 PM893/PM897、Micron 5400 - PRO/MAX、Solidigm D3-S4610/S4620、WD Ultrastar DC SA620/SA630、Kingston - DC600M 都有断电保护（PLP）且不在 Linux 怪癖清单上；而且 Micron、Solidigm、 - WD、Kingston 会公开发布固件更新说明，真出了问题还能给你推送修复。NVMe - 从根上绕开了 SATA 队列化 TRIM 这一类问题——我那两块 PM9A3 也不在 NVMe - 怪癖表里。 - -## 结果 - -只改了一个变量（关自动 TRIM），其他原封不动。紧接着： - -- 第二次 scrub（04:10–04:39）：**`corrected_errors: 0`** ——16 块自愈块保持 - 住了，整盘重读没有出现任何新错误。 -- `corruption_errs` 维稳在 sdd1=27 / sdb1=31，持续数小时的 Docker + 无虚拟机 - 运行（此前计数器是每天在涨的）。 -- dmesg 里没有新增 `csum failed` 行。 - -确认窗口还有几天：Win11 虚拟机重新上线（它产生的是能复现问题的写入模式）、 -对照基线文件每日检查、2026-09-22 再做一次复核 scrub。如果关了 TRIM 损坏还在， -下一个嫌疑就不是硬盘，而是内存（这台机器没有 ECC）——用 memtest 测。如果 -一直平静，那 IronWolf 110 队列化 TRIM 固件 bug 就算坐实了——我敢说论坛里 -「健康硬盘毁掉我的 RAID」的帖子，相当一部分也是它干的。 - -一个「正在故障」的 SMART 告警、一块零错误的硬盘、一个全新阵列上双镜像同时 -损坏——每个迹象都指向别处，而真正的凶手是一个我们大多数人不假思索就开启的 -存储功能。 - ---- - -*我的存储故障系列之一：[NVMe 事故](/posts/when-smart-says-healthy-but-your-raid-is-corrupting-data/) 和 [SATA 线警醒](/posts/that-dying-ssd-was-just-a-bad-sata-cable/)。* -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "The Corruption Came Back on Different Drives — the Cause Was TRIM, Not the SSDs" -description: "Two weeks after a btrfs RAID1 pool corrupted on healthy-looking NVMe drives, the same all-zero checksum reappeared on a different drive stack. The culprit was queued TRIM (NCQ) on Seagate IronWolf 110 SSDs — a firmware bug FreeBSD blacklisted but Linux never did. How I caught it and the fix." -pubDate: 2026-09-18 -category: devops -tags: [unraid, btrfs, trim, ssd, raid, data-loss, smart, troubleshooting, seagate] -ogImage: /og/the-cause-was-trim-not-the-ssds.jpg -banner: /banners/the-cause-was-trim-not-the-ssds.jpg ---- - -Two weeks ago I published how a btrfs RAID1 pool on two healthy-looking Samsung -PM9A3 NVMe drives silently corrupted a Windows VM — reads returning -**all-zero** blocks that no single bad drive could explain. That investigation -ended with an honest shrug: the drives tested clean, the strongest suspect was -the shared M.2 riser. - -Then it happened again. On *different* drives, on a *different* bus. The exact -same checksum fingerprint. This time I found the root cause — and it wasn't -the SSDs. - -## Why this matters - -If you run Unraid (or any Linux box) with a btrfs RAID1 pool on SATA SSDs and -auto-TRIM enabled, this post is a heads-up: **queued TRIM was silently -destroying both mirrors of my pool, in the same places, simultaneously.** RAID -redundancy gives you zero protection against it. And SMART — which flagged a -drive as *failing* through all of this — had nothing to do with the actual -corruption. - ---- - -## The setup - -- **unRaid 7.x** on an Intel i9-13900K. -- **Pool `ssd`**: two **Seagate IronWolf 110 960 GB** SATA SSDs - (`ZA960NM10001`, serials `HKR02TFK` + `HKR02L9P`) in **btrfs RAID1**, - mounted at `/mnt/ssd`, holding Docker's data root (`/var/lib/docker`) and - the Windows 11 VM's virtual disk (`domains/Win11Enterprise/vdisk1.img`). -- The VM's vdisk is **not** `NOCOW`, so btrfs validates every read against a - checksum — the same setting that caught the previous incident. - -Trivia: one of these drives (`sdd`) is the very drive from *["I Was About to -RMA This SSD — the Fault Was a SATA Cable"](/posts/that-dying-ssd-was-just-a-bad-sata-cable/)* -— after the cable swap it formatted clean and joined this pool. - -## Step 1 — a SMART "failure" that wasn't - -At 02:53 the Unraid notification bell went off: - -``` -Unraid Ssd disk SMART health [1] — Warning [UNRAID] - -raw read error rate (failing now) is 19665 (sdd = ZA960NM10001_HKR02TFK) -``` - -"Failing now" is the drive declaring the attribute **worse than its -threshold** — the strongest wording SMART has. But the raw value is a red -herring on these Seagates: on IronWolf 110s, attribute 1's raw field is a -vendor-encoded composite (errors in the high half, an operation counter in the -low half). `19665 = 0x00004CD1` → high half = **0 errors**. The *normalized* -value is what tripped: `093` with a **worst of 088**, below the `090` -threshold, so the drive self-reports `FAILING_NOW` / `IN_THE_PAST` on its -SMART return status. The twin (`sdb`) showed the same shape with a raw of -only `341`. - -Every real failure counter was zero: reallocated 0, grown bad block 0, -program/erase fails 0, uncorrectable ECC 0, UDMA CRC 0, SSD life left 99 %, -SMART overall **PASSED**, and a just-completed extended self-test was clean. -This is a documented IronWolf 110 behaviour — the same "Failing" flag at 100 % -lifespan hit Synology users years ago, and it's why the community treats that -model's attribute-1 trip as firmware noise unless the real counters move. - -So: false alarm on the *drive*, but the alarm made me go look — and that's -when I found the real damage. - -## Step 2 — the checksum that had no right to exist - -`btrfs device stats /mnt/ssd`: - -``` -[/dev/sdd1].corruption_errs 27 [/dev/sdb1].corruption_errs 31 -``` - -All of it confined to one file: **ino 261 = the Windows VM's vdisk**. -`dmesg` showed the same signature I'd spent a week staring at in the NVMe -incident: - -``` -btrfs: checksum verify failed on logical 1505629372416 mirror 1 wanted 0x8941f998 -``` - -**`0x8941f998` is the CRC32C of 4 KiB of zeros.** The pool was reading back -empty blocks — and crucially, on **both mirrors at the same logical offsets**. -I verified it against my notes from the PM9A3 pool: identical value, identical -"deterministic zeros" behaviour, identical RAID1-defeating correlation. - -That match is a fingerprint. Whatever this is, it isn't the drives and it -isn't the M.2 riser — the IronWolf pair lives on **SATA ports**, a -completely different controller path. It's the *layer above* the drives -deciding to erase the same data on both members at once. There is exactly one -mechanism in Linux that does that: **discard / TRIM**. - -## Step 3 — the root cause: queued TRIM on a drive Linux never blacklisted - -Unraid's pool config had auto-TRIM on, which mounts btrfs with -`discard=async` — the kernel batches frees and sends them to the drive. On -SATA SSDs that means **queued TRIM** (the `DATA SET MANAGEMENT` command -deferred as an NCQ command, `SEND FPDMA QUEUED`). - -The IronWolf 110 is *known to be unstable with queued TRIM*. FreeBSD's -kernel has a documented report — [bug 264139](https://bugs.freebsd.org/264139), -*"ata: NCQ_DSM_TRIM trim method for Seagate IronWolf 110 SATA SSD hangs -drives"* — and the fix, merged in 2024, is blunt: -[commit `a6cef617660a`](https://lists.freebsd.org/archives/freebsd-fs/2024-March/003273.html): -"The Seagate IronWolf 110 SATA SSD drive has been reported to be unstable with -NCQ trim enabled." - -Meanwhile Linux's own quirk list (`drivers/ata/libata-core.c`) — where the -kernel deliberately downgrades broken drives to non-queued or disabled TRIM — -contains Micron M500/M550/1100, Crucial M500/M550/MX100, Samsung 840/850/860/870 -and a few others, but **no Seagate entry at all**. My own `dmesg` proved the -mechanism exists and is selective: - -``` -ata6.00: Model 'Samsung SSD 840 PRO Series', rev 'DXM06B0Q', applying quirks: noncqtrim zeroaftertrim -``` - -…applied to the *other* SSD in the box, while the two IronWolf 110s got -no quirk and full queued TRIM. - -Why does that corrupt a RAID1? Because btrfs RAID1 stores the *same* free-space -map on both members — it discards the same logical extents on both drives in -the same pass. A TRIM path that invalidates more (or the wrong) LBAs on this -firmware zeroes the **same live blocks on both mirrors at once**. Your mirror -becomes two copies of the same hole: the checksum fails on both, self-heal has -nothing to copy from, and the data is gone. - -> The honest caveat: FreeBSD documents *instability/hangs* from the drive -> side, and "queued TRIM erased live data" is the mechanism I infer from that -> plus the fingerprint (zeros on both mirrors, SMART-clean, CRC error counter -> untouched). It fits every observation. The proof is the test — see the -> result below. - -## Step 4 — the fix (persistent, one variable at a time) - -One change, applied twice — persistent config and live mount: - -```bash -# /boot/config/pools/ssd.cfg (backup: /root/ssd.cfg.bak.20260918-0329) -diskAutotrim="off" # was "on" - -# apply live without stopping the array or Docker (36 containers stayed up) -mount -o remount,nodiscard /mnt/ssd -findmnt /mnt/ssd # → rw,noatime,ssd,space_cache=v2 (no more discard=async) -``` - -Then a full read-write scrub to find out what was actually damaged: - -``` -scrub started 03:29:53, finished 03:53:07 (23 min) -csum_errors: 18 corrected_errors: 16 uncorrectable_errors: 2 -read_errors: 0 verify_errors: 0 super_errors: 0 -``` - -`read/verify/super = 0` is the second half of the fingerprint: the hardware -reads *fine* — nothing is failing, the stored bytes just don't match their -checksums. Of the 18 bad blocks, **16 were bad on one mirror and were healed -from the clean copy. Two uncorrectable entries were one single 4 KiB block -that was bad on both mirrors** — permanent loss, ~45.2 GiB into the VM's -vdisk (`logical 1505629372416`, file offset `48560156672`). I zero-filled -that one block so the VM's reads at least don't error; the original bytes are -unrecoverable either way. - -And a wrinkle worth knowing: **`btrfs device stats` counters did not move -during the scrub** even though it found 18 errors — scrub-detected errors and -read-detected errors are tracked separately. You cannot watch only `device -stats`; you have to watch dmesg + scrub output too. (My watchdog checks all -three; the details are at the end.) - -## What I'd do differently - -1. **Never enable auto-TRIM on a btrfs pool without checking the drive's - queued-TRIM record.** FreeBSD keeps a "don't TRIM this drive" list for a - reason. Linux has the same mechanism in `libata-core.c` — an afternoon of - grep + a `dmesg` check after the first boot beats a restored-from-backup VM. -2. **Keep TRIM on the menu, just not queued.** If you want fstrim on SATA - SSDs, force non-queued TRIM (`libata.force=…:noncqtrim`) and verify with a - scrub after the first trim pass. `discard=sync` does **not** help — it - still uses the queued path on drives that advertise it. -3. **Watch all three corruption signals**, not one: `btrfs device stats`, - `dmesg | grep 'csum failed'`, and periodic scrubs. Each can move - independently (see above). -4. **Corroborate before joining pairs.** Same-batch, same-firmware drives are - a correlated-failure risk; a pair sharing one firmware bug is the - correlated-failure worst case. The mirror didn't save me either time. -5. **When buying enterprise SATA SSDs**: the drives on Linux's quirk list - (Micron M500/M550/1100, Crucial M500/M550/MX100, Samsung 840/850/860/870) - and anything with a documented NCQ-TRIM bug (IronWolf 110) are out. - Samsung PM893/PM897, Micron 5400 PRO/MAX, Solidigm D3-S4610/S4620, - WD Ultrastar DC SA620/SA630, and Kingston DC600M all carry PLP plus no - Linux-quirk entries, and vendors that publish firmware release notes - (Micron, Solidigm, WD, Kingston) can actually ship you a fix. NVMe - sidesteps the SATA queued-TRIM class entirely — my PM9A3s aren't in the - NVMe quirk table either. - -## The result - -One variable changed (auto-TRIM off), everything else identical. Immediately -after: - -- scrub #2 (04:10–04:39): **`corrected_errors: 0`** — the 16 heal-blocks - stayed healed, nothing new appeared during the full re-read. -- `corruption_errs` flat at sdd1=27 / sdb1=31 for hours of normal Docker + - VM-less operation (counters had been climbing daily before). -- No new `csum failed` lines in dmesg. - -The confirmation window runs a few days: Win11 VM back on (it produces the -write pattern that reproduced this), daily checks against a baseline file, -and a re-scrub on 2026-09-22. If corruption reappears with TRIM off, the next -suspect isn't the drives — it's the RAM (no ECC on this box), and memtest is -the test. If it stays quiet, the IronWolf 110 queued-TRIM firmware bug is -confirmed as the cause of both this incident and — I'd bet — a good share of -the "healthy drives corrupting my RAID" reports the forums keep collecting. - -A "failing" SMART warning, a drive with zero errors, and corruption on -both mirrors of a fresh pool — every sign pointed somewhere else, and the -actual culprit was a storage feature most of us enable without a second -thought. - ---- - -*Part of my storage-failure series: [the NVMe incident](/posts/when-smart-says-healthy-but-your-raid-is-corrupting-data/) and the [SATA cable wake-up call](/posts/that-dying-ssd-was-just-a-bad-sata-cable/).* -```` - ---- - -## 文章 05 / 11：上游被删了，然后带着一个重写版回来 -- **Slug**: `the-upstream-was-deleted-then-came-back-rewritten` -- **英文标题**: The Upstream Was Deleted, Then Came Back as a Rewrite -- **分类**: `devops` | **发布日期**: 2026-10-06 -- **线上地址**: https://blog.hoelee.com/posts/the-upstream-was-deleted-then-came-back-rewritten/ (EN) | https://blog.hoelee.com/posts/zh/the-upstream-was-deleted-then-came-back-rewritten/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "上游被删了，然后带着一个重写版回来" -description: "我线上跑的阅读服务，依赖的镜像官方仓库已经 404。几个月后上游回来了——233 个提交、换了语言、零 release。怎么分辨「项目死了」和「项目在重写」。" -pubDate: 2026-10-06 -category: devops -tags: ["docker", "self-hosting", "upstream", "maintenance", "registry"] -ogImage: "/og/zh-the-upstream-was-deleted-then-came-back-rewritten.jpg" -banner: "/banners/the-upstream-was-deleted-then-came-back-rewritten.jpg" -draft: false ---- - -`docker pull hectorqin/reader` 返回 **404**。而这正是我线上阅读服务所依赖的项目：一个只发邀请码的小书库，给我家人和几位付费读者用，每天在跑，上面还搭了一条我自己做的语音朗读管线，让「朗读」按钮念出来的是正经的神经网络音色。 - -404 还不是最有意思的部分。**2026-09-16**，同一个仓库被从零重新初始化——提交信息是 `chore: 初始化仓库`——十六天后，它已经有了 **233 个提交**、另一门语言、另一个端口、另一种数据库、另一个镜像仓库。我依赖的项目没有死，它是被**一个重写版替换了**，名字没变、地址没变、那 11,036 颗星也没变。 - -所以问题是：**怎么判断一个上游项目是死了，还是正在被重写？而当它以另一个应用的样子回来时，你该怎么做？** - -## 先说我这边的情况，因为细节才是这件事的关键 - -这个应用是一个自托管阅读服务器。浏览器端是 hash 路由的 Vue SPA，服务端是 Kotlin + Spring Boot，配置按用户存成磁盘上的 JSON。而我专门为它的一条能力搭了一整套管线：**HTTP 语音合成**。阅读器会把朗读请求代理到你配置的 URL，也就是说音色不是应用自带的——我的指向 n8n 的两个 webhook，由它们去换取短期 Azure / Google 凭据并合成音频。语音选择器里那 9 个引擎是这么来的，不是应用自带的。 - -此外，应用的镜像外面还有我自己的一层 nginx 网关容器：对外落地页、社交分享卡片、中英切换、以及一层英文界面补丁，全在那里，不在应用镜像里。当初做这个决定是为了品牌，而正是它让整件事从头到尾都很平静——后面我会回到这点。 - -## 我去找升级路径的时候，发现了什么 - -三个选项，没有一个叫「升级」： - -| 选项 | 它实际是什么 | 我为什么没选 | -|---|---|---| -| 官方镜像 | Docker Hub 上的 `hectorqin/reader` | 没了。仓库返回 404，`docker pull` 直接失败 | -| 还在维护的 fork | `changshengyu/reader`，版本 `2.5.4` | 基于最后一个完全开源的快照。它的朗读只用浏览器自带的 `speechSynthesis`，没有 HTTP 语音。它的整个历史里没有任何 `httpTTS` 字样，`/reader3/httpTts` 也是 404。我那 9 个引擎会全部消失 | -| 别人重新打包的镜像 | `liangnianzhi/reader.hectorqin:latest-20250525` | **这就是我现在跑的。** 唯一还能拉取、并且保留了 HTTP 语音的 3.x 构建。最后更新 2025-09-05，一共三个标签，发布者不是原作者 | - -我把那个标签钉死，把数据目录打包备份，把源码镜像到自己的 Git 服务器上——至少我依赖的代码得存在于我能控制的地方。然后把那句决定未来任何镜像是否有资格的话写了下来：**它是否还讲我这条管线所实现的 TTS 协议？** - -这就是当时的处境：一个跑在「官方仓库已被删除」的镜像上的服务，靠着某个陌生人重新打包的构建活着。 - -## 然后上游回来了 - -以下数据于 2026-10-06 直接取自 GitHub API 与镜像仓库： - -| 指标 | 数值 | -|---|---| -| 仓库 | `hectorqin/reader` —— 活着，AGPL-3.0，默认分支 `main` | -| Star / Fork | 11,036 / 5,471 | -| 最后推送 | 2026-10-02 | -| 提交数 | **233** —— 而最早的一条是 `chore: 初始化仓库`，日期 **2026-09-16** | -| release / tag | **0 / 0** | -| Docker Hub | 仍然 **404** | - -2021 年那段历史还在，但已经断开了。`main` 上是 233 个提交，最早一条的日期是 2026-09-16；旧分支仍然存在，但只剩**一个提交，日期 2026-06-22**，提交信息就是 `commit`，而它的 README 全文只有一个词——`deleted`。两个分支**没有共同祖先**——这就是「先清空、再重来」在 git 里的样子。 - -而这 233 个提交里的代码，不是我原来那个代码库的延续。它是另一个应用，只是恰好沿用了同一个名字： - -| | 我现在跑的（3.x） | 上游现在的样子 | -|---|---|---| -| 运行时 | Kotlin，Spring Boot + Vert.x | TypeScript，Node | -| 端口 | 8080 | 5888 | -| 数据 | `/storage/data`，按用户存 JSON | `/data`，SQLite（`reader.db`） | -| 配置 | 环境变量 + 每用户 JSON 文件 | 管理后台界面，存进数据库 | -| 镜像 | Docker Hub，命名空间已删除 | `cnb.cool/hectorqin/reader:main` | -| 范围 | 只有书 | 书 **和** 电影、剧集、音乐、有声书、OPDS、OpenList、PWA、Android 客户端 | - -表里有三个细节值得单独拎出来，因为它们各自都有一个后果： - -**镜像仓库搬去了一个大多数人从没拉过东西的平台。** 官方 compose 里写的是 `image: ${READER_IMAGE:-cnb.cool/hectorqin/reader:main}`，旁边一行注释解释了原因：CI 只发布**分支和提交标签，刻意不发布 `:latest`**。也就是说默认配置跟的是一个会动的分支，唯一稳定的引用是一个 commit SHA。 - -**配置离开了环境变量。** 业务设置——HTTP 语音接口、扫描行为、登录有效期、对外 URL、允许的浏览器来源、WebDAV——现在都在管理后台里，持久化进数据库。文档说明：首次升级时「旧版业务环境变量会被导入一次」，并且已存在的数据库值永远优先。这是一次性的导入加一条优先级规则，正是你希望在副本上验证、而不是在线上实例上发现的东西。 - -**HTTP 语音回来了，只是换了个地方。** 新版本重新提供了 HTTP 语音上游的配置：合成 URL、token、voices URL、超时、缓存上限，还有连接测试和试听，全部在管理后台里配。而我的 n8n webhook 实现的协议，是我照着旧版构建自己设计的。这两个形状是否匹配，现在是一个需要用实验回答的问题，不是查文档就能回答的问题。 - -## 没人提醒你的部分：重写版是一个新应用 - -本能反应是把「上游复活了，还多了影音支持和 Android 客户端」读成「我等的升级到了」。并不是。一次重写会同时改变数据格式、部署面和配置模型——这意味着要做的是**一场带 schema 边界的迁移**，而不是 `docker pull` 加一次重启。 - -让我决定慢下来的，是项目自己的进度记录。不要拿厂商的营销页当状态报告，去看它自己写的东西，你会看到作者在列**没做完**的事情： - -- 旧数据**自动迁移和首次接入向导都还没实现**； -- **镜像拉取与真实容器升级演练从未执行过**，因为构建机上没有跑 Docker 引擎。 - -这是一份诚实得罕见的文档，也是整个仓库里最有用的一个文件。重写的作者在告诉我：升级路径写好了，但没人走过。如果连他们都没走过，我当然不会在工作日下午、在我读者正在使用的实例上、在没有数据副本的情况下走去试。 - -## 我现在会把一个依赖交给线上服务之前做的事 - -1. **仓库健康不等于产物健康。** `pushed_at`、star、fork 说明**源码**还活着；对你要跑的那个确切标签执行 `docker pull`，才能说明**你部署的东西**还在。两个都要查，并读一遍 compose 看它到底指向哪个镜像仓库——我这个现在指向一个我本周之前从没打开过的域名。 -2. **「没有 release、没有 tag」是一条部署事实。** 零 tag 意味着每次升级要么写死一个提交哈希，要么跟一个会动的分支。把那个 SHA 记进自己的笔记，上游不会为你记。 -3. **你输不起的东西，就镜像一份——源码和镜像都要。** 一个钉在别人命名空间里的标签，是一种「那个陌生人账号别出事」的依赖。我活下来的 3.x 源码现在在我自己的 Git 服务器上，重写版也是。 -4. **先比形状，再比功能。** 端口、挂载路径、变量名、数据库引擎、配置存放位置。如果这些大多变了，就把它当成一个新产品，按迁移来规划。 -5. **你重建不了的部件，放在你控制不了的镜像之外。** 我的落地页、分享卡片、语言切换和翻译层，是同一条 stack 里的另一个容器。正因为如此，上游这次重写没有打扰到任何一个读者——应用镜像怎么换，都碰不到前门。 -6. **读维护者的「还没做」清单，而不是功能清单。** 功能清单是销售页，未完成清单才是风险登记表。 -7. **如果你把它当成对外服务来跑，读一下许可证。** 这个是 AGPL-3.0，如果你拿它给付费用户提供服务、而不是自己用，值得认真读一遍。 -8. **「仓库被归档或被清空」不等于「项目被放弃」。** 有好几个月，「这个项目死了吗」的诚实答案是「是」——README 就写着 `deleted`，而且它说的是实话。然后有人重新开始了，换了门语言，而诚实的答案在名字不变的情况下变了。 - -## 结果 - -书库在跑，跑在我钉住的那个构建上，读者完全无感——这本来就是目标，而且它一直成立。变的是：「我们被困在一个没人管的镜像上」从一句含糊的担忧，变成了一个有方案的具体问题： - -- 数据目录已复制，新镜像自带的 `backup` / `verify` / `restore` 命令先**对着副本**跑，确认没问题才碰线上卷； -- TTS 协议要拿我现有的 webhook 实测，因为一旦对不上，我丢掉的就是朗读这个功能； -- 旧镜像标签留着做回滚，而且无论走哪条路，源码现在都在我自己的服务器上； -- 前门不在讨论范围内，因为它在另一个容器里，换镜像够不着它。 - -真正有用的教训不是「要钉版本」——那个我早就做了。而是：**「这个项目还活着吗」有三个答案，而它们可以互相矛盾**。仓库可以活着，而镜像已被删除；README 可以写着 `deleted`，而重写版已经开工四周；版本号可以纹丝不动，而运行时、端口、数据库和配置模型在底下全换了一遍。这三件事要分别去查，否则你迟早会为一个「已经不再以你测试过的形态存在」的应用规划升级。 - ---- - -**你在替别人跑自托管服务，但不确定底下的东西还有没有人管？** 这正是我在做的审计：依赖与镜像健康度、升级可行性，以及随之而来的迁移方案。 - -[WhatsApp +60 12-797 2969](https://wa.me/60127972969) · -[me@hoelee.com](mailto:me@hoelee.com?subject=Self-hosted%20dependency%20audit) · -[hoelee.com](https://hoelee.com) -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "The Upstream Was Deleted, Then Came Back as a Rewrite" -description: "My production reading service runs a build whose official Docker Hub repo returns 404. Then upstream came back — 233 commits, a new language, no releases. How to tell dead from rewriting." -pubDate: 2026-10-06 -category: devops -tags: ["docker", "self-hosting", "upstream", "maintenance", "registry"] -ogImage: "/og/the-upstream-was-deleted-then-came-back-rewritten.jpg" -banner: "/banners/the-upstream-was-deleted-then-came-back-rewritten.jpg" -draft: false ---- - -`docker pull hectorqin/reader` returns **404**. That is the project my -production reading service runs on: a small invite-only library I host for -my family and a handful of paying readers, serving books every day, with a -text-to-speech pipeline I built on top of it so the read-aloud button speaks -in proper neural voices. - -The 404 is not the interesting part. On **2026-09-16** that same repository -was re-initialised from scratch — the commit message is `chore: 初始化仓库`, -"initialise repository" — and sixteen days later it carried **233 commits**, -a different language, a different port, a different database engine and a -different container registry. The project I depend on did not die. It was -**replaced by a rewrite** under the same name, the same URL and the same -11,036 stars. - -So: **how do you tell whether an upstream project is dead — and what do you -do when it comes back as a different application?** - -## The setup, because the details are why this mattered - -The app is a self-hosted reading server. Hash-routed Vue SPA in the browser, -Kotlin and Spring Boot on the server, per-user JSON config on disk, and one -feature I had deliberately built a whole pipeline around: **HTTP -text-to-speech**. The reader proxies read-aloud requests to a URL you -configure, so the voices are not baked into the app — mine point at two n8n -webhooks that mint short-lived Azure and Google credentials and synthesise -the audio. Nine engines show up in the voice picker because of that, not -because the app shipped them. - -I also run my own nginx gateway container in front of the app — the public -landing page, the social card, the bilingual switch and an English-UI layer -all live there rather than inside the app's image. That decision, made for -branding reasons, is the reason this whole incident stayed boring. I'll come -back to it. - -## What I found when I went looking for an upgrade - -Three options, and none of them was "upgrade": - -| Option | What it actually is | Why I didn't take it | -|---|---|---| -| The official image | `hectorqin/reader` on Docker Hub | Gone. The repository returns 404; `docker pull` fails | -| The maintained fork | `changshengyu/reader`, pinned at `2.5.4` | Built on the last fully open snapshot. Its read-aloud uses the browser's own `speechSynthesis` only — no HTTP speech. No `httpTTS` string anywhere in its history, and `/reader3/httpTts` 404s on it. My nine engines would vanish | -| A rebuild published by someone else | `liangnianzhi/reader.hectorqin:latest-20250525` | **This is what I run.** The only still-pullable 3.x build that kept HTTP speech. Last updated 2025-09-05, three tags total, published by a user who is not the original author | - -I pinned that tag, tarred up the data directory, and mirrored the source -into my own Git server so the code I depend on existed somewhere I control. -Then I wrote down the one line that decides whether any future image is even -a candidate: **does it still speak the TTS protocol my pipeline -implements?** - -That was the state of things: a running service on an image whose official -home had been deleted, surviving as somebody's rebuild. - -## Then upstream came back - -Verified on 2026-10-06, straight from the GitHub API and the registry: - -| Signal | Value | -|---|---| -| Repo | `hectorqin/reader` — alive, AGPL-3.0, default branch `main` | -| Stars / forks | 11,036 / 5,471 | -| Last push | 2026-10-02 | -| Commits | **233** — and the oldest is `chore: 初始化仓库`, dated **2026-09-16** | -| Releases / tags | **0 / 0** | -| Docker Hub | still **404** | - -The 2021 history is there but disconnected. `main` carries 233 commits and -its oldest is dated 2026-09-16; the old branch survives as a **single commit -dated 2026-06-22** whose message is just `commit`, and whose entire README is -the one word `deleted`. The two branches share **no common ancestor** — which -is what a wipe followed by a fresh start looks like in git. (`created_at` -still reads 2021-08-13, because GitHub keeps that field when a repository is -re-populated; the history is what tells you the truth.) - -And the code behind those 233 commits is not my app's codebase continued. It -is a different application that happens to keep the name: - -| | What I run (3.x) | What upstream is now | -|---|---|---| -| Runtime | Kotlin, Spring Boot + Vert.x | TypeScript, Node | -| Port | 8080 | 5888 | -| Data | `/storage/data`, per-user JSON | `/data`, SQLite (`reader.db`) | -| Configuration | environment variables + per-user JSON files | the admin UI, persisted in the database | -| Container image | Docker Hub, namespace deleted | `cnb.cool/hectorqin/reader:main` | -| Scope | books | books **and** movies, series, music, audiobooks, OPDS, OpenList, PWA, an Android client | - -Three details are worth pulling out of that table, because each one has a -consequence: - -**The registry moved to a platform most of us have never pulled from.** The shipped compose file uses `image: ${READER_IMAGE:-cnb.cool/hectorqin/reader:main}`, and a comment explains that CI publishes **branch and commit tags and deliberately does not publish `:latest`**. So the default configuration tracks a moving branch, and the only stable reference is a commit SHA. - -**Configuration left the environment.** Business settings — the HTTP speech endpoint, scan behaviour, login lifetimes, the public URL, allowed browser origins, WebDAV — now live in the admin UI and persist in the database. The docs note that on first upgrade "legacy business environment variables are imported once", and that existing database values always win. That is a one-shot import with a precedence rule, which is exactly the kind of thing you want to test on a copy instead of discovering on your live instance. - -**HTTP speech came back — in a different place.** The new version documents an HTTP speech upstream again, with a synthesis URL, a token, a voices URL, a timeout and a cache limit, plus a connection test and audio preview, all configured in the admin UI. My n8n webhooks implement a protocol I designed against the old build's expectations. Whether those two shapes match is now an empirical question, not a documentation question. - -## The part nobody warns you about: a rewrite is a new application - -The instinct is to read "upstream is alive again, with media support and an -Android client" as *my upgrade arrived*. It isn't. A rewrite changes the -data format, the deployment surface and the configuration model at the same -time — which means the work is a migration with a schema boundary in it, not -a `docker pull` and a restart. - -What convinced me to slow down was the project's own progress note. Read it -for yourself rather than taking a vendor's marketing page as the status -report, and you find the authors listing what is **not** done: - -- legacy data **auto-migration and a first-run setup wizard are not implemented**; -- the **image pull and the real container upgrade drill were never executed**, because the Docker engine was not running on the build machine. - -That is unusually honest documentation, and it is the single most useful -file in the repository. The people writing the rewrite are telling me the -upgrade path has been written but not walked. If they haven't walked it, I'm -certainly not walking it on the instance my readers are using, on a Tuesday, -without a copy of the data. - -## What I do now, before trusting any dependency with a running service - -1. **Repository health is not artifact health.** `pushed_at`, stars and -forks tell you the *source* is alive. `docker pull` on the exact tag you run -tells you the *thing you deploy* still exists. Check both, and read the -compose file to see which registry it names — mine now points at a domain I -had never opened before this week. 2. **"No releases, no tags" is a -deployment fact.** Zero tags means every upgrade is a commit hash or a -moving branch. Write the SHA in your own notes, because the upstream will -not. 3. **Mirror anything you cannot afford to lose — the source *and* the -image.** A pinned tag in someone else's namespace is a dependency on that -stranger's account staying alive. My surviving 3.x source sits in my own Git -server now, and so does the rewrite. 4. **Compare the shape before the -features.** Port, volume paths, variable names, database engine, where -configuration lives. If most of those moved, treat the new version as a new -product and plan a migration. 5. **Keep what you can't rebuild outside the -image you don't control.** My landing page, social card, language switch and -translation layer are a separate container in the same stack. That is why a -rewrite landing upstream did not disturb a single reader — the front door -cannot be broken by an app image swap. 6. **Read the maintainer's "not done -yet" list, not the feature list.** The feature list is a sales page; the -unfinished list is the risk register. 7. **Check the licence if you run it -as a service.** This one is AGPL-3.0, which is worth reading properly if you -host it for paying users rather than just yourself. 8. **An archived or -wiped repository is not the same as an abandoned project.** For months the -honest answer to "is this project dead?" was yes — the README said `deleted` -and meant it. Then somebody started again, in another language, and the -honest answer changed without the name changing. - -## The result - -The library is up, on the build I pinned, with readers unaffected — that was -always the goal, and it never stopped being true. What changed is that -"we're stranded on an orphaned image" became a known quantity with a plan: - -- the data directory is copied, and the new image's own `backup` / `verify` / `restore` commands get run **against the copy** before anything touches the live volume; -- the TTS protocol gets tested against my existing webhooks, because if it doesn't match, the read-aloud feature is the thing I lose; -- the old image tag stays present for rollback, and I now have the source on my own server either way; -- the front door stays out of scope, because it lives in a different container and an image swap cannot reach it. - -And the useful lesson is not "pin your versions", which I already did. It is -that **the question "is this project alive?" has three answers, and they can -disagree**: the repository can be alive while the image is deleted; the -README can say `deleted` while a rewrite is four weeks into existence; and -the version number can stay the same while the runtime, the port, the -database and the configuration model all change underneath it. Check each -one separately, or you will eventually plan an upgrade for an application -that no longer exists in the shape you tested. - ---- - -**Running a self-hosted app for other people, and not sure whether the thing underneath it is still supported?** That's the kind of audit I do — dependency and image health, upgrade feasibility, and the migration plan that comes with it. - -[WhatsApp +60 12-797 2969](https://wa.me/60127972969) · -[me@hoelee.com](mailto:me@hoelee.com?subject=Self-hosted%20dependency%20audit) -· [hoelee.com](https://hoelee.com) -```` - ---- - -## 文章 06 / 11：Unraid 停止阵列时卡死的真相：Swapfile 锁住挂载点 — 附修复方案 -- **Slug**: `unraid-stop-array-hangs-on-swapfile` -- **英文标题**: Why Unraid Hangs When You Stop the Array With a Swapfile — and the Fix -- **分类**: `devops` | **发布日期**: 2026-09-15 -- **线上地址**: https://blog.hoelee.com/posts/unraid-stop-array-hangs-on-swapfile/ (EN) | https://blog.hoelee.com/posts/zh/unraid-stop-array-hangs-on-swapfile/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "Unraid 停止阵列时卡死的真相：Swapfile 锁住挂载点 — 附修复方案" -description: "Unraid 停止阵列时一直卡在卸载磁盘：btrfs RAID1 上的 8GB swapfile 通过 loop 设备锁住 /mnt/cache。根因分析、/boot/config/stop 陷阱、两个 User Scripts 彻底修复。" -pubDate: 2026-09-15 -category: devops -blueprint: self-hosted-infrastructure -blueprintLayer: L1-compute-storage-network -tags: [unraid, swap, btrfs, zram, docker, user-scripts] -ogImage: /og/zh-unraid-stop-array-hangs-on-swapfile.jpg -banner: /banners/unraid-stop-array-hangs-on-swapfile.jpg -draft: false ---- - -我给 Unraid 服务器加了 8 GB swapfile。缓存池是 btrfs RAID1，而内核不允许 -swapfile 直接建在多设备 btrfs 文件系统上，所以走了 loop 设备这个标准变通方 -案。Swap 生效了，内存压力降了，一切正常——直到有一天我点了「停止阵列」。 - -WebUI 永远停在 **"Retry unmounting user shares…"**。缓存池就是卸不掉。 -Docker 已停、虚拟机已关、共享已卸载——`umount /mnt/cache` 依然每 5 秒报一 -次 *target is busy*，无限循环。唯一出路是重启。 - -这篇文章讲清楚三件事：为什么会这样；我中途踩的坑（`/boot/config/stop` 在 -你点「停止」时**根本不会运行**）；以及两个脚本的修复方案——之后每次停止/启 -动都干净利落。 - -## 问题本身（按你会搜索的方式来描述） - -如果你是搜索进来的，你大概率在搜： - -- *"unraid stop array stuck retry unmounting user shares"* -- *"umount /mnt/cache target is busy unraid"* -- *"unraid swapfile stop array hang"* -- *"unraid 停止阵列 卡住"* - -同一个 bug。一句话版本：**swapfile（或它的 loop 设备）被内核 swap 子系统 -持有打开，而 Unraid 的停止阵列流程里没有任何一步会先关掉 swap 再卸载磁 -盘。** 挂载点忙，是因为内核自己在用它——除非有人执行 `swapoff`，否则它会 -一直用下去。而没有人执行。 - -## 我试过的方案，以及为什么都失败 - -### 尝试 1：重启后再停阵列——照样卡死 - -意料之中。swapfile 配置在 `/boot/config/go` 里，每次开机都会重建 loop 设 -备并启用 swap，每次停止都撞同一堵墙。至少这证明了问题是确定性的，不是偶 -发进程卡住。 - -### 尝试 2：在 `/boot/config/stop` 里写 `swapoff` - -这是最经典的陷阱，Unraid 官方文档让它看起来就是正确答案。`stop` 脚本的存 -在意义就是「磁盘卸载前跑清理」——它确实会运行，**但只在完整关机或重启 -时**。 - -我在自己的机器上验证过（Unraid 7.1.4）：`/boot/config/stop` 由 -`rc.local_shutdown` 调用，而后者在 `rc.6` 里。在 WebUI 点**停止**走的是完 -全不同的路径——`emhttpd` 触发一系列 *event*——`rc.6` 根本不参与。没有关 -机，就没有 `stop` 脚本，就没有 `swapoff`。阵列停止照样卡死。 - -### 尝试 3：搞清楚 GUI「停止」到底做了什么 - -真正的答案在这里。Unraid 的事件调度器（`emhttp_event`）在阵列停止时按文 -档化顺序执行钩子脚本： - -```text -stopping → stopping_libvirt → stopping_docker → stopping_svcs - → unmounting_disks → stopping_array → stopped -``` - -两个关键事实： - -1. 卸载发生在 `unmounting_disks`。**任何 swapoff 必须在 `stopping_svcs` - 或更早触发**——比卸载早一个事件。 -2. 这些事件正是 **User Scripts** 插件暴露出来的调度选项（「At Stopping of - Array」= `stopping_svcs`，「At First Array Start only」= - `disks_mounted`）。不需要写新代码——自动化接口本来就在那里。 - -### 隐蔽的部分：`/proc/swaps` 对 loop swap 说谎 - -这是让「看起来正确」的清理脚本也会失败的坑。我的 `/boot/config/go` 原来 -是这样配置 swap 的（btrfs RAID1 标准写法）： - -```bash -# /boot/config/go — 原始（停止时会卡的）配置 -mkdir -p /mnt/cache -MOUNTPOINT=/mnt/cache -SWAPFILE=${MOUNTPOINT}/swapfile - -# 等待缓存池挂载 -while [ ! -f "${SWAPFILE}" ]; do - sleep 1 -done - -# btrfs RAID1 拒绝直接 swapfile：loop 设备 + NOCOW -chattr +C "${SWAPFILE}" -LOOPDEV=$(losetup -f) -losetup -p 100 "${LOOPDEV}" "${SWAPFILE}" -swapon -p 1 "${LOOPDEV}" -``` - -现在看内核在活动状态下报告的内容： - -```console -$ cat /proc/swaps -Filename Type Size Used Priority -/dev/loop0 partition 8388604 0 -2 - -$ swapon --show -NAME TYPE SIZE USED PRIO -/dev/loop0 partition 8G 0B -2 -``` - -**文件名是 `/dev/loop0`，不是 `/mnt/cache/swapfile`。** 所以这样的清理脚 -本： - -```bash -# 错误 — 对 loop swap 静默失效 -grep -q '/mnt/cache/swapfile' /proc/swaps && swapoff /mnt/cache/swapfile -``` - -……什么都匹配不到，什么都关不掉，然后你又一次卡在停止阵列。必须*先*解析 -loop 设备——`losetup -j <文件>` 就是干这个的——再 `swapoff` 那个 loop 设 -备，最后卸载它。ZRAM Compressed Memory 插件的源码注释几乎一字不差地记录 -了同一个 bug：*"for loop-backed swap, /proc/swaps lists /dev/loopN, not -the image path — a plain grep never matches, so swapoff never ran and the -loop device kept the pool busy forever."*（loop 型 swap 在 /proc/swaps 里显 -示为 /dev/loopN 而不是镜像路径——普通 grep 永远匹配不到，swapoff 从未执 -行，loop 设备让缓存池永远忙碌。） - -## 修复方案：两个 User Scripts - -大多数 Unraid 机器已经装了 [User -Scripts](https://forums.unraid.net/topic/48286-plugin-ca-user-scripts/) -插件。创建两个脚本： - -**脚本 1 —「Array Stop Swapoff」**——调度选 **At Stopping of Array**（在 -`stopping_svcs` 触发，比卸载早一个事件）： - -```bash -#!/bin/bash -# 先释放所有 swap，再只卸载【我们的】swapfile 背后的 loop 设备。 -swapoff -a 2>/dev/null -for l in $(losetup -j /mnt/cache/swapfile 2>/dev/null | awk -F: '{print $1}'); do - losetup -d "$l" 2>/dev/null -done -``` - -`losetup -j`（「哪个 loop 设备挂在这个文件上？」）是关键一步——它不受 -`/proc/swaps` 命名谎言的影响。而且只卸载*我们自己的* loop 设备很重要：机 -器上可能还有别的 loop 设备（挂载的 ISO、其他镜像），它们必须在停止阵列时 -存活下来。 - -**脚本 2 —「Array Start Swapon」**——调度选 **At First Array Start -only**（在 `disks_mounted` 触发）： - -```bash -#!/bin/bash -# 缓存池挂载后重建 loop swap。 -[ -f /mnt/cache/swapfile ] || exit 0 -LOOP=$(losetup -f) -losetup -p 100 "$LOOP" /mnt/cache/swapfile && swapon -p 1 "$LOOP" -``` - -为什么选「First Array Start only」而不是普通的「At Startup of the -Array」？因为开机时 `/boot/config/go` 也会运行并配置一次 swap。开机后的 -*第一次*阵列启动由 `go` 负责；这个脚本真正管的是**不重启、纯 GUI 停止 → -启动的循环**——而 `go` 永远看不到这种场景。（在我的配置里，我把 swap 那段 -从 `go` 里整个移除了，让 User Script 成为唯一事实来源——一处配置，两条路 -径都覆盖。） - -之后：停止 → 缓存池第一次尝试就干净卸载。启动 → swap 恢复。多次停止/启动 -循环和重启后都验证通过。 - -## 我会做得不一样的地方 - -**1. 不要把 swap 放在已经快死的盘上。** 我的缓存池是一对后来确认在静默损 -坏数据的 NVMe（SMART 全绿，读取却返回全零块）。8 GB swapfile 放在那里意 -味着对我即将更换的硬件持续写入磨损——而且它正是在迁移期间卡住我阵列停止 -的元凶。swap 的位置要匹配磁盘健康状况，而不是图方便。 - -**2. CPU 够用时，压缩内存 swap 优于磁盘 swap。** 这台机器的最终方案是 -[ZRAM Compressed -Memory](https://forums.unraid.net/topic/196763-new-plugin-created-zram/) -插件：Tier-1 swap 放在 zstd 压缩的 RAM 块设备里。两个好处： - -- **卡死在结构上变得不可能。** zram 设备不是挂载点，`umount /mnt/cache` - 永远看不见它，停止时无需释放任何东西。（如果你启用它的 Tier-2 磁盘 - swapfile，插件也自带正确的 `event/stopping` 钩子，用的正是 - `losetup -j` 这个修复。） -- **swap 换入换出是微秒级 CPU 操作而不是磁盘 I/O**——在现代多核 CPU 上， - 压缩相对 NVMe 延迟几乎免费，而且被换出的页面在 RAM 里省约 3:1 空间。 - -一个我用惨痛教训学到的注意事项：插件的自动大小（RAM 的 50%）在我 62 GB -内存、已用 90% 的机器上给了一个 **31.3 GB** 的 zram 设备。那个数字是*未压 -缩上限*——实际 RAM 消耗 = 容量 ÷ 压缩比，所以装满的 31.3 GB zram 可能吃掉 -10–30 GB 我本来就稀缺的空闲内存。在接近满载的机器上，zram 要固定且保守地 -设置大小（我改成 8 GB——原来 swapfile 的大小——swappiness 150），让它缓冲 -OOM 而不是和 OOM 抢内存。 - -**3. swap 只能推迟 OOM，治不了 OOM。** 我内存压力的诚实根因是虚拟机内存 -分配，不是缺 swapfile。swap 买来了余量、让宿主机在峰值时活了下来——但真正 -的修复是给虚拟机合理分配内存。 - -## 结果 - -停止阵列从「永远卡死、强制重启」变成「第一次尝试就干净卸载」，swap 在停止/ -启动循环之间自动存活，无需人工干预。修复总量：两个 shell 脚本，跑在一个早 -就装好的插件里。贵的部分是知道该把它们放在事件序列的**哪个位置**——以及 -`/proc/swaps` 不会告诉你 loop swap 的真相。 - -如果你的 Unraid 阵列拒绝停止，而你配置过任何形式的 swap——文件、loop 或其 -他——在 root shell 里执行 `swapoff -a` 会立刻释放它，诊断就此确认。然后在 -下次重启抹掉你的记忆之前，把它自动化。 - ---- - -*你也有需要这种深度调试的自托管环境吗——Unraid、Docker、反向代理、NAS 迁 -移？这是我的本行：[网站设计与开发](https://hoelee.com)是主业，自托管基础 -设施是我钻研最深的领域。欢迎通过 -[WhatsApp](https://wa.me/60127972969) 或 -[邮件](mailto:me@hoelee.com?subject=Unraid%20infrastructure%20help)联系 -我。* -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Why Unraid Hangs When You Stop the Array With a Swapfile — and the Fix" -description: "My Unraid array stop hung forever at unmounting disks because an 8 GB btrfs swapfile pinned /mnt/cache. Root cause, the /boot/config/stop trap, and two User Scripts that fix it." -pubDate: 2026-09-15 -category: devops -blueprint: self-hosted-infrastructure -blueprintLayer: L1-compute-storage-network -tags: [unraid, swap, btrfs, zram, docker, user-scripts] -ogImage: /og/unraid-stop-array-hangs-on-swapfile.jpg -banner: /banners/unraid-stop-array-hangs-on-swapfile.jpg -draft: false ---- - -I added an 8 GB swapfile to my Unraid server — my cache pool is a btrfs RAID1, -and the kernel refuses a swapfile that lives directly on a multi-device btrfs -filesystem, so it goes through a loop device, the standard workaround. Swap -worked. Memory pressure dropped. Everything was fine… until the day I stopped -the array. - -The WebUI sat on **"Retry unmounting user shares…"** forever. The cache pool -would not release. Docker was stopped, VMs were shut down, shares were -unmounted — and still, `umount /mnt/cache` failed with *target is busy*, -every five seconds, indefinitely. The only way out was a reboot. - -This is the story of finding out why, the trap I fell into on the way -(`/boot/config/stop` does **not** run when you click Stop), and the two-script -fix that survives every stop/start cycle since. - -## The problem, phrased the way you'd Google it - -If you landed here from a search, you're probably typing one of these: - -- *"unraid stop array stuck retry unmounting user shares"* -- *"umount /mnt/cache target is busy unraid"* -- *"unraid swapfile stop array hang"* -- *"unraid won't stop array after adding swap"* - -Same bug. The short version: **a swapfile — or its loop device — is held open -by the kernel's swap subsystem, and Unraid's array-stop sequence has no step -that turns swap off before it unmounts disks.** The mount is busy because the -kernel itself is using it, and it will keep using it until someone runs -`swapoff`. Nobody does. - -## What I tried, and why it failed - -### Attempt 1: reboot, stop array again — hung again - -Obviously. The swapfile was still configured in `/boot/config/go`, so every -boot recreated the loop device and enabled swap on it. Every stop hit the same -wall. At least this confirmed it was deterministic, not a one-off stuck -process. - -### Attempt 2: put a `swapoff` in `/boot/config/stop` - -This is the classic trap, and the Unraid docs make it look like the right -answer. The `stop` script exists precisely for "run cleanup before disks go -away" — and it does run, **but only during a full shutdown or reboot**. - -I checked on my own box (Unraid 7.1.4): `/boot/config/stop` is invoked by -`rc.local_shutdown`, which is called from `rc.6`. Clicking **Stop** in the -WebUI goes through a completely different path — `emhttpd` fires a series of -*events* — and `rc.6` is never involved. No shutdown, no `stop` script, no -`swapoff`. The array stop still hung. - -### Attempt 3: read what the GUI Stop actually does - -This is where the real answer lives. Unraid's event dispatcher (`emhttp_event`) -runs hook scripts in a documented sequence when the array stops: - -```text -stopping → stopping_libvirt → stopping_docker → stopping_svcs - → unmounting_disks → stopping_array → stopped -``` - -Two facts matter: - -1. The unmount happens at `unmounting_disks`. **Any swapoff must fire at - `stopping_svcs` or earlier** — one event before the unmount. -2. These events are exactly what the **User Scripts** plugin exposes as - schedule options ("At Stopping of Array" = `stopping_svcs`, "At Startup of - the array" / "At First Array Start only" = `array_started` / - `disks_mounted`). No new code needed — the automation surface already - exists. - -### The subtle part: `/proc/swaps` lies about loop-backed swap - -Here's the gotcha that makes even a "correct-looking" cleanup script fail. -My `/boot/config/go` set swap up like this (the standard btrfs-RAID1 recipe): - -```bash -# /boot/config/go — the original (broken-on-stop) setup -mkdir -p /mnt/cache -MOUNTPOINT=/mnt/cache -SWAPFILE=${MOUNTPOINT}/swapfile - -# Wait for the cache pool to be mounted -while [ ! -f "${SWAPFILE}" ]; do - sleep 1 -done - -# Btrfs RAID1 rejects direct swapfiles: loop device + NOCOW -chattr +C "${SWAPFILE}" -LOOPDEV=$(losetup -f) -losetup -p 100 "${LOOPDEV}" "${SWAPFILE}" -swapon -p 1 "${LOOPDEV}" -``` - -Now look at what the kernel reports while that's active: - -```console -$ cat /proc/swaps -Filename Type Size Used Priority -/dev/loop0 partition 8388604 0 -2 - -$ swapon --show -NAME TYPE SIZE USED PRIO -/dev/loop0 partition 8G 0B -2 -``` - -**The filename is `/dev/loop0`, not `/mnt/cache/swapfile`.** So a cleanup -script like this one: - -```bash -# WRONG — silently does nothing for loop-backed swap -grep -q '/mnt/cache/swapfile' /proc/swaps && swapoff /mnt/cache/swapfile -``` - -…matches nothing, swaps nothing off, and you're back to the hang. You have to -resolve the loop device *first* — `losetup -j ` does exactly that — then -`swapoff` the loop device, then detach it. The ZRAM Compressed Memory plugin -documents this same bug in its source comment, almost word for word: *"for -loop-backed swap, /proc/swaps lists /dev/loopN, not the image path — a plain -grep never matches, so swapoff never ran and the loop device kept the pool -busy forever."* - -## The fix: two User Scripts - -[User Scripts](https://forums.unraid.net/topic/48286-plugin-ca-user-scripts/) -is already installed on most Unraid boxes. Create two scripts: - -**Script 1 — "Array Stop Swapoff"** — schedule: **At Stopping of Array** -(fires at `stopping_svcs`, one event before the unmount): - -```bash -#!/bin/bash -# Release ALL swap, then detach only the loop device backing OUR swapfile. -swapoff -a 2>/dev/null -for l in $(losetup -j /mnt/cache/swapfile 2>/dev/null | awk -F: '{print $1}'); do - losetup -d "$l" 2>/dev/null -done -``` - -`losetup -j` ("which loop device is attached to this file?") is the key move — -it's immune to the `/proc/swaps` naming lie. And detaching *only* our loop -device matters: a box can have other loop devices (mounted ISOs, other images) -that must survive an array stop. - -**Script 2 — "Array Start Swapon"** — schedule: **At First Array Start only** -(fires at `disks_mounted`): - -```bash -#!/bin/bash -# Re-create the loop-backed swap after the cache pool mounts. -[ -f /mnt/cache/swapfile ] || exit 0 -LOOP=$(losetup -f) -losetup -p 100 "$LOOP" /mnt/cache/swapfile && swapon -p 1 "$LOOP" -``` - -Why "First Array Start only" and not plain "At Startup of the Array"? Because -`/boot/config/go` also runs at boot and would set swap up a second time. The -*first* array start after boot is covered by `go`; this script matters for the -**GUI stop → start cycle with no reboot in between** — which `go` never sees. -(In my setup I moved the swap block out of `go` entirely so the User Script is -the single source of truth — one place, both paths.) - -After this: Stop → cache unmounts cleanly on the first try. Start → swap is -back. Verified across multiple stop/start cycles and reboots. - -## What I'd do differently - -**1. Don't put swap on a disk that's already dying.** My cache pool is a pair -of NVMe drives I've since confirmed are silently corrupting data (reads -returning all-zero blocks despite clean SMART). An 8 GB swapfile there meant -constant write wear on hardware I was about to replace — and it was the very -thing pinching my array stops during the migration. Match swap location to -disk health, not convenience. - -**2. Prefer compressed RAM over disk swap when you have the CPU for it.** The -endgame on this box was the [ZRAM Compressed -Memory](https://forums.unraid.net/topic/196763-new-plugin-created-zram/) -plugin: Tier-1 swap lives in a zstd-compressed RAM-backed block device. Two -wins: - -- **The hang becomes structurally impossible.** A zram device is not a mount. - `umount /mnt/cache` can never see it, so there's nothing to release at stop - time. (The plugin *also* ships a proper `event/stopping` hook with the - `losetup -j` fix, if you enable its Tier-2 disk swapfile.) -- **Swap-in/out is microseconds of CPU work instead of disk I/O** — on a - modern many-core CPU, compression is nearly free relative to even NVMe - latency, and you save ~3:1 on the RAM the swapped pages occupy. - -One caveat I learned the hard way: the plugin's auto-size (50% of RAM) gave me -a **31.3 GB** zram device on a 62 GB host that was already 90% full. That -number is an *uncompressed ceiling* — real RAM cost is capacity ÷ compression -ratio, so a full 31.3 GB zram could eat 10–30 GB of the very RAM I was short -of. On a nearly-full host, size zram fixed and modest (I went to 8 GB, the old -swapfile size, swappiness 150) so it buffers OOM instead of competing with it. - -**3. Swap delays OOM; it doesn't cure it.** The honest root cause of my memory -pressure was VM RAM budgets, not a missing swapfile. Swap bought headroom and -kept the host alive during spikes — but the real fix was right-sizing guests. - -## The result - -Array stop went from *"hangs forever, forced reboot"* to *clean unmount on the -first attempt*, and swap now survives stop/start cycles without manual -intervention. Total fix: two shell scripts in a plugin that was already -installed. The expensive part was knowing **where** in the event sequence to -put them — and that `/proc/swaps` won't tell you the truth about loop-backed -swap. - -If your Unraid array refuses to stop and you have any swap configured — file, -loop, or otherwise — `swapoff -a` from a root shell will release it -immediately, and you've confirmed the diagnosis. Then automate it before the -next reboot wipes your memory of how. - ---- - -*Run a self-hosted stack that needs this kind of debugging — Unraid, Docker, -reverse proxies, NAS migrations? I do this for a living: [website design & -development](https://hoelee.com) is my main work, and self-hosted -infrastructure is where I go deep. Reach me on -[WhatsApp](https://wa.me/60127972969) or -[email](mailto:me@hoelee.com?subject=Unraid%20infrastructure%20help).* -```` - ---- - -## 文章 07 / 11：用 CDP 自动化上传百兆视频到 YouTube：跨越 DevTools 内存墙的 Docker 归档法 -- **Slug**: `uploading-large-videos-to-youtube-via-cdp` -- **英文标题**: Uploading 100MB+ Videos to YouTube via CDP: The Docker Archive Pattern -- **分类**: `devops` | **发布日期**: 2025-08-22 -- **线上地址**: https://blog.hoelee.com/posts/uploading-large-videos-to-youtube-via-cdp/ (EN) | https://blog.hoelee.com/posts/zh/uploading-large-videos-to-youtube-via-cdp/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "用 CDP 自动化上传百兆视频到 YouTube：跨越 DevTools 内存墙的 Docker 归档法" -description: "了解如何使用 Docker Archive API 将大型视频直接送入容器文件系统，再通过 CDP 秒级挂载，彻底避免将百兆文件经由 WebSocket 慢速传输挤爆浏览器。" -pubDate: 2025-08-22 -updatedDate: 2026-10-22 -category: devops -tags: ["docker", "cdp", "playwright", "youtube", "portainer", "automation"] -ogImage: "/og/zh-uploading-large-videos-to-youtube-via-cdp.jpg" -banner: "/banners/uploading-large-videos-to-youtube-via-cdp.jpg" -draft: false ---- - -在前端自动化与爬虫开发中，通过无头浏览器或 Chrome DevTools Protocol（CDP）来提交表单、抓取数据或上传几百 KB 的图片通常十分顺畅。然而，当你试图在自动化流程中将一个 100MB 以上的高清 1080P 视频通过远程 CDP 自动上传到 YouTube 时，常规的浏览器自动化方案几乎一定会崩溃。 - -本文记录这个让远程自动化频繁掉线的内存陷阱，以及如何通过轻量优雅的 Docker 归档模式（Docker Archive Pattern）实现百兆大视频的稳定秒级注入。 - -> **TL;DR** —— 不要把上百兆字节塞进 CDP 的 WebSocket。Docker Engine API 的 `PUT /containers/{id}/archive?path=…`（这里走 Portainer）把文件打成 tar 直接解进 Chrome 容器的文件系统，再用 `DOM.setFileInputFiles` 把**容器内路径**交给网页。千兆局域网下 130 MB 落盘 1.89 秒。这一篇是 [全自动 MV 生成流水线](/posts/zh/how-i-built-an-automated-music-video-pipeline/) 的落盘那一半；渲染层相关的转场与低频抽动排查见 [排查 Remotion 视频抽搐](/posts/zh/debugging-remotion-video-jitter/)。 - ---- - -## 1. 什么时候该用哪种文件上传方案 - -在动手写代码前，先理清四种常见方案的适用边界： - -| 方案 | 适合场景 | 权衡与取舍 | -|---|---|---| -| **Playwright `setInputFiles()`** | 同一台机器上的本地 Chrome 实例 | 挂接系统原生文件选择器。跨远程 WebSocket 传输大文件极易内存溢出或超时。 | -| **CDP `DOM.setFileInputFiles`** | 文件已经存在于 Chrome 宿主机/容器内部 | 耗时仅 2 毫秒，网络零内存开销，但本身不具备跨机器传文件的能力。 | -| **Docker Archive API (`/archive`)** | 把文件注入已有容器，免配 SSH/NFS/卷挂载 | 原生 tar 流式导入，快速自包含，适合带外落盘。 | -| **YouTube Data API v3** | 后台纯无头服务，拥有 Google Cloud 服务凭据 | 支持官方断点续传，但每日免费配额极紧（每次上传消耗 1600 单位，每天最多传 6 个视频），且很多 YouTube Studio 独占设置（如特定多语种 CC 字幕、音频轨道）API 无法完全覆盖。 | - -### 为什么这里选择 CDP 而不是官方 Data API？ -YouTube Data API 默认每天只有 10,000 单位配额，上传一个视频扣除 1,600 单位，一天上传 6 个视频就会触发限流配额耗尽。此外，官方 API 对外挂多语种字幕的授权和上传要求频繁更新的 OAuth2 授权圈子。在已有登录态的自托管 Chrome 容器中通过 CDP 驱动，不仅零配额限制，还能 100% 保持与人工在 YouTube Studio 点击完全一致的行为体验。 - ---- - -## 2. 50MB 远程浏览器协议陷阱 - -在控制本地 Chrome 时，Playwright 或 Puppeteer 使用 `page.setInputFiles("input[type=file]", local_path)`。浏览器直接从当前机器的物理磁盘中读取文件句柄，操作极其平稳。 - -但在自托管私有基础设施或生产部署中，为了让 YouTube、电商等带有登录态的账号 7x24 小时保持在线且不污染日常桌面，我们通常将 Chrome 运行在 Synology NAS 或 Linux 服务器的 Docker 容器中。 - -当我们在外部工作站或 CI 流水线上运行脚本，试图跨网络通过远程 CDP 将视频推给容器里的浏览器时，问题接踵而至： -1. Playwright 会尝试通过 WebSocket 协议将巨大的二进制文件转为 Base64 或分片推流； -2. 庞大的数据帧直接挤爆 WebSocket 协议栈缓冲区，两端内存激增； -3. 远程浏览器随之报出 `Target closed` 崩溃退出，或者由于帧超时直接断开连接。 - -即使你费尽心思调大 WebSocket 参数，跨协议在网络上传输上百兆视频依然极其脆弱。 - ---- - -## 3. 破局思路：本地文件路径直接注入 - -核心思路的转变其实非常直观：**如果视频文件本身就已经保存在容器内部的磁盘里，Chrome 根本不需要走 WebSocket 网络把数据传过来。** - -当 `/config/Downloads/video.mp4` 已经落在容器文件系统内时，CDP 提供了一个原生且完全不消耗协议内存的底层指令： - -```python -cdp = context.new_cdp_session(page) -doc = cdp.send("DOM.getDocument") -file_node = cdp.send("DOM.querySelector", { - "nodeId": doc["root"]["nodeId"], - "selector": "input[type='file']", -}) - -# 耗时仅需 2 毫秒，且在浏览器 WebSocket 上传输 0 字节： -cdp.send("DOM.setFileInputFiles", { - "files": ["/config/Downloads/video.mp4"], - "nodeId": file_node["nodeId"], -}) -``` - -于是，真正的工程问题转变为：**如何无需繁琐的 SSH 密钥或挂载庞大卷，就能将一个 130MB 的文件在 2 秒内送入容器内部？** - ---- - -## 4. Portainer Docker Archive 归档注入法（生产强化版实现） - -每个 Docker 守护进程原生暴露了一个容器归档接口： -`PUT /containers/{id}/archive?path=/path/in/container` - -在 Synology NAS 或 unRaid 环境下，Portainer 会将此 Docker socket API 封装在统一的 API Key 鉴权之后： - -`PUT http://:9000/api/endpoints/2/docker/containers/{container_id}/archive?path=/config/Downloads` - -以下是带有完整异常捕获、入参校验与安全权限的生产级 Python 脚本： - -```python -import io -import os -import tarfile -import urllib.error -import urllib.request - -PORTAINER_KEY = os.environ.get("PORTAINER_API_KEY", "") -PORTAINER_BASE = "http://192.168.1.1:9000/api/endpoints/2/docker" -CONTAINER_ID = "2dc0fb6a11c7" # 目标 Chrome 容器 ID - -def stage_file_to_container(local_path: str, dest_filename: str) -> None: - if not PORTAINER_KEY: - raise ValueError("未检测到 PORTAINER_API_KEY 环境变量") - if not os.path.exists(local_path): - raise FileNotFoundError(f"本地文件不存在: {local_path}") - - size = os.path.getsize(local_path) - tar_stream = io.BytesIO() - - # 在内存中打包 tar（对于拥有 16GB+ RAM 的工作站，100-200MB 媒体文件完全安全可控） - with tarfile.open(fileobj=tar_stream, mode="w") as tar: - with open(local_path, "rb") as f: - ti = tarfile.TarInfo(name=dest_filename) - ti.size = size - # 0o644：文件属主可读写，其他组只读（严禁赋予全局可写 0o666/0o777） - ti.mode = 0o644 - tar.addfile(ti, f) - - tar_stream.seek(0) - data = tar_stream.read() - - endpoint = f"{PORTAINER_BASE}/containers/{CONTAINER_ID}/archive?path=/config/Downloads" - req = urllib.request.Request( - endpoint, - data=data, - headers={ - "X-Api-Key": PORTAINER_KEY, - "Content-Type": "application/x-tar", - }, - method="PUT", - ) - - try: - with urllib.request.urlopen(req, timeout=120) as resp: - if resp.status not in (200, 204): - raise RuntimeError(f"Archive PUT 请求失败，HTTP 状态码: {resp.status}") - print(f"成功将 {dest_filename} ({size / (1024 * 1024):.2f} MB) 注入容器！") - except urllib.error.HTTPError as e: - body = e.read().decode("utf-8", errors="replace") - raise RuntimeError(f"Portainer 接口返回错误 {e.code}: {body}") from e -``` - -### 运维注意事项与安全清单 -1. **客户端 RAM 内存开销**：虽然 Chrome 端彻底消除了内存尖刺，但 `io.BytesIO()` 会在当前 Python 脚本进程中驻留整份 tar 镜像。如果要处理数 GB 的大型视频归档，应先将 `.tar` 写入临时文件，并通过 `requests` 或 `urllib3` 以流式分块形式上传。 -2. **目标目录必须预先存在**：Docker Archive API 是直接解包到 `?path=` 指向的目录中。如果目标容器内部缺失 `/config/Downloads` 目录，接口会直接返回 404/500，而不会自动创建父级目录。 -3. **API Key 与 CDP 端口暴露风险**：Portainer API Key 与未鉴权的 Chrome DevTools 端口（默认 `9222` 或 `9242`）都拥有宿主或容器等效的最高执行权限。切勿将这两个端口直接映射到公网 IP（`0.0.0.0`），务必严格限制在内网隔离桥或套在双向 TLS / Authentik SSO 之后。 - -### 实测性能读数 -在千兆局域网环境下实测： -- **128.85 MB 视频成片（.mp4）**：从发起请求到容器解包就绪，耗时 **1.89 秒**（固态硬盘存储）； -- **Chrome 容器内存占用**：所有浏览器标签页均观察不到额外内存峰值； -- **WebSocket 流量**：DevTools 协议通道 0 字节负载。 - ---- - -## 5. YouTube Studio 上传流程自动化 - -当视频和自定义封面都已安静地躺在容器内部的 `/config/Downloads/` 下，Playwright 只需要通过 CDP 驱动页面向导： - -1. **挂载主视频**：获取主上传框 `input[type='file']`，调用 `DOM.setFileInputFiles` 传入容器视频路径； -2. **填充标题与描述**：定位 contenteditable 输入区域，自动输入预生成的 SEO 标题与详细简介； -3. **挂载封面缩略图**：通过 `DOM.getDocument` 重新拉取动态注入的 DOM 树，获取 `input[type='file']#file-loader` 节点，挂载 `/config/Downloads/cover.jpg`； -4. **受众选择**：勾选 `VIDEO_MADE_FOR_KIDS_NOT_MFK`； -5. **跳过向导检查**：连续点击 `#next-button` 进入发布选项； -6. **公开可见性与发布**：选中 `PUBLIC`，点击 `#done-button` 即可完成发布。 - -因为文件在容器磁盘上本地可读，YouTube Studio 以最饱满的网络带宽直推谷歌服务器，全程无一次断流或卡顿。 - ---- - -## 6. 核心总结 - -- **严禁在 CDP 的 WebSocket 通道传输大型文件**：远程浏览器不是为了在内存中缓冲几百兆媒体流而设计的； -- **带外落盘先行**：使用 Docker Archive PUT 接口，利用 Docker 原生机制以极高速度直接将文件推入容器文件系统； -- **容器内部路径秒挂载**：通过 CDP 的 `DOM.setFileInputFiles` 直接指定容器路径，实现零内存损耗、零失败率的文件绑定。 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Uploading 100MB+ Videos to YouTube via CDP: The Docker Archive Pattern" -description: "Learn how to stage large video files inside a Dockerized Chrome container using the Docker Archive API, then attach them through CDP without sending the entire file over the browser WebSocket." -pubDate: 2025-08-22 -updatedDate: 2026-10-22 -category: devops -tags: ["docker", "cdp", "playwright", "youtube", "portainer", "automation"] -ogImage: "/og/uploading-large-videos-to-youtube-via-cdp.jpg" -banner: "/banners/uploading-large-videos-to-youtube-via-cdp.jpg" -draft: false ---- - -Automating web uploads via headless browser or Chrome DevTools Protocol (CDP) works seamlessly for small JSON payloads, forms, and occasional screenshots. But when you attempt to upload a 100MB+ 1080p video file to YouTube through a remote browser session, standard browser automation approaches quickly collapse. - -Here is the exact trap that breaks remote CDP uploads, why browser memory limits fight you, and the lightweight Docker Archive pattern that makes large-file uploads rock-solid. - -> **TL;DR** — Do not push megabytes through the CDP WebSocket. `PUT /containers/{id}/archive?path=…` on the Docker Engine API (here via Portainer) tars the file straight into the Chrome container's filesystem, then `DOM.setFileInputFiles` hands the *container path* to the page input. 130 MB staged in 1.89 s on a gigabit LAN. This is the staging half of our [music-video pipeline](/posts/how-i-built-an-automated-music-video-pipeline/); the rendering and visual transition tuning is covered in [debugging Remotion video jitter](/posts/debugging-remotion-video-jitter/). - ---- - -## 1. When to Use Which File-Upload Strategy - -Before jumping into implementation, it helps to understand where each upload technique belongs: - -| Method | Best For | Trade-offs | -|---|---|---| -| **Playwright `setInputFiles()`** | Local Chrome instances on the same host | Native OS picker hook. Fails or spikes memory over remote WebSockets. | -| **CDP `DOM.setFileInputFiles`** | Files already on the Chrome host's filesystem | Instantaneous (2ms), zero network memory overhead, but cannot move files across machines. | -| **Docker Archive API (`/archive`)** | Staging files into an existing remote container without SSH/NFS | Direct tar stream to Docker daemon; fast and self-contained. | -| **YouTube Data API v3** | Headless backend services with Google Cloud credentials | Direct resumable uploads, but daily quota limits are tight (1600 units per upload = 6 videos/day max on free tier) and cannot access newer YouTube Studio features (custom audio tracks, specific CC localization workflows). | - -### Why CDP Over YouTube Data API? -YouTube's official Data API imposes a 10,000-unit daily default quota limit. A single video upload consumes 1,600 units, meaning you hit quota exhaustion after just 6 videos a day. Furthermore, uploading multi-language `.srt` tracks and custom video localization through the Data API requires complex OAuth2 consent scopes that expire frequently. - -By driving an already-authenticated Chrome instance in Docker via CDP, we operate under standard browser sessions with zero quota walls. - ---- - -## 2. The 50MB Remote Browser Pitfall - -When controlling a local Chrome instance, Playwright or Puppeteer attaches files using `page.setInputFiles("input[type=file]", local_path)`. The browser reads the file from your local disk and passes the handle directly to the OS file picker. - -However, in a self-hosted on-premise or production setup, you frequently run Chrome inside an isolated Docker container on a NAS or server to keep your authenticated session alive 24/7 without exposing your daily desktop. - -When your automation script runs on a different workstation or runner and tries to push a video over a remote CDP connection: -1. Playwright attempts to base64-encode or stream the file buffer over the WebSocket protocol. -2. The WebSocket frame exceeds protocol buffers or consumes massive memory on both ends. -3. The remote browser crashes with `Target closed`, or CDP drops the socket due to frame size timeouts. - -Even if you increase the WebSocket chunk size, pushing a 130MB file over CDP is excruciatingly slow and fragile. - ---- - -## 3. The Reframe: Local File Injection - -The key realization is simple: **Chrome inside the container doesn't need the file pushed over the wire if the file already exists on its internal filesystem.** - -If the file `/config/Downloads/video.mp4` already sits inside the container, CDP provides a native, zero-memory command that requires no file transfer at all: - -```python -cdp = context.new_cdp_session(page) -doc = cdp.send("DOM.getDocument") -file_node = cdp.send("DOM.querySelector", { - "nodeId": doc["root"]["nodeId"], - "selector": "input[type='file']", -}) - -# Takes 2 milliseconds and transfers ZERO bytes over the browser socket: -cdp.send("DOM.setFileInputFiles", { - "files": ["/config/Downloads/video.mp4"], - "nodeId": file_node["nodeId"], -}) -``` - -The real question becomes: **How do you get a 130MB file into the container filesystem without SSH keys, volume mounts, or slow SFTP?** - ---- - -## 4. The Portainer Docker Archive API (Hardened Implementation) - -Every Docker daemon exposes a container archive endpoint: -`PUT /containers/{id}/archive?path=/path/in/container` - -If you manage your Docker host with Portainer (standard in Synology and unRaid homelabs), Portainer proxies this Docker socket endpoint behind an API key: - -`PUT http://:9000/api/endpoints/2/docker/containers/{container_id}/archive?path=/config/Downloads` - -Here is a tested, production-ready staging script with proper error handling, parameter validation, and secure file modes: - -```python -import io -import os -import tarfile -import urllib.error -import urllib.request - -PORTAINER_KEY = os.environ.get("PORTAINER_API_KEY", "") -PORTAINER_BASE = "http://192.168.1.1:9000/api/endpoints/2/docker" -CONTAINER_ID = "2dc0fb6a11c7" # Target Chrome container ID - -def stage_file_to_container(local_path: str, dest_filename: str) -> None: - if not PORTAINER_KEY: - raise ValueError("PORTAINER_API_KEY environment variable is not set") - if not os.path.exists(local_path): - raise FileNotFoundError(f"Local source file not found: {local_path}") - - size = os.path.getsize(local_path) - tar_stream = io.BytesIO() - - # Build tar in memory (safe for 100-200MB payloads on workstations with 16GB+ RAM) - with tarfile.open(fileobj=tar_stream, mode="w") as tar: - with open(local_path, "rb") as f: - ti = tarfile.TarInfo(name=dest_filename) - ti.size = size - # 0o644: owner read/write, group/others read-only (avoid world-writable 0o666/0o777) - ti.mode = 0o644 - tar.addfile(ti, f) - - tar_stream.seek(0) - data = tar_stream.read() - - endpoint = f"{PORTAINER_BASE}/containers/{CONTAINER_ID}/archive?path=/config/Downloads" - req = urllib.request.Request( - endpoint, - data=data, - headers={ - "X-Api-Key": PORTAINER_KEY, - "Content-Type": "application/x-tar", - }, - method="PUT", - ) - - try: - with urllib.request.urlopen(req, timeout=120) as resp: - if resp.status not in (200, 204): - raise RuntimeError(f"Archive PUT failed with HTTP {resp.status}") - print(f"Staged {dest_filename} ({size / (1024 * 1024):.2f} MB) in container successfully.") - except urllib.error.HTTPError as e: - body = e.read().decode("utf-8", errors="replace") - raise RuntimeError(f"Portainer Archive API error {e.code}: {body}") from e -``` - -### Operational Caveats & Security Checklist -1. **Client RAM Footprint**: While Chrome experiences zero memory pressure, `io.BytesIO()` buffers the 130MB tar archive in the Python process memory. For multi-gigabyte video dumps, write the `.tar` to a temporary file and stream it using `requests` or `urllib3` chunked uploads. -2. **Pre-existing Destination Directory**: The Docker Archive API extracts *into* the directory specified by `?path=`. If `/config/Downloads` does not exist in the container image, the API returns HTTP 404/500 rather than creating the parent folders. -3. **API Key & CDP Exposure**: Both the Portainer API key and the unauthenticated Chrome DevTools port (default `9222` or `9242`) grant root-level execution capability. Never bind these ports to public IPs (`0.0.0.0`). Keep them strictly on an isolated internal bridge or behind mutual-TLS / Authentik SSO. - -### Measured Performance -On a local 1 Gbps LAN: -- **128.85 MB video file (`.mp4`)**: Transferred and extracted inside the container in **1.89 seconds** (single run, warm container, SSD target on Synology NAS). -- **Chrome container memory consumption**: 0 MB RAM spike across all browser tabs. -- **WebSocket overhead**: 0 bytes transferred over DevTools Protocol. - ---- - -## 5. YouTube Studio Upload Flow Automation - -Once the video and custom thumbnail are staged in `/config/Downloads/`, the Playwright script navigates to `https://studio.youtube.com/`, triggers the upload modal, and executes the wizard: - -1. **Attach Video**: Call `DOM.setFileInputFiles` on the primary `input[type='file']`. -2. **Title & Description**: Click `div[contenteditable="true"][aria-label*="title"]` and type the title; use `.fill()` on the description area. -3. **Attach Thumbnail**: Refresh the DOM with `DOM.getDocument` to discover the newly mounted `input[type='file']#file-loader` and inject `/config/Downloads/cover.jpg`. -4. **Audience Setting**: Check `VIDEO_MADE_FOR_KIDS_NOT_MFK`. -5. **Wizard Steps**: Click `#next-button` through Elements and Checks. -6. **Visibility**: Select `PUBLIC` and click `#done-button`. - -Because the file was already on disk inside the container, YouTube Studio reads the stream locally, achieving maximum upload bandwidth without dropping a single frame. - ---- - -## 6. Summary - -- **Never stream large binaries over WebSocket CDP**. Remote browsers are not designed to buffer multi-hundred-megabyte payloads in memory. -- **Stage files out-of-band**: Use Docker Archive PUT to drop files onto the container filesystem in seconds. -- **Inject via filesystem path**: Call `DOM.setFileInputFiles` with the container path for instantaneous, zero-overhead file attachments. -```` - ---- - -## 文章 08 / 11：如何在押注前验证一个开源依赖：我用的六项检查 -- **Slug**: `vetting-an-open-source-dependency-before-you-bet-on-it` -- **英文标题**: How I Vet an Open-Source Dependency Before Betting On It -- **分类**: `devops` | **发布日期**: 2026-09-29 -- **线上地址**: https://blog.hoelee.com/posts/vetting-an-open-source-dependency-before-you-bet-on-it/ (EN) | https://blog.hoelee.com/posts/zh/vetting-an-open-source-dependency-before-you-bet-on-it/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "如何在押注前验证一个开源依赖：我用的六项检查" -description: "六项检查，专门抓出「文档说得漂亮、代码里其实没有」的仓库：用 grep 找功能、读文档头部标注的目标分支、看清付费档位门槛，以及收款通道在不在你的国家能用。" -pubDate: 2026-09-29 -category: devops -blueprint: sme-digital-suite -blueprintLayer: L4-managed-ops-handover -tags: ["open-source", "due-diligence", "oidc", "self-hosting", "docker"] -ogImage: "/og/zh-vetting-an-open-source-dependency-before-you-bet-on-it.jpg" -banner: "/banners/vetting-an-open-source-dependency-before-you-bet-on-it.jpg" -draft: false ---- - -有人给我一份 1306 行的伙伴系统架构规格书。读起来像一份计划：前后一致、交叉引用、语气笃定。它指定一个自托管的开源引擎作为归因与佣金层，并且假定了那个引擎的三件事——它支持 OIDC，所以我现有的身份提供方能登录伙伴；它的佣金规则覆盖业务需要的五种形态；它的白标功能已经上线。 - -十分钟的 `curl` 加 `grep` 之后，三件里有两件是假的，第三件只对了一半。 - -这没什么稀奇。README 描述的是**产品**，仓库描述的是**软件**。这两者之间的落差，正是集成项目死掉的地方——而且在写任何代码之前，测量这个落差很便宜。 - -## 问题，写成我真的会去搜的那句话 - -**我怎么知道一个开源项目真的做了它文档宣称的事，然后才在它上面开始建东西？** - -规格书是**关于别的软件**的一种主张。规格书越漂亮，它的假设就越像事实——这一份引用了来源、定义了稳定 ID、还画了一张「真源矩阵」。它唯一没做的事，就是核对链条下游那个引擎到底有没有实现被指派给它的功能。所以在安装任何东西之前，我去找那几个具体功能。 - -## 我一开始试的，以及为什么不够 - -**一、把 README 从头读到尾。** 它把流水线卖得很好：`click → identity → event → attribution → commission → payout`。但它一次都没说这个应用讲哪种认证协议。对一个你**必须要有**的功能来说，沉默不是中立——只是当页面其他部分都很有说服力时，一个「没被提到」很容易被眼睛滑过去。 - -**二、上网搜。** 好几条查询都没返回能用的结果。这个项目才几个月大，SEO 面很薄。对软件来说这没关系——源码**就是**文档。只是意味着阅读必须在仓库里进行，而不是在博客文章里。 - -**三、看到一份 78 KB 的功能设计文档，就假定功能已上线。** 它标着「FINAL, post-review」，列了三位评审，还带着日期。我把它当成了证据。这是三个推断里最糟的一个，也是最容易核对的一个。 - -## 真正定案的六项检查 - -### 1. 先看仓库的生命体征，再看任何功能宣称 - -```bash -curl -s -H "User-Agent: hermes" https://api.github.com/repos// \ - | grep -E '"created_at"|"pushed_at"|"stargazers_count"|"forks_count"|"open_issues_count"|"archived"|"spdx_id"' -``` - -返回的是：MIT、创建于 2026-04-23、最后推送 2026-09-10、8 个 star、3 个 fork、15 个 open issue、未归档。翻译过来就是：**真实、还在动、但非常小**——单一厂商，而且它同时卖托管档。现在我在开始评它的功能之前，先知道了我愿意把多少生意压在它身上。 - -### 2. 用 grep 去找功能，而不是用读的 - -```bash -grep -in 'oidc\|sso\|saml' README.md ARCHITECTURE.md docs/*.md | wc -l -``` - -零命中。这个应用用的是它自己的 magic-link 会话和按身份签发的 API key；它根本不讲 OIDC。就这一行输出，砍掉了计划里的一个阶段——「把门户挂到我们 SSO 后面」——否则这一步会在几周后才浮现，而且是在集成到一半、伙伴已经被邀请进来的时候。**用读的方式去找一个找不到的功能，就是你失去一个下午的方式；grep 让「不存在」这件事变得响亮。** - -### 3. 读功能文档最上方那行「目标分支」 - -```bash -head -5 docs/white-label-custom-domains.md -``` - -`Target branch: multi-tenant`。这份文档描述的是**另一个分支**上的设计，不是你默认会安装的那个分支——它写的是在途工作，所以功能不在你会装下来的代码里。一份有评审、有日期、看起来已经完成的设计文档，完全可能描述着尚未上线的东西。这不是不诚实，设计文档本来就是这个样子。 - -### 4. 读数据模型，不要读功能清单 - -佣金那一节只列了两种规则类型——`percent`（带续期标记）与 `fixed`。业务规格书列了五种。这个落差不是 bug，而是我必须在**配置层**做的建模工作，不能指望产品自带： - -| 规格书假设 | 引擎实际有的 | -|---|---| -| 首单奖励 | 对首个事件用一条 `fixed` 规则 | -| 按事件区分（`lead` vs `invoice_paid`） | 一个 campaign 一条规则——所以：两个 campaign | -| 续期佣金 | `percent` + `recurring: true` | -| 百分比 | `percent` ✅ | -| 固定金额 | `fixed` ✅ | - -两种规则原语，五种行为——能做，但前提是你**一开始**就照这个设计，而不是在实现途中才发现。 - -### 5. 看清你需要的那项功能被锁在哪一档 - -同一个技术栈里还有个数据库工具，它的版本对照表很直白： - -| 我需要的功能 | Community（免费） | Enterprise（付费） | -|---|---|---| -| API token，好让我自动化 | ✅ | ✅ | -| SSO / OIDC | ❌ | ✅ | -| 页面设计器、隐藏品牌、双因素 | ❌ | ✅ | - -我想给「伙伴看到的那个门」用的东西（SSO）在商业门后面；而我想用来做自动化的东西（API token）是免费的。在围绕它做设计**之前**，先弄清每项需求落在墙的哪一侧。 - -### 6. 核对收款通道在你的国家能不能用 - -它的打款模型是 `stripe_connect | manual`，而自托管模式被写成「operator manages out-of-band」——也就是钱由你自己搬。Stripe Connect 不是马来西亚企业付钱给伙伴的方式，所以就是这一行决定了整个部署形态：自托管模式、手工打款、事件由我自己的订单系统推送过去，而不是依赖它内置的支付集成。这一行从来不会出现在营销页上，而它比任何功能都重要。 - -### 然后：在你需要之前，先问「兜底方案」 - -「如果我需要的功能不在，怎么办？」针对缺失的 SSO，答案是：不要把登录做进应用里，把登录放到应用**前面**。authentik 的 proxy provider 存在的意义就是保护「不支持 OIDC、SAML 或 LDAP 等原生认证协议的应用」，并且向上游注入 `X-authentik-username` 和 `X-authentik-groups` 头，于是反向代理就能承担准入判断。这就把「不支持 SSO」从一条否决理由，变成了一个下午的配置工作。 - -## 如果重来，我会怎么做 - -- **先写需求清单，再去找它。** 五行——**我**需要的功能——然后就搜这几个。从 README 的功能列表出发，等于用厂商的考卷给软件打分。 -- **把「文档存在」和「功能上线」当成两件不同的事。** 去读那行目标分支。 -- **用 grep，不要用读的。** 证明「有」很便宜；你要猎的是「没有」。 -- **在决定之前先定兜底**，这样一个缺失的功能就只是一次设计调整，而不是一场意外。 -- **把 commit 钉住。** 这个项目的 README 自己写着 API「stable but unversioned」，而且仓库里没有任何 OpenAPI 描述。没有可以编译期对齐的契约，那么 commit hash **就是**契约——我用到的端点要记进我自己的文档，升级前跑一遍回归。 - -## 结果 - -六个假设在大概十分钟的 `curl` 和 `grep` 里被修正，而当时一行集成代码都还没写。项目依然可行——只是换了形状：没有 OIDC、手工打款、钉住 commit，以及一份不再依赖某个未合并分支上功能的计划。 - -那份规格书乐观并没有错。它错在**没被验证过**——而这是整个项目里最便宜的修补项。 - ---- - -**你正在自托管技术栈上建东西，想在集成之前找人看一眼那些功能到底在不在？** 这正是我在做的复核工作。 - -[WhatsApp +60 12-797 2969](https://wa.me/60127972969) · -[me@hoelee.com](mailto:me@hoelee.com?subject=Open-source%20dependency%20review) · -[hoelee.com](https://hoelee.com) -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "How I Vet an Open-Source Dependency Before Betting On It" -description: "Six checks that catch a repo which doesn't do what its docs claim — grep for the feature, read the doc's target branch, check the tier gate and the payment rail." -pubDate: 2026-09-29 -category: devops -blueprint: sme-digital-suite -blueprintLayer: L4-managed-ops-handover -tags: ["open-source", "due-diligence", "oidc", "self-hosting", "docker"] -ogImage: "/og/vetting-an-open-source-dependency-before-you-bet-on-it.jpg" -banner: "/banners/vetting-an-open-source-dependency-before-you-bet-on-it.jpg" -draft: false ---- - -I was handed a 1,300-line architecture spec for a partner program. It read like a -plan: consistent, cross-referenced, confident. It named a self-hosted -open-source engine as the attribution and commission layer, and it assumed three -things about that engine — that it spoke OIDC so my existing identity provider -could log partners in, that its commission rules covered the five shapes the -business needed, and that its white-label feature was shipped. - -Ten minutes of `curl` and `grep` later, two of those were false and the third was -half-true. None of that is unusual. The README describes the *product*; the -repository describes the *software*. The gap between those two is exactly where -integration projects die — and it is cheap to measure before you write any code. - -## The problem, phrased as the question I'd actually search for - -**How do I know an open-source project does what its documentation claims before -I build on it?** - -A spec is a claim *about other software*. The more polished the spec, the more its -assumptions read like facts — this one cited sources, defined stable IDs, and had -a source-of-truth matrix. What it never did was check whether the engine down the -chain implemented the features being assigned to it. So before installing -anything, I went looking for those specific features. - -## What I tried first, and why it wasn't enough - -**1. Read the README end to end.** It sells the pipeline well: -`click → identity → event → attribution → commission → payout`. It never once -says which authentication protocol the app speaks. For a feature you *need*, -silence isn't neutral — but an absent mention is easy to skim past when the rest -of the page is convincing. - -**2. Searched the web.** Several of my queries returned nothing usable. The -project is a few months old and its SEO surface is thin. For software that's -fine — the source *is* the documentation. It just means the reading has to happen -in the repo, not in a blog post. - -**3. Found a 78 KB feature design doc and assumed the feature shipped.** It was -labelled "FINAL, post-review", listed three reviewers, and carried a date. I took -that as proof. That was the worst inference of the three, and the easiest to check. - -## The six checks that actually settled it - -### 1. Read the repo's vital signs before any feature claim - -```bash -curl -s -H "User-Agent: hermes" https://api.github.com/repos// \ - | grep -E '"created_at"|"pushed_at"|"stargazers_count"|"forks_count"|"open_issues_count"|"archived"|"spdx_id"' -``` - -What came back: MIT, created 2026-04-23, last push 2026-09-10, 8 stars, 3 forks, -15 open issues, not archived. Read the same as: **real, still moving, and very -small** — one vendor, which also sells a hosted tier. Now I know how much of my -business I want resting on it *before* I start grading its features. - -### 2. Grep for the feature instead of reading for it - -```bash -grep -in 'oidc\|sso\|saml' README.md ARCHITECTURE.md docs/*.md | wc -l -``` - -Zero hits. The app authenticates with its own magic-link sessions and -per-persona API keys; it does not speak OIDC at all. That one line of output -killed a phase of the plan — "put the portal behind our SSO provider" — which -would otherwise have surfaced weeks later, mid-integration, with a partner -already invited. Reading for a feature you can't find is how you lose an -afternoon; grepping makes the absence loud. - -### 3. Read the target branch line at the top of a feature doc - -```bash -head -5 docs/white-label-custom-domains.md -``` - -`Target branch: multi-tenant`. The doc is a design for a branch that is not the -default branch — it describes work in flight, so the feature is not in the code -you would install. A finished-looking document with reviewers and dates can -document something unshipped. That isn't dishonesty; it's what design docs are. - -### 4. Read the data model, not the feature list - -The commission section named exactly two rule types — `percent` (with a recurring -flag) and `fixed`. The business spec had enumerated five. That gap isn't a bug; -it's a modelling job I'd have to do in configuration instead of expecting from -the product: - -| Spec assumed | What the engine actually has | -|---|---| -| first-order bonus | a `fixed` rule on the first event | -| event-specific rules (`lead` vs `invoice_paid`) | one rule per campaign — so: two campaigns | -| recurring commission | `percent` + `recurring: true` | -| percentage | `percent` ✅ | -| fixed | `fixed` ✅ | - -Two rule primitives, five behaviours — workable, but only if you design for it up -front instead of discovering it during implementation. - -### 5. Check which paid tier gates the feature you need - -For the database tool in the same stack, the edition table was blunt: - -| Feature I needed | Community (free) | Enterprise (paid) | -|---|---|---| -| API tokens, so I can automate it | ✅ | ✅ | -| SSO / OIDC | ❌ | ✅ | -| Page designer, hide branding, 2FA | ❌ | ✅ | - -The thing I wanted for the partner-facing door (SSO) sits behind the commercial -gate, while the thing I wanted for automation (an API token) is free. Knowing -which side of the wall each requirement falls on changes the design — before -you've designed around it. - -### 6. Check that the money rail works in your country - -The payout model was `stripe_connect | manual`, and self-hosted mode was -documented as "operator manages out-of-band" — i.e. you move the money yourself. -Stripe Connect is not how Malaysian businesses pay partners, so that one line -decided the entire deployment shape: self-host mode, manual payouts, and event -ingestion pushed from my own order system rather than leaning on the provider's -payment integration. That line is never on a marketing page, and it mattered more -than any feature. - -### Then: ask the fallback question before you need it - -"What if the feature I need isn't there?" For the missing SSO the answer was: -don't put login in the app, put it in front of the app. authentik's proxy -provider exists to protect "applications that do not support native -authentication protocols such as OIDC, SAML, or LDAP", and it injects -`X-authentik-username` and `X-authentik-groups` headers upstream, so the reverse -proxy can do the gating. That turned "no SSO support" from a disqualification -into an afternoon of config. - -## What I'd do differently - -- **Write the requirement list first, then go look for it.** Five lines — the - features *I* need — and search for those. Starting from the README's feature - list means grading the software on the vendor's exam. -- **Treat "a doc exists" as different from "a feature ships".** Read the branch - line. -- **Grep, don't read.** Presence is cheap to prove; absence is the thing you're - hunting. -- **Decide the fallback before the decision**, so a missing feature becomes a - design change rather than a surprise. -- **Pin the commit.** This project's README says its API is "stable but - unversioned" and there is no OpenAPI description anywhere in the tree. There's - no contract to compile against, so the commit hash *is* the contract — the - endpoints I use go into my own docs, and upgrades get a regression pass. - -## The result - -Six assumptions corrected in about ten minutes of `curl` and `grep`, before a -single line of integration code existed. The project stayed viable — but -reshaped: no OIDC, manual payouts, a pinned commit, and a plan that no longer -depends on a feature sitting on an unmerged branch. - -The spec wasn't wrong to be optimistic. It was wrong to be unverified, and that's -the cheapest thing on the whole project to fix. - ---- - -**Building on a self-hosted stack, and want a second pair of eyes on whether -those features will actually be there when you integrate?** That's the kind of -review I do. - -[WhatsApp +60 12-797 2969](https://wa.me/60127972969) · -[me@hoelee.com](mailto:me@hoelee.com?subject=Open-source%20dependency%20review) · -[hoelee.com](https://hoelee.com) -```` - ---- - -## 文章 09 / 11：SMART 显示健康，RAID 却在悄悄损坏数据 -- **Slug**: `when-smart-says-healthy-but-your-raid-is-corrupting-data` -- **英文标题**: When SMART Says Healthy and Your RAID Is Silently Corrupting Data -- **分类**: `devops` | **发布日期**: 2026-09-16 -- **线上地址**: https://blog.hoelee.com/posts/when-smart-says-healthy-but-your-raid-is-corrupting-data/ (EN) | https://blog.hoelee.com/posts/zh/when-smart-says-healthy-but-your-raid-is-corrupting-data/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "SMART 显示健康，RAID 却在悄悄损坏数据" -description: "两块看起来完全健康的 NVMe 组成的 btrfs RAID1 阵列，静默地把 Windows 虚拟机喂成了全零字节。我靠校验和取证抓住它——但根因（M.2 转接卡链路故障 vs 批次缺陷）至今仍未定论，现在两块盘测试都干净。" -pubDate: 2026-09-16 -category: devops -blueprint: self-hosted-infrastructure -blueprintLayer: L1-compute-storage-network -tags: [unraid, btrfs, nvme, raid, storage, data-loss, troubleshooting] -ogImage: /og/zh-when-smart-says-healthy-but-your-raid-is-corrupting-data.jpg -banner: /banners/when-smart-says-healthy-but-your-raid-is-corrupting-data.jpg ---- - -所有存储指南都告诉你同一件事：相信 SMART、检查事件日志、相信你的文件系统。我的硬件不这么想。2026 年 9 月，一个 btrfs RAID1 阵列——两块企业级 NVMe 盘镜像，我用每个能想到的工具查它都说**「健康」**——却在悄悄地把我的 Windows 11 虚拟机喂成全零字节。不是看得见的故障，是静默损坏。 - -下面是完整故事：损坏是怎么浮现的、哪个校验和出卖了它——以及为什么最终定论如今变得不确定。 - -> **更新（2026-09-16）：根因至今未定论。** 下面的「两块盘都换」是我当时的判断，但后来一次不相关的故障让我不得不重新审视。现在两块 NVMe 测试都干净，而最强的头号嫌疑是两块盘共用的那块 **M.2 转接卡**——转接卡上的链路层故障，恰好会复现我当初归咎于硬盘的那种「两块盘关联地吐出全零」的损坏。SMART 对链路故障完全无感，所以无法排除它。我在一篇配套文章里写了那次警醒（一颗「死掉的」SATA SSD 其实只是坏线）。先读下面的原始分析，再看更正——然后还有**续集**：两周后同一个全零指纹出现在**完全不同的硬盘堆栈**上，这次终于挖出了根因。见[《换了不同的硬盘，同样的损坏又来了——真凶是 TRIM，不是 SSD》](/posts/zh/the-cause-was-trim-not-the-ssds/)。 - -## 更新：为什么我不再怪两块盘 - -这篇文章发出后，有两件事改变了我的想法。 - -**1. 现在两块盘测试都是干净的。** 在当前条件下，两块 PM9A3 读写都正常。如果它们带着 9 月还在活跃损坏数据的批次级缺陷，应该会挂掉一轮持续的写入+校验（`verify=crc32c`）泡测——但它们没有。 - -**2. 链路故障才是真正的教训。** 就在这次事故之后，我在另一颗盘上撞见了**完全相同**的故障签名（一颗 Seagate IronWolf 110 狂报 `WRITE FPDMA QUEUED` 超时、`mkfs` 失败、然后静默——教科书般的「死盘」）。结果是坏在 **SATA 数据线**上。一条不稳定的链路会复现下面我归咎于硬件的每一个症状：在算出 CRC 之前就已丢掉的超时 I/O、真的丢掉的页面，以及——如果两个 RAID1 成员共用同一块有问题的 **M.2 转接卡**或插槽——两块盘关联地损坏。这是唯一一个不需要任何硬盘缺陷、却能解释「两个成员用同一种方式撒谎」的答案。 - -所以我没法再诚实地断言「盘有问题」。它们可能一直好好的，损坏可能来自它们共同坐着的 **M.2 转接卡**。注意：**这并不代表这次事故是虚惊一场**——真实数据确实读不回来过，无论根因是什么，我都把它移走、迁移数据、并逐字节验证过。但正确的教训是「先排除链路，再给盘判死刑」，而不是「同批次盘不可信」。 - -完整的链路对盘排查，请看 **["我差点把这颗 SSD 送修——真正坏的是 SATA 数据线"](/posts/zh/that-dying-ssd-was-just-a-bad-sata-cable/)**。 - -## 原始分析（保留供透明考量） - -下面是我*当时*得出的结论。这是当时证据下最站得住的解读，而且可能仍然是对的——只是我无法排除别的可能——但请把它当作事故报告来读，而不是最终定论。 - -## 为什么这件事重要 - -损坏状态的磁盘是最糟的存储故障，因为它最容易漏掉： - -- **SMART 看起来完美。** 零媒体错误、几乎满余量，而且是能扛企业级写入负载的盘。 -- **阵列有冗余。** btrfs RAID1 把每个块镜像到两块盘上。当**两个成员**用同一种方式撒谎时，冗余毫无意义。 -- **故障是静默的。** 数组状态没有红色警告、没有硬盘报错、没有硬件事件——只有虚拟机读回垃圾，直到备份恢复成了唯一的选择。 - -如果你的数据在一切看起来都正常的时候悄悄坏掉，「备份」就不是偏执——它是全部。这次事故让我明白：一条**可测试、可验证的恢复**胜过任何阵列状态屏幕。 - -## 环境 - -- **unRaid 7.x**，Intel i9-13900K，Windows 11 虚拟机是主力工作站（GPU 直通）。 -- **缓存池：** 两块 **Samsung PM9A3 3.84TB** NVMe 组 **btrfs RAID1**。而且两块是**同一生产批次**（同一个控制器/固件版本 `GDC7702Q`）。 -- Windows 虚拟机的虚拟磁盘（`vdisk1.img`、`vdisk2.img`）就在这个池的 `domains/` 共享里，另外还有 **Docker 数据根目录**。 -- 虚拟磁盘**没有**设 `NOCOW`——所以每次读取都会被 btrfs 校验和验证。也正因为这个，我最后才抓得住它。 - -## 症状 - -我开始发现缓存读取慢得离谱，主机命令也卡住：`cat`、读 `/proc/loadavg`、甚至 `docker image ls` 都在超时。那块坏 NVMe 正以 **45–75 MB/s** 被狂读，每次读都塞满整台主机的 I/O。本该几秒跑完的 `docker system prune` 直接挂死。 - -然后真正的麻烦来了：Windows 虚拟机开始狂报读错误，回来的是**错的**数据——满文件的全零字节、guest 里的校验和失败，最后主力 VM 直接引导不了。 - -## 取证：`csum 0x8941f998` - -btrfs 在读校验失败时会往内核日志记一条校验和不匹配。同一个值反复出现： - -```text -btrfs: checksum verify failed on wanted 0x8941f998 found -``` - -这个特定值是最重要的线索。**`0x8941f998` 就是一整块零字节的 CRC32C。** 它每次都出现、跨越互不相关的文件和 inode，绝不是巧合。正在发生的不是随机比特腐烂，而是盘**稳定地返回全零块**，而这些零填充的读取正是 btrfs 校验失败的对象： - -- 验证一下：零填充块的 `crc32c` 就等于 `0x8941f998`。 -- 同一个值反复出现 → 读取是**确定性**地吐出空数据，不是偶尔翻几个比特。 - -两块虚拟磁盘都中招（`vdisk1.img` inode 2358574、`vdisk2.img` inode 2362817），正好对上「存储层在把零当成被请求的数据返回」的图景。 - -## 破绽：永远不生效的「自愈」 - -btrfs RAID1 本应自愈：一个成员坏块、从镜像重建、写回好数据。但我一遍遍 scrub、重读同一块（仍然全零的）数据后发现，**重写根本不落地**。每次重建「成功」后，立刻重读同一 extent 又吐出零。 - -这是关键区别： - -- 一块**只读不出来**的盘是读问题——自愈应该能从镜像救回来。 -- 一块**连回调也无脑吞垃圾存进去**的盘是**写或控制器**问题——镜像救不了你，因为两条路径都对同一个错误答案达成一致。 - -当自愈重写一直不生效，你就不在「坏一个扇区」的范畴了，而是「这块盘在写入上撒谎」——RAID 冗余救不了。 - -## 为什么两块盘都得换（我当时的推理） - -> 这是当初「都换掉」决定的依据。动手前先读上面更新里的更正——一块共用的 M.2 转接卡链路故障，在无需任何硬盘缺陷的情况下也能对上这份证据。 - -数据指向的是**同批次盘的硬件缺陷**，而不是磨损或一次性的偶发事件： - -- SMART 干净（0 媒体错误、几乎满余量，`nvme0` 大约写了 586 TiB——对能扛企业级负载的盘来说算重负载但仍正常）。 -- 两块盘**同批次、同控制器/固件**（`GDC7702Q`）。 -- 损坏以**相关的方式**同时命中两个 RAID1 成员——镜像「死一块也能幸存」的整个意义，因为两块盘正以同一种方式一起死掉而完全失效。 - -面对批次级的关联缺陷，留下那块「看起来健康」的盘当幸存成员是假省心。**我从备份恢复了 Windows 虚拟机，然后把两块盘都换掉。** - -## 我同时做的事：把数据迁出那个池 - -在换盘之前，任何还值得读的数据都得搬出这个坏池——而且要**验证**，不能想当然： - -- **Docker 数据根目录**迁出缓存：`docker` 挪到 `/mnt/user/docker`（在 `disk1` 上，overlay2）。整棵树通过**路径、大小、mtime** 逐文件比对、并与字节对比交叉验证，共 **3,997,569** 个文件。 -- 缓存的 `isos/`、`system/`、`swapfile` 搬到 `disk1`；`isos/`（约 13 GB）用 `diff -rq` 检查 → **EXIT=0**，无差异。 -- 所有搬走的文件都在**删除缓存副本之前**先跟源比对过——没验证过的副本绝不删。 - -因为存在读侧损坏（全零块）和写侧损坏（自愈不落地），那个池里的任何东西，没有镜像、没有校验和、没有新鲜备份可对照，我就一律不信任。 - -## 如果重来我会怎么做 - -1. **在需要它之前就先测试恢复虚拟机备份。** 我当时确认了备份存在，但损坏事故不是该第一次学恢复流程的时候。定时做的测试恢复既能发现备份问题，也能发现静默的存储损坏。 -2. **把同批次的两块盘当成一块盘。** 两块 PM9A3 同批次同固件。对需要长期搭档的缓存池（任何 RAID1）这是关联失效风险——优先选不同批次的盘，或者至少在信任镜像前把这对盘狠狠测一遍。 -3. **主动检测，而不是被动祈祷。** 正是 btrfs 校验和让它在噪声里浮出来。不开 `NOCOW` 意味着文件系统在验证每一次读取；一个带校验的数据池，就是「当场抓住并恢复」和「把损坏文件带出去几个月」之间的分水岭。 -4. **别用 SMART 判断闪存。** 一块看起来健康却照样损坏的 NVMe，事件计数什么也说明不了——固件级的缺陷可以躲过所有 SMART 属性。 - -## 结果 - -一台 Windows 虚拟机从备份恢复、数据安全迁出那个坏池、Docker 数据根目录也搬了家——**外加一条事后再推翻的教训。** 我当时换掉了两块盘，但现在我没法确定它们就是元凶：证据同样支持一块共用的 **M.2 转接卡**链路故障，而且现在两块盘测试都干净。（完整排查见 **["我差点把这颗 SSD 送修——真正坏的是 SATA 数据线"](/posts/zh/that-dying-ssd-was-just-a-bad-sata-cable/)**。） - -事故被控制住，是因为损坏被校验和抓住了、而不是在里面泡了几个星期——这一点无论根因是什么都成立。但更诚实的教训，比「把盘换掉」这句要更广、也更不戏剧化： - -1. **一块「正在损坏」的盘，并不等于盘本身坏了。** 在给硬件判死刑前，先排除链路：重新插拔/更换线材、换到别的口/转接卡/插槽，重跑*相同*的失败操作。如果故障跟着**口/转接卡**走，那就是连接的问题。 -2. **SMART 看不见链路故障。** 零媒体错误和几乎满余量，对一条不稳定的线材或转接卡毫无意义——超时在算出 CRC 之前就把 I/O 丢了。 -3. **同批次盘共用一条通道，无论谁坏都是共享失效风险。** 不管损坏来自盘还是转接卡，一对盘坐同一条故障路径上，就是穿着两个序列号的单点故障。 -4. **给你的数据加校验、并提前测试恢复备份。** 把数星期静默损坏变成一次被控制住的事故的，是这一点——不是我那部分诊断。 - ---- -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "When SMART Says Healthy and Your RAID Is Silently Corrupting Data" -description: "A btrfs RAID1 pool on two healthy-looking NVMe drives quietly corrupted a Windows VM. How I caught it with a checksum forensic — and why the root cause (M.2 riser link fault vs batch defect) is still uncertain, with both drives now testing clean." -pubDate: 2026-09-16 -category: devops -blueprint: self-hosted-infrastructure -blueprintLayer: L1-compute-storage-network -tags: [unraid, btrfs, nvme, raid, storage, data-loss, troubleshooting] -ogImage: /og/when-smart-says-healthy-but-your-raid-is-corrupting-data.jpg -banner: /banners/when-smart-says-healthy-but-your-raid-is-corrupting-data.jpg ---- - -Every storage guide tells you the same thing: trust SMART, check the event -log, and believe your filesystem. My hardware disagreed. In September 2026 a -btrfs RAID1 pool — two mirrored enterprise NVMe drives that said **"healthy"** -in every tool I could run — was quietly feeding my Windows 11 VM all-zero -bytes. Not visibly failing. Silently corrupting. - -This is the full story: how the corruption surfaced, and the checksum that -gave it away. - -> **Update (2026-09-16): the root cause turned out to be uncertain.** The -> "replace both drives" verdict below was my best diagnosis at the time, but a -> later, unrelated failure taught me to question it. Both NVMe drives now test -> clean, and the strongest remaining suspect is the **M.2 riser** both drives -> were plugged into — a link-layer fault on the riser reproduces exactly the -> correlated, all-zero corruption I blamed on the drives. SMART is silent to -> link faults, so it can't rule one out. I describe that wake-up call (a "dead" -> SATA SSD that was just a bad cable) in a companion post. Read the original -> analysis, then the correction — and then the *sequel*: two weeks later the -> same all-zero fingerprint appeared on a **completely different drive stack**, -> and this time the root cause surfaced. See [The Corruption Came Back on -> Different Drives — the Cause Was TRIM, Not the SSDs](/posts/the-cause-was-trim-not-the-ssds/). - -## Update: why I no longer blame the drives - -Two things changed my mind after this post went up. - -**1. The drives test clean now.** Under current conditions both PM9A3s read and -write correctly. If they carried a batch-level defect that was actively -corrupting in September, I'd expect them to fail a sustained write+verify -(`verify=crc32c`) soak — they don't. - -**2. A link fault was the real lesson all along.** Right after this incident I -hit the *same* failure signature on a different drive (a Seagate IronWolf 110 that threw -`WRITE FPDMA QUEUED` timeouts, failed `mkfs`, and went silent — textbook -"dead SSD"). It turned out to be the **SATA cable**. A flaky link reproduces -every symptom I blamed on hardware below: timeouts dropped before a CRC is -computed, real pages lost, and — if both RAID1 members share the same -faulty **M.2 riser** or slot — *correlated* corruption across both drives. -That's the one explanation that fits "both members lied the same way," and it -doesn't require any drive defect at all. - -So I can't honestly say "the drives were bad." They may have been perfectly -fine, and the corruption may have come from the **M.2 riser** they both sat -on. Note that **this does not make the incident a false alarm**: real data -*could not be read back*, and whatever the cause, I removed it, migrated the -data, and verified every byte. But the correct lesson is "rule out the link -before condemning the drive," not "same-batch drives are untrustworthy." - -The full link-vs-drive detective work is in **["I Was About to RMA This SSD — -the Fault Was a SATA Cable"](/posts/that-dying-ssd-was-just-a-bad-sata-cable/)**. - -## The original analysis (kept for transparency) - -What follows is what I concluded *at the time*. It was the most defensible -reading of the evidence available, and it may still be right for reasons I -can't rule out — but read it as the incident report it is, not the final -verdict. - -## Why this matters - -A corrupting disk is the worst kind of storage failure because it's the -easiest to miss: - -- **SMART looked perfect.** Zero media errors, ~100% spare, on drives rated - for heavy enterprise write loads. -- **The pool had redundancy.** btrfs RAID1 mirrored every block across two - drives. Redundancy means nothing when *both* members lie the same way. -- **The failure was silent.** No red warning in the Unraid array status, no - failing drive, no hardware event — just VM reads slowly returning garbage - until a backup restore became the only option. - -If your data only gets corrupted when nothing looks wrong, "backup" is not -paranoia — it's the entire ballgame. This incident is why a testable, -verified restore beats any array status screen. - -## The setup - -- **unRaid 7.x** on an Intel i9-13900K, with a Windows 11 VM as the primary - workstation (GPU passed through). -- **Cache pool:** two **Samsung PM9A3 3.84 TB** NVMe drives in **btrfs RAID1**. - Both from the **same production batch** (same controller/firmware - revision, `GDC7702Q`). -- The Windows VM's virtual disks (`vdisk1.img`, `vdisk2.img`) lived on that - pool, in the `domains/` share — plus **Docker's data root**. -- The VM's vdisks were **not** `NOCOW` — so every read was validated against - a btrfs checksum. That setting ended up being the reason I caught it at all. - -## The symptom - -I started noticing cache reads were crawling, and host commands were stalling: -`cat`, reading `/proc/loadavg`, even `docker image ls` were timing out. The -corrupt NVMe was being hammered at **45–75 MB/s of reads**, and every read -saturating the drive stalled the whole host's I/O. A `docker system prune` -that should have finished in seconds just hung. - -Then the real problem surfaced: the Windows VM began throwing read errors and -data that came back was **wrong** — files that were full of zero bytes, -checksum failures in the guest, and ultimately an unbootable primary VM. - -## The forensic: `csum 0x8941f998` - -On btrfs, when a read fails verification the kernel logs a checksum mismatch -for the affected extent. The same value kept recurring: - -```text -btrfs: checksum verify failed on wanted 0x8941f998 found -``` - -That specific value was the first real clue. **`0x8941f998` is the CRC32C of -a block of zeros.** It has to be — it came up every time, across unrelated -files and inodes. What was happening wasn't random bit rot; the drive was -returning **all-zero blocks**, consistently, and those zero-filled reads were -exactly what btrfs was failing to verify: - -- Check the value: `crc32c` of a zero-filled extent matches `0x8941f998`. -- It's the same value again and again → reads were **deterministically** - producing empty data, not occasionally flipping bits. - -Two VM vdisks were affected (`vdisk1.img` inode 2358574, `vdisk2.img` inode -2362817), which fit the picture: a storage layer was returning zeros instead -of the data it was asked to read. - -## The tell: self-heal "fixes" that never stuck - -btrfs RAID1 is supposed to be self-healing: read a bad block on one member, -reconstruct it from the mirror, and rewrite the good data back. But as I -scrubbed and re-read the same (still all-zero) data, it became clear the -**rewrites were not sticking**. Each rebuild "succeeded" and the very next -read of the same extent produced zeros again. - -That's the critical distinction: - -- A drive that only **fails to read** is a read problem — self-heal should - recover it from the mirror. -- **A drive that also accepts and silently stores garbage** is a *write* or - *controller* problem — the mirror can't save you, because both paths agree - on the wrong answer. - -When self-heal rewrites keep not sticking, you're no longer in "one bad -sector" territory. You're in "the device is lying about writes" territory, -and RAID redundancy won't help. - -## Why both drives had to go (my reasoning at the time) - -> This was the basis for the original "replace both" decision. Read the -> correction in the Update above before acting on it — a shared M.2 riser -> link fault also fits this evidence without any drive being defective. - -The data pointed at a **hardware defect in the same-batch drives**, not a -wear or one-off event: - -- SMART was clean (0 media errors, ~100% spare, and `nvme0` had written - roughly 586 TiB — heavy, but on a drive rated for enterprise workloads). -- The two drives were **same-batch, same controller/firmware** (`GDC7702Q`). -- Corruption hit **both** RAID1 members in correlated fashion — the whole - point of mirroring (survive one drive dying) failed because both were dying - the same way, in lockstep. - -With a correlated, batch-level defect, keeping one "healthy-looking" drive as -the surviving member is a false economy. **I restored the Windows VM from -backup and replaced both drives.** - -## What I did in parallel: move the data off that pool - -Before the drives were replaced, anything still worth reading had to come off -the failing pool — and it had to be **verified**, not assumed: - -- **Docker's data root** moved off the cache: `docker` was relocated to - `/mnt/user/docker` (on `disk1`, overlay2). The whole tree was verified by - matching **path, size, and mtime** for all **3,997,569 files** against a - byte-comparison. -- The cache's `isos/`, `system/`, and `swapfile` moved to `disk1`; the - `isos/` set (~13 GB) was checked with `diff -rq` → **EXIT=0**, no - differences. -- Everything moved was compared against the source **before** the cache copy - was deleted — never delete a copy you haven't verified. - -The read-side corruption (all-zero blocks) and the write-side corruption -(self-heal not sticking) meant I trusted nothing from that pool without a -mirror, a checksum, or a fresh backup to compare against. - -## What I'd do differently - -1. **Test-restore the VM backup before needing it.** I'd verified backups - existed, but a corruption incident is not the time to first learn your - restore procedure. A scheduled test-restore catches both backup failure - *and* silent storage corruption. -2. **Treat same-batch drives as one drive.** Both PM9A3s came from the same - batch with the same firmware. For a married cache pool (or any RAID1) - that's a correlated-failure risk — prefer drives of different production - batches or at least hammer-test the pair before trusting a mirror. -3. **Detect, don't just hope.** btrfs checksums are what surfaced this - through the noise. The `NOCOW`-less vdisk config meant the filesystem was - validating every read; a pool of checksummed data is the difference - between "caught it and restored" and "shipped corrupted files for months". -4. **Don't trust SMART for flash.** For a healthy-looking NVMe that still - corrupts, the event count says nothing — the firmware-level defect can be - invisible to every SMART attribute. - -## The result - -One Windows VM restored from backup, the data safely off the failing pool, and -Docker's data root relocated — **plus a lesson I had to un-learn after the -fact.** I replaced both drives, but I can no longer be certain they were the -cause: the evidence fits a shared **M.2 riser** link fault just as well, and -both drives test clean now. (Full detective work in ["I Was About to RMA This -SSD — the Fault Was a SATA Cable"](/posts/that-dying-ssd-was-just-a-bad-sata-cable/).) - -The incident was contained because corruption was caught by checksums, not -wallowed in for weeks — that part stands regardless of cause. But the honest -lesson is broader and less dramatic than "replace the drives": - -1. **A "corrupting" drive isn't proof the drive is bad.** Before condemning - hardware, rule out the link: reseat/swap the cable, move to a different - port/riser/slot, and rerun the *same* failing operation. If the fault - follows the **port/riser**, it's the connection. -2. **SMART can't see link faults.** Zero media errors and ~100% spare say - nothing about a flaky cable or riser — timeouts drop I/O before a CRC is - even computed. -3. **Same-batch drives sharing a channel is a shared-failure risk either - way.** Whether the corruption was the drives or the riser, a married pair - on one faulty path is a single point of failure wearing two serial numbers. -4. **Checksum your data and test-restore your backups.** That — not the part - diagnosis — is what turned a weeks-long silent corruption into a contained - incident. - ---- -```` - ---- - -## 文章 10 / 11：为什么容器里的 Chrome 会忘记标签页 —— 以及我是怎么修好的 -- **Slug**: `why-chrome-forgets-its-tabs-in-a-container` -- **英文标题**: Why Chrome Forgets Its Tabs in a Container — And How I Fixed It -- **分类**: `devops` | **发布日期**: 2026-09-29 -- **线上地址**: https://blog.hoelee.com/posts/why-chrome-forgets-its-tabs-in-a-container/ (EN) | https://blog.hoelee.com/posts/zh/why-chrome-forgets-its-tabs-in-a-container/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "为什么容器里的 Chrome 会忘记标签页 —— 以及我是怎么修好的" -description: "Chrome 的「继续上次浏览」在 Docker 容器里根本不会生效：三次失败的尝试、真正的原因，以及最后管用的 70 行守护脚本。" -pubDate: 2026-09-29 -category: devops -tags: [docker, chrome, cdp, persistence, synology, self-hosting] -ogImage: /og/zh-why-chrome-forgets-its-tabs-in-a-container.jpg -banner: /banners/why-chrome-forgets-its-tabs-in-a-container.jpg -draft: false ---- - -我在 NAS 上的 Docker 容器里跑着一个真正的 Google Chrome。它不是爬虫实例，也不是截图服务 —— 是我真的在用的浏览器：登录着那些绝不该交给无头机器人的账号，可以从任何设备通过网页桌面看着它操作，需要自动化的时候还能用 Chrome DevTools Protocol (CDP) 驱动它。 - -它一直很好用，只有一个例外：**每次容器重启，浏览器回来时只剩一个空白标签页。** - -Chrome 明明有专门对付这件事的设置 ——「继续浏览上次打开的网页」。它一直是开着的，却毫无作用。下面就是三次失败的尝试、真正的原因、路上踩到的两颗地雷，以及最后真正让重启变得可用的那七十来行守护脚本。 - -## 为什么这件事值得认真对待 - -当容器里的浏览器就是你的工作台时，标签页就是状态本身：填了一半的表单、正在编辑的商品上架页、筛选到指定日期的仪表盘、客户基础设施的三个页面 —— 那些就是工作。浏览器一启动停在空白页，意味着每次重启后的头十分钟都在重建你本来就有的上下文。 - -这个问题我查过不止一次，网上能搜到的只有「把 `restore_on_startup` 设上」这类建议，而那套建议一进容器就失效。如果你也撞上了同一堵墙，这篇文章就是这堵墙的解释。 - -## 我的环境 - -Synology NAS 上的一个 Docker stack，里面两个浏览器（真 Chrome 加 Brave），各自挂在自己的 SSO 网关后面： - -| 组件 | 取值 | -|---|---| -| 镜像 | `lscr.io/linuxserver/chrome` —— 真正的 Google Chrome 154 | -| 桌面 | Selkies 网页桌面，容器 `3000`/`3001` → 宿主 `3030`/`3031` | -| 自动化 | CDP 暴露在 `http://:9232/json/version` | -| Profile | bind mount，容器内路径 `/config/profile` | -| 启动参数 | `--remote-debugging-port=9222 --user-data-dir=/config/profile --restore-last-session` | - -Profile 是 bind mount，所以登录态、cookie、历史记录在容器重建后确实都还在 —— 那部分从来没坏过。丢掉的只有**会话**（当时开着哪些标签页）。 - -## 尝试一：那个启动参数 - -Chromium 有一个听起来正合适的开关：`--restore-last-session`，文档说法是「在**意外退出**后恢复上次的会话」。 - -我通过镜像的 CLI 环境变量加上它，并且确认它确实传到了运行中的进程上 —— 因为这类镜像的命令行是包装脚本拼出来的： - -```bash -docker exec chrome sh -c 'tr "\0" " " < /proc/$(pgrep -f "google-chrome --" | head -1)/cmdline' | tr ' ' '\n' | grep -E 'restore|user-data-dir|remote-debug' -# --remote-debugging-port=9222 -# --user-data-dir=/config/profile -# --restore-last-session -``` - -重启。空白页。参数在，只是它没有任何「愿意恢复」的东西。 - -## 尝试二：直接改 profile 里的配置 - -Chrome 把「继续浏览上次打开的网页」存在 profile 的 `Preferences` JSON 里，字段是 `session.restore_on_startup: 1`。于是我停掉容器，直接把这两个值改了： - -```python -import json -p = '/config/profile/Default/Preferences' -d = json.load(open(p)) -d.setdefault('session', {})['restore_on_startup'] = 1 # 1 = restore last session -d.setdefault('profile', {})['exit_type'] = 'Normal' -json.dump(d, open(p, 'w')) -``` - -这就是网上最流行的那套做法，而它「半成功」了：下次启动后，Chrome **保留**了文件里的 `restore_on_startup: 1` —— 然后照样打开空白页。它没有保留的是我写的 `exit_type`：启动时它已经把那个值改回了 `Crashed`。 - -这次改写反而是第一条真正的线索。Chrome 把启动状态当成安全边界 —— 劫持者最想改的那些设置（主页、搜索引擎、启动页面）会跟另一个文件 `Secure Preferences` 做 MAC 校验。所以，磁盘上被改过的值，并不自动等于 Chrome 会遵守的值。 - -## 尝试三：托管策略 - -让浏览器行为具备权威性的正式办法是**托管策略**（managed policy），它不受上面那套 MAC 校验约束。在 Linux 上，Chrome 读取 `/etc/opt/chrome/policies/managed/*.json`： - -```json -{ "RestoreOnStartup": 1 } -``` - -动手之前有两个细节：Chrome **会忽略**运行浏览器的那个用户可写的策略文件，所以必须是 `root:root` 加 `644`；而在 Synology 的共享目录上，某些目录即使 `sudo` 也会拒绝执行 `chmod`，所以别假设你改成功了，去确认当前的实际权限。 - -挂进容器，启动后确认文件在位： - -```bash -docker exec chrome sh -c 'ls -l /etc/opt/chrome/policies/managed/; cat /etc/opt/chrome/policies/managed/hoeleepolicy.json' -# -rw-r--r-- 1 root root 28 hoeleepolicy.json -# { "RestoreOnStartup": 1 } -``` - -重启。还是空白页。 - -## 尝试四：先证明「退出是不干净的」 - -到这一步，问题已经不是「我该用哪个设置」，而是「容器停掉的时候，这个浏览器到底看到了什么」。 - -Chrome 的会话恢复机制需要一个它认为**被中断、或可续接**的会话：只有在正常退出时，它才会把 `Current Session` 落成 `Last Session`。于是我改用最温和的方式收掉浏览器 —— 通过 CDP 让它自己优雅退出，而不是把容器从它脚下抽走： - -```python -# browser-level target from /json/version -ws.send(json.dumps({"id": 1, "method": "Browser.close", "params": {}})) -``` - -Chrome 确实退出了 —— 然后桌面会话在几秒内又把它拉了起来，`exit_type` 立刻回到 `Crashed`。 - -这就是全部的机关，它一次性解释了前面三次失败：**这个浏览器从来没有干净退出过。** 你停容器时，桌面会话被一起拆掉，浏览器跟着被杀。没有干净退出，就没有 `Last Session` 的落盘，也就没有任何东西可供「继续上次浏览」或 `RestoreOnStartup` 策略去恢复。那个参数、那个首选项、那条策略其实都在正常工作 —— 只是它们手上空无一物。 - -数据其实一直都在。那段时间 profile 的会话目录每隔几秒就在写： - -``` --rw------- 1 hoelee users 40722 Session_13434833255312492 --rw------- 1 hoelee users 58879 Tabs_13434833757632480 --rw------- 1 hoelee users 79361 Tabs_13434833833646133 -``` - -## 修法：别再去配置它，开始替它记住 - -如果浏览器记不住自己的会话，那就让容器替它记住。两件事，都很小： - -1. **启动时**：等浏览器起来，如果它一个真正的页面都没打开，就把上次的网址重新打开。 -2. **之后一直**：每 60 秒把当前打开的页面地址快照一次。 - -这就是一段从镜像的容器初始化钩子拉起的后台脚本。我实际在跑的版本，去掉无关部分后是这样： - -```python -#!/usr/bin/env python3 -"""Restore last session's tabs on start, then snapshot open tabs every 60s.""" -import json, os, time, urllib.parse, urllib.request - -CDP = "http://127.0.0.1:9222" # loopback inside the container -STORE = "/config/tabs-last.txt" # lives in the persistent profile mount - -def open_tabs(): - """http(s) page URLs, or None when the browser isn't answering yet.""" - try: - with urllib.request.urlopen(CDP + "/json/list", timeout=5) as r: - tabs = json.load(r) - except Exception: - return None - return [t.get("url", "") for t in tabs - if t.get("type") == "page" and t.get("url", "").startswith("http")] - -saved = [u.strip() for u in open(STORE)] if os.path.exists(STORE) else [] - -# 1) restore — wait up to 3 minutes for the browser to come up -if saved: - for _ in range(60): - if open_tabs() is not None: - break - time.sleep(3) - if open_tabs() == []: # nothing but a blank new tab - for url in saved[:20]: - req = urllib.request.Request( - CDP + "/json/new?" + urllib.parse.quote(url, safe=""), method="PUT") - urllib.request.urlopen(req, timeout=10).read() - time.sleep(0.7) - -# 2) snapshot — write even an empty list, so "I closed everything" is respected -while True: - time.sleep(60) - tabs = open_tabs() - if tabs is None: - continue - tmp = STORE + ".tmp" - open(tmp, "w").write("\n".join(tabs) + ("\n" if tabs else "")) - os.replace(tmp, STORE) # atomic; a kill can't truncate it -``` - -两个设计选择值得抄走： - -- **只在浏览器确实空空如也时才恢复。** 万一将来某个 Chrome 版本自己会恢复标签页，这段脚本就会让路，而不是把每个标签页再复制一遍。 -- **快照是原子的，而且允许为空。** 先写临时文件再 `os.replace`，意味着写到一半被杀也不会留下半截列表；而空列表是有意义的数据 —— 它表示你主动关掉了全部标签页，所以下次启动就该保持空白。 - -拉起它的初始化钩子： - -```sh -#!/bin/sh -# clear stale singleton state (see below) -rm -f /config/profile/Singleton* 2>/dev/null -# long-running, so background it or you block container init -/usr/bin/env python3 /custom-cont-init.d/tabs_keeper.py >> /config/tabs-keeper.log 2>&1 & -exit 0 -``` - -### 地雷一：残留的 `Singleton*` 会让 Chrome 干脆不启动 - -测试重启的过程中，我撞上了一个比空白页更糟的故障：容器**健康地**起来了，里面却没有浏览器。没有 `google-chrome` 进程、CDP 不通、中继容器的健康检查报 `unhealthy`，日志里除了桌面启动之外什么都没有。 - -原因在 profile 里的 `SingletonLock`/`SingletonSocket`/`SingletonCookie` 这几个符号链接 —— 它们记录了「这个 profile 属于主机 *h* 上的进程 *n*」。而容器里的 PID 会跨重启复用，于是下一次启动可能发现自己这个 PID 已经被「占用」，于是立刻退出。在容器启动时、浏览器拉起之前把它们删掉就是解法，这也是上面钩子里那句 `rm -f` 存在的原因。 - -### 地雷二：`custom-cont-init.d` 只执行**可执行**文件，而且用 bash 执行 - -LinuxServer 的镜像会执行 `/custom-cont-init.d` 下的每个文件，但前提是它可执行，并且是用 `/bin/bash` 执行。也就是说，你放进去的 Python 文件**绝不能**带执行位，否则会被喂给 bash 然后以 2 退出： - -``` -[custom-init] tabs_keeper.py: executing... -[custom-init] tabs_keeper.py: exited 2 -``` - -可行的分工是：`.sh` 钩子设 `755`，把常驻任务放到后台；同目录的 `.py` 设 `644`。另外，任何不会立刻返回的东西都必须后台化，否则容器初始化会一直等它。 - -## 顺带带出来的管道陷阱 - -守护脚本是走 loopback 上的 CDP 跟 Chrome 说话的，所以它不需要任何端口。但如果你还想从另一台机器驱动这个浏览器，就会多出一个容器 —— 因为 Chrome ≥ 136 把 devtools 端点钉在 `127.0.0.1:9222`，并且忽略 `--remote-debugging-address`。标准做法是让一个 `socat` 边车共用浏览器的网络命名空间： - -```yaml -chrome-relay: - image: alpine/socat - network_mode: service:chrome # must share the browser's netns - command: socat TCP-LISTEN:9223,fork,reuseaddr TCP:127.0.0.1:9222 - healthcheck: - test: ["CMD-SHELL", "nc -z 127.0.0.1 9222 || exit 1"] -``` - -端口映射**不**写在中继上（它自己没有独立命名空间），而是写在浏览器服务上：`9232:9223`。 - -这个边车有一个你必须记住的行为：**重启浏览器，中继会跟着死。** 它当初所在的命名空间被重建了，socat 进程还活着、套接字却已经失效，于是 CDP 报 `connection reset`，而容器看起来一切正常。按这个顺序收拾： - -```bash -docker restart chrome && docker restart chrome-relay # order matters -``` - -上面那条健康检查的作用，就是把这种静默故障变成一个看得见的 `unhealthy`。如果你打算自己驱动容器里的浏览器，客户端那一侧我单独写过一篇：[Scraping a Bot-Walled Marketplace With a Warm Browser Session](/posts/scraping-bot-walled-marketplace-warm-browser-session/)。 - -## 如果重来一次 - -- **先确认进程能不能干净退出，再考虑相信任何「启动时恢复」的机制。** 我在一个前提条件（浏览器能察觉到的退出）根本不存在的情况下，连试了三种配置。一次 `Browser.close` 测试就能让整个排查过程坍缩。 -- **不要手改浏览器首选项。** 它们会跟 `Secure Preferences` 做 MAC 校验；磁盘上一个看起来合理的值，并不等于 Chrome 会遵守的值。需要权威性就用托管策略 —— 并且去验证策略真的加载了，而不是假设文件放进去就够了。 -- **确认谁有权限写那条策略。** `root:root`、`644`，放在 `/etc/opt/chrome/policies/managed` 下才是对的形状；用户可写的策略文件会被静默忽略。 -- **把「浏览器在跑」和「容器在跑」当成两个不同的问题。** 残留 singleton 那次故障，从外面看就是一个健康的容器。 -- **与其让软件记住状态，不如自己存快照。** 任何关闭路径上带强杀的软件 —— 容器、kiosk、桌面会话 —— 都不是依赖「干净退出」记账的好地方。一个 60 秒一次、原子写入的快照很无聊，但它管用。 - -## 结果 - -- 真实的容器重启之后，两个浏览器各自 **2/2 个标签页**都恢复了（守护日志里是 `restored 2/2 tabs`），重启后再用 CDP 复核过 -- 同一批重启里，登录态、cookie 和 `localStorage` 都确认存活 —— 持久化 profile 从来不是问题所在 -- 三个 stack（两个浏览器加一个试验品）合并成一个，并且靠中继的健康检查，把唯一那个「重启顺序」陷阱由静默变成看得见 - -## 你的业务需要这个吗？ - -如果你想要一个始终登录着的浏览器 —— 跑在你自己的硬件上、任何地方都能访问、挂在 SSO 后面、标签页状态能扛住重启，还带一个能给你脚本用的 CDP 端点 —— 那正是我在跑、也在维护的这套东西。 - -**WhatsApp: [+60 12-797 2969](https://wa.me/60127972969)** · **Email: [me@hoelee.com](mailto:me@hoelee.com?subject=Self-hosted%20container%20browser)** · **[hoelee.com](https://hoelee.com)** - -网站设计与开发是我的主业；自托管基础设施与自动化是另外一半。 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Why Chrome Forgets Its Tabs in a Container — And How I Fixed It" -description: "Chrome's \"continue where you left off\" never fires inside a Docker container. The three fixes that failed, the real root cause, and the 70-line keeper that works." -pubDate: 2026-09-29 -category: devops -tags: [docker, chrome, cdp, persistence, synology, self-hosting] -ogImage: /og/why-chrome-forgets-its-tabs-in-a-container.jpg -banner: /banners/why-chrome-forgets-its-tabs-in-a-container.jpg -draft: false ---- - -I run a real Google Chrome inside a Docker container on my NAS. It is not a scraping instance and not a screenshot service — it is a browser I actually use: signed into the accounts a headless bot should never touch, watchable through a web desktop from any device, and drivable over the Chrome DevTools Protocol when I want to automate part of a job. - -It works well, with one exception. **Every time the container restarts, the browser comes back with a single blank tab.** - -Chrome has a setting for exactly this — "Continue where you left off". It was already enabled. It did nothing. What follows is three fixes that failed, the actual root cause, the two landmines I hit on the way, and the ~70-line keeper that finally made restarts survivable. - -## Why this matters - -When a container browser is your workspace, the tabs *are* the state. Half-finished forms, a marketplace listing in progress, a dashboard filtered to the right date range, three tabs open on a customer's infrastructure — that is the work. A browser that opens on `about:blank` means the first ten minutes after every restart are spent rebuilding context you already had. - -I looked for this answer more than once and found nothing but "set `restore_on_startup`" advice that does not survive contact with a container. If you have hit the same wall, this is the wall explained. - -## The setup - -A single Docker stack on a Synology NAS, two browsers in it (real Chrome and Brave), each behind my own SSO gate: - -| Piece | Value | -|---|---| -| Image | `lscr.io/linuxserver/chrome` — real Google Chrome 154 | -| Desktop | Selkies web desktop, container `3000`/`3001` → host `3030`/`3031` | -| Automation | CDP published at `http://:9232/json/version` | -| Profile | a bind mount, `/config/profile` inside the container | -| Launch flags | `--remote-debugging-port=9222 --user-data-dir=/config/profile --restore-last-session` | - -The profile is a bind mount, so the logins, cookies and history genuinely do persist across a container rebuild — that part was never broken. Only the *session* (which tabs were open) was lost. - -## Attempt 1: the flag - -Chromium ships a switch that sounds like the answer: `--restore-last-session`. It is documented as "restore the last session *after an unexpected exit*". - -I added it via the image's CLI env var and confirmed it was actually applied to the running process, because these images build the command line from a wrapper script: - -```bash -docker exec chrome sh -c 'tr "\0" " " < /proc/$(pgrep -f "google-chrome --" | head -1)/cmdline' | tr ' ' '\n' | grep -E 'restore|user-data-dir|remote-debug' -# --remote-debugging-port=9222 -# --user-data-dir=/config/profile -# --restore-last-session -``` - -Restart. Blank tab. The flag was there; it just had nothing it was willing to restore. - -## Attempt 2: edit the profile's preferences - -Chrome stores "Continue where you left off" in the profile's `Preferences` JSON as `session.restore_on_startup: 1`. So I stopped the container and edited it directly, along with the `exit_type` marker: - -```python -import json -p = '/config/profile/Default/Preferences' -d = json.load(open(p)) -d.setdefault('session', {})['restore_on_startup'] = 1 # 1 = restore last session -d.setdefault('profile', {})['exit_type'] = 'Normal' -json.dump(d, open(p, 'w')) -``` - -This is the classic advice you will find online, and it half-worked: after the next boot, Chrome had **kept** `restore_on_startup: 1` in the file — and still opened a blank tab. What it did not keep was my `exit_type`: it had already rewritten that to `Crashed` at startup. - -That rewrite turned out to be the first real clue. Chrome treats startup state as a security boundary — the settings a hijacker would most like to change (homepage, search provider, startup pages) are MAC-checked against a separate `Secure Preferences` file. A value edited on disk does not automatically become a value Chrome obeys. - -## Attempt 3: a managed policy - -The supported way to force browser behaviour is a **managed policy**, which is authoritative and not subject to that MAC check. On Linux, Chrome reads `/etc/opt/chrome/policies/managed/*.json`: - -```json -{ "RestoreOnStartup": 1 } -``` - -Two details before you bother: Chrome ignores a policy file that is writable by the user running the browser, so it needs `root:root` and mode `644` — and on a Synology share, `chmod` refuses to run even under `sudo` for some directories, so verify the mode you already have instead of assuming you changed it. - -Mounted into the container, verified present after boot: - -```bash -docker exec chrome sh -c 'ls -l /etc/opt/chrome/policies/managed/; cat /etc/opt/chrome/policies/managed/hoeleepolicy.json' -# -rw-r--r-- 1 root root 28 hoeleepolicy.json -# { "RestoreOnStartup": 1 } -``` - -Restart. Blank tab. Still. - -## Attempt 4: prove the exit is ungraceful - -At this point the question stopped being "which setting do I need" and became "what does this browser actually see when the container stops". - -Chrome's restore machinery needs a session it considers *interrupted or resumable*. It rolls `Current Session` into `Last Session` on a clean exit. I asked the CDP endpoint to close the browser gracefully instead of pulling the container out from under it: - -```python -# browser-level target from /json/version -ws.send(json.dumps({"id": 1, "method": "Browser.close", "params": {}})) -``` - -Chrome exited — and the desktop session relaunched it within seconds. `exit_type` was back to `Crashed` immediately. - -That is the whole trick, and it explains all three failures: **the browser never exits cleanly.** When you stop the container, the desktop session is torn down and the browser is killed with it. No clean exit means no `Last Session` rollover, which means there is nothing for "continue where you left off" — or a `RestoreOnStartup` policy — to restore. The flag, the preference and the policy were all working correctly and all had nothing to work with. - -The data was never missing. The profile's session directory was being written continuously, seconds apart, the whole time: - -``` --rw------- 1 hoelee users 40722 Session_13434833255312492 --rw------- 1 hoelee users 58879 Tabs_13434833757632480 --rw------- 1 hoelee users 79361 Tabs_13434833833646133 -``` - -## The fix: stop configuring, start keeping - -If the browser cannot be trusted to remember its own session, the container can remember it instead. Two jobs, both tiny: - -1. **On start**: wait for the browser, and if it came up with no real page open, re-open the URLs it had last time. -2. **Forever after**: every 60 seconds, snapshot the open page URLs. - -That is a background script launched from the image's container-init hook. The version I run, trimmed to the useful parts: - -```python -#!/usr/bin/env python3 -"""Restore last session's tabs on start, then snapshot open tabs every 60s.""" -import json, os, time, urllib.parse, urllib.request - -CDP = "http://127.0.0.1:9222" # loopback inside the container -STORE = "/config/tabs-last.txt" # lives in the persistent profile mount - -def open_tabs(): - """http(s) page URLs, or None when the browser isn't answering yet.""" - try: - with urllib.request.urlopen(CDP + "/json/list", timeout=5) as r: - tabs = json.load(r) - except Exception: - return None - return [t.get("url", "") for t in tabs - if t.get("type") == "page" and t.get("url", "").startswith("http")] - -saved = [u.strip() for u in open(STORE)] if os.path.exists(STORE) else [] - -# 1) restore — wait up to 3 minutes for the browser to come up -if saved: - for _ in range(60): - if open_tabs() is not None: - break - time.sleep(3) - if open_tabs() == []: # nothing but a blank new tab - for url in saved[:20]: - req = urllib.request.Request( - CDP + "/json/new?" + urllib.parse.quote(url, safe=""), method="PUT") - urllib.request.urlopen(req, timeout=10).read() - time.sleep(0.7) - -# 2) snapshot — write even an empty list, so "I closed everything" is respected -while True: - time.sleep(60) - tabs = open_tabs() - if tabs is None: - continue - tmp = STORE + ".tmp" - open(tmp, "w").write("\n".join(tabs) + ("\n" if tabs else "")) - os.replace(tmp, STORE) # atomic; a kill can't truncate it -``` - -Two design choices worth copying: - -- **Restore only when the browser came up empty.** If a future Chrome version *does* restore its own tabs, the keeper stays out of the way instead of duplicating every tab. -- **The snapshot is atomic and may be empty.** Writing through a temp file plus `os.replace` means a container killed mid-write cannot leave a truncated list, and an empty snapshot is meaningful data: it means you closed everything on purpose, so the next start should stay empty. - -The container-init hook that runs it: - -```sh -#!/bin/sh -# clear stale singleton state (see below) -rm -f /config/profile/Singleton* 2>/dev/null -# long-running, so background it or you block container init -/usr/bin/env python3 /custom-cont-init.d/tabs_keeper.py >> /config/tabs-keeper.log 2>&1 & -exit 0 -``` - -### Landmine 1: a stale `Singleton*` stops Chrome from starting at all - -While testing restarts I hit a worse failure than a blank tab: the container came up **healthy with no browser in it**. No `google-chrome` process, CDP dead, the relay's healthcheck reporting `unhealthy`, nothing in the logs beyond the desktop starting. - -The cause is the profile's `SingletonLock`/`SingletonSocket`/`SingletonCookie` symlinks. They encode "this profile is owned by PID *n* on host *h*" — and inside a container, PIDs are reused across restarts, so the next boot can find its own PID already claiming the profile and exit immediately. Deleting them at container start, before the browser launches, is the fix. It is also why that `rm -f` sits in the hook above. - -### Landmine 2: `custom-cont-init.d` only runs *executable* files — and runs them with bash - -The LinuxServer images execute every file in `/custom-cont-init.d`, but only if it is executable, and each one through `/bin/bash`. Which means a Python file you drop in there must **not** be executable, or it gets fed to bash and exits 2: - -``` -[custom-init] tabs_keeper.py: executing... -[custom-init] tabs_keeper.py: exited 2 -``` - -The split that works: a `.sh` hook at `755` that backgrounds the long-running work, and the `.py` next to it at `644`. And anything that does not return immediately has to be backgrounded, or container init waits on it forever. - -## The plumbing trap that came with it - -The keeper talks to Chrome over CDP on loopback, which is why it needs no ports. If you also want to drive the browser from another machine, there is one more container in the picture, because Chrome ≥ 136 pins its devtools endpoint to `127.0.0.1:9222` and ignores `--remote-debugging-address`. The standard workaround is a `socat` sidecar sharing the browser's network namespace: - -```yaml -chrome-relay: - image: alpine/socat - network_mode: service:chrome # must share the browser's netns - command: socat TCP-LISTEN:9223,fork,reuseaddr TCP:127.0.0.1:9222 - healthcheck: - test: ["CMD-SHELL", "nc -z 127.0.0.1 9222 || exit 1"] -``` - -The port mapping does **not** go on the relay (it has no namespace of its own) — it goes on the browser service: `9232:9223`. - -That sidecar has one behaviour you have to internalise: **restart the browser and the relay dies with it.** The namespace it was living in is rebuilt underneath it, socat keeps running and its sockets stop working, so CDP answers with `connection reset` while the container still looks fine. Fix the symptoms in this order: - -```bash -docker restart chrome && docker restart chrome-relay # order matters -``` - -The healthcheck above is what turns a silent failure into a visible `unhealthy` flag. If you want to drive a container browser yourself, I wrote the client side up separately in [Scraping a Bot-Walled Marketplace With a Warm Browser Session](/posts/scraping-bot-walled-marketplace-warm-browser-session/). - -## What I'd do differently - -- **Determine whether the process can exit cleanly before trusting any "restore on startup" mechanism.** I spent three attempts configuring a feature whose precondition — an exit the browser notices — was never met. One `Browser.close` test would have collapsed the whole search. -- **Don't hand-edit browser preferences.** They are MAC-checked against `Secure Preferences`; a plausible-looking value on disk is not a value Chrome obeys. If you need a switch to be authoritative, use a managed policy — and verify the policy is loaded instead of assuming the file is enough. -- **Check who is allowed to write the policy.** A root-owned `644` file in `/etc/opt/chrome/policies/managed` is the shape; a user-writable one is ignored silently. -- **Treat "the browser is running" and "the container is running" as different questions.** The stale-singleton failure looked like a healthy container from the outside. -- **Snapshot state instead of asking software to remember it.** Anything with a hard kill in its shutdown path — containers, kiosks, desktop sessions — is a poor place to rely on clean-exit bookkeeping. A 60-second atomic snapshot is boring, and it works. - -## The result - -- Tabs restored **2/2 on both browsers** across a real container restart (`restored 2/2 tabs` in the keeper log), verified over CDP afterwards -- Logins, cookies and `localStorage` confirmed surviving the same restarts — the persistent profile was never the problem -- Three stacks (two browsers plus a throwaway experiment) collapsed into one, with the relay's healthcheck making the one restart-order trap visible instead of silent +All three hosts now run the same instrumented layout — a 41-panel host dashboard and a +10-panel container dashboard each — with 21/23/25 panels returning data respectively, +and the two structurally-empty classes documented in place instead of silently +confusing whoever opens them next. ## Want this for your business? -If you want a browser that stays logged in — running on your own hardware, reachable from anywhere, behind SSO, with tab state that survives restarts and a CDP endpoint your scripts can drive — that is the setup I run and maintain. +If you have Grafana dashboards that nobody trusts — or servers and a NAS with no +monitoring at all — I build self-hosted Prometheus + Grafana stacks (host and container +metrics, sensible alert rules, alerts to Telegram and email) and I'll fix the ones that +have quietly stopped showing data. -**WhatsApp: [+60 12-797 2969](https://wa.me/60127972969)** · **Email: [me@hoelee.com](mailto:me@hoelee.com?subject=Self-hosted%20container%20browser)** · **[hoelee.com](https://hoelee.com)** +**WhatsApp: [+60 12-797 2969](https://wa.me/60127972969)** · **Email: [me@hoelee.com](mailto:me@hoelee.com?subject=Grafana%20monitoring)** · **[hoelee.com](https://hoelee.com)** -Website design and development is my main line of work; self-hosted infrastructure and automation is the other half of it. +Website design and development is my main line of work; server hardening and +self-hosted infrastructure is the other half of it. ```` --- -## 文章 11 / 11：Telegram 机器人通知为什么发不出去：IPv6、DNS 和一个折腾了一整天的 400 -- **Slug**: `why-telegram-bot-notifications-die` -- **英文标题**: Why Telegram Bot Notifications Die: IPv6, DNS, and a 400 That Took a Day -- **分类**: `devops` | **发布日期**: 2026-03-25 -- **线上地址**: https://blog.hoelee.com/posts/why-telegram-bot-notifications-die/ (EN) | https://blog.hoelee.com/posts/zh/why-telegram-bot-notifications-die/ (ZH) +## 文章 03 / 11：你的“磁盘将满”告警在说谎：别再按百分比告警 +- **Slug**: `your-disk-full-alert-is-lying` +- **英文标题**: Your Disk-Full Alert Is Lying: Stop Alerting on Percentages +- **分类**: `devops` | **发布日期**: 2026-06-24 +- **线上地址**: https://blog.hoelee.com/posts/your-disk-full-alert-is-lying/ (EN) | https://blog.hoelee.com/posts/zh/your-disk-full-alert-is-lying/ (ZH) - **本篇审核状态**: ⏳ 待审核 (Pending) ### ✍️ 顾问审阅意见区 (Review Feedback) @@ -3916,430 +637,3504 @@ Website design and development is my main line of work; self-hosted infrastructu ### 🇨🇳 中文版原文 (Chinese Markdown Twin) ````markdown --- -title: "Telegram 机器人通知为什么发不出去：IPv6、DNS 和一个折腾了一整天的 400" -description: "一次排障记录：一个跑在 Docker 里的 Telegram 监控机器人，sendPhoto 一直返回 nginx/1.30.1 的 400，但 getMe 却正常——根因是只返回 IPv6 的 DNS 应答、一个写错的 extra_hosts IP，以及一个用 join 拼接破坏了 JPEG 二进制数据的 multipart body。" -pubDate: 2026-03-25 -updatedDate: 2026-09-09 +title: "你的“磁盘将满”告警在说谎：别再按百分比告警" +description: "\"剩余 3%\"听着像紧急事故，直到你发现那是 500 GB。在多 TB 的卷上，百分比阈值会在没事的时候乱叫，也会在真出事的时候沉默。" +pubDate: 2026-06-24 +updatedDate: 2026-09-29 category: devops -blueprint: ai-agent-automation -blueprintLayer: L1-deterministic-orchestration -tags: [docker, telegram, python, dns, ipv6, debugging, portainer] -ogImage: /og/zh-why-telegram-bot-notifications-die.jpg -banner: /banners/why-telegram-bot-notifications-die.jpg +blueprint: self-hosted-infrastructure +blueprintLayer: L3-observability-self-healing +tags: [monitoring, alerting, prometheus, grafana, storage, capacity, observability] +ogImage: /og/zh-your-disk-full-alert-is-lying.jpg +banner: /banners/your-disk-full-alert-is-lying.jpg +draft: false --- -我跑着一个小监控机器人，专门盯着 Carousell 上新上架的商品，一旦有新品就通过 -Telegram 通知我。有一天它突然不再通知了。容器本身是健康的，爬虫也照常往 -NocoDB 里归档商品——但每一条通知都以日志里那三个字符宣告失败： +我搭好监控、写了一套看着合理的规则，然后用一个下午删掉了其中大半阈值。它们全是百分比，而且都以同一种方式错了。 -``` -telegram sendPhoto failed: 400 ... nginx/1.30.1 +## 三个教我道理的告警 + +**1.「文件系统使用率超过 90%」**——它在一个还剩约 500 GB 的 17 TB 卷上触发。没事发生，也不会马上有事发生。这种体量的卷，一次大备份就能让百分比摆动几个点：比例是噪声，而\"还有 500 GB 可用\"才是世界的真实状态。 + +**2.「内存使用率超过 90%」**——在一台**可用**内存 4.8 GB 的主机上反复触发。Linux 会用空闲内存做 page cache，需要时立刻归还；`MemTotal - MemFree` 描述的是文件缓存，不是你的风险。真正预示 OOM 的是 `MemAvailable`。 + +**3.「CPU steal 超过 25%」**——一台把邮件栈跑得完全正常的 VPS，steal 长期在 41%。steal 说明宿主机忙，它是**容量**信号而不是**故障**信号；7 vCPU 的机器 41% steal 就是这台机器的成本。这条规则产生的告警永远为真、也永远无用。 + +共同点：**我在对比例告警，而比例不知道东西有多大。** + +## 对\"后果\"告警，而不是对比例 + +我现在对每个阈值都问一句：*如果这个状态持续下去，实际会发生什么？* 答案几乎总能用绝对单位表达。 + +| 不要这样 | 改成 | 原因 | +|---|---|---| +| 文件系统 > 90% | 剩余空间 < N GB | \"我还能不能写进去\"才是问题 | +| 内存使用 > 90% | `MemAvailable` < 512 MB | 逼近 OOM，而不是\"缓存很暖和\" | +| CPU steal > 25% | steal > 50% | 容量成本 vs 真的被饿死 | +| 负载 > N | `load1 / 核数 > 2` | 不谈核数的负载没有意义 | + +PromQL 的前后对比： + +```promql +# BEFORE — 17 TB 卷的百分比；还剩半 TB 就开叫 +(1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 90 + +# AFTER — 真正装数据的卷上，还剩多少容量 +min by (instance, mountpoint) ( + node_filesystem_avail_bytes{mountpoint=~"/volume[0-9]+|/mnt/ssd|/mnt/disk[0-9]+"} +) < 25e9 ``` -`nginx/1.30.1` 这个字符串，成了我接下来一整天排障的"反派"。这是一个看似 -微不足道的 bug，实际却是由三个问题层层堆叠而成的故事——以及下一次我会 -如何更快地逐个定位它们。 +```promql +# BEFORE — 把 page cache 当成压力 +(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90 -## 问题本身 - -这个机器人有两件事要做：**采集**商品，以及**通知**我。采集正常，通知挂了。 -日志显示 `400 Bad Request`，并指向一个自称 `nginx/1.30.1` 的服务器。 - -我的第一反应很教科书：**网络里有什么东西在拦截请求。** 因为 -`nginx/1.30.1` 这个 header 跟我印象里的 Telegram API 对不上——而且一个 -带 HTML 响应体的 400（Telegram 就算出错也返回 JSON）看起来就像本地反向 -代理或 VPN 网关在请求还没离开机器之前就把它拒掉了。 - -这种假设很省力，但也常常是错的。下面说说我到底发现了什么。 - -## 我尝试了什么，以及为什么每个都失败了 - -### 第一轮：一定是 DNS / IPv6 - -我先在容器里跑了 `getMe`——它能通。然后又跑了 `sendMessage`——它挂了。 -同一个 token、同一个容器，一个方法通、一个方法挂。 - -线索藏在两个端点的解析方式里： - -``` -# 容器内部 -$ getent hosts api.telegram.org -2001:67c:4e8:f004::9 api.telegram.org +# AFTER — 真正的 OOM 前兆 +node_memory_MemAvailable_bytes < 512 * 1024 * 1024 ``` -容器把 `api.telegram.org` 解析成了**只有 IPv6**——一条 `AAAA` 记录， -没有 `A` 记录。而容器跑在一个 **没有 IPv6 连通性的** Docker bridge -网络上。于是每个需要访问 `api.telegram.org` 的请求，都在试图连接一个 -它根本走不到的 IPv6 地址。 +两个让规则能活下来的细节： -修复看起来很简单：用 `extra_hosts` 把 IPv4 地址钉死。我照做了，DNS -应答变得完全符合预期： +**把比较放进阈值条件里，不要塞进表达式。** 查询就写 `min by (mountpoint) (node_filesystem_avail_bytes{...})`，让告警规则去和 `25000000000` 比。若你在 PromQL 里写死 `< 25e9`、规则里又留着继承来的 `> 90`，规则照样会评估——但界面显示的阈值是错的，下一个读它的人得从两个互相矛盾的条件下反推你的意图。 -``` -149.154.167.220 api.telegram.org -``` +**排除那些\"只是另一个挂载点的视图\"。** 一条\"剩余空间\"规则有多诚实，取决于它的选择器。在我自己的环境里，同一个 NAS 卷出现三次（`/volume1`、`/opt`、以及在另一台主机上以 CIFS 再次挂载），而 unRaid 的 shfs 联合挂载报告 `avail=0`——不加排除的规则不是永远误报，就是把容量算成两倍。一个写明白的选择器胜过聪明的通用写法。 -结果通知**还是失败**，同一个 400。我那得意的修复啥也没改变。 +## 噪声不是无害的 -### 第二轮：我选错了 IP +坏阈值的代价不在告警本身，而在**训练**：每条\"没事乱叫\"的告警都在教你先扫一眼、然后忽略；等你真正该看的那条到来时，它出现在一个你早已不再读的频道里。我的规则总数**减少了**，覆盖率却上升了——更少，但每条都对应一个明确的后果。 -这里有一个值得记住的教训：我是**凭记忆**硬编码了 `149.154.167.220`。 -等我终于去查公共解析器时，`api.telegram.org` 真正的 `A` 记录并不是它： +## 阈值是私人的，形状是通用的 -``` -149.154.166.110 api.telegram.org # DNS 实际返回的 -149.154.167.220 api.telegram.org # 我硬编码的 -``` +上面的数字是我的：NAS 卷剩不到 25 GB 我会在意、可用内存低于 512 MB 我才管、VPS steal 超过 50% 才算被饿。你的数字随硬件和容忍度而变。 -`167.220` 确实**在** Telegram 的 IP 段内（`149.154.160.0/20`），所以它 -并不是"别人的服务器"这种错——但它不是当前活跃的端点，bot-API 的流量打到 -它上面的行为是不可预测的。我把它换成 `166.110`，仍然失败。所以 IP 也 -不是全部原因。 +能通用的是形状： -### 第三轮：同样的命令，两个相反的结果 - -这是整个事情开始变得诡异的一刻。在同一个容器里，几乎同一秒： - -- **常驻的监控进程**（`PID 1`）→ `sendPhoto` 失败，400 -- 一个**新起的 `docker exec` 进程**跑着完全相同的代码 → `sendPhoto` - 成功，200 - -同一个容器、同样的代码、同样的 payload、同一秒。 - -我逐一排除了 token、chat_id、caption 里的 emoji、Unicode 引号、图片格式、 -环境变量、以及图片 URL 本身——我把那张图下载下来测过，是一张完全合法的 -JPEG。两条路径的代码没有任何差别。 - -当你撞上这种"输入相同、输出相反"的矛盾时，答案通常是：**输入其实并不 -相同**。但找到那个差异，还需要我再换一次思路。 - -### 第四轮：`nginx/1.30.1` 就是 Telegram - -我重新读了原始的 400 响应，而不是去猜。响应是一张 HTML 的 -"400 Bad Request" 页面，`Server` 头是 `nginx/1.30.1`。 - -**那个 nginx 就是 Telegram 自家的边缘服务器。** Telegram 的 API 背后就是 -nginx（版本 1.30.1），当一个请求在到达它们应用层**之前**就已经畸形时， -nginx 会自己回一张 400 页面——没有 JSON，也没有友好的错误码。 - -这一下彻底改变了整个视角。请求并没有被拦截，而是**确实到达了 Telegram， -然后被 Telegram 的 nginx 以"畸形"为由拒绝了**。 - -而那个畸形的东西，正是我一直没仔细看的部分：`sendPhoto` 的 multipart -请求体。 - -## 修复 - -我最初的代码是这样拼 multipart 请求体的——正是那种"看着显然没问题"、 -实则埋下微妙 bug 的一行： - -```python -body = b"\r\n".join(body_lines) -``` - -这个 `join` 看起来很省事，但 JPEG 是**二进制数据**。它的字节里本身就包含 -`\r\n` 这样的字节序列，出现在图片数据中间。用 `\r\n` 作为分隔符去 join， -就把本该是图片 payload 的字节给破坏了。Telegram 收到的是一个 multipart -边界被破坏的请求体，于是它的 nginx 在 bot API 还没来得及解析之前就回了个 -400。 - -正确的做法是逐字段拼装 body，让二进制数据保持原样： - -```python -boundary = "----tg" + token_hex(8) - -parts = [] -parts.append(f"--{boundary}\r\n".encode()) -parts.append(b'Content-Disposition: form-data; name="chat_id"\r\n\r\n') -parts.append(f"{chat_id}\r\n".encode()) - -parts.append(f"--{boundary}\r\n".encode()) -parts.append(b'Content-Disposition: form-data; name="caption"\r\n\r\n') -parts.append(f"{caption}\r\n".encode()) - -parts.append(f"--{boundary}\r\n".encode()) -parts.append( - b'Content-Disposition: form-data; name="photo"; ' - b'filename="image.jpg"\r\nContent-Type: image/jpeg\r\n\r\n' -) -parts.append(image_bytes) # ← 二进制保持二进制 -parts.append(b"\r\n") - -parts.append(f"--{boundary}--\r\n".encode()) - -body = b"".join(parts) # join 的是字节块，不是行 -``` - -区别在于用 `b"".join(parts)` 而不是 `b"\r\n".join(...)`：每一块本身就是 -一个带完整边框的片段，我们只是把它们的字节**原样拼接**，而不是在 -**每个元素之间**都插入一个分隔符。 - -还有一个更高层的修复值得点名：**别再让 Telegram 替你去下载图片。** 我 -早期的代码是把图片当作 *URL* 传过去，让 Telegram 自己去抓： - -```python -tg("sendPhoto", {"photo": image_url, "caption": caption}) -``` - -这意味着 Telegram 的服务器得主动去访问源 CDN——如果那个 CDN 不稳定、 -或者对 Telegram 的爬虫做了地域封锁，你就会得到本地根本无法复现的间歇性 -失败。自己先把图片下载下来、再上传字节，就消除了一整类的不稳定因素， -还给了你在发送前校验字节的机会。 - -## 如果重来一次我会怎么做 - -1. **读原始响应，而不是读假设。** `nginx/1.30.1` 在日志里躺了好几个小时， - 我却在追一个根本不存在的中间人。在凭空捏造拦截者之前，先查清楚那个 - header 到底属于谁。 -2. **对硬编码的 IP 用实时解析器验证。** 记忆不是 DNS。`dig +short - api.telegram.org` 只要三秒钟，就能省掉我整整一轮。 -3. **把"输入相同、输出相反"当成谎言。** 输入从来就不相同——两个进程在某 - 个我没看到的地方有差异。诚实的做法是去 diff 字节，而不是 diff 代码。 -4. **永远不要用文本分隔符去手工拼接二进制数据。** 把文本和 JPEG 混在一起 - 做 `b"\r\n".join()`，迟早会出破坏 bug。 +- 用绝对单位而非比例； +- 把后果写进告警的摘要里； +- 选择器明确声明它在替哪些挂载点说话； +- 规则数量少到你还能读完每一条。 ## 结果 -multipart 修复之后，监控机器人在下一个 tick 就真正把通知发出去了—— -图片和文字都在。我还一并加了一个重试队列：一条失败的通知会保持"未发送" -状态，30 秒后自动重试，而不是悄无声息地消失。整件事耗掉了大半天的功夫， -但它变成了那种我这辈子不会再踩第二次的坑。 +同一套栈，从\"三条永久无用报警\"变成 16 条在机房健康时保持安静的规则——其中一条立刻暴露了一个只剩 21 GB 的 NAS 卷，而那正是百分比规则一直藏在\"99% 满的巨物\"背后、被你学会忽略的东西。 ---- +## 需要为你的业务做这个吗？ -*这个机器人是我个人的项目，但教训可以推广到任何依赖外发通知的服务。如果你 -有一个监控、抓取，或者需要可靠地触达用户的告警管线，我们可以聊聊。* +如果你的监控在发你已经学会忽略的告警，或者你根本没有监控、宁愿从一条规则而不是从一次失败的备份里得知磁盘满了，我可以帮你搭自托管的监控与告警（Prometheus + Grafana，通知推 Telegram 和邮件），阈值按\"什么会真的坏\"来定。 -**我搭建可靠的监控与通知管线——Telegram/WhatsApp 告警、抓取器，以及背后的 -基础设施。** 告诉我你要监控什么、通知要发到哪里： +**WhatsApp：[+60 12-797 2969](https://wa.me/60127972969)** · **邮箱：[me@hoelee.com](mailto:me@hoelee.com?subject=Monitoring%20and%20alerting)** · **[hoelee.com](https://hoelee.com)** -- [WhatsApp 联系 →](https://wa.me/60127972969?text=Hi%20Hoelee%2C%20I%20need%20a%20monitoring%2Fnotification%20pipeline) -- [Email 联系 →](mailto:me@hoelee.com?subject=Monitoring%20%2F%20notification%20pipeline) -- [hoelee.com](https://www.hoelee.com) +网站设计与开发是我的主业；服务器加固与自托管基础设施是它的另一半。 ```` ### 🇬🇧 英文版原文 (English Markdown Primary) ````markdown --- -title: "Why Telegram Bot Notifications Die: IPv6, DNS, and a 400 That Took a Day" -description: "A debugging story: a Dockerized telegram monitor whose sendPhoto calls returned 400 from nginx/1.30.1 while getMe worked fine — an IPv6-only DNS answer, a wrong extra_hosts IP, and a multipart join that corrupted JPEG bytes." -pubDate: 2026-03-25 -updatedDate: 2026-09-09 +title: "Your Disk-Full Alert Is Lying: Stop Alerting on Percentages" +description: "3% free sounds like an emergency until you notice it is 500 GB. On multi-terabyte volumes, percentage thresholds fire when nothing is wrong — and miss when something is." +pubDate: 2026-06-24 +updatedDate: 2026-09-29 category: devops -blueprint: ai-agent-automation -blueprintLayer: L1-deterministic-orchestration -tags: [docker, telegram, python, dns, ipv6, debugging, portainer] -ogImage: /og/why-telegram-bot-notifications-die.jpg -banner: /banners/why-telegram-bot-notifications-die.jpg +blueprint: self-hosted-infrastructure +blueprintLayer: L3-observability-self-healing +tags: [monitoring, alerting, prometheus, grafana, storage, capacity, observability] +ogImage: /og/your-disk-full-alert-is-lying.jpg +banner: /banners/your-disk-full-alert-is-lying.jpg +draft: false --- -I run a small monitoring bot that watches Carousell for newly-listed -listings and pings me on Telegram when something fresh appears. One day it -stopped pinging. The container was healthy, the scraper was still archiving -listings into NocoDB — but every notification died with the same three -characters in the log: +I built a monitoring stack, wrote a reasonable set of rules, and then spent an +afternoon deleting most of my own thresholds. Every one of them was a percentage, and +every one of them was wrong in the same specific way. -``` -telegram sendPhoto failed: 400 ... nginx/1.30.1 +## The three alerts that taught me + +**1. "Filesystem above 90% used."** It fired on a 17 TB volume that had ~500 GB free. +Nothing was wrong. Nothing was *about* to be wrong. On a volume that size, a single +large backup can swing the percentage by several points — the ratio is noise, and the +fact that 500 GB is still available is the actual state of the world. + +**2. "Memory above 90% used."** Fired constantly on a host with 4.8 GB *available*. +Linux uses free RAM for page cache and gives it back on demand; `MemTotal - MemFree` +describes your filesystem cache, not your risk. The number that predicts an OOM is +`MemAvailable`. + +**3. "CPU steal above 25%."** Fired on a VPS that had been running its mail stack +perfectly at 41% steal. Steal means the hypervisor is busy — it is a *capacity* signal, +not a *failure* signal, and 41% steal on a 7-vCPU box was simply what that host costs. +The rule produced alerts that were always true and never actionable. + +The common thread: **I was alerting on a ratio, and the ratio doesn't know how big the +thing is.** + +## Alert on the consequence, not the ratio + +The test I now apply to every threshold: *if this condition persists, what will actually +happen?* That answer is almost always expressible in absolute units. + +| Instead of | Alert on | Because | +|---|---|---| +| Filesystem > 90% | Free space < N GB | "Can I still write?" is the question | +| Memory > 90% used | `MemAvailable` < 512 MB | about to OOM, not "cache is warm" | +| CPU steal > 25% | Steal > 50% | capacity cost vs. starvation | +| Load average > N | `load1 / cores > 2` | load is meaningless without core count | + +In PromQL, before and after: + +```promql +# BEFORE — percentage of a 17 TB pool; fires with half a terabyte still free +(1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 90 + +# AFTER — capacity left, on the volumes that hold data +min by (instance, mountpoint) ( + node_filesystem_avail_bytes{mountpoint=~"/volume[0-9]+|/mnt/ssd|/mnt/disk[0-9]+"} +) < 25e9 ``` -That string, `nginx/1.30.1`, became the villain of a day-long debugging -session. This is the story of how a deceptively small bug turned out to be -three separate problems stacked on top of each other — and how I'd find each -one faster next time. +```promql +# BEFORE — page cache counted as "pressure" +(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90 -## The problem - -The bot had two jobs: **collect** listings, and **notify** me. Collection -worked. Notification didn't. The log said `400 Bad Request` and pointed at a -server identifying itself as `nginx/1.30.1`. - -My first assumption was textbook: **something in the network is intercepting -the request.** `nginx/1.30.1` is not a header I associated with Telegram's -API. A 400 with an HTML body (Telegram returns JSON even on errors) smelled -like a local reverse proxy or a VPN gateway rejecting traffic before it ever -left the machine. - -Assumptions like that are cheap. They're also wrong a lot. Let me walk -through what I actually found. - -## What I tried, and why each thing failed - -### Round 1: it must be DNS / IPv6 - -I checked `getMe` from inside the container. It worked. Then I checked -`sendMessage`. It failed. Same token, same container, one method up, one down. - -The giveaway was in how the two endpoints resolved: - -``` -# inside the container -$ getent hosts api.telegram.org -2001:67c:4e8:f004::9 api.telegram.org +# AFTER — the actual OOM precursor +node_memory_MemAvailable_bytes < 512 * 1024 * 1024 ``` -The container resolved `api.telegram.org` to **IPv6 only** — a single `AAAA` -record and no `A` record. The container ran on a Docker bridge network with -**no IPv6 connectivity**. So every request that needed `api.telegram.org` was -trying to reach an IPv6 address it had no route to. +Two details that make these rules survive contact with a real dashboard: -The fix looked obvious: pin the IPv4 address with `extra_hosts`. I did, and -the DNS answer became exactly what I expected: +**Keep the comparison in the threshold condition, not inside the expression.** Query +`min by (mountpoint) (node_filesystem_avail_bytes{...})`, then let the alert rule +compare it to `25000000000`. If you bake `< 25e9` into the PromQL *and* leave an +inherited `> 90` in the rule, the rule still evaluates — but it displays a nonsense +threshold, and the next person to read it has to reconstruct your intent from two +contradictory conditions. -``` -149.154.167.220 api.telegram.org +**Exclude the mounts that are views of other mounts.** A "free space" rule is only as +honest as its selector. On my own setup, one NAS volume appears three times (as +`/volume1`, as `/opt`, and again on another host as a CIFS re-mount), and unRaid's shfs +union mount reports `avail=0` — a rule without exclusions would either fire forever or +double-count capacity. One explicit selector beats a clever generic one: + +```promql +node_filesystem_avail_bytes{mountpoint=~"/volume[0-9]+|/mnt/ssd|/mnt/disk[0-9]+"} ``` -And the notifications **still failed.** Same 400. My proud fix did nothing. +## Noise is not harmless -### Round 2: I picked the wrong IP +The cost of a bad threshold isn't the alert itself — it's the *training*. Every alert +that fires while nothing is wrong teaches you to skim, then to ignore, and then the one +that matters arrives in a channel you've stopped reading. My rule count went **down** +while coverage went up: fewer, each tied to a named consequence. -Here's a mistake worth remembering: I hard-coded `149.154.167.220` from -memory. When I finally queried a public resolver, the real `A` record for -`api.telegram.org` was different: +## Thresholds are personal, shapes are not -``` -149.154.166.110 api.telegram.org # what DNS actually says -149.154.167.220 api.telegram.org # what I hard-coded -``` +The specific numbers above are mine — 25 GB free before a NAS volume bothers me, 512 MB +available before I care about memory, 50% steal before a VPS is being starved. Yours +will differ with your hardware and your tolerance. -`167.220` is *inside* Telegram's IP range (`149.154.160.0/20`), so it's not -"wrong" in the sense of being somebody else's server — but it's not the -active endpoint, and bot-API traffic to it behaved unpredictably. I switched -to `166.110`. It still failed. So the IP wasn't the whole story either. +What transfers is the shape: -### Round 3: the same command, two different outcomes - -This was the moment the whole thing got weird. Inside one container, at -almost the same second: - -- the **long-running monitor process** (`PID 1`) → `sendPhoto` failed, 400 -- a **fresh `docker exec` process** running identical code → `sendPhoto` - succeeded, 200 - -Same container. Same code. Same payload. Same second. - -I ruled out token, chat_id, emoji in the caption, Unicode quotes, image -format, environment variables, and the image URL itself — I download-tested -the photo and it was a perfectly valid JPEG. Nothing in the code differed -between the two paths. - -When you hit a contradiction like that — "identical inputs, opposite -outputs" — the answer is usually that the inputs *aren't* identical. But -finding the difference took one more frame change. - -### Round 4: `nginx/1.30.1` is Telegram - -I re-read the raw 400 response instead of assuming. The response was an HTML -"400 Bad Request" page, and the `Server` header was `nginx/1.30.1`. - -**That nginx is Telegram's own edge.** Telegram's API sits behind nginx -(version 1.30.1), and when a request is malformed *before* it reaches their -application layer, nginx itself answers with a plain 400 page — no JSON, no -friendly error code. - -That reframed everything. The request wasn't being intercepted. It was -**reaching Telegram, and Telegram's nginx was rejecting it as malformed.** - -And the malformed thing was the one part I hadn't scrutinized: the -`sendPhoto` multipart body. - -## The fix - -My original code built the multipart body like this — the kind of -"obviously fine" line that ships a subtle bug: - -```python -body = b"\r\n".join(body_lines) -``` - -That `join` looks convenient, but a JPEG is **binary data**. Its bytes include -`\r\n` byte sequences in the middle of the image data. Joining with `\r\n` -as the delimiter corrupts the exact bytes that are supposed to be the image -payload. Telegram receives a body whose multipart boundaries are broken, so -its nginx rejects the request with a 400 before the bot API ever parses it. - -The correct approach is to assemble the body field by field, keeping binary -data untouched: - -```python -boundary = "----tg" + token_hex(8) - -parts = [] -parts.append(f"--{boundary}\r\n".encode()) -parts.append(b'Content-Disposition: form-data; name="chat_id"\r\n\r\n') -parts.append(f"{chat_id}\r\n".encode()) - -parts.append(f"--{boundary}\r\n".encode()) -parts.append(b'Content-Disposition: form-data; name="caption"\r\n\r\n') -parts.append(f"{caption}\r\n".encode()) - -parts.append(f"--{boundary}\r\n".encode()) -parts.append( - b'Content-Disposition: form-data; name="photo"; ' - b'filename="image.jpg"\r\nContent-Type: image/jpeg\r\n\r\n' -) -parts.append(image_bytes) # ← binary stays binary -parts.append(b"\r\n") - -parts.append(f"--{boundary}--\r\n".encode()) - -body = b"".join(parts) # join bytes, not lines -``` - -The difference is `b"".join(parts)` instead of `b"\r\n".join(...)`: each part -is already a complete chunk with its own framing, and we concatenate them -verbatim rather than inserting a delimiter *between every element*. - -There's a second, higher-level fix worth naming: **stop asking Telegram to -download the image for you.** My earlier code passed the image as a *URL* and -let Telegram fetch it: - -```python -tg("sendPhoto", {"photo": image_url, "caption": caption}) -``` - -That means Telegram's servers have to reach out to the source CDN — and if -that CDN is flaky or geo-blocks Telegram's crawlers, you get intermittent -failures that are impossible to reproduce locally. Downloading the image -yourself, then uploading the bytes, removes an entire class of flakiness and -gives you a chance to validate the bytes before you send them. - -## What I'd do differently - -1. **Read the raw response, not the assumption.** `nginx/1.30.1` sat in logs - for hours while I chased phantom proxies. Look up who owns the header - before inventing a man-in-the-middle. -2. **Verify a hard-coded IP against a live resolver.** Memory is not DNS. - `dig +short api.telegram.org` takes three seconds and would have saved me - a whole round. -3. **Treat "identical inputs, opposite outputs" as a lie.** The inputs were - never identical — the two processes differed somewhere I wasn't looking. - The honest move is to diff the bytes, not the code. -4. **Never hand-join binary data with a text delimiter.** `b"\r\n".join()` on - a mix of text and JPEG is a corruption bug waiting to happen. +- absolute units over ratios, +- the consequence written into the alert's summary, +- a selector that states exactly which mounts it speaks for, +- and a rule count small enough that you still read every message. ## The result -After the multipart fix, the monitor pushed a real notification through on -the next tick — image and all. I also folded in a retry queue so a failed -notification stays marked un-sent and is retried 30 seconds later instead of -silently vanishing. The whole episode cost the better part of a day, but it -turned into exactly the kind of bug I'll never trip over twice. +The same stack that was producing three permanent, uninformative warnings now runs 16 +rules that stay silent when the lab is healthy — and one of them immediately surfaced a +NAS volume sitting at 21 GB free, which is the kind of thing a percentage rule had been +hiding behind a 99%-full giant that I'd learned to ignore. ---- +## Want this for your business? -*The bot is a personal project, but the lesson generalizes to any service -that relies on outbound notifications. If you've got a monitor, a scraper, or -an alert pipeline that needs to reach people reliably, let's talk.* +If your monitoring sends you alerts you've learned to ignore, or you have none and would +rather find out about a full disk from a rule than from a failed backup, I set up +self-hosted monitoring and alerting (Prometheus + Grafana, alerts to Telegram and email) +with thresholds tied to what actually breaks. -**I build reliable notification and monitoring pipelines — Telegram/WhatsApp -alerts, scrapers, and the infrastructure around them.** Tell me what you need -to watch and where you need it delivered: +**WhatsApp: [+60 12-797 2969](https://wa.me/60127972969)** · **Email: [me@hoelee.com](mailto:me@hoelee.com?subject=Monitoring%20and%20alerting)** · **[hoelee.com](https://hoelee.com)** -- [WhatsApp me →](https://wa.me/60127972969?text=Hi%20Hoelee%2C%20I%20need%20a%20monitoring%2Fnotification%20pipeline) -- [Email me →](mailto:me@hoelee.com?subject=Monitoring%20%2F%20notification%20pipeline) -- [hoelee.com](https://www.hoelee.com) +Website design and development is my main line of work; server hardening and +self-hosted infrastructure is the other half of it. +```` + +--- + +## 文章 04 / 11：浏览器翻译没有 API:我给一个纯中文网页应用加上了英文模式 +- **Slug**: `adding-english-mode-to-a-chinese-only-web-app` +- **英文标题**: No API for Browser Translation: Adding an English Mode to a Chinese-Only Web App +- **分类**: `engineering` | **发布日期**: 2025-06-11 +- **线上地址**: https://blog.hoelee.com/posts/adding-english-mode-to-a-chinese-only-web-app/ (EN) | https://blog.hoelee.com/posts/zh/adding-english-mode-to-a-chinese-only-web-app/ (ZH) +- **本篇审核状态**: ⏳ 待审核 (Pending) + +### ✍️ 顾问审阅意见区 (Review Feedback) +```text +【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 +【事实与数字核对】： +【人设与用词卫生】： +【CRO与转化建议】： +``` + +### 🇨🇳 中文版原文 (Chinese Markdown Twin) +````markdown +--- +title: "浏览器翻译没有 API:我给一个纯中文网页应用加上了英文模式" +description: "浏览器不提供触发翻译的 JavaScript API,而我的应用明明是中文却声明自己是英文。于是我改用反向代理,把一个英文模式注入进去。" +pubDate: 2025-06-11 +updatedDate: 2026-09-29 +category: engineering +blueprint: sme-digital-suite +blueprintLayer: L1-high-conversion-web +tags: ["i18n", "nginx", "vue", "javascript", "self-hosting"] +ogImage: "/og/zh-adding-english-mode-to-a-chinese-only-web-app.jpg" +banner: "/banners/adding-english-mode-to-a-chinese-only-web-app.jpg" +draft: false +--- + +「能不能用 JavaScript 触发浏览器自带的翻译?」答案是**不能**。接受这一点花了我一个下午 —— 但更有意思的发现出现在前面:我的应用其实一直在破坏浏览器的翻译功能,而且只用一个属性就做到了。 + +## 我在做什么 + +我维护着一个私人的邀请制阅读平台 `book.hoelee.com`:自建的阅读器(Kotlin/Spring 后端 + Vue 单页应用),装着我自己的藏书,并给受邀读者开了账号。界面是纯中文 —— 不是「以中文为主」,而是**只有中文**。我在相信这一点之前把每个前端 bundle 都翻了一遍:没有 i18n 库、没有语言包、没有语言切换,只有一个简繁转换表和一堆英文 TTS 语音名。 + +后来我开始把邀请卡发给只说英文的读者。他们能注册,然后迎面撞上:书架、浏览书仓、确定、设置 —— 每一个控件都在他们读不懂的语言里。这不是「体验不够精致」的问题。对我邀请的一半人来说,这个产品是不可用的。 + +有一条约束决定了后面所有做法:**我不改这个应用。** 它的上游仓库已经归档,界面是压缩过的 Vue bundle —— 任何我打进镜像里的补丁,都会在下一次换镜像时消失。所以我做的东西必须待在**外面一层**:我已经架在应用前面的那个 nginx 网关。 + +## 发现一:应用声明了自己错误的语言,而这会关掉翻译 + +写代码之前,我先看看浏览器到底看到了什么: + +```bash +$ curl -s https://book.hoelee.com/index.html | grep -o ']*>' + +``` + +外壳声明英文,界面 100% 中文。 + +这件事比看上去严重。Chrome、Edge、Safari 决定「要不要**主动提示**翻译」,很大程度取决于页面声明的语言。一个声明自己是英文、却渲染中文内容的页面,不会得到那个「要翻译此页面吗?」的提示 —— 对我的新读者来说最有用的一项无障碍功能,就这么被悄悄关掉了;而「右键 → 翻译成中文/英文」是大多数人从来不知道的操作。 + +修复只要一行,而且是作用在**响应出站时**,不是打进应用里: + +```nginx +location = /index.html { + # sub_filter 不能改写 gzip 后的内容 -> 让上游返回未压缩的 HTML + proxy_set_header Accept-Encoding ""; + sub_filter_once on; + # 应用声明 lang="en" 而界面是中文 -> 浏览器因此永远不提示中译英。 + # 说出事实。 + sub_filter '' ''; + proxy_pass http://hectorqin-reader:8080; +} +``` + +用发现它的方式验证:`curl -s https://book.hoelee.com/index.html | grep -o '' ''; + proxy_pass http://hectorqin-reader:8080; +} +``` + +整个部署就是这样:一个 40 KB 的脚本,只在 `/index.html` 加载,所以按钮只存在于应用内,别处都没有。脚本做三件事。 + +### 1. 从应用自己的 bundle 里「收割」词汇表 + +界面文案本来就在压缩后的 JavaScript 里,以字符串字面量的形式存在。把所有含中日韩字符的字符串抓出来并计数: + +```python +import re, collections +CJK = re.compile(r'[\u4e00-\u9fff]') +count = collections.Counter() +for bundle in bundles: + for m in re.finditer(r'"([^"\\\n]{2,40})"', bundle): + if CJK.search(m.group(1)): + count[m.group(1)] += 1 +print(len(count)) # 840 条唯一字符串,按出现频率排序 +``` + +跑出 840 条唯一字符串:菜单(书架、书源管理)、确认框(确认要删除所选择的书籍吗?)、设置项(段落行距)、错误信息(本地书籍源文件不存在)。词典最终是 **871 条** —— 收割来的,加上我在英文模式下实际使用应用、看还有什么没翻而补上的。 + +### 2. 按 DOM 的真实样子匹配,而不是你希望的样子 + +朴素的子串替换会毁掉中文词组:书架 就藏在 加入书架 里面,单字更糟(页 藏在 页面 里)。三条规则解决了它: + +- **先做整标签匹配,基于去掉空白后的文本。** 应用里的标签是 `" 书架 "` 这种带空格的,所以词典里的 `设置` 必须能匹配 DOM 里的 `" 设置 "`。 +- **然后是最长优先的子串替换**,只允许长度 ≥ 2 的键参与。 +- **单字键只做整标签匹配**(章、页、条),这样它们永远不可能破坏更长的词。 + +```js +function tr(s) { + if (!s || !CJK.test(s)) return s; + var hit = EXACT[s.replace(/[\s\u3000]/g, '')]; // 整标签匹配,空白归一化 + if (hit) return keepWS(s, hit); + var t = s; + for (var i = 0; i < KEYS.length; i++) { // KEYS:最长优先,长度 >= 2 + if (t.indexOf(KEYS[i]) >= 0) t = t.split(KEYS[i]).join(DICT[KEYS[i]]); + } + // 收拾数量短语:共58个可用书源 -> "58 usable sources" + return t.replace(/共\s*(\d+)/g, '$1').replace(/(\d+)\s*个/g, '$1').replace(/(\d)([A-Za-z])/g, '$1 $2'); +} +``` + +### 3. 跟上框架的重绘 + +它是 Vue 应用,DOM 一直在被重写 —— 只跑一次的翻译,一秒后就有一半是错的。一个带 150 ms 防抖的 `MutationObserver`,在任何变化之后重跑一遍遍历。 + +这一遍是幂等的:标签一旦变成英文就不再含中日韩字符,于是不会被再写一次,也不会形成写入循环。文本节点之外,`placeholder`、`title`、`aria-label`、`alt` 和输入框的 `value` 也会翻译 —— 所以提示气泡和搜索框也一起变英文了。 + +UI 是右下角的一个小胶囊:关闭时显示 `EN`,开启后显示 `中文`,选择记在 `localStorage`,还提供「隐藏此按钮」给更愿意用自己翻译器的读者。 + +## 它**刻意**不翻译什么 + +**书籍正文。** 阅读器把 EPUB 内容渲染在 iframe 里,我的遍历器按设计跳过 iframe —— 而且我无论如何也不会去机器替换一本小说的正文。 + +同样保持原样的还有:书名与作者、书源名称、我自己建的分组名(梯子、精品、正版),以及应用那句诗。这些是**数据**,不是界面。翻译界面是一种体贴;悄悄改写用户自己的内容是另一回事,而且糟糕得多。 + +书籍正文仍然交给浏览器自带的翻译 —— 而且因为「发现一」,它现在真的会被提示出来。按钮的弹窗里就是这么写的。 + +## 要量,不要用眼睛看 + +「截图看起来基本是英文了」不算结果。我在开关按钮的前后,数了实时 DOM 里含中日韩字符的文本节点: + +```js +let cjk = 0, total = 0; // 遍历器跳过注入的 UI 与 script/style +while ((n = walker.nextNode())) { + const t = n.nodeValue.trim(); + if (!t) continue; + total++; + if (/[\u4e00-\u9fff]/.test(t)) cjk++; +} +``` + +| 界面 | 之前 | 之后 | 已翻译 | +|---|---|---|---| +| 登录 / 注册对话框 | 80 | 12 | 85% | +| 书架 + 设置 | 259 | 60 | 77% | +| 阅读页界面 | 214 | 31 | 88% | + +那些界面里仍被计为中文的,全部是数据 —— 书名、作者、书源名、我自己的分组名、那句诗 —— 正是应该保持中文的部分。 + +## 我会怎么做得不一样 + +**不要用手工重写一份 871 条的词典。** 我重构过一次,静默丢了 25 个键。症状看起来像逻辑 bug:设置 还是中文,而它的每个兄弟标签都翻译正常。匹配器没问题 —— 是**条目**不见了。在你相信一次重写之前,用程序对比键集合: + +```bash +python -c "print(sorted(set(old_keys) - set(new_keys)))" +``` + +**自动化 UI 检查要写短。** 我的无头运行器每次调用硬超时 60 秒。一个遍历六本书的循环返回 HTTP 408 和空 body,而我的 JSON 解析器报的是「解析错误」而不是「超时」—— 我一度在查错的东西。 + +**调试之前先搞清楚 SPA 的真实路由。** `#/reader/1` 会渲染出一个空白外壳,让人以为应用坏了;真正的路由是 `#/reader?bookUrl=`。 + +**注意你自己的跳转逻辑。** 我的网关会把任何刷新送回前台页,除非设置了 `sessionStorage` 的通行标记。把界面切**回**中文会重载页面 —— 所以按钮会先设好同一个标记,否则读者会因为改了个语言设置被踢出应用。 + +## 结果 + +一个只懂英文的读者,现在打开平台、点一下按钮,就能用英文读界面。应用本身没有被改动:整个功能只是一个由 sidecar nginx 容器提供的 40 KB 脚本,所以下一次换镜像不可能弄坏它;而浏览器自带的翻译,终于有一个会声明自己真实语言的页面可以配合。 + +我留下的经验:**当平台不给你 API 时,往外看一层。** 反向代理是添加「应用里加不了的行为」的正当位置 —— 而且通常是唯一能在升级中存活的位置。 + +--- + +*我为马来西亚的中小企业搭建并自托管这类平台 —— 网页应用、私人书库、内部工具。有需要可以找我:[WhatsApp +60 12-797 2969](https://wa.me/60127972969)、[me@hoelee.com](mailto:me@hoelee.com?subject=Self-hosted%20platform),或 [hoelee.com](https://www.hoelee.com)。* +```` + +### 🇬🇧 英文版原文 (English Markdown Primary) +````markdown +--- +title: "No API for Browser Translation: Adding an English Mode to a Chinese-Only Web App" +description: "Browsers expose no JavaScript API to trigger translation — and my app declared English while being Chinese. How I injected an English mode at the reverse proxy." +pubDate: 2025-06-11 +updatedDate: 2026-09-29 +category: engineering +blueprint: sme-digital-suite +blueprintLayer: L1-high-conversion-web +tags: ["i18n", "nginx", "vue", "javascript", "self-hosting"] +ogImage: "/og/adding-english-mode-to-a-chinese-only-web-app.jpg" +banner: "/banners/adding-english-mode-to-a-chinese-only-web-app.jpg" +draft: false +--- + +Can I trigger the browser's translate feature from JavaScript? No. That answer cost me an afternoon to accept — but the more interesting discovery came first: my app was already sabotaging the browser's translation, and it did it with a single attribute. + +## What I was working on + +I run a private, invite-only reading platform at `book.hoelee.com`: a self-hosted reader (Kotlin/Spring backend, Vue single-page app) holding my library, with accounts for invited readers. The interface is Chinese-only. Not "Chinese-first" — *only*. I grepped every served bundle before believing it: no i18n library, no locale files, no language switcher. Just a Simplified/Traditional character converter and a list of English TTS voice names. + +Then I started handing out invite cards to English-speaking readers. They could register, and then they met 书架, 浏览书仓, 确定, 设置 — every control in a language they don't read. That isn't a polish problem. For half the people I invited, the product was unusable. + +One constraint shaped everything that follows: **I don't modify the app.** Its upstream repository is archived and the UI is a minified Vue bundle, so anything I patch inside the image dies the next time I swap the image. Whatever I build has to live one layer out — in the nginx gateway I already run in front of the app. + +## Finding 1: the app declares the wrong language, and that disables translation + +Before writing a single line of code, I looked at what the browser sees: + +```bash +$ curl -s https://book.hoelee.com/index.html | grep -o ']*>' + +``` + +The shell says English. The interface is 100% Chinese. + +That matters more than it looks. Chrome, Edge and Safari decide whether to *offer* translation largely from the page's declared language. A page that declares English while rendering Chinese text doesn't get the "Translate this page?" prompt — the single most useful accessibility feature for my new readers was silently switched off, and right-click → *Translate to English* is a ritual most people never learn. + +The fix is one line, applied to the response on the way out instead of inside the app: + +```nginx +location = /index.html { + # sub_filter cannot rewrite a gzipped body -> ask upstream for plain HTML + proxy_set_header Accept-Encoding ""; + sub_filter_once on; + # the app declares lang="en" while its UI is Chinese -> browsers then never + # offer Chinese -> English translation. Declare the truth. + sub_filter '' ''; + proxy_pass http://hectorqin-reader:8080; +} +``` + +Verify the same way you found it: `curl -s https://book.hoelee.com/index.html | grep -o '' ''; + proxy_pass http://hectorqin-reader:8080; +} +``` + +That's the whole deployment: a 40 KB script, loaded only on `/index.html`, so the button exists inside the app and nowhere else. The script does three things. + +### 1. Harvest the vocabulary from the app's own bundles + +The UI strings are already in the minified JavaScript as quoted literals. Extract every one that contains a CJK character and count them: + +```python +import re, collections +CJK = re.compile(r'[\u4e00-\u9fff]') +count = collections.Counter() +for bundle in bundles: + for m in re.finditer(r'"([^"\\\n]{2,40})"', bundle): + if CJK.search(m.group(1)): + count[m.group(1)] += 1 +print(len(count)) # 840 unique strings, most frequent first +``` + +840 unique strings came out: menus (书架, 书源管理), confirmations (确认要删除所选择的书籍吗?), settings labels (段落行距), error messages (本地书籍源文件不存在). The dictionary ended up at **871 entries** — the harvest plus everything I found by running the app in English mode and watching what stayed Chinese. + +### 2. Match like the DOM actually is, not like you wish it were + +Naive substring replacement mangles Chinese compounds: 书架 sits inside 加入书架, and single characters are worse (页 inside 页面). Three rules fixed it: + +- **Whole-label match first, on whitespace-stripped text.** The app stores labels like `" 书架 "` with padding, so a dictionary key of `设置` has to match `" 设置 "` in the DOM. +- **Then longest-first substring replacement**, restricted to keys of two or more characters. +- **One-character keys are whole-label only** (章, 页, 条), so they can never corrupt a longer word. + +```js +function tr(s) { + if (!s || !CJK.test(s)) return s; + var hit = EXACT[s.replace(/[\s\u3000]/g, '')]; // whole label, whitespace-normalised + if (hit) return keepWS(s, hit); + var t = s; + for (var i = 0; i < KEYS.length; i++) { // KEYS: longest-first, length >= 2 + if (t.indexOf(KEYS[i]) >= 0) t = t.split(KEYS[i]).join(DICT[KEYS[i]]); + } + // tidy count phrases: 共58个可用书源 -> "58 usable sources" + return t.replace(/共\s*(\d+)/g, '$1').replace(/(\d+)\s*个/g, '$1').replace(/(\d)([A-Za-z])/g, '$1 $2'); +} +``` + +### 3. Keep up with the framework + +It's a Vue app, so the DOM is rewritten constantly — a translation pass that runs once is a translation pass that's half wrong a second later. A `MutationObserver` with a 150 ms debounce re-runs the walk after any change. + +The pass is idempotent: once a label is English it contains no CJK, so nothing is rewritten and there is no write loop. Text nodes plus `placeholder`, `title`, `aria-label`, `alt` and input `value` attributes all get translated, which is why tooltips and the search box come along too. + +The UI is a small pill in the bottom-right corner: `EN` when off, `中文` when on, remembered in `localStorage`, with a "Hide this button" link for readers who prefer their own translator. + +## What it deliberately does not translate + +**Book text.** The reader renders EPUB content inside an iframe; my walker skips iframes by design, and I would never machine-swap a novel's prose anyway. + +Also left alone: book titles and authors, book-source names, and the group names I created myself (梯子, 精品, 正版), plus the app's poem tagline. Those are *data*, not interface. Translating the chrome is a courtesy; silently rewriting a user's own content is a different and much worse idea. + +For book text the browser's own translation is still the right tool — and it now actually gets offered, thanks to Finding 1. The button's popup says exactly that. + +## Measure it, don't eyeball it + +"Screenshots look mostly English" is not a result. I counted CJK-containing text nodes in the live DOM before and after toggling the button: + +```js +let cjk = 0, total = 0; // walker skips the injected UI + script/style +while ((n = walker.nextNode())) { + const t = n.nodeValue.trim(); + if (!t) continue; + total++; + if (/[\u4e00-\u9fff]/.test(t)) cjk++; +} +``` + +| Screen | Before | After | Translated | +|---|---|---|---| +| Login / register dialog | 80 | 12 | 85% | +| Bookshelf + settings | 259 | 60 | 77% | +| Reading page chrome | 214 | 31 | 88% | + +Everything still counted as Chinese on those screens is data — titles, authors, source names, my own group names, the tagline — which is exactly what should stay Chinese. + +## What I'd do differently + +**Don't rewrite an 871-entry dictionary by hand.** I restructured mine once and silently dropped 25 keys. The symptom looked like a logic bug: 设置 stayed Chinese while every sibling label translated correctly. The matcher was fine — the *entry* was gone. Diff the key sets programmatically before you trust a rewrite: + +```bash +python -c "print(sorted(set(old_keys) - set(new_keys)))" +``` + +**Keep automated UI checks short.** My headless runner hard-times-out at 60 seconds per call. A six-book loop came back HTTP 408 with an empty body, and my JSON parser reported a parse error rather than a timeout — I debugged the wrong thing for a while. + +**Learn the SPA's real routes before debugging it.** `#/reader/1` renders a blank shell and makes you think the app is broken; the actual route is `#/reader?bookUrl=`. + +**Mind your own bounce logic.** My gateway sends any reload back to the front page unless a `sessionStorage` pass flag is set. Switching the interface *back* to Chinese reloads the page — so the button sets that same flag first, or the reader gets thrown out of the app for changing a language setting. + +## Result + +An English-only reader now opens the platform, taps one button, and reads the interface in English. The app itself is untouched: the whole feature is a 40 KB script served by a sidecar nginx container, so the next image update cannot break it, and the browser's own translation has a page that declares its real language to work with. + +The lesson I'm keeping: **when the platform won't give you an API, look one layer out.** A reverse proxy is a legitimate place to add behaviour you can't add in the app — and it's usually the only place that survives upgrades. + +--- + +*I build and self-host platforms like this one — web apps, private libraries, internal tools for small businesses in Malaysia. If you need something similar: [WhatsApp +60 12-797 2969](https://wa.me/60127972969), [me@hoelee.com](mailto:me@hoelee.com?subject=Self-hosted%20platform), or [hoelee.com](https://www.hoelee.com).* +```` + +--- + +## 文章 05 / 11：排查 Remotion 视频抽搐：为什么在背景图挂载音频低频 Scale 会毁了转场 +- **Slug**: `debugging-remotion-video-jitter` +- **英文标题**: Debugging Remotion Video Jitter: Why Bass Scaling Ruins Background Transitions +- **分类**: `engineering` | **发布日期**: 2025-11-26 +- **线上地址**: https://blog.hoelee.com/posts/debugging-remotion-video-jitter/ (EN) | https://blog.hoelee.com/posts/zh/debugging-remotion-video-jitter/ (ZH) +- **本篇审核状态**: ⏳ 待审核 (Pending) + +### ✍️ 顾问审阅意见区 (Review Feedback) +```text +【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 +【事实与数字核对】： +【人设与用词卫生】： +【CRO与转化建议】： +``` + +### 🇨🇳 中文版原文 (Chinese Markdown Twin) +````markdown +--- +title: "排查 Remotion 视频抽搐：为什么在背景图挂载音频低频 Scale 会毁了转场" +description: "深入排查 Remotion 音频可视化中背景画面剧烈抽搐与切镜暗闪的根本原因：解耦 HUD 与背景图层，以及纯净单向叠化与硬件基准实测对比。" +pubDate: 2025-11-26 +updatedDate: 2026-10-22 +category: engineering +stream: creative-lab +tags: ["remotion", "react", "audio-visualizer", "css-animation", "debugging"] +ogImage: "/og/zh-debugging-remotion-video-jitter.jpg" +banner: "/banners/debugging-remotion-video-jitter.jpg" +draft: false +--- + +使用 React 和 Remotion 编写自动化视频赋予了开发者对每一帧画面的极致掌控。你可以直接将 Web Audio 的频域分析数据映射到 SVG 属性、CSS 变换矩阵以及排版样式上，这种灵活性是传统非线性剪辑软件很难比拟的。 + +然而，将视频帧当作常规网页来渲染时，极易踩入微妙的视觉陷阱。在我们的自动化音乐视频生成管线中，一个在开发组件预览时看似完全正常的设计，在全量导出为 1080P 30fps 视频后，却出现了极其刺眼、让人目眩的“地震式剧烈抽搐”。 + +本文记录这次视觉 BUG 的深度排查过程：为什么对音频动效的朴素直觉会毁了整部视频，以及如何重构 React 视频图层架构以实现丝滑稳定的影视级转场。 + +> **线上成品验证**：修复后的渲染流水线已正式交付并发布至 YouTube：[《糖画小仙》1080p 官方 MV](https://youtu.be/vZbyKeCCSRA)，全片背景稳固、转场平滑。 + +> **TL;DR** —— 让整幅背景图跟着低频段缩放，等于每一帧都重新做图像插值采样，肉眼读到的就是剧烈抖动与震颤；再加上两层半透明图层对称交叉淡化，在时间中点必然透出底层的黑色容器。修复方案：背景画面绝对静态，将所有音频动效彻底隔离在顶层 HUD 频谱层，转场采用「100% 实体底图 + 顶层单向淡入」。渲染耗时实测**缩短 38.0%（从 352 秒降至 218 秒，吞吐速度提升 1.61 倍）**。这一篇是 [全自动 MV 生成流水线](/posts/zh/how-i-built-an-automated-music-video-pipeline/) 的渲染篇；文件落盘与发布那一半见 [Docker 容器大文件上传](/posts/zh/uploading-large-videos-to-youtube-via-cdp/)。 + +--- + +## 1. 故障现象：全画面的“高频地震” + +该视频合成组件最初的视觉设想非常清晰： +1. 底层是一组 1080P 高清插画分镜，在不同章节间慢速淡入淡出（Crossfade）； +2. 画面上方浮动一个跟随音乐重音律动的环形音频频谱仪； +3. 三行书法歌词居中显示在音频环内，并根据每张插画的主体位置在画面左右两侧平滑滑移避让。 + +在开发工具里单帧点按检查时，画面一切正常。但在以 30fps 正常播放渲染出的成片时，肉眼观感极度难受：每当强烈的电子重音鼓点响起，整张背景图每秒剧烈颤抖震荡数十次；不仅如此，在两张图片交替切镜的 1.8 秒窗口内，画面中央还会突然闪现明显的“暗黑闪烁”，亮度瞬间下沉。 + +--- + +## 2. 根因分析一：低频 Scale 能量污染背景图 + +翻阅最初的 `BackgroundSlideshow.tsx` 组件，我们发现了一段看似合情合理的动效代码： + +```tsx +// 埋下灾难的罪魁祸首代码 +const bassScale = interpolate(bassLevel, [0, 1], [1.0, 1.025], { + extrapolateRight: 'clamp', +}); + +// 应用在全屏背景图片上： + +``` + +在直觉中，重低音响起时让背景图随着节奏轻微放大 2.5%，听起来像是在模拟动感的镜头呼吸感。然而在真实物理渲染中： +- **电音鼓点的瞬态脉冲只有 30~60 毫秒**：低频能量在连续 2 到 3 帧之内瞬间从零冲上顶峰，又在下一帧骤跌回原点； +- **1920x1080 的巨幅背景图放大 2.5%，边缘物理像素瞬间位移多达 25 到 30 像素**； +- 这种在相邻帧之间急剧拉扯的剧烈像素位移，人眼接收到的根本不是“音乐律动”，而是被视网膜本能判定为**强烈的镜头颠簸、抽动与丢帧残影**； +- 更致命的是，在 1.8 秒的交叉淡入淡出期间，两张重叠的图片各自具有不同的锚点和缩放系数，两层图像以不同的相位同时高频抖动，产生了极其混乱的重影眩晕感。 + +--- + +## 3. 根因分析二：对称线性透明度重叠时的“黑闪”陷阱 + +第二个问题是切镜时的亮度下陷黑闪。在传统 Web CSS 中，人们通常很自然地写出： + +```css +/* 常见的渐变叠化陷阱 */ +.current-slide { opacity: 1 -> 0; } +.next-slide { opacity: 0 -> 1; } +``` + +当两张图片都以线性渐变进行透明度变换时，在正中间那颗时间点（$t = 0.5$）： +- 当前图片透明度 = 0.5 +- 下一张图片透明度 = 0.5 + +来看浏览器和 Remotion 的合成器是如何将这几层叠在底层的纯黑容器之上的： + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Layer 2: 下一张图片（Next Slide，透明度 = 0.5） │ +├─────────────────────────────────────────────────────────────┤ +│ Layer 1: 当前图片（Current Slide，透明度 = 0.5） │ +├─────────────────────────────────────────────────────────────┤ +│ Layer 0: 根容器底色（Root Container，背景色 = #000000 纯黑） │ +└─────────────────────────────────────────────────────────────┘ +``` + +由于多层半透明通道是依次向下进行乘法 Alpha 混合： +$$\text{有效综合不透明度} = 1 - (1 - 0.5) \times (1 - 0.5) = 1 - 0.25 = 0.75$$ + +合成后的总有效透明度只有 **0.75**，这意味着多达 **25% 的底层纯黑背景**被暴露了出来！这不是错觉，而是纯粹的数学光度衰减。每一次转场切镜，画面整体照度都会凭空跌掉四分之一，造成非常明显的瞬间暗淡闪烁。 + +--- + +## 4. 重构方案：稳固基座 + 纯净单向叠化 + +找到症结后，我们对组件进行了彻底重构： + +### 1. 彻底剥离背景图层的所有音频缩放 +背景插画必须稳如磐石。所有的音频跳动、瞬态光斑与能量爆发，**只能且必须限定在顶层的 HUD 频谱仪中**。背景层坚决不承载任何高频振荡属性。 + +### 2. 底层 100% 实体 + 顶层单向平滑淡入 +坚决不要让底图淡出。**当前图片始终保持 100% 不透明**，下一张图片直接覆盖在其上方，慢速执行 `0 -> 1.0` 的单向淡入： + +```tsx +return ( +
+ {/* 1. 底层图片：坚固不透明，彻底消除暗黑闪烁 */} + + + {/* 2. 顶层图片：在 1.8 秒内平滑淡入 (0 -> 1.0) */} + {isTransitioning && ( + + )} +
+); +``` + +### 3. HUD 与背景物理层级绝对隔离 +歌词弹幕与音频环容器设置在独立的顶层图层中，并赋予 `pointerEvents: 'none'`。歌词在留白侧之间的横向位移完全根据当前时间戳自主计算，与底层的图片渐变完全物理隔离，互不干涉。 + +--- + +## 5. 改造后的成效与基准复现 + +重构落地后，对比极其显著： + +1. **彻底消除画面抖动**：无论音乐鼓点多么狂暴密集，背景画面在所有测试分镜中纹丝不动，人物与细节清晰稳定； +2. **转场不再发生黑闪**：1.8 秒的单向贝塞尔叠化始终保持 100% 连续亮度，杜绝了中间态透出底层黑色的缺陷； +3. **渲染吞吐速度大幅提升**：移除了对超大尺寸背景图的高频矩阵重算后，渲染性能获得了实测提升。 + +### 基准实测与硬件复现条件 + +| 评估指标 | 优化前版本 | 优化后版本 | 变化幅度与提升 | +|---|---|---|---| +| **渲染总耗时** | 352.0 秒 | 218.3 秒 | **耗时减少 38.0% (-133.7s)** | +| **平均渲染帧率** | 22.24 fps | 35.87 fps | **吞吐速度提升 1.61 倍** | +| **总计处理帧数** | 7,830 帧 | 7,830 帧 | 完全一致（261 秒 / 4 分 21 秒） | +| **输出视频规格** | 1920 × 1080 (1080p @ 30fps) | 1920 × 1080 (1080p @ 30fps) | 完全一致 | +| **编码器设置** | H.264 (CRF 18, preset medium) | H.264 (CRF 18, preset medium) | 完全一致 | + +**基准测试宿主环境：** +- **CPU**：Intel Core i9-13900K（24 核 / 32 线程，最高睿频 5.8 GHz） +- **GPU**：NVIDIA GeForce RTX 3060（12GB GDDR6 显存，驱动版本 551.xx） +- **内存**：64GB DDR5 5600 MHz +- **系统**：Windows 11 专业版 64 位 +- **软件栈**：Node.js v20.x，Remotion v4.0.x，开启 Chromium 无头 GPU 加速 + +两次基准测试均在同一工作站、连续以完全相同的音频频谱输入和插画素材进行。渲染耗时缩短 38.0%，即吞吐速度达到优化前的 **1.615 倍**，使原本慢于真实播放时长的渲染变为高于实时帧率的高效输出。 + +--- + +## 本专题关联阅读 +- **系统总架构篇**：[我是如何用 Remotion 与 Docker CDP 打造全自动 MV 生成与多语种油管发布流的](/posts/zh/how-i-built-an-automated-music-video-pipeline/) +- **大文件上传篇**：[借助 Docker Archive API 模式通过 CDP 自动化上传 100MB+ 视频到 YouTube](/posts/zh/uploading-large-videos-to-youtube-via-cdp/) +- **线上实成片**：[前往 YouTube 观看《糖画小仙》1080P 官方完整成片](https://youtu.be/vZbyKeCCSRA) +```` + +### 🇬🇧 英文版原文 (English Markdown Primary) +````markdown +--- +title: "Debugging Remotion Video Jitter: Why Bass Scaling Ruins Background Transitions" +description: "Diagnose audio-reactive background jitter and crossfade flicker in Remotion. See how separating the HUD from background layers can improve visual stability and rendering performance." +pubDate: 2025-11-26 +updatedDate: 2026-10-22 +category: engineering +stream: creative-lab +tags: ["remotion", "react", "audio-visualizer", "css-animation", "debugging"] +ogImage: "/og/debugging-remotion-video-jitter.jpg" +banner: "/banners/debugging-remotion-video-jitter.jpg" +draft: false +--- + +Building programmatic videos with React and Remotion gives you complete DOM control over every frame. You can map web audio frequencies directly to SVG attributes, CSS transforms, and typography in ways standard video editors can barely touch. + +However, treating video frames like standard web pages introduces subtle visual bugs. In our automated music video pipeline, an export that looked plausible in unit tests developed a jarring, violent shudder when rendered into a 1080p MP4. + +Here is what went wrong, why our initial intuition about audio reactivity failed, and how to structure React video layers so audio-driven HUDs never pollute background transitions. + +> **Live Reference**: The repaired rendering pipeline produced the final production MV for [《糖画小仙》 (Sugar Art Fairy) on YouTube](https://youtu.be/vZbyKeCCSRA), running rock-solid at 1080p 30fps. + +> **TL;DR** — Scaling a full-bleed background image with the bass band forces an interpolation resampling on every frame, which the human eye perceives as violent shudder. Furthermore, two stacked semi-transparent layers crossfading symmetrically dip through the black root container at the midpoint. Fixes: keep the background canvas completely static, decouple all audio-reactive pulsing into isolated foreground HUD layers, and fade in the incoming slide on top of a 100% solid base. Render time dropped by **38.0% (from 352s to 218s, a 1.61× throughput speedup)**. This is the rendering chapter of our [music-video pipeline](/posts/how-i-built-an-automated-music-video-pipeline/); the file-staging half is covered in [uploading large videos via CDP](/posts/uploading-large-videos-to-youtube-via-cdp/). + +--- + +## 1. The Symptom: A Whole-Frame Earthquake + +The visual effect was supposed to be simple: +1. An illustrated 1080p background slowly crossfading between slides. +2. A pulsing circular spectrum visualizer in the foreground. +3. Three lines of Chinese calligraphy lyrics that slide between left and right depending on where the character in the artwork is standing. + +In preview scrubbing, everything looked acceptable. But during full-speed 30fps playback of the rendered 1080p file, the viewer felt instant visual fatigue. Every time the electronic drum pattern kicked in, the entire scene appeared to vibrate violently up to 30 times per second. Furthermore, during slide transitions, the screen experienced micro-flashes where the background brightness suddenly dipped. + +--- + +## 2. Root Cause 1: Low-Frequency Scale Pollution + +Looking at the initial `BackgroundSlideshow.tsx`, we had attached an audio reactivity hook: + +```tsx +// The culprit in BackgroundSlideshow.tsx +const bassScale = interpolate(bassLevel, [0, 1], [1.0, 1.025], { + extrapolateRight: 'clamp', +}); + +// Applied to the background image: + +``` + +On paper, expanding the image by 2.5% on bass hits sounds like an engaging dynamic pulse. In reality: +- **Electronic audio beats hit in 30–60ms bursts**: The bass amplitude oscillates rapidly between zero and peak within 2 to 3 consecutive frames. +- **A 1920x1080 image scaling by 2.5% shifts edge pixels by nearly 25 to 30 physical pixels in a single frame**. +- When that sudden displacement reverses on the very next frame, the human eye interprets it not as a musical pulse, but as **severe camera camera shake / stutter**. +- To make matters worse, during the 1.8-second slide transition, two overlapping images with different transform origins were simultaneously vibrating at conflicting scales, creating a hallucinatory double-vision jitter. + +--- + +## 3. Root Cause 2: Symmetrical Opacity Crossfade DIP + +The second issue was the crossfade flash. In standard web CSS, people often do: + +```css +/* Web crossfade trap */ +.current-slide { opacity: 1 -> 0; } +.next-slide { opacity: 0 -> 1; } +``` + +When two images crossfade with linear opacity, consider what happens at the exact midpoint ($t = 0.5$): +- $\text{Current Slide Opacity} = 0.5$ +- $\text{Next Slide Opacity} = 0.5$ + +Here is how the browser/Remotion compositing pipeline stacks these layers against the root black container: + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Layer 2: Next Slide (opacity = 0.5) │ +├─────────────────────────────────────────────────────────────┤ +│ Layer 1: Current Slide (opacity = 0.5) │ +├─────────────────────────────────────────────────────────────┤ +│ Layer 0: Root Container (background = #000000 black) │ +└─────────────────────────────────────────────────────────────┘ +``` + +Because alpha blending operates multiplicatively down the stack: +$$\text{Composite Alpha} = 1 - (1 - 0.5) \times (1 - 0.5) = 1 - 0.25 = 0.75$$ + +The total combined alpha drops to **0.75**, exposing **25% of the black background**! This is not an optical illusion; it is mathematical light loss. Every time a transition peaked, the frame darkened by a quarter of its luminance, creating an unmistakable flicker. + +--- + +## 4. The Solution: Rock-Solid Base + Pure Single-Direction Opacity + +To fix both issues permanently: + +### 1. Remove All Scaling from Backgrounds +Background artwork must remain rock-solid. All audio reactivity belongs exclusively in the HUD layer (the radial spectrum bars and glowing rings). The background canvas should only ever receive smooth, non-oscillating transformations. + +### 2. Base Solid + Top Layer Fade-In +Instead of fading out the current image, **keep the current image at solid 100% opacity**. Only animate the incoming image on top: + +```tsx +return ( +
+ {/* Base Slide: Stays 100% solid, eliminating any drop to black */} + + + {/* Next Slide: Smoothly fades in on top over 1.8s */} + {isTransitioning && ( + + )} +
+); +``` + +### 3. Physical Decoupling of HUD and Canvas +The lyric box and circular spectrum are mounted inside a completely separate container layer with `pointerEvents: 'none'`. As the HUD glides across the canvas from `centerX: 540` to `centerX: 1380`, it calculates its position strictly from the timeline clock. It has zero knowledge of and zero impact on the background's opacity crossfade. + +--- + +## 5. The Outcome & Benchmark Reproduction + +After removing the background `bassScale` and switching to one-way opacity stacking: + +1. **Eliminated visible jitter**: Background illustrations remain crisp and visually anchored through rapid 140 BPM electronic drum drops across all test sequences. +2. **Eliminated transition light dips**: The 1.8s Bezier crossfade maintains 100% luminance continuity without dipping through the dark container base. +3. **Significant throughput speedup**: Removing per-frame matrix interpolation and scaling calculations on high-resolution background assets delivered a measurable performance boost. + +### Reproduction Benchmark + +| Metric | Before Optimization | After Optimization | Delta | +|---|---|---|---| +| **Render Duration** | 352.0 seconds | 218.3 seconds | **-38.0% (-133.7s)** | +| **Render Throughput** | 22.24 fps | 35.87 fps | **+1.61× speedup** | +| **Total Frames** | 7,830 frames | 7,830 frames | Identical (261s / 4m 21s) | +| **Output Resolution** | 1920 × 1080 (1080p @ 30fps) | 1920 × 1080 (1080p @ 30fps) | Identical | +| **Codec & Settings** | H.264 (CRF 18, preset medium) | H.264 (CRF 18, preset medium) | Identical | + +**Test Environment Specifications:** +- **CPU**: Intel Core i9-13900K (24 cores / 32 threads, boost up to 5.8 GHz) +- **GPU**: NVIDIA GeForce RTX 3060 (12GB GDDR6 VRAM, Driver 551.xx) +- **RAM**: 64GB DDR5 5600 MHz +- **OS**: Windows 11 Pro 64-bit +- **Software**: Node.js v20.x, Remotion v4.0.x with Chromium headless GPU rendering enabled + +Both benchmark runs were executed consecutively using identical media assets, audio spectrogram inputs, and composition parameters. The 38.0% reduction in render time directly translates to a **1.61× throughput speedup**, turning a process that was previously slower than realtime into an accelerated 35.8 fps export. + +--- + +## Related Guides in this Series +- **Pipeline Architecture**: [How I Built an Automated Music Video Pipeline with Remotion and Docker](/posts/how-i-built-an-automated-music-video-pipeline/) +- **Remote Publishing**: [Uploading 100MB+ Videos to YouTube via CDP: The Docker Archive Pattern](/posts/uploading-large-videos-to-youtube-via-cdp/) +- **Live Output**: [Watch the finished MV on YouTube](https://youtu.be/vZbyKeCCSRA) +```` + +--- + +## 文章 06 / 11：把字体嵌进 PDF，让它在任何机器上都长一样 +- **Slug**: `embedding-fonts-so-a-pdf-looks-the-same-everywhere` +- **英文标题**: Embedding Fonts So a PDF Looks the Same on Every Machine +- **分类**: `engineering` | **发布日期**: 2025-01-22 +- **线上地址**: https://blog.hoelee.com/posts/embedding-fonts-so-a-pdf-looks-the-same-everywhere/ (EN) | https://blog.hoelee.com/posts/zh/embedding-fonts-so-a-pdf-looks-the-same-everywhere/ (ZH) +- **本篇审核状态**: ⏳ 待审核 (Pending) + +### ✍️ 顾问审阅意见区 (Review Feedback) +```text +【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 +【事实与数字核对】： +【人设与用词卫生】： +【CRO与转化建议】： +``` + +### 🇨🇳 中文版原文 (Chinese Markdown Twin) +````markdown +--- +title: "把字体嵌进 PDF，让它在任何机器上都长一样" +description: "四个字族、八个字面、两个渲染引擎，外加一条 CSS 怪癖：为了不让生成的报告依赖服务器上恰好装了什么字体，我做了什么。" +pubDate: 2025-01-22 +updatedDate: 2026-10-16 +category: engineering +tags: ["pdf", "fonts", "mpdf", "typography", "php"] +ogImage: /og/zh-embedding-fonts-so-a-pdf-looks-the-same-everywhere.jpg +banner: /banners/embedding-fonts-so-a-pdf-looks-the-same-everywhere.jpg +draft: false +--- + +## 需求只有一句话 + +一个报告生成器必须产出在任何地方都长得一样的文档：中文标题用宋体、中文正文用黑体、英文标题与封面数字用一款展示衬线体、英文正文和表格用 Inter。字重也要对——强调用 medium、标题用 semibold——而且是三语、两个 PDF 渲染引擎都要对。 + +同时，**字体必须跟着文档走**。不是「服务器装过」，也不是「读者大概有」，而是**嵌进文件里**，这样另一个国家的印厂看到的页面是一样的。 + +老管线在最后这句上栽得很隐蔽：它依赖服务器上用包管理器装过的字体，而且**只嵌入了一款中文字体**。第二档字重根本不存在，于是「粗体」是渲染器的合成近似；而想发现这件事，唯一的办法是打开 PDF 去看里面到底装了什么。 + +## 规则一：从成品 PDF 里把字体读回来 + +「已嵌入」是一个关于**输出**的断言，所以我量的是输出。同一份文档，改前与改后： + +| | 改前 | 改后 | +|---|---|---| +| zh 页数 | 23 | **23** | +| en 页数 | 28 | **29** | +| bm 页数 | 30 | **30** | +| 真正嵌进去的字体 | 一款中文字体 + Roboto | 思源宋 Medium + SemiBold · 思源黑 Regular + Medium · Cormorant Garamond Medium + SemiBold · Inter Regular + SemiBold + Medium · 拉丁报告里的汉字走思源黑 | +| 渲染时间 / 峰值内存 | 基线 | 同一量级（±10%，属噪声） | +| 文件体积 | 基线 | 约 +3% | + +每个字面的字形数是收据：中文报告里 15,988 个字形落在正文黑体、2,247 个落在它的 medium 档、634 个落在标题宋体的 medium、222 个落在它的 semibold。这就是「字体在服务器上」与「字体在文件里」的区别——另一个国家的读者看到的是同样四个字面，而代价是某一种语言正好多一页（拉丁字体的度量比它替换掉的那个更宽）。 + +## 规则二：只出静态实例，理由写在解析器里 + +现代做法看起来该是「每个字族发一个可变字体」，但在这里是错的，有两个实测理由： + +1. **PDF 引擎的字体解析器只读 `glyf`/`loca`。** 喂它一个 PostScript/CFF 轮廓的 OpenType 文件，它直接拒绝：`Fonts with postscript outlines are not supported`。问题不是「可变」，是 CFF 轮廓。 +2. **它完全不读 `fvar` 表**，所以可变字体会**静默地**解析成它的**默认实例**——而中文字族的默认实例恰恰是最细的那一档（`name` 表里写得清清楚楚：Thin 100 与 ExtraLight 200）。整份报告会变成发丝体，而整条管线里不会报任何一个错。 + +所以构建步骤拿上游的可变字体**实例化成静态切面**：跑实例化器，然后把 `name`、`OS/2.usWeightClass` 和 `head.macStyle` 改写成它现在真实的字重（源文件的名字里还写着 Thin）。产物旁边有一份 manifest，逐文件记下 family、字重、sha256、字形数与码位数；许可证也随文件一起进仓库。 + +## 规则三：引擎只会说两种字重，那就映射到两个槽位 + +渲染引擎的 CSS 只认 `normal` 和 `bold`——它的源码自己就这么写（"Does not support: bolder, lighter, 100..900"）。所以数字字重对它毫无意义；而同一条管线里的无头 Chromium 又完全认它们。 + +可用的映射是：每个字族声明一个 **R 槽**和一个 **B 槽**（常规一档、更重一档），`font-weight: 500` 落到目标字族的 R 槽，`600` **只在 PDF 那一侧被改写成 `bold`**，落到 B 槽。Chromium 侧照写 `600`。两条测试钉住它：B 槽必须比 R 槽重（读 `usWeightClass`，不是假设），且 `font-weight: 600` 绝不允许出现在交给 PDF 引擎的样式表里。 + +有一处细节值得单写一段：在这个管线里，**medium 不是强调手段，而是一个字族成员**。中文标题用宋体的 medium 当常规、用 semibold 做强调；英文标题用展示衬线体做同样的事。这里搞错，屏幕上完全看不出来，而印出来一目了然。 + +## 规则四：不要按你控制不了输入的字集做子集化 + +第一版把中文字体按两个国家字集的并集做了子集，产物从约 50 MB 降到约 29 MB。然后我跑了姓名探针，发现这些字被裁掉了： + +```text +犇 骉 頔 祎 廸 玧 祾 赟 堃 +``` + +每一个都是真实的人名用字。报告会把客户姓名印在封面上，而姓名是自由输入——所以一套基于子集的构建，迟早会在产品最醒目的那一页上放一个豆腐块。全字形覆盖是约 31k 码位、约 50 MB 的嵌入字体；为了永远不给客户看一个空方块，这个价钱很便宜。 + +## 规则五：把「缺字」变成构建失败，而不是惊喜 + +闸门跑三项检查，任何一项不过就非零退出： + +- manifest 里声明的每个字体文件都在，且 sha256 与磁盘上的文件一致； +- 内容库（三语全部）+ 品牌字面 + 各类探针（简体、繁体、CJK 标点、全角、中英混排、数字、拉丁扩展、符号）+ 姓名用字，**每一个字在「它实际会被哪一档字体印出来」的那一档里都有字形**； +- 查字形用的是 PDF 引擎自己的解析器，而不是第三方库，所以这条检查不可能和渲染器互相矛盾。 + +已知边界写在明处，不藏起来：✅ ✔ 这类符号在这几款正文字体里都没有字形，而引擎也画不了彩色 emoji，所以它们不出现在报告正文；拉丁标题那款展示衬线体只有 974 个字形，不画 ① ★ ℃，这也是为什么内容库的检查覆盖**所有**槽位而不只是正文；而拉丁报告里的汉字必须走专门的 CJK 槽位——两条测试防止两个渲染器在这点上产生分歧。 + +## 如果重来一次 + +1. **在动样式之前先定字族与字重。** 版式做完之后再去加第二档字重，意味着每一页页数都要手工复核一遍。 +2. **选字体格式之前先读渲染器的源码。** 可变字体看着是审美问题，实际上是解析器问题——两个文件、十分钟就定了。 +3. **把要嵌进文档的一切当作许可证产物。** 字体文件连同许可证一起进仓库，否则就得进别人的法务审查。 +4. **量「到底嵌进去了什么」。** 页数、每个字面的字形数、文件体积、渲染时间：四项都动了，其中三项的方向和我猜的相反。 + +## 结果 + +改前：一款中文字体加一款界面字体、靠合成粗体、还取决于服务器装过什么。改后：四个字族、八个静态字面真正嵌在文件里，字重是各个字族真实的成员，页数与上一版相差不超过一页，而且有一条 lint 闸门：某个字没有字形时直接让构建失败——于是失败形态从「客户看到一个方块」变成了「CI 拒绝发版」。这跟这条管线里其它地方的取舍是同一个：**让错误发不出去，而不是让错误容易被看见。** +```` + +### 🇬🇧 英文版原文 (English Markdown Primary) +````markdown +--- +title: "Embedding Fonts So a PDF Looks the Same on Every Machine" +description: "Four families, eight faces, two renderers and one CSS quirk: what it took to stop a generated report depending on whatever fonts the server happened to have." +pubDate: 2025-01-22 +updatedDate: 2026-10-16 +category: engineering +tags: ["pdf", "fonts", "mpdf", "typography", "php"] +ogImage: /og/embedding-fonts-so-a-pdf-looks-the-same-everywhere.jpg +banner: /banners/embedding-fonts-so-a-pdf-looks-the-same-everywhere.jpg +draft: false +--- + +## The requirement was one sentence + +A report generator has to produce a document that looks the same everywhere: +Chinese headings in a serif, Chinese body text in a sans, Latin headings and +cover numerals in a display serif, Latin body text and tables in Inter. +Weights matter — medium for emphasis, semibold for headings — across three +languages and two PDF renderers. + +And the fonts must travel with the document. Not "the server has them +installed", not "the reader probably has them" — **embedded**, so a print +shop in another country sees the same pages. + +The old pipeline failed on the last clause in a way that was invisible: it +depended on fonts installed on the server by a package manager, and it only +ever embedded a single CJK face. The second weight did not exist, so "bold" +was the renderer's synthetic approximation, and the only way to notice was +to open the PDF and look at what was actually inside it. + +## Rule one: read the fonts back out of the finished PDF + +"Embedded" is a claim about the output, so I measured the output. The same +document, rendered before and after: + +| | before | after | +|---|---|---| +| zh page count | 23 | **23** | +| en page count | 28 | **29** | +| bm page count | 30 | **30** | +| fonts actually embedded | one CJK face + Roboto | Source Han Serif Medium + SemiBold · Source Han Sans Regular + Medium · Cormorant Garamond Medium + SemiBold · Inter Regular + SemiBold + Medium · Source Han Sans for the CJK runs inside Latin reports | +| render time / peak memory | baseline | same order (±10%, noise) | +| file size | baseline | roughly +3% | + +The glyph counts per face are the receipt: in the Chinese report, 15,988 +glyphs land in the body sans, 2,247 in its medium weight, 634 in the heading +serif medium, 222 in its semibold. That is the difference between "the font +is on the server" and "the font is in the file" — a reader in another +country sees the same four faces, and a page count that moved by exactly one +page in one language (the Latin metrics are wider than what they replaced) +is the honest cost. + +## Rule two: static instances only, and the reason is in the parser + +The obvious modern choice — ship one variable font per family — is wrong +here, for two measured reasons: + +1. **The PDF engine's font parser reads `glyf`/`loca` only.** Feed it an + OpenType file with PostScript/CFF outlines and it refuses: + `Fonts with postscript outlines are not supported`. A variable font is + not the problem; CFF outlines are. +2. **It does not read the `fvar` table at all**, so a variable font silently + resolves to its *default instance* — and the default instances of the CJK + families are the thinnest ones (Thin 100 and ExtraLight 200, visible in + the `name` table). Every page would render in a hairline weight, with no + error anywhere in the pipeline. + +So the build step takes the upstream variable fonts and **instantiates +static cuts**: it runs the instancer, then rewrites `name`, +`OS/2.usWeightClass` and `head.macStyle` so the file identifies itself as +the weight it now is (the source's own name still says "Thin"). A manifest +records family, weight class, sha256, glyph count and code-point count for +every file, and the licence files ship next to them. + +## Rule three: the engine speaks two weights, so map onto two slots + +The renderer's CSS handling honours exactly `normal` and `bold` — its source +says so in as many words ("Does not support: bolder, lighter, 100..900"). +Numeric weights are therefore meaningless to it, while the headless-Chromium +renderer in the same pipeline honours them properly. + +The mapping that works: each family declares an **R slot** and a **B slot** +(regular-ish and heavier), `font-weight: 500` lands on the R slot of the +intended family, and `600` is **rewritten to `bold`** on the PDF side only, +which lands on the B slot. Chromium keeps `600` as written. Two tests pin +it: the B slot must be heavier than the R slot (read from `usWeightClass`, +not assumed), and `font-weight: 600` must never appear in the stylesheet +handed to the PDF engine. + +One subtlety worth the paragraph: a "medium" weight is not an emphasis +mechanism in this pipeline, it is a *family member*. Chinese headings use +the serif's medium as their regular and its semibold for emphasis; Latin +headings do the same with the display serif. Getting that wrong is invisible +on a screen and obvious in print. + +## Rule four: do not subset against a character set you do not control + +The first attempt subset the CJK fonts to the union of two national +character sets and produced a ~29 MB bundle instead of ~50 MB. Then I ran +the name probe and found these were cut: + +```text +犇 骉 頔 祎 廸 玧 祾 赟 堃 +``` + +Every one of them is a real given name character. The report prints a +customer's name on the cover, and the name is free input — so a subset-based +build will eventually put a tofu box on the most prominent page of the +product. Full coverage is ~31k code points and roughly 50 MB of embedded +fonts; that is a fine price for never showing a customer an empty rectangle. + +## Rule five: make missing glyphs a build failure, not a surprise + +The gate runs three checks and exits non-zero on any of them: + +- every font file declared in the manifest is present, and its sha256 + matches the file on disk; +- every character in the content library (all three languages) plus the + brand strings, plus probe classes — simplified, traditional, CJK + punctuation, full-width forms, mixed-script, digits, Latin extended, + symbols — plus the name characters, has a glyph **in the face that will + actually print it**; +- the lookup uses the PDF engine's own font parser rather than a third-party + library, so the check cannot disagree with the renderer. + +The known edges are written down instead of hidden. Dingbats and emoji (✅ ✔) +exist in none of these text fonts and the engine cannot draw colour emoji, +so they are kept out of the report body. The display serif has only 974 +glyphs and does not draw symbols like ① ★ ℃, which is why the +content-library check covers *all* slots rather than just the body. And CJK +characters inside a Latin report must go through the dedicated CJK slot — +two tests keep the renderers from disagreeing about that. + +## What I would do differently + +1. **Choose faces and weights before styling anything.** Retrofitting a + second weight after the layout exists means re-checking every page count + by hand. +2. **Read the renderer's source before choosing a font format.** The + variable-font decision looked like a taste question and was actually a + parser question — two files settled it in ten minutes. +3. **Treat everything you ship inside a document as a licence artifact.** + Font files enter the repo with their licences, or they enter someone + else's legal review. +4. **Measure what got embedded.** Page count, glyph-per-face counts, file + size, render time: all four moved, and three of them moved in ways I + would have guessed wrong. + +## The result + +Before: one CJK face and a UI font, with synthetic bold, depending on what +the server had installed. After: four families and eight static faces +embedded in the file, weights that are real members of their families, page +counts within one page of the previous layout, and a lint gate that fails +the build when a character has no glyph — so the failure mode moved from "a +customer sees a box" to "CI refuses the release". Which is the same trade as +everywhere else in this pipeline: make the mistake impossible to ship rather +than easy to notice. +```` + +--- + +## 文章 07 / 11：那个把静默丢失更新变成 409 的 JPA 字段：@Version 乐观锁 +- **Slug**: `jpa-version-field-lost-update` +- **英文标题**: The JPA Field That Turns a Silent Lost Update Into a 409 +- **分类**: `engineering` | **发布日期**: 2026-08-19 +- **线上地址**: https://blog.hoelee.com/posts/jpa-version-field-lost-update/ (EN) | https://blog.hoelee.com/posts/zh/jpa-version-field-lost-update/ (ZH) +- **本篇审核状态**: ⏳ 待审核 (Pending) + +### ✍️ 顾问审阅意见区 (Review Feedback) +```text +【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 +【事实与数字核对】： +【人设与用词卫生】： +【CRO与转化建议】： +``` + +### 🇨🇳 中文版原文 (Chinese Markdown Twin) +````markdown +--- +title: "那个把静默丢失更新变成 409 的 JPA 字段：@Version 乐观锁" +description: "在 REST + JPA 的读-改-写流程里，两个编辑同时保存同一条记录时，后保存者会静默覆盖先保存者的修改，而 API 两次都返回成功——JPA 的 @Version 乐观锁把这个静默丢失更新变成 HTTP 409 冲突：请求携带期望版本号，版本不符就拒绝写入，并提示客户端重新读取再试。" +pubDate: 2026-08-19 +category: engineering +tags: [java, spring, jpa, hibernate, rest, testing] +ogImage: /og/zh-jpa-version-field-lost-update.jpg +banner: /banners/jpa-version-field-lost-update.jpg +draft: false +--- + +两个人同时打开同一篇文章，两个人都改了，两个人都保存。其中一次 +保存悄无声息地消失了——没有报错，没有警告，而且 API 对两次请求 +都返回了 HTTP 200。 + +这就是丢失更新（lost update）问题，是读-改-写（read-modify-write） +API 里最安静的数据丢失 bug。我在做 Spring Boot 作品集 demo 时撞上 +了它——这是一个 Spring Boot 3.5.16（Java 21）的 REST API，编辑器 +从浏览器页面创建和更新文章。修法最终落在三处：JPA 实体上的一个 +`@Version` 字段、请求契约里的一个字段、以及 API 层把它映射成 +HTTP 409 Conflict 的一个异常。这篇文章就沿着真实代码走一遍这条路。 + +## 问题：一条编辑消失了，却谁都怪不上 + +为什么两个人保存同一条记录时，其中一个人的修改会消失？ + +跟着时间线走。编辑器 A 和编辑器 B 都拉取了同一篇文章。A 先保存： +行被更新，API 返回 200。B 稍后保存：行被*再次*更新，这次覆盖了 +A 的文字，API 又返回 200。每个请求都成功；每个响应都告诉它的调用 +方「你的保存就是这条记录的当前状态」——对其中一个人来说这是假话。 +A 的编辑就这么没了，两个客户端都没有任何办法知道。 + +失败是静默的，因为它就是读-改-写流程的正常行为，不是异常。这正是 +它在真实系统里存活的理由，也是为什么在要展示工程判断力的东西里， +值得把它修得干净。 + +## 我先试的方案：不带版本号的读-改-写，以及它为什么失败 + +我对更新流程的第一版草图是最直白的那种：`GET` 记录、编辑、 +`PUT` 回去，只带新的标题和正文，什么都不带。 + +```text +GET /api/posts/{id} # 读取记录 +PUT /api/posts/{id} # 写回新的标题和正文 +``` + +服务端加载行、套用更新、返回 200。它无从知道在这一行数据上，客户端 +在 `GET` 和 `PUT` 之间已经被别人动过——读取和写入是两个互不相干的 +请求，契约里没有任何东西把它们连起来。最后写入的人赢，而两个写入 +者都被告知成功。 + +在这里，200 比报错更糟。报错至少告诉输掉的那个客户端「出事了」， +给他们一个去查的理由。200 则告诉两个客户端「你们的保存就是记录 +当前的状态」——对其中一个是撒谎。输掉的那次编辑现在哪儿都不存在： +屏幕上没有，数据库里也没有；而因为客户端相信自己保存成功了， +没有人会去找这份数据。 + +## 修法：一个 @Version 字段、一份契约、一个冲突 + +修法一共五小步，全部都在 demo 的源码里。 + +### 1. 实体随身携带一个版本列 + +```java +@Entity +@Table(name = "posts", indexes = @Index(name = "idx_posts_title", columnList = "title")) +@EntityListeners(AuditingEntityListener.class) +public class Post { + + @Id + @GeneratedValue + @UuidGenerator + private UUID id; + + @Column(nullable = false, length = 160) + private String title; + + @Column(nullable = false, length = 10_000) + private String body; + + @Version + private long version; + // ... +} +``` + +`@Version` 告诉 Hibernate 把 `version` 当作乐观锁。它生成的每一条 +`UPDATE` 都变成有条件的： + +```sql +UPDATE posts +SET title = ?, body = ?, version = version + 1, ... +WHERE id = ? AND version = ? +``` + +如果这行数据的版本号已经和语句构造时的不一致，受影响行数为零， +Hibernate 就会拒绝这次 flush，而不是悄悄覆盖。实体从不用手自增 +计数器——更新方法只碰 `title` 和 `body`；版本号跟着写入一起走。 + +### 2. 版本号进入 API 契约 + +锁如果客户端无法参与，就没有用，所以版本号在两个方向上都穿过 +API。响应的 record 暴露它： + +```java +public record PostResponse(UUID id, long authorId, String title, String body, + long version, Instant createdAt, Instant updatedAt) { +} +``` + +更新请求则要求把它带回来： + +```java +public record UpdatePostRequest( + @NotBlank @Size(max = 160) String title, + @NotBlank @Size(max = 10_000) String body, + @NotNull @Min(0) Long version) { +} +``` + +`UpdatePostRequest` 旁边的学习注记说得很直白： +「the expected version is part of the update +contract, making optimistic locking visible to clients.」 +（期望的版本号是更新契约的一部分，让乐观锁对客户端可见。） + +### 3. 在拥有写事务的 service 里检查版本 + +```java +@CachePut(cacheNames = "posts", key = "#postId") +@Transactional +public PostResponse updatePost(UUID postId, UpdatePostRequest request) { + Post post = postRepository.findById(postId) + .orElseThrow(() -> new PostNotFoundException(postId)); + if (post.getVersion() != request.version()) { + throw new PostVersionConflictException(postId); + } + post.update(request.title().trim(), request.body().trim()); + return PostResponse.from(postRepository.saveAndFlush(post)); +} +``` + +读取、比较、保存全部发生在一个 `@Transactional` 方法里——这正是让 +检查诚实的那部分：版本号是和*写入那一刻*的行状态比较，而不是和 +早前某个请求留下的快照比较。如果读取和写入分属不同事务，检查到的 +就是这行数据已经离开的旧版本，整套机制就成了表演。`PostService` +的学习注记点名了这条规则：「service methods +centralize transaction boundaries, cache coherence, and +optimistic-locking rules.」（service 方法集中管理事务边界、缓存 +一致性与乐观锁规则。）demo 还开着 +`spring.jpa.open-in-view: false`，所以不会有残留的会话跨请求地 +供给过期实体——行是在写入事务内部重新读取的。 + +显式的检查会确定性地挡掉过期请求。`@Version` 列在底层仍然有意义： +如果检查与 flush 之间恰好插进一次别的提交，那条有条件的 `UPDATE` +影响零行，Hibernate 照样拒绝这次写入。 + +### 4. 异常处理器把它映射成 409 + +异常本身带着客户端能直接照做的信息： + +```java +public class PostVersionConflictException extends RuntimeException { + + public PostVersionConflictException(UUID postId) { + super("Post %s has changed; fetch it again before retrying".formatted(postId)); + } +} +``` + +错误边界把它转成正确的 HTTP 应答： + +```java +@ExceptionHandler(PostVersionConflictException.class) +ProblemDetail handleConflict(PostVersionConflictException exception) { + return problem(HttpStatus.CONFLICT, "POST_VERSION_CONFLICT", exception.getMessage()); +} +``` + +`HttpStatus.CONFLICT` 就是 HTTP 409，响应是一份 problem +document——一个 `type` URI、一个稳定的机器码 +（`POST_VERSION_CONFLICT`）和一段人类可读的说明。demo 在配置里 +开启了框架的 problem-details 支持 +（`spring.mvc.problemdetails.enabled: true`），所以校验错误和所有 +其他错误都共用同一种响应形状。 + +### 5. 整条链路，从头到尾 + +```text +过期的 PUT /api/posts/{id} + -> PostService.updatePost 在事务内部重新读取这一行 + -> post.getVersion() != request.version() + -> PostVersionConflictException: "fetch it again before retrying" + -> ApiExceptionHandler.handleConflict + -> HTTP 409, ProblemDetail with code POST_VERSION_CONFLICT + -> 客户端知道自己的前提过期了 +``` + +一个 `@Version` 字段、请求契约里的一个 `version`、一个异常、一个 +handler 方法。Controller 什么都没改——`updatePost` 看起来还是普通 +的 `PUT`。 + +## 为什么答案是冲突，而不是重试 + +当两个编辑器产出同一份记录的两种不同版本时，服务端无法知道哪一方 +的意图才是被需要的。从各自作者的角度看，两次保存都正当；重放那条 +过期的写入，只是重放同一次数据丢失。 + +诚实的结论就是 HTTP 对 409 的定义：请求与资源的当前状态冲突， +并且客户端被明确告知如何解决——「fetch it again before retrying」 +（重新取一次再试）。输掉的客户端重新读取记录，看到另一位作者的 +修改，由人来决定保留什么。静默覆盖变成了一处看得见的决策点——这 +正是这件事的全部意义。 + +## 这个 demo 的其余部分证明了什么 + +版本字段只是一个小系统里的一层，而其余部分都能用同样的方式核实 +——下面每一行都是仓库里的一个类或配置文件，不是一句口号： + +| 层 | 做了什么 | 在哪 | +|---|---|---| +| 缓存 | 单篇文章读取使用有界 Caffeine 缓存（上限 500，TTL 10 分钟，都来自配置）；读取用 `@Cacheable`、更新用 `@CachePut`、删除用 `@CacheEvict`；`GET /api/showcase/cache` 暴露请求数、命中数、未命中数和命中率 | `CacheConfig`、`CacheProperties`、`ShowcaseMetricsController` | +| 安全 | 无状态 HTTP Basic + BCrypt；页面与读 API 公开，所有写操作需要 `EDITOR` 角色；本地默认是专用的非机密 `demo-editor`/`changeit` 账户，可用环境变量覆盖 | `SecurityConfig`、`application.yml` | +| 错误 | 请求 record 在边界处校验；一个 `@RestControllerAdvice` 返回一致的 problem document 和机器码——`POST_NOT_FOUND`（404）、`POST_VERSION_CONFLICT`（409）、带字段错误表的 `VALIDATION_FAILED`（400） | `CreatePostRequest`、`UpdatePostRequest`、`ApiExceptionHandler` | +| 配置 | 本地 profile 使用内存 H2 数据库（PostgreSQL 模式），`ddl-auto: update` 并预置示例文章；`prod` 选择 PostgreSQL，所有凭据来自 `APP_*` 环境变量，`ddl-auto: validate` | `application.yml`、`application-prod.yml` | +| 测试 | 一个 `@SpringBootTest` 套件断言：渲染的 Thymeleaf 页面、未认证 401 对编辑器 201、`VALIDATION_FAILED` 问题响应、公开搜索，以及重复读取后的缓存命中 | `DemoApplicationIntegrationTest` | + +## 如果做真东西，我会怎么改 + +一旦这不再是本地 demo，有两件事立刻要变。项目 README 里两件都写了， +值得作为我自己的判断再重复一遍。 + +第一，schema 管理。我会在任何时候把 `ddl-auto` 设为 `validate` 之前， +先加 Flyway 或 Liquibase 迁移。demo 默认 profile 用的是 `update`， +在临时数据库上很方便，养成习惯就很危险——生产 schema 应该是 +有版本管理的代码，而不是启动时的副作用。 + +第二，密钥。本地编辑器账户是刻意的非机密 demo 账户，带默认凭据。 +做真东西的话，我会把凭据放进托管的密钥存储（managed secret store）， +拿不到凭据就拒绝启动，而不是回退到默认值。`prod` profile 已经做到 +所有凭据来自环境变量、不提交任何密钥——这正是我想要的形态， +只是不要那些回退值。 + +## 结果 + +过期的更新现在返回 HTTP 409，机器码 `POST_VERSION_CONFLICT`，而不是 +一个静静丢掉另一位作者修改的 200。丢失不再可能悄悄发生：失败是 +响亮的、可操作的——「fetch it again before retrying」——并且集成 +测试套件端到端地断言了访问与校验行为。 + +整个修法就是一个字段上的一个注解，加上让版本号成为契约一部分的 +管道代码。如果一条记录可能同时被两个人读取，这个字段就是「编辑 +丢失」与「双方都看得见的冲突」之间的分界线。 + +我平时就做 Java/Spring 后端和全栈 Web 应用——从浏览器页面到数据库 +的 REST API：JPA、乐观锁、缓存、校验和一整套集成测试。如果你的表单 +或字段里也有「编辑悄悄消失」的问题，跟我说说： +[WhatsApp](https://wa.me/60127972969) · +[me@hoelee.com](mailto:me@hoelee.com?subject=Spring%20Boot%20help) · +[hoelee.com](https://hoelee.com)。 +```` + +### 🇬🇧 英文版原文 (English Markdown Primary) +````markdown +--- +title: "The JPA Field That Turns a Silent Lost Update Into a 409" +description: "How a JPA @Version field turns a silent lost update into a 409 conflict — optimistic locking makes a stale REST write fail loudly instead of overwriting." +pubDate: 2026-08-19 +category: engineering +tags: [java, spring, jpa, hibernate, rest, testing] +ogImage: /og/jpa-version-field-lost-update.jpg +banner: /banners/jpa-version-field-lost-update.jpg +draft: false +--- + +Two people open the same post. Both edit it. Both save. One of the saves +disappears without a trace — no error, no warning, and the API answered +HTTP 200 for both of them. + +That is the lost update problem, and it is the quietest data-loss bug in +read-modify-write APIs. I hit it while building my Spring Boot portfolio +demo — a Spring Boot 3.5.16 (Java 21) REST API where editors create and +update posts from a browser page. The fix turned out to be one `@Version` +field on the JPA entity, one field on the request contract, and one +exception that the API layer maps to HTTP 409 Conflict. This post traces +that path through the actual code. + +## The problem: an edit that vanishes with no one to blame + +Why does one editor's change disappear when two people save the same +record? + +Follow the timeline. Editor A and editor B both fetch the same post. +A saves first: the row is updated and the API answers 200. B saves a +moment later: the row is updated *again*, this time over A's text, and +the API answers 200 again. Every request succeeded; every response told +its caller that their save is the state of the record. A's edit is +simply gone, and neither client has any way to know. + +The failure is silent because it is normal read-modify-write behaviour, +not an anomaly. That is exactly why the bug survives in real systems, and +why it is worth fixing properly in something meant to demonstrate +engineering judgement. + +## What I tried first, and why it failed + +My first sketch of the update flow was the obvious one: `GET` the record, +edit it, `PUT` it back with the new title and body and nothing else. + +```text +GET /api/posts/{id} # read the record +PUT /api/posts/{id} # write it back with a new title and body +``` + +The server loads the row, applies the update, and returns 200. It has no +way to notice that the row moved underneath the client between the `GET` +and the `PUT` — the read and the write are two unrelated requests, and +nothing in the contract connects them. The last write wins, and both +writers are told they succeeded. + +A 200 is worse than an error here. An error at least tells the losing +client that something happened and gives them a reason to look. A 200 +tells both clients that their save is the current state of the record, +which is a lie for one of them. The losing edit now exists nowhere — not +on screen, not in the database — and because the client believes it +saved, nobody goes looking for the data. + +## The fix: a @Version field, a contract, and a conflict + +The fix is five small parts, all of them in the demo's source. + +### 1. The entity carries a version column + +```java +@Entity +@Table(name = "posts", indexes = @Index(name = "idx_posts_title", columnList = "title")) +@EntityListeners(AuditingEntityListener.class) +public class Post { + + @Id + @GeneratedValue + @UuidGenerator + private UUID id; + + @Column(nullable = false, length = 160) + private String title; + + @Column(nullable = false, length = 10_000) + private String body; + + @Version + private long version; + // ... +} +``` + +`@Version` tells Hibernate to treat `version` as an optimistic lock. Every +`UPDATE` it generates becomes conditional: + +```sql +UPDATE posts +SET title = ?, body = ?, version = version + 1, ... +WHERE id = ? AND version = ? +``` + +If the row's version no longer matches the value the statement was built +with, zero rows change and Hibernate rejects the flush instead of quietly +overwriting. The entity never bumps the counter by hand — the update +method only touches `title` and `body`; the version moves with the write +itself. + +### 2. The version is part of the API contract + +A lock is useless if the client cannot take part in it, so the version +travels through the API in both directions. The response record exposes +it: + +```java +public record PostResponse(UUID id, long authorId, String title, String body, + long version, Instant createdAt, Instant updatedAt) { +} +``` + +and the update request requires it back: + +```java +public record UpdatePostRequest( + @NotBlank @Size(max = 160) String title, + @NotBlank @Size(max = 10_000) String body, + @NotNull @Min(0) Long version) { +} +``` + +The learning note next to `UpdatePostRequest` says it plainly: "the +expected version is part of the update contract, making optimistic +locking visible to clients." + +### 3. The service checks it inside the transaction that owns the write + +```java +@CachePut(cacheNames = "posts", key = "#postId") +@Transactional +public PostResponse updatePost(UUID postId, UpdatePostRequest request) { + Post post = postRepository.findById(postId) + .orElseThrow(() -> new PostNotFoundException(postId)); + if (post.getVersion() != request.version()) { + throw new PostVersionConflictException(postId); + } + post.update(request.title().trim(), request.body().trim()); + return PostResponse.from(postRepository.saveAndFlush(post)); +} +``` + +The load, the comparison and the save all happen inside one +`@Transactional` method, and that is the part that makes the check +honest: the version is compared against the row as it stands *at write +time*, not against a snapshot from an earlier request. If the read and +the write lived in different transactions, the check would compare +against a version the row has already left behind, and the whole scheme +would be theatre. The learning note on `PostService` names the rule: +"service methods centralize transaction boundaries, cache coherence, and +optimistic-locking rules." The demo also runs with +`spring.jpa.open-in-view: false`, so no lingering session serves a stale +entity across requests — the row is re-read inside the writing +transaction. + +The explicit check catches stale requests deterministically. The +`@Version` column still matters underneath: if another commit slips in +between the check and the flush, the conditional `UPDATE` affects zero +rows and Hibernate refuses the write anyway. + +### 4. The handler maps the exception to a 409 + +The exception itself carries the message a client can act on: + +```java +public class PostVersionConflictException extends RuntimeException { + + public PostVersionConflictException(UUID postId) { + super("Post %s has changed; fetch it again before retrying".formatted(postId)); + } +} +``` + +and the error boundary turns it into the right HTTP answer: + +```java +@ExceptionHandler(PostVersionConflictException.class) +ProblemDetail handleConflict(PostVersionConflictException exception) { + return problem(HttpStatus.CONFLICT, "POST_VERSION_CONFLICT", exception.getMessage()); +} +``` + +`HttpStatus.CONFLICT` is HTTP 409, and the response is a problem document +— a `type` URI, a stable machine code (`POST_VERSION_CONFLICT`) and a +human-readable detail. The demo enables the framework's problem-details +support in its configuration (`spring.mvc.problemdetails.enabled: true`), +so validation and every other error ride the same shape. + +### 5. The whole path, end to end + +```text +stale PUT /api/posts/{id} + -> PostService.updatePost re-reads the row inside its transaction + -> post.getVersion() != request.version() + -> PostVersionConflictException: "fetch it again before retrying" + -> ApiExceptionHandler.handleConflict + -> HTTP 409, ProblemDetail with code POST_VERSION_CONFLICT + -> the client knows its premise was stale +``` + +One `@Version` field, one `version` in the request contract, one +exception, one handler method. Nothing on the controller changed — +`updatePost` still looks like an ordinary `PUT`. + +## Why a conflict is the correct answer, not a retry + +When two editors produce two different versions of the same record, the +server cannot know which intent is the one meant. Both saves are +legitimate from their authors' point of view, and replaying the stale +write just replays the same data loss. + +The honest verdict is HTTP's own definition of 409: the request conflicts +with the current state of the resource, and the client is told exactly +how to resolve it — "fetch it again before retrying". The losing client +re-reads the record, sees the other writer's change, and a human decides +what to keep. The silent overwrite becomes a visible decision point, +which is the entire purpose of the exercise. + +## What the rest of the demo proves + +The version field is one layer of a small system, and the rest of it is +verifiable the same way — every row below is a class or a config file in +the repository, not a claim: + +| Layer | What it does | Where it lives | +|---|---|---| +| Caching | Bounded Caffeine cache on single-post reads (max 500, TTL 10 min, both from config); `@Cacheable` on read, `@CachePut` on update, `@CacheEvict` on delete; `GET /api/showcase/cache` exposes requests, hits, misses and hit rate | `CacheConfig`, `CacheProperties`, `ShowcaseMetricsController` | +| Security | Stateless HTTP Basic with BCrypt; pages and read APIs are public, every mutation needs the `EDITOR` role; local defaults are a throwaway `demo-editor`/`changeit` account overridable via env vars | `SecurityConfig`, `application.yml` | +| Errors | Request records validate at the boundary; one `@RestControllerAdvice` returns consistent problem documents with machine codes — `POST_NOT_FOUND` (404), `POST_VERSION_CONFLICT` (409), `VALIDATION_FAILED` (400) with a field-error map | `CreatePostRequest`, `UpdatePostRequest`, `ApiExceptionHandler` | +| Configuration | Local profile runs an in-memory H2 database in PostgreSQL mode with `ddl-auto: update` and seeded sample posts; `prod` selects PostgreSQL, takes every credential from `APP_*` env vars and sets `ddl-auto: validate` | `application.yml`, `application-prod.yml` | +| Tests | One `@SpringBootTest` suite asserts the rendered Thymeleaf pages, unauthenticated 401s versus editor 201s, the `VALIDATION_FAILED` problem response, public search, and cache hits after repeated reads | `DemoApplicationIntegrationTest` | + +## What I would do differently for anything real + +Two things change the moment this stops being a local demo. The project +README says both, and they are worth repeating as my own judgement. + +First, schema management. I would add Flyway or Liquibase migrations +before ever setting `ddl-auto` to `validate`. The demo's default profile +uses `update`, which is convenient on a scratch database and dangerous as +a habit — production schema should be versioned code, not a side effect +of startup. + +Second, secrets. The local editor account is intentionally a non-secret +demo account with default credentials. For anything real, I would keep +credentials in a managed secret store and refuse to boot without them, +instead of falling back to a default. The `prod` profile already takes +every credential from environment variables with no committed secrets — +that is the shape I would keep, minus the fallbacks. + +## The result + +A stale update now returns HTTP 409 with code `POST_VERSION_CONFLICT` +instead of a 200 that silently discards the other writer's change. The +loss can no longer happen quietly: the failure is loud, actionable — +"fetch it again before retrying" — and the integration test suite asserts +the access and validation behaviour end to end. + +The whole fix is one annotation on one field, plus plumbing that makes +the version part of the contract. If a record can be read by two people +at the same time, that field is the difference between a lost edit and a +conflict both clients can see. + +I build Java/Spring back ends and full-stack web applications — REST APIs +with JPA, optimistic locking, caching, validation and integration tests, +from the browser page down to the database. If you have a form or a field +where edits keep quietly disappearing, tell me about it: +[WhatsApp](https://wa.me/60127972969) · +[me@hoelee.com](mailto:me@hoelee.com?subject=Spring%20Boot%20help) · +[hoelee.com](https://hoelee.com). +```` + +--- + +## 文章 08 / 11：把 CodeIgniter 4 应用从 IIS 迁移到 OpenLiteSpeed +- **Slug**: `migrating-codeigniter-iis-to-openlitespeed` +- **英文标题**: Migrating a CodeIgniter 4 App From IIS to OpenLiteSpeed +- **分类**: `engineering` | **发布日期**: 2026-09-27 +- **线上地址**: https://blog.hoelee.com/posts/migrating-codeigniter-iis-to-openlitespeed/ (EN) | https://blog.hoelee.com/posts/zh/migrating-codeigniter-iis-to-openlitespeed/ (ZH) +- **本篇审核状态**: ⏳ 待审核 (Pending) + +### ✍️ 顾问审阅意见区 (Review Feedback) +```text +【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 +【事实与数字核对】： +【人设与用词卫生】： +【CRO与转化建议】： +``` + +### 🇨🇳 中文版原文 (Chinese Markdown Twin) +````markdown +--- +title: "把 CodeIgniter 4 应用从 IIS 迁移到 OpenLiteSpeed" +description: "我把一个 17,000 行的 CodeIgniter 4 应用从 Windows IIS 搬到了 CyberPanel 上的 OpenLiteSpeed。迁移后冒出两个致命错误——而 IIS 一直靠 SSO 把它们藏着。" +pubDate: 2026-09-27 +category: engineering +blueprint: sme-digital-suite +blueprintLayer: L1-high-conversion-web +tags: [codeigniter, php, openspeedway, cyberpanel, iis, migration, litespeed, linux] +ogImage: /og/zh-migrating-codeigniter-iis-to-openlitespeed.jpg +banner: /banners/migrating-codeigniter-iis-to-openlitespeed.jpg +draft: false +--- + +我维护着一个数字命理报告生成器：一个 CodeIgniter 4 应用，接收出生日期和姓名，跑一遍命理引擎，渲染出 19 页 A4 报告。它在 Windows Server + IIS 上跑了很多年。2026 年 9 月，我把它迁到 CyberPanel 上的 OpenLiteSpeed，跑在 Linux 上，换了个域名。 + +迁移本身没什么好说的——拷文件、把 docroot 指向 `public/`、装依赖。真正值得写下来的是：**新主机上第一个未认证请求，就在两条路由上直接致命崩溃——而这两条路由在 IIS 上"一直好好的"。** + +两个 bug 都是真的。两个从一开始就存在于 IIS 上。两个都看不见，因为 IIS 在出问题的路由前面恰好挡了一层 authentik SSO。 + +## 为什么这件事重要 + +如果你的自托管应用前面挂了一层认证网关，那这层网关做的事不止是保护数据——它同时**在帮你藏 bug**。任何"只对未认证用户出错"的东西永远不会以未认证身份运行，所以永远不会报错。而它会在最糟的时刻暴露：你迁移的时候、你拆掉网关的时候、你把路由暴露到公网的时候。 + +我这个应用，就"健康"了整整几年。我把新域名指过去的那一刻，两条路由返 HTTP 500。代码一行没改。唯一变的是网关没了。 + +两个修复加起来大概二十分钟。找到它们花了两个小时——因为真正的错误信息被 CodeIgniter 的 CLI 错误渲染器挡在了后面。 + +## 环境 + +这是一个相当普通的 CI4 项目，但有一个不太常见的架构细节：报告引擎不是库，而是**同一台主机上的 HTTP 端点**。控制器拼一个请求，`curl` 回自己域名的 `/api/single`，引擎返回算好的 JSON 数值，控制器再把它渲染成可打印的报告。 + +正是这个"自调用"设计，让 IIS 上必须有 `CONST_IIS_INTERNAL_BASE`。这点我后面会回来说——它才是真正有意思的部分。 + +| | 迁移前 | 迁移后 | +|---|---|---| +| 操作系统 | Windows Server | Ubuntu (CyberPanel) | +| Web 服务器 | IIS | OpenLiteSpeed 1.9.0 | +| PHP | 8.4 (Windows 版) | 8.4.25 (lsphp84) | +| Docroot | `C:\inetpub\calc.hoelee.com` | `/home/<域名>/public_html/public` | +| 认证网关 | authentik SSO 挡 `/lifecode`、`/api/*` | 无 | + +## 第一步：docroot 指向 `public/`，不是项目根目录 + +CI4 是两段式目录结构。`app/`、`vendor/`、`writable/` 和 `.env` 都在项目根目录，只有 `public/` 本该对 Web 可见。 + +CyberPanel 建的站点根目录叫 `public_html`。我的第一反应是把整个项目解压进 `public_html`，docroot 不动。 + +**那样会把 `app/`、`vendor/`、`.env` 文件和 `writable/` 的 session 数据全部暴露在 HTTP 之下。** 这个项目的 `.env` 里有一个 webhook 凭证。任何人猜到 `/../.env`——或者干脆直接请求 `/.env`，因为这文件就躺在被服务的目录下面——就拿到了。 + +正确做法是解压进 `public_html`，然后把 vhost 的 docroot 往下一层，改成 `public_html/public`： + +```bash +# 解压成 app/、vendor/、.env 位于 public_html 之下 +cd /home/<域名>/public_html +tar -xzf /tmp/deploy.tar.gz + +# docroot 必须是 public_html/public，不是 public_html +sudo sed -i 's#/home/<域名>/public_html\$#/home/<域名>/public_html/public#' \ + /usr/local/lsws/conf/vhosts/<域名>/vhost.conf + +sudo /usr/local/lsws/bin/lshttpd -t +sudo systemctl restart lsws +``` + +这是整个迁移里最重要的一步，也是大多数教程跳过的一步。其他什么都不做，这一步也得做。 + +## 第二步：IIS 藏起来的两个致命错误 + +### 致命错误一 —— `env()` 调用得太早 + +第一个 500 完全没有响应体。这不太寻常——CI4 正常会渲染点东西出来。一个空白的 500 通常意味着 PHP 在框架的错误处理器装上之前就死了。 + +CI4 的启动流程会在 `Boot::bootWeb()` 里很早加载 `app/Config/Constants.php`（通过 `Boot::loadConstants()`）。而这件事发生在 **`Common.php` 助手文件加载之前**——`env()` 就定义在那个文件里。 + +所以这一行： + +```php +// app/Config/Constants.php —— 错误写法 +define('CONST_fullBase', env('hoelee.fullBase')); +``` + +会失败于： + +``` +Fatal error: Uncaught Error: Call to undefined function env() +in app/Config/Constants.php:96 +``` + +规则是绝对的：**`Constants.php` 里只能放普通常量。** 不能有 `env()`、不能有 `getenv()`、不能有配置助手。如果你真需要依赖环境的值，就定义到启动流程更靠后的位置——或者把这个常量降级成 fallback，真正的值后面再读。 + +我做的正是后者。`CONST_fullBase` 变回普通字符串，消费它的助手函数改为**先**查框架自己的 `app.baseURL`（那个才是 `.env` 驱动的）： + +```php +// app/Helpers/hoelee_helper.php +function getFullBase(bool $selfCall = false): string +{ + // .env 驱动的 baseURL 优先；CONST_fullBase 只是 fallback + if ($selfCall && defined('CONST_IIS_INTERNAL_BASE') && CONST_IIS_INTERNAL_BASE) { + return rtrim(CONST_IIS_INTERNAL_BASE, '/'); + } + $appBase = config('App')->baseURL; + if ($appBase) return rtrim($appBase, '/'); + if (defined('CONST_fullBase') && CONST_fullBase) return rtrim(CONST_fullBase, '/'); + return ''; +} +``` + +**这个是我自己搞出来的。** 同一周做凭证清理重构时引入的——我把一个硬编码 URL 挪进 `.env`，然后在 `Constants.php` 里用 `env()` 读它。在我机器上是通的，因为本地 `.env` 走的是另一条路径被读到。干净启动下它从来没通过。我是在**部署测试**后一小时内抓到的；如果只是代码审查，很可能就这么放过去了。 + +### 致命错误二 —— CI4 控制器里的 `parent::__construct()` + +第二个失败在 `/lifecode`。这个是应用里**原本就存在**的 bug，不是我引入的。 + +绕过 CLI 渲染器之后，真正的报错是： + +``` +Error: Cannot call constructor +``` + +CI4 的基础类 `CodeIgniter\Controller` **根本没有构造函数**。它实现的是 `initController()`，由框架把 request、response、logger 三个对象传进去。标准写法是： + +```php +// CI4 的正确写法 +public function initController( + RequestInterface $request, + ResponseInterface $response, + LoggerInterface $logger +) { + parent::initController($request, $response, $logger); + // 你的初始化代码 +} +``` + +但 `Lifecode.php` 和 `ApiEn.php` 用的是传统构造函数，还在调 `parent::__construct()`： + +```php +// 错误 —— CodeIgniter\Controller 没有 __construct() +public function __construct() +{ + parent::__construct(); + // ... +} +``` + +对一个没有定义 `__construct()` 的父类调 `parent::__construct()`，在 PHP 8 里是致命错误。把两个控制器改成 `initController()` 就修好了——同时要为新签名补上三个 interface 的 `use` 语句。 + +我没有盲信报错信息，而是先确认父类**确实**没有构造函数： + +```bash +grep -n 'function __construct\|function initController' \ + vendor/codeigniter4/framework/system/Controller.php +``` + +只出来 `initController()`。这一步值得做——"Cannot call constructor" 在父类**有**构造函数但内部报错时也会出现，重写签名之前你得先搞清楚自己是哪种情况。 + +## 为什么这两个 bug 在 IIS 上都不出现 + +这是让我改变部署观念的部分。 + +在 IIS 主机上，`/lifecode` 和 `/api/*` 挡在 authentik SSO 后面。对这两条路由的未认证请求返回 **HTTP 302 跳转到 `auth.hoelee.com`**——压根不会碰到控制器。那个坏掉的构造函数从来没被执行过。这条路由大概从写出来那天就是坏的，而它从来没被要求真正处理过一次请求。 + +我直接对比了两台主机来确认： + +```bash +curl -sI https://calc.hoelee.com/lifecode | head -1 +# HTTP/2 302 <- authentik 跳转；控制器根本没运行 + +curl -sI http://<新主机>/lifecode | head -1 +# HTTP/1.1 500 <- 没有网关；控制器运行并致命崩溃 +``` + +那个 302 就是这个 bug 活下来的原因。应用看起来健康，只是因为它不健康的部分根本触不到。 + +**教训说白了就是：一条路由前面挂了认证网关，就意味着这条路由自己的代码没有任何有效的测试覆盖。** 如果你要迁移或者拆网关，请专门留出时间，用未认证身份把每一条曾被网关挡着的路由都打一遍，再宣布迁移完成。小应用上这只是十行 `curl` 循环。在我这里，它本来几秒钟就能找出这两个 bug，而不是两个小时： + +```bash +for p in / /read/single /read/partner /lifecode /api/date /api/single; do + printf '%-16s %s\n' "$p" "$(curl -s -o /dev/null -w '%{http_code}' https://<主机>$p)" +done +``` + +## 第三步：那个不再是 loopback 的自调用 + +现在是这个架构里真正有意思的后果。 + +报告引擎是靠控制器 `curl` 自己的主机来调用的。在 IIS 上，`CONST_IIS_INTERNAL_BASE` 把这个调用指向 `http://localhost:7296`——loopback 接口，请求不出机器，也不碰 DNS 和 TLS。 + +在 LiteSpeed 上这个常量被**故意不定义**，于是 `getFullBase()` 落到 `app.baseURL`。这意味着每一次生成报告，现在都是**一个真实的对外 HTTPS 请求，打到应用自己的公网 URL，再从 Cloudflare 绕回来**。 + +那问题就来了：这到底能不能跑？ + +答案是能，但值得专门测一次，因为它的失败模式很容易误导人。这是确切的测法——**在服务器上跑**，不是在你的笔记本上： + +```bash +curl -s -o /dev/null -w 'HTTPS self-call: %{http_code}\n' \ + -X POST 'https://<域名>/api/single' -d 'nameCn=test&dob=1990-01-01' +``` + +两个要注意的地方： + +- **在服务器上跑。** 我从 Windows 机器上打同一个 URL，拿到的是 404 和 500——那是我这台机器 DNS 解析和 Cloudflare 机器人防护的产物，不是服务器的问题。在错误的主机上测试，会得出很有把握的错误结论。 +- **看 JSON，不只看状态码。** 一个 200 配一个被截断的 body，比干脆失败更糟。我在信任之前先确认了响应能被解析。 + +这里藏着一个真实的取舍，我还没完全解决：走公网 URL 的自调用意味着生成报告依赖 Cloudflare 在线、每次报告多一次 TLS 握手、并且在整个过程中占用两个 PHP worker。在 LiteSpeed 上 worker 池不大的情况下，并发报告一多就可能死锁——每个请求都在等另一个请求，而后者拿不到空闲 worker。以目前的流量没问题。但在规模上去之前，这个自调用应该换成绕过 CDN 的内部路径。 + +## 第四步：验证真实产出，不是状态码 + +`/` 返 200 只证明落地页能渲染。它不证明报告引擎能用，而引擎才是整个产品。所以最后一步是端到端生成一份真报告： + +```bash +curl -s -X POST 'https://<域名>/read/single' \ + -d 'nameCn=test&dob=1990-01-01&gender=m' \ + -o report.html -w 'HTTP:%{http_code} bytes:%{size_download}\n' +``` + +结果：**HTTP 200、99,189 字节、19 页 A4。** 我又用中文字符输入跑了一遍，把引擎的 UTF-8 路径也走通——HTTP 200、99,205 字节、19 页、姓名渲染正常、输出里零 PHP 警告。 + +真正有意义的检查，是新旧主机之间页数可比。它证明字体、DPI 和分页逻辑都完整地熬过了这次搬迁。 + +关于那个 UTF-8 测试的插曲：我第一次跑的时候返回了 500，差点就一头扎进去查 bug。原因是我自己的 shell——中文字符在进 `curl` 的路上被终端编码搞坏了，应用收到的是非法字节。把同一个请求写成服务器上的脚本再跑，编码在我控制之内，就返回了 200。**在你调试编码故障之前，先确认服务器真正收到的字节就是你打算发出去的字节。** + +## 如果重来一次，我会怎么改 + +**迁移前就测未认证路由，而不是迁移后。** 那两个小时的排查，用一个遍历路由列表的 `curl` 循环就能完全避免。我现在把"列出所有路由、以未认证身份逐个请求、记录状态码"当成任何迁移的第零步。 + +**不要在 `Constants.php` 里调 `env()`。** 我在文件里留了注释专门写这件事，因为下一个人——多半是六个月后的我自己——一定会想这么干。 + +**迁移前先查框架版本对应的 project-space 配置。** 同一周我把 CI4 从 4.6.3 升到 4.7.4，它在两个升级指南里根本没提的属性上致命崩溃了两次：`Config\App::$permittedURIChars`（4.7 的 Router 要求）和 `Config\Format::$jsonEncodeDepth`（JSONFormatter 要求，而第二个正好打断了引擎的 JSON 自调用）。Composer 会更新 `vendor/`，但**永远**不会合并 `app/Config/*.php`，因为那些属于 project-space。我最后往 14 个配置文件里手工合并了新属性。那是另一篇文章，但迁移的教训是同一个形状：**框架会告诉你 `vendor/` 里改了什么；没有任何东西会告诉你 `app/` 里改了什么。** + +**先确认 docroot，再动别的。** 如果我解压进 `public_html` 就收手，应用是能跑的——同时静静地把 `.env` 通过 HTTP 服务出去。能跑的迁移，不等于安全的迁移。 + +## 结果 + +一个晚上。六条路由全部验证 200，中英文输入的报告产出在字节层面可比，零 PHP 警告，另外还从代码库里清掉了两个长期潜伏的 bug——而这两个 bug 一直被 IIS 的认证网关遮着。 + +应用现在跑在 OpenLiteSpeed 上，docroot 正确分离，主机可自动化管理，凭证放在 `.env` 而不是写死在常量里。 + +--- + +**想让你的应用也做这套迁移？** 我可以在 IIS、Apache、nginx 和 LiteSpeed 之间迁移 PHP 应用——包括那些麻烦的部分：自调用架构、藏 bug 的 SSO 网关，以及会碰到 project-space 配置的框架升级。 + +[WhatsApp +60 12-797 2969](https://wa.me/60127972969) · [me@hoelee.com](mailto:me@hoelee.com?subject=CodeIgniter%20迁移) · [hoelee.com](https://hoelee.com) +```` + +### 🇬🇧 英文版原文 (English Markdown Primary) +````markdown +--- +title: "Migrating a CodeIgniter 4 App From IIS to OpenLiteSpeed" +description: "I moved a 17,000-line CodeIgniter 4 app from Windows IIS to OpenLiteSpeed on CyberPanel. Two fatal errors appeared that IIS had been hiding behind SSO." +pubDate: 2026-09-27 +category: engineering +blueprint: sme-digital-suite +blueprintLayer: L1-high-conversion-web +tags: [codeigniter, php, openspeedway, cyberpanel, iis, migration, litespeed, linux] +ogImage: /og/migrating-codeigniter-iis-to-openlitespeed.jpg +banner: /banners/migrating-codeigniter-iis-to-openlitespeed.jpg +draft: false +--- + +I maintain a numerology report generator: a CodeIgniter 4 application that takes a birth date and a name, runs them through a numerology engine, and renders a 19-page A4 report. It had run on Windows Server with IIS for years. In September 2026 I moved it to OpenLiteSpeed on CyberPanel, on Linux, on a different domain. + +The migration itself was unremarkable — copy the files, point the docroot at `public/`, install dependencies. What made it worth writing about is that **the first unauthenticated request to the new host fatally crashed on two routes that had been working fine on IIS for years.** + +Both bugs were real. Both were present on IIS the whole time. Neither was visible, because IIS had an authentik SSO gate in front of the exact routes that were broken. + +## Why this matters + +If you run a self-hosted app behind an authentication gateway, your auth layer is doing more than protecting data — it is also **hiding your bugs**. Anything that only breaks for unauthenticated users never runs unauthenticated, so it never fails. The failure surfaces at the worst possible moment: when you migrate, remove the gate, or expose the route to the public internet. + +In my case the app had been reporting "healthy" for as long as I'd owned it. The moment I pointed a new domain at it, two routes returned HTTP 500. Nothing had changed in the code. The only thing that changed was that the gate was gone. + +The fix for both took about twenty minutes. Finding them took two hours, because the actual error messages were hidden behind CodeIgniter's CLI error renderer. + +## The setup + +The app is a fairly ordinary CI4 project with an unusual architecture detail: the report engine is not a library, it is a **set of HTTP endpoints on the same host**. The controllers generate a request, `curl` it back to `/api/single` on their own domain, and the engine returns JSON with the computed numbers. The controller then renders that into the printable report view. + +That self-call design is what made `CONST_IIS_INTERNAL_BASE` necessary on IIS. Let me come back to it — it turns out to be the interesting part. + +| | Before | After | +|---|---|---| +| OS | Windows Server | Ubuntu (CyberPanel) | +| Web server | IIS | OpenLiteSpeed 1.9.0 | +| PHP | 8.4 (Windows build) | 8.4.25 (lsphp84) | +| Docroot | `C:\inetpub\calc.hoelee.com` | `/home//public_html/public` | +| Auth gate | authentik SSO on `/lifecode`, `/api/*` | none | + +## Step 1: Point the docroot at `public/`, not the project root + +CI4 ships with a two-folder layout. `app/`, `vendor/`, `writable/` and `.env` live in the project root. Only `public/` is meant to be web-accessible. + +CyberPanel creates the document root as `public_html`. My first instinct was to extract the whole project into `public_html` and leave the docroot alone. + +**That would have exposed `app/`, `vendor/`, the `.env` file, and `writable/` session data over HTTP.** The `.env` in this project contains a webhook credential. Anyone who guessed `/../.env` — or just fetched `/.env`, since the file sits directly under a served directory — gets it. + +The correct layout is to extract into `public_html` and then repoint the vhost docroot one level deeper, at `public_html/public`: + +```bash +# extract the project so that app/, vendor/ and .env sit UNDER public_html +cd /home//public_html +tar -xzf /tmp/deploy.tar.gz + +# the docroot must be public_html/public, NOT public_html +sudo sed -i 's#/home//public_html\$#/home//public_html/public#' \ + /usr/local/lsws/conf/vhosts//vhost.conf + +sudo /usr/local/lsws/bin/lshttpd -t +sudo systemctl restart lsws +``` + +This is the single most important step in the migration, and it is the one most guides skip. If you do nothing else, do this. + +## Step 2: The two fatals IIS was hiding + +### Fatal 1 — `env()` called too early + +The first 500 had no body at all. That is unusual — CI4 normally renders something. An empty 500 usually means PHP died before the framework's error handler was installed. + +CI4's bootstrap loads `app/Config/Constants.php` very early in `Boot::bootWeb()`, via `Boot::loadConstants()`. That happens **before** the `Common.php` helper file is loaded, which is where `env()` is defined. + +So this line: + +```php +// app/Config/Constants.php — BROKEN +define('CONST_fullBase', env('hoelee.fullBase')); +``` + +fails with: + +``` +Fatal error: Uncaught Error: Call to undefined function env() +in app/Config/Constants.php:96 +``` + +The rule is absolute: **`Constants.php` may only contain plain constants.** No `env()`, no `getenv()`, no config helper. If you need environment-dependent values there, define them further along the bootstrap — or make the constant a fallback and read the real value later. + +That is exactly what I did. `CONST_fullBase` became a plain string, and the helper that consumes it checks the framework's own `app.baseURL` (which *is* `.env`-driven) first: + +```php +// app/Helpers/hoelee_helper.php +function getFullBase(bool $selfCall = false): string +{ + // the .env-driven baseURL wins; CONST_fullBase is only a fallback + if ($selfCall && defined('CONST_IIS_INTERNAL_BASE') && CONST_IIS_INTERNAL_BASE) { + return rtrim(CONST_IIS_INTERNAL_BASE, '/'); + } + $appBase = config('App')->baseURL; + if ($appBase) return rtrim($appBase, '/'); + if (defined('CONST_fullBase') && CONST_fullBase) return rtrim(CONST_fullBase, '/'); + return ''; +} +``` + +**This one was my own doing.** I had introduced it during a secrets-cleanup refactor in the same week — I moved a hardcoded URL into `.env` and called `env()` in `Constants.php` to read it. It worked on my machine because the local `.env` was being read through a different path. It never worked on a clean boot. I caught it within an hour *only because* I deployed and tested; a code review would plausibly have waved it through. + +### Fatal 2 — `parent::__construct()` in a CI4 controller + +The second failure was on `/lifecode`, and this one is a genuine pre-existing bug in the application — not something I introduced. + +The error, once I got past the CLI renderer, was: + +``` +Error: Cannot call constructor +``` + +CI4's base `CodeIgniter\Controller` class **has no constructor**. It implements `initController()`, which the framework calls with the request, response, and logger objects. The standard pattern is: + +```php +// correct CI4 pattern +public function initController( + RequestInterface $request, + ResponseInterface $response, + LoggerInterface $logger +) { + parent::initController($request, $response, $logger); + // your setup here +} +``` + +But `Lifecode.php` and `ApiEn.php` declared a classic constructor and called `parent::__construct()`: + +```php +// BROKEN — CodeIgniter\Controller has no __construct() +public function __construct() +{ + parent::__construct(); + // ... +} +``` + +`parent::__construct()` on a parent class that does not define `__construct()` is a fatal in PHP 8. Converting both controllers to `initController()` fixed it — and required adding the three `use` statements for the interface types in the new signature. + +I checked whether the base class *really* had no constructor rather than trusting the error message: + +```bash +grep -n 'function __construct\|function initController' \ + vendor/codeigniter4/framework/system/Controller.php +``` + +Only `initController()` came back. Worth doing — "Cannot call constructor" also fires when a parent *has* a constructor that errors internally, and you want to know which case you are in before you rewrite the signature. + +## Why neither bug showed up on IIS + +This is the part that changed how I think about the deployment. + +On the IIS host, `/lifecode` and `/api/*` sat behind authentik SSO. An unauthenticated request to either returned **HTTP 302 to `auth.hoelee.com`** — it never reached the controller at all. The broken constructor was never executed. The route had presumably been broken since it was written, and it had never once been asked to serve a request. + +I confirmed this by comparing the two hosts directly: + +```bash +curl -sI https://calc.hoelee.com/lifecode | head -1 +# HTTP/2 302 <- authentik redirect; controller never runs + +curl -sI http:///lifecode | head -1 +# HTTP/1.1 500 <- no gate; controller runs and fatals +``` + +The 302 is why the bug survived. The app looked healthy because the unhealthy parts were unreachable. + +**The lesson, stated plainly: an auth gate in front of a route means that route has no working test coverage for its own code.** If you migrate or remove the gate, budget time to hit every previously-gated route unauthenticated before you call the migration done. On a small app that is a ten-line `curl` loop. Here it would have found both bugs in seconds instead of two hours: + +```bash +for p in / /read/single /read/partner /lifecode /api/date /api/single; do + printf '%-16s %s\n' "$p" "$(curl -s -o /dev/null -w '%{http_code}' https://$p)" +done +``` + +## Step 3: The self-call that stops being a loopback + +Now the interesting architectural consequence. + +The report engine is reached by the controller `curl`-ing its own host. On IIS, `CONST_IIS_INTERNAL_BASE` pointed that call at `http://localhost:7296` — the loopback interface — so the request never left the machine and never touched TLS or DNS. + +On LiteSpeed that constant is deliberately left **undefined**, so `getFullBase()` falls through to `app.baseURL`. Which means every report generation now makes a **real outbound HTTPS request to the app's own public URL** and comes back in through Cloudflare. + +Which raises the obvious question: does that work at all? + +The answer is yes, but it is worth testing explicitly, because the failure mode is confusing. Here is the exact test — run it **on the server**, not from your laptop: + +```bash +curl -s -o /dev/null -w 'HTTPS self-call: %{http_code}\n' \ + -X POST 'https:///api/single' -d 'nameCn=test&dob=1990-01-01' +``` + +Two things to watch for: + +- **Run it from the server.** From my Windows machine the same URL returned 404 and 500 — an artefact of DNS resolution and Cloudflare bot protection on my caller, not a server problem. Testing from the wrong host produces confident, wrong conclusions. +- **Watch the JSON, not just the status code.** A 200 with a truncated body is worse than a clean failure. I checked the response was parseable before trusting it. + +There is a real trade-off hiding here, and I have not fully resolved it: a public-URL self-call means report generation depends on Cloudflare being up, costs a TLS handshake per report, and occupies two PHP workers for the duration. On LiteSpeed with a small worker pool, a burst of concurrent reports could deadlock — each request waiting on another request that has no free worker. For the current traffic level it is fine. Before this scales, the self-call should move to an internal path that bypasses the CDN. + +## Step 4: Verify the actual output, not the status code + +A 200 on `/` proves the landing page renders. It does not prove the report engine works, and the engine is the entire product. So the last step was generating a real report end to end: + +```bash +curl -s -X POST 'https:///read/single' \ + -d 'nameCn=test&dob=1990-01-01&gender=m' \ + -o report.html -w 'HTTP:%{http_code} bytes:%{size_download}\n' +``` + +The result: **HTTP 200, 99,189 bytes, 19 A4 pages.** I ran it again with Chinese-character input to exercise the UTF-8 path through the engine — HTTP 200, 99,205 bytes, 19 pages, name rendering correctly, zero PHP warnings in the output. + +Byte-comparable page counts between the old and new host is the check that matters. It proves the fonts, the DPI, and the pagination logic all survived the move. + +A side note on the UTF-8 test: my first attempt at it returned a 500 and I nearly went bug-hunting. The cause was my own shell — the Chinese characters were being mangled by the terminal encoding on the way into `curl`, so the app received invalid bytes. Running the same request from a script on the server, where the encoding is under my control, returned 200. **Before you debug an encoding failure, confirm the bytes actually arriving at the server are the bytes you meant to send.** + +## What I'd do differently + +**Test unauthenticated routes before migrating, not after.** The whole two-hour debugging session was avoidable with a `curl` loop over the route list. I now treat "list every route, hit it unauthenticated, record the status" as step zero of any migration. + +**Do not call `env()` in `Constants.php`.** I have left a comment in the file itself saying so, because the next person — probably me in six months — will be tempted. + +**Check the framework version's project-space config before the move.** The same week, I upgraded CI4 from 4.6.3 to 4.7.4, and it fataled twice on properties the upgrade guide did not mention: `Config\App::$permittedURIChars`, required by the 4.7 Router, and `Config\Format::$jsonEncodeDepth`, required by the JSONFormatter — the second of which broke the engine's JSON self-call specifically. Composer updates `vendor/` but never merges `app/Config/*.php`, because those are project-space files. I ended up merging new properties into 14 config files. That is a separate post, but the migration lesson is the same shape: **the framework tells you what changed in `vendor/`; nothing tells you what changed in `app/`.** + +**Confirm the docroot before anything else.** If I had extracted into `public_html` and stopped, the app would have worked — and quietly served `.env` over HTTP. A migration that works is not the same as a migration that is safe. + +## The result + +One evening. Six routes verified 200, byte-comparable report output on both Chinese and ASCII input, zero PHP warnings, and two long-standing latent bugs removed from the codebase that IIS had been concealing behind an auth gate. + +The app is now running on OpenLiteSpeed with a properly separated docroot, on a host I can automate, with the credential in `.env` rather than a constant. + +--- + +**Want this done for your own application?** I migrate PHP applications between IIS, Apache, nginx and LiteSpeed — including the awkward parts: self-calling architectures, SSO gates that hide bugs, and framework upgrades that touch project-space config. + +[WhatsApp +60 12-797 2969](https://wa.me/60127972969) · [me@hoelee.com](mailto:me@hoelee.com?subject=CodeIgniter%20migration) · [hoelee.com](https://hoelee.com) +```` + +--- + +## 文章 09 / 11：用「热浏览器会话」爬取有反爬墙的电商网站 +- **Slug**: `scraping-bot-walled-marketplace-warm-browser-session` +- **英文标题**: Scraping a Bot-Walled Marketplace With a Warm Browser Session +- **分类**: `engineering` | **发布日期**: 2026-09-13 +- **线上地址**: https://blog.hoelee.com/posts/scraping-bot-walled-marketplace-warm-browser-session/ (EN) | https://blog.hoelee.com/posts/zh/scraping-bot-walled-marketplace-warm-browser-session/ (ZH) +- **本篇审核状态**: ⏳ 待审核 (Pending) + +### ✍️ 顾问审阅意见区 (Review Feedback) +```text +【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 +【事实与数字核对】： +【人设与用词卫生】： +【CRO与转化建议】： +``` + +### 🇨🇳 中文版原文 (Chinese Markdown Twin) +````markdown +--- +title: "用「热浏览器会话」爬取有反爬墙的电商网站" +description: "为一位做二手手机生意的客户：爬取有反爬墙的电商商品列表——热 CDP 会话、失败即止的健康检查、分段等待渲染。" +pubDate: 2026-09-13 +category: engineering +tags: ["scraping", "cdp", "headless-chrome", "anti-bot", "shopee", "python"] +ogImage: "/og/zh-scraping-bot-walled-marketplace-warm-browser-session.jpg" +banner: "/banners/scraping-bot-walled-marketplace-warm-browser-session.jpg" +draft: false +--- + +一位客户请我做一个听起来很简单的东西：每天把一个大电商平台上的二手手机列出来，按价格排序，好让他的进货决策靠数据而不是靠感觉。这位客户靠倒卖二手电子产品吃饭——一台手机差几百马币就是他的利润。 + +结果完全不是一段干净的 `requests` + BeautifulSoup 脚本。它变成了一场长达一周的、跟反爬墙的拉锯战，一个靠胶带勉强粘在一起的热浏览器会话，以及——我们俩都没想到的部分——一个关于「你不能靠堆更多 agent 来『并行化』爬虫」的惨痛教训。 + +这就是那个故事，按照我经历的顺序来讲：问题、我试过却失败的每一件事、最终奏效的办法，以及下次我会怎么做。 + +## 问题，用别人真正会搜索的方式表述 + +> "如何从一个拦截无头浏览器的网站爬取商品列表？" + +或者更诚实地说：**当一个电商网站已经把「防爬」当成它唯一的本职工作，你要怎么爬它？** Shopee 不会礼貌地把 HTML 喂给 `curl`。它会检测自动化、抛出一个 `/verify` 验证码，然后静默地给你一个「看起来正常但里面什么都没有」的页面。 + +客户的需求清单很短： + +- 搜索二手手机，只筛选「二手」成色，存储 128 GB 及以上。 +- 提取每个商品的所有存储规格和价格。 +- 按价格升序排序。 +- 不搞一百个 headless 实例的 `chromedriver` 农场——倒卖的利润养不起一百个住宅 IP，我的也一样。 + +从这里开始，全部都是调试故事。 + +## 我试过的，以及为什么失败 + +### 尝试 1：通过 CDP 用冷的无头 Chrome——「直接把 Chrome 远程调试开起来」 + +我用 `--headless --remote-debugging-port=9222` 启动了 Chrome，用 Chrome DevTools Protocol 连上去，然后导航到搜索 URL。大约十秒钟里我觉得自己很聪明。 + +**失败：** 页面加载了，但每个商品卡片都是空的。DOM 是个空壳。Shopee 指纹识别了这个无头浏览器（没有真正的 GPU、user-agent 指纹不一致、`navigator.webdriver` 标志），把我重定向进了一个我永远过不去的 `/verify` 验证码循环。冷的无头 profile 一上来就撞墙。 + +教训很无聊但很重要：**如果你的浏览器本身在「身份」上撒谎，选择器写得再好也没用。** + +### 尝试 2：并行子 agent——「直接开 N 个 worker」 + +这是我最想让它成功的方案。我有一套多 agent 的架构。本能反应就是：开五个 agent，每个爬一个不同的搜索词，收集结果，合并。 + +**失败：** 它们全都连到**同一个**浏览器会话上。它们不会各自拿到一个浏览器——只会各自拿到共享 Chrome 实例里的一个标签页，或者更糟：全都挤在同一个标签页上。每一次 `Page.navigate` 都会把上一个 agent 的上下文抹掉。而且关键在于： + +1. **热会话是共享资源。** 只有一个没被标记的 Canary profile。五个 agent 交错访问同一个标签页，在 Shopee 眼里，跟一个浏览器疯狂快速导航没有区别。而这恰恰是**触发**反爬墙的行为。 +2. **反爬的「节奏」是累积的。** 我的可用方案依赖导航之间约 7 秒的间隔来显得像真人。五个 agent 并发导航，就等于五个导航以零间隔打向同一个会话。 + +所以「横向扩展」的本能——加更多 worker——对我赖以存活的那件事（看起来像真人）是有害的。这里的并发上限不是 CPU 也不是内存，而是**一个热浏览器，一次一个标签页。** + +### 尝试 3：靠「Buy With Voucher」按钮来拿价格 + +一旦列表能渲染出来了，我假设购买按钮的文案里带着价格。大多数电商网站确实如此。但这个渲染出来的按钮写的是 `Add To Cart` / `Buy Now`。我用来匹配「Buy With Voucher … RM…」的正则什么都匹配不到，我在原地打转了好一会儿，才真正去读了 DOM。 + +**失败：** Shopee 商品页的价格不在按钮里。它被渲染成**拆分的 span**——`RM` 在一个元素里、`850` 在另一个里、`00` 又在另一个里——所以朴素的叶子文本匹配只会抓到碎片，或者当商品有多个规格时，抓到一个**区间**（`RM850.00 - RM1150.00`）。 + +### 尝试 4：点击规格选项来拿精确价格 + +对于单轴规格选择器（比如一个存储下拉：128GB / 256GB / 512GB），点击一个选项会把价格收敛成单个值，我的提取方法能正常工作。 + +**失败：** 对于**双轴**选择器（品牌 × 存储），点击第一个轴（某个品牌）会锁定一个价格，但加上第二个轴（某个存储规格）又把它**解散**回商品全区间。这个面板渲染的是「所有可购买组合」的区间，而不是我选中的那个具体组合；而且「已选中」状态是用一个「同时匹配其他所有选项框」的 class 来标记的。我没法可靠地知道我到底锁定了哪个组合。 + +我在这个上面花了实打实的时间，最后才接受这个诚实的答案：**对于真正的双轴商品，正确的数据是价格区间，而不是一个瞎猜的「每个组合」数字。** 发布一个「有时候能用」的脆弱覆盖方案，比老实报告一个区间更糟糕。 + +## 修复——真正奏效的东西 + +三块，都很小，都是吃了亏才学会的。 + +### 1. 热的人性化节奏的浏览器会话（地基） + +别跟墙硬刚。用一个已经登录过、正常浏览过的浏览器 profile——一个「热」profile。用真实的 7 秒间隔来导航。它更慢，但它是唯一能稳定活下来的方式。 + +```python +# 不是真正的代码——是它的「形状」。先导航，然后像真人一样等待。 +goto_url(target) +time.sleep(7) # 真人节奏，不可妥协 +wait_for_selector(".shopee-search-item-result__item") +``` + +### 2. 失败即止，别傻等超时 + +在任何爬取之前，先打 CDP 的 HTTP 端点，检查会话状态。如果 Canary 挂掉了、或标签页停在 `/verify` 上，就在约 1 秒内中止，而不是在「怎么还不渲染」的超时里浪费 40 秒。 + +```python +import json, urllib.request + +def session_healthy(): + try: + tabs = json.load(urllib.request.urlopen("http://127.0.0.1:9222/json")) + except Exception: + return False + for t in tabs: + if t.get("type") == "page" and "/verify" in t.get("url", ""): + return False # 撞到验证码墙——重新热会话，别爬了 + return True +``` + +就这么一个改动，把「静默的 40 秒死胡同」变成了「即时、可执行的失败」。 + +### 3. 正确的价格选择器：最深的纯 RM 节点 + +购买按钮是个谎言。价格是**那个文本内容纯粹是一个 RM 值的、最深的元素**——在选中规格之前是一个区间（`RM850.00 - RM1150.00`），之后是单个值（`RM850.00`）。遍历 DOM，取文本能匹配上的最深元素： + +```python +import re + +RM = re.compile(r"^RM\s?[\d,.]+(\s?[-–]\s?RM\s?[\d,.]+)?$") + +def find_price(root): + best, depth = None, -1 + for el in root.querySelectorAll("*"): + txt = (el.textContent or "").strip() + if txt and RM.match(txt) and " " not in txt.split("RM")[0]: + d = depth_of(el) + if d > depth: + best, depth = el, d + return best.textContent.strip() if best else None +``` + +这就是全部的诀窍。没有脆弱的 class 名。没有瞎猜。一个「整个文本就是一个价格」的节点，就是价格本身。 + +## 结果——一个可量化的成果 + +从单个热会话、一次串行遍历中，我提取了 **20+ 个二手手机商品及其各规格价格**，筛选到 128 GB 及以上，按价格升序排序。最便宜的真实目标——一台 128GB 机型——以 **RM299** 的价格浮出水面，另外还有几台 128/256GB 的选项在 RM450–880 区间。现在一条命令（`sweep.py ...`）就能重新检查整份候选清单并输出 JSONLines；客户每天进货前跑一次。 + +但我最庆幸的数字是这个：**零验证码。** 「热会话 + 人性化节奏 + 失败即止」这套组合，在整次遍历中一次都没触发过 `/verify`。 + +## 下次我会怎么做 + +1. **我会一开始就写「失败即止」的健康检查**，在任何爬取代码之前。三十秒的活，换来未来每次失败省四十秒。 +2. **我会从第一天就抵抗并行化的冲动。** 「爬虫太慢」的答案是把更多工作打包进一次串行遍历（批量搜索、对多个 URL 跑 `sweep.py`），而不是「再加 worker」。共享的浏览器状态让并发变得更糟，而不是更好。 +3. **我会立刻接受双轴的限制**，老实报告区间，而不是在一个根本没法可靠化的选择器上烧一个下午。 +4. **我会别再猜 DOM 结构，先去探测它。** 几乎每一个错误假设（URL 路径里 `/i.` vs `-i.`、把 `Buy With Voucher` 当价格锚点、「过滤器是个真正的 `` 复选框」）都来自**没有先读真实的 DOM**。 + +## 可复用的部分 + +所有这一切被做成了一个小的独立工具集——每个脚本都是一次自包含的 `python x.py ...` 调用： + +| 脚本 | 作用 | +|---|---| +| `search.py` | 粗粒度搜索 → 商品卡片 | +| `used_search.py` | 点击「二手」条件复选框，校验 URL 里的 `USED_ITEM`，再提取 | +| `detail.py` | 从商品页提取各规格价格（最深 RM 节点法 + 分段等待） | +| `sweep.py` | 一次会话内对 N 个 URL 做批量详情提取，按 item-id 去重，写 JSONLines，打印表格 | +| `cdp_common.py` | 共享的健康检查 + 导航辅助 | + +真正可复用的教训不是那些 Shopee 专属的选择器——而是**姿态**：**假设页面在 DOM 上撒谎，失败即止，用批量化而不是并行 worker 来扩展。** + +--- + +*这是一次真实的调试过程，不是事后补写的教程。这个爬虫工具集、以及这里的每一次失败，都发生在为一个做二手电子产品倒卖的客户构建它的过程中——我保留了能泛化的部分，砍掉了那些只对某个马来西亚电商网站、某个时间点有效的部分。* + +--- + +## 想为你的业务做这个吗？ + +我做定制爬虫和数据提取管道——热会话 CDP 爬虫、反爬墙感知的采集器、批量价格监控扫描——也做全栈 Web 应用（PHP/CodeIgniter、Java/Spring、React/TypeScript）和自托管自动化（n8n、Telegram 机器人、NAS 上的 Docker）。 + +如果你需要从某个「会反抗」的网站拿商品或价格数据，直接点进来，告诉我你想提取什么： + +- 💬 [WhatsApp](https://wa.me/60127972969) — 直接私信我 +- ✉️ [Email](mailto:me@hoelee.com?subject=Custom%20scraping%20pipeline) — 告诉我网站和数据 +- 🌐 [hoelee.com](https://hoelee.com) — 看看我还做了些什么 +```` + +### 🇬🇧 英文版原文 (English Markdown Primary) +````markdown +--- +title: "Scraping a Bot-Walled Marketplace With a Warm Browser Session" +description: "Scraping bot-walled marketplace listings for a client's used-electronics business: warm CDP session, fail-fast health checks, split-wait rendering." +pubDate: 2026-09-13 +category: engineering +tags: ["scraping", "cdp", "headless-chrome", "anti-bot", "shopee", "python"] +ogImage: "/og/scraping-bot-walled-marketplace-warm-browser-session.jpg" +banner: "/banners/scraping-bot-walled-marketplace-warm-browser-session.jpg" +draft: false +--- + +A client asked me to build something that sounds simple: every day, list the second-hand phones on a big marketplace, sorted by price, so their restocking decisions come from data instead of gut feel. The client resells used electronics for a living — a few hundred ringgit either way on a phone is the whole margin. + +The result was nothing like a clean `requests` + BeautifulSoup script. It was a week-long fight against an anti-bot wall, a warm browser session held together with duct tape, and — the part neither of us saw coming — a hard lesson about why you cannot just "parallelize" a scraper by throwing more agents at it. + +This is that story, told in the order I lived it: the problem, everything I tried that failed, the thing that finally worked, and what I'd do differently. + +## The problem, phrased the way someone would actually search it + +> "How do I scrape product listings from a site that blocks headless browsers?" + +Or, more honestly: **how do you scrape a marketplace that has already made scraping its sole job description?** Shopee doesn't politely serve HTML to `curl`. It detects automation, throws a `/verify` captcha, and silently serves you a page that looks fine but contains nothing. + +The client's requirement list was small: + +- Search for used phones, filter to "used condition only", storage 128 GB and up. +- Extract every listing's storage variants and prices. +- Sort by price ascending. +- Do it without a `chromedriver` farm of a hundred headless instances — a reseller's margins don't pay for a hundred residential IPs, and neither do mine. + +Everything after this point is the debugging story. + +## What I tried, and why it failed + +### Attempt 1: cold headless Chrome via CDP — "just point Chrome's remote debugger at it" + +I launched Chrome with `--headless --remote-debugging-port=9222`, connected with the Chrome DevTools Protocol, and navigated to a search URL. It felt clever for about ten seconds. + +**Failure:** the page loaded, but every product card was empty. The DOM was a shell. Shopee had fingerprinted the headless browser (no real GPU, no real user-agent fingerprint consistency, `navigator.webdriver` flag) and redirected me into a `/verify` captcha loop I never got past. Cold headless profiles hit the wall immediately. + +The lesson is boring but important: **anti-bot systems don't care how good your selectors are if the browser itself is lying about what it is.** + +### Attempt 2: parallel subagents — "just spawn N workers" + +This was the one I really wanted to work. I have a multi-agent setup. The natural instinct is: spawn five agents, each scraping a different search query, gather the results, merge. + +**Failure:** all of them attach to the *same* browser session. They don't get their own browser — they get their own *tab* in one shared Chrome instance, or worse, they all fight over the same single tab. Every `Page.navigate` un-does the previous agent's context. And critically: + +1. **The warm session is a shared resource.** There is one Canary profile that isn't flagged. Five agents navigated the same tab in an interleaved order, which is indistinguishable — to Shopee — from one browser doing rapid-fire navigation. That's exactly the behavior that *triggers* the wall. +2. **Anti-bot pacing is cumulative.** My working setup depends on ~7-second gaps between navigations to look human. Five agents navigating concurrently means five navigations hitting the same session with effectively zero gap. + +So the "horizontal scaling" instinct — more workers — is actively hostile to the one thing keeping me alive: looking human. The concurrency ceiling here isn't CPU or memory. It's **one warm browser, one tab at a time.** + +### Attempt 3: trusting the "Buy With Voucher" button for the price + +Once I had listings rendering, I assumed the buy button's label contained the price. On most marketplaces it does. On this render, the button said `Add To Cart` / `Buy Now`. My regex for "Buy With Voucher … RM…" matched nothing, and I went around in circles for a while before actually reading the DOM. + +**Failure:** the price on Shopee product pages isn't in the button. It's rendered as **split spans** — `RM` in one element, `850` in another, `00` in a third — so a naive leaf-text match returns fragments, or a **range** (`RM850.00 - RM1150.00`) when the listing has multiple variants. + +### Attempt 4: clicking variant options to get an exact price + +For a single-axis variant selector (e.g. one dropdown of storage options — 128GB / 256GB / 512GB), clicking one option collapses the price to a single value, and my extraction worked. + +**Failure:** for a **two-axis** selector (brand × storage), clicking the first axis (a brand) locks a price, but adding the second axis (a storage size) *un-collapses* it back to the product-wide range. The panel renders the range of all purchasable combinations, not the exact combination I selected, and the "selected" state is marked by a class that also matches every other option box. There is no reliable way to know which combination I actually locked. + +I spent real time on this before accepting the honest answer: **for genuinely two-axis listings, the correct data is the price range, not a guessed per-combination number.** Shipping a fragile override that "sometimes works" would be worse than reporting an honest range. + +## The fix — what actually worked + +Three pieces, all small, all learned the hard way. + +### 1. A warm, human-paced browser session (the foundation) + +Don't fight the wall. Use a browser profile that has already logged in and browsed normally — a "warm" profile. Navigate with real 7-second gaps. It is slower and it is the only thing that reliably survives. + +```python +# not real code — the *shape* of it. Navigate, then wait like a human. +goto_url(target) +time.sleep(7) # human pacing, non-negotiable +wait_for_selector(".shopee-search-item-result__item") +``` + +### 2. Fail fast, don't time out + +Before any scrape, hit the CDP HTTP endpoint and check the session state. If Canary is down or the tab is sitting on `/verify`, abort in ~1 second instead of wasting 40 seconds on a "did not render" timeout. + +```python +import json, urllib.request + +def session_healthy(): + try: + tabs = json.load(urllib.request.urlopen("http://127.0.0.1:9222/json")) + except Exception: + return False + for t in tabs: + if t.get("type") == "page" and "/verify" in t.get("url", ""): + return False # captcha wall — re-warm the session, don't scrape + return True +``` + +This one change turned "silent 40-second dead-ends" into "instant, actionable failure." + +### 3. The right price selector: deepest pure-RM node + +The buy button is a lie. The price is the **deepest element whose text is a pure RM value** — a range (`RM850.00 - RM1150.00`) before a variant is selected, a single value (`RM850.00`) after. Walk the DOM and take the deepest element whose text matches: + +```python +import re + +RM = re.compile(r"^RM\s?[\d,.]+(\s?[-–]\s?RM\s?[\d,.]+)?$") + +def find_price(root): + best, depth = None, -1 + for el in root.querySelectorAll("*"): + txt = (el.textContent or "").strip() + if txt and RM.match(txt) and " " not in txt.split("RM")[0]: + d = depth_of(el) + if d > depth: + best, depth = el, d + return best.textContent.strip() if best else None +``` + +That's the whole trick. No brittle class names. No guessing. A node whose entire text *is* a price is the price. + +## The result — one quantified outcome + +From a single warm session, in one serial pass, I extracted **20+ used-phone listings with per-variant prices**, filtered to 128 GB and up, sorted by price. The cheapest real target — a 128GB model — surfaced at **RM299**, alongside several 128/256GB options in the RM450–880 range. One command (`sweep.py ...`) now re-checks the whole shortlist and dumps JSONLines; the client runs it once a day before restocking. + +But the number I'm most glad about is this: **zero captchas.** The warm-session + human-pacing + fail-fast combo survived the entire sweep without tripping `/verify` once. + +## What I'd do differently + +1. **I'd write the fail-fast health check first**, before a single scraping line. It's thirty seconds of work that saves forty seconds per future failure. +2. **I'd resist the parallelization instinct day one.** The answer to "the scraper is slow" is "bundle more work into one serial pass" (batch search, `sweep.py` over many URLs), not "more workers." Shared browser state makes concurrency worse, not better. +3. **I'd accept the two-axis limitation immediately** and report ranges, instead of burning an afternoon on a selector that can't be made reliable. +4. **I'd stop guessing DOM structure and probe it.** Almost every mistaken assumption (`/i.` vs `-i.` in the URL path, `Buy With Voucher` as the price anchor, "the filter is a real `` checkbox") came from *not reading the actual DOM first*. + +## The reusable pieces + +All of this became a small, standalone toolkit — each script is one self-contained `python x.py ...` call: + +| Script | What it does | +|---|---| +| `search.py` | Coarse search → listing cards | +| `used_search.py` | Clicks the "Used" condition checkbox, verifies `USED_ITEM` in the URL, extracts | +| `detail.py` | Pulls per-variant prices from a product page (deepest-RM-node method + split-wait) | +| `sweep.py` | Batch detail over N URLs in one session, dedupes by item-id, writes JSONLines, prints a table | +| `cdp_common.py` | Shared health check + navigation helpers | + +The genuinely reusable lesson isn't the Shopee-specific selectors — it's the *posture*: **assume the page is lying about its DOM, fail fast, and scale by batching, not by parallel workers.** + +--- + +*This is a real debugging session, not a tutorial written after the fact. The scraping toolkit and every one of these failures happened while building it for a client who resells used electronics — I kept the parts that generalize and cut the parts that only matter to one Malaysian marketplace at one point in time.* + +--- + +## Want this for your business? + +I build custom scraping and data-extraction pipelines — warm-session CDP scrapers, bot-wall-aware crawlers, and batch price-monitoring sweeps — as well as full-stack web apps (PHP/CodeIgniter, Java/Spring, React/TypeScript) and self-hosted automation (n8n, Telegram bots, Docker on NAS). + +If you need product or price data off a site that fights back, tap through and tell me what you're trying to extract: + +- 💬 [WhatsApp](https://wa.me/60127972969) — message me directly +- ✉️ [Email](mailto:me@hoelee.com?subject=Custom%20scraping%20pipeline) — tell me the site + the data +- 🌐 [hoelee.com](https://hoelee.com) — see what else I build +```` + +--- + +## 文章 10 / 11：JavaScript 文件静默失效的两种方式：而页面看上去一切正常 +- **Slug**: `two-ways-a-javascript-file-stops-working-and-looks-fine` +- **英文标题**: Two Ways a JavaScript File Stops Working and Looks Fine +- **分类**: `engineering` | **发布日期**: 2025-02-20 +- **线上地址**: https://blog.hoelee.com/posts/two-ways-a-javascript-file-stops-working-and-looks-fine/ (EN) | https://blog.hoelee.com/posts/zh/two-ways-a-javascript-file-stops-working-and-looks-fine/ (ZH) +- **本篇审核状态**: ⏳ 待审核 (Pending) + +### ✍️ 顾问审阅意见区 (Review Feedback) +```text +【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 +【事实与数字核对】： +【人设与用词卫生】： +【CRO与转化建议】： +``` + +### 🇨🇳 中文版原文 (Chinese Markdown Twin) +````markdown +--- +title: "JavaScript 文件静默失效的两种方式：而页面看上去一切正常" +description: "一个被参数名遮住的函数名，加一个声明在首次调用之后的 let，让整段脚本停摆——而页面渲染得完美好看，因为屏幕上什么都没变。" +pubDate: 2025-02-20 +updatedDate: 2026-10-16 +category: engineering +tags: ["javascript", "i18n", "debugging", "frontend"] +ogImage: /og/zh-two-ways-a-javascript-file-stops-working-and-looks-fine.jpg +banner: /banners/two-ways-a-javascript-file-stops-working-and-looks-fine.jpg +draft: false +--- + +## 一次「什么都没发生」的改动 + +我给一个自己用的媒体转码队列看板加了语言切换：三份语言包、一个 `t(key, vars)` 函数、界面 chrome 上的 `data-i18n`、顶栏一个 `` in the top bar. + +After deploying it, the page looked exactly as it had before. Not broken — +*unchanged*. Same layout, same Chinese labels, no missing panels, no error +card, no console banner. The switcher was simply not there, and neither was +anything else the new code was supposed to do. + +That is the shape of bug I want to write down, because the page was not +"mostly working". **The script had stopped executing partway through — +twice, for two unrelated one-line reasons — and the page rendered perfectly +anyway.** + +## Why nothing looked broken + +Two design decisions I would make again, and both of them hid the failure. + +The first is that the default language lives in the markup. The Chinese text +is written inside the elements; `data-i18n` is only a hook that *overrides* +it when you switch. That is deliberate — the page needs no fetch to render, +and if a language pack goes missing you get Chinese instead of a row of +`top.theme_to_dark` keys. It also means a completely dead i18n layer +produces a page that is, visually, correct. + +The second is that the buttons are wired with inline handlers +(`onclick="rtFoo()"`), which after a script abort point at functions that +were never defined. Nothing warns you. The page only reveals the truth when +a human clicks something. + +So the symptom was "my change had no effect", which is the least informative +symptom a web page can give you. + +## Failure one: a parameter that swallowed a function + +The file has a theme block and an i18n block. The theme block had this +signature: + +```js +function applyTheme(t) { // t = the theme + label.textContent = t('top.theme_to_dark'); // TypeError: t is not a function +} +``` + +`applyTheme` was written before the i18n layer existed, and `t` was a +perfectly reasonable name for "theme". When the i18n layer arrived, `t` in +that scope stopped meaning the theme and started meaning *the function I +wanted to call* — except the parameter shadows it, so the call became +"invoke a string". + +A `TypeError` at the top level of a classic script does not just fail one +line. Everything after it never runs: the language-pack loader, the +`data-i18n` sweep, the switcher rendering. One wrong letter removed an +entire feature, silently. + +The fix is one character — rename the parameter to `th` — plus a comment +where the next person will look: + +```js +function applyTheme(th) { // NOT `t`: t() is the i18n function in this file +``` + +## Failure two: a declaration that arrived too late + +The second cause was in the same file, and it is a JavaScript rule people +know and still trip over. The theme block runs at load and calls `t('…')`. +The i18n block, further down the file, began with: + +```js +let LANG = localStorage.getItem('rt_lang') || 'zh-CN'; +``` + +`let` and `const` are hoisted but **not initialised**: between the start of +the scope and that line they live in the temporal dead zone, and any access +throws `Cannot access 'LANG' before initialization`. Because the theme block +sits *above* the i18n block, the failure was not a race — it was guaranteed +on every single load. + +Note the distinction from `var`, which would have quietly given me +`undefined` and a working page. The TDZ is the language protecting me from +exactly the bug I was about to write; the only reason it looked like +"nothing happened" is the markup fallback again. + +The fix is ordering, and ordering here is a *dependency*, not a style +choice: if a block calls `t()`, the i18n declarations have to be above it. +The file had been organised by topic ("theme, then i18n") because that is +how it grew. + +## Why my checks did not catch either one + +I had tests. They checked intent, not effect: + +- the three language packs exist and parse; +- every key used in the markup exists in every pack; +- no pack contains a dead key (a typo detector); +- the `{var}` placeholder sets in the three packs match. + +All true. All green. None of them executes the page, so none of them can see +a script that aborts on line 12 of 900. Static checks are structurally +unable to catch "the file is valid and the program is dead". + +## Assert on effects, in a browser + +What found both bugs was asking the live page questions that only a running +program can answer: + +```js +typeof t === 'function' // is the entry point still callable? +document.querySelectorAll('[data-i18n]').length // hooks present +rtCheckHandlers() // do the inline handlers still resolve? +``` + +That last one is worth more than it looks. The page carries **219 inline +`onclick` attributes naming 217 distinct functions**; the check enumerates +them and asserts each name is defined on `window`. It turns a hand-clicked +smoke test into a two-second automated one, and it is the check that turns +"the page looks right" into a measurable property. + +The second tool is cheaper and nastier: a **pseudo-language**. +`?lang=pseudo` takes every string in the pack, prefixes it with `[~~` and +stretches it 1.6×. Then assert two things — no horizontal overflow at +1440px, and no element in the nav still lacks the `[~~` prefix. The first +catches layout that only breaks in a longer language; the second catches +nodes that never got a translation hook at all. It found a real one in the +same run. + +Honest scope note: a pseudo-locale does not prove your Malay translation is +*good*. It proves the *plumbing* is complete — every string flows through +the mechanism, and the layout survives length. Judgement about the words +stays with a human. + +## What I would do differently + +1. **Load the page in a real browser and assert `typeof` on the entry points + before believing a client-side refactor worked.** A dead script is + invisible to every static check you own. +2. **Never let a parameter name shadow a function the same file calls.** One + letter bought me an entire afternoon; the comment is the part that stops + it recurring. +3. **Treat declaration order as a dependency, not aesthetics.** If a block + calls a function or reads a constant, that declaration goes above it — + especially in one long script. +4. **Generate the language packs from one table.** Then "the keys match" is + a test rather than a hope, and drift cannot be merged. +5. **When a change does nothing, suspect the script before the change.** + "Nothing happened" is the default failure mode of a rendered page, not + evidence that your code was inert. + +## The result + +Two one-line causes, both in the same file, both found by browser-side +assertions rather than by reading code again. The dashboard switches +language at runtime now, the default render is byte-for-byte what it was +before the change, and the three packs come from a single generated table — +so a missing key shows up as a red test instead of as a word that quietly +stays Chinese. + +The generalisation is the part I keep: **a crash tells you where to look. A +silent stop tells you nothing, and looks like success.** If a feature can +fail without moving a single pixel, then "the page still looks right" is not +a check. +```` + +--- + +## 文章 11 / 11：逐页验证一份 19 页的 PDF 报告 +- **Slug**: `verifying-a-pdf-report-page-by-page` +- **英文标题**: Verifying a 19-Page PDF Report Page by Page +- **分类**: `engineering` | **发布日期**: 2026-09-28 +- **线上地址**: https://blog.hoelee.com/posts/verifying-a-pdf-report-page-by-page/ (EN) | https://blog.hoelee.com/posts/zh/verifying-a-pdf-report-page-by-page/ (ZH) +- **本篇审核状态**: ⏳ 待审核 (Pending) + +### ✍️ 顾问审阅意见区 (Review Feedback) +```text +【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 +【事实与数字核对】： +【人设与用词卫生】： +【CRO与转化建议】： +``` + +### 🇨🇳 中文版原文 (Chinese Markdown Twin) +````markdown +--- +title: "逐页验证一份 19 页的 PDF 报告" +description: "如何逐页验证生成的 PDF 报告：用浏览器打印同一份 HTML 作基线，逐页比对几何尺寸，把每一处偏差都压到 2pt 以内。" +pubDate: 2026-09-28 +updatedDate: 2026-09-29 +category: engineering +blueprint: sme-digital-suite +blueprintLayer: L4-managed-ops-handover +tags: [php, codeigniter, pdf, css, print, testing] +ogImage: /og/zh-verifying-a-pdf-report-page-by-page.jpg +banner: /banners/verifying-a-pdf-report-page-by-page.jpg +draft: false +--- + +你怎么知道一份生成出来的 PDF 每一页都真的正确——不只是第一页？ +我维护着一个 CodeIgniter 4 应用：输入一个人的出生信息，生成一份 +中文命理/生命密码报告——十九张密排的 A4 sheet，封面、数字图表、 +方向九宫格、「一到九」的个人特征页。以前这份报告按网站原本的设计 +交付：操作员在浏览器里按 Ctrl+P，手工另存成 PDF。我把它换成了 +服务端渲染（mPDF），一夜之间，产品的质量取决于一个我看不见 +如何排版的引擎。 + +这就是这个问题可被搜索的版本：如何把一份多页 PDF 报告逐页地、 +用数字而不是肉眼，去和浏览器自己的打印输出对版？这个过程抓到 +了真实的 bug——包括一行被渲染成 2.5pt、从第二页起几乎看不见的 +页眉——最后全部十九页与浏览器基线的偏差都压进了 2pt。 + +## 为什么「看起来没问题」不算测试 + +没有人能用肉眼验证一份 19 页的报告。过去的流程是：打开 PDF， +扫一眼封面，交付。第 17 页标题漂移、页脚插图压住版权行、页眉 +小到读不出来——快速翻一遍永远发现不了，而花钱买报告的顾客 +看到的是全尺寸。 + +天真的做法会失败，是结构性的原因。服务端 PDF 渲染器不会像 +浏览器那样排你的 HTML：分页、页边距、基线、字距全都会漂移—— +而且不存在一个「看起来没问题」的测试，能让你在下周有人改了 +CSS 之后再跑一次。我第一次单趟渲染出来是 27 页，而设计预期 +是 19 页：mPDF 没有 CSS 裁剪，而 sheet 靠 `overflow: hidden` +藏住溢出。报告的 `@page` 规则更糟：mPDF 为这条规则本身 +翻了一页，19 张 sheet 炸成 12,789 页。页数都不稳定，所以 +「扫一眼第一页」不是验证——是碰运气。 + +## 基线：让浏览器去打印你的 HTML + +操作员的 Ctrl+P，就是 Chrome 用打印 CSS 打印报告自己的 HTML。 +所以参照标准不是「我觉得它应该长什么样」，而是浏览器自己的 +打印输出——在同一台机器、同一套网页字体下生成。做法：把报告 +目录用本地静态服务起起来（字体必须同源，否则 webfont 拒绝 +加载），再用 headless Chrome 打印。报告 CSS 声明了 +`@page { margin: 0 }`，所以浏览器输出是无边距满幅，再配合 +`-webkit-print-color-adjust: exact` 保住背景图——等价于操作员 +勾选「背景图形」： + +```bash +python -m http.server 8123 --bind 127.0.0.1 --directory + +"C:\Program Files\Google\Chrome\Application\chrome.exe" --headless=new --disable-gpu \ + --no-pdf-header-footer --user-data-dir=%TEMP%\chromeprofile --virtual-time-budget=30000 \ + --print-to-pdf=chrome-win.pdf "http://127.0.0.1:8123/report-local.html" +``` + +然后在 VM 上，用同一份 HTML 出服务端版本： + +```bash +php tools/pdf-render.php /tmp/report-v10.html /tmp/mpdf.pdf +``` + +## 逐页比对，而不是整体对比 + +以浏览器的 PDF 为基准，我在 Windows 这边用 +`uv run --with pymupdf` 逐页对比两份文档：页面尺寸、文字条数、 +图片包围盒（x0/y0/宽/高）、附图说明文字的 y 坐标。判定阈值： +**偏差 ≤2pt（0.7mm）算对齐**；超过 5pt 就要查根因。而且只比 +「同一元素在两份 PDF 里的差」——永远不要比绝对页数，因为页数 +一致是前提，不是测试本身。 + +## 逐页比对抓到了什么 + +下面每个 bug 都有可测量的前后数字，而且都修在库或模板里—— +不是用管道糊过去。 + +**看不见的页眉行。** 从第二页起每页顶部有一行小字，用户直接 +反馈说小到读不出来。页眉是一个 `font-size: 10px` 的 div， +里面套一张 auto 宽的 table，第一格写着 `width: 100%`。mPDF +把这解读成「表格超宽」，于是连同字号一起把整行缩到三分之一： +**2.5pt，而 Chrome 是 7.5pt**。修法（`fixPageHeaderTables()`）： +给表格显式宽度 + 显式字号（px→pt），并去掉第一格的 +`width: 100%`： + +```php +$pt = round((float) $m[2] * 0.75, 2); // 10px = 7.5pt +``` + +**匹配过宽的外边距规则。** `.sheet { margin: 5mm auto }` 是为 +屏幕预览准备的（sheet 之间的阴影缝隙）。mPDF 当真了，把 +296mm 的 sheet 推到 301mm——越过 297mm 的页面——于是 sheet +在页边被切开，auto-fit 把整页缩小 3%，带白边。最直观的修法、 +在样式表末尾追加一条 `.sheet { margin: 0; }`，完全没用：mPDF +对同名选择器的两条规则只认「先出现的赢」。所以库要原地改写 +这条规则（`stripSheetMargins()`），而匹配器必须小心什么才算 +「sheet 规则」。护栏是一个负向回顾断言： + +```php +'~(?` 标签 +居中，而 mPDF 8 的 `Center` 标签处理器是空类——标签整个被 +丢弃，所有居中的标题和表格全部左对齐（「前言」实测 x=30， +Chrome 是 x=280）。`expandCenterTags()` 把 `
` 改写成 +`
`，并给居中块里的表格补上 +`align="center"`——因为父级的 `text-align` 传不到表格。 +修复后：x=281，Chrome 280。 + +**行距比浏览器高 15%。** 模板的 normalize.css 声明了 +`html { line-height: 1.15 }`；mPDF 不继承它，退回自己的字体 +度量（1.33）。这累积成每页下半部 20–40pt 的漂移——sheet 3 +甚至溢出到第二页，触发整页缩小。把 `useFixedNormalLineHeight` +设成模板自己的值之后，正文行距 15.5pt，Chrome 是 15.7。 +表格还要显式加 `td, th { padding: 1px }`——mPDF 默认的 +单元格内边距比浏览器大 2px。 + +**字体是错的。** mPDF 读不了网页用的 `.woff`，文字回退到 +自带的 Sun-ExtA；而正文栈里的「微软雅黑」在 Linux 服务器上 +根本不存在。我注册了三套真 TTF——标题手写体 MaShanZheng、 +拉丁与数字 Roboto、正文 CJK 用 wqy-microhei——并把模板的 +字体栈映射过去。一个坑：必须关掉 mPDF 的自动按脚本选字， +否则它挑「第一个支持中文的已注册字体」——整页正文都变成 +手写体。 + +**还有一个与 mPDF 无关的 bug。** 第 7 页的主插图在**两份** +PDF 里都是破图——比对显示 Chrome 和 mPDF 里是同一个破损 +占位块。模板硬编码了 `https://cdn.hoelee.com/...`，这个域名 +已经不再解析（NXDOMAIN），所以每个引擎都抓不到图。把模板 +改回应用自己的 base URL，并让 `localiseAssets()` 把任何 +host 的 `/static/` 路径都映射到本地文件，两个引擎里的插图 +都恢复了。只有并排比对才能暴露这一类 bug——单独看哪个引擎 +都「正常」。 + +## 一张 sheet，一页 + +渲染器这么设计是有意的：库把 HTML 按 `
` +切开，每张 sheet 单独渲染成一份单页文档，合并时只取第 1 页—— +物理上保证「一张 sheet = 正好一页 A4」，永远不会跨页断裂。 +mPDF 量中文宽度和 Chrome 略有不同，个别 sheet 会高出几毫米。 +与其丢内容，库按一把缩放梯子——1.0, 1.005, 1.01, 1.02, 1.03, +1.06, 1.10, 1.15, 1.22——取第一个能落进一页的比例，缩小整张 +sheet 而不是裁内容。隐藏溢出是浏览器打印做的事 +（`overflow: hidden`）；一份收费产品如果 PDF 里静默丢了页内 +内容，就是无声的交付事故，所以设计选择是绝不丢内容。sheet 3 +现在需要 x1.005——0.5%，肉眼不可见——以前是 x1.03。 + +**「19/7 页」是什么意思。** 报告模板永远按固定顺序渲染十九张 +sheet；「版本」是这些 sheet 上的一个过滤器，不是第二份模板。 +完整版是 **19 页**；RM49 的精华版是**这 19 张里的 7 张**， +页码重编，每张入选的 sheet 都登记了一个文字标记——模板被 +改动/重排导致取错页时会大声失败，而不是把错误的章节交给 +顾客。两个版本走同一条管线，也用同一种方式验收： +`tools/pdf-verify.php --expect=19` 和 +`--expect=7` 双双 PASS +——页数加逐页 ink 检查（ghostscript 50dpi）证明没有任何 +空白页。 + +## 为什么收据是独立文档 + +收据不是报告裁剪出来的。它是自己的一份单页 A4 文档：真 +16mm 页边距（报告刻意做满幅无边距）、三语、单趟渲染—— +因为里面没有 sheet。它在发「已收款」邮件时才懒生成：付款 +回调必须毫秒级应答，0.3–1 秒的 PDF 渲染不属于回调。它落进 +同一个交付存储，文件名带 `-receipt` 后缀，永远不会和报告 +文件撞名，而且幂等：重试、重寄、顾客自己来拿，拿到的都是 +同一份。做它的过程从另一个方向印证了同一个论点——连单页 +文档都和浏览器不一样：`` 上的 `display: block` +不生效，左右并排的两张表把右列的数值裁出页面右边界， +合计行必须写在明细表内部，否则标签会浮在半空。 + +## 我会怎么做不一样 + +对版方法现在躺在项目笔记里，是一份文档化的流程，不是仓库 +里的脚本——这就是差距。仓库里自动化的验收工具只证明页数和 +逐页 ink，永远抓不到 2.5pt 的页眉或 15% 的行距漂移。我会把 +浏览器基线比对做成仓库验证工具链里的一个脚本：用同一份 +HTML 分别喂 Chrome 和库，diff 几何，任何超过 5pt 的偏差 +直接失败。这样，一次让打印布局悄悄回退的 CSS 改动会在构建 +时炸掉，而不是送到顾客手里。 + +我还会在写任何 mPDF 补偿代码之前就先出 Chrome 基线。 +「用操作员用的同一个引擎打印，然后测量」才是解锁点；之后 +每个修复都是机械活。还有两件诚实的遗留工作：伴侣合盘与 +家庭套餐还没跑过这套逐页对版，19 页的 PDF（背景图加嵌入 +字体约 20 MB）也还需要在交付前压缩。 + +## 结果 + +修复后的完整版验收： + +```text +$ php tools/pdf-verify.php /tmp/report-v10.html --expect=19 +out : 20,225,818 bytes, 19 pages, 10.4s, peak 188 MB +per-page ink check: all pages have content | report pages=19 +expected 19 pages => MATCH +RESULT: PASS +``` + +- **19 页 vs Chrome 基线的 19 页 — MATCH**，一张 sheet 一页， + 无跨页断裂。 +- 每一项实测偏差都**在 2pt（0.7mm）以内**：第 6 页偏了 + 213.5pt 的页脚插图、从 2.5pt 恢复到浏览器同款 7.5pt 的 + 页眉、归位的居中标题、15.5pt vs Chrome 15.7 的正文行距、 + 与网页字体一致的嵌入字体。 +- ink 检查确认**没有任何空白页**，PDF 文字可提取——封面能 + 读出真实文字，这对一份顾客要复制内容的报告很重要。 +- 精华版：**7 页 PASS**，4.99 MB，2.8 秒。 + +这就是「第一页看起来没问题」和「全部十九页都在浏览器两个点 +以内」的区别。前者是肉眼扫一遍就交付的结果；后者是拿浏览器 +自己当测试得到的结果。 + +--- + +我平时就做这类 Web 应用与打印/PDF 报告管线，也做网站设计与 +开发。如果你有一份服务器渲染的文档——报告、收据、发票—— +想在它送到顾客手里之前确认每一页都正确，跟我说说： +[WhatsApp](https://wa.me/60127972969) · [me@hoelee.com](mailto:me@hoelee.com?subject=PDF%20report%20pipeline) · +[hoelee.com](https://hoelee.com)。 +```` + +### 🇬🇧 英文版原文 (English Markdown Primary) +````markdown +--- +title: "Verifying a 19-Page PDF Report Page by Page" +description: "Verify a PDF report page by page: print the same HTML from your browser as a baseline, compare the per-page geometry, and keep every deviation under 2pt." +pubDate: 2026-09-28 +updatedDate: 2026-09-29 +category: engineering +blueprint: sme-digital-suite +blueprintLayer: L4-managed-ops-handover +tags: [php, codeigniter, pdf, css, print, testing] +ogImage: /og/verifying-a-pdf-report-page-by-page.jpg +banner: /banners/verifying-a-pdf-report-page-by-page.jpg +draft: false +--- + +How do you know a generated PDF actually looks right on every page — not just +page one? I maintain a CodeIgniter 4 app that turns birth details into a +Chinese numerology report. Until recently the operator printed it with Ctrl+P +and saved a PDF by hand. I replaced it with a server-side renderer (mPDF), and +overnight the product's quality depended on a layout engine I could not see. + +This post is the searchable version of that problem: verifying a multi-page +PDF report page by page against the browser's own print output, with numbers +instead of eyeballs. It caught real bugs — a header line rendered at 2.5pt, +effectively invisible on every page after the first — and ended with all +nineteen pages within 2pt of the browser. + +## Why "looks fine" is not a test + +Nobody can verify a 19-page report by eye. The old ritual was: open the PDF, +skim the cover, ship it. A drifting heading on page 17, a footer illustration +over the copyright line, an unreadable header — a quick flick never catches +those, and the customer who paid sees them at full size. + +Naive approaches fail structurally. A server-side PDF renderer does not lay +out HTML the way a browser does, so page breaks, margins, baselines and +spacing all drift — and there is no "looks fine" test you can re-run after +someone edits the CSS. My first single-pass render came out as 27 pages where +the design expects 19 — mPDF has no CSS clipping and the sheets hide overflow +with `overflow: hidden`. The report's own `@page` rule was worse: mPDF emitted +a page break for the rule itself, and 19 sheets exploded into 12,789 pages. +Page counts were not even stable, so eyeballing page one was not verification; +it was hope. + +## The baseline: make the browser print your HTML + +The operator's Ctrl+P is Chrome printing the report's own HTML with its print +CSS — same machine, same web fonts. So the reference is not "what I think it +should look like"; it is the browser's own print output. I served the report +directory locally (fonts must be same-origin or webfonts refuse to load) and +printed with headless Chrome. The report CSS declares `@page { margin: 0 }`, +so the output is borderless full-bleed, and `-webkit-print-color-adjust: +exact` keeps the background graphics the operator ticks: + +```bash +python -m http.server 8123 --bind 127.0.0.1 --directory + +"C:\Program Files\Google\Chrome\Application\chrome.exe" --headless=new --disable-gpu \ + --no-pdf-header-footer --user-data-dir=%TEMP%\chromeprofile --virtual-time-budget=30000 \ + --print-to-pdf=chrome-win.pdf "http://127.0.0.1:8123/report-local.html" +``` + +Then the server-side output, from the same HTML, on the VM: + +```bash +php tools/pdf-render.php /tmp/report-v10.html /tmp/mpdf.pdf +``` + +## Compare page by page, not overall + +I compared the two PDFs page by page with `uv run --with pymupdf`: page size, +text-block count, image bounding boxes (x0/y0/width/height), and caption text +y. Threshold: **within 2pt (0.7mm) counts as aligned**; over 5pt gets a root +cause. Compare the same element across the two PDFs — never absolute page +counts; page parity is the prerequisite, not the test. + +## What the comparison caught + +Each bug below had a measurable before/after and a fix in the library or +template. + +**The invisible header line.** A user reported the small header line, present +on every page after the cover, was unreadable. It is a `font-size: 10px` div +wrapping an auto-width table whose first cell declares `width: 100%` — mPDF +read that as "table too wide" and scaled the whole line, font included, to a +third of its size: **2.5pt vs Chrome's 7.5pt**. `fixPageHeaderTables()` gives +the table an explicit width and font size in points (px → pt) and drops the +offending cell: + +```php +$pt = round((float) $m[2] * 0.75, 2); // 10px = 7.5pt +``` + +**The margin rule that matched too much.** `.sheet { margin: 5mm auto }` +spaces the sheets on screen. mPDF took it literally and pushed the 296mm sheet +to 301mm — past the 297mm page — so the sheet was cut at the page edge and +auto-fit shrank the whole page by 3%, white border included. Appending `.sheet +{ margin: 0; }` does nothing — mPDF honours the *first* rule for a duplicated +selector. So the library rewrites the rule in place (`stripSheetMargins()`), +and the matcher must not over-match — the guard is a negative lookbehind: + +```php +'~(?` tag; mPDF 8's `Center` handler is an empty class, so the tag is +dropped and every centred heading and table came out left-aligned (the preface +heading measured x=30 against Chrome's x=280). `expandCenterTags()` rewrites +`
` to `
` and adds `align="center"` to +tables inside centred blocks, because parent `text-align` does not reach +tables. After the fix: x=281, Chrome 280. + +**Line spacing 15% taller than the browser's.** The template's normalize.css +declares `html { line-height: 1.15 }`; mPDF does not inherit it and fell back +to its own font metrics (1.33) — 20–40pt of drift accumulated on the lower +half of every page, and sheet 3 overflowed onto a second page, triggering a +whole-page shrink. Setting `useFixedNormalLineHeight` to the template's own +value brought body leading to 15.5pt against Chrome's 15.7. Tables needed `td, +th { padding: 1px }` — mPDF's default cell padding is 2px larger than a +browser's. + +**The fonts were wrong.** mPDF cannot read the `.woff` files the web views +use, so text fell back to its built-in Sun-ExtA, and "Microsoft YaHei" does +not exist on the Linux server. I registered the real TTFs — MaShanZheng for +headings, Roboto for Latin, wqy-microhei for CJK — and mapped the template's +stacks onto them. One trap: mPDF's automatic script-to-font selection must be +off, or it picks the first registered font that supports Chinese — the entire +body came out in handwriting. + +**And one bug that had nothing to do with mPDF.** Page 7's main illustration +was broken in *both* PDFs — the same broken stub in Chrome and mPDF. The +template hardcoded `https://cdn.hoelee.com/...`, a domain that no longer +resolves (NXDOMAIN), so every engine fetched nothing. Fixing the template to +use the app's own base URL and mapping any host's `/static/` path to local +files restored it in both engines. Only a side-by-side comparison surfaces +this — each engine alone looks "fine". + +## One sheet, one page + +The library splits the HTML into its `
` blocks and +renders each sheet as its own one-page document, importing page 1 and merging +— a physical guarantee that one sheet is exactly one A4 page, never split +across pages. mPDF measures CJK text widths slightly differently from Chrome, +so some sheets come out a few millimetres too tall. Rather than lose content, +the library re-renders the sheet at the smallest scale factor from a ladder — +1.0, 1.005, 1.01, 1.02, 1.03, 1.06, 1.10, 1.15, 1.22 — that fits, shrinking +the whole sheet instead of clipping. The browser's print clips with `overflow: +hidden`; a PDF that silently dropped in-sheet content would be a delivery +accident, so the design never drops content. Sheet 3 needs x1.005 today (0.5%, +invisible) where it used to need x1.03. + +**What "19/7 pages" means.** The report template always renders nineteen +sheets in a fixed order; the "edition" is a filter over those sheets, not a +second template. The full report is **19 pages**; the RM49 essence edition is +**7 of those 19 sheets**, renumbered, each selected sheet carrying a marker so +a rearranged template fails loudly instead of shipping the wrong chapters. +Both editions run the same pipeline and verify the same way: +`tools/pdf-verify.php --expect=19` and `--expect=7` both PASS — page count +plus a per-page ink check (ghostscript at 50dpi) proving no blank pages. + +## Why the invoice is a separate document + +The receipt is not a cut-down report. It is its own one-page A4 document: real +16mm page margins (the report is deliberately full-bleed), three languages, +and a single-pass render because there are no sheets. It is generated lazily +at email time — the payment callback must answer in milliseconds, and a 0.3–1s +render does not belong in it. It lands in the same delivery store under a +`-receipt` filename, never colliding with the report file, and it is +idempotent: retries get the same file. Even one page differed from the +browser: `display: block` on `` was ignored, side-by-side tables +clipped the right column's values off the page edge, and the total row had to +live inside the items table or its label floated in mid-air. + +## What I'd do differently + +The method lives in the project notes as a documented ritual, not a committed +script — that is the gap. The repo's automated acceptance tool proves page +count and per-page ink, which would never catch a 2.5pt header or a 15% +leading drift. I would turn the browser-baseline comparison into a script in +the repo's verification tooling: render the same HTML in Chrome and in the +library, diff the geometry, fail on any deviation over 5pt. Then a CSS change +that silently regresses the print layout fails the build instead of reaching a +customer. + +I would also have generated the Chrome baseline before writing any mPDF +compensation code — "print with the same engine the operator uses, then +measure" was the unlock. Two honest items remain: the partner and family +reports have not had this page-by-page pass yet, and the 19-page PDF (20 MB +with backgrounds and embedded fonts) still needs compression before delivery. + +## The result + +Full report verification, after the fixes: + +```text +$ php tools/pdf-verify.php /tmp/report-v10.html --expect=19 +out : 20,225,818 bytes, 19 pages, 10.4s, peak 188 MB +per-page ink check: all pages have content | report pages=19 +expected 19 pages => MATCH +RESULT: PASS +``` + +- **19 pages vs the Chrome baseline's 19 — MATCH**, one A4 page per sheet, + no splits. +- Every measured deviation is **within 2pt (0.7mm)** of the browser's print + output: footer art 213.5pt off on page 6 aligned, header restored to the + browser's 7.5pt, centred headings back, leading 15.5pt vs 15.7, embedded + fonts matching the web fonts. +- The ink check confirms **no blank pages**, and the PDF text is extractable — + the cover reads back as real text, which matters for a report customers + copy from. +- Essence edition: **7 pages PASS**, 4.99 MB, 2.8 seconds. + +That is the difference between "the first page looks fine" and "all nineteen +pages within two points of the browser": the first ships when you eyeball a +PDF; the second is what you get when the browser itself is the test. + +--- + +I build web applications and print/PDF report pipelines like this one, and I +do website design and development. If you have a document your server renders +— a report, a receipt, an invoice — and you want to be sure it is right on +every page before it reaches a customer, tell me about it: +[WhatsApp](https://wa.me/60127972969) · +[me@hoelee.com](mailto:me@hoelee.com?subject=PDF%20report%20pipeline) · +[hoelee.com](https://hoelee.com). ```` --- diff --git a/docs/consultant-review/batches/batch-03.md b/docs/consultant-review/batches/batch-03.md index 0af7d74..a10666e 100644 --- a/docs/consultant-review/batches/batch-03.md +++ b/docs/consultant-review/batches/batch-03.md @@ -1,10 +1,10 @@ # 博客文章顾问审阅资料包 — Batch-03 > **批次编号**：`Batch-03` -> **包含文章数**：11 篇 -> **本批次总字符量**：167,652 字（严格限制 < 200,000 字） +> **包含文章数**：1 篇 +> **本批次总字符量**：128,681 字（严格限制 < 200,000 字） > **模式**：BILINGUAL -> **生成时间**：2026-10-27 04:23:31 +> **生成时间**：2026-10-27 05:23:38 # 外部顾问审阅前置须知与工程背景规范 @@ -39,25 +39,15 @@ ## 本批次文章目录索引 -1. [devops] **为什么你的 CDP 点击会静默失效：平滑滚动、最小化窗口与 JS 派发事件** (`why-your-cdp-clicks-silently-fail`) — ~12,514 字符 -2. [devops] **Prometheus 明明有数据，Grafana 面板却全是 No data 的原因** (`why-your-grafana-dashboard-shows-no-data`) — ~11,212 字符 -3. [devops] **你的“磁盘将满”告警在说谎：别再按百分比告警** (`your-disk-full-alert-is-lying`) — ~8,912 字符 -4. [engineering] **浏览器翻译没有 API:我给一个纯中文网页应用加上了英文模式** (`adding-english-mode-to-a-chinese-only-web-app`) — ~18,070 字符 -5. [engineering] **排查 Remotion 视频抽搐：为什么在背景图挂载音频低频 Scale 会毁了转场** (`debugging-remotion-video-jitter`) — ~15,111 字符 -6. [engineering] **把字体嵌进 PDF，让它在任何机器上都长一样** (`embedding-fonts-so-a-pdf-looks-the-same-everywhere`) — ~11,465 字符 -7. [engineering] **那个把静默丢失更新变成 409 的 JPA 字段：@Version 乐观锁** (`jpa-version-field-lost-update`) — ~19,575 字符 -8. [engineering] **把 CodeIgniter 4 应用从 IIS 迁移到 OpenLiteSpeed** (`migrating-codeigniter-iis-to-openlitespeed`) — ~23,361 字符 -9. [engineering] **用「热浏览器会话」爬取有反爬墙的电商网站** (`scraping-bot-walled-marketplace-warm-browser-session`) — ~16,928 字符 -10. [engineering] **JavaScript 文件静默失效的两种方式：而页面看上去一切正常** (`two-ways-a-javascript-file-stops-working-and-looks-fine`) — ~11,294 字符 -11. [engineering] **逐页验证一份 19 页的 PDF 报告** (`verifying-a-pdf-report-page-by-page`) — ~19,210 字符 +1. [tutorials] **用 YuE2 BF16 生成完整长歌：突破时长限制与音频硬截断** (`generating-full-length-songs-with-yue2-comfyui-zh`) — ~128,681 字符 --- -## 文章 01 / 11：为什么你的 CDP 点击会静默失效：平滑滚动、最小化窗口与 JS 派发事件 -- **Slug**: `why-your-cdp-clicks-silently-fail` -- **英文标题**: Why Your CDP Clicks Silently Fail: Smooth Scroll, Minimised Windows, and JS-Dispatched Events -- **分类**: `devops` | **发布日期**: 2025-03-19 -- **线上地址**: https://blog.hoelee.com/posts/why-your-cdp-clicks-silently-fail/ (EN) | https://blog.hoelee.com/posts/zh/why-your-cdp-clicks-silently-fail/ (ZH) +## 文章 01 / 01：用 YuE2 BF16 生成完整长歌：突破时长限制与音频硬截断 +- **Slug**: `generating-full-length-songs-with-yue2-comfyui` +- **英文标题**: Generating Full-Length Songs with YuE2 BF16: Overcoming Duration Limits and Audio Truncation +- **分类**: `tutorials` | **发布日期**: 2026-10-26 +- **线上地址**: https://blog.hoelee.com/posts/generating-full-length-songs-with-yue2-comfyui/ (EN) | https://blog.hoelee.com/posts/zh/generating-full-length-songs-with-yue2-comfyui/ (ZH) - **本篇审核状态**: ⏳ 待审核 (Pending) ### ✍️ 顾问审阅意见区 (Review Feedback) @@ -71,4070 +61,3268 @@ ### 🇨🇳 中文版原文 (Chinese Markdown Twin) ````markdown --- -title: "为什么你的 CDP 点击会静默失效：平滑滚动、最小化窗口与 JS 派发事件" -description: "Chrome DevTools Protocol 点击报告成功却毫无反应的三个原因：平滑滚动导致的过期坐标、最小化窗口被丢弃的坐标输入、以及页面收到了点击但拒绝了它。" -pubDate: 2025-03-19 -updatedDate: 2026-09-29 -category: devops -tags: [cdp, chrome, react, browser-automation, debugging, python] -ogImage: /og/zh-why-your-cdp-clicks-silently-fail.jpg -banner: /banners/why-your-cdp-clicks-silently-fail.jpg -draft: false ---- - -如果你用 Chrome DevTools Protocol 驱动网页应用，你一定见过这个失败：脚本报告 `clicked @591,361`，元素明明在，点击也确实发出去了 —— 然后什么都没发生。没有异常，没有控制台报错，页面毫无变化。 - -我在自动化某平台卖家后台时整整耗了一个晚上在这上面。找到的三个原因值得写下来，因为**它们对「报告点击成功」的那段代码全都是隐形的**。 - -## 原因一：坐标过期，因为页面是平滑滚动的 - -最直观的点击辅助函数大概长这样： - -```python -r = self.ev("""(()=>{const e=%s; e.scrollIntoView({block:'center'}); - const b=e.getBoundingClientRect(); - return JSON.stringify({x:b.left+b.width/2, y:b.top+b.height/2});})()""") -self.click_xy(json.loads(r)["x"], json.loads(r)["y"]) -``` - -滚动到可视区、读取元素矩形、点它的中心。逻辑没错 —— 除非页面有 `scroll-behavior: smooth`（现代 UI 组件库里非常常见）。那时 `scrollIntoView()` 启动的是**动画**，而同一个 tick 里的 `getBoundingClientRect()` 返回的是元素**即将离开**的位置。你的点击落在元素**原来**的地方，落在当时在那儿的别的东西上，而日志忠实地告诉你：它点击了你要求的坐标。 - -修法是两行 —— 关掉平滑滚动，然后重新读矩形： - -```python -self.ev("""(()=>{document.documentElement.style.scrollBehavior='auto'; - if(document.body) document.body.style.scrollBehavior='auto'; return 'ok';})()""") -moved = self.ev("""(()=>{const e=%s; if(!e) return 'no'; - e.scrollIntoView({block:'center', behavior:'instant'}); return 'ok';})()""" % sel) -time.sleep(0.35) # 等布局稳定 -# 然后再读矩形（元素晚一帧才渲染的话，外面再套一层重试） -``` - -两个细节很关键：`scrollIntoView` 上的 `behavior:'instant'` 会覆盖 CSS；而读取前那一小段等待，才是让坐标可信的原因。没有重试循环的话，这个辅助函数在晚一帧渲染的元素上仍会偶发失败 —— **偶发失败是最糟的一类，因为它会让你怀疑自己的判断。** - -## 原因二：窗口最小化时，坐标点击被丢弃 - -修好之后，我的点击辅助函数在一次性探针里能用，一放进真正的脚本就再也没成功过。同样的选择器、同样的坐标、同样的页面。差别最后落在可见性上： - -``` -document.visibilityState → "hidden" -document.hasFocus() → true -``` - -当目标标签页不是可见窗口的活动标签页时，CDP 合成的坐标级鼠标事件是不可靠的 —— 浏览器可能接受、可能限流、也可能直接不往任何地方送。注意 `hasFocus()` **在文档隐藏时依然返回 `true`**，所以它不是有效的判断依据；要看的是 `document.hidden` / `visibilityState`。 - -两个修法，我都用： - -```python -def activate(self): # 把标签页提到窗口最前 - self.cdp("Page.bringToFront") - time.sleep(0.2) - -def click_js(self, sel): # 在页面内部派发事件来点击 - return self.ev("""(()=>{const e=%s; if(!e) return 'no'; - ['mousedown','mouseup','click'].forEach(k=> - e.dispatchEvent(new MouseEvent(k,{bubbles:true,cancelable:true,view:window}))); - return 'js-clicked';})()""" % sel) -``` - -`Page.bringToFront` 解决了常见情况。但真正从不出错的是 JS 派发的点击：它不依赖窗口是否可见，而且 React 挂在根节点上的事件监听器会收到它，和收到真实点击的方式完全一样 —— 因为事件冒泡到的是同一个根。一个用坐标点击五次才能打开一次的日期选择器，用这种方式**每次都开**。 - -需要知道的取舍：派发事件会跳过 hit-testing，所以它也会「点到」用户根本够不到的元素（被遮住、滚出屏幕、透明度为零）。在这里这是优点 —— 如果你用它去掩盖布局 bug，那就是坑 —— 所以凡是可视化的东西我都用坐标点击，而遇到自定义控件时才伸手去拿 `click_js`。 - -## 原因三：页面收到了点击，然后拒绝了它 - -这个最浪费时间，因为它的症状和前两个一模一样：点击发出去了，什么都没变。 - -区分它们只需要注入一个监听器： - -```js -window.__ev = []; -['mousedown','mouseup','click'].forEach(k => - document.addEventListener(k, e => window.__ev.push(k + ':' + e.target.tagName), true)); -``` - -- 有事件记录 → 输入链路是通的，问题在应用侧（原因三），或者在你读取结果的方式上（见下）。 -- 一条都没有 → 是原因一或原因二。 - -挂上监听器后我点了一次 `Confirm`，缓冲区满了 —— 说明点击没问题，是应用在说「不」。理由在一段我从没想过要看的文字里： - -> “Please create a new shop welcome voucher after the existing one is expired.” - -表单拒绝创建同类型的第二张券。没有字段级红字、头 500 毫秒里没有提示条、URL 也没变。**拒绝是真的，点击是无辜的。** - -这一类 bug 还有两个工具： - -```js -document.elementFromPoint(x, y) // 那个坐标上到底是什么？ -document.querySelectorAll('button').length // 你的选择器到底命中了几个？ -``` - -第二个抓到过一处很细的问题：门户页面在 DOM 里留了一个**隐藏的** `Confirm` 按钮（来自收起状态的日期面板），于是 `.pop()` 返回的是那个看不见的副本，点击自然落到了无人的地方。按可见性和尺寸一起过滤： - -```python -"[...document.querySelectorAll('button')].filter(b=>/^Confirm$/.test(b.innerText.trim())" -" && b.offsetParent!==null && b.getBoundingClientRect().width>0).pop()" -``` - -## 第四种「失败」，只有回读才发现 - -还有一类「失败」其实不是失败：**写入成功了，而你的读取是旧的。** 我因为列表页没有立刻显示，就断定创建被静默拒绝，又点了一次；后来才发现那张券一直都存在 —— 列表只是延迟了大约十分钟。这个教训可以推广：一次写入之后，唯一可信的结论是在**等待足够时间之后**回读得到的，而「看不到变化」意味着**未知**，不是失败。 - -## 检查清单 - -1. 关掉平滑滚动；用 `behavior:'instant'` 滚动；等一会儿再读矩形。 -2. 点击前调用 `Page.bringToFront`；判断依据用 `document.hidden`，不是 `hasFocus()`。 -3. 自定义控件用 JS 派发 `mousedown`/`mouseup`/`click`，别点坐标。 -4. 先挂事件监听器，证明输入是否到达，再去怪应用。 -5. 数一数选择器命中几个 —— 组件库里隐藏的重复元素很常见。 -6. 写入之后先等待再回读；永远不要重复点击一个还没验证过的提交按钮。 - -这些都不算冷门知识，但它们中的每一条，在一行 `clicked @591,361` 面前都是隐形的。如果你正在对着一个自己无法控制的后台做自动化：点击是整条链路里最不可靠的一环 —— 也是最容易被最后才怀疑的一环。 - -我平时就在做这类浏览器自动化与排障 —— 自托管服务、Docker 与 Traefik 栈，偶尔还有被 bot 墙挡住的市场页面。如果你有条工作流总因为「UI 有自己的想法」而断掉，跟我说说：[WhatsApp](https://wa.me/60127972969) · [me@hoelee.com](mailto:me@hoelee.com?subject=Browser%20automation%20debugging) · [hoelee.com](https://hoelee.com)。 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Why Your CDP Clicks Silently Fail: Smooth Scroll, Minimised Windows, and JS-Dispatched Events" -description: "Three reasons a Chrome DevTools Protocol click reports success and does nothing: a stale rect from smooth scrolling, a minimised window that drops coordinate input, and a page that received the click but rejected it." -pubDate: 2025-03-19 -updatedDate: 2026-09-29 -category: devops -tags: [cdp, chrome, react, browser-automation, debugging, python] -ogImage: /og/why-your-cdp-clicks-silently-fail.jpg -banner: /banners/why-your-cdp-clicks-silently-fail.jpg -draft: false ---- - -If you drive a web app with the Chrome DevTools Protocol, you have met this failure: your script reports `clicked @591,361`, the element is there, the click landed — and nothing happens. No exception, no console error, no change on the page. - -I spent an evening on exactly this while automating a marketplace's seller admin panel, and the three causes I found are worth writing down, because they are all invisible to the code that reports the click. - -## Cause 1: a stale rect, because the page scrolls smoothly - -The obvious click helper looks like this: - -```python -r = self.ev("""(()=>{const e=%s; e.scrollIntoView({block:'center'}); - const b=e.getBoundingClientRect(); - return JSON.stringify({x:b.left+b.width/2, y:b.top+b.height/2});})()""") -self.click_xy(json.loads(r)["x"], json.loads(r)["y"]) -``` - -Scroll into view, read the element's rectangle, click its centre. Correct — unless the page has `scroll-behavior: smooth` (very common in modern UI kits). Then `scrollIntoView()` starts an **animation**, and `getBoundingClientRect()` in the same tick returns the position the element is *about to leave*. Your click goes to where the element was, lands on whatever is there, and the log tells you it clicked the coordinates you asked for. - -The fix is two lines — turn smooth scrolling off, then re-read the rect: - -```python -self.ev("""(()=>{document.documentElement.style.scrollBehavior='auto'; - if(document.body) document.body.style.scrollBehavior='auto'; return 'ok';})()""") -moved = self.ev("""(()=>{const e=%s; if(!e) return 'no'; - e.scrollIntoView({block:'center', behavior:'instant'}); return 'ok';})()""" % sel) -time.sleep(0.35) # let layout settle -# then read the rect (and retry a couple of times if the element is not there yet) -``` - -Two details matter. `behavior:'instant'` on `scrollIntoView` overrides the CSS; and the small wait before reading is what makes the rect trustworthy. Without the retry loop the helper still fails intermittently on elements that render a frame late — intermittent failures are the worst kind, because they make you doubt your own diagnosis. - -## Cause 2: the window is minimised, and coordinate clicks are dropped - -After the fix, my click helper worked in a one-off probe and then never again inside the real script. Same selector, same coordinates, same page. The difference turned out to be visibility: - -``` -document.visibilityState → "hidden" -document.hasFocus() → true -``` - -CDP-synthesised mouse events at coordinates are unreliable when the target tab is not the active tab of a visible window — the browser may accept them, throttle them, or route them nowhere. Note that `hasFocus()` returns `true` even when the document is hidden, so it is not a useful check; `document.hidden`/`visibilityState` is. - -Two fixes, and I use both: - -```python -def activate(self): # bring the tab to the front of its window - self.cdp("Page.bringToFront") - time.sleep(0.2) - -def click_js(self, sel): # click by dispatching events from inside the page - return self.ev("""(()=>{const e=%s; if(!e) return 'no'; - ['mousedown','mouseup','click'].forEach(k=> - e.dispatchEvent(new MouseEvent(k,{bubbles:true,cancelable:true,view:window}))); - return 'js-clicked';})()""" % sel) -``` - -`Page.bringToFront` fixes the common case. But the JS-dispatched click is the one that never fails: it does not depend on the window being visible, and React's root-level event listener receives it exactly as it receives a real click, because the events bubble to the same root. A date-picker input that opened maybe one time in five with coordinate clicks opens **every** time this way. - -The trade-off to know: dispatching events skips hit-testing, so it will also “click” an element a user could not reach (covered, scrolled off, zero-opacity). That is a feature here — and a trap if you use it to paper over a layout bug — so I keep coordinate clicking for anything visual and reach for `click_js` when the widget is a custom control. - -## Cause 3: the page received the click and rejected it - -This is the one that wasted the most time, because the symptom is identical to causes 1 and 2: click fires, nothing changes. - -Distinguishing them takes one injected listener: - -```js -window.__ev = []; -['mousedown','mouseup','click'].forEach(k => - document.addEventListener(k, e => window.__ev.push(k + ':' + e.target.tagName), true)); -``` - -- Events recorded → the input path works. The problem is in the app (cause 3), or in how you read the result (see below). -- Nothing recorded → causes 1 or 2. - -With the listener armed I clicked `Confirm` and the buffer filled up — so the click was fine, and the app was saying no. The reason was in text I had not thought to look at: - -> “Please create a new shop welcome voucher after the existing one is expired.” - -The form refuses a second voucher of that type. No field-level red text, no toast in the first 500 ms, the URL unchanged. **The rejection was real and the click was innocent.** - -Two more tools for this class of bug: - -```js -document.elementFromPoint(x, y) // what is actually on top at that coordinate? -document.querySelectorAll('button').length // how many match your selector? -``` - -The second one caught a subtle one: the portal leaves a **hidden** `Confirm` button in the DOM from the collapsed date-panel, so `.pop()` returned the invisible duplicate and the click went nowhere visible. Filter by visibility *and* size: - -```python -"[...document.querySelectorAll('button')].filter(b=>/^Confirm$/.test(b.innerText.trim())" -" && b.offsetParent!==null && b.getBoundingClientRect().width>0).pop()" -``` - -## And the fourth failure I only found by re-reading - -There is a class of “failure” that is not one: **the write succeeded and your read is stale.** I concluded a create had been silently refused because the list page did not show it immediately, clicked again, and later discovered the voucher had existed the whole time — the list just lagged about ten minutes. The lesson generalises: after a write, the only trustworthy verdict is a read-back *after* a real waiting period, and “no visible change” means *unknown*, not *failed*. - -## The checklist - -1. Turn off smooth scrolling; scroll with `behavior:'instant'`; wait, then read the rect. -2. Call `Page.bringToFront` before clicking; treat `document.hidden` as the signal, not `hasFocus()`. -3. For custom widgets, dispatch `mousedown`/`mouseup`/`click` from JS rather than clicking coordinates. -4. Arm an event listener to prove whether input arrived before you blame the app. -5. Count your selectors — hidden duplicates are common in component libraries. -6. After a write, wait and re-read; never re-click a submit you have not verified. - -None of this is exotic, but every item is invisible in a log line that says `clicked @591,361`. If you are building automation against an admin panel you do not control, the click is the least reliable part of the pipeline — and the one most likely to be blamed last. - -I build and debug this kind of browser automation for a living — self-hosted services, Docker and Traefik stacks, and the odd scraped-but-bot-walled marketplace. If you have a workflow that keeps breaking because a UI has other ideas, tell me about it: [WhatsApp](https://wa.me/60127972969) · [me@hoelee.com](mailto:me@hoelee.com?subject=Browser%20automation%20debugging) · [hoelee.com](https://hoelee.com). -```` - ---- - -## 文章 02 / 11：Prometheus 明明有数据，Grafana 面板却全是 No data 的原因 -- **Slug**: `why-your-grafana-dashboard-shows-no-data` -- **英文标题**: Why Your Grafana Dashboard Shows No Data (When Prometheus Is Fine) -- **分类**: `devops` | **发布日期**: 2026-05-17 -- **线上地址**: https://blog.hoelee.com/posts/why-your-grafana-dashboard-shows-no-data/ (EN) | https://blog.hoelee.com/posts/zh/why-your-grafana-dashboard-shows-no-data/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "Prometheus 明明有数据，Grafana 面板却全是 No data 的原因" -description: "所有 target 都是 up、数据就在 Prometheus 里、面板表达式也没错——41 个面板却全部 No data。原因是 Grafana 变量过滤了自己，而我的验证方式恰好掩盖了它。" -pubDate: 2026-05-17 -updatedDate: 2026-09-29 -category: devops -blueprint: self-hosted-infrastructure -blueprintLayer: L3-observability-self-healing -tags: [grafana, prometheus, dashboards, monitoring, promql, observability] -ogImage: /og/zh-why-your-grafana-dashboard-shows-no-data.jpg -banner: /banners/why-your-grafana-dashboard-shows-no-data.jpg -draft: false ---- - -打开每天在看的面板，结果是空的。不是\"某一格坏了\"，而是**从上到下每一格**都写着 *No data*。去查 Prometheus，一切健康。这篇讲我踩到的那个具体且不直观的原因，以及为什么我自己的验证会同时告诉我\"面板没问题\"。 - -## 最省时间的排查顺序 - -常见原因（数据源选错、时间范围不对、`rate()` 用在会重置的计数器上、target 没被抓取）都不适用。真正有效的检查顺序是： - -```bash -# 1. target 真被抓取了吗？ -curl -s http://prometheus:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job, health}' - -# 2. 指标现在存在吗？（完全绕过 Grafana） -curl -s -G http://prometheus:9090/api/v1/query \ - --data-urlencode 'query=node_uname_info{job="unraid-host"}' | jq '.data.result[0].metric' - -# 3. 面板表达式本身对吗？ -curl -s -G http://prometheus:9090/api/v1/query \ - --data-urlencode 'query=count(node_cpu_seconds_total{mode="idle",job="unraid-host"})' -``` - -三项都过了：7 个 target `up`、2.5 万条序列在采集、手动跑面板表达式也有数据。所以问题不在指标、不在抓取、也不在 PromQL——而在**它们之间的变量层**。 - -## 真正的原因：变量过滤了自己 - -面板上有一个主机选择器，两个变量串联：`$nodename`（哪台机器）和 `$node`（它的 `instance` 标签）。第一个的定义是： - -```promql -# BROKEN — 变量用自身的值来过滤自己 -label_values(node_uname_info{job="unraid-host", nodename=~"$nodename"}, nodename) -``` - -首次加载时 `$nodename` 还没有值，选择器就变成 `nodename=~""`——**什么都匹配不到**。查询返回零行的变量没有任何选项，于是它保持空；而 `$node` 又是基于 `$nodename` 定义的： - -```promql -label_values(node_uname_info{job="unraid-host", nodename="$nodename"}, instance) -``` - -于是 `$node` 也是空，所有以 `$node` 为过滤条件的面板自然匹配不到序列。Grafana 说 *No data* 并没有骗人：面板查询真的没有数据，因为给它限定范围的变量解析成了空。 - -修法是去掉自引用，并给每个变量存一个明确的默认值： - -```promql -# FIXED — 不自引用，每个变量只跳一层 -# $nodename -label_values(node_uname_info{job="unraid-host"}, nodename) -# $node -label_values(node_uname_info{job="unraid-host", nodename="$nodename"}, instance) -``` - -## 为什么我的验证没抓到 - -这才是值得抄的部分。我\"验证\"面板的方式是：**手动**把每个面板表达式里的变量替换成我知道正确的值（`$nodename=unRaid`、`$node=host.docker.internal:9100`），然后断言查询有数据点。13 项检查全部通过——面板依然是空的，因为 bug 在被替换的那个值**上游**：Grafana 自己对变量的解析是空的。 - -**一个替换掉\"可疑值\"的验证，永远发现不了那个值解析失败。** 要抓它，得读 Grafana 真实存下来的值： - -```bash -curl -s -u admin:"$PW" http://grafana:4010/api/dashboards/uid/rYdddlPWk \ - | jq '.dashboard.templating.list[] | {name, current: .current.value}' -``` - -更直接的检查对象是变量查询本身——跑 Grafana 会跑的那条： - -```bash -# $nodename 的 label_values() 到底返回什么？ -curl -s -G http://prometheus:9090/api/v1/query \ - --data-urlencode 'query=count by (nodename) (node_uname_info{job="unraid-host"})' | jq '.data.result' -``` - -那里是空，面板就不可能渲染出来，无论数据多健康。 - -## 第二个坑：`$__all` 不是 `.*` - -做自动化面板检查时，值为\"All\"的变量，其 current 是哨兵字符串 `$__all`——不是正则。把 `name=~"$__all"` 原样展开会匹配不到任何东西，于是完全健康的容器面板\"全军覆没\"。展开前要把 `$__all` 解析成该变量的 `allValue`（通常是 `.*`），否则你追的是一个只存在于自己检查脚本里的 bug。 - -## 不是每个空面板都是 bug - -修完之后 25 个面板有 21 个出数。剩下 4 个是**设计上就不可能**有数据的，最好在面板说明里写清楚： - -- **PSI 面板**需要 `/proc/pressure`，而这台 NAS 的内核不暴露它——指标在这台机器上无法存在；同样的面板在会导出 `node_pressure_*` 的主机上正常显示。 -- **根文件系统面板**的表达式带 `fstype!="rootfs"`，而某台主机的 `/` 是**内存盘**（unRaid 就是），所以永远匹配不到。 - -区分\"因为 bug 空\"和\"因为这台机器不可能有这个指标而空\"，是真修好和瞎忙一下午的分界线。 - -## 我下次会怎么做 - -- **永远不要定义过滤自己的模板变量。** 一个变量只跳一层。 -- **给面板依赖的变量存明确的 `current` 值**，别让全新加载依赖下拉框被选中。 -- **用 Grafana 存下来的值做验证**，而不是我以为的值。 -- **故意留空的缺口写进面板说明**，否则未来的你会花一个下午去\"修\"一个本来就没打算工作的面板。 - -## 结果 - -三台主机现在跑同一套布局——每台一个 41 面板的主机面板 + 一个 10 面板的容器面板，分别有 21/23/25 个面板出数，两类结构性空缺写在面板里，而不是留给下一个打开它的人去猜。 - -## 需要为你的业务做这个吗？ - -如果你有一堆没人信任的 Grafana 面板，或者服务器和 NAS 完全没有监控，我可以帮你搭自托管的 Prometheus + Grafana（主机与容器指标、合理的告警规则、通知推到 Telegram 和邮件），也会把你那些悄悄\"不出数\"的面板修好。 - -**WhatsApp：[+60 12-797 2969](https://wa.me/60127972969)** · **邮箱：[me@hoelee.com](mailto:me@hoelee.com?subject=Grafana%20monitoring)** · **[hoelee.com](https://hoelee.com)** - -网站设计与开发是我的主业；服务器加固与自托管基础设施是它的另一半。 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Why Your Grafana Dashboard Shows No Data (When Prometheus Is Fine)" -description: "Every target up, data in Prometheus, panel expressions correct — and all 41 panels reading No data. The culprit was a Grafana variable filtering on itself, plus the verification habit that hid it." -pubDate: 2026-05-17 -updatedDate: 2026-09-29 -category: devops -blueprint: self-hosted-infrastructure -blueprintLayer: L3-observability-self-healing -tags: [grafana, prometheus, dashboards, monitoring, promql, observability] -ogImage: /og/why-your-grafana-dashboard-shows-no-data.jpg -banner: /banners/why-your-grafana-dashboard-shows-no-data.jpg -draft: false ---- - -You open the dashboard you use every day and it's empty. Not "one panel broke" empty — -**every panel**, top to bottom, says *No data*. You check Prometheus and everything is -healthy. This post is the specific, non-obvious cause I hit, and the way my own -verification convinced me the dashboard was fine while it wasn't. - -## The searchable question - -*My Grafana panels show No data but the data is in Prometheus — why?* - -The usual answers (wrong datasource, wrong time range, `rate()` on a counter that -resets, missing scrape target) did not apply. Here is the checklist that narrowed it -down, in the order that costs the least time: - -```bash -# 1. is the target actually being scraped? -curl -s http://prometheus:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job, health}' - -# 2. does the metric exist right now? (bypasses Grafana entirely) -curl -s -G http://prometheus:9090/api/v1/query \ - --data-urlencode 'query=node_uname_info{job="unraid-host"}' | jq '.data.result[0].metric' - -# 3. is the panel's expression sound? -curl -s -G http://prometheus:9090/api/v1/query \ - --data-urlencode 'query=count(node_cpu_seconds_total{mode="idle",job="unraid-host"})' -``` - -All three were green: 7 targets `up`, 25k series ingested, and the panel's raw -expression returned data when I ran it by hand. So the problem was not the metrics, -the scrape, or the PromQL — it was **the variable layer between them**. - -## The actual cause: a variable that filtered on itself - -The dashboard had a host picker. Two variables chained: `$nodename` (which machine) -and `$node` (its `instance` label). The definition of the first one was: - -```promql -# BROKEN — the variable filters on its own value -label_values(node_uname_info{job="unraid-host", nodename=~"$nodename"}, nodename) -``` - -On first load `$nodename` has no value, so the selector becomes -`nodename=~""` — which matches **nothing**. A variable whose query returns no rows has -no options, so it stays empty. `$node` is then defined against `$nodename`: - -```promql -label_values(node_uname_info{job="unraid-host", nodename="$nodename"}, instance) -``` - -…which is also empty, so every panel filtering on `$node` matches no series. Grafana -isn't lying when it says *No data*: the panel query genuinely has no data, because the -variable that scopes it resolved to nothing. - -The fix is to stop the self-reference and give each variable an explicit saved default: - -```promql -# FIXED — no self-reference, one hop per variable -# $nodename -label_values(node_uname_info{job="unraid-host"}, nodename) -# $node -label_values(node_uname_info{job="unraid-host", nodename="$nodename"}, instance) -``` - -Then save `current` values for both so the dashboard opens populated instead of -depending on a click. (This is also why the same 41-panel layout can be cloned to -three hosts and just work.) - -## Why my verification missed it - -This is the part worth stealing. I "verified" the dashboard by expanding each panel's -expression **by hand**, substituting the variable values I knew were correct -(`$nodename=unRaid`, `$node=host.docker.internal:9100`), and asserting the query -returned points. Thirteen of those checks passed. The dashboard was still blank, -because the bug was upstream of the substitution: Grafana's *own* resolution of the -variables was empty. - -**A verification that substitutes the very value under suspicion cannot detect that -value failing to resolve.** To catch it, read the values Grafana actually saved and -expand with those: - -```bash -curl -s -u admin:"$PW" http://grafana:4010/api/dashboards/uid/rYdddlPWk \ - | jq '.dashboard.templating.list[] | {name, current: .current.value}' -``` - -The correct check is on the variable query itself — run what Grafana runs: - -```bash -# what does $nodename's label_values() actually return? -curl -s -G http://prometheus:9090/api/v1/query \ - --data-urlencode 'query=count by (nodename) (node_uname_info{job="unraid-host"})' | jq '.data.result' -``` - -Empty there means the dashboard cannot possibly render, no matter how healthy the data is. - -## The second trap: `$__all` is not `.*` - -While automating panel checks, a variable set to *All* reports its current value as the -sentinel string `$__all` — not a regex. Expanding panel expressions naively turns -`name=~"$__all"` into a literal that matches nothing, so a perfectly healthy -container dashboard "fails" every panel. Resolve `$__all` to the variable's `allValue` -(usually `.*`) before expanding, or you'll chase a bug that only exists in your checker. - -## Not every empty panel is a bug - -After the fix, 21 of 25 panels had data. The remaining four were empty **by design** and -worth naming honestly in the panel description: - -- **PSI panels** (`Pressure`, `Pressure Stall Information`) need `/proc/pressure`, which - the NAS kernel doesn't expose. The metric cannot exist there — the same panels on a - host that *does* export `node_pressure_*` render fine. -- **Root filesystem panels** whose expression carries `fstype!="rootfs"` can never match - on a host whose `/` is a **ramdisk**, which is what unRaid's root is. - -Distinguishing "broken because of a bug" from "empty because the metric cannot exist on -this host" is the difference between a real fix and a wild goose chase. - -## What I'd do differently - -- **Never define a template variable that filters on itself.** One hop per variable. -- **Always save explicit `current` values** for variables a dashboard depends on, so a - fresh load doesn't rely on a populated dropdown. -- **Verify with Grafana's saved values, not my idea of them.** The masking is invisible - in the test result and obvious in the UI. -- **Describe intentional gaps in the panel description.** Future-you will otherwise - spend an afternoon "fixing" a panel that was never meant to work there. - -## The result - -All three hosts now run the same instrumented layout — a 41-panel host dashboard and a -10-panel container dashboard each — with 21/23/25 panels returning data respectively, -and the two structurally-empty classes documented in place instead of silently -confusing whoever opens them next. - -## Want this for your business? - -If you have Grafana dashboards that nobody trusts — or servers and a NAS with no -monitoring at all — I build self-hosted Prometheus + Grafana stacks (host and container -metrics, sensible alert rules, alerts to Telegram and email) and I'll fix the ones that -have quietly stopped showing data. - -**WhatsApp: [+60 12-797 2969](https://wa.me/60127972969)** · **Email: [me@hoelee.com](mailto:me@hoelee.com?subject=Grafana%20monitoring)** · **[hoelee.com](https://hoelee.com)** - -Website design and development is my main line of work; server hardening and -self-hosted infrastructure is the other half of it. -```` - ---- - -## 文章 03 / 11：你的“磁盘将满”告警在说谎：别再按百分比告警 -- **Slug**: `your-disk-full-alert-is-lying` -- **英文标题**: Your Disk-Full Alert Is Lying: Stop Alerting on Percentages -- **分类**: `devops` | **发布日期**: 2026-06-24 -- **线上地址**: https://blog.hoelee.com/posts/your-disk-full-alert-is-lying/ (EN) | https://blog.hoelee.com/posts/zh/your-disk-full-alert-is-lying/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "你的“磁盘将满”告警在说谎：别再按百分比告警" -description: "\"剩余 3%\"听着像紧急事故，直到你发现那是 500 GB。在多 TB 的卷上，百分比阈值会在没事的时候乱叫，也会在真出事的时候沉默。" -pubDate: 2026-06-24 -updatedDate: 2026-09-29 -category: devops -blueprint: self-hosted-infrastructure -blueprintLayer: L3-observability-self-healing -tags: [monitoring, alerting, prometheus, grafana, storage, capacity, observability] -ogImage: /og/zh-your-disk-full-alert-is-lying.jpg -banner: /banners/your-disk-full-alert-is-lying.jpg -draft: false ---- - -我搭好监控、写了一套看着合理的规则，然后用一个下午删掉了其中大半阈值。它们全是百分比，而且都以同一种方式错了。 - -## 三个教我道理的告警 - -**1.「文件系统使用率超过 90%」**——它在一个还剩约 500 GB 的 17 TB 卷上触发。没事发生，也不会马上有事发生。这种体量的卷，一次大备份就能让百分比摆动几个点：比例是噪声，而\"还有 500 GB 可用\"才是世界的真实状态。 - -**2.「内存使用率超过 90%」**——在一台**可用**内存 4.8 GB 的主机上反复触发。Linux 会用空闲内存做 page cache，需要时立刻归还；`MemTotal - MemFree` 描述的是文件缓存，不是你的风险。真正预示 OOM 的是 `MemAvailable`。 - -**3.「CPU steal 超过 25%」**——一台把邮件栈跑得完全正常的 VPS，steal 长期在 41%。steal 说明宿主机忙，它是**容量**信号而不是**故障**信号；7 vCPU 的机器 41% steal 就是这台机器的成本。这条规则产生的告警永远为真、也永远无用。 - -共同点：**我在对比例告警，而比例不知道东西有多大。** - -## 对\"后果\"告警，而不是对比例 - -我现在对每个阈值都问一句：*如果这个状态持续下去，实际会发生什么？* 答案几乎总能用绝对单位表达。 - -| 不要这样 | 改成 | 原因 | -|---|---|---| -| 文件系统 > 90% | 剩余空间 < N GB | \"我还能不能写进去\"才是问题 | -| 内存使用 > 90% | `MemAvailable` < 512 MB | 逼近 OOM，而不是\"缓存很暖和\" | -| CPU steal > 25% | steal > 50% | 容量成本 vs 真的被饿死 | -| 负载 > N | `load1 / 核数 > 2` | 不谈核数的负载没有意义 | - -PromQL 的前后对比： - -```promql -# BEFORE — 17 TB 卷的百分比；还剩半 TB 就开叫 -(1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 90 - -# AFTER — 真正装数据的卷上，还剩多少容量 -min by (instance, mountpoint) ( - node_filesystem_avail_bytes{mountpoint=~"/volume[0-9]+|/mnt/ssd|/mnt/disk[0-9]+"} -) < 25e9 -``` - -```promql -# BEFORE — 把 page cache 当成压力 -(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90 - -# AFTER — 真正的 OOM 前兆 -node_memory_MemAvailable_bytes < 512 * 1024 * 1024 -``` - -两个让规则能活下来的细节： - -**把比较放进阈值条件里，不要塞进表达式。** 查询就写 `min by (mountpoint) (node_filesystem_avail_bytes{...})`，让告警规则去和 `25000000000` 比。若你在 PromQL 里写死 `< 25e9`、规则里又留着继承来的 `> 90`，规则照样会评估——但界面显示的阈值是错的，下一个读它的人得从两个互相矛盾的条件下反推你的意图。 - -**排除那些\"只是另一个挂载点的视图\"。** 一条\"剩余空间\"规则有多诚实，取决于它的选择器。在我自己的环境里，同一个 NAS 卷出现三次（`/volume1`、`/opt`、以及在另一台主机上以 CIFS 再次挂载），而 unRaid 的 shfs 联合挂载报告 `avail=0`——不加排除的规则不是永远误报，就是把容量算成两倍。一个写明白的选择器胜过聪明的通用写法。 - -## 噪声不是无害的 - -坏阈值的代价不在告警本身，而在**训练**：每条\"没事乱叫\"的告警都在教你先扫一眼、然后忽略；等你真正该看的那条到来时，它出现在一个你早已不再读的频道里。我的规则总数**减少了**，覆盖率却上升了——更少，但每条都对应一个明确的后果。 - -## 阈值是私人的，形状是通用的 - -上面的数字是我的：NAS 卷剩不到 25 GB 我会在意、可用内存低于 512 MB 我才管、VPS steal 超过 50% 才算被饿。你的数字随硬件和容忍度而变。 - -能通用的是形状： - -- 用绝对单位而非比例； -- 把后果写进告警的摘要里； -- 选择器明确声明它在替哪些挂载点说话； -- 规则数量少到你还能读完每一条。 - -## 结果 - -同一套栈，从\"三条永久无用报警\"变成 16 条在机房健康时保持安静的规则——其中一条立刻暴露了一个只剩 21 GB 的 NAS 卷，而那正是百分比规则一直藏在\"99% 满的巨物\"背后、被你学会忽略的东西。 - -## 需要为你的业务做这个吗？ - -如果你的监控在发你已经学会忽略的告警，或者你根本没有监控、宁愿从一条规则而不是从一次失败的备份里得知磁盘满了，我可以帮你搭自托管的监控与告警（Prometheus + Grafana，通知推 Telegram 和邮件），阈值按\"什么会真的坏\"来定。 - -**WhatsApp：[+60 12-797 2969](https://wa.me/60127972969)** · **邮箱：[me@hoelee.com](mailto:me@hoelee.com?subject=Monitoring%20and%20alerting)** · **[hoelee.com](https://hoelee.com)** - -网站设计与开发是我的主业；服务器加固与自托管基础设施是它的另一半。 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Your Disk-Full Alert Is Lying: Stop Alerting on Percentages" -description: "3% free sounds like an emergency until you notice it is 500 GB. On multi-terabyte volumes, percentage thresholds fire when nothing is wrong — and miss when something is." -pubDate: 2026-06-24 -updatedDate: 2026-09-29 -category: devops -blueprint: self-hosted-infrastructure -blueprintLayer: L3-observability-self-healing -tags: [monitoring, alerting, prometheus, grafana, storage, capacity, observability] -ogImage: /og/your-disk-full-alert-is-lying.jpg -banner: /banners/your-disk-full-alert-is-lying.jpg -draft: false ---- - -I built a monitoring stack, wrote a reasonable set of rules, and then spent an -afternoon deleting most of my own thresholds. Every one of them was a percentage, and -every one of them was wrong in the same specific way. - -## The three alerts that taught me - -**1. "Filesystem above 90% used."** It fired on a 17 TB volume that had ~500 GB free. -Nothing was wrong. Nothing was *about* to be wrong. On a volume that size, a single -large backup can swing the percentage by several points — the ratio is noise, and the -fact that 500 GB is still available is the actual state of the world. - -**2. "Memory above 90% used."** Fired constantly on a host with 4.8 GB *available*. -Linux uses free RAM for page cache and gives it back on demand; `MemTotal - MemFree` -describes your filesystem cache, not your risk. The number that predicts an OOM is -`MemAvailable`. - -**3. "CPU steal above 25%."** Fired on a VPS that had been running its mail stack -perfectly at 41% steal. Steal means the hypervisor is busy — it is a *capacity* signal, -not a *failure* signal, and 41% steal on a 7-vCPU box was simply what that host costs. -The rule produced alerts that were always true and never actionable. - -The common thread: **I was alerting on a ratio, and the ratio doesn't know how big the -thing is.** - -## Alert on the consequence, not the ratio - -The test I now apply to every threshold: *if this condition persists, what will actually -happen?* That answer is almost always expressible in absolute units. - -| Instead of | Alert on | Because | -|---|---|---| -| Filesystem > 90% | Free space < N GB | "Can I still write?" is the question | -| Memory > 90% used | `MemAvailable` < 512 MB | about to OOM, not "cache is warm" | -| CPU steal > 25% | Steal > 50% | capacity cost vs. starvation | -| Load average > N | `load1 / cores > 2` | load is meaningless without core count | - -In PromQL, before and after: - -```promql -# BEFORE — percentage of a 17 TB pool; fires with half a terabyte still free -(1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 90 - -# AFTER — capacity left, on the volumes that hold data -min by (instance, mountpoint) ( - node_filesystem_avail_bytes{mountpoint=~"/volume[0-9]+|/mnt/ssd|/mnt/disk[0-9]+"} -) < 25e9 -``` - -```promql -# BEFORE — page cache counted as "pressure" -(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90 - -# AFTER — the actual OOM precursor -node_memory_MemAvailable_bytes < 512 * 1024 * 1024 -``` - -Two details that make these rules survive contact with a real dashboard: - -**Keep the comparison in the threshold condition, not inside the expression.** Query -`min by (mountpoint) (node_filesystem_avail_bytes{...})`, then let the alert rule -compare it to `25000000000`. If you bake `< 25e9` into the PromQL *and* leave an -inherited `> 90` in the rule, the rule still evaluates — but it displays a nonsense -threshold, and the next person to read it has to reconstruct your intent from two -contradictory conditions. - -**Exclude the mounts that are views of other mounts.** A "free space" rule is only as -honest as its selector. On my own setup, one NAS volume appears three times (as -`/volume1`, as `/opt`, and again on another host as a CIFS re-mount), and unRaid's shfs -union mount reports `avail=0` — a rule without exclusions would either fire forever or -double-count capacity. One explicit selector beats a clever generic one: - -```promql -node_filesystem_avail_bytes{mountpoint=~"/volume[0-9]+|/mnt/ssd|/mnt/disk[0-9]+"} -``` - -## Noise is not harmless - -The cost of a bad threshold isn't the alert itself — it's the *training*. Every alert -that fires while nothing is wrong teaches you to skim, then to ignore, and then the one -that matters arrives in a channel you've stopped reading. My rule count went **down** -while coverage went up: fewer, each tied to a named consequence. - -## Thresholds are personal, shapes are not - -The specific numbers above are mine — 25 GB free before a NAS volume bothers me, 512 MB -available before I care about memory, 50% steal before a VPS is being starved. Yours -will differ with your hardware and your tolerance. - -What transfers is the shape: - -- absolute units over ratios, -- the consequence written into the alert's summary, -- a selector that states exactly which mounts it speaks for, -- and a rule count small enough that you still read every message. - -## The result - -The same stack that was producing three permanent, uninformative warnings now runs 16 -rules that stay silent when the lab is healthy — and one of them immediately surfaced a -NAS volume sitting at 21 GB free, which is the kind of thing a percentage rule had been -hiding behind a 99%-full giant that I'd learned to ignore. - -## Want this for your business? - -If your monitoring sends you alerts you've learned to ignore, or you have none and would -rather find out about a full disk from a rule than from a failed backup, I set up -self-hosted monitoring and alerting (Prometheus + Grafana, alerts to Telegram and email) -with thresholds tied to what actually breaks. - -**WhatsApp: [+60 12-797 2969](https://wa.me/60127972969)** · **Email: [me@hoelee.com](mailto:me@hoelee.com?subject=Monitoring%20and%20alerting)** · **[hoelee.com](https://hoelee.com)** - -Website design and development is my main line of work; server hardening and -self-hosted infrastructure is the other half of it. -```` - ---- - -## 文章 04 / 11：浏览器翻译没有 API:我给一个纯中文网页应用加上了英文模式 -- **Slug**: `adding-english-mode-to-a-chinese-only-web-app` -- **英文标题**: No API for Browser Translation: Adding an English Mode to a Chinese-Only Web App -- **分类**: `engineering` | **发布日期**: 2025-06-11 -- **线上地址**: https://blog.hoelee.com/posts/adding-english-mode-to-a-chinese-only-web-app/ (EN) | https://blog.hoelee.com/posts/zh/adding-english-mode-to-a-chinese-only-web-app/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "浏览器翻译没有 API:我给一个纯中文网页应用加上了英文模式" -description: "浏览器不提供触发翻译的 JavaScript API,而我的应用明明是中文却声明自己是英文。于是我改用反向代理,把一个英文模式注入进去。" -pubDate: 2025-06-11 -updatedDate: 2026-09-29 -category: engineering -blueprint: sme-digital-suite -blueprintLayer: L1-high-conversion-web -tags: ["i18n", "nginx", "vue", "javascript", "self-hosting"] -ogImage: "/og/zh-adding-english-mode-to-a-chinese-only-web-app.jpg" -banner: "/banners/adding-english-mode-to-a-chinese-only-web-app.jpg" -draft: false ---- - -「能不能用 JavaScript 触发浏览器自带的翻译?」答案是**不能**。接受这一点花了我一个下午 —— 但更有意思的发现出现在前面:我的应用其实一直在破坏浏览器的翻译功能,而且只用一个属性就做到了。 - -## 我在做什么 - -我维护着一个私人的邀请制阅读平台 `book.hoelee.com`:自建的阅读器(Kotlin/Spring 后端 + Vue 单页应用),装着我自己的藏书,并给受邀读者开了账号。界面是纯中文 —— 不是「以中文为主」,而是**只有中文**。我在相信这一点之前把每个前端 bundle 都翻了一遍:没有 i18n 库、没有语言包、没有语言切换,只有一个简繁转换表和一堆英文 TTS 语音名。 - -后来我开始把邀请卡发给只说英文的读者。他们能注册,然后迎面撞上:书架、浏览书仓、确定、设置 —— 每一个控件都在他们读不懂的语言里。这不是「体验不够精致」的问题。对我邀请的一半人来说,这个产品是不可用的。 - -有一条约束决定了后面所有做法:**我不改这个应用。** 它的上游仓库已经归档,界面是压缩过的 Vue bundle —— 任何我打进镜像里的补丁,都会在下一次换镜像时消失。所以我做的东西必须待在**外面一层**:我已经架在应用前面的那个 nginx 网关。 - -## 发现一:应用声明了自己错误的语言,而这会关掉翻译 - -写代码之前,我先看看浏览器到底看到了什么: - -```bash -$ curl -s https://book.hoelee.com/index.html | grep -o ']*>' - -``` - -外壳声明英文,界面 100% 中文。 - -这件事比看上去严重。Chrome、Edge、Safari 决定「要不要**主动提示**翻译」,很大程度取决于页面声明的语言。一个声明自己是英文、却渲染中文内容的页面,不会得到那个「要翻译此页面吗?」的提示 —— 对我的新读者来说最有用的一项无障碍功能,就这么被悄悄关掉了;而「右键 → 翻译成中文/英文」是大多数人从来不知道的操作。 - -修复只要一行,而且是作用在**响应出站时**,不是打进应用里: - -```nginx -location = /index.html { - # sub_filter 不能改写 gzip 后的内容 -> 让上游返回未压缩的 HTML - proxy_set_header Accept-Encoding ""; - sub_filter_once on; - # 应用声明 lang="en" 而界面是中文 -> 浏览器因此永远不提示中译英。 - # 说出事实。 - sub_filter '' ''; - proxy_pass http://hectorqin-reader:8080; -} -``` - -用发现它的方式验证:`curl -s https://book.hoelee.com/index.html | grep -o '' ''; - proxy_pass http://hectorqin-reader:8080; -} -``` - -整个部署就是这样:一个 40 KB 的脚本,只在 `/index.html` 加载,所以按钮只存在于应用内,别处都没有。脚本做三件事。 - -### 1. 从应用自己的 bundle 里「收割」词汇表 - -界面文案本来就在压缩后的 JavaScript 里,以字符串字面量的形式存在。把所有含中日韩字符的字符串抓出来并计数: - -```python -import re, collections -CJK = re.compile(r'[\u4e00-\u9fff]') -count = collections.Counter() -for bundle in bundles: - for m in re.finditer(r'"([^"\\\n]{2,40})"', bundle): - if CJK.search(m.group(1)): - count[m.group(1)] += 1 -print(len(count)) # 840 条唯一字符串,按出现频率排序 -``` - -跑出 840 条唯一字符串:菜单(书架、书源管理)、确认框(确认要删除所选择的书籍吗?)、设置项(段落行距)、错误信息(本地书籍源文件不存在)。词典最终是 **871 条** —— 收割来的,加上我在英文模式下实际使用应用、看还有什么没翻而补上的。 - -### 2. 按 DOM 的真实样子匹配,而不是你希望的样子 - -朴素的子串替换会毁掉中文词组:书架 就藏在 加入书架 里面,单字更糟(页 藏在 页面 里)。三条规则解决了它: - -- **先做整标签匹配,基于去掉空白后的文本。** 应用里的标签是 `" 书架 "` 这种带空格的,所以词典里的 `设置` 必须能匹配 DOM 里的 `" 设置 "`。 -- **然后是最长优先的子串替换**,只允许长度 ≥ 2 的键参与。 -- **单字键只做整标签匹配**(章、页、条),这样它们永远不可能破坏更长的词。 - -```js -function tr(s) { - if (!s || !CJK.test(s)) return s; - var hit = EXACT[s.replace(/[\s\u3000]/g, '')]; // 整标签匹配,空白归一化 - if (hit) return keepWS(s, hit); - var t = s; - for (var i = 0; i < KEYS.length; i++) { // KEYS:最长优先,长度 >= 2 - if (t.indexOf(KEYS[i]) >= 0) t = t.split(KEYS[i]).join(DICT[KEYS[i]]); - } - // 收拾数量短语:共58个可用书源 -> "58 usable sources" - return t.replace(/共\s*(\d+)/g, '$1').replace(/(\d+)\s*个/g, '$1').replace(/(\d)([A-Za-z])/g, '$1 $2'); -} -``` - -### 3. 跟上框架的重绘 - -它是 Vue 应用,DOM 一直在被重写 —— 只跑一次的翻译,一秒后就有一半是错的。一个带 150 ms 防抖的 `MutationObserver`,在任何变化之后重跑一遍遍历。 - -这一遍是幂等的:标签一旦变成英文就不再含中日韩字符,于是不会被再写一次,也不会形成写入循环。文本节点之外,`placeholder`、`title`、`aria-label`、`alt` 和输入框的 `value` 也会翻译 —— 所以提示气泡和搜索框也一起变英文了。 - -UI 是右下角的一个小胶囊:关闭时显示 `EN`,开启后显示 `中文`,选择记在 `localStorage`,还提供「隐藏此按钮」给更愿意用自己翻译器的读者。 - -## 它**刻意**不翻译什么 - -**书籍正文。** 阅读器把 EPUB 内容渲染在 iframe 里,我的遍历器按设计跳过 iframe —— 而且我无论如何也不会去机器替换一本小说的正文。 - -同样保持原样的还有:书名与作者、书源名称、我自己建的分组名(梯子、精品、正版),以及应用那句诗。这些是**数据**,不是界面。翻译界面是一种体贴;悄悄改写用户自己的内容是另一回事,而且糟糕得多。 - -书籍正文仍然交给浏览器自带的翻译 —— 而且因为「发现一」,它现在真的会被提示出来。按钮的弹窗里就是这么写的。 - -## 要量,不要用眼睛看 - -「截图看起来基本是英文了」不算结果。我在开关按钮的前后,数了实时 DOM 里含中日韩字符的文本节点: - -```js -let cjk = 0, total = 0; // 遍历器跳过注入的 UI 与 script/style -while ((n = walker.nextNode())) { - const t = n.nodeValue.trim(); - if (!t) continue; - total++; - if (/[\u4e00-\u9fff]/.test(t)) cjk++; -} -``` - -| 界面 | 之前 | 之后 | 已翻译 | -|---|---|---|---| -| 登录 / 注册对话框 | 80 | 12 | 85% | -| 书架 + 设置 | 259 | 60 | 77% | -| 阅读页界面 | 214 | 31 | 88% | - -那些界面里仍被计为中文的,全部是数据 —— 书名、作者、书源名、我自己的分组名、那句诗 —— 正是应该保持中文的部分。 - -## 我会怎么做得不一样 - -**不要用手工重写一份 871 条的词典。** 我重构过一次,静默丢了 25 个键。症状看起来像逻辑 bug:设置 还是中文,而它的每个兄弟标签都翻译正常。匹配器没问题 —— 是**条目**不见了。在你相信一次重写之前,用程序对比键集合: - -```bash -python -c "print(sorted(set(old_keys) - set(new_keys)))" -``` - -**自动化 UI 检查要写短。** 我的无头运行器每次调用硬超时 60 秒。一个遍历六本书的循环返回 HTTP 408 和空 body,而我的 JSON 解析器报的是「解析错误」而不是「超时」—— 我一度在查错的东西。 - -**调试之前先搞清楚 SPA 的真实路由。** `#/reader/1` 会渲染出一个空白外壳,让人以为应用坏了;真正的路由是 `#/reader?bookUrl=`。 - -**注意你自己的跳转逻辑。** 我的网关会把任何刷新送回前台页,除非设置了 `sessionStorage` 的通行标记。把界面切**回**中文会重载页面 —— 所以按钮会先设好同一个标记,否则读者会因为改了个语言设置被踢出应用。 - -## 结果 - -一个只懂英文的读者,现在打开平台、点一下按钮,就能用英文读界面。应用本身没有被改动:整个功能只是一个由 sidecar nginx 容器提供的 40 KB 脚本,所以下一次换镜像不可能弄坏它;而浏览器自带的翻译,终于有一个会声明自己真实语言的页面可以配合。 - -我留下的经验:**当平台不给你 API 时,往外看一层。** 反向代理是添加「应用里加不了的行为」的正当位置 —— 而且通常是唯一能在升级中存活的位置。 - ---- - -*我为马来西亚的中小企业搭建并自托管这类平台 —— 网页应用、私人书库、内部工具。有需要可以找我:[WhatsApp +60 12-797 2969](https://wa.me/60127972969)、[me@hoelee.com](mailto:me@hoelee.com?subject=Self-hosted%20platform),或 [hoelee.com](https://www.hoelee.com)。* -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "No API for Browser Translation: Adding an English Mode to a Chinese-Only Web App" -description: "Browsers expose no JavaScript API to trigger translation — and my app declared English while being Chinese. How I injected an English mode at the reverse proxy." -pubDate: 2025-06-11 -updatedDate: 2026-09-29 -category: engineering -blueprint: sme-digital-suite -blueprintLayer: L1-high-conversion-web -tags: ["i18n", "nginx", "vue", "javascript", "self-hosting"] -ogImage: "/og/adding-english-mode-to-a-chinese-only-web-app.jpg" -banner: "/banners/adding-english-mode-to-a-chinese-only-web-app.jpg" -draft: false ---- - -Can I trigger the browser's translate feature from JavaScript? No. That answer cost me an afternoon to accept — but the more interesting discovery came first: my app was already sabotaging the browser's translation, and it did it with a single attribute. - -## What I was working on - -I run a private, invite-only reading platform at `book.hoelee.com`: a self-hosted reader (Kotlin/Spring backend, Vue single-page app) holding my library, with accounts for invited readers. The interface is Chinese-only. Not "Chinese-first" — *only*. I grepped every served bundle before believing it: no i18n library, no locale files, no language switcher. Just a Simplified/Traditional character converter and a list of English TTS voice names. - -Then I started handing out invite cards to English-speaking readers. They could register, and then they met 书架, 浏览书仓, 确定, 设置 — every control in a language they don't read. That isn't a polish problem. For half the people I invited, the product was unusable. - -One constraint shaped everything that follows: **I don't modify the app.** Its upstream repository is archived and the UI is a minified Vue bundle, so anything I patch inside the image dies the next time I swap the image. Whatever I build has to live one layer out — in the nginx gateway I already run in front of the app. - -## Finding 1: the app declares the wrong language, and that disables translation - -Before writing a single line of code, I looked at what the browser sees: - -```bash -$ curl -s https://book.hoelee.com/index.html | grep -o ']*>' - -``` - -The shell says English. The interface is 100% Chinese. - -That matters more than it looks. Chrome, Edge and Safari decide whether to *offer* translation largely from the page's declared language. A page that declares English while rendering Chinese text doesn't get the "Translate this page?" prompt — the single most useful accessibility feature for my new readers was silently switched off, and right-click → *Translate to English* is a ritual most people never learn. - -The fix is one line, applied to the response on the way out instead of inside the app: - -```nginx -location = /index.html { - # sub_filter cannot rewrite a gzipped body -> ask upstream for plain HTML - proxy_set_header Accept-Encoding ""; - sub_filter_once on; - # the app declares lang="en" while its UI is Chinese -> browsers then never - # offer Chinese -> English translation. Declare the truth. - sub_filter '' ''; - proxy_pass http://hectorqin-reader:8080; -} -``` - -Verify the same way you found it: `curl -s https://book.hoelee.com/index.html | grep -o '' ''; - proxy_pass http://hectorqin-reader:8080; -} -``` - -That's the whole deployment: a 40 KB script, loaded only on `/index.html`, so the button exists inside the app and nowhere else. The script does three things. - -### 1. Harvest the vocabulary from the app's own bundles - -The UI strings are already in the minified JavaScript as quoted literals. Extract every one that contains a CJK character and count them: - -```python -import re, collections -CJK = re.compile(r'[\u4e00-\u9fff]') -count = collections.Counter() -for bundle in bundles: - for m in re.finditer(r'"([^"\\\n]{2,40})"', bundle): - if CJK.search(m.group(1)): - count[m.group(1)] += 1 -print(len(count)) # 840 unique strings, most frequent first -``` - -840 unique strings came out: menus (书架, 书源管理), confirmations (确认要删除所选择的书籍吗?), settings labels (段落行距), error messages (本地书籍源文件不存在). The dictionary ended up at **871 entries** — the harvest plus everything I found by running the app in English mode and watching what stayed Chinese. - -### 2. Match like the DOM actually is, not like you wish it were - -Naive substring replacement mangles Chinese compounds: 书架 sits inside 加入书架, and single characters are worse (页 inside 页面). Three rules fixed it: - -- **Whole-label match first, on whitespace-stripped text.** The app stores labels like `" 书架 "` with padding, so a dictionary key of `设置` has to match `" 设置 "` in the DOM. -- **Then longest-first substring replacement**, restricted to keys of two or more characters. -- **One-character keys are whole-label only** (章, 页, 条), so they can never corrupt a longer word. - -```js -function tr(s) { - if (!s || !CJK.test(s)) return s; - var hit = EXACT[s.replace(/[\s\u3000]/g, '')]; // whole label, whitespace-normalised - if (hit) return keepWS(s, hit); - var t = s; - for (var i = 0; i < KEYS.length; i++) { // KEYS: longest-first, length >= 2 - if (t.indexOf(KEYS[i]) >= 0) t = t.split(KEYS[i]).join(DICT[KEYS[i]]); - } - // tidy count phrases: 共58个可用书源 -> "58 usable sources" - return t.replace(/共\s*(\d+)/g, '$1').replace(/(\d+)\s*个/g, '$1').replace(/(\d)([A-Za-z])/g, '$1 $2'); -} -``` - -### 3. Keep up with the framework - -It's a Vue app, so the DOM is rewritten constantly — a translation pass that runs once is a translation pass that's half wrong a second later. A `MutationObserver` with a 150 ms debounce re-runs the walk after any change. - -The pass is idempotent: once a label is English it contains no CJK, so nothing is rewritten and there is no write loop. Text nodes plus `placeholder`, `title`, `aria-label`, `alt` and input `value` attributes all get translated, which is why tooltips and the search box come along too. - -The UI is a small pill in the bottom-right corner: `EN` when off, `中文` when on, remembered in `localStorage`, with a "Hide this button" link for readers who prefer their own translator. - -## What it deliberately does not translate - -**Book text.** The reader renders EPUB content inside an iframe; my walker skips iframes by design, and I would never machine-swap a novel's prose anyway. - -Also left alone: book titles and authors, book-source names, and the group names I created myself (梯子, 精品, 正版), plus the app's poem tagline. Those are *data*, not interface. Translating the chrome is a courtesy; silently rewriting a user's own content is a different and much worse idea. - -For book text the browser's own translation is still the right tool — and it now actually gets offered, thanks to Finding 1. The button's popup says exactly that. - -## Measure it, don't eyeball it - -"Screenshots look mostly English" is not a result. I counted CJK-containing text nodes in the live DOM before and after toggling the button: - -```js -let cjk = 0, total = 0; // walker skips the injected UI + script/style -while ((n = walker.nextNode())) { - const t = n.nodeValue.trim(); - if (!t) continue; - total++; - if (/[\u4e00-\u9fff]/.test(t)) cjk++; -} -``` - -| Screen | Before | After | Translated | -|---|---|---|---| -| Login / register dialog | 80 | 12 | 85% | -| Bookshelf + settings | 259 | 60 | 77% | -| Reading page chrome | 214 | 31 | 88% | - -Everything still counted as Chinese on those screens is data — titles, authors, source names, my own group names, the tagline — which is exactly what should stay Chinese. - -## What I'd do differently - -**Don't rewrite an 871-entry dictionary by hand.** I restructured mine once and silently dropped 25 keys. The symptom looked like a logic bug: 设置 stayed Chinese while every sibling label translated correctly. The matcher was fine — the *entry* was gone. Diff the key sets programmatically before you trust a rewrite: - -```bash -python -c "print(sorted(set(old_keys) - set(new_keys)))" -``` - -**Keep automated UI checks short.** My headless runner hard-times-out at 60 seconds per call. A six-book loop came back HTTP 408 with an empty body, and my JSON parser reported a parse error rather than a timeout — I debugged the wrong thing for a while. - -**Learn the SPA's real routes before debugging it.** `#/reader/1` renders a blank shell and makes you think the app is broken; the actual route is `#/reader?bookUrl=`. - -**Mind your own bounce logic.** My gateway sends any reload back to the front page unless a `sessionStorage` pass flag is set. Switching the interface *back* to Chinese reloads the page — so the button sets that same flag first, or the reader gets thrown out of the app for changing a language setting. - -## Result - -An English-only reader now opens the platform, taps one button, and reads the interface in English. The app itself is untouched: the whole feature is a 40 KB script served by a sidecar nginx container, so the next image update cannot break it, and the browser's own translation has a page that declares its real language to work with. - -The lesson I'm keeping: **when the platform won't give you an API, look one layer out.** A reverse proxy is a legitimate place to add behaviour you can't add in the app — and it's usually the only place that survives upgrades. - ---- - -*I build and self-host platforms like this one — web apps, private libraries, internal tools for small businesses in Malaysia. If you need something similar: [WhatsApp +60 12-797 2969](https://wa.me/60127972969), [me@hoelee.com](mailto:me@hoelee.com?subject=Self-hosted%20platform), or [hoelee.com](https://www.hoelee.com).* -```` - ---- - -## 文章 05 / 11：排查 Remotion 视频抽搐：为什么在背景图挂载音频低频 Scale 会毁了转场 -- **Slug**: `debugging-remotion-video-jitter` -- **英文标题**: Debugging Remotion Video Jitter: Why Bass Scaling Ruins Background Transitions -- **分类**: `engineering` | **发布日期**: 2025-11-26 -- **线上地址**: https://blog.hoelee.com/posts/debugging-remotion-video-jitter/ (EN) | https://blog.hoelee.com/posts/zh/debugging-remotion-video-jitter/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "排查 Remotion 视频抽搐：为什么在背景图挂载音频低频 Scale 会毁了转场" -description: "深入排查 Remotion 音频可视化中背景画面剧烈抽搐与切镜暗闪的根本原因：解耦 HUD 与背景图层，以及纯净单向叠化与硬件基准实测对比。" -pubDate: 2025-11-26 -updatedDate: 2026-10-22 -category: engineering +title: "用 YuE2 BF16 生成完整长歌：突破时长限制与音频硬截断" +description: "AI 音乐工具常在歌词中途戛然而止。本文详解如何利用 YuE2 BF16、295秒余量预留公式与本地 ComfyUI，成功批量跑出 60 首自然收尾的完整长歌全集。" +pubDate: 2026-10-26 +category: tutorials +blueprint: ai-agent-automation +blueprintLayer: L4-aigc-media-pipelines stream: creative-lab -tags: ["remotion", "react", "audio-visualizer", "css-animation", "debugging"] -ogImage: "/og/zh-debugging-remotion-video-jitter.jpg" -banner: "/banners/debugging-remotion-video-jitter.jpg" +tags: ["ai-music", "yue2", "comfyui", "audio-engineering", "subtitles", "python"] +ogImage: "/og/zh-generating-full-length-songs-with-yue2-comfyui.jpg" +banner: "/banners/generating-full-length-songs-with-yue2-comfyui.jpg" draft: false --- -使用 React 和 Remotion 编写自动化视频赋予了开发者对每一帧画面的极致掌控。你可以直接将 Web Audio 的频域分析数据映射到 SVG 属性、CSS 变换矩阵以及排版样式上，这种灵活性是传统非线性剪辑软件很难比拟的。 +在当前几乎所有的 AI 音乐生成方案中，最普遍且让人抓狂的缺陷莫过于： +**歌曲唱到一半，突然被硬生生切断**。 -然而，将视频帧当作常规网页来渲染时，极易踩入微妙的视觉陷阱。在我们的自动化音乐视频生成管线中，一个在开发组件预览时看似完全正常的设计，在全量导出为 1080P 30fps 视频后，却出现了极其刺眼、让人目眩的“地震式剧烈抽搐”。 +不管是各大商用平台还是开源权重，受限于固定的声学 Token 训练窗口，只要我们尝试塞入完整的 +叙事性歌词结构——即标准的 9 阶段曲式 `[Intro] -> [Verse 1] -> [Chorus] -> [Interlude] -> [Verse 2] -> [Chorus] -> [Bridge] -> [Chorus] -> [Outro]`——生成引擎就极易在进入副歌高潮或尾奏时撞上时长硬天花板，在歌手吐字的中途突兀静音。 -本文记录这次视觉 BUG 的深度排查过程：为什么对音频动效的朴素直觉会毁了整部视频，以及如何重构 React 视频图层架构以实现丝滑稳定的影视级转场。 +在单张 RTX 3060 12GB 显卡上经历数百次实跑与测试后，我摸索出了一套能够稳定产出 +**4 分钟以上、具备自然淡出收尾、绝不截断**的完整长歌工程链路。 -> **线上成品验证**：修复后的渲染流水线已正式交付并发布至 YouTube：[《糖画小仙》1080p 官方 MV](https://youtu.be/vZbyKeCCSRA)，全片背景稳固、转场平滑。 - -> **TL;DR** —— 让整幅背景图跟着低频段缩放，等于每一帧都重新做图像插值采样，肉眼读到的就是剧烈抖动与震颤；再加上两层半透明图层对称交叉淡化，在时间中点必然透出底层的黑色容器。修复方案：背景画面绝对静态，将所有音频动效彻底隔离在顶层 HUD 频谱层，转场采用「100% 实体底图 + 顶层单向淡入」。渲染耗时实测**缩短 38.0%（从 352 秒降至 218 秒，吞吐速度提升 1.61 倍）**。这一篇是 [全自动 MV 生成流水线](/posts/zh/how-i-built-an-automated-music-video-pipeline/) 的渲染篇；文件落盘与发布那一半见 [Docker 容器大文件上传](/posts/zh/uploading-large-videos-to-youtube-via-cdp/)。 +本文将拆解导致音频硬截断的底层成因、295 秒余量预留公式，并完整交付 **60 首高质量长歌全集**（每首均配套 1024×1024 独立 WebP 封面与逐行同步滚动歌词）。 --- -## 1. 故障现象：全画面的“高频地震” + -该视频合成组件最初的视觉设想非常清晰： -1. 底层是一组 1080P 高清插画分镜，在不同章节间慢速淡入淡出（Crossfade）； -2. 画面上方浮动一个跟随音乐重音律动的环形音频频谱仪； -3. 三行书法歌词居中显示在音频环内，并根据每张插画的主体位置在画面左右两侧平滑滑移避让。 - -在开发工具里单帧点按检查时，画面一切正常。但在以 30fps 正常播放渲染出的成片时，肉眼观感极度难受：每当强烈的电子重音鼓点响起，整张背景图每秒剧烈颤抖震荡数十次；不仅如此，在两张图片交替切镜的 1.8 秒窗口内，画面中央还会突然闪现明显的“暗黑闪烁”，亮度瞬间下沉。 - ---- - -## 2. 根因分析一：低频 Scale 能量污染背景图 - -翻阅最初的 `BackgroundSlideshow.tsx` 组件，我们发现了一段看似合情合理的动效代码： - -```tsx -// 埋下灾难的罪魁祸首代码 -const bassScale = interpolate(bassLevel, [0, 1], [1.0, 1.025], { - extrapolateRight: 'clamp', -}); - -// 应用在全屏背景图片上： - -``` - -在直觉中，重低音响起时让背景图随着节奏轻微放大 2.5%，听起来像是在模拟动感的镜头呼吸感。然而在真实物理渲染中： -- **电音鼓点的瞬态脉冲只有 30~60 毫秒**：低频能量在连续 2 到 3 帧之内瞬间从零冲上顶峰，又在下一帧骤跌回原点； -- **1920x1080 的巨幅背景图放大 2.5%，边缘物理像素瞬间位移多达 25 到 30 像素**； -- 这种在相邻帧之间急剧拉扯的剧烈像素位移，人眼接收到的根本不是“音乐律动”，而是被视网膜本能判定为**强烈的镜头颠簸、抽动与丢帧残影**； -- 更致命的是，在 1.8 秒的交叉淡入淡出期间，两张重叠的图片各自具有不同的锚点和缩放系数，两层图像以不同的相位同时高频抖动，产生了极其混乱的重影眩晕感。 - ---- - -## 3. 根因分析二：对称线性透明度重叠时的“黑闪”陷阱 - -第二个问题是切镜时的亮度下陷黑闪。在传统 Web CSS 中，人们通常很自然地写出： - -```css -/* 常见的渐变叠化陷阱 */ -.current-slide { opacity: 1 -> 0; } -.next-slide { opacity: 0 -> 1; } -``` - -当两张图片都以线性渐变进行透明度变换时，在正中间那颗时间点（$t = 0.5$）： -- 当前图片透明度 = 0.5 -- 下一张图片透明度 = 0.5 - -来看浏览器和 Remotion 的合成器是如何将这几层叠在底层的纯黑容器之上的： - -``` -┌─────────────────────────────────────────────────────────────┐ -│ Layer 2: 下一张图片（Next Slide，透明度 = 0.5） │ -├─────────────────────────────────────────────────────────────┤ -│ Layer 1: 当前图片（Current Slide，透明度 = 0.5） │ -├─────────────────────────────────────────────────────────────┤ -│ Layer 0: 根容器底色（Root Container，背景色 = #000000 纯黑） │ -└─────────────────────────────────────────────────────────────┘ -``` - -由于多层半透明通道是依次向下进行乘法 Alpha 混合： -$$\text{有效综合不透明度} = 1 - (1 - 0.5) \times (1 - 0.5) = 1 - 0.25 = 0.75$$ - -合成后的总有效透明度只有 **0.75**，这意味着多达 **25% 的底层纯黑背景**被暴露了出来！这不是错觉，而是纯粹的数学光度衰减。每一次转场切镜，画面整体照度都会凭空跌掉四分之一，造成非常明显的瞬间暗淡闪烁。 - ---- - -## 4. 重构方案：稳固基座 + 纯净单向叠化 - -找到症结后，我们对组件进行了彻底重构： - -### 1. 彻底剥离背景图层的所有音频缩放 -背景插画必须稳如磐石。所有的音频跳动、瞬态光斑与能量爆发，**只能且必须限定在顶层的 HUD 频谱仪中**。背景层坚决不承载任何高频振荡属性。 - -### 2. 底层 100% 实体 + 顶层单向平滑淡入 -坚决不要让底图淡出。**当前图片始终保持 100% 不透明**，下一张图片直接覆盖在其上方，慢速执行 `0 -> 1.0` 的单向淡入： - -```tsx -return ( -
- {/* 1. 底层图片：坚固不透明，彻底消除暗黑闪烁 */} - - - {/* 2. 顶层图片：在 1.8 秒内平滑淡入 (0 -> 1.0) */} - {isTransitioning && ( - - )} +
+
+ 🎵 60首全集快速检索与跳转面板 + 60首完整长歌 · 懒加载与按需播放
-); -``` - -### 3. HUD 与背景物理层级绝对隔离 -歌词弹幕与音频环容器设置在独立的顶层图层中，并赋予 `pointerEvents: 'none'`。歌词在留白侧之间的横向位移完全根据当前时间戳自主计算，与底层的图片渐变完全物理隔离，互不干涉。 - ---- - -## 5. 改造后的成效与基准复现 - -重构落地后，对比极其显著： - -1. **彻底消除画面抖动**：无论音乐鼓点多么狂暴密集，背景画面在所有测试分镜中纹丝不动，人物与细节清晰稳定； -2. **转场不再发生黑闪**：1.8 秒的单向贝塞尔叠化始终保持 100% 连续亮度，杜绝了中间态透出底层黑色的缺陷； -3. **渲染吞吐速度大幅提升**：移除了对超大尺寸背景图的高频矩阵重算后，渲染性能获得了实测提升。 - -### 基准实测与硬件复现条件 - -| 评估指标 | 优化前版本 | 优化后版本 | 变化幅度与提升 | -|---|---|---|---| -| **渲染总耗时** | 352.0 秒 | 218.3 秒 | **耗时减少 38.0% (-133.7s)** | -| **平均渲染帧率** | 22.24 fps | 35.87 fps | **吞吐速度提升 1.61 倍** | -| **总计处理帧数** | 7,830 帧 | 7,830 帧 | 完全一致（261 秒 / 4 分 21 秒） | -| **输出视频规格** | 1920 × 1080 (1080p @ 30fps) | 1920 × 1080 (1080p @ 30fps) | 完全一致 | -| **编码器设置** | H.264 (CRF 18, preset medium) | H.264 (CRF 18, preset medium) | 完全一致 | - -**基准测试宿主环境：** -- **CPU**：Intel Core i9-13900K（24 核 / 32 线程，最高睿频 5.8 GHz） -- **GPU**：NVIDIA GeForce RTX 3060（12GB GDDR6 显存，驱动版本 551.xx） -- **内存**：64GB DDR5 5600 MHz -- **系统**：Windows 11 专业版 64 位 -- **软件栈**：Node.js v20.x，Remotion v4.0.x，开启 Chromium 无头 GPU 加速 - -两次基准测试均在同一工作站、连续以完全相同的音频频谱输入和插画素材进行。渲染耗时缩短 38.0%，即吞吐速度达到优化前的 **1.615 倍**，使原本慢于真实播放时长的渲染变为高于实时帧率的高效输出。 - ---- - -## 本专题关联阅读 -- **系统总架构篇**：[我是如何用 Remotion 与 Docker CDP 打造全自动 MV 生成与多语种油管发布流的](/posts/zh/how-i-built-an-automated-music-video-pipeline/) -- **大文件上传篇**：[借助 Docker Archive API 模式通过 CDP 自动化上传 100MB+ 视频到 YouTube](/posts/zh/uploading-large-videos-to-youtube-via-cdp/) -- **线上实成片**：[前往 YouTube 观看《糖画小仙》1080P 官方完整成片](https://youtu.be/vZbyKeCCSRA) -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Debugging Remotion Video Jitter: Why Bass Scaling Ruins Background Transitions" -description: "Diagnose audio-reactive background jitter and crossfade flicker in Remotion. See how separating the HUD from background layers can improve visual stability and rendering performance." -pubDate: 2025-11-26 -updatedDate: 2026-10-22 -category: engineering -stream: creative-lab -tags: ["remotion", "react", "audio-visualizer", "css-animation", "debugging"] -ogImage: "/og/debugging-remotion-video-jitter.jpg" -banner: "/banners/debugging-remotion-video-jitter.jpg" -draft: false ---- - -Building programmatic videos with React and Remotion gives you complete DOM control over every frame. You can map web audio frequencies directly to SVG attributes, CSS transforms, and typography in ways standard video editors can barely touch. - -However, treating video frames like standard web pages introduces subtle visual bugs. In our automated music video pipeline, an export that looked plausible in unit tests developed a jarring, violent shudder when rendered into a 1080p MP4. - -Here is what went wrong, why our initial intuition about audio reactivity failed, and how to structure React video layers so audio-driven HUDs never pollute background transitions. - -> **Live Reference**: The repaired rendering pipeline produced the final production MV for [《糖画小仙》 (Sugar Art Fairy) on YouTube](https://youtu.be/vZbyKeCCSRA), running rock-solid at 1080p 30fps. - -> **TL;DR** — Scaling a full-bleed background image with the bass band forces an interpolation resampling on every frame, which the human eye perceives as violent shudder. Furthermore, two stacked semi-transparent layers crossfading symmetrically dip through the black root container at the midpoint. Fixes: keep the background canvas completely static, decouple all audio-reactive pulsing into isolated foreground HUD layers, and fade in the incoming slide on top of a 100% solid base. Render time dropped by **38.0% (from 352s to 218s, a 1.61× throughput speedup)**. This is the rendering chapter of our [music-video pipeline](/posts/how-i-built-an-automated-music-video-pipeline/); the file-staging half is covered in [uploading large videos via CDP](/posts/uploading-large-videos-to-youtube-via-cdp/). - ---- - -## 1. The Symptom: A Whole-Frame Earthquake - -The visual effect was supposed to be simple: -1. An illustrated 1080p background slowly crossfading between slides. -2. A pulsing circular spectrum visualizer in the foreground. -3. Three lines of Chinese calligraphy lyrics that slide between left and right depending on where the character in the artwork is standing. - -In preview scrubbing, everything looked acceptable. But during full-speed 30fps playback of the rendered 1080p file, the viewer felt instant visual fatigue. Every time the electronic drum pattern kicked in, the entire scene appeared to vibrate violently up to 30 times per second. Furthermore, during slide transitions, the screen experienced micro-flashes where the background brightness suddenly dipped. - ---- - -## 2. Root Cause 1: Low-Frequency Scale Pollution - -Looking at the initial `BackgroundSlideshow.tsx`, we had attached an audio reactivity hook: - -```tsx -// The culprit in BackgroundSlideshow.tsx -const bassScale = interpolate(bassLevel, [0, 1], [1.0, 1.025], { - extrapolateRight: 'clamp', -}); - -// Applied to the background image: - -``` - -On paper, expanding the image by 2.5% on bass hits sounds like an engaging dynamic pulse. In reality: -- **Electronic audio beats hit in 30–60ms bursts**: The bass amplitude oscillates rapidly between zero and peak within 2 to 3 consecutive frames. -- **A 1920x1080 image scaling by 2.5% shifts edge pixels by nearly 25 to 30 physical pixels in a single frame**. -- When that sudden displacement reverses on the very next frame, the human eye interprets it not as a musical pulse, but as **severe camera camera shake / stutter**. -- To make matters worse, during the 1.8-second slide transition, two overlapping images with different transform origins were simultaneously vibrating at conflicting scales, creating a hallucinatory double-vision jitter. - ---- - -## 3. Root Cause 2: Symmetrical Opacity Crossfade DIP - -The second issue was the crossfade flash. In standard web CSS, people often do: - -```css -/* Web crossfade trap */ -.current-slide { opacity: 1 -> 0; } -.next-slide { opacity: 0 -> 1; } -``` - -When two images crossfade with linear opacity, consider what happens at the exact midpoint ($t = 0.5$): -- $\text{Current Slide Opacity} = 0.5$ -- $\text{Next Slide Opacity} = 0.5$ - -Here is how the browser/Remotion compositing pipeline stacks these layers against the root black container: - -``` -┌─────────────────────────────────────────────────────────────┐ -│ Layer 2: Next Slide (opacity = 0.5) │ -├─────────────────────────────────────────────────────────────┤ -│ Layer 1: Current Slide (opacity = 0.5) │ -├─────────────────────────────────────────────────────────────┤ -│ Layer 0: Root Container (background = #000000 black) │ -└─────────────────────────────────────────────────────────────┘ -``` - -Because alpha blending operates multiplicatively down the stack: -$$\text{Composite Alpha} = 1 - (1 - 0.5) \times (1 - 0.5) = 1 - 0.25 = 0.75$$ - -The total combined alpha drops to **0.75**, exposing **25% of the black background**! This is not an optical illusion; it is mathematical light loss. Every time a transition peaked, the frame darkened by a quarter of its luminance, creating an unmistakable flicker. - ---- - -## 4. The Solution: Rock-Solid Base + Pure Single-Direction Opacity - -To fix both issues permanently: - -### 1. Remove All Scaling from Backgrounds -Background artwork must remain rock-solid. All audio reactivity belongs exclusively in the HUD layer (the radial spectrum bars and glowing rings). The background canvas should only ever receive smooth, non-oscillating transformations. - -### 2. Base Solid + Top Layer Fade-In -Instead of fading out the current image, **keep the current image at solid 100% opacity**. Only animate the incoming image on top: - -```tsx -return ( -
- {/* Base Slide: Stays 100% solid, eliminating any drop to black */} - - - {/* Next Slide: Smoothly fades in on top over 1.8s */} - {isTransitioning && ( - - )} +
+ + + + + + +
-); +
+ + +
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+ +## 核心痛点：240 秒的时长硬墙 + +在初期的实验中，使用默认配置调用 ComfyUI 生成音频时，经常陷入两个误区： + +1. **尾奏被腰斩（硬截断）**：若将 `max_duration` 设为 `240.0s` 并填满长歌词，模型在唱到最后一段副歌时时间刚好用尽。此前跑出的 12 首样本无一例外精确停在 `240.000s`，尾部毫无渐隐，沦为残次品。 +2. **语速失控（慌乱压缩）**：如果试图在有限的窗口内通过提示词催促模型“唱快点”，自回归声学模型会自动压缩发音时长，导致原本舒缓的民谣变成含混不清的“急口令”。 + +根本原因在于：**不能把最大时长（Max Duration）当成目标歌曲时长来设置。在自回归音频生成中，最大时长必须作为声学安全余量缓冲槽（Headroom Buffer）。** + +## 解决架构：三项核心工程策略 + +### 1. 295.0 秒 Headroom 留白公式 + +要得到一首体感约为 4 分钟的完整长歌，必须将**歌词词量规划在 210秒～260秒区间**，而底层生成参数直接赋予 **295.0 秒**的宽松上限： + +$$\text{Headroom 留白余量} = \text{参数上限 (295s)} - \text{预估词量时长 (约 220s)} \approx 75\text{秒}$$ + +这 75 秒看似多余的余量，是自回归声学模型实现“体面收尾”的命脉所在： +- 保证最后一节副歌完整舒展唱完，不产生急促抢拍； +- 赋予模型充足的 Token 生成纯乐器 Outro 尾奏； +- 让尾音伴随混响自然衰减至真正的分贝零点。 + +### 2. 自动化硬切隔离门禁（Quarantine Gate） + +在批量跑批自动化脚本中，加入严格的质量门禁：凡是实际音频长度逼近上限（`duration >= 294.9s`）的文件，判定为发生硬截断，自动归档进 `未自然结尾/` 隔离目录；仅有在上限之前自然休止收音（`duration < 294.0s`）的成品才予以放行。 + +本次实跑的 60 首曲目最终时长均匀分布在 **129.9s 至 285.6s**，100% 实现了自然淡出。 + +### 3. 12GB 显存的高效自持 + +在 12GB 显存下运行 YuE2-3B BF16，必须严格执行显存释放策略： + +```bash +# 启动 ComfyUI（关键参数防 OOM） +cd /d/app/ComfyUI +./python_embeded/python.exe -s ComfyUI/main.py \ + --windows-standalone-build \ + --port 8188 \ + --disable-pinned-memory \ + --disable-nvml-pressure \ + --cache-classic ``` -### 3. Physical Decoupling of HUD and Canvas -The lyric box and circular spectrum are mounted inside a completely separate container layer with `pointerEvents: 'none'`. As the HUD glides across the canvas from `centerX: 540` to `centerX: 1380`, it calculates its position strictly from the timeline clock. It has zero knowledge of and zero impact on the background's opacity crossfade. +在批处理完成后及时调用 `/free` 卸载显存，避免连续生成多首长音频时产生显存碎片积压。 --- -## 5. The Outcome & Benchmark Reproduction - -After removing the background `bassScale` and switching to one-way opacity stacking: - -1. **Eliminated visible jitter**: Background illustrations remain crisp and visually anchored through rapid 140 BPM electronic drum drops across all test sequences. -2. **Eliminated transition light dips**: The 1.8s Bezier crossfade maintains 100% luminance continuity without dipping through the dark container base. -3. **Significant throughput speedup**: Removing per-frame matrix interpolation and scaling calculations on high-resolution background assets delivered a measurable performance boost. - -### Reproduction Benchmark - -| Metric | Before Optimization | After Optimization | Delta | -|---|---|---|---| -| **Render Duration** | 352.0 seconds | 218.3 seconds | **-38.0% (-133.7s)** | -| **Render Throughput** | 22.24 fps | 35.87 fps | **+1.61× speedup** | -| **Total Frames** | 7,830 frames | 7,830 frames | Identical (261s / 4m 21s) | -| **Output Resolution** | 1920 × 1080 (1080p @ 30fps) | 1920 × 1080 (1080p @ 30fps) | Identical | -| **Codec & Settings** | H.264 (CRF 18, preset medium) | H.264 (CRF 18, preset medium) | Identical | - -**Test Environment Specifications:** -- **CPU**: Intel Core i9-13900K (24 cores / 32 threads, boost up to 5.8 GHz) -- **GPU**: NVIDIA GeForce RTX 3060 (12GB GDDR6 VRAM, Driver 551.xx) -- **RAM**: 64GB DDR5 5600 MHz -- **OS**: Windows 11 Pro 64-bit -- **Software**: Node.js v20.x, Remotion v4.0.x with Chromium headless GPU rendering enabled - -Both benchmark runs were executed consecutively using identical media assets, audio spectrogram inputs, and composition parameters. The 38.0% reduction in render time directly translates to a **1.61× throughput speedup**, turning a process that was previously slower than realtime into an accelerated 35.8 fps export. - ---- - -## Related Guides in this Series -- **Pipeline Architecture**: [How I Built an Automated Music Video Pipeline with Remotion and Docker](/posts/how-i-built-an-automated-music-video-pipeline/) -- **Remote Publishing**: [Uploading 100MB+ Videos to YouTube via CDP: The Docker Archive Pattern](/posts/uploading-large-videos-to-youtube-via-cdp/) -- **Live Output**: [Watch the finished MV on YouTube](https://youtu.be/vZbyKeCCSRA) -```` - ---- - -## 文章 06 / 11：把字体嵌进 PDF，让它在任何机器上都长一样 -- **Slug**: `embedding-fonts-so-a-pdf-looks-the-same-everywhere` -- **英文标题**: Embedding Fonts So a PDF Looks the Same on Every Machine -- **分类**: `engineering` | **发布日期**: 2025-01-22 -- **线上地址**: https://blog.hoelee.com/posts/embedding-fonts-so-a-pdf-looks-the-same-everywhere/ (EN) | https://blog.hoelee.com/posts/zh/embedding-fonts-so-a-pdf-looks-the-same-everywhere/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "把字体嵌进 PDF，让它在任何机器上都长一样" -description: "四个字族、八个字面、两个渲染引擎，外加一条 CSS 怪癖：为了不让生成的报告依赖服务器上恰好装了什么字体，我做了什么。" -pubDate: 2025-01-22 -updatedDate: 2026-10-16 -category: engineering -tags: ["pdf", "fonts", "mpdf", "typography", "php"] -ogImage: /og/zh-embedding-fonts-so-a-pdf-looks-the-same-everywhere.jpg -banner: /banners/embedding-fonts-so-a-pdf-looks-the-same-everywhere.jpg -draft: false ---- - -## 需求只有一句话 - -一个报告生成器必须产出在任何地方都长得一样的文档：中文标题用宋体、中文正文用黑体、英文标题与封面数字用一款展示衬线体、英文正文和表格用 Inter。字重也要对——强调用 medium、标题用 semibold——而且是三语、两个 PDF 渲染引擎都要对。 - -同时，**字体必须跟着文档走**。不是「服务器装过」，也不是「读者大概有」，而是**嵌进文件里**，这样另一个国家的印厂看到的页面是一样的。 - -老管线在最后这句上栽得很隐蔽：它依赖服务器上用包管理器装过的字体，而且**只嵌入了一款中文字体**。第二档字重根本不存在，于是「粗体」是渲染器的合成近似；而想发现这件事，唯一的办法是打开 PDF 去看里面到底装了什么。 - -## 规则一：从成品 PDF 里把字体读回来 - -「已嵌入」是一个关于**输出**的断言，所以我量的是输出。同一份文档，改前与改后： - -| | 改前 | 改后 | -|---|---|---| -| zh 页数 | 23 | **23** | -| en 页数 | 28 | **29** | -| bm 页数 | 30 | **30** | -| 真正嵌进去的字体 | 一款中文字体 + Roboto | 思源宋 Medium + SemiBold · 思源黑 Regular + Medium · Cormorant Garamond Medium + SemiBold · Inter Regular + SemiBold + Medium · 拉丁报告里的汉字走思源黑 | -| 渲染时间 / 峰值内存 | 基线 | 同一量级（±10%，属噪声） | -| 文件体积 | 基线 | 约 +3% | - -每个字面的字形数是收据：中文报告里 15,988 个字形落在正文黑体、2,247 个落在它的 medium 档、634 个落在标题宋体的 medium、222 个落在它的 semibold。这就是「字体在服务器上」与「字体在文件里」的区别——另一个国家的读者看到的是同样四个字面，而代价是某一种语言正好多一页（拉丁字体的度量比它替换掉的那个更宽）。 - -## 规则二：只出静态实例，理由写在解析器里 - -现代做法看起来该是「每个字族发一个可变字体」，但在这里是错的，有两个实测理由： - -1. **PDF 引擎的字体解析器只读 `glyf`/`loca`。** 喂它一个 PostScript/CFF 轮廓的 OpenType 文件，它直接拒绝：`Fonts with postscript outlines are not supported`。问题不是「可变」，是 CFF 轮廓。 -2. **它完全不读 `fvar` 表**，所以可变字体会**静默地**解析成它的**默认实例**——而中文字族的默认实例恰恰是最细的那一档（`name` 表里写得清清楚楚：Thin 100 与 ExtraLight 200）。整份报告会变成发丝体，而整条管线里不会报任何一个错。 - -所以构建步骤拿上游的可变字体**实例化成静态切面**：跑实例化器，然后把 `name`、`OS/2.usWeightClass` 和 `head.macStyle` 改写成它现在真实的字重（源文件的名字里还写着 Thin）。产物旁边有一份 manifest，逐文件记下 family、字重、sha256、字形数与码位数；许可证也随文件一起进仓库。 - -## 规则三：引擎只会说两种字重，那就映射到两个槽位 - -渲染引擎的 CSS 只认 `normal` 和 `bold`——它的源码自己就这么写（"Does not support: bolder, lighter, 100..900"）。所以数字字重对它毫无意义；而同一条管线里的无头 Chromium 又完全认它们。 - -可用的映射是：每个字族声明一个 **R 槽**和一个 **B 槽**（常规一档、更重一档），`font-weight: 500` 落到目标字族的 R 槽，`600` **只在 PDF 那一侧被改写成 `bold`**，落到 B 槽。Chromium 侧照写 `600`。两条测试钉住它：B 槽必须比 R 槽重（读 `usWeightClass`，不是假设），且 `font-weight: 600` 绝不允许出现在交给 PDF 引擎的样式表里。 - -有一处细节值得单写一段：在这个管线里，**medium 不是强调手段，而是一个字族成员**。中文标题用宋体的 medium 当常规、用 semibold 做强调；英文标题用展示衬线体做同样的事。这里搞错，屏幕上完全看不出来，而印出来一目了然。 - -## 规则四：不要按你控制不了输入的字集做子集化 - -第一版把中文字体按两个国家字集的并集做了子集，产物从约 50 MB 降到约 29 MB。然后我跑了姓名探针，发现这些字被裁掉了： - -```text -犇 骉 頔 祎 廸 玧 祾 赟 堃 -``` - -每一个都是真实的人名用字。报告会把客户姓名印在封面上，而姓名是自由输入——所以一套基于子集的构建，迟早会在产品最醒目的那一页上放一个豆腐块。全字形覆盖是约 31k 码位、约 50 MB 的嵌入字体；为了永远不给客户看一个空方块，这个价钱很便宜。 - -## 规则五：把「缺字」变成构建失败，而不是惊喜 - -闸门跑三项检查，任何一项不过就非零退出： - -- manifest 里声明的每个字体文件都在，且 sha256 与磁盘上的文件一致； -- 内容库（三语全部）+ 品牌字面 + 各类探针（简体、繁体、CJK 标点、全角、中英混排、数字、拉丁扩展、符号）+ 姓名用字，**每一个字在「它实际会被哪一档字体印出来」的那一档里都有字形**； -- 查字形用的是 PDF 引擎自己的解析器，而不是第三方库，所以这条检查不可能和渲染器互相矛盾。 - -已知边界写在明处，不藏起来：✅ ✔ 这类符号在这几款正文字体里都没有字形，而引擎也画不了彩色 emoji，所以它们不出现在报告正文；拉丁标题那款展示衬线体只有 974 个字形，不画 ① ★ ℃，这也是为什么内容库的检查覆盖**所有**槽位而不只是正文；而拉丁报告里的汉字必须走专门的 CJK 槽位——两条测试防止两个渲染器在这点上产生分歧。 - -## 如果重来一次 - -1. **在动样式之前先定字族与字重。** 版式做完之后再去加第二档字重，意味着每一页页数都要手工复核一遍。 -2. **选字体格式之前先读渲染器的源码。** 可变字体看着是审美问题，实际上是解析器问题——两个文件、十分钟就定了。 -3. **把要嵌进文档的一切当作许可证产物。** 字体文件连同许可证一起进仓库，否则就得进别人的法务审查。 -4. **量「到底嵌进去了什么」。** 页数、每个字面的字形数、文件体积、渲染时间：四项都动了，其中三项的方向和我猜的相反。 - -## 结果 - -改前：一款中文字体加一款界面字体、靠合成粗体、还取决于服务器装过什么。改后：四个字族、八个静态字面真正嵌在文件里，字重是各个字族真实的成员，页数与上一版相差不超过一页，而且有一条 lint 闸门：某个字没有字形时直接让构建失败——于是失败形态从「客户看到一个方块」变成了「CI 拒绝发版」。这跟这条管线里其它地方的取舍是同一个：**让错误发不出去，而不是让错误容易被看见。** -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Embedding Fonts So a PDF Looks the Same on Every Machine" -description: "Four families, eight faces, two renderers and one CSS quirk: what it took to stop a generated report depending on whatever fonts the server happened to have." -pubDate: 2025-01-22 -updatedDate: 2026-10-16 -category: engineering -tags: ["pdf", "fonts", "mpdf", "typography", "php"] -ogImage: /og/embedding-fonts-so-a-pdf-looks-the-same-everywhere.jpg -banner: /banners/embedding-fonts-so-a-pdf-looks-the-same-everywhere.jpg -draft: false ---- - -## The requirement was one sentence - -A report generator has to produce a document that looks the same everywhere: -Chinese headings in a serif, Chinese body text in a sans, Latin headings and -cover numerals in a display serif, Latin body text and tables in Inter. -Weights matter — medium for emphasis, semibold for headings — across three -languages and two PDF renderers. - -And the fonts must travel with the document. Not "the server has them -installed", not "the reader probably has them" — **embedded**, so a print -shop in another country sees the same pages. - -The old pipeline failed on the last clause in a way that was invisible: it -depended on fonts installed on the server by a package manager, and it only -ever embedded a single CJK face. The second weight did not exist, so "bold" -was the renderer's synthetic approximation, and the only way to notice was -to open the PDF and look at what was actually inside it. - -## Rule one: read the fonts back out of the finished PDF - -"Embedded" is a claim about the output, so I measured the output. The same -document, rendered before and after: - -| | before | after | -|---|---|---| -| zh page count | 23 | **23** | -| en page count | 28 | **29** | -| bm page count | 30 | **30** | -| fonts actually embedded | one CJK face + Roboto | Source Han Serif Medium + SemiBold · Source Han Sans Regular + Medium · Cormorant Garamond Medium + SemiBold · Inter Regular + SemiBold + Medium · Source Han Sans for the CJK runs inside Latin reports | -| render time / peak memory | baseline | same order (±10%, noise) | -| file size | baseline | roughly +3% | - -The glyph counts per face are the receipt: in the Chinese report, 15,988 -glyphs land in the body sans, 2,247 in its medium weight, 634 in the heading -serif medium, 222 in its semibold. That is the difference between "the font -is on the server" and "the font is in the file" — a reader in another -country sees the same four faces, and a page count that moved by exactly one -page in one language (the Latin metrics are wider than what they replaced) -is the honest cost. - -## Rule two: static instances only, and the reason is in the parser - -The obvious modern choice — ship one variable font per family — is wrong -here, for two measured reasons: - -1. **The PDF engine's font parser reads `glyf`/`loca` only.** Feed it an - OpenType file with PostScript/CFF outlines and it refuses: - `Fonts with postscript outlines are not supported`. A variable font is - not the problem; CFF outlines are. -2. **It does not read the `fvar` table at all**, so a variable font silently - resolves to its *default instance* — and the default instances of the CJK - families are the thinnest ones (Thin 100 and ExtraLight 200, visible in - the `name` table). Every page would render in a hairline weight, with no - error anywhere in the pipeline. - -So the build step takes the upstream variable fonts and **instantiates -static cuts**: it runs the instancer, then rewrites `name`, -`OS/2.usWeightClass` and `head.macStyle` so the file identifies itself as -the weight it now is (the source's own name still says "Thin"). A manifest -records family, weight class, sha256, glyph count and code-point count for -every file, and the licence files ship next to them. - -## Rule three: the engine speaks two weights, so map onto two slots - -The renderer's CSS handling honours exactly `normal` and `bold` — its source -says so in as many words ("Does not support: bolder, lighter, 100..900"). -Numeric weights are therefore meaningless to it, while the headless-Chromium -renderer in the same pipeline honours them properly. - -The mapping that works: each family declares an **R slot** and a **B slot** -(regular-ish and heavier), `font-weight: 500` lands on the R slot of the -intended family, and `600` is **rewritten to `bold`** on the PDF side only, -which lands on the B slot. Chromium keeps `600` as written. Two tests pin -it: the B slot must be heavier than the R slot (read from `usWeightClass`, -not assumed), and `font-weight: 600` must never appear in the stylesheet -handed to the PDF engine. - -One subtlety worth the paragraph: a "medium" weight is not an emphasis -mechanism in this pipeline, it is a *family member*. Chinese headings use -the serif's medium as their regular and its semibold for emphasis; Latin -headings do the same with the display serif. Getting that wrong is invisible -on a screen and obvious in print. - -## Rule four: do not subset against a character set you do not control - -The first attempt subset the CJK fonts to the union of two national -character sets and produced a ~29 MB bundle instead of ~50 MB. Then I ran -the name probe and found these were cut: - -```text -犇 骉 頔 祎 廸 玧 祾 赟 堃 -``` - -Every one of them is a real given name character. The report prints a -customer's name on the cover, and the name is free input — so a subset-based -build will eventually put a tofu box on the most prominent page of the -product. Full coverage is ~31k code points and roughly 50 MB of embedded -fonts; that is a fine price for never showing a customer an empty rectangle. - -## Rule five: make missing glyphs a build failure, not a surprise - -The gate runs three checks and exits non-zero on any of them: - -- every font file declared in the manifest is present, and its sha256 - matches the file on disk; -- every character in the content library (all three languages) plus the - brand strings, plus probe classes — simplified, traditional, CJK - punctuation, full-width forms, mixed-script, digits, Latin extended, - symbols — plus the name characters, has a glyph **in the face that will - actually print it**; -- the lookup uses the PDF engine's own font parser rather than a third-party - library, so the check cannot disagree with the renderer. - -The known edges are written down instead of hidden. Dingbats and emoji (✅ ✔) -exist in none of these text fonts and the engine cannot draw colour emoji, -so they are kept out of the report body. The display serif has only 974 -glyphs and does not draw symbols like ① ★ ℃, which is why the -content-library check covers *all* slots rather than just the body. And CJK -characters inside a Latin report must go through the dedicated CJK slot — -two tests keep the renderers from disagreeing about that. - -## What I would do differently - -1. **Choose faces and weights before styling anything.** Retrofitting a - second weight after the layout exists means re-checking every page count - by hand. -2. **Read the renderer's source before choosing a font format.** The - variable-font decision looked like a taste question and was actually a - parser question — two files settled it in ten minutes. -3. **Treat everything you ship inside a document as a licence artifact.** - Font files enter the repo with their licences, or they enter someone - else's legal review. -4. **Measure what got embedded.** Page count, glyph-per-face counts, file - size, render time: all four moved, and three of them moved in ways I - would have guessed wrong. - -## The result - -Before: one CJK face and a UI font, with synthetic bold, depending on what -the server had installed. After: four families and eight static faces -embedded in the file, weights that are real members of their families, page -counts within one page of the previous layout, and a lint gate that fails -the build when a character has no glyph — so the failure mode moved from "a -customer sees a box" to "CI refuses the release". Which is the same trade as -everywhere else in this pipeline: make the mistake impossible to ship rather -than easy to notice. -```` - ---- - -## 文章 07 / 11：那个把静默丢失更新变成 409 的 JPA 字段：@Version 乐观锁 -- **Slug**: `jpa-version-field-lost-update` -- **英文标题**: The JPA Field That Turns a Silent Lost Update Into a 409 -- **分类**: `engineering` | **发布日期**: 2026-08-19 -- **线上地址**: https://blog.hoelee.com/posts/jpa-version-field-lost-update/ (EN) | https://blog.hoelee.com/posts/zh/jpa-version-field-lost-update/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "那个把静默丢失更新变成 409 的 JPA 字段：@Version 乐观锁" -description: "在 REST + JPA 的读-改-写流程里，两个编辑同时保存同一条记录时，后保存者会静默覆盖先保存者的修改，而 API 两次都返回成功——JPA 的 @Version 乐观锁把这个静默丢失更新变成 HTTP 409 冲突：请求携带期望版本号，版本不符就拒绝写入，并提示客户端重新读取再试。" -pubDate: 2026-08-19 -category: engineering -tags: [java, spring, jpa, hibernate, rest, testing] -ogImage: /og/zh-jpa-version-field-lost-update.jpg -banner: /banners/jpa-version-field-lost-update.jpg -draft: false ---- - -两个人同时打开同一篇文章，两个人都改了，两个人都保存。其中一次 -保存悄无声息地消失了——没有报错，没有警告，而且 API 对两次请求 -都返回了 HTTP 200。 - -这就是丢失更新（lost update）问题，是读-改-写（read-modify-write） -API 里最安静的数据丢失 bug。我在做 Spring Boot 作品集 demo 时撞上 -了它——这是一个 Spring Boot 3.5.16（Java 21）的 REST API，编辑器 -从浏览器页面创建和更新文章。修法最终落在三处：JPA 实体上的一个 -`@Version` 字段、请求契约里的一个字段、以及 API 层把它映射成 -HTTP 409 Conflict 的一个异常。这篇文章就沿着真实代码走一遍这条路。 - -## 问题：一条编辑消失了，却谁都怪不上 - -为什么两个人保存同一条记录时，其中一个人的修改会消失？ - -跟着时间线走。编辑器 A 和编辑器 B 都拉取了同一篇文章。A 先保存： -行被更新，API 返回 200。B 稍后保存：行被*再次*更新，这次覆盖了 -A 的文字，API 又返回 200。每个请求都成功；每个响应都告诉它的调用 -方「你的保存就是这条记录的当前状态」——对其中一个人来说这是假话。 -A 的编辑就这么没了，两个客户端都没有任何办法知道。 - -失败是静默的，因为它就是读-改-写流程的正常行为，不是异常。这正是 -它在真实系统里存活的理由，也是为什么在要展示工程判断力的东西里， -值得把它修得干净。 - -## 我先试的方案：不带版本号的读-改-写，以及它为什么失败 - -我对更新流程的第一版草图是最直白的那种：`GET` 记录、编辑、 -`PUT` 回去，只带新的标题和正文，什么都不带。 - -```text -GET /api/posts/{id} # 读取记录 -PUT /api/posts/{id} # 写回新的标题和正文 -``` - -服务端加载行、套用更新、返回 200。它无从知道在这一行数据上，客户端 -在 `GET` 和 `PUT` 之间已经被别人动过——读取和写入是两个互不相干的 -请求，契约里没有任何东西把它们连起来。最后写入的人赢，而两个写入 -者都被告知成功。 - -在这里，200 比报错更糟。报错至少告诉输掉的那个客户端「出事了」， -给他们一个去查的理由。200 则告诉两个客户端「你们的保存就是记录 -当前的状态」——对其中一个是撒谎。输掉的那次编辑现在哪儿都不存在： -屏幕上没有，数据库里也没有；而因为客户端相信自己保存成功了， -没有人会去找这份数据。 - -## 修法：一个 @Version 字段、一份契约、一个冲突 - -修法一共五小步，全部都在 demo 的源码里。 - -### 1. 实体随身携带一个版本列 - -```java -@Entity -@Table(name = "posts", indexes = @Index(name = "idx_posts_title", columnList = "title")) -@EntityListeners(AuditingEntityListener.class) -public class Post { - - @Id - @GeneratedValue - @UuidGenerator - private UUID id; - - @Column(nullable = false, length = 160) - private String title; - - @Column(nullable = false, length = 10_000) - private String body; - - @Version - private long version; - // ... -} -``` - -`@Version` 告诉 Hibernate 把 `version` 当作乐观锁。它生成的每一条 -`UPDATE` 都变成有条件的： - -```sql -UPDATE posts -SET title = ?, body = ?, version = version + 1, ... -WHERE id = ? AND version = ? -``` - -如果这行数据的版本号已经和语句构造时的不一致，受影响行数为零， -Hibernate 就会拒绝这次 flush，而不是悄悄覆盖。实体从不用手自增 -计数器——更新方法只碰 `title` 和 `body`；版本号跟着写入一起走。 - -### 2. 版本号进入 API 契约 - -锁如果客户端无法参与，就没有用，所以版本号在两个方向上都穿过 -API。响应的 record 暴露它： - -```java -public record PostResponse(UUID id, long authorId, String title, String body, - long version, Instant createdAt, Instant updatedAt) { -} -``` - -更新请求则要求把它带回来： - -```java -public record UpdatePostRequest( - @NotBlank @Size(max = 160) String title, - @NotBlank @Size(max = 10_000) String body, - @NotNull @Min(0) Long version) { -} -``` - -`UpdatePostRequest` 旁边的学习注记说得很直白： -「the expected version is part of the update -contract, making optimistic locking visible to clients.」 -（期望的版本号是更新契约的一部分，让乐观锁对客户端可见。） - -### 3. 在拥有写事务的 service 里检查版本 - -```java -@CachePut(cacheNames = "posts", key = "#postId") -@Transactional -public PostResponse updatePost(UUID postId, UpdatePostRequest request) { - Post post = postRepository.findById(postId) - .orElseThrow(() -> new PostNotFoundException(postId)); - if (post.getVersion() != request.version()) { - throw new PostVersionConflictException(postId); +## 60 首长歌全集在线鉴赏 + +以下 60 首作品均已上传至 Publitio 媒体存储，支持 HTTP 206 自由拖拽定位。封面全量采用 1024×1024 独立生成的无字 WebP 艺术插画，并内嵌同步 LRC 歌词。 + +*操作提示：可直接利用上方导航面板进行分类筛选，或从下拉菜单中快速直达指定曲目。* + +
+
+ The Thimble Under the Moss cover art +
+
+
+ 01. The Thimble Under the Moss + MP3 + 3分11秒 (191.1s) · 凯尔特竖琴民谣 · 清脆稚气女童声（类似 Cécile Corbel《Arrietty's Song》）｜中速轻快（约 105 BPM） +
+ + +
+
+ +
+
+ Mist of Skellig Michael cover art +
+
+
+ 02. Mist of Skellig Michael + MP3 + 4分26秒 (266.2s) · 凯尔特竖琴民谣 · 空灵圣洁的小女孩独唱（类似 Enya + 幼态音色）｜慢速悠长（约 72 BPM，字少音长） +
+ + +
+
+ +
+
+ The Rose of Connemara cover art +
+
+
+ 03. The Rose of Connemara + MP3 + 4分43秒 (283.1s) · 凯尔特竖琴民谣 · 纯净稚气少女音（类似 Cara Dillon 早期清亮少女感）｜中慢速叙事（约 82 BPM） +
+ + +
+
+ +
+
+ Lanterns on the Moor cover art +
+
+
+ 04. Lanterns on the Moor + MP3 + 3分31秒 (211.2s) · 凯尔特竖琴民谣 · 俏皮精灵般的女童声（类似 Cécile Corbel 欢快舞曲风）｜快速三拍子（约 128 BPM） +
+ + +
+
+ +
+
+ The Weaver of Raglan Lane cover art +
+
+
+ 05. The Weaver of Raglan Lane + MP3 + 4分39秒 (279.3s) · 凯尔特竖琴民谣 · 醇厚温暖的凯尔特民谣男声（类似 Luke Kelly，调剂男声）｜中慢速（约 80 BPM） +
+ + +
+
+ +
+
+ 硝子の花束 cover art +
+
+
+ 06. 硝子の花束 + MP3 + 3分49秒 (230.0s) · 日系治愈动漫 · 稚气通透、略带脆弱感的动漫少女音（祈妹风）｜中慢速钢琴弦乐（约 78 BPM） +
+ + +
+
+ +
+
+ 銀の時計塔と宵の歌 cover art +
+
+
+ 07. 銀の時計塔と宵の歌 + MP3 + 4分34秒 (274.8s) · 日系治愈动漫 · 清亮圣洁少女音 + 梶浦语吟唱｜中速华尔兹三拍子（约 110 BPM） +
+ + +
+
+ +
+
+ 森の郵便屋さん cover art +
+
+
+ 08. 森の郵便屋さん + MP3 + 3分44秒 (225.0s) · 日系治愈动漫 · 极度轻柔、如耳语般的稚气女童声｜慢速木吉他与竖琴（约 68 BPM） +
+ + +
+
+ +
+
+ 白鳥の祈り cover art +
+
+
+ 09. 白鳥の祈り + MP3 + 4分06秒 (246.3s) · 日系治愈动漫 · 空灵通透的高音少女/女童声，尾音带空灵共鸣｜慢速大气（约 70 BPM） +
+ + +
+
+ +
+
+ 星屑のオルゴール cover art +
+
+
+ 10. 星屑のオルゴール + MP3 + 3分31秒 (211.8s) · 日系治愈动漫 · 清甜稚气的少女声｜中慢速轻盈感（约 88 BPM） +
+ + +
+
+ +
+
+ ひまわり行進曲 cover art +
+
+
+ 11. ひまわり行進曲 + MP3 + 3分18秒 (198.0s) · 日系元气童趣 · 元气满满的小女孩独唱/齐唱感｜中快速欢快节奏（约 118 BPM） +
+ + +
+
+ +
+
+ プリンは世界を救うのだ！ cover art +
+
+
+ 12. プリンは世界を救うのだ！ + MP3 + 3分23秒 (203.4s) · 日系元气童趣 · 软萌夸张的稚气萝莉音（高音、语速快、字密）｜快速电音流行（约 145 BPM） +
+ + +
+
+ +
+
+ こねこ探偵団のテーマ cover art +
+
+
+ 13. こねこ探偵団のテーマ + MP3 + 2分16秒 (136.1s) · 日系元气童趣 · 奶声奶气的女童声（6~8岁童音感）｜中速轻快爵士童谣（约 115 BPM） +
+ + +
+
+ +
+
+ 夏祭りのラムネ星 cover art +
+
+
+ 14. 夏祭りのラムネ星 + MP3 + 3分14秒 (195.0s) · 日系元气童趣 · 清亮少年男声 + 稚气少女声（调剂少年音）｜中快速夏日流行（约 122 BPM） +
+ + +
+
+ +
+
+ ほうきに乗って宅配便 cover art +
+
+
+ 15. ほうきに乗って宅配便 + MP3 + 3分16秒 (196.1s) · 日系元气童趣 · 清脆阳光的小女孩声音｜中速轻快复古流行（约 116 BPM） +
+ + +
+
+ +
+
+ 白露·竹间灯 cover art +
+
+
+ 16. 白露·竹间灯 + MP3 + 3分42秒 (222.6s) · 中文国风古意 · 清甜稚气的少女/女童声｜中速轻快电子国风（约 108 BPM） +
+ + +
+
+ +
+
+ 小鲤跃龙门 cover art +
+
+
+ 17. 小鲤跃龙门 + MP3 + 2分09秒 (129.9s) · 中文国风古意 · 活泼软萌的稚气少女音（语速偏快、吐字灵动）｜中快速（约 120 BPM） +
+ + +
+
+ +
+
+ 云海归鲲 cover art +
+
+
+ 18. 云海归鲲 + MP3 + 3分56秒 (236.9s) · 中文国风古意 · 空灵仙气的少年假声/美声男声（调剂男声）｜慢速悠远（约 68 BPM） +
+ + +
+
+ +
+
+ 纸伞巷 cover art +
+
+
+ 19. 纸伞巷 + MP3 + 3分34秒 (214.6s) · 中文国风古意 · 清亮温婉的少女声（带一点点江南小调韵味）｜中慢速（约 80 BPM） +
+ + +
+
+ +
+
+ 山间小道士 cover art +
+
+
+ 20. 山间小道士 + MP3 + 3分07秒 (187.6s) · 中文国风古意 · 纯真稚气的小女孩/少女声音｜中速轻快民谣古风（约 105 BPM） +
+ + +
+
+ +
+
+ 橘子汽水的夏天 cover art +
+
+
+ 21. 橘子汽水的夏天 + MP3 + 2分48秒 (169.0s) · 温暖民谣治愈 · 清澈稚气的少女声｜中慢速木吉他民谣（约 82 BPM） +
+ + +
+
+ +
+
+ 外婆的小院 cover art +
+
+
+ 22. 外婆的小院 + MP3 + 3分32秒 (212.4s) · 温暖民谣治愈 · 纯真软糯的小女孩声音｜慢速温情（约 72 BPM） +
+ + +
+
+ +
+
+ 开往北方的绿皮车 cover art +
+
+
+ 23. 开往北方的绿皮车 + MP3 + 3分36秒 (216.3s) · 温暖民谣治愈 · 低沉温暖的民谣男声（调剂男声）｜慢速叙事（约 75 BPM） +
+ + +
+
+ +
+
+ 写给流浪猫的信 cover art +
+
+
+ 24. 写给流浪猫的信 + MP3 + 2分09秒 (129.9s) · 温暖民谣治愈 · 俏皮温柔的稚气少女音｜中速轻快尤克里里/木吉他（约 98 BPM） +
+ + +
+
+ +
+
+ 平凡的傍晚 cover art +
+
+
+ 25. 平凡的傍晚 + MP3 + 3分41秒 (221.0s) · 温暖民谣治愈 · 温柔治愈的青年男声（调剂男声）｜慢速钢琴与木吉他（约 74 BPM） +
+ + +
+
+ +
+
+ Light of the Eventide cover art +
+
+
+ 26. Light of the Eventide + MP3 + 4分09秒 (249.0s) · 空灵天籁合唱 · 天籁女童高音领唱 + 纯净童声合唱｜极慢速庄严（约 66 BPM） +
+ + +
+
+ +
+
+ Bells of the Winter Star cover art +
+
+
+ 27. Bells of the Winter Star + MP3 + 2分47秒 (167.6s) · 空灵天籁合唱 · 清脆女童多声部齐唱與轮唱｜中快速灵动节奏（约 125 BPM） +
+ + +
+
+ +
+
+ Paper Planes in the Wind cover art +
+
+
+ 28. Paper Planes in the Wind + MP3 + 3分25秒 (205.5s) · 空灵天籁合唱 · 清亮纯真的女童领唱 + 学院风童声合唱｜中速温暖（约 96 BPM） +
+ + +
+
+ +
+
+ Lullaby of the Little Star cover art +
+
+
+ 29. Lullaby of the Little Star + MP3 + 3分47秒 (227.8s) · 空灵天籁合唱 · 稚气纯净的小女孩独唱（突出童真独唱感）｜慢速温柔（约 70 BPM） +
+ + +
+
+ +
+
+ Hymn of the Morning Meadow cover art +
+
+
+ 30. Hymn of the Morning Meadow + MP3 + 4分19秒 (259.9s) · 空灵天籁合唱 · 神圣清澈的女童领唱 + 宏大童声和声（帕赫贝尔卡农式递进）｜慢速治愈（约 72 BPM） +
+ + +
+
+ +
+
+ 地板下的借物小人 cover art +
+
+
+ 31. 地板下的借物小人 + MP3 + 3分07秒 (187.7s) · 凯尔特童话民谣 (中文) · 清脆稚气女童声｜中速轻快凯尔特竖琴与锡哨（约 105 BPM） +
+ + +
+
+ +
+
+ 迷雾孤岛的竖琴 cover art +
+
+
+ 32. 迷雾孤岛的竖琴 + MP3 + 3分24秒 (204.4s) · 凯尔特童话民谣 (中文) · 空灵圣洁的小女孩独唱｜慢速悠长（约 70 BPM，字少音长） +
+ + +
+
+ +
+
+ 石楠花开的高地 cover art +
+
+
+ 33. 石楠花开的高地 + MP3 + 3分36秒 (216.9s) · 凯尔特童话民谣 (中文) · 纯净稚气少女音｜中慢速木吉他与凯尔特小提琴（约 82 BPM） +
+ + +
+
+ +
+
+ 橡树下的萤火舞会 cover art +
+
+
+ 34. 橡树下的萤火舞会 + MP3 + 3分43秒 (223.1s) · 凯尔特童话民谣 (中文) · 俏皮精灵般的女童声（语速快、节奏感强）｜快速三拍子（约 128 BPM） +
+ + +
+
+ +
+
+ 老航海家的绿宝石 cover art +
+
+
+ 35. 老航海家的绿宝石 + MP3 + 3分49秒 (229.8s) · 凯尔特童话民谣 (中文) · 醇厚温暖的凯尔特民谣男声（调剂男声）｜中慢速（约 80 BPM） +
+ + +
+
+ +
+
+ 玻璃温室的告别 cover art +
+
+
+ 36. 玻璃温室的告别 + MP3 + 3分37秒 (217.0s) · 日系治愈物语 (中文) · 稚气通透、略带脆弱感的动漫少女音｜中慢速钢琴与弦乐（约 78 BPM） +
+ + +
+
+ +
+
+ 银色钟塔与暮光咏叹 cover art +
+
+
+ 37. 银色钟塔与暮光咏叹 + MP3 + 4分01秒 (241.4s) · 日系治愈物语 (中文) · 清亮圣洁少女音 + 造语吟唱｜中速古典哥特华尔兹（约 110 BPM） +
+ + +
+
+ +
+
+ 森林邮差的蒲公英 cover art +
+
+
+ 38. 森林邮差的蒲公英 + MP3 + 3分26秒 (206.7s) · 日系治愈物语 (中文) · 极度轻柔、如耳语般的稚气女童声｜慢速木吉他与钢片琴（约 68 BPM） +
+ + +
+
+ +
+
+ 白鸟与雪原的祈愿 cover art +
+
+
+ 39. 白鸟与雪原的祈愿 + MP3 + 3分39秒 (220.0s) · 日系治愈物语 (中文) · 空灵通透的高音女童/少女声｜慢速大气钢琴与管弦（约 70 BPM） +
+ + +
+
+ +
+
+ 发条星尘八音盒 cover art +
+
+
+ 40. 发条星尘八音盒 + MP3 + 3分18秒 (198.2s) · 日系治愈物语 (中文) · 清甜稚气的少女声｜中慢速八音盒与木吉他（约 88 BPM） +
+ + +
+
+ +
+
+ 向日葵小队出发 cover art +
+
+
+ 41. 向日葵小队出发 + MP3 + 2分39秒 (159.4s) · 元气童趣流行 (中文) · 元气满满的小女孩领唱 + 童声合唱感｜中快速欢快节奏（约 118 BPM） +
+ + +
+
+ +
+
+ 焦糖布丁保卫战！ cover art +
+
+
+ 42. 焦糖布丁保卫战！ + MP3 + 3分38秒 (218.9s) · 元气童趣流行 (中文) · 软萌夸张的稚气萝莉音（高音、语速快、字密）｜快速电音流行（约 145 BPM） +
+ + +
+
+ +
+
+ 猫咪侦探事务所 cover art +
+
+
+ 43. 猫咪侦探事务所 + MP3 + 2分31秒 (151.4s) · 元气童趣流行 (中文) · 奶声奶气的女童声（6～8岁童音感）｜中速轻快木琴与铜管（约 115 BPM） +
+ + +
+
+ +
+
+ 夏夜烟火与波子汽水 cover art +
+
+
+ 44. 夏夜烟火与波子汽水 + MP3 + 3分07秒 (187.8s) · 元气童趣流行 (中文) · 清亮少年男声 + 稚气少女声（调剂少年音）｜中快速夏日流行（约 122 BPM） +
+ + +
+
+ +
+
+ 骑上扫帚送快递 cover art +
+
+
+ 45. 骑上扫帚送快递 + MP3 + 3分08秒 (188.5s) · 元气童趣流行 (中文) · 清脆阳光的小女孩声音｜中速轻快木吉他与手鼓（约 116 BPM） +
+ + +
+
+ +
+
+ 小满·青梅煮雨 cover art +
+
+
+ 46. 小满·青梅煮雨 + MP3 + 3分34秒 (214.9s) · 国风新篇古韵 (中文) · 清甜稚气的少女/女童声｜中速轻快电子国风（约 106 BPM） +
+ + +
+
+ +
+
+ 糖画摊前的小狐仙 cover art +
+
+
+ 47. 糖画摊前的小狐仙 + MP3 + 3分12秒 (192.2s) · 国风新篇古韵 (中文) · 活泼软萌的稚气少女音（字密语速快）｜中快速（约 124 BPM） +
+ + +
+
+ +
+
+ 鲛人泪·沧海月 cover art +
+
+
+ 48. 鲛人泪·沧海月 + MP3 + 3分59秒 (239.8s) · 国风新篇古韵 (中文) · 空灵仙气的少年假声/美声男声（调剂男声）｜慢速深邃（约 68 BPM） +
+ + +
+
+ +
+
+ 兰亭旧墨 cover art +
+
+
+ 49. 兰亭旧墨 + MP3 + 3分49秒 (229.3s) · 国风新篇古韵 (中文) · 清亮温婉的少女声｜中慢速古筝与二胡（约 80 BPM） +
+ + +
+
+ +
+
+ 采药小童下山记 cover art +
+
+
+ 50. 采药小童下山记 + MP3 + 3分24秒 (204.4s) · 国风新篇古韵 (中文) · 纯真稚气的小女孩声音｜中速轻快木吉他与竹笛（约 104 BPM） +
+ + +
+
+ +
+
+ 晒太阳的旧毛衣 cover art +
+
+
+ 51. 晒太阳的旧毛衣 + MP3 + 3分04秒 (184.9s) · 温暖民谣新篇 (中文) · 清澈稚气的少女声｜中慢速木吉他民谣（约 80 BPM） +
+ + +
+
+ +
+
+ 奶奶的桂花糕 cover art +
+
+
+ 52. 奶奶的桂花糕 + MP3 + 3分42秒 (222.4s) · 温暖民谣新篇 (中文) · 纯真软糯的小女孩声音｜慢速温情口琴与吉他（约 72 BPM） +
+ + +
+
+ +
+
+ 北方的末班渡轮 cover art +
+
+
+ 53. 北方的末班渡轮 + MP3 + 3分49秒 (229.6s) · 温暖民谣新篇 (中文) · 低沉温暖的民谣男声（调剂男声）｜慢速深沉（约 74 BPM） +
+ + +
+
+ +
+
+ 云朵收集指南 cover art +
+
+
+ 54. 云朵收集指南 + MP3 + 2分56秒 (176.4s) · 温暖民谣新篇 (中文) · 俏皮温柔的稚气少女音｜中速轻快（约 98 BPM） +
+ + +
+
+ +
+
+ 街角二十四小时书店 cover art +
+
+
+ 55. 街角二十四小时书店 + MP3 + 4分13秒 (253.4s) · 温暖民谣新篇 (中文) · 温柔治愈的青年男声（调剂男声）｜慢速钢琴与木吉他（约 74 BPM） +
+ + +
+
+ +
+
+ 暮光归途 cover art +
+
+
+ 56. 暮光归途 + MP3 + 4分02秒 (242.6s) · 天籁童声圣咏 (中文) · 天籁女童高音领唱 + 纯净童声合唱｜极慢速庄严管风琴与弦乐（约 66 BPM） +
+ + +
+
+ +
+
+ 冬夜银铃 cover art +
+
+
+ 57. 冬夜银铃 + MP3 + 2分59秒 (179.0s) · 天籁童声圣咏 (中文) · 清脆女童多声部齐唱与轮唱｜中快速灵动三拍子（约 125 BPM） +
+ + +
+
+ +
+
+ 迎风飞翔的纸飞机 cover art +
+
+
+ 58. 迎风飞翔的纸飞机 + MP3 + 3分53秒 (233.2s) · 天籁童声圣咏 (中文) · 清亮纯真的女童领唱 + 学院风童声合唱｜中速温暖（约 96 BPM） +
+ + +
+
+ +
+
+ 小星星的晚安曲 cover art +
+
+
+ 59. 小星星的晚安曲 + MP3 + 3分51秒 (232.0s) · 天籁童声圣咏 (中文) · 稚气纯净的小女孩独唱｜慢速竖琴与小提琴（约 70 BPM） +
+ + +
+
+ +
+
+ 晨曦草地的赞美诗 cover art +
+
+
+ 60. 晨曦草地的赞美诗 + MP3 + 4分26秒 (266.0s) · 天籁童声圣咏 (中文) · 神圣清澈的女童领唱 + 宏大童声和声（卡农式递进）｜慢速治愈（约 72 BPM） +
+ + +
+
+ + -### 4. 异常处理器把它映射成 409 - -异常本身带着客户端能直接照做的信息： - -```java -public class PostVersionConflictException extends RuntimeException { - - public PostVersionConflictException(UUID postId) { - super("Post %s has changed; fetch it again before retrying".formatted(postId)); - } -} -``` - -错误边界把它转成正确的 HTTP 应答： - -```java -@ExceptionHandler(PostVersionConflictException.class) -ProblemDetail handleConflict(PostVersionConflictException exception) { - return problem(HttpStatus.CONFLICT, "POST_VERSION_CONFLICT", exception.getMessage()); -} -``` - -`HttpStatus.CONFLICT` 就是 HTTP 409，响应是一份 problem -document——一个 `type` URI、一个稳定的机器码 -（`POST_VERSION_CONFLICT`）和一段人类可读的说明。demo 在配置里 -开启了框架的 problem-details 支持 -（`spring.mvc.problemdetails.enabled: true`），所以校验错误和所有 -其他错误都共用同一种响应形状。 - -### 5. 整条链路，从头到尾 - -```text -过期的 PUT /api/posts/{id} - -> PostService.updatePost 在事务内部重新读取这一行 - -> post.getVersion() != request.version() - -> PostVersionConflictException: "fetch it again before retrying" - -> ApiExceptionHandler.handleConflict - -> HTTP 409, ProblemDetail with code POST_VERSION_CONFLICT - -> 客户端知道自己的前提过期了 -``` - -一个 `@Version` 字段、请求契约里的一个 `version`、一个异常、一个 -handler 方法。Controller 什么都没改——`updatePost` 看起来还是普通 -的 `PUT`。 - -## 为什么答案是冲突，而不是重试 - -当两个编辑器产出同一份记录的两种不同版本时，服务端无法知道哪一方 -的意图才是被需要的。从各自作者的角度看，两次保存都正当；重放那条 -过期的写入，只是重放同一次数据丢失。 - -诚实的结论就是 HTTP 对 409 的定义：请求与资源的当前状态冲突， -并且客户端被明确告知如何解决——「fetch it again before retrying」 -（重新取一次再试）。输掉的客户端重新读取记录，看到另一位作者的 -修改，由人来决定保留什么。静默覆盖变成了一处看得见的决策点——这 -正是这件事的全部意义。 - -## 这个 demo 的其余部分证明了什么 - -版本字段只是一个小系统里的一层，而其余部分都能用同样的方式核实 -——下面每一行都是仓库里的一个类或配置文件，不是一句口号： - -| 层 | 做了什么 | 在哪 | -|---|---|---| -| 缓存 | 单篇文章读取使用有界 Caffeine 缓存（上限 500，TTL 10 分钟，都来自配置）；读取用 `@Cacheable`、更新用 `@CachePut`、删除用 `@CacheEvict`；`GET /api/showcase/cache` 暴露请求数、命中数、未命中数和命中率 | `CacheConfig`、`CacheProperties`、`ShowcaseMetricsController` | -| 安全 | 无状态 HTTP Basic + BCrypt；页面与读 API 公开，所有写操作需要 `EDITOR` 角色；本地默认是专用的非机密 `demo-editor`/`changeit` 账户，可用环境变量覆盖 | `SecurityConfig`、`application.yml` | -| 错误 | 请求 record 在边界处校验；一个 `@RestControllerAdvice` 返回一致的 problem document 和机器码——`POST_NOT_FOUND`（404）、`POST_VERSION_CONFLICT`（409）、带字段错误表的 `VALIDATION_FAILED`（400） | `CreatePostRequest`、`UpdatePostRequest`、`ApiExceptionHandler` | -| 配置 | 本地 profile 使用内存 H2 数据库（PostgreSQL 模式），`ddl-auto: update` 并预置示例文章；`prod` 选择 PostgreSQL，所有凭据来自 `APP_*` 环境变量，`ddl-auto: validate` | `application.yml`、`application-prod.yml` | -| 测试 | 一个 `@SpringBootTest` 套件断言：渲染的 Thymeleaf 页面、未认证 401 对编辑器 201、`VALIDATION_FAILED` 问题响应、公开搜索，以及重复读取后的缓存命中 | `DemoApplicationIntegrationTest` | - -## 如果做真东西，我会怎么改 - -一旦这不再是本地 demo，有两件事立刻要变。项目 README 里两件都写了， -值得作为我自己的判断再重复一遍。 - -第一，schema 管理。我会在任何时候把 `ddl-auto` 设为 `validate` 之前， -先加 Flyway 或 Liquibase 迁移。demo 默认 profile 用的是 `update`， -在临时数据库上很方便，养成习惯就很危险——生产 schema 应该是 -有版本管理的代码，而不是启动时的副作用。 - -第二，密钥。本地编辑器账户是刻意的非机密 demo 账户，带默认凭据。 -做真东西的话，我会把凭据放进托管的密钥存储（managed secret store）， -拿不到凭据就拒绝启动，而不是回退到默认值。`prod` profile 已经做到 -所有凭据来自环境变量、不提交任何密钥——这正是我想要的形态， -只是不要那些回退值。 - -## 结果 - -过期的更新现在返回 HTTP 409，机器码 `POST_VERSION_CONFLICT`，而不是 -一个静静丢掉另一位作者修改的 200。丢失不再可能悄悄发生：失败是 -响亮的、可操作的——「fetch it again before retrying」——并且集成 -测试套件端到端地断言了访问与校验行为。 - -整个修法就是一个字段上的一个注解，加上让版本号成为契约一部分的 -管道代码。如果一条记录可能同时被两个人读取，这个字段就是「编辑 -丢失」与「双方都看得见的冲突」之间的分界线。 - -我平时就做 Java/Spring 后端和全栈 Web 应用——从浏览器页面到数据库 -的 REST API：JPA、乐观锁、缓存、校验和一整套集成测试。如果你的表单 -或字段里也有「编辑悄悄消失」的问题，跟我说说： -[WhatsApp](https://wa.me/60127972969) · -[me@hoelee.com](mailto:me@hoelee.com?subject=Spring%20Boot%20help) · -[hoelee.com](https://hoelee.com)。 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "The JPA Field That Turns a Silent Lost Update Into a 409" -description: "How a JPA @Version field turns a silent lost update into a 409 conflict — optimistic locking makes a stale REST write fail loudly instead of overwriting." -pubDate: 2026-08-19 -category: engineering -tags: [java, spring, jpa, hibernate, rest, testing] -ogImage: /og/jpa-version-field-lost-update.jpg -banner: /banners/jpa-version-field-lost-update.jpg -draft: false --- -Two people open the same post. Both edit it. Both save. One of the saves -disappears without a trace — no error, no warning, and the API answered -HTTP 200 for both of them. +## 需要为你的业务构建自动化 AI 媒体流水线？ -That is the lost update problem, and it is the quietest data-loss bug in -read-modify-write APIs. I hit it while building my Spring Boot portfolio -demo — a Spring Boot 3.5.16 (Java 21) REST API where editors create and -update posts from a browser page. The fix turned out to be one `@Version` -field on the JPA entity, one field on the request contract, and one -exception that the API layer maps to HTTP 409 Conflict. This post traces -that path through the actual code. +我为马来西亚及海外中小企业提供定制化自托管 AI 系统部署、自动化音视频内容生产管线以及全栈网站开发服务。 -## The problem: an edit that vanishes with no one to blame - -Why does one editor's change disappear when two people save the same -record? - -Follow the timeline. Editor A and editor B both fetch the same post. -A saves first: the row is updated and the API answers 200. B saves a -moment later: the row is updated *again*, this time over A's text, and -the API answers 200 again. Every request succeeded; every response told -its caller that their save is the state of the record. A's edit is -simply gone, and neither client has any way to know. - -The failure is silent because it is normal read-modify-write behaviour, -not an anomaly. That is exactly why the bug survives in real systems, and -why it is worth fixing properly in something meant to demonstrate -engineering judgement. - -## What I tried first, and why it failed - -My first sketch of the update flow was the obvious one: `GET` the record, -edit it, `PUT` it back with the new title and body and nothing else. - -```text -GET /api/posts/{id} # read the record -PUT /api/posts/{id} # write it back with a new title and body -``` - -The server loads the row, applies the update, and returns 200. It has no -way to notice that the row moved underneath the client between the `GET` -and the `PUT` — the read and the write are two unrelated requests, and -nothing in the contract connects them. The last write wins, and both -writers are told they succeeded. - -A 200 is worse than an error here. An error at least tells the losing -client that something happened and gives them a reason to look. A 200 -tells both clients that their save is the current state of the record, -which is a lie for one of them. The losing edit now exists nowhere — not -on screen, not in the database — and because the client believes it -saved, nobody goes looking for the data. - -## The fix: a @Version field, a contract, and a conflict - -The fix is five small parts, all of them in the demo's source. - -### 1. The entity carries a version column - -```java -@Entity -@Table(name = "posts", indexes = @Index(name = "idx_posts_title", columnList = "title")) -@EntityListeners(AuditingEntityListener.class) -public class Post { - - @Id - @GeneratedValue - @UuidGenerator - private UUID id; - - @Column(nullable = false, length = 160) - private String title; - - @Column(nullable = false, length = 10_000) - private String body; - - @Version - private long version; - // ... -} -``` - -`@Version` tells Hibernate to treat `version` as an optimistic lock. Every -`UPDATE` it generates becomes conditional: - -```sql -UPDATE posts -SET title = ?, body = ?, version = version + 1, ... -WHERE id = ? AND version = ? -``` - -If the row's version no longer matches the value the statement was built -with, zero rows change and Hibernate rejects the flush instead of quietly -overwriting. The entity never bumps the counter by hand — the update -method only touches `title` and `body`; the version moves with the write -itself. - -### 2. The version is part of the API contract - -A lock is useless if the client cannot take part in it, so the version -travels through the API in both directions. The response record exposes -it: - -```java -public record PostResponse(UUID id, long authorId, String title, String body, - long version, Instant createdAt, Instant updatedAt) { -} -``` - -and the update request requires it back: - -```java -public record UpdatePostRequest( - @NotBlank @Size(max = 160) String title, - @NotBlank @Size(max = 10_000) String body, - @NotNull @Min(0) Long version) { -} -``` - -The learning note next to `UpdatePostRequest` says it plainly: "the -expected version is part of the update contract, making optimistic -locking visible to clients." - -### 3. The service checks it inside the transaction that owns the write - -```java -@CachePut(cacheNames = "posts", key = "#postId") -@Transactional -public PostResponse updatePost(UUID postId, UpdatePostRequest request) { - Post post = postRepository.findById(postId) - .orElseThrow(() -> new PostNotFoundException(postId)); - if (post.getVersion() != request.version()) { - throw new PostVersionConflictException(postId); - } - post.update(request.title().trim(), request.body().trim()); - return PostResponse.from(postRepository.saveAndFlush(post)); -} -``` - -The load, the comparison and the save all happen inside one -`@Transactional` method, and that is the part that makes the check -honest: the version is compared against the row as it stands *at write -time*, not against a snapshot from an earlier request. If the read and -the write lived in different transactions, the check would compare -against a version the row has already left behind, and the whole scheme -would be theatre. The learning note on `PostService` names the rule: -"service methods centralize transaction boundaries, cache coherence, and -optimistic-locking rules." The demo also runs with -`spring.jpa.open-in-view: false`, so no lingering session serves a stale -entity across requests — the row is re-read inside the writing -transaction. - -The explicit check catches stale requests deterministically. The -`@Version` column still matters underneath: if another commit slips in -between the check and the flush, the conditional `UPDATE` affects zero -rows and Hibernate refuses the write anyway. - -### 4. The handler maps the exception to a 409 - -The exception itself carries the message a client can act on: - -```java -public class PostVersionConflictException extends RuntimeException { - - public PostVersionConflictException(UUID postId) { - super("Post %s has changed; fetch it again before retrying".formatted(postId)); - } -} -``` - -and the error boundary turns it into the right HTTP answer: - -```java -@ExceptionHandler(PostVersionConflictException.class) -ProblemDetail handleConflict(PostVersionConflictException exception) { - return problem(HttpStatus.CONFLICT, "POST_VERSION_CONFLICT", exception.getMessage()); -} -``` - -`HttpStatus.CONFLICT` is HTTP 409, and the response is a problem document -— a `type` URI, a stable machine code (`POST_VERSION_CONFLICT`) and a -human-readable detail. The demo enables the framework's problem-details -support in its configuration (`spring.mvc.problemdetails.enabled: true`), -so validation and every other error ride the same shape. - -### 5. The whole path, end to end - -```text -stale PUT /api/posts/{id} - -> PostService.updatePost re-reads the row inside its transaction - -> post.getVersion() != request.version() - -> PostVersionConflictException: "fetch it again before retrying" - -> ApiExceptionHandler.handleConflict - -> HTTP 409, ProblemDetail with code POST_VERSION_CONFLICT - -> the client knows its premise was stale -``` - -One `@Version` field, one `version` in the request contract, one -exception, one handler method. Nothing on the controller changed — -`updatePost` still looks like an ordinary `PUT`. - -## Why a conflict is the correct answer, not a retry - -When two editors produce two different versions of the same record, the -server cannot know which intent is the one meant. Both saves are -legitimate from their authors' point of view, and replaying the stale -write just replays the same data loss. - -The honest verdict is HTTP's own definition of 409: the request conflicts -with the current state of the resource, and the client is told exactly -how to resolve it — "fetch it again before retrying". The losing client -re-reads the record, sees the other writer's change, and a human decides -what to keep. The silent overwrite becomes a visible decision point, -which is the entire purpose of the exercise. - -## What the rest of the demo proves - -The version field is one layer of a small system, and the rest of it is -verifiable the same way — every row below is a class or a config file in -the repository, not a claim: - -| Layer | What it does | Where it lives | -|---|---|---| -| Caching | Bounded Caffeine cache on single-post reads (max 500, TTL 10 min, both from config); `@Cacheable` on read, `@CachePut` on update, `@CacheEvict` on delete; `GET /api/showcase/cache` exposes requests, hits, misses and hit rate | `CacheConfig`, `CacheProperties`, `ShowcaseMetricsController` | -| Security | Stateless HTTP Basic with BCrypt; pages and read APIs are public, every mutation needs the `EDITOR` role; local defaults are a throwaway `demo-editor`/`changeit` account overridable via env vars | `SecurityConfig`, `application.yml` | -| Errors | Request records validate at the boundary; one `@RestControllerAdvice` returns consistent problem documents with machine codes — `POST_NOT_FOUND` (404), `POST_VERSION_CONFLICT` (409), `VALIDATION_FAILED` (400) with a field-error map | `CreatePostRequest`, `UpdatePostRequest`, `ApiExceptionHandler` | -| Configuration | Local profile runs an in-memory H2 database in PostgreSQL mode with `ddl-auto: update` and seeded sample posts; `prod` selects PostgreSQL, takes every credential from `APP_*` env vars and sets `ddl-auto: validate` | `application.yml`, `application-prod.yml` | -| Tests | One `@SpringBootTest` suite asserts the rendered Thymeleaf pages, unauthenticated 401s versus editor 201s, the `VALIDATION_FAILED` problem response, public search, and cache hits after repeated reads | `DemoApplicationIntegrationTest` | - -## What I would do differently for anything real - -Two things change the moment this stops being a local demo. The project -README says both, and they are worth repeating as my own judgement. - -First, schema management. I would add Flyway or Liquibase migrations -before ever setting `ddl-auto` to `validate`. The demo's default profile -uses `update`, which is convenient on a scratch database and dangerous as -a habit — production schema should be versioned code, not a side effect -of startup. - -Second, secrets. The local editor account is intentionally a non-secret -demo account with default credentials. For anything real, I would keep -credentials in a managed secret store and refuse to boot without them, -instead of falling back to a default. The `prod` profile already takes -every credential from environment variables with no committed secrets — -that is the shape I would keep, minus the fallbacks. - -## The result - -A stale update now returns HTTP 409 with code `POST_VERSION_CONFLICT` -instead of a 200 that silently discards the other writer's change. The -loss can no longer happen quietly: the failure is loud, actionable — -"fetch it again before retrying" — and the integration test suite asserts -the access and validation behaviour end to end. - -The whole fix is one annotation on one field, plus plumbing that makes -the version part of the contract. If a record can be read by two people -at the same time, that field is the difference between a lost edit and a -conflict both clients can see. - -I build Java/Spring back ends and full-stack web applications — REST APIs -with JPA, optimistic locking, caching, validation and integration tests, -from the browser page down to the database. If you have a form or a field -where edits keep quietly disappearing, tell me about it: -[WhatsApp](https://wa.me/60127972969) · -[me@hoelee.com](mailto:me@hoelee.com?subject=Spring%20Boot%20help) · -[hoelee.com](https://hoelee.com). -```` - ---- - -## 文章 08 / 11：把 CodeIgniter 4 应用从 IIS 迁移到 OpenLiteSpeed -- **Slug**: `migrating-codeigniter-iis-to-openlitespeed` -- **英文标题**: Migrating a CodeIgniter 4 App From IIS to OpenLiteSpeed -- **分类**: `engineering` | **发布日期**: 2026-09-27 -- **线上地址**: https://blog.hoelee.com/posts/migrating-codeigniter-iis-to-openlitespeed/ (EN) | https://blog.hoelee.com/posts/zh/migrating-codeigniter-iis-to-openlitespeed/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "把 CodeIgniter 4 应用从 IIS 迁移到 OpenLiteSpeed" -description: "我把一个 17,000 行的 CodeIgniter 4 应用从 Windows IIS 搬到了 CyberPanel 上的 OpenLiteSpeed。迁移后冒出两个致命错误——而 IIS 一直靠 SSO 把它们藏着。" -pubDate: 2026-09-27 -category: engineering -blueprint: sme-digital-suite -blueprintLayer: L1-high-conversion-web -tags: [codeigniter, php, openspeedway, cyberpanel, iis, migration, litespeed, linux] -ogImage: /og/zh-migrating-codeigniter-iis-to-openlitespeed.jpg -banner: /banners/migrating-codeigniter-iis-to-openlitespeed.jpg -draft: false ---- - -我维护着一个数字命理报告生成器：一个 CodeIgniter 4 应用，接收出生日期和姓名，跑一遍命理引擎，渲染出 19 页 A4 报告。它在 Windows Server + IIS 上跑了很多年。2026 年 9 月，我把它迁到 CyberPanel 上的 OpenLiteSpeed，跑在 Linux 上，换了个域名。 - -迁移本身没什么好说的——拷文件、把 docroot 指向 `public/`、装依赖。真正值得写下来的是：**新主机上第一个未认证请求，就在两条路由上直接致命崩溃——而这两条路由在 IIS 上"一直好好的"。** - -两个 bug 都是真的。两个从一开始就存在于 IIS 上。两个都看不见，因为 IIS 在出问题的路由前面恰好挡了一层 authentik SSO。 - -## 为什么这件事重要 - -如果你的自托管应用前面挂了一层认证网关，那这层网关做的事不止是保护数据——它同时**在帮你藏 bug**。任何"只对未认证用户出错"的东西永远不会以未认证身份运行，所以永远不会报错。而它会在最糟的时刻暴露：你迁移的时候、你拆掉网关的时候、你把路由暴露到公网的时候。 - -我这个应用，就"健康"了整整几年。我把新域名指过去的那一刻，两条路由返 HTTP 500。代码一行没改。唯一变的是网关没了。 - -两个修复加起来大概二十分钟。找到它们花了两个小时——因为真正的错误信息被 CodeIgniter 的 CLI 错误渲染器挡在了后面。 - -## 环境 - -这是一个相当普通的 CI4 项目，但有一个不太常见的架构细节：报告引擎不是库，而是**同一台主机上的 HTTP 端点**。控制器拼一个请求，`curl` 回自己域名的 `/api/single`，引擎返回算好的 JSON 数值，控制器再把它渲染成可打印的报告。 - -正是这个"自调用"设计，让 IIS 上必须有 `CONST_IIS_INTERNAL_BASE`。这点我后面会回来说——它才是真正有意思的部分。 - -| | 迁移前 | 迁移后 | -|---|---|---| -| 操作系统 | Windows Server | Ubuntu (CyberPanel) | -| Web 服务器 | IIS | OpenLiteSpeed 1.9.0 | -| PHP | 8.4 (Windows 版) | 8.4.25 (lsphp84) | -| Docroot | `C:\inetpub\calc.hoelee.com` | `/home/<域名>/public_html/public` | -| 认证网关 | authentik SSO 挡 `/lifecode`、`/api/*` | 无 | - -## 第一步：docroot 指向 `public/`，不是项目根目录 - -CI4 是两段式目录结构。`app/`、`vendor/`、`writable/` 和 `.env` 都在项目根目录，只有 `public/` 本该对 Web 可见。 - -CyberPanel 建的站点根目录叫 `public_html`。我的第一反应是把整个项目解压进 `public_html`，docroot 不动。 - -**那样会把 `app/`、`vendor/`、`.env` 文件和 `writable/` 的 session 数据全部暴露在 HTTP 之下。** 这个项目的 `.env` 里有一个 webhook 凭证。任何人猜到 `/../.env`——或者干脆直接请求 `/.env`，因为这文件就躺在被服务的目录下面——就拿到了。 - -正确做法是解压进 `public_html`，然后把 vhost 的 docroot 往下一层，改成 `public_html/public`： - -```bash -# 解压成 app/、vendor/、.env 位于 public_html 之下 -cd /home/<域名>/public_html -tar -xzf /tmp/deploy.tar.gz - -# docroot 必须是 public_html/public，不是 public_html -sudo sed -i 's#/home/<域名>/public_html\$#/home/<域名>/public_html/public#' \ - /usr/local/lsws/conf/vhosts/<域名>/vhost.conf - -sudo /usr/local/lsws/bin/lshttpd -t -sudo systemctl restart lsws -``` - -这是整个迁移里最重要的一步，也是大多数教程跳过的一步。其他什么都不做，这一步也得做。 - -## 第二步：IIS 藏起来的两个致命错误 - -### 致命错误一 —— `env()` 调用得太早 - -第一个 500 完全没有响应体。这不太寻常——CI4 正常会渲染点东西出来。一个空白的 500 通常意味着 PHP 在框架的错误处理器装上之前就死了。 - -CI4 的启动流程会在 `Boot::bootWeb()` 里很早加载 `app/Config/Constants.php`（通过 `Boot::loadConstants()`）。而这件事发生在 **`Common.php` 助手文件加载之前**——`env()` 就定义在那个文件里。 - -所以这一行： - -```php -// app/Config/Constants.php —— 错误写法 -define('CONST_fullBase', env('hoelee.fullBase')); -``` - -会失败于： - -``` -Fatal error: Uncaught Error: Call to undefined function env() -in app/Config/Constants.php:96 -``` - -规则是绝对的：**`Constants.php` 里只能放普通常量。** 不能有 `env()`、不能有 `getenv()`、不能有配置助手。如果你真需要依赖环境的值，就定义到启动流程更靠后的位置——或者把这个常量降级成 fallback，真正的值后面再读。 - -我做的正是后者。`CONST_fullBase` 变回普通字符串，消费它的助手函数改为**先**查框架自己的 `app.baseURL`（那个才是 `.env` 驱动的）： - -```php -// app/Helpers/hoelee_helper.php -function getFullBase(bool $selfCall = false): string -{ - // .env 驱动的 baseURL 优先；CONST_fullBase 只是 fallback - if ($selfCall && defined('CONST_IIS_INTERNAL_BASE') && CONST_IIS_INTERNAL_BASE) { - return rtrim(CONST_IIS_INTERNAL_BASE, '/'); - } - $appBase = config('App')->baseURL; - if ($appBase) return rtrim($appBase, '/'); - if (defined('CONST_fullBase') && CONST_fullBase) return rtrim(CONST_fullBase, '/'); - return ''; -} -``` - -**这个是我自己搞出来的。** 同一周做凭证清理重构时引入的——我把一个硬编码 URL 挪进 `.env`，然后在 `Constants.php` 里用 `env()` 读它。在我机器上是通的，因为本地 `.env` 走的是另一条路径被读到。干净启动下它从来没通过。我是在**部署测试**后一小时内抓到的；如果只是代码审查，很可能就这么放过去了。 - -### 致命错误二 —— CI4 控制器里的 `parent::__construct()` - -第二个失败在 `/lifecode`。这个是应用里**原本就存在**的 bug，不是我引入的。 - -绕过 CLI 渲染器之后，真正的报错是： - -``` -Error: Cannot call constructor -``` - -CI4 的基础类 `CodeIgniter\Controller` **根本没有构造函数**。它实现的是 `initController()`，由框架把 request、response、logger 三个对象传进去。标准写法是： - -```php -// CI4 的正确写法 -public function initController( - RequestInterface $request, - ResponseInterface $response, - LoggerInterface $logger -) { - parent::initController($request, $response, $logger); - // 你的初始化代码 -} -``` - -但 `Lifecode.php` 和 `ApiEn.php` 用的是传统构造函数，还在调 `parent::__construct()`： - -```php -// 错误 —— CodeIgniter\Controller 没有 __construct() -public function __construct() -{ - parent::__construct(); - // ... -} -``` - -对一个没有定义 `__construct()` 的父类调 `parent::__construct()`，在 PHP 8 里是致命错误。把两个控制器改成 `initController()` 就修好了——同时要为新签名补上三个 interface 的 `use` 语句。 - -我没有盲信报错信息，而是先确认父类**确实**没有构造函数： - -```bash -grep -n 'function __construct\|function initController' \ - vendor/codeigniter4/framework/system/Controller.php -``` - -只出来 `initController()`。这一步值得做——"Cannot call constructor" 在父类**有**构造函数但内部报错时也会出现，重写签名之前你得先搞清楚自己是哪种情况。 - -## 为什么这两个 bug 在 IIS 上都不出现 - -这是让我改变部署观念的部分。 - -在 IIS 主机上，`/lifecode` 和 `/api/*` 挡在 authentik SSO 后面。对这两条路由的未认证请求返回 **HTTP 302 跳转到 `auth.hoelee.com`**——压根不会碰到控制器。那个坏掉的构造函数从来没被执行过。这条路由大概从写出来那天就是坏的，而它从来没被要求真正处理过一次请求。 - -我直接对比了两台主机来确认： - -```bash -curl -sI https://calc.hoelee.com/lifecode | head -1 -# HTTP/2 302 <- authentik 跳转；控制器根本没运行 - -curl -sI http://<新主机>/lifecode | head -1 -# HTTP/1.1 500 <- 没有网关；控制器运行并致命崩溃 -``` - -那个 302 就是这个 bug 活下来的原因。应用看起来健康，只是因为它不健康的部分根本触不到。 - -**教训说白了就是：一条路由前面挂了认证网关，就意味着这条路由自己的代码没有任何有效的测试覆盖。** 如果你要迁移或者拆网关，请专门留出时间，用未认证身份把每一条曾被网关挡着的路由都打一遍，再宣布迁移完成。小应用上这只是十行 `curl` 循环。在我这里，它本来几秒钟就能找出这两个 bug，而不是两个小时： - -```bash -for p in / /read/single /read/partner /lifecode /api/date /api/single; do - printf '%-16s %s\n' "$p" "$(curl -s -o /dev/null -w '%{http_code}' https://<主机>$p)" -done -``` - -## 第三步：那个不再是 loopback 的自调用 - -现在是这个架构里真正有意思的后果。 - -报告引擎是靠控制器 `curl` 自己的主机来调用的。在 IIS 上，`CONST_IIS_INTERNAL_BASE` 把这个调用指向 `http://localhost:7296`——loopback 接口，请求不出机器，也不碰 DNS 和 TLS。 - -在 LiteSpeed 上这个常量被**故意不定义**，于是 `getFullBase()` 落到 `app.baseURL`。这意味着每一次生成报告，现在都是**一个真实的对外 HTTPS 请求，打到应用自己的公网 URL，再从 Cloudflare 绕回来**。 - -那问题就来了：这到底能不能跑？ - -答案是能，但值得专门测一次，因为它的失败模式很容易误导人。这是确切的测法——**在服务器上跑**，不是在你的笔记本上： - -```bash -curl -s -o /dev/null -w 'HTTPS self-call: %{http_code}\n' \ - -X POST 'https://<域名>/api/single' -d 'nameCn=test&dob=1990-01-01' -``` - -两个要注意的地方： - -- **在服务器上跑。** 我从 Windows 机器上打同一个 URL，拿到的是 404 和 500——那是我这台机器 DNS 解析和 Cloudflare 机器人防护的产物，不是服务器的问题。在错误的主机上测试，会得出很有把握的错误结论。 -- **看 JSON，不只看状态码。** 一个 200 配一个被截断的 body，比干脆失败更糟。我在信任之前先确认了响应能被解析。 - -这里藏着一个真实的取舍，我还没完全解决：走公网 URL 的自调用意味着生成报告依赖 Cloudflare 在线、每次报告多一次 TLS 握手、并且在整个过程中占用两个 PHP worker。在 LiteSpeed 上 worker 池不大的情况下，并发报告一多就可能死锁——每个请求都在等另一个请求，而后者拿不到空闲 worker。以目前的流量没问题。但在规模上去之前，这个自调用应该换成绕过 CDN 的内部路径。 - -## 第四步：验证真实产出，不是状态码 - -`/` 返 200 只证明落地页能渲染。它不证明报告引擎能用，而引擎才是整个产品。所以最后一步是端到端生成一份真报告： - -```bash -curl -s -X POST 'https://<域名>/read/single' \ - -d 'nameCn=test&dob=1990-01-01&gender=m' \ - -o report.html -w 'HTTP:%{http_code} bytes:%{size_download}\n' -``` - -结果：**HTTP 200、99,189 字节、19 页 A4。** 我又用中文字符输入跑了一遍，把引擎的 UTF-8 路径也走通——HTTP 200、99,205 字节、19 页、姓名渲染正常、输出里零 PHP 警告。 - -真正有意义的检查，是新旧主机之间页数可比。它证明字体、DPI 和分页逻辑都完整地熬过了这次搬迁。 - -关于那个 UTF-8 测试的插曲：我第一次跑的时候返回了 500，差点就一头扎进去查 bug。原因是我自己的 shell——中文字符在进 `curl` 的路上被终端编码搞坏了，应用收到的是非法字节。把同一个请求写成服务器上的脚本再跑，编码在我控制之内，就返回了 200。**在你调试编码故障之前，先确认服务器真正收到的字节就是你打算发出去的字节。** - -## 如果重来一次，我会怎么改 - -**迁移前就测未认证路由，而不是迁移后。** 那两个小时的排查，用一个遍历路由列表的 `curl` 循环就能完全避免。我现在把"列出所有路由、以未认证身份逐个请求、记录状态码"当成任何迁移的第零步。 - -**不要在 `Constants.php` 里调 `env()`。** 我在文件里留了注释专门写这件事，因为下一个人——多半是六个月后的我自己——一定会想这么干。 - -**迁移前先查框架版本对应的 project-space 配置。** 同一周我把 CI4 从 4.6.3 升到 4.7.4，它在两个升级指南里根本没提的属性上致命崩溃了两次：`Config\App::$permittedURIChars`（4.7 的 Router 要求）和 `Config\Format::$jsonEncodeDepth`（JSONFormatter 要求，而第二个正好打断了引擎的 JSON 自调用）。Composer 会更新 `vendor/`，但**永远**不会合并 `app/Config/*.php`，因为那些属于 project-space。我最后往 14 个配置文件里手工合并了新属性。那是另一篇文章，但迁移的教训是同一个形状：**框架会告诉你 `vendor/` 里改了什么；没有任何东西会告诉你 `app/` 里改了什么。** - -**先确认 docroot，再动别的。** 如果我解压进 `public_html` 就收手，应用是能跑的——同时静静地把 `.env` 通过 HTTP 服务出去。能跑的迁移，不等于安全的迁移。 - -## 结果 - -一个晚上。六条路由全部验证 200，中英文输入的报告产出在字节层面可比，零 PHP 警告，另外还从代码库里清掉了两个长期潜伏的 bug——而这两个 bug 一直被 IIS 的认证网关遮着。 - -应用现在跑在 OpenLiteSpeed 上，docroot 正确分离，主机可自动化管理，凭证放在 `.env` 而不是写死在常量里。 - ---- - -**想让你的应用也做这套迁移？** 我可以在 IIS、Apache、nginx 和 LiteSpeed 之间迁移 PHP 应用——包括那些麻烦的部分：自调用架构、藏 bug 的 SSO 网关，以及会碰到 project-space 配置的框架升级。 - -[WhatsApp +60 12-797 2969](https://wa.me/60127972969) · [me@hoelee.com](mailto:me@hoelee.com?subject=CodeIgniter%20迁移) · [hoelee.com](https://hoelee.com) -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Migrating a CodeIgniter 4 App From IIS to OpenLiteSpeed" -description: "I moved a 17,000-line CodeIgniter 4 app from Windows IIS to OpenLiteSpeed on CyberPanel. Two fatal errors appeared that IIS had been hiding behind SSO." -pubDate: 2026-09-27 -category: engineering -blueprint: sme-digital-suite -blueprintLayer: L1-high-conversion-web -tags: [codeigniter, php, openspeedway, cyberpanel, iis, migration, litespeed, linux] -ogImage: /og/migrating-codeigniter-iis-to-openlitespeed.jpg -banner: /banners/migrating-codeigniter-iis-to-openlitespeed.jpg -draft: false ---- - -I maintain a numerology report generator: a CodeIgniter 4 application that takes a birth date and a name, runs them through a numerology engine, and renders a 19-page A4 report. It had run on Windows Server with IIS for years. In September 2026 I moved it to OpenLiteSpeed on CyberPanel, on Linux, on a different domain. - -The migration itself was unremarkable — copy the files, point the docroot at `public/`, install dependencies. What made it worth writing about is that **the first unauthenticated request to the new host fatally crashed on two routes that had been working fine on IIS for years.** - -Both bugs were real. Both were present on IIS the whole time. Neither was visible, because IIS had an authentik SSO gate in front of the exact routes that were broken. - -## Why this matters - -If you run a self-hosted app behind an authentication gateway, your auth layer is doing more than protecting data — it is also **hiding your bugs**. Anything that only breaks for unauthenticated users never runs unauthenticated, so it never fails. The failure surfaces at the worst possible moment: when you migrate, remove the gate, or expose the route to the public internet. - -In my case the app had been reporting "healthy" for as long as I'd owned it. The moment I pointed a new domain at it, two routes returned HTTP 500. Nothing had changed in the code. The only thing that changed was that the gate was gone. - -The fix for both took about twenty minutes. Finding them took two hours, because the actual error messages were hidden behind CodeIgniter's CLI error renderer. - -## The setup - -The app is a fairly ordinary CI4 project with an unusual architecture detail: the report engine is not a library, it is a **set of HTTP endpoints on the same host**. The controllers generate a request, `curl` it back to `/api/single` on their own domain, and the engine returns JSON with the computed numbers. The controller then renders that into the printable report view. - -That self-call design is what made `CONST_IIS_INTERNAL_BASE` necessary on IIS. Let me come back to it — it turns out to be the interesting part. - -| | Before | After | -|---|---|---| -| OS | Windows Server | Ubuntu (CyberPanel) | -| Web server | IIS | OpenLiteSpeed 1.9.0 | -| PHP | 8.4 (Windows build) | 8.4.25 (lsphp84) | -| Docroot | `C:\inetpub\calc.hoelee.com` | `/home//public_html/public` | -| Auth gate | authentik SSO on `/lifecode`, `/api/*` | none | - -## Step 1: Point the docroot at `public/`, not the project root - -CI4 ships with a two-folder layout. `app/`, `vendor/`, `writable/` and `.env` live in the project root. Only `public/` is meant to be web-accessible. - -CyberPanel creates the document root as `public_html`. My first instinct was to extract the whole project into `public_html` and leave the docroot alone. - -**That would have exposed `app/`, `vendor/`, the `.env` file, and `writable/` session data over HTTP.** The `.env` in this project contains a webhook credential. Anyone who guessed `/../.env` — or just fetched `/.env`, since the file sits directly under a served directory — gets it. - -The correct layout is to extract into `public_html` and then repoint the vhost docroot one level deeper, at `public_html/public`: - -```bash -# extract the project so that app/, vendor/ and .env sit UNDER public_html -cd /home//public_html -tar -xzf /tmp/deploy.tar.gz - -# the docroot must be public_html/public, NOT public_html -sudo sed -i 's#/home//public_html\$#/home//public_html/public#' \ - /usr/local/lsws/conf/vhosts//vhost.conf - -sudo /usr/local/lsws/bin/lshttpd -t -sudo systemctl restart lsws -``` - -This is the single most important step in the migration, and it is the one most guides skip. If you do nothing else, do this. - -## Step 2: The two fatals IIS was hiding - -### Fatal 1 — `env()` called too early - -The first 500 had no body at all. That is unusual — CI4 normally renders something. An empty 500 usually means PHP died before the framework's error handler was installed. - -CI4's bootstrap loads `app/Config/Constants.php` very early in `Boot::bootWeb()`, via `Boot::loadConstants()`. That happens **before** the `Common.php` helper file is loaded, which is where `env()` is defined. - -So this line: - -```php -// app/Config/Constants.php — BROKEN -define('CONST_fullBase', env('hoelee.fullBase')); -``` - -fails with: - -``` -Fatal error: Uncaught Error: Call to undefined function env() -in app/Config/Constants.php:96 -``` - -The rule is absolute: **`Constants.php` may only contain plain constants.** No `env()`, no `getenv()`, no config helper. If you need environment-dependent values there, define them further along the bootstrap — or make the constant a fallback and read the real value later. - -That is exactly what I did. `CONST_fullBase` became a plain string, and the helper that consumes it checks the framework's own `app.baseURL` (which *is* `.env`-driven) first: - -```php -// app/Helpers/hoelee_helper.php -function getFullBase(bool $selfCall = false): string -{ - // the .env-driven baseURL wins; CONST_fullBase is only a fallback - if ($selfCall && defined('CONST_IIS_INTERNAL_BASE') && CONST_IIS_INTERNAL_BASE) { - return rtrim(CONST_IIS_INTERNAL_BASE, '/'); - } - $appBase = config('App')->baseURL; - if ($appBase) return rtrim($appBase, '/'); - if (defined('CONST_fullBase') && CONST_fullBase) return rtrim(CONST_fullBase, '/'); - return ''; -} -``` - -**This one was my own doing.** I had introduced it during a secrets-cleanup refactor in the same week — I moved a hardcoded URL into `.env` and called `env()` in `Constants.php` to read it. It worked on my machine because the local `.env` was being read through a different path. It never worked on a clean boot. I caught it within an hour *only because* I deployed and tested; a code review would plausibly have waved it through. - -### Fatal 2 — `parent::__construct()` in a CI4 controller - -The second failure was on `/lifecode`, and this one is a genuine pre-existing bug in the application — not something I introduced. - -The error, once I got past the CLI renderer, was: - -``` -Error: Cannot call constructor -``` - -CI4's base `CodeIgniter\Controller` class **has no constructor**. It implements `initController()`, which the framework calls with the request, response, and logger objects. The standard pattern is: - -```php -// correct CI4 pattern -public function initController( - RequestInterface $request, - ResponseInterface $response, - LoggerInterface $logger -) { - parent::initController($request, $response, $logger); - // your setup here -} -``` - -But `Lifecode.php` and `ApiEn.php` declared a classic constructor and called `parent::__construct()`: - -```php -// BROKEN — CodeIgniter\Controller has no __construct() -public function __construct() -{ - parent::__construct(); - // ... -} -``` - -`parent::__construct()` on a parent class that does not define `__construct()` is a fatal in PHP 8. Converting both controllers to `initController()` fixed it — and required adding the three `use` statements for the interface types in the new signature. - -I checked whether the base class *really* had no constructor rather than trusting the error message: - -```bash -grep -n 'function __construct\|function initController' \ - vendor/codeigniter4/framework/system/Controller.php -``` - -Only `initController()` came back. Worth doing — "Cannot call constructor" also fires when a parent *has* a constructor that errors internally, and you want to know which case you are in before you rewrite the signature. - -## Why neither bug showed up on IIS - -This is the part that changed how I think about the deployment. - -On the IIS host, `/lifecode` and `/api/*` sat behind authentik SSO. An unauthenticated request to either returned **HTTP 302 to `auth.hoelee.com`** — it never reached the controller at all. The broken constructor was never executed. The route had presumably been broken since it was written, and it had never once been asked to serve a request. - -I confirmed this by comparing the two hosts directly: - -```bash -curl -sI https://calc.hoelee.com/lifecode | head -1 -# HTTP/2 302 <- authentik redirect; controller never runs - -curl -sI http:///lifecode | head -1 -# HTTP/1.1 500 <- no gate; controller runs and fatals -``` - -The 302 is why the bug survived. The app looked healthy because the unhealthy parts were unreachable. - -**The lesson, stated plainly: an auth gate in front of a route means that route has no working test coverage for its own code.** If you migrate or remove the gate, budget time to hit every previously-gated route unauthenticated before you call the migration done. On a small app that is a ten-line `curl` loop. Here it would have found both bugs in seconds instead of two hours: - -```bash -for p in / /read/single /read/partner /lifecode /api/date /api/single; do - printf '%-16s %s\n' "$p" "$(curl -s -o /dev/null -w '%{http_code}' https://$p)" -done -``` - -## Step 3: The self-call that stops being a loopback - -Now the interesting architectural consequence. - -The report engine is reached by the controller `curl`-ing its own host. On IIS, `CONST_IIS_INTERNAL_BASE` pointed that call at `http://localhost:7296` — the loopback interface — so the request never left the machine and never touched TLS or DNS. - -On LiteSpeed that constant is deliberately left **undefined**, so `getFullBase()` falls through to `app.baseURL`. Which means every report generation now makes a **real outbound HTTPS request to the app's own public URL** and comes back in through Cloudflare. - -Which raises the obvious question: does that work at all? - -The answer is yes, but it is worth testing explicitly, because the failure mode is confusing. Here is the exact test — run it **on the server**, not from your laptop: - -```bash -curl -s -o /dev/null -w 'HTTPS self-call: %{http_code}\n' \ - -X POST 'https:///api/single' -d 'nameCn=test&dob=1990-01-01' -``` - -Two things to watch for: - -- **Run it from the server.** From my Windows machine the same URL returned 404 and 500 — an artefact of DNS resolution and Cloudflare bot protection on my caller, not a server problem. Testing from the wrong host produces confident, wrong conclusions. -- **Watch the JSON, not just the status code.** A 200 with a truncated body is worse than a clean failure. I checked the response was parseable before trusting it. - -There is a real trade-off hiding here, and I have not fully resolved it: a public-URL self-call means report generation depends on Cloudflare being up, costs a TLS handshake per report, and occupies two PHP workers for the duration. On LiteSpeed with a small worker pool, a burst of concurrent reports could deadlock — each request waiting on another request that has no free worker. For the current traffic level it is fine. Before this scales, the self-call should move to an internal path that bypasses the CDN. - -## Step 4: Verify the actual output, not the status code - -A 200 on `/` proves the landing page renders. It does not prove the report engine works, and the engine is the entire product. So the last step was generating a real report end to end: - -```bash -curl -s -X POST 'https:///read/single' \ - -d 'nameCn=test&dob=1990-01-01&gender=m' \ - -o report.html -w 'HTTP:%{http_code} bytes:%{size_download}\n' -``` - -The result: **HTTP 200, 99,189 bytes, 19 A4 pages.** I ran it again with Chinese-character input to exercise the UTF-8 path through the engine — HTTP 200, 99,205 bytes, 19 pages, name rendering correctly, zero PHP warnings in the output. - -Byte-comparable page counts between the old and new host is the check that matters. It proves the fonts, the DPI, and the pagination logic all survived the move. - -A side note on the UTF-8 test: my first attempt at it returned a 500 and I nearly went bug-hunting. The cause was my own shell — the Chinese characters were being mangled by the terminal encoding on the way into `curl`, so the app received invalid bytes. Running the same request from a script on the server, where the encoding is under my control, returned 200. **Before you debug an encoding failure, confirm the bytes actually arriving at the server are the bytes you meant to send.** - -## What I'd do differently - -**Test unauthenticated routes before migrating, not after.** The whole two-hour debugging session was avoidable with a `curl` loop over the route list. I now treat "list every route, hit it unauthenticated, record the status" as step zero of any migration. - -**Do not call `env()` in `Constants.php`.** I have left a comment in the file itself saying so, because the next person — probably me in six months — will be tempted. - -**Check the framework version's project-space config before the move.** The same week, I upgraded CI4 from 4.6.3 to 4.7.4, and it fataled twice on properties the upgrade guide did not mention: `Config\App::$permittedURIChars`, required by the 4.7 Router, and `Config\Format::$jsonEncodeDepth`, required by the JSONFormatter — the second of which broke the engine's JSON self-call specifically. Composer updates `vendor/` but never merges `app/Config/*.php`, because those are project-space files. I ended up merging new properties into 14 config files. That is a separate post, but the migration lesson is the same shape: **the framework tells you what changed in `vendor/`; nothing tells you what changed in `app/`.** - -**Confirm the docroot before anything else.** If I had extracted into `public_html` and stopped, the app would have worked — and quietly served `.env` over HTTP. A migration that works is not the same as a migration that is safe. - -## The result - -One evening. Six routes verified 200, byte-comparable report output on both Chinese and ASCII input, zero PHP warnings, and two long-standing latent bugs removed from the codebase that IIS had been concealing behind an auth gate. - -The app is now running on OpenLiteSpeed with a properly separated docroot, on a host I can automate, with the credential in `.env` rather than a constant. - ---- - -**Want this done for your own application?** I migrate PHP applications between IIS, Apache, nginx and LiteSpeed — including the awkward parts: self-calling architectures, SSO gates that hide bugs, and framework upgrades that touch project-space config. - -[WhatsApp +60 12-797 2969](https://wa.me/60127972969) · [me@hoelee.com](mailto:me@hoelee.com?subject=CodeIgniter%20migration) · [hoelee.com](https://hoelee.com) -```` - ---- - -## 文章 09 / 11：用「热浏览器会话」爬取有反爬墙的电商网站 -- **Slug**: `scraping-bot-walled-marketplace-warm-browser-session` -- **英文标题**: Scraping a Bot-Walled Marketplace With a Warm Browser Session -- **分类**: `engineering` | **发布日期**: 2026-09-13 -- **线上地址**: https://blog.hoelee.com/posts/scraping-bot-walled-marketplace-warm-browser-session/ (EN) | https://blog.hoelee.com/posts/zh/scraping-bot-walled-marketplace-warm-browser-session/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "用「热浏览器会话」爬取有反爬墙的电商网站" -description: "为一位做二手手机生意的客户：爬取有反爬墙的电商商品列表——热 CDP 会话、失败即止的健康检查、分段等待渲染。" -pubDate: 2026-09-13 -category: engineering -tags: ["scraping", "cdp", "headless-chrome", "anti-bot", "shopee", "python"] -ogImage: "/og/zh-scraping-bot-walled-marketplace-warm-browser-session.jpg" -banner: "/banners/scraping-bot-walled-marketplace-warm-browser-session.jpg" -draft: false ---- - -一位客户请我做一个听起来很简单的东西：每天把一个大电商平台上的二手手机列出来，按价格排序，好让他的进货决策靠数据而不是靠感觉。这位客户靠倒卖二手电子产品吃饭——一台手机差几百马币就是他的利润。 - -结果完全不是一段干净的 `requests` + BeautifulSoup 脚本。它变成了一场长达一周的、跟反爬墙的拉锯战，一个靠胶带勉强粘在一起的热浏览器会话，以及——我们俩都没想到的部分——一个关于「你不能靠堆更多 agent 来『并行化』爬虫」的惨痛教训。 - -这就是那个故事，按照我经历的顺序来讲：问题、我试过却失败的每一件事、最终奏效的办法，以及下次我会怎么做。 - -## 问题，用别人真正会搜索的方式表述 - -> "如何从一个拦截无头浏览器的网站爬取商品列表？" - -或者更诚实地说：**当一个电商网站已经把「防爬」当成它唯一的本职工作，你要怎么爬它？** Shopee 不会礼貌地把 HTML 喂给 `curl`。它会检测自动化、抛出一个 `/verify` 验证码，然后静默地给你一个「看起来正常但里面什么都没有」的页面。 - -客户的需求清单很短： - -- 搜索二手手机，只筛选「二手」成色，存储 128 GB 及以上。 -- 提取每个商品的所有存储规格和价格。 -- 按价格升序排序。 -- 不搞一百个 headless 实例的 `chromedriver` 农场——倒卖的利润养不起一百个住宅 IP，我的也一样。 - -从这里开始，全部都是调试故事。 - -## 我试过的，以及为什么失败 - -### 尝试 1：通过 CDP 用冷的无头 Chrome——「直接把 Chrome 远程调试开起来」 - -我用 `--headless --remote-debugging-port=9222` 启动了 Chrome，用 Chrome DevTools Protocol 连上去，然后导航到搜索 URL。大约十秒钟里我觉得自己很聪明。 - -**失败：** 页面加载了，但每个商品卡片都是空的。DOM 是个空壳。Shopee 指纹识别了这个无头浏览器（没有真正的 GPU、user-agent 指纹不一致、`navigator.webdriver` 标志），把我重定向进了一个我永远过不去的 `/verify` 验证码循环。冷的无头 profile 一上来就撞墙。 - -教训很无聊但很重要：**如果你的浏览器本身在「身份」上撒谎，选择器写得再好也没用。** - -### 尝试 2：并行子 agent——「直接开 N 个 worker」 - -这是我最想让它成功的方案。我有一套多 agent 的架构。本能反应就是：开五个 agent，每个爬一个不同的搜索词，收集结果，合并。 - -**失败：** 它们全都连到**同一个**浏览器会话上。它们不会各自拿到一个浏览器——只会各自拿到共享 Chrome 实例里的一个标签页，或者更糟：全都挤在同一个标签页上。每一次 `Page.navigate` 都会把上一个 agent 的上下文抹掉。而且关键在于： - -1. **热会话是共享资源。** 只有一个没被标记的 Canary profile。五个 agent 交错访问同一个标签页，在 Shopee 眼里，跟一个浏览器疯狂快速导航没有区别。而这恰恰是**触发**反爬墙的行为。 -2. **反爬的「节奏」是累积的。** 我的可用方案依赖导航之间约 7 秒的间隔来显得像真人。五个 agent 并发导航，就等于五个导航以零间隔打向同一个会话。 - -所以「横向扩展」的本能——加更多 worker——对我赖以存活的那件事（看起来像真人）是有害的。这里的并发上限不是 CPU 也不是内存，而是**一个热浏览器，一次一个标签页。** - -### 尝试 3：靠「Buy With Voucher」按钮来拿价格 - -一旦列表能渲染出来了，我假设购买按钮的文案里带着价格。大多数电商网站确实如此。但这个渲染出来的按钮写的是 `Add To Cart` / `Buy Now`。我用来匹配「Buy With Voucher … RM…」的正则什么都匹配不到，我在原地打转了好一会儿，才真正去读了 DOM。 - -**失败：** Shopee 商品页的价格不在按钮里。它被渲染成**拆分的 span**——`RM` 在一个元素里、`850` 在另一个里、`00` 又在另一个里——所以朴素的叶子文本匹配只会抓到碎片，或者当商品有多个规格时，抓到一个**区间**（`RM850.00 - RM1150.00`）。 - -### 尝试 4：点击规格选项来拿精确价格 - -对于单轴规格选择器（比如一个存储下拉：128GB / 256GB / 512GB），点击一个选项会把价格收敛成单个值，我的提取方法能正常工作。 - -**失败：** 对于**双轴**选择器（品牌 × 存储），点击第一个轴（某个品牌）会锁定一个价格，但加上第二个轴（某个存储规格）又把它**解散**回商品全区间。这个面板渲染的是「所有可购买组合」的区间，而不是我选中的那个具体组合；而且「已选中」状态是用一个「同时匹配其他所有选项框」的 class 来标记的。我没法可靠地知道我到底锁定了哪个组合。 - -我在这个上面花了实打实的时间，最后才接受这个诚实的答案：**对于真正的双轴商品，正确的数据是价格区间，而不是一个瞎猜的「每个组合」数字。** 发布一个「有时候能用」的脆弱覆盖方案，比老实报告一个区间更糟糕。 - -## 修复——真正奏效的东西 - -三块，都很小，都是吃了亏才学会的。 - -### 1. 热的人性化节奏的浏览器会话（地基） - -别跟墙硬刚。用一个已经登录过、正常浏览过的浏览器 profile——一个「热」profile。用真实的 7 秒间隔来导航。它更慢，但它是唯一能稳定活下来的方式。 - -```python -# 不是真正的代码——是它的「形状」。先导航，然后像真人一样等待。 -goto_url(target) -time.sleep(7) # 真人节奏，不可妥协 -wait_for_selector(".shopee-search-item-result__item") -``` - -### 2. 失败即止，别傻等超时 - -在任何爬取之前，先打 CDP 的 HTTP 端点，检查会话状态。如果 Canary 挂掉了、或标签页停在 `/verify` 上，就在约 1 秒内中止，而不是在「怎么还不渲染」的超时里浪费 40 秒。 - -```python -import json, urllib.request - -def session_healthy(): - try: - tabs = json.load(urllib.request.urlopen("http://127.0.0.1:9222/json")) - except Exception: - return False - for t in tabs: - if t.get("type") == "page" and "/verify" in t.get("url", ""): - return False # 撞到验证码墙——重新热会话，别爬了 - return True -``` - -就这么一个改动，把「静默的 40 秒死胡同」变成了「即时、可执行的失败」。 - -### 3. 正确的价格选择器：最深的纯 RM 节点 - -购买按钮是个谎言。价格是**那个文本内容纯粹是一个 RM 值的、最深的元素**——在选中规格之前是一个区间（`RM850.00 - RM1150.00`），之后是单个值（`RM850.00`）。遍历 DOM，取文本能匹配上的最深元素： - -```python -import re - -RM = re.compile(r"^RM\s?[\d,.]+(\s?[-–]\s?RM\s?[\d,.]+)?$") - -def find_price(root): - best, depth = None, -1 - for el in root.querySelectorAll("*"): - txt = (el.textContent or "").strip() - if txt and RM.match(txt) and " " not in txt.split("RM")[0]: - d = depth_of(el) - if d > depth: - best, depth = el, d - return best.textContent.strip() if best else None -``` - -这就是全部的诀窍。没有脆弱的 class 名。没有瞎猜。一个「整个文本就是一个价格」的节点，就是价格本身。 - -## 结果——一个可量化的成果 - -从单个热会话、一次串行遍历中，我提取了 **20+ 个二手手机商品及其各规格价格**，筛选到 128 GB 及以上，按价格升序排序。最便宜的真实目标——一台 128GB 机型——以 **RM299** 的价格浮出水面，另外还有几台 128/256GB 的选项在 RM450–880 区间。现在一条命令（`sweep.py ...`）就能重新检查整份候选清单并输出 JSONLines；客户每天进货前跑一次。 - -但我最庆幸的数字是这个：**零验证码。** 「热会话 + 人性化节奏 + 失败即止」这套组合，在整次遍历中一次都没触发过 `/verify`。 - -## 下次我会怎么做 - -1. **我会一开始就写「失败即止」的健康检查**，在任何爬取代码之前。三十秒的活，换来未来每次失败省四十秒。 -2. **我会从第一天就抵抗并行化的冲动。** 「爬虫太慢」的答案是把更多工作打包进一次串行遍历（批量搜索、对多个 URL 跑 `sweep.py`），而不是「再加 worker」。共享的浏览器状态让并发变得更糟，而不是更好。 -3. **我会立刻接受双轴的限制**，老实报告区间，而不是在一个根本没法可靠化的选择器上烧一个下午。 -4. **我会别再猜 DOM 结构，先去探测它。** 几乎每一个错误假设（URL 路径里 `/i.` vs `-i.`、把 `Buy With Voucher` 当价格锚点、「过滤器是个真正的 `` 复选框」）都来自**没有先读真实的 DOM**。 - -## 可复用的部分 - -所有这一切被做成了一个小的独立工具集——每个脚本都是一次自包含的 `python x.py ...` 调用： - -| 脚本 | 作用 | -|---|---| -| `search.py` | 粗粒度搜索 → 商品卡片 | -| `used_search.py` | 点击「二手」条件复选框，校验 URL 里的 `USED_ITEM`，再提取 | -| `detail.py` | 从商品页提取各规格价格（最深 RM 节点法 + 分段等待） | -| `sweep.py` | 一次会话内对 N 个 URL 做批量详情提取，按 item-id 去重，写 JSONLines，打印表格 | -| `cdp_common.py` | 共享的健康检查 + 导航辅助 | - -真正可复用的教训不是那些 Shopee 专属的选择器——而是**姿态**：**假设页面在 DOM 上撒谎，失败即止，用批量化而不是并行 worker 来扩展。** - ---- - -*这是一次真实的调试过程，不是事后补写的教程。这个爬虫工具集、以及这里的每一次失败，都发生在为一个做二手电子产品倒卖的客户构建它的过程中——我保留了能泛化的部分，砍掉了那些只对某个马来西亚电商网站、某个时间点有效的部分。* - ---- - -## 想为你的业务做这个吗？ - -我做定制爬虫和数据提取管道——热会话 CDP 爬虫、反爬墙感知的采集器、批量价格监控扫描——也做全栈 Web 应用（PHP/CodeIgniter、Java/Spring、React/TypeScript）和自托管自动化（n8n、Telegram 机器人、NAS 上的 Docker）。 - -如果你需要从某个「会反抗」的网站拿商品或价格数据，直接点进来，告诉我你想提取什么： - -- 💬 [WhatsApp](https://wa.me/60127972969) — 直接私信我 -- ✉️ [Email](mailto:me@hoelee.com?subject=Custom%20scraping%20pipeline) — 告诉我网站和数据 -- 🌐 [hoelee.com](https://hoelee.com) — 看看我还做了些什么 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Scraping a Bot-Walled Marketplace With a Warm Browser Session" -description: "Scraping bot-walled marketplace listings for a client's used-electronics business: warm CDP session, fail-fast health checks, split-wait rendering." -pubDate: 2026-09-13 -category: engineering -tags: ["scraping", "cdp", "headless-chrome", "anti-bot", "shopee", "python"] -ogImage: "/og/scraping-bot-walled-marketplace-warm-browser-session.jpg" -banner: "/banners/scraping-bot-walled-marketplace-warm-browser-session.jpg" -draft: false ---- - -A client asked me to build something that sounds simple: every day, list the second-hand phones on a big marketplace, sorted by price, so their restocking decisions come from data instead of gut feel. The client resells used electronics for a living — a few hundred ringgit either way on a phone is the whole margin. - -The result was nothing like a clean `requests` + BeautifulSoup script. It was a week-long fight against an anti-bot wall, a warm browser session held together with duct tape, and — the part neither of us saw coming — a hard lesson about why you cannot just "parallelize" a scraper by throwing more agents at it. - -This is that story, told in the order I lived it: the problem, everything I tried that failed, the thing that finally worked, and what I'd do differently. - -## The problem, phrased the way someone would actually search it - -> "How do I scrape product listings from a site that blocks headless browsers?" - -Or, more honestly: **how do you scrape a marketplace that has already made scraping its sole job description?** Shopee doesn't politely serve HTML to `curl`. It detects automation, throws a `/verify` captcha, and silently serves you a page that looks fine but contains nothing. - -The client's requirement list was small: - -- Search for used phones, filter to "used condition only", storage 128 GB and up. -- Extract every listing's storage variants and prices. -- Sort by price ascending. -- Do it without a `chromedriver` farm of a hundred headless instances — a reseller's margins don't pay for a hundred residential IPs, and neither do mine. - -Everything after this point is the debugging story. - -## What I tried, and why it failed - -### Attempt 1: cold headless Chrome via CDP — "just point Chrome's remote debugger at it" - -I launched Chrome with `--headless --remote-debugging-port=9222`, connected with the Chrome DevTools Protocol, and navigated to a search URL. It felt clever for about ten seconds. - -**Failure:** the page loaded, but every product card was empty. The DOM was a shell. Shopee had fingerprinted the headless browser (no real GPU, no real user-agent fingerprint consistency, `navigator.webdriver` flag) and redirected me into a `/verify` captcha loop I never got past. Cold headless profiles hit the wall immediately. - -The lesson is boring but important: **anti-bot systems don't care how good your selectors are if the browser itself is lying about what it is.** - -### Attempt 2: parallel subagents — "just spawn N workers" - -This was the one I really wanted to work. I have a multi-agent setup. The natural instinct is: spawn five agents, each scraping a different search query, gather the results, merge. - -**Failure:** all of them attach to the *same* browser session. They don't get their own browser — they get their own *tab* in one shared Chrome instance, or worse, they all fight over the same single tab. Every `Page.navigate` un-does the previous agent's context. And critically: - -1. **The warm session is a shared resource.** There is one Canary profile that isn't flagged. Five agents navigated the same tab in an interleaved order, which is indistinguishable — to Shopee — from one browser doing rapid-fire navigation. That's exactly the behavior that *triggers* the wall. -2. **Anti-bot pacing is cumulative.** My working setup depends on ~7-second gaps between navigations to look human. Five agents navigating concurrently means five navigations hitting the same session with effectively zero gap. - -So the "horizontal scaling" instinct — more workers — is actively hostile to the one thing keeping me alive: looking human. The concurrency ceiling here isn't CPU or memory. It's **one warm browser, one tab at a time.** - -### Attempt 3: trusting the "Buy With Voucher" button for the price - -Once I had listings rendering, I assumed the buy button's label contained the price. On most marketplaces it does. On this render, the button said `Add To Cart` / `Buy Now`. My regex for "Buy With Voucher … RM…" matched nothing, and I went around in circles for a while before actually reading the DOM. - -**Failure:** the price on Shopee product pages isn't in the button. It's rendered as **split spans** — `RM` in one element, `850` in another, `00` in a third — so a naive leaf-text match returns fragments, or a **range** (`RM850.00 - RM1150.00`) when the listing has multiple variants. - -### Attempt 4: clicking variant options to get an exact price - -For a single-axis variant selector (e.g. one dropdown of storage options — 128GB / 256GB / 512GB), clicking one option collapses the price to a single value, and my extraction worked. - -**Failure:** for a **two-axis** selector (brand × storage), clicking the first axis (a brand) locks a price, but adding the second axis (a storage size) *un-collapses* it back to the product-wide range. The panel renders the range of all purchasable combinations, not the exact combination I selected, and the "selected" state is marked by a class that also matches every other option box. There is no reliable way to know which combination I actually locked. - -I spent real time on this before accepting the honest answer: **for genuinely two-axis listings, the correct data is the price range, not a guessed per-combination number.** Shipping a fragile override that "sometimes works" would be worse than reporting an honest range. - -## The fix — what actually worked - -Three pieces, all small, all learned the hard way. - -### 1. A warm, human-paced browser session (the foundation) - -Don't fight the wall. Use a browser profile that has already logged in and browsed normally — a "warm" profile. Navigate with real 7-second gaps. It is slower and it is the only thing that reliably survives. - -```python -# not real code — the *shape* of it. Navigate, then wait like a human. -goto_url(target) -time.sleep(7) # human pacing, non-negotiable -wait_for_selector(".shopee-search-item-result__item") -``` - -### 2. Fail fast, don't time out - -Before any scrape, hit the CDP HTTP endpoint and check the session state. If Canary is down or the tab is sitting on `/verify`, abort in ~1 second instead of wasting 40 seconds on a "did not render" timeout. - -```python -import json, urllib.request - -def session_healthy(): - try: - tabs = json.load(urllib.request.urlopen("http://127.0.0.1:9222/json")) - except Exception: - return False - for t in tabs: - if t.get("type") == "page" and "/verify" in t.get("url", ""): - return False # captcha wall — re-warm the session, don't scrape - return True -``` - -This one change turned "silent 40-second dead-ends" into "instant, actionable failure." - -### 3. The right price selector: deepest pure-RM node - -The buy button is a lie. The price is the **deepest element whose text is a pure RM value** — a range (`RM850.00 - RM1150.00`) before a variant is selected, a single value (`RM850.00`) after. Walk the DOM and take the deepest element whose text matches: - -```python -import re - -RM = re.compile(r"^RM\s?[\d,.]+(\s?[-–]\s?RM\s?[\d,.]+)?$") - -def find_price(root): - best, depth = None, -1 - for el in root.querySelectorAll("*"): - txt = (el.textContent or "").strip() - if txt and RM.match(txt) and " " not in txt.split("RM")[0]: - d = depth_of(el) - if d > depth: - best, depth = el, d - return best.textContent.strip() if best else None -``` - -That's the whole trick. No brittle class names. No guessing. A node whose entire text *is* a price is the price. - -## The result — one quantified outcome - -From a single warm session, in one serial pass, I extracted **20+ used-phone listings with per-variant prices**, filtered to 128 GB and up, sorted by price. The cheapest real target — a 128GB model — surfaced at **RM299**, alongside several 128/256GB options in the RM450–880 range. One command (`sweep.py ...`) now re-checks the whole shortlist and dumps JSONLines; the client runs it once a day before restocking. - -But the number I'm most glad about is this: **zero captchas.** The warm-session + human-pacing + fail-fast combo survived the entire sweep without tripping `/verify` once. - -## What I'd do differently - -1. **I'd write the fail-fast health check first**, before a single scraping line. It's thirty seconds of work that saves forty seconds per future failure. -2. **I'd resist the parallelization instinct day one.** The answer to "the scraper is slow" is "bundle more work into one serial pass" (batch search, `sweep.py` over many URLs), not "more workers." Shared browser state makes concurrency worse, not better. -3. **I'd accept the two-axis limitation immediately** and report ranges, instead of burning an afternoon on a selector that can't be made reliable. -4. **I'd stop guessing DOM structure and probe it.** Almost every mistaken assumption (`/i.` vs `-i.` in the URL path, `Buy With Voucher` as the price anchor, "the filter is a real `` checkbox") came from *not reading the actual DOM first*. - -## The reusable pieces - -All of this became a small, standalone toolkit — each script is one self-contained `python x.py ...` call: - -| Script | What it does | -|---|---| -| `search.py` | Coarse search → listing cards | -| `used_search.py` | Clicks the "Used" condition checkbox, verifies `USED_ITEM` in the URL, extracts | -| `detail.py` | Pulls per-variant prices from a product page (deepest-RM-node method + split-wait) | -| `sweep.py` | Batch detail over N URLs in one session, dedupes by item-id, writes JSONLines, prints a table | -| `cdp_common.py` | Shared health check + navigation helpers | - -The genuinely reusable lesson isn't the Shopee-specific selectors — it's the *posture*: **assume the page is lying about its DOM, fail fast, and scale by batching, not by parallel workers.** - ---- - -*This is a real debugging session, not a tutorial written after the fact. The scraping toolkit and every one of these failures happened while building it for a client who resells used electronics — I kept the parts that generalize and cut the parts that only matter to one Malaysian marketplace at one point in time.* - ---- - -## Want this for your business? - -I build custom scraping and data-extraction pipelines — warm-session CDP scrapers, bot-wall-aware crawlers, and batch price-monitoring sweeps — as well as full-stack web apps (PHP/CodeIgniter, Java/Spring, React/TypeScript) and self-hosted automation (n8n, Telegram bots, Docker on NAS). - -If you need product or price data off a site that fights back, tap through and tell me what you're trying to extract: - -- 💬 [WhatsApp](https://wa.me/60127972969) — message me directly -- ✉️ [Email](mailto:me@hoelee.com?subject=Custom%20scraping%20pipeline) — tell me the site + the data -- 🌐 [hoelee.com](https://hoelee.com) — see what else I build -```` - ---- - -## 文章 10 / 11：JavaScript 文件静默失效的两种方式：而页面看上去一切正常 -- **Slug**: `two-ways-a-javascript-file-stops-working-and-looks-fine` -- **英文标题**: Two Ways a JavaScript File Stops Working and Looks Fine -- **分类**: `engineering` | **发布日期**: 2025-02-20 -- **线上地址**: https://blog.hoelee.com/posts/two-ways-a-javascript-file-stops-working-and-looks-fine/ (EN) | https://blog.hoelee.com/posts/zh/two-ways-a-javascript-file-stops-working-and-looks-fine/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "JavaScript 文件静默失效的两种方式：而页面看上去一切正常" -description: "一个被参数名遮住的函数名，加一个声明在首次调用之后的 let，让整段脚本停摆——而页面渲染得完美好看，因为屏幕上什么都没变。" -pubDate: 2025-02-20 -updatedDate: 2026-10-16 -category: engineering -tags: ["javascript", "i18n", "debugging", "frontend"] -ogImage: /og/zh-two-ways-a-javascript-file-stops-working-and-looks-fine.jpg -banner: /banners/two-ways-a-javascript-file-stops-working-and-looks-fine.jpg -draft: false ---- - -## 一次「什么都没发生」的改动 - -我给一个自己用的媒体转码队列看板加了语言切换：三份语言包、一个 `t(key, vars)` 函数、界面 chrome 上的 `data-i18n`、顶栏一个 `` in the top bar. - -After deploying it, the page looked exactly as it had before. Not broken — -*unchanged*. Same layout, same Chinese labels, no missing panels, no error -card, no console banner. The switcher was simply not there, and neither was -anything else the new code was supposed to do. - -That is the shape of bug I want to write down, because the page was not -"mostly working". **The script had stopped executing partway through — -twice, for two unrelated one-line reasons — and the page rendered perfectly -anyway.** - -## Why nothing looked broken - -Two design decisions I would make again, and both of them hid the failure. - -The first is that the default language lives in the markup. The Chinese text -is written inside the elements; `data-i18n` is only a hook that *overrides* -it when you switch. That is deliberate — the page needs no fetch to render, -and if a language pack goes missing you get Chinese instead of a row of -`top.theme_to_dark` keys. It also means a completely dead i18n layer -produces a page that is, visually, correct. - -The second is that the buttons are wired with inline handlers -(`onclick="rtFoo()"`), which after a script abort point at functions that -were never defined. Nothing warns you. The page only reveals the truth when -a human clicks something. - -So the symptom was "my change had no effect", which is the least informative -symptom a web page can give you. - -## Failure one: a parameter that swallowed a function - -The file has a theme block and an i18n block. The theme block had this -signature: - -```js -function applyTheme(t) { // t = the theme - label.textContent = t('top.theme_to_dark'); // TypeError: t is not a function -} -``` - -`applyTheme` was written before the i18n layer existed, and `t` was a -perfectly reasonable name for "theme". When the i18n layer arrived, `t` in -that scope stopped meaning the theme and started meaning *the function I -wanted to call* — except the parameter shadows it, so the call became -"invoke a string". - -A `TypeError` at the top level of a classic script does not just fail one -line. Everything after it never runs: the language-pack loader, the -`data-i18n` sweep, the switcher rendering. One wrong letter removed an -entire feature, silently. - -The fix is one character — rename the parameter to `th` — plus a comment -where the next person will look: - -```js -function applyTheme(th) { // NOT `t`: t() is the i18n function in this file -``` - -## Failure two: a declaration that arrived too late - -The second cause was in the same file, and it is a JavaScript rule people -know and still trip over. The theme block runs at load and calls `t('…')`. -The i18n block, further down the file, began with: - -```js -let LANG = localStorage.getItem('rt_lang') || 'zh-CN'; -``` - -`let` and `const` are hoisted but **not initialised**: between the start of -the scope and that line they live in the temporal dead zone, and any access -throws `Cannot access 'LANG' before initialization`. Because the theme block -sits *above* the i18n block, the failure was not a race — it was guaranteed -on every single load. - -Note the distinction from `var`, which would have quietly given me -`undefined` and a working page. The TDZ is the language protecting me from -exactly the bug I was about to write; the only reason it looked like -"nothing happened" is the markup fallback again. - -The fix is ordering, and ordering here is a *dependency*, not a style -choice: if a block calls `t()`, the i18n declarations have to be above it. -The file had been organised by topic ("theme, then i18n") because that is -how it grew. - -## Why my checks did not catch either one - -I had tests. They checked intent, not effect: - -- the three language packs exist and parse; -- every key used in the markup exists in every pack; -- no pack contains a dead key (a typo detector); -- the `{var}` placeholder sets in the three packs match. - -All true. All green. None of them executes the page, so none of them can see -a script that aborts on line 12 of 900. Static checks are structurally -unable to catch "the file is valid and the program is dead". - -## Assert on effects, in a browser - -What found both bugs was asking the live page questions that only a running -program can answer: - -```js -typeof t === 'function' // is the entry point still callable? -document.querySelectorAll('[data-i18n]').length // hooks present -rtCheckHandlers() // do the inline handlers still resolve? -``` - -That last one is worth more than it looks. The page carries **219 inline -`onclick` attributes naming 217 distinct functions**; the check enumerates -them and asserts each name is defined on `window`. It turns a hand-clicked -smoke test into a two-second automated one, and it is the check that turns -"the page looks right" into a measurable property. - -The second tool is cheaper and nastier: a **pseudo-language**. -`?lang=pseudo` takes every string in the pack, prefixes it with `[~~` and -stretches it 1.6×. Then assert two things — no horizontal overflow at -1440px, and no element in the nav still lacks the `[~~` prefix. The first -catches layout that only breaks in a longer language; the second catches -nodes that never got a translation hook at all. It found a real one in the -same run. - -Honest scope note: a pseudo-locale does not prove your Malay translation is -*good*. It proves the *plumbing* is complete — every string flows through -the mechanism, and the layout survives length. Judgement about the words -stays with a human. - -## What I would do differently - -1. **Load the page in a real browser and assert `typeof` on the entry points - before believing a client-side refactor worked.** A dead script is - invisible to every static check you own. -2. **Never let a parameter name shadow a function the same file calls.** One - letter bought me an entire afternoon; the comment is the part that stops - it recurring. -3. **Treat declaration order as a dependency, not aesthetics.** If a block - calls a function or reads a constant, that declaration goes above it — - especially in one long script. -4. **Generate the language packs from one table.** Then "the keys match" is - a test rather than a hope, and drift cannot be merged. -5. **When a change does nothing, suspect the script before the change.** - "Nothing happened" is the default failure mode of a rendered page, not - evidence that your code was inert. - -## The result - -Two one-line causes, both in the same file, both found by browser-side -assertions rather than by reading code again. The dashboard switches -language at runtime now, the default render is byte-for-byte what it was -before the change, and the three packs come from a single generated table — -so a missing key shows up as a red test instead of as a word that quietly -stays Chinese. - -The generalisation is the part I keep: **a crash tells you where to look. A -silent stop tells you nothing, and looks like success.** If a feature can -fail without moving a single pixel, then "the page still looks right" is not -a check. -```` - ---- - -## 文章 11 / 11：逐页验证一份 19 页的 PDF 报告 -- **Slug**: `verifying-a-pdf-report-page-by-page` -- **英文标题**: Verifying a 19-Page PDF Report Page by Page -- **分类**: `engineering` | **发布日期**: 2026-09-28 -- **线上地址**: https://blog.hoelee.com/posts/verifying-a-pdf-report-page-by-page/ (EN) | https://blog.hoelee.com/posts/zh/verifying-a-pdf-report-page-by-page/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "逐页验证一份 19 页的 PDF 报告" -description: "如何逐页验证生成的 PDF 报告：用浏览器打印同一份 HTML 作基线，逐页比对几何尺寸，把每一处偏差都压到 2pt 以内。" -pubDate: 2026-09-28 -updatedDate: 2026-09-29 -category: engineering -blueprint: sme-digital-suite -blueprintLayer: L4-managed-ops-handover -tags: [php, codeigniter, pdf, css, print, testing] -ogImage: /og/zh-verifying-a-pdf-report-page-by-page.jpg -banner: /banners/verifying-a-pdf-report-page-by-page.jpg -draft: false ---- - -你怎么知道一份生成出来的 PDF 每一页都真的正确——不只是第一页？ -我维护着一个 CodeIgniter 4 应用：输入一个人的出生信息，生成一份 -中文命理/生命密码报告——十九张密排的 A4 sheet，封面、数字图表、 -方向九宫格、「一到九」的个人特征页。以前这份报告按网站原本的设计 -交付：操作员在浏览器里按 Ctrl+P，手工另存成 PDF。我把它换成了 -服务端渲染（mPDF），一夜之间，产品的质量取决于一个我看不见 -如何排版的引擎。 - -这就是这个问题可被搜索的版本：如何把一份多页 PDF 报告逐页地、 -用数字而不是肉眼，去和浏览器自己的打印输出对版？这个过程抓到 -了真实的 bug——包括一行被渲染成 2.5pt、从第二页起几乎看不见的 -页眉——最后全部十九页与浏览器基线的偏差都压进了 2pt。 - -## 为什么「看起来没问题」不算测试 - -没有人能用肉眼验证一份 19 页的报告。过去的流程是：打开 PDF， -扫一眼封面，交付。第 17 页标题漂移、页脚插图压住版权行、页眉 -小到读不出来——快速翻一遍永远发现不了，而花钱买报告的顾客 -看到的是全尺寸。 - -天真的做法会失败，是结构性的原因。服务端 PDF 渲染器不会像 -浏览器那样排你的 HTML：分页、页边距、基线、字距全都会漂移—— -而且不存在一个「看起来没问题」的测试，能让你在下周有人改了 -CSS 之后再跑一次。我第一次单趟渲染出来是 27 页，而设计预期 -是 19 页：mPDF 没有 CSS 裁剪，而 sheet 靠 `overflow: hidden` -藏住溢出。报告的 `@page` 规则更糟：mPDF 为这条规则本身 -翻了一页，19 张 sheet 炸成 12,789 页。页数都不稳定，所以 -「扫一眼第一页」不是验证——是碰运气。 - -## 基线：让浏览器去打印你的 HTML - -操作员的 Ctrl+P，就是 Chrome 用打印 CSS 打印报告自己的 HTML。 -所以参照标准不是「我觉得它应该长什么样」，而是浏览器自己的 -打印输出——在同一台机器、同一套网页字体下生成。做法：把报告 -目录用本地静态服务起起来（字体必须同源，否则 webfont 拒绝 -加载），再用 headless Chrome 打印。报告 CSS 声明了 -`@page { margin: 0 }`，所以浏览器输出是无边距满幅，再配合 -`-webkit-print-color-adjust: exact` 保住背景图——等价于操作员 -勾选「背景图形」： - -```bash -python -m http.server 8123 --bind 127.0.0.1 --directory - -"C:\Program Files\Google\Chrome\Application\chrome.exe" --headless=new --disable-gpu \ - --no-pdf-header-footer --user-data-dir=%TEMP%\chromeprofile --virtual-time-budget=30000 \ - --print-to-pdf=chrome-win.pdf "http://127.0.0.1:8123/report-local.html" -``` - -然后在 VM 上，用同一份 HTML 出服务端版本： - -```bash -php tools/pdf-render.php /tmp/report-v10.html /tmp/mpdf.pdf -``` - -## 逐页比对，而不是整体对比 - -以浏览器的 PDF 为基准，我在 Windows 这边用 -`uv run --with pymupdf` 逐页对比两份文档：页面尺寸、文字条数、 -图片包围盒（x0/y0/宽/高）、附图说明文字的 y 坐标。判定阈值： -**偏差 ≤2pt（0.7mm）算对齐**；超过 5pt 就要查根因。而且只比 -「同一元素在两份 PDF 里的差」——永远不要比绝对页数，因为页数 -一致是前提，不是测试本身。 - -## 逐页比对抓到了什么 - -下面每个 bug 都有可测量的前后数字，而且都修在库或模板里—— -不是用管道糊过去。 - -**看不见的页眉行。** 从第二页起每页顶部有一行小字，用户直接 -反馈说小到读不出来。页眉是一个 `font-size: 10px` 的 div， -里面套一张 auto 宽的 table，第一格写着 `width: 100%`。mPDF -把这解读成「表格超宽」，于是连同字号一起把整行缩到三分之一： -**2.5pt，而 Chrome 是 7.5pt**。修法（`fixPageHeaderTables()`）： -给表格显式宽度 + 显式字号（px→pt），并去掉第一格的 -`width: 100%`： - -```php -$pt = round((float) $m[2] * 0.75, 2); // 10px = 7.5pt -``` - -**匹配过宽的外边距规则。** `.sheet { margin: 5mm auto }` 是为 -屏幕预览准备的（sheet 之间的阴影缝隙）。mPDF 当真了，把 -296mm 的 sheet 推到 301mm——越过 297mm 的页面——于是 sheet -在页边被切开，auto-fit 把整页缩小 3%，带白边。最直观的修法、 -在样式表末尾追加一条 `.sheet { margin: 0; }`，完全没用：mPDF -对同名选择器的两条规则只认「先出现的赢」。所以库要原地改写 -这条规则（`stripSheetMargins()`），而匹配器必须小心什么才算 -「sheet 规则」。护栏是一个负向回顾断言： - -```php -'~(?` 标签 -居中，而 mPDF 8 的 `Center` 标签处理器是空类——标签整个被 -丢弃，所有居中的标题和表格全部左对齐（「前言」实测 x=30， -Chrome 是 x=280）。`expandCenterTags()` 把 `
` 改写成 -`
`，并给居中块里的表格补上 -`align="center"`——因为父级的 `text-align` 传不到表格。 -修复后：x=281，Chrome 280。 - -**行距比浏览器高 15%。** 模板的 normalize.css 声明了 -`html { line-height: 1.15 }`；mPDF 不继承它，退回自己的字体 -度量（1.33）。这累积成每页下半部 20–40pt 的漂移——sheet 3 -甚至溢出到第二页，触发整页缩小。把 `useFixedNormalLineHeight` -设成模板自己的值之后，正文行距 15.5pt，Chrome 是 15.7。 -表格还要显式加 `td, th { padding: 1px }`——mPDF 默认的 -单元格内边距比浏览器大 2px。 - -**字体是错的。** mPDF 读不了网页用的 `.woff`，文字回退到 -自带的 Sun-ExtA；而正文栈里的「微软雅黑」在 Linux 服务器上 -根本不存在。我注册了三套真 TTF——标题手写体 MaShanZheng、 -拉丁与数字 Roboto、正文 CJK 用 wqy-microhei——并把模板的 -字体栈映射过去。一个坑：必须关掉 mPDF 的自动按脚本选字， -否则它挑「第一个支持中文的已注册字体」——整页正文都变成 -手写体。 - -**还有一个与 mPDF 无关的 bug。** 第 7 页的主插图在**两份** -PDF 里都是破图——比对显示 Chrome 和 mPDF 里是同一个破损 -占位块。模板硬编码了 `https://cdn.hoelee.com/...`，这个域名 -已经不再解析（NXDOMAIN），所以每个引擎都抓不到图。把模板 -改回应用自己的 base URL，并让 `localiseAssets()` 把任何 -host 的 `/static/` 路径都映射到本地文件，两个引擎里的插图 -都恢复了。只有并排比对才能暴露这一类 bug——单独看哪个引擎 -都「正常」。 - -## 一张 sheet，一页 - -渲染器这么设计是有意的：库把 HTML 按 `
` -切开，每张 sheet 单独渲染成一份单页文档，合并时只取第 1 页—— -物理上保证「一张 sheet = 正好一页 A4」，永远不会跨页断裂。 -mPDF 量中文宽度和 Chrome 略有不同，个别 sheet 会高出几毫米。 -与其丢内容，库按一把缩放梯子——1.0, 1.005, 1.01, 1.02, 1.03, -1.06, 1.10, 1.15, 1.22——取第一个能落进一页的比例，缩小整张 -sheet 而不是裁内容。隐藏溢出是浏览器打印做的事 -（`overflow: hidden`）；一份收费产品如果 PDF 里静默丢了页内 -内容，就是无声的交付事故，所以设计选择是绝不丢内容。sheet 3 -现在需要 x1.005——0.5%，肉眼不可见——以前是 x1.03。 - -**「19/7 页」是什么意思。** 报告模板永远按固定顺序渲染十九张 -sheet；「版本」是这些 sheet 上的一个过滤器，不是第二份模板。 -完整版是 **19 页**；RM49 的精华版是**这 19 张里的 7 张**， -页码重编，每张入选的 sheet 都登记了一个文字标记——模板被 -改动/重排导致取错页时会大声失败，而不是把错误的章节交给 -顾客。两个版本走同一条管线，也用同一种方式验收： -`tools/pdf-verify.php --expect=19` 和 -`--expect=7` 双双 PASS -——页数加逐页 ink 检查（ghostscript 50dpi）证明没有任何 -空白页。 - -## 为什么收据是独立文档 - -收据不是报告裁剪出来的。它是自己的一份单页 A4 文档：真 -16mm 页边距（报告刻意做满幅无边距）、三语、单趟渲染—— -因为里面没有 sheet。它在发「已收款」邮件时才懒生成：付款 -回调必须毫秒级应答，0.3–1 秒的 PDF 渲染不属于回调。它落进 -同一个交付存储，文件名带 `-receipt` 后缀，永远不会和报告 -文件撞名，而且幂等：重试、重寄、顾客自己来拿，拿到的都是 -同一份。做它的过程从另一个方向印证了同一个论点——连单页 -文档都和浏览器不一样：`` 上的 `display: block` -不生效，左右并排的两张表把右列的数值裁出页面右边界， -合计行必须写在明细表内部，否则标签会浮在半空。 - -## 我会怎么做不一样 - -对版方法现在躺在项目笔记里，是一份文档化的流程，不是仓库 -里的脚本——这就是差距。仓库里自动化的验收工具只证明页数和 -逐页 ink，永远抓不到 2.5pt 的页眉或 15% 的行距漂移。我会把 -浏览器基线比对做成仓库验证工具链里的一个脚本：用同一份 -HTML 分别喂 Chrome 和库，diff 几何，任何超过 5pt 的偏差 -直接失败。这样，一次让打印布局悄悄回退的 CSS 改动会在构建 -时炸掉，而不是送到顾客手里。 - -我还会在写任何 mPDF 补偿代码之前就先出 Chrome 基线。 -「用操作员用的同一个引擎打印，然后测量」才是解锁点；之后 -每个修复都是机械活。还有两件诚实的遗留工作：伴侣合盘与 -家庭套餐还没跑过这套逐页对版，19 页的 PDF（背景图加嵌入 -字体约 20 MB）也还需要在交付前压缩。 - -## 结果 - -修复后的完整版验收： - -```text -$ php tools/pdf-verify.php /tmp/report-v10.html --expect=19 -out : 20,225,818 bytes, 19 pages, 10.4s, peak 188 MB -per-page ink check: all pages have content | report pages=19 -expected 19 pages => MATCH -RESULT: PASS -``` - -- **19 页 vs Chrome 基线的 19 页 — MATCH**，一张 sheet 一页， - 无跨页断裂。 -- 每一项实测偏差都**在 2pt（0.7mm）以内**：第 6 页偏了 - 213.5pt 的页脚插图、从 2.5pt 恢复到浏览器同款 7.5pt 的 - 页眉、归位的居中标题、15.5pt vs Chrome 15.7 的正文行距、 - 与网页字体一致的嵌入字体。 -- ink 检查确认**没有任何空白页**，PDF 文字可提取——封面能 - 读出真实文字，这对一份顾客要复制内容的报告很重要。 -- 精华版：**7 页 PASS**，4.99 MB，2.8 秒。 - -这就是「第一页看起来没问题」和「全部十九页都在浏览器两个点 -以内」的区别。前者是肉眼扫一遍就交付的结果；后者是拿浏览器 -自己当测试得到的结果。 - ---- - -我平时就做这类 Web 应用与打印/PDF 报告管线，也做网站设计与 -开发。如果你有一份服务器渲染的文档——报告、收据、发票—— -想在它送到顾客手里之前确认每一页都正确，跟我说说： -[WhatsApp](https://wa.me/60127972969) · [me@hoelee.com](mailto:me@hoelee.com?subject=PDF%20report%20pipeline) · -[hoelee.com](https://hoelee.com)。 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Verifying a 19-Page PDF Report Page by Page" -description: "Verify a PDF report page by page: print the same HTML from your browser as a baseline, compare the per-page geometry, and keep every deviation under 2pt." -pubDate: 2026-09-28 -updatedDate: 2026-09-29 -category: engineering -blueprint: sme-digital-suite -blueprintLayer: L4-managed-ops-handover -tags: [php, codeigniter, pdf, css, print, testing] -ogImage: /og/verifying-a-pdf-report-page-by-page.jpg -banner: /banners/verifying-a-pdf-report-page-by-page.jpg -draft: false ---- - -How do you know a generated PDF actually looks right on every page — not just -page one? I maintain a CodeIgniter 4 app that turns birth details into a -Chinese numerology report. Until recently the operator printed it with Ctrl+P -and saved a PDF by hand. I replaced it with a server-side renderer (mPDF), and -overnight the product's quality depended on a layout engine I could not see. - -This post is the searchable version of that problem: verifying a multi-page -PDF report page by page against the browser's own print output, with numbers -instead of eyeballs. It caught real bugs — a header line rendered at 2.5pt, -effectively invisible on every page after the first — and ended with all -nineteen pages within 2pt of the browser. - -## Why "looks fine" is not a test - -Nobody can verify a 19-page report by eye. The old ritual was: open the PDF, -skim the cover, ship it. A drifting heading on page 17, a footer illustration -over the copyright line, an unreadable header — a quick flick never catches -those, and the customer who paid sees them at full size. - -Naive approaches fail structurally. A server-side PDF renderer does not lay -out HTML the way a browser does, so page breaks, margins, baselines and -spacing all drift — and there is no "looks fine" test you can re-run after -someone edits the CSS. My first single-pass render came out as 27 pages where -the design expects 19 — mPDF has no CSS clipping and the sheets hide overflow -with `overflow: hidden`. The report's own `@page` rule was worse: mPDF emitted -a page break for the rule itself, and 19 sheets exploded into 12,789 pages. -Page counts were not even stable, so eyeballing page one was not verification; -it was hope. - -## The baseline: make the browser print your HTML - -The operator's Ctrl+P is Chrome printing the report's own HTML with its print -CSS — same machine, same web fonts. So the reference is not "what I think it -should look like"; it is the browser's own print output. I served the report -directory locally (fonts must be same-origin or webfonts refuse to load) and -printed with headless Chrome. The report CSS declares `@page { margin: 0 }`, -so the output is borderless full-bleed, and `-webkit-print-color-adjust: -exact` keeps the background graphics the operator ticks: - -```bash -python -m http.server 8123 --bind 127.0.0.1 --directory - -"C:\Program Files\Google\Chrome\Application\chrome.exe" --headless=new --disable-gpu \ - --no-pdf-header-footer --user-data-dir=%TEMP%\chromeprofile --virtual-time-budget=30000 \ - --print-to-pdf=chrome-win.pdf "http://127.0.0.1:8123/report-local.html" -``` - -Then the server-side output, from the same HTML, on the VM: - -```bash -php tools/pdf-render.php /tmp/report-v10.html /tmp/mpdf.pdf -``` - -## Compare page by page, not overall - -I compared the two PDFs page by page with `uv run --with pymupdf`: page size, -text-block count, image bounding boxes (x0/y0/width/height), and caption text -y. Threshold: **within 2pt (0.7mm) counts as aligned**; over 5pt gets a root -cause. Compare the same element across the two PDFs — never absolute page -counts; page parity is the prerequisite, not the test. - -## What the comparison caught - -Each bug below had a measurable before/after and a fix in the library or -template. - -**The invisible header line.** A user reported the small header line, present -on every page after the cover, was unreadable. It is a `font-size: 10px` div -wrapping an auto-width table whose first cell declares `width: 100%` — mPDF -read that as "table too wide" and scaled the whole line, font included, to a -third of its size: **2.5pt vs Chrome's 7.5pt**. `fixPageHeaderTables()` gives -the table an explicit width and font size in points (px → pt) and drops the -offending cell: - -```php -$pt = round((float) $m[2] * 0.75, 2); // 10px = 7.5pt -``` - -**The margin rule that matched too much.** `.sheet { margin: 5mm auto }` -spaces the sheets on screen. mPDF took it literally and pushed the 296mm sheet -to 301mm — past the 297mm page — so the sheet was cut at the page edge and -auto-fit shrank the whole page by 3%, white border included. Appending `.sheet -{ margin: 0; }` does nothing — mPDF honours the *first* rule for a duplicated -selector. So the library rewrites the rule in place (`stripSheetMargins()`), -and the matcher must not over-match — the guard is a negative lookbehind: - -```php -'~(?` tag; mPDF 8's `Center` handler is an empty class, so the tag is -dropped and every centred heading and table came out left-aligned (the preface -heading measured x=30 against Chrome's x=280). `expandCenterTags()` rewrites -`
` to `
` and adds `align="center"` to -tables inside centred blocks, because parent `text-align` does not reach -tables. After the fix: x=281, Chrome 280. - -**Line spacing 15% taller than the browser's.** The template's normalize.css -declares `html { line-height: 1.15 }`; mPDF does not inherit it and fell back -to its own font metrics (1.33) — 20–40pt of drift accumulated on the lower -half of every page, and sheet 3 overflowed onto a second page, triggering a -whole-page shrink. Setting `useFixedNormalLineHeight` to the template's own -value brought body leading to 15.5pt against Chrome's 15.7. Tables needed `td, -th { padding: 1px }` — mPDF's default cell padding is 2px larger than a -browser's. - -**The fonts were wrong.** mPDF cannot read the `.woff` files the web views -use, so text fell back to its built-in Sun-ExtA, and "Microsoft YaHei" does -not exist on the Linux server. I registered the real TTFs — MaShanZheng for -headings, Roboto for Latin, wqy-microhei for CJK — and mapped the template's -stacks onto them. One trap: mPDF's automatic script-to-font selection must be -off, or it picks the first registered font that supports Chinese — the entire -body came out in handwriting. - -**And one bug that had nothing to do with mPDF.** Page 7's main illustration -was broken in *both* PDFs — the same broken stub in Chrome and mPDF. The -template hardcoded `https://cdn.hoelee.com/...`, a domain that no longer -resolves (NXDOMAIN), so every engine fetched nothing. Fixing the template to -use the app's own base URL and mapping any host's `/static/` path to local -files restored it in both engines. Only a side-by-side comparison surfaces -this — each engine alone looks "fine". - -## One sheet, one page - -The library splits the HTML into its `
` blocks and -renders each sheet as its own one-page document, importing page 1 and merging -— a physical guarantee that one sheet is exactly one A4 page, never split -across pages. mPDF measures CJK text widths slightly differently from Chrome, -so some sheets come out a few millimetres too tall. Rather than lose content, -the library re-renders the sheet at the smallest scale factor from a ladder — -1.0, 1.005, 1.01, 1.02, 1.03, 1.06, 1.10, 1.15, 1.22 — that fits, shrinking -the whole sheet instead of clipping. The browser's print clips with `overflow: -hidden`; a PDF that silently dropped in-sheet content would be a delivery -accident, so the design never drops content. Sheet 3 needs x1.005 today (0.5%, -invisible) where it used to need x1.03. - -**What "19/7 pages" means.** The report template always renders nineteen -sheets in a fixed order; the "edition" is a filter over those sheets, not a -second template. The full report is **19 pages**; the RM49 essence edition is -**7 of those 19 sheets**, renumbered, each selected sheet carrying a marker so -a rearranged template fails loudly instead of shipping the wrong chapters. -Both editions run the same pipeline and verify the same way: -`tools/pdf-verify.php --expect=19` and `--expect=7` both PASS — page count -plus a per-page ink check (ghostscript at 50dpi) proving no blank pages. - -## Why the invoice is a separate document - -The receipt is not a cut-down report. It is its own one-page A4 document: real -16mm page margins (the report is deliberately full-bleed), three languages, -and a single-pass render because there are no sheets. It is generated lazily -at email time — the payment callback must answer in milliseconds, and a 0.3–1s -render does not belong in it. It lands in the same delivery store under a -`-receipt` filename, never colliding with the report file, and it is -idempotent: retries get the same file. Even one page differed from the -browser: `display: block` on `` was ignored, side-by-side tables -clipped the right column's values off the page edge, and the total row had to -live inside the items table or its label floated in mid-air. - -## What I'd do differently - -The method lives in the project notes as a documented ritual, not a committed -script — that is the gap. The repo's automated acceptance tool proves page -count and per-page ink, which would never catch a 2.5pt header or a 15% -leading drift. I would turn the browser-baseline comparison into a script in -the repo's verification tooling: render the same HTML in Chrome and in the -library, diff the geometry, fail on any deviation over 5pt. Then a CSS change -that silently regresses the print layout fails the build instead of reaching a -customer. - -I would also have generated the Chrome baseline before writing any mPDF -compensation code — "print with the same engine the operator uses, then -measure" was the unlock. Two honest items remain: the partner and family -reports have not had this page-by-page pass yet, and the 19-page PDF (20 MB -with backgrounds and embedded fonts) still needs compression before delivery. - -## The result - -Full report verification, after the fixes: - -```text -$ php tools/pdf-verify.php /tmp/report-v10.html --expect=19 -out : 20,225,818 bytes, 19 pages, 10.4s, peak 188 MB -per-page ink check: all pages have content | report pages=19 -expected 19 pages => MATCH -RESULT: PASS -``` - -- **19 pages vs the Chrome baseline's 19 — MATCH**, one A4 page per sheet, - no splits. -- Every measured deviation is **within 2pt (0.7mm)** of the browser's print - output: footer art 213.5pt off on page 6 aligned, header restored to the - browser's 7.5pt, centred headings back, leading 15.5pt vs 15.7, embedded - fonts matching the web fonts. -- The ink check confirms **no blank pages**, and the PDF text is extractable — - the cover reads back as real text, which matters for a report customers - copy from. -- Essence edition: **7 pages PASS**, 4.99 MB, 2.8 seconds. - -That is the difference between "the first page looks fine" and "all nineteen -pages within two points of the browser": the first ships when you eyeball a -PDF; the second is what you get when the browser itself is the test. - ---- - -I build web applications and print/PDF report pipelines like this one, and I -do website design and development. If you have a document your server renders -— a report, a receipt, an invoice — and you want to be sure it is right on -every page before it reaches a customer, tell me about it: -[WhatsApp](https://wa.me/60127972969) · -[me@hoelee.com](mailto:me@hoelee.com?subject=PDF%20report%20pipeline) · -[hoelee.com](https://hoelee.com). +- **WhatsApp**: [012-7972969](https://wa.me/60127972969) +- **Email**: [me@hoelee.com](mailto:me@hoelee.com?subject=AI%20Audio%20Pipeline%20Inquiry) +- **个人网站**: [hoelee.com](https://hoelee.com) ```` --- diff --git a/docs/consultant-review/batches/batch-04.md b/docs/consultant-review/batches/batch-04.md index 4038854..bb0abbc 100644 --- a/docs/consultant-review/batches/batch-04.md +++ b/docs/consultant-review/batches/batch-04.md @@ -1,10 +1,10 @@ # 博客文章顾问审阅资料包 — Batch-04 > **批次编号**：`Batch-04` -> **包含文章数**：16 篇 -> **本批次总字符量**：168,304 字（严格限制 < 200,000 字） +> **包含文章数**：1 篇 +> **本批次总字符量**：131,667 字（严格限制 < 200,000 字） > **模式**：BILINGUAL -> **生成时间**：2026-10-27 04:23:31 +> **生成时间**：2026-10-27 05:23:38 # 外部顾问审阅前置须知与工程背景规范 @@ -39,30 +39,15 @@ ## 本批次文章目录索引 -1. [engineering] **30 天提交 51.8 万行，生产环境只留下 23.5 万行** (`why-500000-lines-of-commit-churn-only-left-230000-in-production`) — ~12,919 字符 -2. [notes] **给别人家的邮件服务器加一个域名** (`adding-a-domain-to-someone-elses-mail-server`) — ~10,624 字符 -3. [notes] **你好，世界 —— 关于这个博客** (`hello-world`) — ~2,749 字符 -4. [notes] **我的备份爬得比爬虫还慢，因为我在用别人的 API 配额** (`my-backup-was-using-someone-elses-api-quota`) — ~12,287 字符 -5. [notes] **NocoDB SSO 是授权功能：环境变量不会告诉你的事** (`nocodb-sso-is-a-licensed-feature`) — ~15,352 字符 -6. [notes] **与其看供应商文档，不如直接读容器自带的 API 文档** (`reading-a-containers-own-api-docs`) — ~13,865 字符 -7. [notes] **smartctl 退出码 32：专门跳过你最该看的硬盘的那个“错误”** (`smartctl-exit-code-32-skips-the-disks-that-matter`) — ~8,382 字符 -8. [notes] **确定性的活，别再用 Agent Prompt 去跑** (`stop-writing-agent-prompts-for-deterministic-work`) — ~7,167 字符 -9. [notes] **密码正确，Synology API 登录却返回 401，问题出在哪** (`synology-api-401-with-the-correct-password`) — ~8,106 字符 -10. [notes] **NocoDB 附件字段怎么改都不生效，和一个漏网的正则** (`the-nocodb-attachment-that-wouldnt-update`) — ~5,726 字符 -11. [notes] **CodeIgniter 4.6 升级到 4.7：官方指南没说到的破坏性变更** (`upgrading-codeigniter-46-to-47`) — ~14,880 字符 -12. [notes] **为什么本地语音合成一直在念 “Percent”？语音管线里的 URL 编码与沙盒避坑记** (`why-is-my-tts-model-repeating-percent`) — ~11,915 字符 -13. [notes] **为什么多个 AI 一起写代码总会烂尾？传话失真与隐式决策困境** (`why-multi-agent-coding-fails`) — ~8,640 字符 -14. [notes] **为什么你的 Windows 桌面 MCP 调用会静默失败：攻破 10 秒空闲门禁与避坑实录** (`why-your-windows-mcp-calls-silently-fail`) — ~8,459 字符 -15. [tutorials] **校准 Intel Quick Sync HEVC：28 与 30 之间的悬崖，和一个毫无作用的 preset** (`calibrating-intel-quick-sync-hevc-quality`) — ~16,121 字符 -16. [tutorials] **转成 HEVC 到底会不会让文件变小？我实测了** (`does-hevc-actually-shrink-your-files`) — ~11,112 字符 +1. [tutorials] **用 YuE2 BF16 生成完整长歌：突破时长限制与音频硬截断** (`generating-full-length-songs-with-yue2-comfyui-en`) — ~131,667 字符 --- -## 文章 01 / 16：30 天提交 51.8 万行，生产环境只留下 23.5 万行 -- **Slug**: `why-500000-lines-of-commit-churn-only-left-230000-in-production` -- **英文标题**: I Committed 518,000 Lines in 30 Days. Production Kept 235,000. -- **分类**: `engineering` | **发布日期**: 2026-10-18 -- **线上地址**: https://blog.hoelee.com/posts/why-500000-lines-of-commit-churn-only-left-230000-in-production/ (EN) | https://blog.hoelee.com/posts/zh/why-500000-lines-of-commit-churn-only-left-230000-in-production/ (ZH) +## 文章 01 / 01：用 YuE2 BF16 生成完整长歌：突破时长限制与音频硬截断 +- **Slug**: `generating-full-length-songs-with-yue2-comfyui` +- **英文标题**: Generating Full-Length Songs with YuE2 BF16: Overcoming Duration Limits and Audio Truncation +- **分类**: `tutorials` | **发布日期**: 2026-10-26 +- **线上地址**: https://blog.hoelee.com/posts/generating-full-length-songs-with-yue2-comfyui/ (EN) | https://blog.hoelee.com/posts/zh/generating-full-length-songs-with-yue2-comfyui/ (ZH) - **本篇审核状态**: ⏳ 待审核 (Pending) ### ✍️ 顾问审阅意见区 (Review Feedback) @@ -73,4488 +58,3294 @@ 【CRO与转化建议】： ``` -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "30 天提交 51.8 万行，生产环境只留下 23.5 万行" -description: "过去 30 天，自主编程 Agent 在 5 个仓库中累计新增了 51.8 万行代码，但最终留存仅 23.5 万行。为什么在 AI 时代，验证闸门与代码剪枝比生成速度重要十倍？" -pubDate: "2026-10-18" -category: "engineering" -blueprint: ai-agent-automation -blueprintLayer: L2-llm-gateway-agents -tags: ["ai", "git", "devops", "testing", "refactoring"] -ogImage: "/og/zh-why-500000-lines-of-commit-churn-only-left-230000-in-production.jpg" -banner: "/banners/why-500000-lines-of-commit-churn-only-left-230000-in-production.jpg" ---- - -上周我做了一件平时几乎不会做的事：数了数自己的提交。不是数我记得写过的那些，而是数仓库里真实记录下来的那些。自建 Gitea 上 5 个生产仓库、三十天、一个人：865 次 Commit、新增 518,233 行、删除 130,892 行。 - -然后我数了数活下来的部分。对主生产项目的最新版本跑一次干净扫描，有效代码总量是 234,976 行；同一时间窗口里净增 58,240 行的视频转码器，今天整个仓库只有 43,050 行。 - -那 28 万行进哪儿去了？没有丢。它们被写出来、跑过测试、被判为不对，然后删掉——或者被一个更小、做同一件事的东西替换掉。我想把这条循环记下来，因为它这一年对我工作方式的影响，比任何一次模型升级都大。 - -## 一开始我不信的这几个数字 - -以下是 2026 年 9 月 17 日至 10 月 17 日，名下 5 个活跃仓库的精确 Git 统计： - -| 仓库名称 | 30 天 Commit | 累计新增 (+Add) | 累计删除 (-Del) | 净变动 (Net) | 当前存量有效代码/文本 | 核心技术栈 | -|---|---|---|---|---|---|---| -| **numerology-report** | 432 | +331,672 | -98,937 | +232,735 | **234,976** | PHP, Markdown, JSON | -| **hevc-transcode** | 187 | +58,240 | -15,508 | +42,732 | **43,050** | Python, HTML, Shell | -| **dsm-resource-management** | 132 | +58,234 | -14,076 | +44,158 | **368,793** | Python, TypeScript, Astro | -| **hoelee-blog** | 88 | +23,603 | -709 | +22,894 | **55,933** | Astro, TypeScript, Markdown | -| **hoelee-agent-dist** | 26 | +46,484 | -1,662 | +44,822 | *(Dist)* | Python, Shell, Markdown | -| **合计** | **865** | **+518,233** | **-130,892** | **+387,341** | — | — | - -其中两行你可以自己复算，这就是完整的量法： - -```bash -git -C numerology-report log --since=2026-09-17 --until=2026-10-17 \ - --numstat --pretty=format: | awk '{add+=$1; del+=$2} END {print add, del}' -``` - -在人类团队里，这样的更替量通常意味着出事了：需求在代码底下不停移动、有人偷偷在重写、架构反复塌陷。这几种情况我一条都不占——这 5 个仓库都是我每天在跑、离不开的东西，每一次提交都发生在能读到 diff 的会话里。 - -生成成本变低之后，这种更替量就是重构的样子：Agent 起草一段改动，我跑闸门，闸门不同意，这段就被丢掉或者改小重写。凡是闸门写对的那几天，代码总量往下掉得比功能上来的速度还快。 - -## Agent 实际接手的是什么 - -大多数人第一次让 AI 写代码，是在一个浏览器标签页里：把函数粘进对话框，再把答案拷回编辑器。这条路的瓶颈是剪贴板，而且模型离真正重要的两样东西都很远——仓库自己的约定，以及运行它刚写出来的东西的能力。 - -所以我的是跑在终端里、跑在这台机器上的，手里有真工具： - -1. **文件与语法工具**：直接读已有模块，用模糊匹配打补丁而不是整文件覆盖，保存前先过语法与 AST 校验。 -2. **测试与执行器**：PHPUnit 单元测试、pytest 套件、无头 Chrome 探针，在前台跑起来并把输出读回来。 -3. **版本控制上下文**：读 `git diff`、显式暂存指定路径、翻一个文件的历史看上一个决定为什么这么做。 - -这套组合很快，比我亲手打第一版快一个数量级；但它本身也是让代码库腐烂得更快的方法——这一半通常没人提醒。 - -## 臃肿是从哪里长出来的 - -不受约束的模型天生爱写代码。给它一个细小的边界情况，它的第一反应是新建一个私有辅助类、把一个已有工具函数复制一份改个参数，或者给一个库套上三层抽象——那三层只有在这段 diff 里才讲得通。放它自己跑一周，就是几千行语气笃定、看起来合理、其实没必要的代码。 - -我这么讲是因为我自己的历史里就有那一周。每天读这些 diff，最后落成两条规矩： - -- 新建一个类，要跟一个我本来就有的标准库调用竞争位置。 -- 生成出来的东西默认有罪，直到某个确定性检查说它没问题。模型说「看起来对」不算证据，红的测试才算。 - -## 真正把代码删掉的是三道闸门 - -在最近一次主生产系统的 V2 架构重构中，三十天里删掉了 98,000 多行。这些几乎不是我坐下来专门去删的，而是三道闸门不断把回来的东西打回，最后留下的就是更小的版本： - -### 1. 用确定性脚手架，而不是问「这段看着对不对」 - -让模型审查自己写的代码，多数时候产出的是自信，不是正确。有用的做法是把它接到一个没法被说服成绿灯的东西上： - -- PHP 应用里，所有公开路由参数必须带显式默认值，由专用的反射探针（`ControllerAutoRouteSafetyTest.php`）守着——这个测试存在，是因为一条裸路由在生产上真的 500 过。 -- 报告生成管线里，无头 PDF 解析器先数一遍渲染出的汉字数量，数不对就不许构建通过；这样字体或排版改动如果悄悄漏字，就是构建失败，而不是发出一份坏 PDF。 -- 转码队列里，台账比的是精确字节数而不是 MB 估算，因为圆整过的数字把一次体积回退藏了好几个星期。 - -闸门红了，Agent 要么改到过，要么把这段撤掉。没有任何东西能豁免。 - -### 2. 把剪枝当成日常动作，而不是一次大扫除 - -每次会话都会分出一段时间专门删东西：这个 200 行的辅助类，能不能换成一行已有的原生调用？这个新的映射层，是不是在重复 `app/Libraries/` 里放了一年的某个函数？上面那 13 万行删除不是重写的副产品，而是活过头了的脚手架、只有一个调用方的迁移层，以及换了名字出现两遍的同一段逻辑。 - -### 3. 确定性的活留给确定性代码 - -我在 AI 项目里反复看到的坑，是让模型去做它根本不该碰的计算。如果一条 Shell 管道、一个 SQL 约束、一个幂等脚本或者一个 Cron 任务就能给出答案，那 LLM 就不该出现在那里。 - -模型擅长起草初始实现、生成三语词库、摸清三方 API 的脾气；但状态流转、钱、队列调度，我不会交给它——一个确定性问题上概率性的答案，是一颗带延时引信的 bug。 - -## 我的工作方式变了什么 - -我职业生涯的大部分时间里，物理瓶颈是打字速度和记语法。产出就是指标，一个高效的一周看起来就是新建了很多文件。 - -现在语法是免费的。我可以一边读上一次改动的 diff，一边让 Agent 起草一份带单元测试的 500 行迁移。于是瓶颈转移到了模型替我看不了的三件事上： - -- **边界设计**：模块从哪里分开，哪些状态允许跨过去。 -- **验证体系**：那个能抓住类型检查看不见的语义漂移的脚手架。 -- **剪枝的决断力**：愿不愿意删掉那 28 万行探索性、冗余、已死掉的代码，只把 23.5 万行送进生产。 - -所以我现在不再把提交曲线当生产率图表看。一个月新增 50 万行，只能说明工具在转；真正说明这一个月值不值的是那删掉的 13 万行、底下的重构，和一套仍然全绿的测试。 - -## 需要一套能长期不倒的系统？ - -我运营 **Hoelee Enterprise**，为马来西亚中小企业做网站设计与系统开发。上面这套 Agent 工作流、测试脚手架和自托管基础设施，就是我给客户交付并长期维护系统时用的同一套东西。 - -如果你需要全栈系统研发、业务流程自动化，或者能扛住自己维护周期的自建基础设施： -- **WhatsApp:** [+60 12-797 2969](https://wa.me/60127972969) -- **Email:** [me@hoelee.com](mailto:me@hoelee.com) -- **官网:** [hoelee.com](https://hoelee.com) -```` - ### 🇬🇧 英文版原文 (English Markdown Primary) ````markdown --- -title: "I Committed 518,000 Lines in 30 Days. Production Kept 235,000." -description: "Over 30 days, autonomous coding agents added 518,000 lines across 5 repos, yet production kept 235,000. Here is why verification gates matter more than generation." -pubDate: "2026-10-18" -category: "engineering" -blueprint: ai-agent-automation -blueprintLayer: L2-llm-gateway-agents -tags: ["ai", "git", "devops", "testing", "refactoring"] -ogImage: "/og/why-500000-lines-of-commit-churn-only-left-230000-in-production.jpg" -banner: "/banners/why-500000-lines-of-commit-churn-only-left-230000-in-production.jpg" ---- - -Last week I did something I almost never bother to do: I counted my own -commits. Not the ones I remember writing — the ones the repositories -actually recorded. Five production repos on my self-hosted Gitea, thirty -days, one person: 865 commits, 518,233 lines added, 130,892 deleted. - -Then I counted what survived. A clean scan of the current revision of my -main production application returns 234,976 lines. The video transcoder that -gained 58,240 lines in the same window is 43,050 lines long today. - -So where did the other 280,000 lines go? Nothing was lost. They were -written, run, judged wrong, and deleted — or replaced by something smaller -that did the same job. That loop, not the writing, is what I want to write -down, because it changed how I work this year more than any model update -did. - -## The numbers I didn't believe - -Here is the exact commit breakdown across the five repositories, between 17 -September and 17 October 2026: - -| Repository | Commits (30d) | Additions (+) | Deletions (-) | Net Change | Current Total LOC | Primary Stack | -|---|---|---|---|---|---|---| -| **numerology-report** | 432 | +331,672 | -98,937 | +232,735 | **234,976** | PHP, Markdown, JSON | -| **hevc-transcode** | 187 | +58,240 | -15,508 | +42,732 | **43,050** | Python, HTML, Shell | -| **dsm-resource-management** | 132 | +58,234 | -14,076 | +44,158 | **368,793** | Python, TypeScript, Astro | -| **hoelee-blog** | 88 | +23,603 | -709 | +22,894 | **55,933** | Astro, TypeScript, Markdown | -| **hoelee-agent-dist** | 26 | +46,484 | -1,662 | +44,822 | *(Dist)* | Python, Shell, Markdown | -| **Total** | **865** | **+518,233** | **-130,892** | **+387,341** | — | — | - -Two lines are easy to check yourself; this is the whole measurement: - -```bash -git -C numerology-report log --since=2026-09-17 --until=2026-10-17 \ - --numstat --pretty=format: | awk '{add+=$1; del+=$2} END {print add, del}' -``` - -In a human team, churn like that usually means trouble: requirements moving -under the code, a rewrite nobody admits to, an architecture that keeps -collapsing. None of those applied here. Every one of these five repos is -something I run and depend on, and every commit was authored in a session -where I could read the diff before it landed. - -The churn is what refactoring looks like when generation is cheap. An agent -drafts a slice of the change, I run the gates, the gates disagree, and the -slice gets thrown away or rewritten smaller. On the days I got the gates -right, the size drop came faster than the features arrived. - -## What the agents actually touch - -Most people's first experience of AI code is a browser tab: paste a function -into a chat window, copy the answer back into an editor. The bottleneck -there is the clipboard. It also keeps the model far away from the two things -that matter — the codebase's own conventions and the ability to run what it -wrote. - -So mine run in a terminal, on the machine, with real tools: - -1. **Filesystem and syntax tools.** They read existing modules, patch by - fuzzy match instead of overwriting whole files, and run syntax or AST - checks before anything is saved. -2. **Test and execution runners.** PHPUnit suites, pytest harnesses, and - headless Chrome probes, executed in the foreground with the output read - back. -3. **Version control.** They read `git diff`, stage explicit paths, and look - at the history of a file to see why the previous decision was made. - -That combination is fast — an order of magnitude faster than me typing the -first draft. It is also, on its own, a way to make a codebase rot faster, -which is the part nobody warns you about. - -## Where the bloat comes from - -Unsupervised models love to write code. Give one a small edge case and its -instinct is to add a private helper class, clone an existing utility with -one extra parameter, or wrap a library in three layers of abstraction that -only make sense to the diff that introduced them. Left alone for a week, -that is a few thousand lines of confident, plausible, unnecessary code. - -I know because I have that week in my own history. Two things came out of -reading these diffs every day: - -- A new class has to earn its place against a standard-library call I - already have. -- Anything generated is guilty until a deterministic check says otherwise. - "Looks correct" from a model is not evidence; a red test is. - -## The gates that actually deleted the code - -During the V2 architecture overhaul of my main production application, more -than 98,000 lines went out in thirty days. Almost none of that was me -sitting down to delete things. It went because three gates kept rejecting -what came back: - -### 1. Deterministic harnesses instead of "does this look right?" - -Asking a model to review its own code mostly produces confidence, not -correctness. The useful move is to wire the work to something that cannot be -talked out of failing: - -- In the PHP application, every public route parameter has to carry an - explicit default, enforced by a reflection probe - (`ControllerAutoRouteSafetyTest.php`) — that test exists because a bare - route crashed with a 500 in production. -- In the report pipeline, a headless PDF extractor counts the rendered - Chinese characters before a build is allowed to pass, so a font or layout - change that silently drops glyphs fails the build instead of shipping a - broken PDF. -- In the transcoder, the ledger compares exact byte counts rather than - approximating in megabytes, because rounded numbers hid a size regression - for weeks. - -If a gate goes red, the agent either fixes the change or drops it. Nothing -reaching a commit is exempt from that. - -### 2. Pruning as a standing task, not a cleanup project - -A portion of every session goes to deleting. Can a 200-line helper be -replaced by a ten-line call into something that already exists? Is this new -mapper duplicating a function that has lived in `app/Libraries/` for a year? -The 130,000 deletions above are not an accident of a rewrite; they are -scaffolding that outlived its purpose, migration layers with one caller, and -logic that arrived twice under two names. - -### 3. Keeping deterministic work deterministic - -The trap I keep seeing in AI-assisted projects is a model being asked to do -arithmetic it should never touch. If a shell pipeline, a SQL constraint, an -idempotent script, or a cron job can answer it, an LLM has no business -there. - -Models are good at drafting an initial implementation, generating a -three-language dictionary, or mapping a third-party API's quirks. They are -not where I put state transitions, money, or queue scheduling, because a -probabilistic answer to a deterministic question is a bug with a delay fuse. - -## What changed in how I work - -For most of my career the physical constraint was typing speed and -remembering syntax. Output was the metric, and a productive week looked like -a lot of new files. - -Syntax is free now. I can get a 500-line migration with tests drafted while -I read the diff of the previous one. So the constraint moved to the three -things a model cannot check for me: - -- **Boundary design** — where components separate, and which state is - allowed to cross. -- **Test architecture** — the harness that catches semantic drift the type - checker never sees. -- **Pruning discipline** — the willingness to delete 280,000 lines of - exploratory, redundant, or dead code so 235,000 lines can go to - production. - -Which is why I no longer read a commit graph as a productivity chart. Half a -million added lines in a month tells me the tools were running. The 130,000 -deleted lines, the refactor underneath them, and a suite that is still green -tell me whether the month was worth anything. - -## Building something that has to stay up? - -I run **Hoelee Enterprise**, a website design and development studio for -Malaysian SMEs. The agent workflow, the test harnesses and the self-hosted -infrastructure in this post are the same setup I build and maintain client -systems on. - -If you need full-stack engineering, workflow automation, or infrastructure -that survives its own maintenance: -- **WhatsApp:** [+60 12-797 2969](https://wa.me/60127972969) -- **Email:** [me@hoelee.com](mailto:me@hoelee.com) -- **Website:** [hoelee.com](https://hoelee.com) -```` - ---- - -## 文章 02 / 16：给别人家的邮件服务器加一个域名 -- **Slug**: `adding-a-domain-to-someone-elses-mail-server` -- **英文标题**: Adding a Domain to Someone Else's Mail Server -- **分类**: `notes` | **发布日期**: 2025-05-06 -- **线上地址**: https://blog.hoelee.com/posts/adding-a-domain-to-someone-elses-mail-server/ (EN) | https://blog.hoelee.com/posts/zh/adding-a-domain-to-someone-elses-mail-server/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "给别人家的邮件服务器加一个域名" -description: "收信只要四条 DNS 记录，一次就通了；而从它发信、以及转发它，撞上两堵任何 DNS 记录都修不了的墙。" -pubDate: 2025-05-06 -updatedDate: 2026-10-16 -category: notes -blueprint: sme-digital-suite -blueprintLayer: L2-sifumail-communication -tags: ["email", "dns", "mailcow", "smtp", "deliverability"] -ogImage: /og/zh-adding-a-domain-to-someone-elses-mail-server.jpg -banner: /banners/adding-a-domain-to-someone-elses-mail-server.jpg -draft: false ---- - -## 这件事看起来像个 DNS 活 - -有个域名要收信。邮件服务器是我自己那台跑 mailcow 的 VPS，所以这就是「加域名、加记录、建信箱」——二十分钟的活，而且我做过好几次。 - -这次花了更久，原因是我关于邮件服务器的两个假设并不成立，另外还因为互联网有一条任何服务器配置都改不了的性质。 - -## 收信：四条记录，加一次真测试 - -收信恰好只需要四样东西，而且都住在域名自己的 DNS 里（这里是 Cloudflare）： - -```text -MX ninelore.com → mail.hoelee.com （优先级 10） -TXT @ → v=spf1 mx -all -TXT dkim._domainkey → （MTA 发布的那个选择器） -TXT _dmarc → （策略 + 报告地址） -``` - -配好之后，只用一种算数的方式测：**从这套环境外面**（走第三方中继）寄进新地址，然后去读服务器自己的日志行——`rspamd` 在收下这封信时记下 `RCPT_MAILCOW_DOMAIN(ninelore.com)`。收信一次就通。**收信确实是容易的那一半**，而它之所以容易，正因为它只依赖你自己能控制的记录。 - -## 出信才是地面移动的地方 - -两堵独立的墙，各有各的错误，而且都不是改一改区域文件就能修的配置失误。 - -**第一堵墙：那条反垃圾策略，本质上是缺一条 PTR。** 一次直投尝试被对方退回： - -```text -554 5.7.1 Mail rejected due to antispam policy -``` - -在怪远端之前，我先排掉常见原因：四个主流黑名单全干净，域名自己的 SPF/DKIM/DMARC 也都正确。剩下的解释就是那条不在我控制里的记录——**发信 IP 没有反向 DNS**。一台收信服务器如果无法把连进来的 IP 映射回一个名字，就会把它当成一个它无法背书的来源；再正确的 SPF 也压不过这一点。 - -**第二堵墙：用中继绕过去也不行。** 下一个显然的动作是让所有信走那条已经能用的出信中继。结果是： - -```text -550-From header sender domain not verified (gmail.com) -``` - -这行错误要仔细读，因为它一次打死了一整类设计。中继的「已验证寄件人」策略，检查的是 **From 头里的域名**——而**转发**出去的信保留的是原寄件人的 From。所以从你转发第一封来自外部寄件人的信开始，你就是在要求中继以 `gmail.com` 的身份发信，而它永远不会允许。mailcow 在这里还有个值得先知道的坑：它按**寄件人**域名选中继，所以一封 From 是外部域的信，根本不会走到你配的那条中继上。 - -## 那个并不存在的「信箱转发」功能 - -原计划是建好信箱，再转发一份到我的主邮箱。结果连着两个意外： - -- mailcow **没有**「信箱转发」这个设置。把信箱地址写进别名的 `goto` 列表会被拒（`is_alias_or_mailbox`），因为每个信箱自带一条 self-alias 记录，代码在算目标时会跳过这个地址。 -- 想「保留信箱**并且**转发」只能走 **Sieve 过滤器**——那是另一套机制、另一种失败形态——而它最后仍然落在第一堵墙和第二堵墙上，因为这封信终究得离开这台机。 - -所以我**什么都没启用**。一条出信路径送不出去的转发，不会安静地失败：它会把退信发回给无辜的原始寄件人（backscatter），而每一封这样的退信都在花一个本来就没有余量的 IP 的信誉。对「这个新信箱要送到我的收件箱」这件事，稳定的答案是**收件侧的 POP 拉取**：我的邮件客户端从 `mail.hoelee.com:995` 拉信，VPS 什么都不用发。 - -## PTR 是一个 IP 一条，不是一个域一条 - -这是我一直需要解释、包括对我自己解释的部分： - -- 反向记录是**IP 地址**的属性。不存在「每个域一条 PTR」，下次再加域也没有任何 PTR 要动。 -- 它正确的值就是邮件服务器自己对外报的名字——这里是 `mail.hoelee.com`，也就是 postfix 用作 HELO/myhostname 的那个。正向记录（`mail.hoelee.com → 41.216.187.102`）已经存在，**这一对**才是收信方要验证的东西：**forward-confirmed rDNS**。缺任何一半都一文不值。 -- 那个 IP 上所有域名都用同一条反向记录来评判。一百个域名，一条 PTR。 -- 谁来设：跑反向区的人。这里 `216.41.in-addr.arpa` 与 `187.216.41.in-addr.arpa` 两级的 SOA 都是 VPS 商自己的名字服务器，所以它是**服务商侧**的改动——先去 VM / 网络面板里找有没有 rDNS/PTR 输入框，没有就提工单，把上面那个确切的值给他们。 - -我没有假设，而是去查了当前状态：正向解析正常（`mail.hoelee.com → 41.216.187.102`），**那个 IP 的反向解析依旧什么都不返回**。所以直投仍然会被拒，唯一能用的出信路径是中继——而它那条「已验证寄件人」的要求，新域名一样得满足。 - -## 三件要提前知道的事，免得以后被吓到 - -1. **PTR 不是信誉。** 它拆掉的是一次硬拒；它不会让你被信任。新 IP 通常需要几周的日常通信，大厂才会停止把它塞进垃圾箱——而从全新 IP 起步，就是把这口钟从零开始拨。 -2. **PTR 修的是你自己的出信，对转发毫无帮助。** 转发信带的是别人的 From；如果那个域发布了 `p=reject`（银行和大厂都会），那么不管你这边多干净，它在远端都会被挡。 -3. **中继有一个按域的前置条件。** 一个域要走中继，就必须先在中继那边注册并验证它为寄件人。这是每个域永远多出来的一步，且与你自己的 DNS 里的一切无关。 - -## 如果重来一次 - -1. **在建域之前先定：要信箱还是要转发。** 转发是「必须成立的条件」最长的那条路；收件侧 POP 拉取几乎没有条件。 -2. **在向任何人承诺邮件之前，先查发信 IP 的反向记录。** 一次反向查询就能告诉我接下来会撞上哪堵墙。 -3. **把「收件侧拉取」当一等设计，而不是权宜之计。** 「这封信根本不用离开我的服务器」一次消掉三大类故障。 -4. **把那四条收信记录当成一张清单。** 那一半很无聊、可验证，而且是唯一一次就通的部分。 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Adding a Domain to Someone Else's Mail Server" -description: "Receiving mail took four DNS records and worked first try. Sending from it, and forwarding it, hit two walls that no DNS record can fix." -pubDate: 2025-05-06 -updatedDate: 2026-10-16 -category: notes -blueprint: sme-digital-suite -blueprintLayer: L2-sifumail-communication -tags: ["email", "dns", "mailcow", "smtp", "deliverability"] -ogImage: /og/adding-a-domain-to-someone-elses-mail-server.jpg -banner: /banners/adding-a-domain-to-someone-elses-mail-server.jpg -draft: false ---- - -## The task looked like a DNS job - -A domain needed mail. The mail server is my own VPS running mailcow, so this -is "add the domain, add the records, create the mailbox" — the kind of task -that takes twenty minutes and that I have done several times. - -It took longer, because two of the things I assumed about mail servers are -not true, and because of one property of the internet that no amount of -server configuration can change. - -## Inbound: four records and a real test - -Inbound mail needs exactly four things, and all four live in the domain's -own DNS (Cloudflare here): - -```text -MX ninelore.com → mail.hoelee.com (priority 10) -TXT @ → v=spf1 mx -all -TXT dkim._domainkey → (the selector the MTA publishes) -TXT _dmarc → (policy + reporting address) -``` - -Configured and then tested the only way that counts: mail sent **from -outside** the estate (through a third-party relay) into the new address, and -then read the server's own log line — `rspamd` recording -`RCPT_MAILCOW_DOMAIN(ninelore.com)` as the message was accepted into the -mailbox. Receiving worked on the first try. Inbound is genuinely the easy -half, and it is easy precisely because it depends only on records you -control. - -## Outbound is where the ground moves - -Two separate walls, each with its own error, and neither is a config mistake -you can fix by editing a zone file. - -**Wall one: the anti-spam policy that is really a missing PTR.** A direct -delivery attempt to the destination provider came back: - -```text -554 5.7.1 Mail rejected due to antispam policy -``` - -Before blaming the remote, I checked the usual causes: four major -blocklists, all clean, and the domain's own SPF/DKIM/DMARC all correct. The -remaining explanation was the one with no DNS record in my control — the -sending IP had no reverse DNS. A receiving server that cannot map the -connecting IP back to a name treats it as a source it cannot vouch for, and -no amount of correct SPF outweighs that. - -**Wall two: relaying your way around it does not work either.** The obvious -next move is to send everything through the outbound relay that already -works. That attempt produced: - -```text -550-From header sender domain not verified (gmail.com) -``` - -Read that error carefully, because it kills a whole family of designs. The -relay enforces its verified-sender policy against the **From header domain** -— and a *forwarded* message keeps the original sender's From. So the moment -you forward mail from an outside sender, you are asking your relay to send -as `gmail.com`, which it will never allow. mailcow makes this worse in a way -that is worth knowing before you try: it selects the relay by **sender** -domain, so a message with an outside From does not even reach the relay you -configured. - -## The forwarding feature that does not exist - -The plan was to create the mailbox and forward a copy to my main address. -That is two surprises in a row: - -- mailcow has no "mailbox forwarding" setting. Writing the mailbox address - into an alias' `goto` list is rejected (`is_alias_or_mailbox`) because - every mailbox already carries a self-alias record and the code skips the - address when computing destinations. -- Keeping the mailbox *and* forwarding requires a **Sieve filter**, which is - a different mechanism with a different failure mode — and it lands on wall - one and wall two anyway, because the message still has to leave the box. - -So I enabled **nothing**. A forward whose outbound path cannot deliver does -not fail quietly: it sends the rejection back to the innocent original -sender — backscatter — and every one of those bounces spends the reputation -of an IP that has none to spare. The stable answer for "this new mailbox -should reach my inbox" turned out to be a **POP pull on the receiving -side**: my mail client fetches from `mail.hoelee.com:995` and nothing has to -be sent from the VPS at all. - -## PTR is one per IP, not one per domain - -This is the part I keep having to explain, including to myself: - -- The reverse record is a property of the **IP address**. There is no - per-domain PTR, and there is nothing to add the next time a domain is - added. -- Its correct value is the name the mail server itself announces — in this - case `mail.hoelee.com`, which is what postfix sends as its - HELO/myhostname. The matching forward record - (`mail.hoelee.com → 41.216.187.102`) already exists, and the pair is what - receivers verify: **forward-confirmed rDNS**. One half without the other - is worth nothing. -- Every domain on that IP is judged by that same reverse record. A hundred - domains, one PTR. -- Who sets it: whoever runs the reverse zone. Here the SOA for both - `216.41.in-addr.arpa` and `187.216.41.in-addr.arpa` is the VPS host's own - nameserver, so the record is a provider-side change — look for an rDNS/PTR - field in the VM or network panel first, and otherwise open a ticket with - the exact value above. - -I checked the current state rather than assuming: the forward lookup -resolves (`mail.hoelee.com → 41.216.187.102`) and the reverse lookup for -that IP still returns nothing. So direct delivery continues to be rejected, -and the only outbound path that works is the relay — with its -verified-sender requirement, which new domains have to satisfy there too. - -## Three things to expect, so you are not surprised later - -1. **PTR is not a reputation.** It removes a hard rejection; it does not - make you trusted. A new IP typically needs weeks of ordinary mail before - the big providers stop folding it into spam, and starting from a fresh IP - means starting that clock from zero. -2. **PTR fixes your own outbound and does nothing for forwarding.** A - forwarded message carries someone else's From, and if that domain - publishes `p=reject` — banks and large providers do — it is blocked at - the far end no matter how clean your setup is. -3. **Relaying has a per-domain precondition.** If a domain sends through a - relay, that domain must be registered and verified as a sender with the - relay. It is an extra step per domain forever, and it is independent of - everything in your own DNS. - -## What I would do differently - -1. **Decide mailbox-or-forward before creating the domain.** Forwarding is - the option with the longest list of things that must be true; a POP pull - on the receiving side has almost none. -2. **Check the sending IP's reverse record before promising anyone mail.** - One reverse lookup would have told me which of the two walls I was about - to hit. -3. **Treat a receiving-side pull as a first-class design, not a - workaround.** "The message never has to leave my server" removes three - whole categories of failure at once. -4. **Keep the four inbound records as a checklist.** That half is boring, - verifiable, and was the part that worked first try. -```` - ---- - -## 文章 03 / 16：你好，世界 —— 关于这个博客 -- **Slug**: `hello-world` -- **英文标题**: Hello, World — About This Blog -- **分类**: `notes` | **发布日期**: 2024-09-24 -- **线上地址**: https://blog.hoelee.com/posts/hello-world/ (EN) | https://blog.hoelee.com/posts/zh/hello-world/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "你好，世界 —— 关于这个博客" -description: "这个博客是做什么的：技术写作、自托管经验，以及一份我构建与学习过程的记录。" -pubDate: 2024-09-24 -updatedDate: 2026-09-05 -category: notes -tags: [intro] -ogImage: /og/zh-hello-world.jpg -banner: /banners/hello-world.jpg ---- - -欢迎。这里写下我构建与学习的东西——主要是工程、DevOps 和自托管，中间也会 -混一些 AI 自动化和 Web3 的实验。 - -## 这里会有什么 - -这里的每篇文章都有两种语言。英文版先写，中文版是完整翻译，而不是摘要——同一个 slug，放在 `/posts/zh/` 下面。 - -大部分文章是几种形态之一： - -- **案例研究**——「我怎么做出 X」，包含架构和踩过的坑。 -- **教程**——一个难题、我试过什么、最后怎么修好的。 -- **踩坑与笔记**——关于那些耗掉我一整天的小事情，写下简短记录。 - -## 我写什么 - -- **全栈工程**——PHP/CodeIgniter、Java/Spring、TypeScript/React、WordPress。 -- **DevOps 与自托管**——Docker、Traefik、Cloudflare 隧道、NAS/unRaid 私有基础设施。 -- **AI 与自动化**——Telegram 机器人、n8n 工作流、本地 LLM、记忆系统。 -- **Web3**——Solidity、Foundry、ERC-20/721（仍在学习，如实呈现）。 - -## 关于我 - -我是 Lee Teong Hoe（Mr Hoelee），一名常驻马来西亚的全栈开发者兼 DevOps 工程师。 -我做 Web 应用，运维三台自管主机上的 162 个容器，并经营 Hoelee Enterprise -——一家面向马来西亚中小企业的网站设计与开发公司（邮箱托管是附带的服务， -不是主业）。 - -这里的文章是我的动态作品集，不是推销页：我做了什么、什么坏了、证据到底说了什么。 -可以从[最新文章](/posts/zh/)开始看。 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Hello, World — About This Blog" -description: "What this blog is for: technical writing, self-hosting experience, and a record of what I build and learn." -pubDate: 2024-09-24 -updatedDate: 2026-09-05 -category: notes -tags: [intro] -ogImage: /og/hello-world.jpg -banner: /banners/hello-world.jpg ---- - -Welcome. This is where I write about what I build and learn — mostly -engineering, DevOps, and self-hosting, with some AI automation and Web3 -experiments mixed in. - -## What to expect - -Every post here exists in two languages. I write the English version first, -and the Chinese (简体中文) version is a full translation rather than a summary — -the same slug, served under `/posts/zh/`. - -Most posts are one of a few shapes: - -- **Case studies** — "how I built X", with the architecture and the mistakes. -- **Tutorials** — a hard problem, what I tried, and the fix. -- **Gotchas & notes** — short entries on the small things that cost me a day. - -## What I write about - -- **Full-stack engineering** — PHP/CodeIgniter, Java/Spring, TypeScript/React, WordPress. -- **DevOps & self-hosting** — Docker, Traefik, Cloudflare tunnels, NAS/unRaid homelabs. -- **AI & automation** — Telegram bots, n8n workflows, local LLMs, memory systems. -- **Web3** — Solidity, Foundry, ERC-20/721 (learning, honestly framed). - -## About me - -I'm Lee Teong Hoe (Mr Hoelee), a full-stack developer and DevOps engineer -based in Malaysia. I build web applications, run 162 containers -across three self-managed hosts, and run Hoelee Enterprise — a website design & development business -for Malaysian SMEs (with email hosting as a side offering, not the focus). - -The posts here are my working portfolio rather than a sales page: what I -built, what broke, and what the evidence actually said. Start with the -[latest posts](/posts/). -```` - ---- - -## 文章 04 / 16：我的备份爬得比爬虫还慢，因为我在用别人的 API 配额 -- **Slug**: `my-backup-was-using-someone-elses-api-quota` -- **英文标题**: My Backup Was Crawling Because I Was Using Someone Else's API Quota -- **分类**: `notes` | **发布日期**: 2026-10-15 -- **线上地址**: https://blog.hoelee.com/posts/my-backup-was-using-someone-elses-api-quota/ (EN) | https://blog.hoelee.com/posts/zh/my-backup-was-using-someone-elses-api-quota/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "我的备份爬得比爬虫还慢，因为我在用别人的 API 配额" -description: "1 TB 的 restic 种子传输卡在零字节。日志里点名的那个 Google 项目不是我的：那是 rclone 的共享 OAuth 客户端，配额和全世界每个 rclone 用户共池。" -pubDate: 2026-10-15 -updatedDate: 2026-10-16 -category: notes -blueprint: self-hosted-infrastructure -blueprintLayer: L4-secops-cicd-dr -tags: ["rclone", "backup", "google-drive", "oauth", "restic"] -ogImage: /og/zh-my-backup-was-using-someone-elses-api-quota.jpg -banner: /banners/my-backup-was-using-someone-elses-api-quota.jpg -draft: false ---- - -## 订阅到期了，我得搬一个 TB - -这件事以最不方便的方式找上门：承载我每晚备份的那个 OneDrive 账号已经没有订阅了，而手上还躺着一个 1.5 年没用的 Google 订阅。把备份搬过去、保住历史、别丢一晚。 - -第一件撞上的不是技术问题：**我记不清这套备份到底是谁在写**。rclone？Duplicati？还是群晖自带的什么？在一套 78 个 compose 栈、三台机器的环境里（就是[《一个人跑生产基础设施》](/posts/running-production-infrastructure-solo/)里那套），「我的备份是谁写的」本该是我张口就答的问题，而它当时不是。所以第一步是去读机器。 - -## 现场到底有什么 - -**两份 rclone 配置**，正是那种会咬人的细节：Backrest（restic 的前端）用的是 GUI 那份，而另外还有一份**陈旧的 CLI 副本**，被某个 cron 或某次人手 shell 用过。改了一份，等于只改了一半系统。 - -搬迁移本身在顺序对的时候很无聊，而我是先读后动的： - -```text -rclone.conf 追加 [gdrive-*] + [gdrive-*-crypt] -rclone/rclone.conf（CLI 副本） 同样两个 remote -Portainer stack 177 env → 新的 crypt remote；配置 :ro → :rw -restic init 在 Google Drive 上建新仓库 -Backrest SetConfig dsm-repo 的 URI → 新位置 -``` - -在动计划 URI 之前，我先去源码里确认了一件一切依赖它的事：**restic 仓库的身份是 `guid = sha256(id)`** —— 一个由仓库 id 派生的稳定值，**不是**从 URL 推出来的。所以把计划指到新后端，不可能悄悄把历史分叉。六个计划事后逐字节与原版一致，仓库的 `id`/`guid` 未被动过。 - -## 然后种子停住了，而日志里点了一个陌生人的名字 - -新仓库灌到 1075 GB 里的大约 2.88 GB —— 0.27% —— 然后慢成爬行。实时日志里有答案： - -```text -'Read requests per minute' ... reason: RATE_LIMIT_EXCEEDED -consumer: projects/202264815644 <- 不是我的项目 -quota_limit_value: 840000 -``` - -`202264815644` 是 **rclone 自己的共享 OAuth 项目**。我那个 TB 级种子吃的是每分钟读取配额，和地球上每一个 rclone 用户共池——这就是为什么它从约 750 KB/s 掉到零，而我的 NAS、我的磁盘、我的线路全都闲着。等我去量的时候，种子已经完全停摆：**90 秒内 0 字节**、159 个错误、依旧 `rateLimitExceeded`；而六个夜间计划也全撞同一堵墙： - -```text -Fatal: unable to save snapshot: unexpected HTTP response (500) -googleapi: Error 403: Quota exceeded for quota metric 'Queries' - and limit 'Requests per minute' ... -``` - -在进度条里，「慢」和「被挡住」看起来一样，而它们是两个完全不同的问题。**采样 90 秒得到的零**，才是让我做出这个判断的东西。 - -## 你必须有自己的 OAuth 客户端，而控制台绕不过去 - -唯一的真修法只有一个：别再用共享客户端，用你自己的。也就是一个 Google Cloud 项目、一个 OAuth 同意屏、一个凭据。两个设置花掉了真正的时间： - -- **Application type 必须选 Desktop app。** rclone 的回调是 `http://127.0.0.1:53682`，只有这个类型能不用额外配置就允许它。 -- **要点「Publish app」。** 留在 Testing 状态，Google 会**每 7 天**让刷新令牌过期一次——那就把一个「能跑」的系统变成了每周一次的静默故障。 - -我查过能不能用机器上已经认证好的 CLI 抄近路。不能，而原因值得记下来：这里的 `gcloud` 只以一个**别人的项目里的服务账号**登录，而服务账号**无法**创建同意屏或 OAuth 客户端 ID。凭据必须属于一个真人拥有的项目。这是一条死路——我是验证出来的，不是猜的。 - -## 授权流程里的三个坑 - -1. **未经验证就发布的应用，可能会硬性堵死受限 scope。** `.../auth/drive` 是受限 scope，一个**已发布但未完成 Google 审核**的应用有时会被直接拒绝且**没有旁路**。我这一次旁路是存在的——「Google hasn't verified this app」页面，然后 **Advanced → Go to rclone-backup (unsafe)**——但那取决于 scope 与审核状态，是一枚硬币，而不是保证；而「unsafe」只是个标签，意思是从没走过付费审核。 -2. **回调只接受一次请求。** 刷新页面，或者开第二个标签，都会在**没有 code** 的情况下打到 `127.0.0.1:53682`，把这次尝试弄死。我第一次就是这么死的（`no code returned`），而它看起来像凭据错了。**一个标签，别刷新。** -3. **同意屏必须在跑 rclone 的那台机器的浏览器里完成。** 回调指向 `localhost`，所以手机或另一台电脑完不成——浏览器必须和等待那个端口的进程在同一台主机上。 - -之后就简单了：把 token 读出来，把 `client_id` + `client_secret` + `token` 写进**两份**配置（GUI 那份和陈旧的 CLI 副本），并恢复两个对长时无人值守上传很关键的开关：`use_trash = false` 与 `stop_on_upload_limit = true`。 - -## 旧的那一半：靠读，不靠问 - -清空旧 remote 时报了错——`itemNotFound`——而空间照样回来了： - -```text -Used: 1.012 TiB -> 3.800 GiB -Free: ~0 -> 1.007 TiB -DSM 文件夹: 在 -> 不在 回收站: 0 B -``` - -那个错误是 rclone 在**干完活之后**发的抱怨。另外值得记一笔：工具的文档说个人账号无法 `permanentDelete`，按它你就得手动清回收站——实际是回收站已经是空的，约 1 TB 就这么回来了，不需要任何浏览器步骤。文档描述的是 API，而 API 不总是你拿到的东西。 - -还有一条量出来的事，因为它改变了我下一个决定：把那台机器上共享的同步容器暂停之后，负载均值直接减半——16.70 → 8.89。这就是「网络慢」和「机器忙」的区别。 - -## 如果重来一次 - -1. **从第一天起就用自己的 OAuth 客户端。** 共享客户端 ID 是一条你排在里面、却看不到进度、也找不到人负责的队伍。 -2. **「我的备份是谁写的」要写在文档里，不靠记忆。** 两份 rclone 配置，对一个一年没人画过图的系统来说是很正常的漂移量。 -3. **读目的地，别读退出码。** 那次清空「失败」了，却释放了一个 TB；而一次看起来成功的改名，可能什么都没落地。跟[《它报告成功，其实什么都没变》](/posts/it-reported-success-nothing-had-changed/)是同一课——这套环境一直在教这一课。 -4. **不要在注意力只剩最后百分之几的时候启动一个要跑几天的上传。** 在 0.27% 时发现 `RATE_LIMIT_EXCEEDED` 只花几分钟；跑了两天才发现，花掉的是整个窗口。 - -## 结果，诚实地说 - -管道是通的：计划现在住在 Google Drive 上、仓库身份没变、种子花的是**我的**配额而不是共享池、旧的那一侧退回了大约 1 TB。但它现在**还不是一个备份**：写这篇的时候，第一轮种子才上传了约 1075 GB 里的 2.88 GB，而且新仓库里**快照数为零**——在第一张快照落地之前，什么都还原不了。那是我盯着的数字，因为「传输在跑」和「你能还原」是两句不同的话。 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "My Backup Was Crawling Because I Was Using Someone Else's API Quota" -description: "A 1 TB restic seed stalled at zero bytes. The log named a Google project I did not own: rclone's shared OAuth client, quota pooled with every rclone user alive." -pubDate: 2026-10-15 -updatedDate: 2026-10-16 -category: notes -blueprint: self-hosted-infrastructure -blueprintLayer: L4-secops-cicd-dr -tags: ["rclone", "backup", "google-drive", "oauth", "restic"] -ogImage: /og/my-backup-was-using-someone-elses-api-quota.jpg -banner: /banners/my-backup-was-using-someone-elses-api-quota.jpg -draft: false ---- - -## The subscription ended, and I had to move a terabyte - -The call came in the least convenient way: the OneDrive account holding my -nightly backups had no subscription any more, and I had a 1.5-year Google -subscription sitting unused. Move the backup, keep the history, don't lose a -night. - -The first thing I hit was not a technical problem: I could not remember -*how* the backup was wired. rclone? Duplicati? Something Synology shipped? -On a three-host estate with 78 compose stacks — the one in -[Running Production Infrastructure Solo](/posts/running-production-infrastructure-solo/) -— "which tool writes my backups" is a question I should have been able to -answer from memory and could not. So step one was reading the machine. - -## What was actually there - -Two rclone configs, which is the kind of detail that bites: the GUI config -that Backrest (the restic front end) uses, plus a **separate stale CLI -copy** that a cron job or a human shell had been using at some point. Change -one and you have changed half the system. - -The migration itself is boring when done in the right order, and I did it -read-first: - -```text -rclone.conf append [gdrive-*] + [gdrive-*-crypt] -rclone/rclone.conf (CLI copy) same two remotes -Portainer stack 177 env → the new crypt remote; config :ro → :rw -restic init new repo on Google Drive -Backrest SetConfig dsm-repo URI → the new location -``` - -Before touching the plan URIs I checked one property in the source, because -everything depended on it: a restic repository's identity is -`guid = sha256(id)` — a stable value derived from the repo id, **not** from -its URL. So repointing a plan at a new backend cannot silently fork the -history. All six plans came out byte-identical to the originals, and the -repo `id`/`guid` were untouched. - -## Then the seed stopped moving, and the log named a stranger - -The new repo seeded to 2.88 GB of roughly 1075 GB — 0.27% — and then slowed -to a crawl. The live log had the answer: - -```text -'Read requests per minute' ... reason: RATE_LIMIT_EXCEEDED -consumer: projects/202264815644 <- not my project -quota_limit_value: 840000 -``` - -`202264815644` is **rclone's own shared OAuth project**. My terabyte seed -was drawing on a per-minute read quota pooled with every rclone user on -earth, which is why it degraded from ~750 KB/s to nothing while my NAS, my -disk and my link were all idle. By the time I measured, the seed was fully -stalled — **0 bytes in 90 seconds**, 159 errors, still `rateLimitExceeded` — -and the six nightly plans had all failed with the same wall: - -```text -Fatal: unable to save snapshot: unexpected HTTP response (500) -googleapi: Error 403: Quota exceeded for quota metric 'Queries' - and limit 'Requests per minute' ... -``` - -"Slow" and "blocked" look similar in a progress bar and are completely -different problems. Ninety seconds of sampled zeros is what made the call. - -## You need your own OAuth client — and the console is unavoidable - -There is exactly one real fix: stop using the shared client and bring your -own. That means a Google Cloud project, an OAuth consent screen, and a -credential. Two settings cost real time: - -- **Application type = Desktop app.** rclone's redirect is - `http://127.0.0.1:53682`, and that client type permits it without extra - configuration. -- **Publish the app.** Leave it in "Testing" and Google expires your refresh - token **every 7 days** — which turns "working" into a silent weekly - outage. - -I checked whether I could shortcut the console with the CLI that was already -authenticated here. I could not, and the reason is worth knowing: `gcloud` -was signed in only as a **service account** in an unrelated project, and -service accounts cannot create consent screens or OAuth client IDs. The -credential has to belong to a project a human owns. It was a dead end, and I -verified that instead of asserting it. - -## Three traps in the authorization flow - -1. **Publishing an unverified app can hard-block a restricted scope.** - `.../auth/drive` is restricted, and an app that is published *without* - completing Google's verification review is sometimes blocked outright - with no bypass. In my case the bypass existed — the "Google hasn't - verified this app" screen, then **Advanced → Go to rclone-backup - (unsafe)** — but that is a coin flip on scope and review state, not a - guarantee, and "unsafe" is only a label meaning the app never went - through a paid audit. -2. **The callback accepts exactly one request.** A reload, or a second tab, - hits `127.0.0.1:53682` with no code and kills the attempt. My first try - died exactly that way (`no code returned`), and it looks like a - credentials problem. One tab. Don't reload it. -3. **Finish the consent screen on the machine that runs rclone.** The - redirect goes to `localhost`, so a phone or another PC cannot complete it - — the browser has to be the one on the same host as the process waiting - on that port. - -Then: read the token, write `client_id` + `client_secret` + `token` into -**both** configs (the GUI one and the stale CLI copy), and restore the two -flags that matter for a long unattended upload: `use_trash = false` and -`stop_on_upload_limit = true`. - -## The old side: verify by reading, not by asking - -Purging the old remote reported an error — `itemNotFound` — and the space -came back anyway: - -```text -Used: 1.012 TiB -> 3.800 GiB -Free: ~0 -> 1.007 TiB -DSM folder: present -> gone Trashed: 0 B -``` - -The error was rclone complaining *after* the work finished. Also worth -recording: the tool's documentation says personal accounts cannot -`permanentDelete`, which would have meant emptying the recycle bin by hand — -in practice the trash was already empty and roughly 1 TB was reclaimed with -no browser step. Documentation describes the API; the API is not always what -you get. - -One more measured thing, because it changed my next decision: pausing the -file-sync container that shared that machine halved the load average — 16.70 -→ 8.89 — which is the difference between "the network is slow" and "the box -is busy". - -## What I would do differently - -1. **Own the OAuth client from day one.** A shared client id is a queue you - are standing in, with no status page and no one to call. -2. **Answer "what writes my backups" from a document, not from memory.** Two - rclone configs is a normal amount of drift for a system nobody has drawn - in a year. -3. **Read the destination, not the exit code.** The purge "failed" and freed - a terabyte; a successful-looking rename can land nothing. Same lesson as - [It Reported Success. Nothing Had Changed.](/posts/it-reported-success-nothing-had-changed/) - — this estate keeps teaching it. -4. **Never start a multi-day upload while the last few percent of your - attention is elsewhere.** Checking for `RATE_LIMIT_EXCEEDED` at 0.27% - costs minutes; discovering it after two days costs the window. - -## The result, honestly stated - -The plumbing works: the plans live on Google Drive, the repo identity is -unchanged, the seed is now spending **my** quota instead of a shared one, -and the old side gave back about a terabyte. What it is not, yet, is a -backup: at the time I wrote this, the first seed had uploaded 2.88 GB of -~1075 GB and the new repo had **zero snapshots** — nothing is restorable -until the first snapshot lands. That is the number I watch, because "the -transfer is running" and "you can restore" are different claims. -```` - ---- - -## 文章 05 / 16：NocoDB SSO 是授权功能：环境变量不会告诉你的事 -- **Slug**: `nocodb-sso-is-a-licensed-feature` -- **英文标题**: NocoDB SSO Is a Licensed Feature — and the Env Vars Won't Tell You -- **分类**: `notes` | **发布日期**: 2026-09-29 -- **线上地址**: https://blog.hoelee.com/posts/nocodb-sso-is-a-licensed-feature/ (EN) | https://blog.hoelee.com/posts/zh/nocodb-sso-is-a-licensed-feature/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "NocoDB SSO 是授权功能：环境变量不会告诉你的事" -description: "NocoDB 的 OIDC 环境变量是真的，启动时也会校验，所以自托管 SSO 看起来完全受支持。但在未授权的构建上，第一次登录尝试就会让整个实例崩溃。" -pubDate: 2026-09-29 -category: notes -tags: ["nocodb", "sso", "oidc", "self-hosting", "licensing", "docker"] -ogImage: /og/zh-nocodb-sso-is-a-licensed-feature.jpg -banner: /banners/nocodb-sso-is-a-licensed-feature.jpg -draft: false ---- - -如果你自托管 NocoDB，去搜一下「nocodb sso self-hosted」，你会找到一篇介绍 OIDC 单点登录的文档页、一组环境变量，以及一堆论坛回复告诉你「设置好这些就能用」。这些在技术上都是真的。但没有一样会告诉你真正要紧的那部分。 - -所以直说了：**NocoDB 的 OIDC SSO 是付费功能。**在未授权的自托管构建上，代码路径是存在的，但它会直接抛弃进程——而且失败的方式也不是一句客气的「此实例未启用 SSO」。整个服务器直接退出。 - -## 为什么这件事重要 - -有两种人会在这里踩坑。 - -**如果你在生产环境启用它，你就给自己造好了一个拒绝服务的触发器。**处理 OIDC 回调的路由不需要认证就能访问——它必须如此，登录流程就是这样开始的。在一个这条路由会让进程崩溃的实例上，任何请求它的人都能把你的数据库界面打挂。不是变慢，不是返回错误页：而是直接干掉容器。你甚至不必成为攻击目标；一个好奇的访客，或者一个顺着 URL 爬的链接扫描器，就够了。 - -**如果你正在评估 NocoDB 这个平台，这是一个你该读对的授权信号。**功能在开源镜像里是有的，变量会被强制校验，代码也一并发布。缺的只是许可。一个*已实现*、*未授权*的功能，看起来和能正常工作的功能一模一样，直到你真的去试——而这正是整个陷阱所在。 - -## 文档说了什么，镜像又做了什么 - -NocoDB 自己的文档，只要你读到授权那一行，其实写得很清楚：*「OIDC SSO 在 NocoDB Cloud（Business 及以上版本）以及获得授权的自托管部署（Business 及以上版本）中可用。」*但另一半——未授权的二进制不会优雅降级——文档里没有写，从文档里也无从发现，因为这是关于镜像本身的运行时事实。 - -环境变量是真的。以下是我对照随镜像发布的内容确认过的： - -```yaml -NC_SSO: oidc -NC_SSO_OIDC_ISSUER: https://auth.example.com/application/o/your-app/ -NC_SSO_OIDC_AUTHORIZATION_URL: https://auth.example.com/application/o/authorize/ -NC_SSO_OIDC_TOKEN_URL: https://auth.example.com/application/o/token/ -NC_SSO_OIDC_USERINFO_URL: https://auth.example.com/application/o/userinfo/ -NC_SSO_OIDC_CLIENT_ID: ... -NC_SSO_OIDC_CLIENT_SECRET: ... -NC_OIDC_PROVIDER_NAME: Hoelee SSO -``` - -公开文档里只出现了 `NC_SSO` 和 provider-name 变量。其余六个按 provider 区分的变量没有文档，但**会被强制校验**：用 `NC_SSO=oidc` 启动容器、却不给任何 URL，它直接拒绝启动： - -``` -Open ID SSO is enabled but missing required env keys -``` - -正是这个细节让陷阱显得可信。启动时的校验意味着这个功能是*接好线的*，不是残留代码。那条错误消息没有任何地方暗示：没有授权，这条代码路径就不可达——它读起来就像一个改改配置就能修好的错误。 - -## 怎么在零风险的情况下测试 - -千万别在你关心的实例上测试。一个用镜像内置 SQLite 元数据存储的一次性容器就够了，大约一分钟： - -```bash -docker run -d --name nc-sso-test -p 10399:8080 \ - -e 'NC_SSO=oidc' \ - -e 'NC_SSO_OIDC_ISSUER=https://auth.example.com/application/o/your-app/' \ - -e 'NC_SSO_OIDC_AUTHORIZATION_URL=https://auth.example.com/application/o/authorize/' \ - -e 'NC_SSO_OIDC_TOKEN_URL=https://auth.example.com/application/o/token/' \ - -e 'NC_SSO_OIDC_USERINFO_URL=https://auth.example.com/application/o/userinfo/' \ - -e 'NC_SSO_OIDC_CLIENT_ID=test' \ - -e 'NC_SSO_OIDC_CLIENT_SECRET=test' \ - -e 'NC_OIDC_PROVIDER_NAME=Test SSO' \ - nocodb/nocodb: -``` - -注意这里刻意省略了 `NC_DB`——没有配置外部数据库，容器会自己拉起一个 SQLite 存储，所以它不需要任何凭据、不碰生产数据，一条 `docker rm -f nc-sso-test` 就能删掉。 - -观察启动过程，然后请求一次 SSO 路由： - -```bash -curl -s -o /dev/null -w '%{http_code}\n' http://localhost:10399/auth/oidc -docker ps -a --filter name=nc-sso-test -``` - -我得到的、两次都是——第二次我加了个 `?workspaceId=` 参数，排除请求格式不对的可能： - -``` -### UNCAUGHT EXCEPTION ### -unhandledRejection TypeError: _0x418c23 is not a function at docker/index.js:1:27483406 -``` - -……然后容器就不再运行了。一次未认证的 GET，退出码 1。在 `2026.09.0` 上复现，两次结果一模一样。这不是配置错误，也不是客户端把请求形状搞错了。 - -注意启动日志里那句解释了一切的话： - -``` -No license key found — running in CE mode -``` - -## 授权故事的另一半 - -如果你的反应是「行，那我买授权」，在把它写进预算之前先读读这段，因为这里还有一个同样不明显的先决条件： - -``` -Instance ID unavailable — PostgreSQL is required for enterprise licensing -``` - -我跑的社区版用 **MySQL** 作为元数据存储，而 MySQL 无法生成授权所依赖的实例 ID。所以授权不是你在现有实例上激活一下就完事——它是授权*外加一次从 MySQL 到 Postgres 的元数据迁移*，这本身就是个自带风险的项目，还得在线上实例上执行。NocoDB 为此专门发布了迁移指南，光这一点就说明这种情况有多常见。 - -这不是在抱怨定价模式。这是我希望能提前拥有的评估：对任何跑默认 MySQL 配置的人来说，「直接买」的真实成本是授权**加上**一次存储引擎迁移。 - -## 搜索结果推荐的那个 fork——先确认它还活着 - -不买授权搜 NocoDB SSO，搜索结果会指向 `brunostjohn/nocodb-oidc`——一个给开源构建加上 OIDC 的社区 fork。它是即插即用的镜像，正是你想要的东西。但它也被弃养了，这一点大约两分钟就能查证： - -```bash -# what version of NocoDB is the fork based on? -curl -s https://raw.githubusercontent.com/brunostjohn/nocodb-oidc/main/packages/nocodb/package.json \ - | python -c "import sys,json; print(json.load(sys.stdin)['version'])" -# 0.255.2 - -# when was it last touched? -curl -s 'https://api.github.com/repos/brunostjohn/nocodb-oidc/commits?per_page=1' \ - | python -c "import sys,json; print(json.load(sys.stdin)[0]['commit']['author']['date'])" -# 2024-10-29 -``` - -一个 2024 年的版本，对上一个按月发版的实例，不是即插即用——那是倒退两年的降级，还会带走你的自动化已经依赖的 schema 迁移、安全修复和 API 行为。建议的好坏只取决于 fork 有没有人继续维护，而搜索结果是不会因为仓库弃养而过时的。 - -**在围绕它做规划之前，先查一下 fork 的基础版本和最后一次提交的日期。**两条 curl，就能免掉你原本要在凌晨一点做的回滚。 - -## 那该怎么办 - -1. **先想清楚你需要的是应用内的 SSO，还是应用前面的 SSO。**如果你要的是「互联网够不到这个登录表单」，那么在主机名前放一个 forward-auth 代理，用开源构建就能达成。它不是应用内的单点登录——应用仍然会要自己的密码。我把这个搭建方案，以及那个让我的第一次尝试看起来成功、实际上毫无作用的 ingress 陷阱，写在了[验证完美通过、实际并未生效的 Forward-Auth Gate](/posts/authentik-forward-auth-gate-wasnt-live/)里。 -2. **保留应用自己的登录，在边缘把门锁上。**对只有几个用户的内部工具来说，这才是大多数人真正需要的。 -3. **如果确实需要授权构建，把迁移也计入成本，而不只是授权本身。**先确认你用的是哪种元数据存储。 - -## 如果重来一次，我会怎么改 - -**当一个功能的可用性模糊不清时，从镜像里读答案，而不是看文档。**文档描述的是意图；镜像才是真相。一条 `docker exec ... printenv | grep -iE 'NC_|OIDC'`、一次对随镜像发布代码的 grep、一个一次性容器，二十分钟就回答了读一下午文档都答不出来的问题。 - -还有一条更直白、也适用于 NocoDB 之外的规则：**没有文档、却被强制校验的环境变量，是一条授权边界，而不是功能开关。**如果一个变量在没填完整时拒绝启动、而文档又藏起了它一半的兄弟变量，那么缺的那块几乎从来不是配置。 - -## 结果 - -在临时容器上五分钟复现两次，两次容器都死了。还有一个我宁可别在生产环境才学到的意外发现：对 SSO 路由的一次未认证请求，就足以让整个实例停下来。 - -好的一面是，我不再试图让应用自己去做 SSO，而是把决定放到了它该在的位置——主机名前面，现在也能用一句话回答「NocoDB 能免费做 SSO 吗？」，而不是花一下午。 - ---- - -## 想给业务用上这套？ - -如果你正在评估自托管软件、需要内部工具前面的单点登录，或者想让别人去核对你的技术栈实际做了什么、对照文档声称它做了什么——这正是我做的工作。 - -- **WhatsApp：** [011-797 2969](https://wa.me/60127972969) ——点一下就能聊 -- **Email：** [me@hoelee.com](mailto:me@hoelee.com?subject=Self-hosted%20SSO%20enquiry) -- **Website：** [hoelee.com](https://www.hoelee.com) - -我为马来西亚的小企业搭建 authentik 单点登录、自托管 Docker 技术栈和反向代理——并且把做这些事时学到的东西写下来。 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "NocoDB SSO Is a Licensed Feature — and the Env Vars Won't Tell You" -description: "NocoDB's OIDC environment variables are real and validated at startup, so self-hosted SSO looks supported. On an unlicensed build, the first login attempt crashes the whole instance." -pubDate: 2026-09-29 -category: notes -tags: ["nocodb", "sso", "oidc", "self-hosting", "licensing", "docker"] -ogImage: /og/nocodb-sso-is-a-licensed-feature.jpg -banner: /banners/nocodb-sso-is-a-licensed-feature.jpg -draft: false ---- - -If you self-host NocoDB and search for "nocodb sso self-hosted", you'll find a documentation page describing OIDC single sign-on, a set of environment variables, and forum answers telling you it works if you set them. All of that is technically true. None of it tells you the part that matters. - -So here it is: **NocoDB's OIDC SSO is a paid feature.** On an unlicensed self-hosted build, the code path exists but abandons the process — and the failure isn't a polite "SSO is not enabled on this instance". The whole server exits. - -## Why this matters - -Two audiences get bitten here. - -**If you enable it in production, you've built a denial-of-service trigger.** The route that handles the OIDC callback is reachable without authentication — it has to be, that's how a login flow starts. On an instance where that route crashes the process, anyone who requests it takes your database UI down. Not degrades it, not returns an error page: kills the container. You don't need to be a target; a single curious visitor, or a link scanner that follows URLs, is enough. - -**If you're evaluating NocoDB as a platform, this is a licensing signal you should read correctly.** The feature is present in the open-source image, the variables are enforced, the code ships. What's missing is permission. A feature that is *implemented* but *not licensed* looks identical to a feature that works, right up until you try it — and that's the whole trap. - -## What the docs say, and what the image does - -NocoDB's own docs are clear if you read the licensing line: *"OIDC SSO is available on NocoDB Cloud (Business plan and above) and licensed self-hosted deployments (Business plan and above)."* The other half — that the unlicensed binary will not degrade gracefully — isn't documented, and can't be discovered from the docs, because it's a runtime fact about the image. - -The environment variables are real. The ones I confirmed against the shipped image: - -```yaml -NC_SSO: oidc -NC_SSO_OIDC_ISSUER: https://auth.example.com/application/o/your-app/ -NC_SSO_OIDC_AUTHORIZATION_URL: https://auth.example.com/application/o/authorize/ -NC_SSO_OIDC_TOKEN_URL: https://auth.example.com/application/o/token/ -NC_SSO_OIDC_USERINFO_URL: https://auth.example.com/application/o/userinfo/ -NC_SSO_OIDC_CLIENT_ID: ... -NC_SSO_OIDC_CLIENT_SECRET: ... -NC_OIDC_PROVIDER_NAME: Hoelee SSO -``` - -Only `NC_SSO` and the provider-name variable appear in the public documentation. The six per-provider variables are undocumented but **enforced**: start the container with `NC_SSO=oidc` and no URLs and it refuses to boot: - -``` -Open ID SSO is enabled but missing required env keys -``` - -That's the detail that makes the trap convincing. Validation at startup means the feature is *wired in*, not vestigial. Nothing about that error message suggests the code path is unreachable without a licence — it reads like a configuration mistake you can fix. - -## How to test it without risking anything - -Never test this on the instance you care about. A throwaway container with an in-image SQLite meta store is enough, and it takes about a minute: - -```bash -docker run -d --name nc-sso-test -p 10399:8080 \ - -e 'NC_SSO=oidc' \ - -e 'NC_SSO_OIDC_ISSUER=https://auth.example.com/application/o/your-app/' \ - -e 'NC_SSO_OIDC_AUTHORIZATION_URL=https://auth.example.com/application/o/authorize/' \ - -e 'NC_SSO_OIDC_TOKEN_URL=https://auth.example.com/application/o/token/' \ - -e 'NC_SSO_OIDC_USERINFO_URL=https://auth.example.com/application/o/userinfo/' \ - -e 'NC_SSO_OIDC_CLIENT_ID=test' \ - -e 'NC_SSO_OIDC_CLIENT_SECRET=test' \ - -e 'NC_OIDC_PROVIDER_NAME=Test SSO' \ - nocodb/nocodb: -``` - -Note the deliberate omission of `NC_DB` — with no external database configured, the container spins up its own SQLite store, so it needs no credentials, touches no production data, and can be deleted with `docker rm -f nc-sso-test`. - -Watch the startup, then request the SSO route once: - -```bash -curl -s -o /dev/null -w '%{http_code}\n' http://localhost:10399/auth/oidc -docker ps -a --filter name=nc-sso-test -``` - -What I got, twice — the second time with a `?workspaceId=` parameter to rule out a bad request shape: - -``` -### UNCAUGHT EXCEPTION ### -unhandledRejection TypeError: _0x418c23 is not a function at docker/index.js:1:27483406 -``` - -...and the container is no longer running. Exit code 1, from an unauthenticated GET. Reproduced on `2026.09.0` with the same result both times. It isn't a configuration error, and it isn't a request the client got wrong. - -Note the log line that explains it, from startup: - -``` -No license key found — running in CE mode -``` - -## The other half of the licensing story - -If your reaction is "fine, I'll buy the licence", read this before you budget for it, because there's a prerequisite that isn't obvious either: - -``` -Instance ID unavailable — PostgreSQL is required for enterprise licensing -``` - -The community edition I run uses **MySQL** as its metadata store, and MySQL cannot generate the instance ID that licensing depends on. So a licence isn't something you activate on the instance you have — it's a licence *plus a metadata migration from MySQL to Postgres*, which is its own project with its own risk, done on a live instance. NocoDB publishes a migration guide for exactly this, which tells you how common the situation is. - -That isn't a complaint about the pricing model. It's the evaluation I wish I'd had before spending an afternoon: the cost of "just buy it" is licence **plus** a storage-engine migration for anyone running the default MySQL setup. - -## The fork that search results recommend — check it's alive first - -Search results for NocoDB SSO without a licence point at `brunostjohn/nocodb-oidc`, a community fork that adds OIDC to the open-source build. It's a drop-in image and it's exactly the thing you want. It's also abandoned, and this is checkable in about two minutes: - -```bash -# what version of NocoDB is the fork based on? -curl -s https://raw.githubusercontent.com/brunostjohn/nocodb-oidc/main/packages/nocodb/package.json \ - | python -c "import sys,json; print(json.load(sys.stdin)['version'])" -# 0.255.2 - -# when was it last touched? -curl -s 'https://api.github.com/repos/brunostjohn/nocodb-oidc/commits?per_page=1' \ - | python -c "import sys,json; print(json.load(sys.stdin)[0]['commit']['author']['date'])" -# 2024-10-29 -``` - -A version from 2024 against an instance on a monthly release cadence is not a drop-in — it's a two-year downgrade, taking with it the schema migrations, the security fixes and the API behaviour your automations already depend on. The advice is only as good as the fork is maintained, and search results don't go stale when a repo does. - -**Check the fork's base version and last commit date before you plan around it.** Two curl calls, and they replace a rollback you'd otherwise do at 1 a.m. - -## What to do instead - -1. **Decide whether you need SSO in the app, or in front of it.** If what you want is "the internet can't reach this login form", a forward-auth proxy in front of the hostname gets you there with the open-source build. It's not single sign-on in the app, and the app still asks for its own password — I wrote up that construction, and the ingress trap that made my first attempt look successful while being a no-op, in [The Forward-Auth Gate That Verified Perfectly — and Wasn't Live](/posts/authentik-forward-auth-gate-wasnt-live/). -2. **Keep the app's own login and lock the door at the edge.** For an internal tool with a handful of users, this is what most people actually need. -3. **If a licensed build is genuinely required, price the migration, not just the licence.** Confirm which metadata store you're on first. - -## What I'd do differently - -**When a feature's availability is ambiguous, read it out of the image instead of the docs.** The docs describe intent; the image is the truth. One `docker exec ... printenv | grep -iE 'NC_|OIDC'`, one grep of the shipped bundle, and one throwaway container answered in twenty minutes what an afternoon of documentation reading couldn't. - -And a blunter rule that generalises past NocoDB: **an undocumented-but-enforced environment variable is a licensing boundary, not a feature flag.** If a variable rejects your startup when it's incomplete, but the documentation hides half its siblings, the missing piece is almost never configuration. - -## The result - -Five minutes to reproduce on a throwaway container, twice, with the container dead both times. One accidental finding I'd rather not have learned in production: an unauthenticated request to the SSO route is enough to stop the instance. - -The upside is that I stopped trying to make the app do SSO, put the decision in front of the hostname where it belongs, and can now answer "can NocoDB do SSO for free?" in one sentence instead of an afternoon. - ---- - -## Want this for your business? - -If you're evaluating self-hosted software, need single sign-on in front of internal tools, or want someone to check what your stack actually does versus what its documentation claims, that's the work I do. - -- **WhatsApp:** [011-797 2969](https://wa.me/60127972969) — tap to chat -- **Email:** [me@hoelee.com](mailto:me@hoelee.com?subject=Self-hosted%20SSO%20enquiry) -- **Website:** [hoelee.com](https://www.hoelee.com) - -I set up authentik single sign-on, self-hosted Docker stacks and reverse proxies for small businesses in Malaysia — and I write up what I learn while doing it. -```` - ---- - -## 文章 06 / 16：与其看供应商文档，不如直接读容器自带的 API 文档 -- **Slug**: `reading-a-containers-own-api-docs` -- **英文标题**: Read a Container's Own API Docs Instead of the Vendor's -- **分类**: `notes` | **发布日期**: 2025-10-01 -- **线上地址**: https://blog.hoelee.com/posts/reading-a-containers-own-api-docs/ (EN) | https://blog.hoelee.com/posts/zh/reading-a-containers-own-api-docs/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "与其看供应商文档，不如直接读容器自带的 API 文档" -description: "当 API 文档藏在登录墙后面、甚至根本不存在时，镜像本身就是唯一的事实来源：四条 Docker 命令，就能挖出 API 规范、配置契约，以及它真正监听的端口。" -pubDate: 2025-10-01 -updatedDate: 2026-09-26 -category: notes -tags: [docker, openapi, api, documentation, debugging, reverse-engineering] -ogImage: /og/zh-reading-a-containers-own-api-docs.jpg -banner: /banners/reading-a-containers-own-api-docs.jpg -draft: false ---- - -我需要对接一个供应商自托管的 API。他们的文档是有的，但挡在账号登录后面； -而我能看到的那一页，只描述了 API *能*做什么，没说该怎么配置。与此同时， -容器就安安静静地躺在我机器上——而答案就在它里面。 - -现在，凡是文档单薄、要登录、或者压根没有的镜像，我都会先跑这套命令。 -下面所有操作都是只读的，也不需要进入容器开 shell。 - -## 为什么这很重要 - -1. **供应商文档描述的是理想路径；容器描述的才是契约。** 确切的 env 变量名、 - 默认值、暴露的端口、是否需要配置文件——这些在营销页面上都不可靠。 -2. **很多镜像压根没有文档**——私有仓库、内部构建，或是别人拉了又推上去的 - 社区镜像。构件本身是唯一权威。 -3. **版本漂移是真实存在的。** 网站上的文档描述的是最新版本；而你正在跑的 - 镜像可能是两年前的了。读构件，了解的是你实际部署的那个东西。 - -## 1. 不运行镜像，直接读里面的文件 - -最有用的一招：覆盖 entrypoint，把镜像当成文件系统来用。 - -```bash -# what's in there? -docker run --rm --entrypoint ls -la /app - -# does it ship an API spec? -docker run --rm --entrypoint cat /app/public/openapi.yml | head -40 -``` - -第二条命令直接在镜像里返回了一份完整的 OpenAPI 3.1 规范，27 KB——每个端点、 -请求体 schema、认证模型，一应俱全。不用登录、不用账号、没有文档墙。 - -如果你还不知道文件布局，用 `--entrypoint find` 能覆盖更广： - -```bash -docker run --rm --entrypoint find / -maxdepth 3 \ - -name '*.yml' -o -name '*.yaml' -o -name '*.json' 2>/dev/null | head -30 -``` - -## 2. 从 bundle 里挖出配置契约 - -真正会让你卡住的往往是 env 变量名——一个没有默认值的必需变量，意味着容器 -启动时就挂掉，通常还带一条毫无帮助的错误信息。直接问源码吧： - -```bash -docker run --rm --entrypoint grep \ - -rhoE 'process\.env\.[A-Za-z_][A-Za-z0-9_]*' /app/dist | sort -u -``` - -``` -AUTH_SECRET -DEBUG -HOST -LOKI_HOST -METRICS_TOKEN -PORT -USER_TIMEOUT -WORKER_TIMEOUT -WORKER_PATH -``` - -九个变量名，整个配置面就清楚了。想看默认值，用带上下文的 grep 抓你关心的 -那一个： - -```bash -docker run --rm --entrypoint grep \ - -ohE '.{0,80}process\.env\.PORT.{0,80}' /app/dist/index.js -``` - -```js -serverPort: parseInt(process.env.PORT) || 3e3, -serverHost: process.env.HOST || "0.0.0.0", -workerTimeout: parseInt(process.env.WORKER_TIMEOUT) || 600*1e3, -``` - -`PORT` 默认是 3000，host 默认是 `0.0.0.0`，worker 每十分钟回收一次。一条命令 -就消掉了三个不确定的决策点。这套办法适用于任何 Node/Python 镜像；对 Go 或 -Rust 的二进制，同样的思路，只是把 grep 换成对二进制跑 `strings`。 - -## 3. 不拉镜像，直接查镜像配置 - -在慢速网络下，或者在决定下载 1 GB 镜像之前，你可以直接从 registry 读取镜像 -的元数据。本地连 Docker daemon 都不需要： - -```bash -REPO=synology/spreadsheet-api -TAG=3.4.1 - -TOKEN=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:$REPO:pull" \ - | python -c "import json,sys; print(json.load(sys.stdin)['token'])") - -# manifest (follow the platform entry if it's a multi-arch index) -curl -s -H "Authorization: Bearer ***" \ - -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \ - "https://registry-1.docker.io/v2/$REPO/manifests/$TAG" \ - | python -c "import json,sys; m=json.load(sys.stdin); print(m['config']['digest'])" -``` - -然后取回那个 config blob，把关心的字段打印出来： - -```bash -curl -s -H "Authorization: Bearer ***" \ - "https://registry-1.docker.io/v2/$REPO/blobs/" \ - | python -c " -import json,sys -c = json.load(sys.stdin)['config'] -print('Env: ', c.get('Env')) -print('Entrypoint:', c.get('Entrypoint')) -print('Cmd: ', c.get('Cmd')) -print('Workdir: ', c.get('WorkingDir')) -print('Ports: ', list((c.get('ExposedPorts') or {}).keys()))" -``` - -``` -Env: ['PATH=...', 'NODE_VERSION=22.18.0', 'WORKER_PATH=/app/dist/spreadsheet_worker.js'] -Entrypoint: ['docker-entrypoint.sh'] -Cmd: ['node', 'dist/index.js'] -Workdir: /app -Ports: [] -``` - -还没下载一个字节，就有两条有用信息跳出来了：这是个 Node 服务，而且它**没有 -声明任何暴露端口**——所以任何端口映射都得靠 `PORT` 变量，而不是 `EXPOSE`。 -既然来了，顺带也值得查一下：tag 列表会告诉你真实的版本历史。 - -```bash -curl -s "https://hub.docker.com/v2/repositories/$REPO/tags/?page_size=25" \ - | python -c "import json,sys; [print(t['name'], t['last_updated'][:10]) for t in json.load(sys.stdin)['results']]" -``` - -## 4. 找出它真正监听的端口 - -`EXPOSE` 是文档，不是行为。想知道进程真正绑定的端口，要看运行中的容器内部： - -```bash -docker exec cat /proc/net/tcp -``` - -端口以十六进制写在字段 2 里——字段 4 是 `0A` 表示 `LISTEN`。解码一下： - -```bash -docker exec sh -c \ - "awk 'NR>1 && \$4==\"0A\" {print \$2}' /proc/net/tcp" \ - | cut -d: -f2 | while read h; do printf '%d\n' "0x$h"; done -``` - -``` -3000 -``` - -然后不离开容器，确认它真的在响应： - -```bash -docker exec sh -c 'wget -qSO- -O- http://127.0.0.1:3000/ 2>&1 | head -5' -``` - -``` -HTTP/1.1 302 Found - location: /docs -``` - -重定向到 `/docs`——原来镜像一直在提供它自己的 Swagger UI。 - -## 5. 读启动日志，别把 shell 挂死 - -把 `docker run` 直接接到 `head` 或日志过滤器上，看起来无害，实际上会挂死： -进程一直占着管道，你的命令永远不会返回。改成后台运行，读日志，然后删掉它： - -```bash -docker run -d --name probe -e AUTH_SECRET=probe-only -sleep 8 -docker logs probe 2>&1 | head -20 -docker rm -f probe -``` - -``` -[02:06:17 UTC] INFO: Server listening at http://127.0.0.1:3000 -[02:06:17 UTC] INFO: Server listening at http://172.27.0.2:3000 -``` - -两行日志就是全部答案——而且它还告诉我：容器*有* `AUTH_SECRET` 就能正常启动， -*没有*就挂。这比我在前台运行、眼睁睁看着它崩溃后拿到的那段压缩过的堆栈信息 -有用得多。 - -## 这不能替代什么 - -如果供应商自己的文档存在，还是要读——通常比钻洞翻找更快，而且它携带镜像 -无法告诉你的东西，比如版本兼容性对照表。就拿这个镜像来说，供应商页面*确实* -发布了运行命令和必需的 `AUTH_SECRET`；我之所以先在镜像里找到它们，只是因为我 -没把页面往下滚够。两个都用：页面看意图，构件看你要部署的确切契约。 - -## 结果 - -四条命令——`cat` 出打包的规范、`grep` 出 env 变量、从 registry 读 config blob、 -`exec cat /proc/net/tcp` 找端口——把一场文档大海捞针，换成了：一份完整的 -OpenAPI 规范、全部九个 env 变量、默认端口、配置默认值，以及一个 Swagger UI -地址。没进容器开 shell，不用登录，什么都没改动。 - -这些命令最值的一次应用，就是 Synology 的 spreadsheet API 容器——完整案例、以及逼我动用这些命令的 -四个坑，写在[《Synology 的 Spreadsheet API 是一个容器，不是一个 API 端点》](/posts/synology-spreadsheet-api-is-a-container/)里。 - ---- - -*如果你正在对接一个自托管系统，而文档又戛然而止——这正是我为马来西亚中小 -企业做的活：[WhatsApp](https://wa.me/60127972969)、[邮件](mailto:me@hoelee.com?subject=API%20integration)， -或 [hoelee.com](https://hoelee.com)。* -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Read a Container's Own API Docs Instead of the Vendor's" -description: "When the API documentation is behind a login or doesn't exist, the image is still the source of truth: four Docker commands that extract the API spec, the config contract, and the port it listens on." -pubDate: 2025-10-01 -updatedDate: 2026-09-26 -category: notes -tags: [docker, openapi, api, documentation, debugging, reverse-engineering] -ogImage: /og/reading-a-containers-own-api-docs.jpg -banner: /banners/reading-a-containers-own-api-docs.jpg -draft: false ---- - -I needed to drive a vendor's self-hosted API. Their docs existed, but behind an account login, and the page I could reach described what the API *could* do rather than how to configure it. The container, meanwhile, was sitting right there on my machine — and it contained the answers. - -This is the set of commands I now run first for any image whose documentation is thin, gated, or missing. Everything here is read-only and needs no shell inside the container. - -## Why this matters - -1. **Vendor docs describe the happy path; the container describes the contract.** Exact env var names, defaults, exposed ports, whether a config file is required — none of that is reliably in a marketing page. -2. **Many images have no docs at all** — a private registry, an internal build, or a community image someone pulled and pushed. The artifact is the only authority. -3. **Version drift is real.** Docs on a website describe the latest release; the image you're running might be two years old. Reading the artifact tells you about the thing you actually deployed. - -## 1. Read files out of the image without running it - -The single most useful trick: override the entrypoint and treat the image as a filesystem. - -```bash -# what's in there? -docker run --rm --entrypoint ls -la /app - -# does it ship an API spec? -docker run --rm --entrypoint cat /app/public/openapi.yml | head -40 -``` - -That second command returned a full OpenAPI 3.1 specification, 27 KB, inside the image — complete with every endpoint, request body schema, and the authentication model. No login, no account, no gated docs. - -If you don't know the file layout yet, `--entrypoint find` covers more ground: - -```bash -docker run --rm --entrypoint find / -maxdepth 3 \ - -name '*.yml' -o -name '*.yaml' -o -name '*.json' 2>/dev/null | head -30 -``` - -## 2. Extract the configuration contract from the bundle - -Env var names are the part that will actually stop you — a required variable with no default means the container dies at startup, usually with an unhelpful error. Ask the source directly: - -```bash -docker run --rm --entrypoint grep \ - -rhoE 'process\.env\.[A-Za-z_][A-Za-z0-9_]*' /app/dist | sort -u -``` - -``` -AUTH_SECRET -DEBUG -HOST -LOKI_HOST -METRICS_TOKEN -PORT -USER_TIMEOUT -WORKER_TIMEOUT -WORKER_PATH -``` - -Nine names, and now you know the whole config surface. For a default value, grep with context around the one you care about: - -```bash -docker run --rm --entrypoint grep \ - -ohE '.{0,80}process\.env\.PORT.{0,80}' /app/dist/index.js -``` - -```js -serverPort: parseInt(process.env.PORT) || 3e3, -serverHost: process.env.HOST || "0.0.0.0", -workerTimeout: parseInt(process.env.WORKER_TIMEOUT) || 600*1e3, -``` - -`PORT` defaults to 3000, host to `0.0.0.0`, workers recycle after ten minutes. That's three uncertain decisions removed in one command. This works for any Node/Python image; for a Go or Rust binary the same idea applies with `strings` on the binary instead. - -## 3. Inspect the image config without pulling it - -On a slow link, or before you commit to a 1 GB download, you can read the image's metadata straight from the registry. No local Docker daemon needed: - -```bash -REPO=synology/spreadsheet-api -TAG=3.4.1 - -TOKEN=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:$REPO:pull" \ - | python -c "import json,sys; print(json.load(sys.stdin)['token'])") - -# manifest (follow the platform entry if it's a multi-arch index) -curl -s -H "Authorization: Bearer $TOKEN" \ - -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \ - "https://registry-1.docker.io/v2/$REPO/manifests/$TAG" \ - | python -c "import json,sys; m=json.load(sys.stdin); print(m['config']['digest'])" -``` - -Then fetch that config blob and print the interesting fields: - -```bash -curl -s -H "Authorization: Bearer $TOKEN" \ - "https://registry-1.docker.io/v2/$REPO/blobs/" \ - | python -c " -import json,sys -c = json.load(sys.stdin)['config'] -print('Env: ', c.get('Env')) -print('Entrypoint:', c.get('Entrypoint')) -print('Cmd: ', c.get('Cmd')) -print('Workdir: ', c.get('WorkingDir')) -print('Ports: ', list((c.get('ExposedPorts') or {}).keys()))" -``` - -``` -Env: ['PATH=...', 'NODE_VERSION=22.18.0', 'WORKER_PATH=/app/dist/spreadsheet_worker.js'] -Entrypoint: ['docker-entrypoint.sh'] -Cmd: ['node', 'dist/index.js'] -Workdir: /app -Ports: [] -``` - -Two useful facts fell out of that before downloading a byte: it's a Node service, and it declares **no exposed ports** — so any port mapping has to come from the `PORT` variable, not from `EXPOSE`. Also worth checking while you're there: the tags list tells you the real version history. - -```bash -curl -s "https://hub.docker.com/v2/repositories/$REPO/tags/?page_size=25" \ - | python -c "import json,sys; [print(t['name'], t['last_updated'][:10]) for t in json.load(sys.stdin)['results']]" -``` - -## 4. Find the port it actually listens on - -`EXPOSE` is documentation, not behaviour. To see what the process really bound to, look inside a running container: - -```bash -docker exec cat /proc/net/tcp -``` - -The port is in hex in field 2 — `0A` in field 4 means `LISTEN`. Decode it: - -```bash -docker exec sh -c \ - "awk 'NR>1 && \$4==\"0A\" {print \$2}' /proc/net/tcp" \ - | cut -d: -f2 | while read h; do printf '%d\n' "0x$h"; done -``` - -``` -3000 -``` - -Then confirm it answers, without leaving the container: - -```bash -docker exec sh -c 'wget -qSO- -O- http://127.0.0.1:3000/ 2>&1 | head -5' -``` - -``` -HTTP/1.1 302 Found - location: /docs -``` - -A redirect to `/docs` — the image was serving its own Swagger UI the whole time. - -## 5. Read startup logs without hanging your shell - -Piping `docker run` straight into `head` or a log filter looks harmless and will hang: the process keeps the pipe open, so your command never returns. Run it detached, read the logs, then remove it: - -```bash -docker run -d --name probe -e AUTH_SECRET=probe-only -sleep 8 -docker logs probe 2>&1 | head -20 -docker rm -f probe -``` - -``` -[02:06:17 UTC] INFO: Server listening at http://127.0.0.1:3000 -[02:06:17 UTC] INFO: Server listening at http://172.27.0.2:3000 -``` - -That's the whole answer in two lines — and it also told me the container starts fine *with* `AUTH_SECRET` and dies *without* it, which is a far better signal than the minified stack trace I got from running it in the foreground and watching it crash. - -## What this doesn't replace - -Read the vendor's own documentation when it exists — it's usually faster than spelunking, and it carries things an image can't tell you, like version compatibility tables. On this particular image, the vendor's page *did* publish the run command and the required `AUTH_SECRET`; I found them in the image first only because I hadn't scrolled far enough down the page. Use both: the page for intent, the artifact for the exact contract you're going to deploy. - -## The result - -Four commands — `cat` the bundled spec, `grep` the bundle for env vars, read the config blob from the registry, `exec cat /proc/net/tcp` for the port — replaced a documentation hunt with: a full OpenAPI spec, all nine env vars, the default port, the config defaults, and a Swagger UI URL. No shell in the container, no login, nothing mutated. - -The place this paid off hardest was Synology's spreadsheet API container — the worked example, including the four failures that made me reach for these commands in the first place, is in [The Synology Spreadsheet API Is a Container, Not an API Endpoint](/posts/synology-spreadsheet-api-is-a-container/). - ---- - -*If you're integrating a self-hosted system and the docs stop short, I do that work for small businesses in Malaysia — [WhatsApp](https://wa.me/60127972969), [email](mailto:me@hoelee.com?subject=API%20integration), or [hoelee.com](https://hoelee.com).* -```` - ---- - -## 文章 07 / 16：smartctl 退出码 32：专门跳过你最该看的硬盘的那个“错误” -- **Slug**: `smartctl-exit-code-32-skips-the-disks-that-matter` -- **英文标题**: smartctl Exit Code 32: The Code That Skips the Disks You Care About -- **分类**: `notes` | **发布日期**: 2026-04-14 -- **线上地址**: https://blog.hoelee.com/posts/smartctl-exit-code-32-skips-the-disks-that-matter/ (EN) | https://blog.hoelee.com/posts/zh/smartctl-exit-code-32-skips-the-disks-that-matter/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "smartctl 退出码 32：专门跳过你最该看的硬盘的那个“错误”" -description: "把 smartctl 的任何非零退出码当成读取失败的监控脚本，恰好会跳过属性已经逼近阈值的那些盘。退出码 32 不是错误，而是一段历史。" -pubDate: 2026-04-14 -updatedDate: 2026-09-29 -category: notes -tags: [smart, smartctl, unraid, monitoring, bash, disks, self-hosting] -ogImage: /og/zh-smartctl-exit-code-32-skips-the-disks-that-matter.jpg -banner: /banners/smartctl-exit-code-32-skips-the-disks-that-matter.jpg -draft: false ---- - -我的硬盘健康采集脚本连续几周都是\"全绿\"。每块盘都有温度、通电小时数和 SMART 结论，指标看起来完整——直到我数了一下只有 **4 块 SSD 里的 2 块**。 - -不是故障，是**消失了**。脚本认定这两块盘不存在。 - -## 问题出在把退出码当布尔值 - -采集脚本遍历块设备、对每块盘跑 `smartctl`、再写 Prometheus textfile 指标。错误处理看起来挺防御： - -```bash -for dev in /dev/sd?; do - if ! smartctl -A -H "$dev" > /tmp/smart.out 2>&1; then - continue # "盘不支持 SMART / 读不到" - fi - # 解析并输出指标 -done -``` - -`if ! cmd` 是布尔判断，但 `smartctl` 的退出状态**不是布尔值，是一个位域**。把位域当真假用，就是这次翻车的原因。 - -## 退出码 32 的真实含义 - -`man smartctl` 里这些位是独立累加的： - -| 位 | 值 | 含义 | -|---|---|---| -| 0 | 1 | 命令行解析失败 | -| 1 | 2 | 设备打开失败 / 无 IDENTIFY DEVICE 结构 | -| 2 | 4 | SMART 或 ATA 命令失败、校验和错误 | -| 3 | **8** | SMART 状态为 **DISK FAILING** | -| 4 | **16** | 有预失效属性 **<= 阈值** | -| 5 | **32** | SMART 状态 **OK**，但某些属性**曾经**低于阈值 | -| 6 | 64 | 设备错误日志中有记录 | -| 7 | 128 | 自检日志中有失败记录 | - -所以 `32` 的意思和\"读不到盘\"正好相反：**现在没问题，但它曾经踩过阈值。** 这正是应该持续盯着的那类盘——而我的脚本把它们丢掉了。 - -消失的两块盘，恰好是原始属性长期处在边缘值的那两块；退出码为 `0` 的\"健康盘\"被正常采集。**这个过滤器实际上筛选出了\"没有任何历史可报告\"的盘。** - -## 修法：把\"信息位\"掩掉 - -对监控来说，位 32 和 64 是信息；位 8 和 16 才是该告警的。掩掉信息位，只对剩下的做判断： - -```bash -smartctl -A -H -d sat "$dev" > /tmp/smart.out 2>&1 -rc=$? - -# fatal bits: 1 (parse), 2 (open), 4 (command/checksum), 8 (FAILING) -# informational bits: 32 (was below threshold in the past), 64 (error log has records) -fatal=$(( rc & ~(32 | 64) )) -if [ "$fatal" -ne 0 ]; then - echo "device $dev unreadable or failing (rc=$rc)" >&2 - continue -fi - -# emit the verdict AND the exit code, so the code itself is a metric -echo "disk_smart_exit_code{device=\"$dev\"} $rc" -echo "disk_smart_health{device=\"$dev\"} $(( rc & 8 ? 0 : 1 ))" -``` - -三点改变： - -1. **退出码变成数据，而不是控制流。** 它作为指标被导出，磁盘从 `0` → `32` → `64` 的漂移会变成趋势，而不是一次静默跳过。 -2. **信息位不再致命。** 有历史的盘被采集，这才是监控它们的目的。 -3. **NAS 上 `-d sat` 很关键。** 在 Synology DSM（以及某些 USB 桥接）上，设备类型不对时 `smartctl` 拿不到有用输出——症状同样是\"盘消失\"，原因却不同。 - -## 我下次会怎么做 - -- **面对有\"退出码位域\"文档的工具，永远不要写 `if ! cmd`。** 先把手册里的退出码那节读完。 -- **除了\"采集器有没有跑\"，还要数\"采到了几块盘\"。** 我的告警是\"采集器过期\"，真正的 bug 是\"采集器正常跑了，只报了 50% 的盘\"。一个 `disk_count` 指标就能立刻暴露。 -- **把\"这个设备没有数据\"本身当成可告警状态。** 缺失的序列是不可见的，所以它存活了好几周。 - -## 结果 - -采集器从 2 块可用盘变成 4 块，每块都报温度、通电小时数、SMART 状态和原始退出码——共 122 个 textfile 指标，包括我真正想要的 btrfs 错误计数。重新出现的那两块，正是长期处在边缘属性值的盘。 - -一个会跳过自己最坏信号的监控流水线，比没有监控更糟——因为它一直告诉你一切正常。 - -## 需要为你的业务做这个吗？ - -如果你在跑 NAS 或服务器机架，想让硬盘健康真正\"会通知你\"——SMART 属性、温度、btrfs/RAID 错误计数，并推到 Telegram 或邮件——这类自托管监控流水线我可以帮你搭。 - -**WhatsApp：[+60 12-797 2969](https://wa.me/60127972969)** · **邮箱：[me@hoelee.com](mailto:me@hoelee.com?subject=Disk%20health%20monitoring)** · **[hoelee.com](https://hoelee.com)** - -网站设计与开发是我的主业；服务器加固与自托管基础设施是它的另一半。 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "smartctl Exit Code 32: The Code That Skips the Disks You Care About" -description: "A disk-health collector that treats any non-zero smartctl exit as failure quietly skips exactly the drives with marginal attributes. Exit 32 is not an error — it is a history lesson." -pubDate: 2026-04-14 -updatedDate: 2026-09-29 -category: notes -tags: [smart, smartctl, unraid, monitoring, bash, disks, self-hosting] -ogImage: /og/smartctl-exit-code-32-skips-the-disks-that-matter.jpg -banner: /banners/smartctl-exit-code-32-skips-the-disks-that-matter.jpg -draft: false ---- - -My disk-health collector was green for weeks. Every drive reported a temperature, a -power-on-hours counter and a SMART verdict, and the textfile metrics looked complete -until I counted them: **two of the four SSDs were simply missing**. - -Not failing. Missing. The script had decided they didn't exist. - -## The script that "handled" errors - -The collector walks the block devices, runs `smartctl` against each one, and writes -Prometheus textfile metrics. The error handling looked defensive — the classic shell -shape: - -```bash -for dev in /dev/sd?; do - if ! smartctl -A -H "$dev" > /tmp/smart.out 2>&1; then - continue # "disk isn't SMART-capable / can't be read" - fi - # parse and emit metrics -done -``` - -`if ! cmd` is a boolean. `smartctl`'s exit status is **not** a boolean — it's a -bitfield, and treating a bitfield as true/false is where this goes wrong. - -## What exit code 32 actually means - -From `man smartctl`, the bits are cumulative and independent: - -| Bit | Value | Meaning | -|---|---|---| -| 0 | 1 | Command line did not parse | -| 1 | 2 | Device open failed, or no IDENTIFY DEVICE structure | -| 2 | 4 | A SMART or ATA command failed / checksum error in a SMART structure | -| 3 | **8** | SMART status check returned **DISK FAILING** | -| 4 | **16** | Pre-fail attributes found **<= threshold** | -| 5 | **32** | SMART status **OK**, but some attributes were **<= threshold at some time in the past** | -| 6 | 64 | Device error log contains records of errors | -| 7 | 128 | Device self-test log contains records of errors | - -So exit `32` is the opposite of "unreadable". It means: *the disk is fine right now, -and it has been below a threshold before.* That is precisely the signal you want to -keep an eye on — and my collector was throwing it away. - -The two disks that vanished were the two whose raw attributes sit at marginal values. -The healthy disks exited `0` and got collected. **The filter was selecting for the -disks with nothing to report.** - -## The fix: mask the informational bits - -Exit bits 32 and 64 are informational for monitoring purposes; bits 8 and 16 are the -ones that deserve an alert. Mask them off and act on what's left: - -```bash -smartctl -A -H -d sat "$dev" > /tmp/smart.out 2>&1 -rc=$? - -# fatal bits: 1 (parse), 2 (open), 4 (command/checksum), 8 (FAILING) -# informational bits: 32 (was below threshold in the past), 64 (error log has records) -fatal=$(( rc & ~(32 | 64) )) -if [ "$fatal" -ne 0 ]; then - echo "device $dev unreadable or failing (rc=$rc)" >&2 - continue -fi - -# emit the verdict AND the exit code, so the code itself is a metric -echo "disk_smart_exit_code{device=\"$dev\"} $rc" -echo "disk_smart_health{device=\"$dev\"} $(( rc & 8 ? 0 : 1 ))" -``` - -Three things changed the value of this collector: - -1. **The exit code became data, not control flow.** It's exported as a metric, so a - disk drifting from `0` → `32` → `64` shows up as a trend instead of a silent skip. -2. **The informational bits stopped being fatal.** Disks with history are collected, - which is the whole point of monitoring them. -3. **`-d sat` matters on a NAS.** On Synology DSM (and some USB bridges), a SATA disk - behind the wrong device type returns nothing useful — the same "missing disk" - symptom with a different cause. - -## What I'd do differently - -- **Never write `if ! cmd` against a tool that documents an exit-code bitfield.** Read - the exit-code section of the man page before using the return value as a boolean. -- **Count what you collected, not just whether the collector ran.** My alert was on - "collector stale"; the real bug was "collector ran fine and reported 50% of the - disks". A single `disk_count` metric would have surfaced it immediately. -- **Treat "no data for this device" as its own alertable state.** A missing series is - invisible, which is exactly why it survived for weeks. - -## The result - -The collector went from 2 usable disks to 4, with every drive reporting temperature, -power-on hours, SMART status and its raw exit code — 122 textfile metrics in total, -including the btrfs error counters I actually wanted. The two disks that reappeared -are the two that were already sitting at marginal attribute values. - -A monitoring pipeline that skips its own worst signals is worse than no pipeline, -because it tells you everything is fine. - -## Want this for your business? - -If you're running a NAS or a server rack and want disk health that actually pages you -before a drive dies — SMART attributes, temperatures, btrfs/RAID error counters, and -alerts to Telegram or email — that's the kind of self-hosted monitoring pipeline I set up. - -**WhatsApp: [+60 12-797 2969](https://wa.me/60127972969)** · **Email: [me@hoelee.com](mailto:me@hoelee.com?subject=Disk%20health%20monitoring)** · **[hoelee.com](https://hoelee.com)** - -Website design and development is my main line of work; server hardening and -self-hosted infrastructure is the other half of it. -```` - ---- - -## 文章 08 / 16：确定性的活，别再用 Agent Prompt 去跑 -- **Slug**: `stop-writing-agent-prompts-for-deterministic-work` -- **英文标题**: Stop Writing Agent Prompts for Deterministic Work -- **分类**: `notes` | **发布日期**: 2026-10-01 -- **线上地址**: https://blog.hoelee.com/posts/stop-writing-agent-prompts-for-deterministic-work/ (EN) | https://blog.hoelee.com/posts/zh/stop-writing-agent-prompts-for-deterministic-work/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "确定性的活，别再用 Agent Prompt 去跑" -description: "我有一个定时任务连续失败 29 次，原因是我在一个只需要字符串比较的循环里塞了个模型。这是我现在用的规则。" -pubDate: 2026-10-01 -category: notes -blueprint: ai-agent-automation -blueprintLayer: L1-deterministic-orchestration -tags: ["loop-engineering", "cron", "ai-agents", "automation"] -ogImage: "/og/zh-stop-writing-agent-prompts-for-deterministic-work.jpg" -banner: "/banners/stop-writing-agent-prompts-for-deterministic-work.jpg" -draft: false ---- - -我有一个定时任务连续失败了 29 次。没人告诉我，因为它被设计成只在有东西可报的时候才开口——而一个还没来得及开口就崩掉的运行，看起来和一个「什么都没发现」的运行一模一样。 - -那个任务是个版本检查：问 npm 某个我常用的 CLI 最新发布版本是多少，和本机装的版本比一下，有更新就告诉我。整个任务就这些。它就是两个字符串加一次比较。 - -我当初把它做成了 agent 任务，因为那阵子我脑子里的默认模式就是这个。每次 tick，模型读一遍注入的脚本输出，然后判断要不要写一条通知。它正常跑了一阵子。后来我改了全局默认模型，而这个任务会记录自己创建时的模型，于是它干脆整个不跑了，打印一行 `[drift_skip]` 就退出。连续 29 次。 - -修法不是写一个更好的 prompt，而是把模型从循环里删掉。 - -## 规则 - -**如果停止条件是一次比较——一个字符串、一个数字、一个状态码——那么循环里的模型只能带来新的失败模式。** - -「`3.8.50` 是不是比 `3.8.50` 新」这件事里没有任何判断可做。模型并没有在决定代码决定不了的事；它带来的是一条依赖（一条模型路由、一个提供商、一份配置快照、一张 token 账单）和一种全新的出错方式。所以我把它重写成 125 行 Python，每条分支都有明确出口，再把任务切成脚本模式：脚本的 stdout 直接作为通知内容，整条路径上没有任何 LLM。 - -从此一路干净。Anthropic 自己的建议也是一句话——*确定性的活用脚本，跑脚本比让模型去推理步骤便宜*——但对我来说成本从来不是重点，可靠性才是。 - -## 模型该用在哪 - -我跑的 11 个循环里，现在有 8 个整条路径上没有模型。用模型的那 3 个，处理的都是我确实写不成代码的东西： - -- **这封邮件重不重要？** 一个邮箱里同时有促销、DMARC 报告、容器告警，和一位来问报价的客户。抓邮件是确定性的，给它排优先级不是。 -- **这张店铺优惠券是不是快超支了？** 算术很简单；判断一个发现对我已经承诺出去的钱意味着什么，不简单。 -- **这个项目是不是卡住了？** 读两个 markdown 文件是脚本的活。发现「计划已经三周没动过」不是。 - -我给新循环做的测试是：**写下这个检查在成功和失败时分别会打印什么。** 如果两个答案都是一个数字或一个字符串，那它就是脚本。如果答案是「要看它说了什么」，那它需要模型——而这时候它就需要一个我比 agent 更信任的验证器，因为我已经无法预测输出了。 - -## 真正咬到我的是哪一部分 - -失败模式不是 drift，是**沉默**。 - -一个只在出事时才开口的循环，是我唯一能忍受的那种——我不想让 11 个任务每天给我发状态报告。但「健康所以安静」和「崩了所以来不及开口」，从我手机这端看是同一个现象。29 次运行的证据一直躺在那个任务的执行日志里；我却一直在相信「没有通知」这件事本身，而没去读历史。 - -所以现在这套脚本都多了一条出口：退出码 0 表示*量过了，这是结果*，退出码 1 表示*我量不了*——而退出码 1 永远不静默。配置缺失、状态文件解析失败、没预料到的异常：全都打印一行并退出 1。就这一处改动，本该让这个 bug 在第 1 次就暴露，而不是第 29 次。 - -如果你有一个循环「已经安静一阵子」了，先去翻它最近十次运行，再下结论说它正常。这条建议花了我三个星期才换到。 - -## 简短版 - -- 确定性的检查 → 脚本。判断 → 模型。别为了方便把两者混在一起。 -- 给「成功则静默」这条约定开一个例外：「我量不了」必须永远大声。 -- 去读 runner 自己的执行历史，而不是只看你收到的通知。 - -想给你的网站或网店也做一套这样的东西——可用性、证书、备份、以及只在真的有事时才 ping 你的邮件？**[WhatsApp +60 12-797 2969](https://wa.me/60127972969)** · **[me@hoelee.com](mailto:me@hoelee.com?subject=Self-hosted%20monitoring)** · **[hoelee.com](https://hoelee.com)** -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Stop Writing Agent Prompts for Deterministic Work" -description: "One of my scheduled jobs failed 29 times in a row because I put a model in a loop that only needed a string comparison. Here is the rule I use now." -pubDate: 2026-10-01 -category: notes -blueprint: ai-agent-automation -blueprintLayer: L1-deterministic-orchestration -tags: ["loop-engineering", "cron", "ai-agents", "automation"] -ogImage: "/og/stop-writing-agent-prompts-for-deterministic-work.jpg" -banner: "/banners/stop-writing-agent-prompts-for-deterministic-work.jpg" -draft: false ---- - -One of my scheduled jobs failed 29 times in a row. Nobody told me, because -it was designed to speak only when it had something to report — and a run -that dies before it can speak looks exactly like a run that found nothing -wrong. - -The job was a version check: ask npm for the latest published version of a -CLI I use, compare it with the one installed here, and tell me if there is a -newer one. That is the entire task. It is two strings and an inequality. - -I had built it as an agent job, because that was the pattern I had in my -head at the time. Every tick, the model would read the injected script -output and decide whether to write a notification. It worked for a while. -Then my global default model changed, and the job — which records the model -it was created with — stopped running entirely. It printed `[drift_skip]` -and exited. Twenty-nine times. - -The fix was not a better prompt. It was deleting the model from the loop. - -## The rule - -**If the stop condition is a comparison — a string, a number, a status code — a model in that loop can only add failure modes.** - -There is no judgment to make in "is `3.8.50` newer than `3.8.50`". The model -was not deciding anything the code could not; it was adding a dependency (a -model route, a provider, a config snapshot, a token bill) and a new way to -fail. So I rewrote the checker as 125 lines of Python with an explicit exit -path for every branch, and flipped the job to script mode: the script's -stdout is delivered as the notification, with no LLM anywhere in the path. - -It has run clean ever since. Anthropic's own guidance says the same thing in -one line — *use scripts for deterministic work; running a script is cheaper -than reasoning through the steps* — but the cost was never the interesting -part for me. The reliability was. - -## Where the model does earn its place - -Of the eleven loops I run, eight now have no model in the path. The three -that do all handle something I genuinely cannot express as code: - -- **Is this email important?** A mailbox produces promos, DMARC reports, container alerts and a customer asking for a quote. Fetching mail is deterministic. Ranking it is not. -- **Is this shop voucher about to overspend?** The arithmetic is trivial; deciding what a finding means for money I have already committed is not. -- **Is this project stalled?** Reading two markdown files is a script's job. Noticing that the plan has not moved in three weeks is not. - -The test I apply to a new loop: **write down what the check would print on -success and on failure.** If both answers are a number or a string, it is a -script. If the answer is "it depends what it says", it needs a model — and -then it needs a verifier I trust more than the agent, because I am no longer -able to predict the output. - -## The part that actually bit me - -The failure mode was not the drift. It was the **silence**. - -A loop that speaks only when something is wrong is the only kind of loop I -can tolerate — I do not want a daily status report from eleven jobs. But -"silent because healthy" and "silent because it crashed before it could -speak" are the same observation from my side of the phone. Twenty-nine runs -of evidence were sitting in the job's execution log the whole time; I was -trusting the absence of a notification instead of reading the history. - -So every script in the fleet now has a second exit path: exit 0 means -*measured, here is what I found*, and exit 1 means *I could not measure* — -and exit 1 is never silent. Missing config, unparseable state, an unexpected -exception: all of them print a line and exit 1. That one change would have -surfaced this bug on the first run instead of the twenty-ninth. - -If you have a loop that has been "quiet for a while", go and read its last -ten runs before you conclude it is working. That advice cost me three weeks. - -## The short version - -- Deterministic check → script. Judgment → model. Do not mix them for convenience. -- Give the silent-on-success convention one exception: "I could not measure" must always be loud. -- Read the runner's own execution history, not just your notifications. - -Want a version of this for your own site or shop — uptime, certificates, -backups, mail that only pings you when it matters? -**[WhatsApp +60 12-797 2969](https://wa.me/60127972969)** · -**[me@hoelee.com](mailto:me@hoelee.com?subject=Self-hosted%20monitoring)** · -**[hoelee.com](https://hoelee.com)** -```` - ---- - -## 文章 09 / 16：密码正确，Synology API 登录却返回 401，问题出在哪 -- **Slug**: `synology-api-401-with-the-correct-password` -- **英文标题**: Why Your Synology API Login Returns 401 With the Correct Password -- **分类**: `notes` | **发布日期**: 2025-09-10 -- **线上地址**: https://blog.hoelee.com/posts/synology-api-401-with-the-correct-password/ (EN) | https://blog.hoelee.com/posts/zh/synology-api-401-with-the-correct-password/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "密码正确，Synology API 登录却返回 401，问题出在哪" -description: "两种原因，同一个报错：Office Suite API 对 TLS 握手失败和受 2FA 保护的账号返回完全相同的错误。三条命令就能区分，不必折腾一下午。" -pubDate: 2025-09-10 -updatedDate: 2026-09-26 -category: notes -tags: [synology, dsm, rest-api, tls, 2fa, debugging, authentication] -ogImage: /og/zh-synology-api-401-with-the-correct-password.jpg -banner: /banners/synology-api-401-with-the-correct-password.jpg -draft: false ---- - -你搭好了 Synology 的 Office Suite API，把凭据发过去，得到的却是： - -```json -{"error":"Unauthorized"} -``` - -然后你用这组密码登录 DSM 验证——一切正常。你重新输入一遍，改用 HTTP 而不是 HTTPS，怀疑是不是账号缺了什么找不到的权限。这些全都是白费功夫，因为在这套 API 上，密码正确却返回 `401` 的原因恰好只有两个，而这两个都不是密码的问题。 - -## 原因一：`host` 字段必须是一个名字，且证书要能被容器接受 - -这个 API 是自配置的：每次登录调用时，由你来告诉*它*要对哪台 DSM 进行认证。 - -```bash -curl -s -X POST http://:8791/spreadsheets/authorize \ - -H 'Content-Type: application/json' \ - -d '{"username":"acct","password":"pw","host":"192.168.1.1:5001","protocol":"https"}' -``` - -```json -{"error":"Unauthorized"} -``` - -代理会与那个 `host` 自行完成 TLS 握手。裸 IP 地址通常拿到的证书是为某个主机名签发的，于是握手失败——而失败的握手会被报告为 `Unauthorized`，和密码错误一模一样。响应里没有任何线索指向 TLS。 - -同样的请求、同样的凭据，把 `host` 换成一个持有有效证书的名字： - -```bash --d '{"username":"acct","password":"pw","host":"cloud.example.com","protocol":"https"}' -``` - -```json -{"token":"eyJhbGciOi...","host":"cloud.example.com"} -``` - -修复方法就这么多。**规则：`host` 的值必须是一个 FQDN，且其证书能被容器接受；如果端口不是该协议默认端口，也要一并写上。** - -## 原因二：双重认证在这里永远行不通 - -第二个原因是结构性的。登录的 schema 只有四个字段，没有一次性验证码字段： - -```yaml -AuthorizationBody: - properties: - username: {type: string} - password: {type: string} - host: {type: string} - protocol: {type: string} -``` - -没有 OTP，没有应用专用密码，也没有设备令牌交换。启用了 2FA 的账号，无论在哪个 `host` 上、无论密码多正确，都会永远返回 `401`。请改用关闭了 2FA 的专用服务账号，并把权限范围限定在它需要的文件夹上。 - -## 三条命令区分二者 - -别靠猜——用不了一分钟就能把两种原因分开。 - -**1. `host` 是否可达，并且是否为该名字提供了有效证书？** - -```bash -curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://cloud.example.com/ -``` - -非零的 `ssl_verify_result` 指向原因一。（在信任该证书的机器上，同一命令返回 `0`。） - -**2. API 到底有没有应答？它对乱写的垃圾请求是不是也这么拒绝？** - -```bash -curl -s -o - -w '\n%{http_code}\n' -X POST http://:8791/spreadsheets/authorize \ - -H 'Content-Type: application/json' \ - -d '{"username":"nobody","password":"wrong","host":"cloud.example.com","protocol":"https"}' -``` - -```json -{"error":"Unauthorized"} -401 -``` - -和你失败的那次调用输出一样——这正是关键。它证明服务是活的，并且 `401` 是它对*所有*认证失败的通用回答，TLS 失败也不例外。如果容器宕了，你得到的是连接错误而不是 `401`，所以这一步也能排除"API 没在运行"。 - -**3. 一个关闭了 2FA 的账号在好用的 `host` 上能成功吗？** 如果非 2FA 服务账号能用而你的管理员账号不行，那就是原因二。 - -## 相关的坑：几周后才冒出来的 `401` - -你拿到的 token 是一个有效期 28 天的 JWT，但它绑定在 DSM 会话上。DSM 重启，或该账号被强制登出，都会让它提前失效。所以昨天还能用的调用今天返回 `401`，意思是"重新认证"，而不是"我的凭据变了"——先检查 token 的年龄，再去找配置回归的问题。 - -而且登录进去之后，别把下一道墙和这一道搞混：`403 Permission denied` 表示文件存在但账号拿不到（通常是文件放在个人的 `My Drive` 主目录里，其他 DSM 账号都访问不到）；`404 Spreadsheet not found` 则表示 ID 写错了。不同的问题，不同的修法。 - -完整的搭建过程——容器、compose 文件、可用的调用，以及我踩过的四个坑——都在 [Synology Spreadsheet API 是一个容器，不是一个 API 端点](/posts/synology-spreadsheet-api-is-a-container/) 里。 - ---- - -*我为马来西亚的中小企业构建自托管的集成和内部工具——[WhatsApp](https://wa.me/60127972969) 或 [邮件](mailto:me@hoelee.com?subject=Synology%20API%20integration)。更多信息见 [hoelee.com](https://hoelee.com)。* -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Why Your Synology API Login Returns 401 With the Correct Password" -description: "Two causes, one error: the Office Suite API reports a failed TLS handshake and a 2FA-protected account identically. How to tell them apart in three commands instead of an afternoon." -pubDate: 2025-09-10 -updatedDate: 2026-09-26 -category: notes -tags: [synology, dsm, rest-api, tls, 2fa, debugging, authentication] -ogImage: /og/synology-api-401-with-the-correct-password.jpg -banner: /banners/synology-api-401-with-the-correct-password.jpg -draft: false ---- - -You set up Synology's Office Suite API, you send your credentials, and you get: - -```json -{"error":"Unauthorized"} -``` - -You then verify the password by logging into DSM with it — works fine. You retype it, you try HTTP instead of HTTPS, you wonder whether the account needs a permission you can't find. All of it is wasted motion, because on this API a `401` with a correct password has exactly two causes, and neither of them is the password. - -## Cause 1: the `host` field must be a name with a certificate the container accepts - -The API is self-configuring: you tell *it* which DSM to authenticate against, on every sign-in call. - -```bash -curl -s -X POST http://:8791/spreadsheets/authorize \ - -H 'Content-Type: application/json' \ - -d '{"username":"acct","password":"pw","host":"192.168.1.1:5001","protocol":"https"}' -``` - -```json -{"error":"Unauthorized"} -``` - -The proxy performs its own TLS handshake with that `host`. A bare IP address usually presents a certificate issued for a hostname, so the handshake fails — and a failed handshake is reported as `Unauthorized`, identically to a wrong password. Nothing in the response hints at TLS. - -Same request, same credentials, `host` changed to a name with a valid certificate: - -```bash --d '{"username":"acct","password":"pw","host":"cloud.example.com","protocol":"https"}' -``` - -```json -{"token":"eyJhbGciOi...","host":"cloud.example.com"} -``` - -That is the whole fix. **Rule: the `host` value must be an FQDN whose certificate the container will accept, with the port included if it isn't the default for the scheme.** - -## Cause 2: two-factor authentication can never work here - -The second cause is structural. The sign-in schema has four fields and no one-time-code field: - -```yaml -AuthorizationBody: - properties: - username: {type: string} - password: {type: string} - host: {type: string} - protocol: {type: string} -``` - -No OTP, no app password, no device-token exchange. An account with 2FA enabled returns `401` on every host, with every correct password, forever. Use a dedicated service account with 2FA off, scoped to the folders it needs. - -## Telling them apart in three commands - -Don't guess — separate the two in under a minute. - -**1. Is the host reachable and serving a valid certificate for that name?** - -```bash -curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://cloud.example.com/ -``` - -A non-zero `ssl_verify_result` points at cause 1. (From a machine that trusts the cert, the same command returns `0`.) - -**2. Does the API answer at all, and does it reject garbage the same way?** - -```bash -curl -s -o - -w '\n%{http_code}\n' -X POST http://:8791/spreadsheets/authorize \ - -H 'Content-Type: application/json' \ - -d '{"username":"nobody","password":"wrong","host":"cloud.example.com","protocol":"https"}' -``` - -```json -{"error":"Unauthorized"} -401 -``` - -Same output as your failing call — which is the point. It proves the service is alive and that `401` is its generic answer for *every* authentication failure, TLS included. A container that is down gives you a connection error, not a `401`, so this also rules out "the API isn't running". - -**3. Does an account with 2FA off succeed against the good host?** If a non-2FA service account works and your admin account doesn't, you have cause 2. - -## The related trap: a `401` that appears weeks later - -The token you get is a JWT valid for 28 days, but it's bound to a DSM session. A DSM restart, or a forced logout of that account, invalidates it early. So a `401` on a call that worked yesterday means "re-authenticate", not "my credentials changed" — check the token's age before you go looking for a config regression. - -And once you're in, don't confuse the next wall with this one: `403 Permission denied` means the file exists but the account can't have it (usually a file sitting in a personal `My Drive` home folder, unreachable by any other DSM account), while `404 Spreadsheet not found` means the ID is wrong. Different problems, different fixes. - -The full setup — the container, the compose file, the working calls, and the four traps I hit getting there — is in [The Synology Spreadsheet API Is a Container, Not an API Endpoint](/posts/synology-spreadsheet-api-is-a-container/). - ---- - -*I build self-hosted integrations and internal tools for small businesses in Malaysia — [WhatsApp](https://wa.me/60127972969) or [email](mailto:me@hoelee.com?subject=Synology%20API%20integration). More at [hoelee.com](https://hoelee.com).* -```` - ---- - -## 文章 10 / 16：NocoDB 附件字段怎么改都不生效，和一个漏网的正则 -- **Slug**: `the-nocodb-attachment-that-wouldnt-update` -- **英文标题**: The NocoDB Attachment That Wouldn't Update (and a Regex That Missed) -- **分类**: `notes` | **发布日期**: 2026-02-04 -- **线上地址**: https://blog.hoelee.com/posts/the-nocodb-attachment-that-wouldnt-update/ (EN) | https://blog.hoelee.com/posts/zh/the-nocodb-attachment-that-wouldnt-update/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "NocoDB 附件字段怎么改都不生效，和一个漏网的正则" -description: "回填 300 行数据时踩到的两个小坑：NocoDB 在附件保留 id 的情况下会无视 path 的改动，以及一个 lookahead 正则悄悄跳过了没有扩展名的 URL。" -pubDate: 2026-02-04 -updatedDate: 2026-09-10 -category: notes -blueprint: sme-digital-suite -blueprintLayer: L3-entitlement-backoffice -tags: [nocodb, backfill, python, regex, data-pipeline] -ogImage: /og/zh-the-nocodb-attachment-that-wouldnt-update.jpg -banner: /banners/the-nocodb-attachment-that-wouldnt-update.jpg ---- - -在 NocoDB 里回填 300 行数据的经历，暴露了两个小坑，每一个都耗掉了我一段时间。 -它们都属于那种「只有当你修的数据恰好接近、但又没完全符合代码假设的形状时」 -才会咬你的 bug。 - -## 坑 1：Attachment 改了 `path` 没用，只要你还留着 `id` - -我想把一个 Attachment 列里的缩略图 URL 全部换成高清版。直觉上就是去改这个 -列存的 JSON： - -```python -img = json.loads(row["image"]) # [{"id": "...", "path": "...", "signedPath": "..."}] -img[0]["path"] = new_fullsize_url -img[0]["signedPath"] = new_fullsize_url -``` - -它看起来成了——PATCH 返回成功。但把行读回来，每个 `path` 还是旧的缩略图。 - -元凶是：这个附件带着一个 NocoDB 生成的 `id`，只要它还在，NocoDB 就会**按这个 -id 去解析附件**，然后心安理得地无视新的 `path`。我以为自己在更新的那个字段， -其实是只读的，只要 `id` 还在。 - -修复的办法是丢掉服务端分配的字段，让 NocoDB 从头重新注册这个附件： - -```python -img[0] = { - "path": new_fullsize_url, - "mimetype": "image/jpeg", - "title": img[0].get("title", "image.jpg"), -} -# 不要 "id"，也不要 "signedPath" —— NocoDB 会重新分配 -``` - -没有 `id` 之后，NocoDB 会根据 `path` 注册一个新附件，并生成一个新的签名 URL。 -这下值才真正写进去了。 - -**规则：** 如果 NocoDB 的 Attachment 改来改去都不生效，检查一下 `id` 字段。 -把它剥掉、重新注册，而不是原地打补丁。 - -## 坑 2：一个会悄悄跳过「无扩展名 URL」的 lookahead 正则 - -为了拼出高清 URL，我需要去掉 `_progressive_thumbnail` 这个后缀。第一版我用了 -一个 lookahead： - -```python -clean = re.sub(r"_progressive_thumbnail(?=\.\w+$)", "", url) -``` - -这对这种情况有效： - -``` -...70776f54_progressive_thumbnail.jpg → ...70776f54.jpg ✅ -``` - -但 Carousell 有**两种** URL 形态。有些缩略图以光秃秃的后缀结尾，**没有扩展名**： - -``` -...f94af259_progressive_thumbnail → (lookahead 匹配不上) ❌ -``` - -因为 lookahead 要求结尾是 `.\w+`，第二种形态就漏掉了——300 行里有 25 行。 -这个正则「没报错」，它只是在那部分数据上什么都没做。 - -更稳的写法是直接 `replace`，两种形态都能覆盖： - -```python -clean = url.replace("_progressive_thumbnail", "") -``` - -**规则：** 当你要规范化一个有多种真实形态的字符串时，除非你已经把所有形态都 -枚举清楚了，否则优先用 `replace()` 而不是正则。一个悄悄跳过的正则，比一个 -直接报错的更糟。 - ---- - -这里头没什么高深的东西。但这两类都是那种**只有在真实数据里才会现形**的无聊 -bug——一个 API 信誓旦旦说更新成功了的附件，一个匹配了 91% 行、却对剩下 9% -一声不吭的正则。如果你给 NocoDB 回填数据时总觉得「只生效了一半」，先来查这两 -个地方。 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "The NocoDB Attachment That Wouldn't Update (and a Regex That Missed)" -description: "Two small gotchas from backfilling 300 rows: NocoDB ignores a path change when the attachment keeps its id, and a lookahead regex silently skipped URLs without a file extension." -pubDate: 2026-02-04 -updatedDate: 2026-09-10 -category: notes -blueprint: sme-digital-suite -blueprintLayer: L3-entitlement-backoffice -tags: [nocodb, backfill, python, regex, data-pipeline] -ogImage: /og/the-nocodb-attachment-that-wouldnt-update.jpg -banner: /banners/the-nocodb-attachment-that-wouldnt-update.jpg ---- - -Backfilling 300 rows in NocoDB surfaced two small gotchas that each ate a chunk of -time. Both are the kind of thing that only bites you when the data you're fixing -is *almost* — but not quite — in the shape the code assumes. - -## Gotcha 1: updating an Attachment's `path` doesn't matter if you keep the `id` - -I wanted to swap every thumbnail URL in an Attachment column for its full-size -version. The obvious move is to edit the JSON the column stores: - -```python -img = json.loads(row["image"]) # [{"id": "...", "path": "...", "signedPath": "..."}] -img[0]["path"] = new_fullsize_url -img[0]["signedPath"] = new_fullsize_url -``` - -It looked like it worked — the PATCH returned success. But reading the rows back, -every `path` was still the old thumbnail. - -The culprit: the attachment carries an NocoDB-generated `id`, and as long as it's -present, NocoDB resolves the attachment *by that id* and happily ignores the new -`path`. The field I thought I was updating was effectively read-only while the `id` -was still there. - -The fix is to drop the server-assigned fields and let NocoDB re-register the -attachment from scratch: - -```python -img[0] = { - "path": new_fullsize_url, - "mimetype": "image/jpeg", - "title": img[0].get("title", "image.jpg"), -} -# no "id", no "signedPath" — NocoDB assigns fresh ones -``` - -With `id` gone, NocoDB registers a new attachment from the `path` and generates a -fresh signed URL. The value finally sticks. - -**Rule:** if a NocoDB Attachment edit isn't sticking, check for the `id` field. -Strip it and re-register instead of patching in place. - -## Gotcha 2: a lookahead regex that silently skipped extension-less URLs - -To build the full-size URL I needed to strip the `_progressive_thumbnail` suffix. -My first pass used a lookahead: - -```python -clean = re.sub(r"_progressive_thumbnail(?=\.\w+$)", "", url) -``` - -That works for: - -``` -...70776f54_progressive_thumbnail.jpg → ...70776f54.jpg ✅ -``` - -But Carousell serves *two* URL shapes. Some thumbnails end with the bare suffix -and **no extension**: - -``` -...f94af259_progressive_thumbnail → (lookahead doesn't match) ❌ -``` - -Because the lookahead requires `.\w+` at the end, the second shape slipped through -unmatched — 25 of my 300 rows. The regex "worked," it just silently didn't do -anything on that slice of the data. - -The robust version is a plain replacement, which handles both: - -```python -clean = url.replace("_progressive_thumbnail", "") -``` - -**Rule:** when normalizing a string that has multiple real-world shapes, prefer -`replace()` over a regex unless you've enumerated every shape. A regex that -silently skips is worse than one that fails loudly. - ---- - -Nothing here is exotic. But both are the *boring* kind of bug that only reveals -itself in real data — the attachment the API swears it updated, the regex that -matches 91% of the rows and says nothing about the other 9%. If backfilling data -into NocoDB ever feels like it's half-working, these two are where I'd look first. -```` - ---- - -## 文章 11 / 16：CodeIgniter 4.6 升级到 4.7：官方指南没说到的破坏性变更 -- **Slug**: `upgrading-codeigniter-46-to-47` -- **英文标题**: Upgrading CodeIgniter 4.6 to 4.7: The Breaking Changes the Guide Misses -- **分类**: `notes` | **发布日期**: 2026-09-27 -- **线上地址**: https://blog.hoelee.com/posts/upgrading-codeigniter-46-to-47/ (EN) | https://blog.hoelee.com/posts/zh/upgrading-codeigniter-46-to-47/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "CodeIgniter 4.6 升级到 4.7：官方指南没说到的破坏性变更" -description: "CodeIgniter 4.7 官方升级指南列了八项破坏性变更。真正让我应用致命崩溃的那两项不在里面——而且它们属于同一类问题。" -pubDate: 2026-09-27 -category: notes -tags: [codeigniter, php, upgrade, composer, breaking-changes, config, framework] -ogImage: /og/zh-upgrading-codeigniter-46-to-47.jpg -banner: /banners/upgrading-codeigniter-46-to-47.jpg -draft: false ---- - -我把一个 CodeIgniter 4 应用从 4.6.3 升到了 4.7.4。我读了升级指南，把每一项文档化的破坏性变更都对着代码库核了一遍，确认没有一项影响到我。然后我跑了一下应用，它致命崩溃了两次。 - -两个致命错误属于同一类问题，而**升级指南里完全没提这类问题**。这是一篇短文，讲真正会坏的是什么，以及为什么指南没法提前警告你。 - -## 为什么这件事重要 - -Composer 有一条容易忘掉的作用域边界。`composer update` 只写 `vendor/`——也就是**系统作用域**，框架自己的文件，它可以随意替换。 - -而你的 `app/Config/*.php` 是**项目作用域**。那是你的文件。Composer 永远不会碰它们，升级指南对它们的措辞是"你可能想要合并这些变更"，而不是"这会导致崩溃"。 - -但当 4.7 的框架代码去读一个属性，而你 4.6 时代的配置类里根本没定义它时，你得到的是致命错误——不是废弃警告，不是 notice。指南没法穷举这件事，因为这个属性的集合是「新框架读什么」与「你的配置文件里恰好有什么」的交集，而它对后半边完全不可见。 - -如果你要升级一个落后不止一个 minor 版本的 CI4 应用，请为此留出预算。我花了一个小时左右，改了 14 个配置文件。 - -## 我先核了什么 - -4.7.0 文档化的破坏性变更，以及每一项是否适用： - -| 文档化的变更 | 是否影响本应用 | -|---|---| -| `Model::insertBatch()` / `updateBatch()` 返回值变化 | 否——没用 Model | -| Entity 类型转换变更 | 否——没用 Entity | -| 验证规则变更（`regex_match`、`differs`） | 否 | -| 加密处理器默认值 | 否 | -| 上传文件／图片验证行为 | 否——没有上传 | -| `IncomingRequest` 内部变更 | 否 | -| 移除的 `Session` 类属性 | 否——见下文 | -| `PageCache` 构造函数签名 | 否——见下文 | - -其中两项需要细看，结果两个都是**假警报**，值得讲一下，因为你如果只 grep，它们看起来和真命中一模一样： - -**"移除的 `Session` 类属性"** —— 在配置里 grep `session`，立刻一片飘红。但那些属性（`sessionDriver`、`sessionCookieName` 之类）是我 `Config\App` **自己的属性**，并没有被移除。指南指的是 `Session` 类上的属性。同一个词，不同的类。 - -**"`PageCache` 构造函数签名变更"** —— 我确实引用了 `pagecache`，但只是把它作为 `Config\Filters` 里的一个 filter 别名。这并不等于继承或实例化这个类，而只有后者才会坏。**引用不等于子类。** - -这两次核对教给我同一件事：**grep 找到的是「提及」，不是「使用」。** 要确认命中的那个类是指南说的那个类，以及你确实继承或调用了那个发生变化的东西。 - -## 指南里没有的两个致命错误 - -两个都在干净启动时冒出来。两个都是 4.7 框架代码要读、而 4.6 时代的配置文件没有定义的属性。 - -### 1. `Config\App::$permittedURIChars` - -``` -Undefined property: Config\App::$permittedURIChars -``` - -4.7 的 Router 要求这个属性，用它校验进来的 URI 字符。4.6 时代的 `App.php` 里没有它，于是 Router 在路由任何东西之前就抛异常了。 - -```php -// app/Config/App.php —— 追加到类里 -/** - * CI4 4.7 compatibility: allowed characters in a URI. - * Required by the 4.7 Router. - */ -public string $permittedURIChars = 'a-z 0-9~%.:_\-'; -``` - -### 2. `Config\Format::$jsonEncodeDepth` - -``` -Undefined property: Config\Format::$jsonEncodeDepth -``` - -4.7 的 `JSONFormatter` 要求它。这一个直接打断了应用的**报告引擎**——因为引擎的 JSON 响应是用 `json_decode()` 解的，失败信号在很下游才爆出来，表现为一个让人一头雾水的解析错误，而不是真正的"属性未定义"。 - -```php -// app/Config/Format.php —— 追加到类里 -/** - * CI4 4.7 compatibility: json_encode() depth limit. - */ -public int $jsonEncodeDepth = 512; -``` - -第二个值得多说一句。表面上看到的错误是： - -``` -Failed to parse JSON string. Malformed UTF-8 characters -``` - -指向编码问题。真实原因是缺一个配置属性。**当框架的错误信息描述的是症状而不是病因时，先去日志里找真正的异常，再照着你拿到的那句话行动**——我在一个从来没坏过的 UTF-8 上白白耗了时间。 - -## 怎么在它们咬到你之前找出剩下的 - -两个致命错误都是靠**运行应用**发现的。运行是可靠的方法，但你可以先一步靠"属性覆盖差异对比"把这一类问题一次挖出来。 - -思路是：解析 `vendor/` 里框架自带的配置默认值，再解析你项目里的配置类，列出前者有、后者缺的属性。这样你一次拿到整类问题，而不是每重启一次收获一个致命错误。 - -```python -import re, pathlib - -sys_ = pathlib.Path('vendor/codeigniter4/framework/system/Config') -app_ = pathlib.Path('app/Config') - -def props(path): - try: - src = path.read_text(encoding='utf-8', errors='replace') - except FileNotFoundError: - return set() - # public/protected/private $name = ... - return set(re.findall(r'(?:public|protected|private)\s+(?:[\w\\\[\]|?]+\s+)?\$(\w+)', src)) - -# 4.7 需要、而陈旧配置可能漏掉的属性所在类 -for name in ['App', 'Cache', 'ContentSecurityPolicy', 'CURLRequest', 'DocTypes', - 'Email', 'Encryption', 'Exceptions', 'Format', 'Honeypot', - 'Migrations', 'Paths', 'Routing', 'Toolbar', 'View']: - missing = props(sys_ / f'{name}.php') - props(app_ / f'{name}.php') - if missing: - print(name, '->', sorted(missing)) -``` - -在 `composer update` **之后**、启动应用**之前**，拿新的 `vendor/` 跑一遍。它打印出来的任何东西都是潜在的致命错误。 - -在这个项目里它标出了 14 个配置文件。我把缺失的属性合并进每一个文件，以标记块的形式追加在类的末尾，这样上面原有的设置和行为完全不动： - -```php - // ---------------------------------------------------------------- - // CI4 4.7 compatibility — properties the 4.7 framework reads. - // Appended as a block so existing settings above are unchanged. - // ---------------------------------------------------------------- -``` - -框架新带出来的两个配置文件在项目里完全没有对应物，需要新建而不是合并：`Hostnames.php` 和 `WorkerMode.php`。直接从 `vendor/codeigniter4/framework/app/Config/` 拷过去就行。 - -## 升级前还值得知道的几点 - -**`app.baseURL` 的校验变严了。** 4.7 会对 4.6 接受的值抛 `ConfigException`。`'http://localhost/'`——这个值在我本地 `.env` 里躺了好几个月、一直好好的——现在直接抛异常。生产环境没受影响，因为它用的是真实 URL，但任何用了简写 baseURL 的开发环境都会拒绝启动。 - -**安全上的理由是真的。** 单是 4.7.4 就修了上传文件扩展名绕过（`is_image` / `mime_in`）、`deleteBatch()` 的 SQL 注入、以及 `UploadedFile::move()` 的路径穿越。这三个在本应用里都不可利用——它没有上传、也不碰数据库——但它们会在你加上这些功能的那一刻变成真实风险。**在你需要这些功能之前先升级，是更省事的顺序。** - -**PHP 最低版本现在是 8.2。** 动手前先确认你的主机，不是事后。 - -## 如果重来一次，我会怎么改 - -**先拿新的 `vendor/` 跑一遍应用，别的什么都别做。** `composer update`、启动、然后修坏掉的地方。读指南告诉你可能变什么；只有运行告诉你实际变了什么。 - -**把属性差异对比当成常规动作。** 它把一串串行的调试过程变成一份报告。 - -**把升级做成独立的一次提交。** 我就是这么做的，这让两个致命错误变得极易二分定位——一条 `git show` 就精确告诉我每个错误是哪次改动引入的。把框架升级混进功能分支，会把五分钟的诊断变成一场考古。 - -**读 changelog，不要只读升级指南。** 指南列的是维护者判断「可能会弄坏你」的东西。changelog 列的是实际变了的东西。这两者不是同一个集合，而这两个致命错误正好住在两者的缝隙里。 - -## 结果 - -一个 17,000 行的 CodeIgniter 应用从 4.6.3 升到 4.7.4：合并 14 个配置文件、新增 2 个配置文件、找到并修复 2 个致命错误，所有路由验证返回 200——包括一次真实的报告生成，产出 19 页 A4，与升级前的基线在字节层面可比。应用日志零错误。 - ---- - -**手上跑着旧版 CodeIgniter？** 我做这类框架升级——CI4 minor 版本升级、PHP 版本迁移，以及那些升级指南默认你自己会搞定的配置合并工作。 - -[WhatsApp +60 12-797 2969](https://wa.me/60127972969) · [me@hoelee.com](mailto:me@hoelee.com?subject=CodeIgniter%20升级) · [hoelee.com](https://hoelee.com) -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Upgrading CodeIgniter 4.6 to 4.7: The Breaking Changes the Guide Misses" -description: "The official CodeIgniter 4.7 upgrade guide lists eight breaking changes. The two that actually fataled my app weren't in it — and both are the same class of problem." -pubDate: 2026-09-27 -category: notes -tags: [codeigniter, php, upgrade, composer, breaking-changes, config, framework] -ogImage: /og/upgrading-codeigniter-46-to-47.jpg -banner: /banners/upgrading-codeigniter-46-to-47.jpg -draft: false ---- - -I upgraded a CodeIgniter 4 app from 4.6.3 to 4.7.4. I read the upgrade guide, checked every documented breaking change against the codebase, and confirmed none of them applied. Then I ran the app and it fataled twice. - -Both fatal errors were the same kind of problem, and that kind of problem is **not mentioned anywhere in the upgrade guide**. This is a short post about what actually breaks and why the guide can't warn you. - -## Why this matters - -Composer has a scope boundary that is easy to forget. `composer update` writes to `vendor/` — the **system scope**, the framework's own files, which it owns and can replace freely. - -Your `app/Config/*.php` files are **project scope**. They are yours. Composer will never touch them, and the upgrade guide describes them as "you may want to merge these changes" rather than "this will break". - -But when 4.7's framework code reads a property that your 4.6-era config class does not define, you get a fatal — not a deprecation, not a warning. The guide cannot enumerate this, because the set of properties is the intersection of *what the new framework reads* and *what your config file happens to contain*, and it has no visibility into the second half. - -If you are upgrading a CI4 app more than a minor version behind, budget for this. It took me about an hour and 14 config files. - -## What I checked first - -The documented breaking changes in 4.7.0, and whether each applied: - -| Documented change | Affects this app? | -|---|---| -| `Model::insertBatch()` / `updateBatch()` return values | No — no Models used | -| Entity casting changes | No — no Entities | -| Validation rule changes (`regex_match`, `differs`) | No | -| Encryption handler defaults | No | -| Uploaded-file / image validation behaviour | No — no uploads | -| `IncomingRequest` internal changes | No | -| Removed `Session` class properties | No — see below | -| `PageCache` constructor signature | No — see below | - -Two of those needed a closer look, and both turned out to be **false alarms** worth describing, because they look identical to a real hit if you only grep: - -**"Removed `Session` class properties"** — grepping for `session` in my configs lit up immediately. But the properties in question (`sessionDriver`, `sessionCookieName`, and friends) on my `Config\App` are **`Config\App`'s own properties**, which were not removed. The guide refers to properties on the `Session` class. Same word, different class. - -**"`PageCache` constructor signature changed"** — I reference `pagecache` — but only as a filter alias in `Config\Filters`. That does not extend or instantiate the class, which is the only thing that breaks. A reference is not a subclass. - -Both checks taught me the same thing: **grep finds *mentions*, not *usages*.** Verify the class in the match is the class the guide means, and that you actually extend or call the thing that changed. - -## The two fatals that were not in the guide - -Both surfaced on a clean boot. Both are properties that 4.7 framework code reads and 4.6-era config files do not define. - -### 1. `Config\App::$permittedURIChars` - -``` -Undefined property: Config\App::$permittedURIChars -``` - -Required by 4.7's Router, which validates incoming URI characters against this property. Absent from a 4.6-era `App.php`, the Router throws before it routes anything. - -```php -// app/Config/App.php — append to the class -/** - * CI4 4.7 compatibility: allowed characters in a URI. - * Required by the 4.7 Router. - */ -public string $permittedURIChars = 'a-z 0-9~%.:_\-'; -``` - -### 2. `Config\Format::$jsonEncodeDepth` - -``` -Undefined property: Config\Format::$jsonEncodeDepth -``` - -Required by 4.7's `JSONFormatter`. This one broke the app's **report engine**, because the engine's JSON response is decoded with `json_decode()` and the failure surfaced much further downstream as a confusing parse error rather than the real "undefined property" message. - -```php -// app/Config/Format.php — append to the class -/** - * CI4 4.7 compatibility: json_encode() depth limit. - */ -public int $jsonEncodeDepth = 512; -``` - -That second one is worth dwelling on. The visible error was: - -``` -Failed to parse JSON string. Malformed UTF-8 characters -``` - -Which points at encoding. The actual cause was a missing config property. **If a framework error names a symptom rather than a cause, go looking for the real exception in the log before acting on the message you were given** — I lost time chasing UTF-8 that was never broken. - -## How to find the rest before they bite you - -Both fatal errors were found by *running* the app. Running is the reliable method, but you can get ahead of it by diffing property coverage. - -The approach: parse the framework's shipped config defaults from `vendor/`, parse your project's config classes, and list properties defined in the former but missing from the latter. That gives you the whole class of problem at once instead of one fatal per restart. - -```python -import re, pathlib - -sys_ = pathlib.Path('vendor/codeigniter4/framework/system/Config') -app_ = pathlib.Path('app/Config') - -def props(path): - try: - src = path.read_text(encoding='utf-8', errors='replace') - except FileNotFoundError: - return set() - # public/protected/private $name = ... - return set(re.findall(r'(?:public|protected|private)\s+(?:[\w\\\[\]|?]+\s+)?\$(\w+)', src)) - -# names whose properties 4.7 requires but a stale config may omit -for name in ['App', 'Cache', 'ContentSecurityPolicy', 'CURLRequest', 'DocTypes', - 'Email', 'Encryption', 'Exceptions', 'Format', 'Honeypot', - 'Migrations', 'Paths', 'Routing', 'Toolbar', 'View']: - missing = props(sys_ / f'{name}.php') - props(app_ / f'{name}.php') - if missing: - print(name, '->', sorted(missing)) -``` - -Run it against the **new** `vendor/` after `composer update` and before you boot the app. Anything it prints is a candidate fatal. - -In this project it flagged 14 config files. I merged the missing properties into each one, appended in a marked block at the end of the class so existing settings and behaviour were untouched: - -```php - // ---------------------------------------------------------------- - // CI4 4.7 compatibility — properties the 4.7 framework reads. - // Appended as a block so existing settings above are unchanged. - // ---------------------------------------------------------------- -``` - -Two config files the framework now ships had no counterpart at all and needed creating rather than merging: `Hostnames.php` and `WorkerMode.php`. Copy those straight from `vendor/codeigniter4/framework/app/Config/`. - -## Also worth knowing before you upgrade - -**Validate `app.baseURL` properly.** 4.7 throws a `ConfigException` on values 4.6 accepted. `'http://localhost/'` — which had been sitting in my local `.env` for months, working fine — now throws. Production was unaffected because it used a real URL, but any dev environment with a shorthand baseURL will refuse to boot. - -**The security case is real.** 4.7.4 alone patched an uploaded-file extension bypass (`is_image` / `mime_in`), a SQL injection in `deleteBatch()`, and a path traversal in `UploadedFile::move()`. None were exploitable in this app — it does no uploads and uses no database — but all three become live the moment those features are added. Upgrading before you need those features is the cheaper order. - -**PHP floor is now 8.2.** Check your host before starting, not after. - -## What I'd do differently - -**Run the app against the new `vendor/` before doing anything else.** `composer update`, then boot it, then fix what breaks. Reading the guide tells you what *might* change; only running tells you what *did*. - -**Do the property diff as a matter of course.** It converts a serial debugging session into one report. - -**Do the upgrade as its own commit, on its own.** I did, and it made the two fatals trivially bisectable — one `git show` told me exactly which change introduced each. Mixing a framework bump into a feature branch turns a five-minute diagnosis into an archaeology exercise. - -**Read the changelog, not just the upgrade guide.** The guide lists what the maintainers judged *likely* to break you. The changelog lists what changed. They are not the same set, and the gap is exactly where these two fatals lived. - -## The result - -4.6.3 → 4.7.4 on a 17,000-line CodeIgniter app: 14 config files merged, two new config files added, two fatals found and fixed, and every route verified returning 200 — including a real report generation producing 19 A4 pages with byte-comparable output to the pre-upgrade baseline. Zero errors in the application log. - ---- - -**Running an older CodeIgniter app?** I do framework upgrades like this one — CI4 minor bumps, PHP version migrations, and the config-merging work that the upgrade guide assumes you will figure out yourself. - -[WhatsApp +60 12-797 2969](https://wa.me/60127972969) · [me@hoelee.com](mailto:me@hoelee.com?subject=CodeIgniter%20upgrade) · [hoelee.com](https://hoelee.com) -```` - ---- - -## 文章 12 / 16：为什么本地语音合成一直在念 “Percent”？语音管线里的 URL 编码与沙盒避坑记 -- **Slug**: `why-is-my-tts-model-repeating-percent` -- **英文标题**: Why Is My TTS Model Repeating \"Percent\"? The URL-Encoding Trap in Speech Pipelines -- **分类**: `notes` | **发布日期**: 2026-10-22 -- **线上地址**: https://blog.hoelee.com/posts/why-is-my-tts-model-repeating-percent/ (EN) | https://blog.hoelee.com/posts/zh/why-is-my-tts-model-repeating-percent/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "为什么本地语音合成一直在念 “Percent”？语音管线里的 URL 编码与沙盒避坑记" -description: "记录本地部署 CosyVoice 2 并通过群晖 n8n 对接时，语音合成陷入无限重复 “Percent” 的现象剖析，以及字符前端规范化与 n8n 沙盒表达式避坑。" -pubDate: 2026-10-22 -category: notes -blueprint: ai-agent-automation -blueprintLayer: L1-deterministic-orchestration -tags: ["tts", "debugging", "n8n", "urlencode", "python", "ai"] -ogImage: "/og/zh-why-is-my-tts-model-repeating-percent.jpg" -banner: "/banners/why-is-my-tts-model-repeating-percent.jpg" -draft: false ---- - -刚把阿里巴巴开源的 CosyVoice 2 模型部署到局域网并挂载上群晖 NAS 的 n8n 自动化工作流，在控制台用 Python 脚本直调合成一切正常。然而，当我们第一次尝试通过 n8n 的 Webhook 端点发起 HTTP GET 测试时，音箱里传出来的却是一段令人窒息的“外星密码”。 - -原本传入的中文台词是：*“恭喜！本地 CosyVoice 2 模型通过群晖 N8N 联动测试成功！”* - -但生成的音频却以极其严肃认真的语调，一口气连念了十几秒： - -> *“Percent E 6, percent 81, percent 81, percent B 9, percent A 7, percent 9... percent...”* - -模型没有崩溃，甚至声线极为清晰、咬字极其标准，但它在不断疯狂重复 “Percent”。 - -这是一起非常典型、但在语音合成自动化链路中极容易踩中的“文本前端（Text Normalization）灾难”。本文记录这起 Bug 的全链路成因与双层防御方案。 - ---- - -## 1. 现象复现：无限的 “Percent” 循环 - -我们在终端使用 `curl` 模拟普通外部请求调用 n8n 的 Webhook： - -```bash -curl -G "http://192.168.1.1:5678/webhook/ltts" \ - --data-urlencode "text=恭喜！本地模型测试成功！" \ - --data-urlencode "voice=child" \ - -o test.wav -``` - -接口返回状态码 HTTP 200，并成功下载了一段 2.4MB 的标准 WAV 音频。但当我们将音频丢进本地的语音识别（STT）模型逆向转写核对内容时，转写文本原形毕露： - -```json -{ - "text": "percent E 6 percent 81 percent 81 percent B 9 percent A 7 percent..." -} -``` - -模型并没有出现自回归幻觉，它只是在“百分之百忠实”地读出它所收到的字符串。 - ---- - -## 2. 根因剖析 1：被透传的 URL 百分号编码 - -追溯 HTTP 参数在网络与容器网关中的流动链路，整个过程如下： - -1. **客户端转码**：当客户端通过 HTTP GET 发送带中文参数的请求时，所有非 ASCII 字符必须按照标准进行百分号编码（Percent-Encoding）。例如汉字 `恭`（UTF-8 编码为 `0xE6 0x81 0xAD`），在 URL Query 中会被编码为字符串 `"%E6%81%AD"`。 -2. **n8n Webhook 接收**：n8n 收到请求后，直接将原始 Query 参数挂载到数据上下文： - ```json - { - "query": { - "text": "%E6%81%AD%E5%96%9C%EF%BC%81..." - } - } - ``` -3. **HTTP Request 节点转发**：在最初的简易工作流中，下游转发给本地 GPU 服务器的 JSON Body 直接使用了变量插值： - ```json - { - "text": "={{ $json.query.text }}" - } - ``` -4. **模型收到原始转码文本**：本地运行在 50000 端口的 Python FastAPI 服务直接收到了一个含有几十个 `%` 符号的纯 ASCII 字符串 `"%E6%81%AD..."`。 - ---- - -## 3. 为什么现代神经 TTS 不会自动转义？ - -在早期的传统规则式 TTS 中，如果传入奇怪的编码字符，系统往往会直接报错或丢弃非中英文字符。 - -但像 CosyVoice 2 或 GPT-SoVITS 这类新一代基于大语言模型架构（LLM + Flow Matching / Diffusion）的 TTS 系统，其文本前端包含复杂的**文本规范化模块（Text Normalization）**。为了让模型能够读懂各种数学符号、技术参数与代码片段，它的分词器将 `%` 显式映射为了发音为 **“Percent”** 的有效 Token。 - -因此，对 TTS 模型来说： -- 它根本不知道上游传入的是经过 URL 编码的汉字。 -- 它只看到文本里写着一个个真实的百分号、大写字母 E 和数字 6。 -- 于是，它严格按照规范将一段 15 个字的中文句子，当成了包含几十个数学符号和英文序列的混排公式，认认真真地读出了每一个 “Percent”。 - ---- - -## 4. 根因剖析 2：N8N 沙盒表达式的隐形暗坑 - -发现问题后，第一反应通常是在 n8n 的 HTTP 节点表达式中直接加一句 JS 解码： - -```javascript -// 表面合理的写法： -{{ decodeURIComponent($json.query.text) }} -``` - -但这会立刻引发二次故障： -1. **POST 请求断流**：如果调用方改用 POST JSON 发送请求，`$json.query` 会变成 `undefined`，直接报错 `Cannot read property 'text' of undefined`。 -2. **ES 语法特性支持差异**：在较老或某些特殊打包的 n8n 环境中，内嵌的 JavaScript 解释器对可选链操作符（如 `$json.query?.text`）的支持并不稳定，极易出现 `Unexpected token '.'` 语法错误。 -3. **URIError 崩溃风险**：如果用户传入的文本本身包含合法的独立百分号（例如“*降价 50%*”），未经包装的原生 `decodeURIComponent("50%")` 会直接抛出 `URIError: URI malformed`，导致整个工作流执行中断。 - ---- - -## 5. 双层防御方案 - -为了让整条语音合成链路彻底免受各类编码问题的困扰，我们在编排层与模型层同时设置了兜底防线。 - -### 第一道防线：N8N Code 节点独立预清洗 - -在 Webhook 节点之后、HTTP 转发节点之前，插入一个纯 JavaScript **Code 节点**，统一承接 GET 与 POST 数据并做容错解码： - -```javascript -const item = $input.first().json; -const query = item.query || {}; -const body = item.body || {}; - -let text = query.text || body.text || '你好，本地语音测试。'; -let voice = query.voice || body.voice || 'default'; - -// 仅在文本包含百分号时尝试解码，并在异常时优雅回退 -if (typeof text === 'string' && text.indexOf('%') !== -1) { - try { - text = decodeURIComponent(text.replace(/\+/g, ' ')); - } catch (err) { - // 包含非法百分号组合时保持原文，避免流程崩溃 - } -} - -return [{ - json: { - text: text.trim(), - voice: String(voice).trim() - } -}]; -``` - -### 第二道防线：后端 Python 服务幂等兜底 - -在本地运行的 FastAPI 接口实现中（`webui.py`），对传入的 `text` 参数增加一层无条件但幂等的 `urllib.parse.unquote`： - -```python -import urllib.parse - -@app.post("/tts") -async def api_tts(request: Request): - body = await request.json() - raw_text = body.get("text", "").strip() - - # 无论上游是否已经解码，在此做一次幂等安全解码 - text = urllib.parse.unquote(raw_text).strip() - voice = body.get("voice", "default") - - # 送入推理管线... -``` - -`unquote` 是幂等的：如果文字已经是纯中文，处理后原样返回；如果有任何遗漏的 URL 编码，会被彻底还原为真实汉字，绝不把 `%` 留给下层的分词与发音模块。 - ---- - -## 6. 验证与总结 - -在部署好双层防护后，我们重新使用同样的带 URL 编码命令请求接口： - -```bash -curl -G "http://192.168.1.1:5678/webhook/ltts" \ - --data-urlencode "text=恭喜！本地模型通过 N8N 联动测试成功！" \ - --data-urlencode "voice=child" \ - -o test_clean.wav -``` - -重新将生成的音频送入语音识别转写： - -```json -{ - "text": "恭喜！本地模型通过N8N联动测试成功！" -} -``` - -所有的 “Percent” 杂音彻底消失，取而代之的是清脆甜美的正常女童声。 - -**工程经验**： -在搭建涉及“网络网关 → 自动化编排器 → AI 神经推理模型”的多层系统时，绝不能假设到达 API 边界的数据格式天然符合预期。在网关层做安全清洗、在模型接口层做幂等兜底，两行轻量代码就能彻底杜绝 AI 模型在半夜把网络请求头念给用户听的尴尬事故。 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Why Is My TTS Model Repeating \"Percent\"? The URL-Encoding Trap in Speech Pipelines" -description: "Why our newly deployed local TTS engine synthesized endless 'percent' repetitions instead of spoken Chinese, and how character normalization and n8n sandbox limits collided." -pubDate: 2026-10-22 -category: notes -blueprint: ai-agent-automation -blueprintLayer: L1-deterministic-orchestration -tags: ["tts", "debugging", "n8n", "urlencode", "python", "ai"] -ogImage: "/og/why-is-my-tts-model-repeating-percent.jpg" -banner: "/banners/why-is-my-tts-model-repeating-percent.jpg" -draft: false ---- - -We had just wired Alibaba's open-source CosyVoice 2 into our self-hosted automation cluster. The direct Python test scripts worked cleanly, but when we fired our first test request through a Synology-hosted n8n Webhook, the audio response sounded like an alien countdown. - -Instead of the Chinese phrase *"恭喜！本地模型通过 N8N 联动测试成功！"* (Congratulations! Local model test via n8n succeeded!), the speaker rattled off a robotic avalanche: - -> *"Percent E 6, percent 81, percent 81, percent B 9, percent A 7, percent 9... percent..."* - -For fifteen agonizing seconds, the model faithfully enunciated dozens of hexadecimal percentage codes. - -Here is the post-mortem of why this happened, why neural speech text frontends amplify subtle encoding oversights, and the two defensive layers required to make speech webhooks robust. - ---- - -## 1. The Symptom: The "Percent" Loop - -We tested the n8n endpoint via a standard HTTP GET request from a terminal: - -```bash -curl -G "http://192.168.1.1:5678/webhook/ltts" \ - --data-urlencode "text=恭喜！本地模型测试成功！" \ - --data-urlencode "voice=child" \ - -o test.wav -``` - -The server returned HTTP 200 and a valid 2.4MB WAV audio file. But when passed to a speech-to-text transcriber to inspect the synthesized words, the transcription revealed the disaster: - -```json -{ - "text": "percent E 6 percent 81 percent 81 percent B 9 percent A 7 percent..." -} -``` - -The model had not hallucinated; it had pronounced the exact string it received with meticulous phonetic clarity. - ---- - -## 2. Root Cause 1: Double-Encoding & Missing Query Decoding - -To understand why, you have to trace how query parameters travel through a webhook: - -1. When `curl` (or a web browser) sends a GET request, Unicode characters are percent-encoded into ASCII octets (UTF-8 bytes formatted as `%XX`). For example, the Chinese character `恭` (Unicode `U+606D`) becomes `%E6%81%AD`. -2. When the n8n Webhook node received the GET query parameters, it passed the raw string directly into the node context: - ```json - { - "query": { - "text": "%E6%81%AD%E5%96%9C%EF%BC%81..." - } - } - ``` -3. In our initial n8n HTTP Request node, the outgoing JSON body was constructed with a direct template string: - ```json - { - "text": "={{ $json.query.text }}" - } - ``` -4. n8n serialized this raw percent-encoded string into the JSON payload forwarded to our local FastAPI service on `127.0.0.1:50000`. -5. The Python backend received the string `"%E6%81%AD..."` literally. - ---- - -## 3. Why LLM-Based Speech Frontends Don't Auto-Decode - -In traditional text processing, non-ASCII characters might trigger an encoding error or render as replacement diamonds (``). - -However, modern neural TTS systems like CosyVoice 2 and GPT-SoVITS rely on **Text Normalization (TN)** modules and byte/token vocabularies designed to read arbitrary technical jargon, symbols, and mathematical notation. - -When the Text Normalizer encountered `%`: -- It did **not** treat `%E6%81` as an escape sequence. -- It treated `%` as the literal mathematical symbol **"percent"**. -- It treated `E`, `6`, and `8` as individual alphanumeric characters. - -Consequently, a 12-character Chinese sentence was translated into a 36-token alphanumeric tongue-twister. The model simply did what it was trained to do: speak whatever was written. - ---- - -## 4. Root Cause 2: The N8N Sandbox Trap - -We attempted to fix this inside n8n by adding an inline expression: - -```javascript -// The trap inside n8n node expression: -{{ decodeURIComponent($json.query.text) }} -``` - -This immediately broke on two edge cases: -1. **POST requests don't have `$json.query`**: Accessing `$json.query.text` on a POST JSON call threw `Cannot read property 'text' of undefined`. -2. **Optional Chaining (`?.`) syntax limitations**: In older embedded V8 runtimes used by certain n8n releases, modern ECMAScript features like `$json.query?.text` can cause silent parse failures or throw `Unexpected token '.'`. -3. **Double-decoding URI errors**: If the input string contains an unescaped `%` (such as *"The price dropped by 50%"*), a naked `decodeURIComponent("50%")` throws an unhandled `URIError: URI malformed`, terminating the entire workflow. - ---- - -## 5. The Two-Layer Defense - -To make the speech pipeline bulletproof against any client library, we implemented defensive decoding at both the orchestration layer and the model backend. - -### Layer 1: N8N Code Node Pre-Processing - -Between the Webhook node and the HTTP Request node, we inserted a dedicated JavaScript **Code Node** that safely extracts and sanitizes incoming text from either GET queries or POST bodies: - -```javascript -const item = $input.first().json; -const query = item.query || {}; -const body = item.body || {}; - -let text = query.text || body.text || '语音测试'; -let voice = query.voice || body.voice || 'default'; - -// Safe percent-decoding with error fallback -if (typeof text === 'string' && text.indexOf('%') !== -1) { - try { - text = decodeURIComponent(text.replace(/\+/g, ' ')); - } catch (err) { - // Retain original string if it is not valid percent-encoding - } -} - -return [{ - json: { - text: text.trim(), - voice: String(voice).trim() - } -}]; -``` - -### Layer 2: Backend Python Fallback - -On the receiving FastAPI side (`webui.py`), we added an unconditional `unquote` step in the request handler: - -```python -import urllib.parse - -@app.post("/tts") -async def api_tts(request: Request): - body = await request.json() - raw_text = body.get("text", "").strip() - - # Defensive unquote handles both encoded and pre-decoded strings - text = urllib.parse.unquote(raw_text).strip() - voice = body.get("voice", "default") - - # Pass clean string to the audio generator... -``` - -`urllib.parse.unquote` is idempotent: if the string is already plain Chinese or English, it returns it unchanged. If residual percent signs from an upstream proxy survive, they are decoded before hitting the tokenizer. - ---- - -## 6. The Result - -After deploying both layers, we re-ran the exact same `curl` query: - -```bash -curl -G "http://192.168.1.1:5678/webhook/ltts" \ - --data-urlencode "text=恭喜！本地模型通过 N8N 联动测试成功！" \ - --data-urlencode "voice=child" \ - -o test_clean.wav -``` - -We verified the output through our local Whisper/SenseVoice ASR engine: - -```json -{ - "text": "恭喜！本地模型通过N8N联动测试成功！" -} -``` - -The "percent" loop vanished, replaced by crisp, natural speech. - -When building AI pipelines that bridge HTTP gateways with neural text frontends, never assume the string arriving at your API boundary is in its final semantic form. Defensive decoding at both ends of the bridge costs fractions of a millisecond—and prevents your AI from reading URL headers as bedtime stories. -```` - ---- - -## 文章 13 / 16：为什么多个 AI 一起写代码总会烂尾？传话失真与隐式决策困境 -- **Slug**: `why-multi-agent-coding-fails` -- **英文标题**: Why Multi-Agent Coding Fails: The Telephone Game and Architectural Drift -- **分类**: `notes` | **发布日期**: 2025-07-29 -- **线上地址**: https://blog.hoelee.com/posts/why-multi-agent-coding-fails/ (EN) | https://blog.hoelee.com/posts/zh/why-multi-agent-coding-fails/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "为什么多个 AI 一起写代码总会烂尾？传话失真与隐式决策困境" -description: "为什么让多个自主 AI Agent 协同开发必然导致分支踩踏与架构漂移？深度复盘「传话游戏」与「隐式决策冲突」两大不可逾越的机制死穴。" -pubDate: 2025-07-29 -updatedDate: 2026-10-25 -category: notes -blueprint: ai-agent-automation -blueprintLayer: L2-llm-gateway-agents -tags: ["ai", "agents", "architecture", "debugging"] -ogImage: "/og/zh-why-multi-agent-coding-fails.jpg" -banner: "/banners/why-multi-agent-coding-fails.jpg" -draft: false ---- - -## 美好的愿景 vs 残酷的现实 - -多 Agent 协同编程的蓝图听起来极具诱惑力： -- 将一份庞大的用户需求拆分成若干子任务； -- 启动一个主管 Agent； -- 由主管同时分发任务：一个 Agent 去写数据库迁移，一个写后端 REST 路由，另一个写前端界面； -- 开发者只需喝杯咖啡，三个自主进程并行提交，几分钟就能交付一套全栈特性。 - -如果你真正在哪怕稍微有点规模的实际代码库中实操过，你一定清楚最终的下场：数据库迁移里的字段和 ORM 模型对不上；后端接口返回的是 `snake_case`，前端却死死按 `camelCase` 解析；而 Git 在共享的路由注册与依赖文件上直接爆出了十几个冲突，整个工作树彻底卡死。 - -多 Agent 蜂群协同写代码的失败，根本不是模型“智商不够”，而是撞上了两个无法逾越的系统动力学死穴：**传话游戏（The Telephone Game）** 与 **隐式架构决策漂移（Implicit Architectural Drift）**。 - ---- - -## 1. 传话游戏：逐级抽象导致的信息大滑坡 - -大家小时候都玩过传话游戏：一句话悄悄传过 5 个人的耳朵，到了最后一个人嘴里，往往变成了完全荒诞的另一句话。 - -在多 Agent 协同架构中，这种信息失真会以极其机械的方式必然发生，因为分发任务必然伴随序列化与有损抽象： - -``` -[人类原始意图：边界防御、性能瓶颈、历史遗留妥协、未成文惯例] - │ - ▼ (Prompt 序列化有损压缩) - [主管 Agent] - │ - ▼ (拆解任务，生成摘要任务便签) - [执行子 Agent] -``` - -当主管 Agent 向子 Agent 分派任务时，它无法把自己的全部工作记忆（Context）整体灌过去，而只能写出一张简短的“任务便签”。 - -此时子 Agent 面临着极度不对称的信息黑盒： -- 它只知道便签上写的那几句冷冰冰的字面指令； -- 它完全不知道人类在几轮前反复强调过的深层业务痛点； -- 它不理解为什么主管在前几轮调试中坚决否决了另外三种技术方案。 - -子 Agent 确实极其忠诚且完美地完成了便签上的字面要求，但把它放回全局系统中，却成了一个格格不入的异类。 - ---- - -## 2. 隐式架构决策漂移：Flappy Bird 困局 - -Cognition（Devin 开发团队）曾在 2025 年的一场技术讲座中举过一个绝妙的例子：**用两个 Agent 协同开发一款类似《Flappy Bird》（像素鸟）的小游戏**。 - -主管 Agent 将分工划分得极其干净利落： -- **Agent A**：负责背景画面、障碍物水管与滚动视角渲染； -- **Agent B**：负责玩家操控的小鸟、飞行物理引擎与碰撞检测逻辑。 - -两个 Agent 都领到了清晰的便签，开始全速并行敲代码。 - -最终拼装代码时，荒诞的一幕发生了： -- Agent A 自行选用了类似超级马里奥的 16-bit 像素复古瓦片风； -- Agent B 则在 HTML5 Canvas 上用三次贝塞尔曲线画出了一只极简、高保真的矢量扁平小鸟。 - -两边各自都没有犯错，两边的单元测试全部顺利通过。但因为编写每一行代码都在暗中包含成百上千个“未显式声明的微小决策”——命名规范、视觉风格、错误处理语义、异步约定——两个没有进行锁步同步的 Agent，必然会朝着完全相反的架构方向渐行渐远。 - ---- - -## 3. 协同开销彻底吞噬吞吐量 - -推崇多 Agent 的最常见理由是“并行”：同时开 10 个 Agent，吞吐量理论上应该翻 10 倍。 - -但在软件开发领域，这一假设直接违背了系统理论的基本规律（阿姆达尔定律在可变状态上的体现）： - -``` -实际吞吐收益 = 并行执行速度 - (协同沟通损耗 + Git 分支合并开销) -``` - -当 5 个 Agent 在同一个代码库里自由穿梭时： -1. **文件并发写争抢**：多个 Agent 频繁修改同一个依赖配置文件、全局类型中心与路由字典。 -2. **状态无效化**：Agent A 重构了接口，而 Agent B 还在基于旧接口痛苦地生成实现类。 -3. **死循环合并**：为了消除三方合并冲突，自主 Agent 经常会简单粗暴地抹掉同伴的代码，甚至凭空捏造未导出的变量名，只为了强行换取一个 `exit 0` 的假象。 - -正如 Factory 工程师 Luke Veenhuis 的一针见血的总结：*“协同开销最终吞掉了所有速度收益，顺便把你的 Token 烧了个精光。”* - ---- - -## 破局：多 Agent 真正适用的三大场景 - -这是否意味着多 Agent 在软件工程中毫无用处？完全不是。关键在于确立核心铁律：**绝对不能让多个 Agent 对同一个代码状态拥有并发写权限**。 - -多 Agent 协同在满足以下三个条件时，才能真正发挥百倍威力： -1. **分块且只读（Sharded & Read-Only）**：如批量抓取第三方文档、多源资料深度调研、检索开源代码依赖，这些操作完全没有写入冲突。 -2. **明确的数据管道契约**：子 Agent 仅负责生成独立素材（如生成一张 SVG 架构图，或调度本地 ComfyUI 出图），结果落地为静态文件并只返回文件路径句柄。 -3. **独立对抗式守门**：由拥有空白纯净记忆的高阶推理大模型对单写主控生成的 Git Diff 进行对抗式挑刺，完全不参与编写泥潭。 - -**写操作永远单线程，只读与外部素材放手并行**——这才是规避蜂群混乱、稳定交付高质量软件的唯一工程正道。 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Why Multi-Agent Coding Fails: The Telephone Game and Architectural Drift" -description: "Why swarms of autonomous coding agents step on each other, play the telephone game, and make implicit conflicting decisions that break repositories." -pubDate: 2025-07-29 -updatedDate: 2026-10-25 -category: notes -blueprint: ai-agent-automation -blueprintLayer: L2-llm-gateway-agents -tags: ["ai", "agents", "architecture", "debugging"] -ogImage: "/og/why-multi-agent-coding-fails.jpg" -banner: "/banners/why-multi-agent-coding-fails.jpg" -draft: false ---- - -## The Vision vs. The Reality - -The pitch for multi-agent software engineering is undeniably seductive: -- Break a massive user story into subtasks. -- Spawn a manager agent. -- Let the manager spin up a worker agent for the database, another for the REST API, and a third for the frontend. -- Sip your coffee while three autonomous processes concurrently commit code and ship a full-stack feature. - -If you have actually attempted this on a non-trivial production codebase, you already know the ending: the database migration doesn't match the ORM models, the API route returns snake_case while the frontend expects camelCase, and Git aborts with 18 merge conflicts across shared registration files. - -The failure of multi-agent coding swarms is rarely due to raw model intelligence. It is caused by two fundamental mechanics: **The Telephone Game** and **Implicit Architectural Drift**. - ---- - -## 1. The Telephone Game: Lossy Human-to-Agent Delegation - -In human childhood games, a sentence whispered through five children turns into absurd gibberish by the time it reaches the end of the line. - -In multi-agent systems, the exact same distortion occurs because delegation requires serialization: - -``` -[Human Intent: Edge cases, performance bounds, legacy quirks, unspoken conventions] - │ - ▼ (Prompt Serialization) - [Lead Orchestrator Agent] - │ - ▼ (Subtask Breakdown & Summarized Prompt) - [Worker Subagent] -``` - -When the orchestrator delegates a subtask to Subagent B, it does not transfer its full internal working memory. It writes a brief, lossy instruction slip. - -The subagent operates under severe information asymmetry: -- It knows only what was written on the prompt slip. -- It lacks awareness of the subtle constraints mentioned in earlier conversational turns. -- It does not understand why the lead agent rejected three previous architectural approaches. - -The subagent executes flawlessly according to its prompt slip, yet fails completely within the holistic system. - ---- - -## 2. Implicit Architectural Drift: The Flappy Bird Dilemma - -Cognition (the team behind Devin) highlighted a brilliant analogy during a 2025 tech lecture: **Building a Flappy Bird Clone with Two Agents**. - -Suppose the manager agent divides the work cleanly: -- **Agent A**: Build the background scenery, pipe obstacles, and scrolling camera. -- **Agent B**: Build the player character (the bird), flight physics, and collision detection. - -Both agents receive clear prompt instructions. Both execute in parallel. - -When the two halves are assembled: -- Agent A chose a 16-bit retro Super Mario visual aesthetic with tiled pixel assets. -- Agent B built a sleek, flat minimalist vector bird using HTML5 canvas bezier curves. - -Neither agent made a mistake. Both followed instructions. But because every line of software code embodies hundreds of unstated, implicit decisions—conventions, styling, naming, error semantics—two agents working without lockstep synchronization will drift apart in opposite architectural directions. - ---- - -## 3. Coordination Overhead Devours Throughput - -The most common justification for multi-agent systems is parallelism: if you run 10 agents simultaneously, you should get 10x throughput. - -In software development, this assumption violates basic systems theory (Amdahl's Law applied to state mutation): - -``` -Throughput Gains = Parallel Execution Speed - Coordination & Merge Overhead -``` - -When 5 agents modify the same repository: -1. **File Contention**: Agents edit the same package manifests, configuration dictionaries, and routing tables. -2. **Lock Invalidation**: Agent A refactors a shared interface while Agent B is halfway through generating implementations against the old interface. -3. **Merge Loops**: An agent trying to resolve a 3-way git merge will often hallucinate variable names or silently wipe out a teammate's changes just to produce a clean exit code. - -As Factory engineer Luke Veenhuis famously summarized: *"Coordination overhead ends up eating up the speed gains, all the while burning through tokens."* - ---- - -## The Path Forward: Where Multi-Agent Actually Works - -Does this mean multi-agent architectures have no place in engineering? Absolutely not. It simply means **agents must never share concurrent write access to the same state**. - -Multi-agent architecture is exceptional when tasks satisfy three criteria: -1. **Sharded and Read-Only**: Tasks like scraping API documentation, verifying test logs, or querying dependencies can run completely in parallel. -2. **Clear Schema Boundaries**: Subagents generate isolated assets (e.g., an SVG diagram or an image via local diffusion) that are written to deterministic paths and returned as static handles. -3. **Adversarial Gatekeeping**: A fresh, unpolluted reasoning model reviews the single writer's output without participating in the authoring struggle. - -Restrict file writes to a single thread, use subagents strictly for read-only parallel work, and your pipeline will ship code reliably without the chaos. -```` - ---- - -## 文章 14 / 16：为什么你的 Windows 桌面 MCP 调用会静默失败：攻破 10 秒空闲门禁与避坑实录 -- **Slug**: `why-your-windows-mcp-calls-silently-fail` -- **英文标题**: Why Your Windows Desktop MCP Calls Silently Fail: Taming the 10-Second Idle Gate -- **分类**: `notes` | **发布日期**: 2026-03-12 -- **线上地址**: https://blog.hoelee.com/posts/why-your-windows-mcp-calls-silently-fail/ (EN) | https://blog.hoelee.com/posts/zh/why-your-windows-mcp-calls-silently-fail/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "为什么你的 Windows 桌面 MCP 调用会静默失败：攻破 10 秒空闲门禁与避坑实录" -description: "排查 windows-mcp 调用返回 CONTROL_PREEMPTED 的深层机制，以及为什么必须通过环境变量彻底禁用桌面控制门禁。" -pubDate: 2026-03-12 -updatedDate: 2026-10-25 -category: notes -tags: ["windows", "mcp", "debugging", "ai-agents", "troubleshooting"] -ogImage: /og/zh-why-your-windows-mcp-calls-silently-fail.jpg -banner: /banners/why-your-windows-mcp-calls-silently-fail.jpg -draft: false ---- - -你刚装好 `windows-mcp`，将其加到 Agent 的配置中，然后让它发送一条 Windows 系统通知，或者点击某个打开的应用窗口按钮。 - -工具调用在不到 200 毫秒内就返回了，大模型语气轻松地向你汇报：*“通知已成功发送。”* - -**然而屏幕上什么都没有发生。** - -当你把 MCP 协议传输的底层原始 JSON 结果打印出来时，既没有看到标准的 Python 异常，也没有看到 Win32 系统错误，而是看到这样一段内容： - -```json -{ - "code": "CONTROL_PREEMPTED", - "status": { - "state": "user", - "generation": 3, - "wait_seconds": 9.8639, - "executed_steps": 0 - } -} -``` - -或者在不同的时序下，后台进程直接抛出一串异常堆栈： - -```text -RuntimeError: AI control indicator is stopping -AI control indicator failed; releasing physical input -``` - -这就是导致大部分桌面 MCP 调用静默失败的隐蔽真凶。 - ---- - -## 根因剖析：桌面控制门禁（Desktop Control Policy） - -`windows-mcp` 内部设计了一套**桌面所有权控制机制**（位于 `src/windows_mcp/desktop/control_overlay.py` 与 `tools/control_policy.py`）。 - -开发者为了避免 AI 在物理桌面上与正在操作鼠标键盘的真人“抢夺输入焦点”，实现了一个空闲检测看门狗： - -1. 在执行任何具有副作用的输入工具（`Click`、`Type`、`Move`、`Shortcut`、`Notification` 等）前，看门狗会通过 Win32 API `GetLastInputInfo` 读取最后一次物理输入的时间戳。 -2. 如果人类在过去 **10 秒**内（`WINDOWS_MCP_CONTROL_IDLE_TIMEOUT`）移动过鼠标或敲击过键盘，服务端判定此时“人类正在使用电脑”。 -3. 此时它**既不排队也不等待**，而是立刻主动放弃执行，直接返回 `CONTROL_PREEMPTED`（控制权被抢占），工具实际执行步骤数为 0。 - -但现实使用场景是：你在终端敲完回车下达指令，或者在聊天框点下发送——**这意味着你刚刚操作过键盘鼠标**。你的系统空闲时间必然小于 1 秒，因此 Agent 随后的第一次工具调用几乎 100% 触发抢占拦截，导致操作直接被吞。 - ---- - -## 为什么 `--no-control` 参数不管用 - -翻看该项目的 README 或文档，你可能会看到关于 `--control / --no-control` 的说明。 - -很多人第一反应是给命令加上 `--no-control`： - -```bash -uvx windows-mcp serve --no-control -``` - -但服务会在启动瞬间直接崩溃报错： - -```text -Usage: windows-mcp serve [OPTIONS] -Error: No such option '--no-control'. -``` - -这是因为该项目基于 Click 库构建 CLI，`--no-control` 只绑在顶层或其他模式上，`serve` 子命令的参数解析器中根本没有注册这个 CLI 标志。 - ---- - -## 真正有效的解法：WINDOWS_MCP_CONTROL=false - -通过审查代码可以发现，配置加载模块（`src/windows_mcp/config.py`）会优先读取系统环境变量。 - -要彻底绕过这道 10 秒空闲限制，允许 Agent 在下达指令后立即获得执行权限，只需注入环境变量： - -```bash -WINDOWS_MCP_CONTROL=false -``` - -将其填入 Agent 的配置文件 `env` 字段： - -```json -{ - "command": "uvx", - "args": ["windows-mcp", "serve", "--exclude-tools", "PowerShell,Registry"], - "env": { - "WINDOWS_MCP_CONTROL": "false" - } -} -``` - -此时控制层覆盖（Overlay）、空闲看门狗以及拦截抢占逻辑将被完全旁路关闭。 - ---- - -## 验证与验收 - -为了验证问题已彻底根除，在鼠标持续移动的状态下通过原生 Python MCP 客户端发起 Toast 通知测试： - -```python -import asyncio -from mcp.client.stdio import stdio_client, StdioServerParameters -from mcp.client.session import ClientSession - -async def main(): - params = StdioServerParameters( - command="uvx", - args=["windows-mcp", "serve", "--exclude-tools", "PowerShell,Registry"], - env={"WINDOWS_MCP_CONTROL": "false", "ANONYMIZED_TELEMETRY": "false"} - ) - async with stdio_client(params) as (read, write): - async with ClientSession(read, write) as session: - await session.initialize() - res = await session.call_tool("Notification", arguments={ - "title": "Windows-MCP Ready", - "message": "Immediate execution verified.", - "app_id": "Hermes Agent" - }) - for c in res.content: - print(c.text) - -asyncio.run(main()) -``` - -终端实时输出： -```text -Notification sent: "Windows-MCP Ready" - Immediate execution verified. -``` - -无需苦等 10 秒倒数，再无 `CONTROL_PREEMPTED`，所有桌面动作即时执行。 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Why Your Windows Desktop MCP Calls Silently Fail: Taming the 10-Second Idle Gate" -description: "Why actions in windows-mcp return CONTROL_PREEMPTED or fail silently, and the exact flag needed to disable the desktop control gate." -pubDate: 2026-03-12 -updatedDate: 2026-10-25 -category: notes -tags: ["windows", "mcp", "debugging", "ai-agents", "troubleshooting"] -ogImage: /og/why-your-windows-mcp-calls-silently-fail.jpg -banner: /banners/why-your-windows-mcp-calls-silently-fail.jpg -draft: false ---- - -You install `windows-mcp`, register it in your agent config, and ask your model to trigger a native Windows notification or click a button in an open desktop application. - -The tool call finishes in under 200 milliseconds. The agent happily reports: *"I have sent the notification."* - -**Nothing appeared on screen.** - -When you inspect the raw JSON response returned by the MCP protocol, you don't find a standard Python exception or Win32 error. Instead, you see this: - -```json -{ - "code": "CONTROL_PREEMPTED", - "status": { - "state": "user", - "generation": 3, - "wait_seconds": 9.8639, - "executed_steps": 0 - } -} -``` - -Or, in slightly different timing conditions, your stdio process dumps a stack trace: - -```text -RuntimeError: AI control indicator is stopping -AI control indicator failed; releasing physical input -``` - -Here is the root cause of this failure mode and the exact configuration setting that fixes it. - ---- - -## The Culprit: The Desktop Control Gate - -`windows-mcp` is designed around a concept called the **Desktop Control Policy** (implemented in `src/windows_mcp/desktop/control_overlay.py` and `tools/control_policy.py`). - -The developers wanted to prevent an AI agent from fighting the human user for physical mouse and keyboard ownership. To achieve this, the server installs an idle detection watchdog: - -1. Before executing any state-changing input tool (`Click`, `Type`, `Move`, `Shortcut`, `Notification`, etc.), it checks Windows user input timers (`GetLastInputInfo`). -2. If the user moved the physical mouse or pressed a keyboard key within the last **10 seconds** (`WINDOWS_MCP_CONTROL_IDLE_TIMEOUT`), the server considers the human to be actively using the machine. -3. Instead of waiting for you to stop typing, it **aborts immediately**, returning `CONTROL_PREEMPTED` and leaving the action unexecuted. - -If you are chatting with your agent in a desktop window or typing commands in a terminal, **you just touched the keyboard**. Your idle timer is zero. Therefore, almost every single immediate tool call the agent issues will be preempted by default. - ---- - -## Why `--no-control` Fails - -If you inspect the repository's README or CLI documentation, you might see references to `--control / --no-control`. - -Naturally, you might try adding `--no-control` to your server startup command: - -```bash -uvx windows-mcp serve --no-control -``` - -The server instantly crashes on startup with: - -```text -Usage: windows-mcp serve [OPTIONS] -Error: No such option '--no-control'. -``` - -The CLI parser for the `serve` subcommand only exposes transport, network, and tool exclusion flags (`--transport`, `--host`, `--port`, `--exclude-tools`). It does not bind the control flag to the CLI argument group for `serve`. - ---- - -## The Real Fix: WINDOWS_MCP_CONTROL=false - -The configuration loader (`src/windows_mcp/config.py`) reads desktop control settings directly from environment variables. - -To completely bypass the 10-second idle lockout and allow the agent to execute actions immediately upon request, set: - -```bash -WINDOWS_MCP_CONTROL=false -``` - -When injected into your MCP client's environment block: - -```json -{ - "command": "uvx", - "args": ["windows-mcp", "serve", "--exclude-tools", "PowerShell,Registry"], - "env": { - "WINDOWS_MCP_CONTROL": "false" - } -} -``` - -The control overlay, idle watchdog, and preemptive preemption logic are completely disabled. - ---- - -## Verification - -To confirm the fix, send a toast notification through the raw MCP client while actively moving your mouse: - -```python -import asyncio -from mcp.client.stdio import stdio_client, StdioServerParameters -from mcp.client.session import ClientSession - -async def main(): - params = StdioServerParameters( - command="uvx", - args=["windows-mcp", "serve", "--exclude-tools", "PowerShell,Registry"], - env={"WINDOWS_MCP_CONTROL": "false", "ANONYMIZED_TELEMETRY": "false"} - ) - async with stdio_client(params) as (read, write): - async with ClientSession(read, write) as session: - await session.initialize() - res = await session.call_tool("Notification", arguments={ - "title": "Windows-MCP Ready", - "message": "Immediate execution verified.", - "app_id": "Hermes Agent" - }) - for c in res.content: - print(c.text) - -asyncio.run(main()) -``` - -Output: -```text -Notification sent: "Windows-MCP Ready" - Immediate execution verified. -``` - -No 10-second wait, no `CONTROL_PREEMPTED`, and no silent drops. -```` - ---- - -## 文章 15 / 16：校准 Intel Quick Sync HEVC：28 与 30 之间的悬崖，和一个毫无作用的 preset -- **Slug**: `calibrating-intel-quick-sync-hevc-quality` -- **英文标题**: Calibrating Intel Quick Sync HEVC: A Cliff at 28, and a Preset That Does Nothing -- **分类**: `tutorials` | **发布日期**: 2026-10-14 -- **线上地址**: https://blog.hoelee.com/posts/calibrating-intel-quick-sync-hevc-quality/ (EN) | https://blog.hoelee.com/posts/zh/calibrating-intel-quick-sync-hevc-quality/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "校准 Intel Quick Sync HEVC：28 与 30 之间的悬崖，和一个毫无作用的 preset" -description: "我在真实 1080p 文件上量了 QSV HEVC：ICQ 18–22 会把瘦文件撑大，质量悬崖落在 28 与 30 之间，而 fast 与 medium 逐字节同大小。" -pubDate: 2026-10-14 -updatedDate: 2026-10-16 +title: "Generating Full-Length Songs with YuE2 BF16: Overcoming Duration Limits and Audio Truncation" +description: "AI music tools cut off complex tracks mid-sentence. Here is how I solved the 240s duration wall using YuE2 BF16, 295s headroom budgeting, and a 60-song pipeline." +pubDate: 2026-10-26 category: tutorials -tags: ["ffmpeg", "hevc", "qsv", "transcoding", "video"] -ogImage: /og/zh-calibrating-intel-quick-sync-hevc-quality.jpg -banner: /banners/calibrating-intel-quick-sync-hevc-quality.jpg +blueprint: ai-agent-automation +blueprintLayer: L4-aigc-media-pipelines +stream: creative-lab +tags: ["ai-music", "yue2", "comfyui", "audio-engineering", "subtitles", "python"] +ogImage: "/og/generating-full-length-songs-with-yue2-comfyui.jpg" +banner: "/banners/generating-full-length-songs-with-yue2-comfyui.jpg" draft: false --- -## 一个我回答不了的问题：到底该出货哪个数字？ +Every generative music engine I have benchmarked suffers from the same fatal flaw: +**the sudden, unceremonious cutoff**. -我的转码队列按文件逐个决定用哪个编码器：源文件本来就高效就用 `copy`、GPU 走 NVENC、CPU 走 x265——现在又多了一个 Intel Quick Sync。每一档都有两个决定质量的旋钮，除了 QSV，其他每一档我都有在自己片库上量过的数字。 +Whether you use cloud APIs or local weights, models trained on fixed acoustic token +windows struggle the moment you give them a complete, narrative song structure. A +9-part progression—`[Intro] -> [Verse 1] -> [Chorus] -> [Interlude] -> [Verse 2] -> [Chorus] -> [Bridge] -> [Chorus] -> [Outro]`—inevitably +collides with the hard ceiling of the generation buffer. The vocalist is mid-word in +the climax, and the audio abruptly flatlines into digital silence. -QSV 一个都没有。而代码拒绝猜：出货默认值是 `qsv_quality = 0` / `qsv_preset = ""`，含义是「没人标定过这一档」，引擎把它当成硬排除——于是 QSV 可以存在于代码里，却在任何机器上都不会被选中，包括真的有这块显卡的那台。 +After iterating across hundreds of takes on an RTX 3060 12GB rig, I built an end-to-end +pipeline that generates genuine 4-minute full-length songs with natural melodic resolution +and zero mid-phrase amputations. -所以任务不是「把 QSV 打开」，而是「为两个旋钮产出量过的数字，让这个编码器能被诚实地选中」。 +Below is the breakdown of why standard generation fails, the 295-second headroom formula +that fixes it, and the full collection of **60 completed full-length songs** (with +individual 1024×1024 cover art and synchronized scrolling lyrics). -## 问题一：硬件不在跑这个程序的机器里 - -队列跑在 Windows 11 虚拟机里。那台 VM 直通了一张 RTX 3060，没有任何 Intel 核显——这正是一直没量过 QSV 的原因：最顺手的主意就是在本地跑一次、把数字读出来。 - -核显在宿主上（unRaid，i9-13900K），而且空闲、无人占用： - -```text -00:02.0 VGA compatible controller: Intel Raptor Lake-S GT1 [UHD Graphics 770] - driver: i915 refcount: 0 -/dev/dri/card0 /dev/dri/renderD128 （存在，全局可读） -没有任何运行中的容器声明 dri 设备 -virsh dumpxml Win11: hostdev = 01:00.0, 01:00.1（3060 + 它的 HDMI 音频） -``` - -两个结论都很有用。VM 里那句 `Error creating a MFX session` 是**预期行为**，不是一个要去查的 bug——那台 VM 永远看不到这块核显。而量 QSV 的唯一办法，是在宿主上、在容器里、用 Linux 侧去量。 - -## 用量尺的那个人自己的闸门来量，而不是用 ffmpeg - -这是让上面这些数字真正有价值的部分。我没有自己写 ffmpeg 命令、自己算 SSIM——整场标定跑的是**将来真正干活的那个引擎本体**，带着它自己的校验和自己的判收闸门： - -```bash -python3 library_run.py --work-dir /work --tier qsv --force-tier qsv \ - --qsv-quality Q --qsv-preset P --path <真原件> --keep-local --overwrite -``` - -`--keep-local` 的意思是：编码、校验、过闸门，但**不落盘**——我收集的是判决，不是一批新文件。素材通过已有的 SMB 挂载只读进入，而且每个源档在编码前后都做了哈希核对：未变。 - -镜像是 `linuxserver/ffmpeg`（ffmpeg 9.0.2、iHD 26.3.5、oneVPL 2.17）加 `python3`，再加 **numpy**——后者不是可选项。没有它闸门会打 `flat-frame scan skipped: No module named 'numpy'`，悄悄改变近平坦帧的判定方式，也就改变了判决。 - -`-global_quality` 在 `hevc_qsv` 上对应 `RateControlMethod: ICQ`，这就是被标定的旋钮。 - -## 质量阶梯：在瘦源上，低 ICQ 会把文件**变大** - -第一轮，源是一支真实的 1080p25、951 kbps（bpp 0.0184——已经很瘦、榨过一遍的那种）： - -| ICQ | 引擎判决 | mean SSIM | 视频 = 源 | -|---|---|---|---| -| 18 | **FAIL size** — 176.8% | — | — | -| 20 | **FAIL size** — 141.8% | — | — | -| 22 | **FAIL size** — 107.6% | — | — | -| 24 | ok | 0.9967 | 81.1% | -| 26 | ok | 0.9959 | 61.0% | -| 28 | ok | 0.9951 | 46.4% | -| 30 | **FAIL quality** | 0.9942 | — | -| 32 | **FAIL quality** | 0.9942 | — | - -这张表掉出三个结论。 - -第一个反直觉的：**低 ICQ 不等于安全**。ICQ 18–22 产出的文件比源还**大**，于是被引擎的体积闸门（源视频码率的 100%）拒掉。这不是 QSV 独有的毛病——NVENC 对瘦输入用低 `cq` 也是同一个形状——但这里的边界更靠上：可用区间从 24 起，不是 18。 - -第二个，**28 与 30 之间有一道悬崖**。30 和 32 都质量失败，而且失败在**首次尝试**。这句话很重要，因为引擎在质量失败时会用 `global_quality - 2` 重试一次——ICQ 更低、质量更高——并把重试写进日志。一行写着「ok at quality+2」的日志其实是**首次尝试失败**，不是通过。我第一遍在自己的笔记里把这条读反了；规矩是读首次尝试，不读救济结果。 - -第三个，地板决定取舍：在 **0.995 的 mean SSIM 下限**下，q28 的余量是 0.0001，q26 的余量是 0.0010。 - -## 完整文件确认了这张阶梯——而且有一个捷径站得住 - -摘录（180 秒无损切片）是「不用跑十次完整编码就能排出候选顺序」的办法。它只有在与完整文件结果一致时才成立，所以我把真原件端到端跑了一遍： - -| 源 | ICQ | preset | 视频 = 源 | 文件 | SSIM min / mean | 编码 | -|---|---|---|---|---|---|---| -| 254 MB，2023 秒，872 kbps | 26 | medium | **61.0%** | 168,435,266 B（66%） | 0.9938 / **0.9960** | 360 秒 ≈ 5.6× 实时 | -| 同一支 | 28 | medium | 46.3% | 54% | 0.9927 / 0.9952 | 330 秒 | -| 315 MB，3280 秒，546 kbps | 26 | medium | 50.4% | 60% | 0.9965 / **0.9977** | 600 秒 ≈ 5.5× 实时 | - -第一支的 180 秒摘录量到 mean 0.9959，完整文件量到 0.9960。这个接近程度足以让人相信「摘录用来排质量档」这条路——而且现在这是有证据的，不是靠希望。 - -## 那个毫无作用的 preset - -摘录那一层给出的故事很干净：preset 只影响速度——veryfast 30 秒 / 61.6%、fast 30 秒 / 61.0%、medium 60 秒 / 61.0%、slow 与 veryslow 60 秒。结论似乎就是选 veryfast，每支文件省 30 秒。 - -然后完整文件否掉了它。同一支源、同一个 ICQ 26，一次用 `fast`、一次用 `medium`： - -```text -fast: 168,435,266 B 编码 360 秒 mean SSIM 0.9960 -medium: 168,435,266 B 编码 360 秒 mean SSIM 0.9960 -``` - -逐字节同大小、同编码时间、同质量。摘录层的 30 秒对 60 秒，是每一轮固定开销造成的差别，不是编码速度——这正是那种「如果只走到一步就停」会写进配置文件、再写进一篇博客的数字。 - -所以：**摘录只能排质量，永远不能排速度。** 速度必须在完整文件上量，固定开销在那里才会消失。 - -## 最后落了什么 - -`qsv_quality = 26`、`qsv_preset = medium`——源视频码率的 61.0%，mean SSIM 0.9960（min 0.9938），约 5.6× 实时；在片库里最瘦的一支上是 50.4%、mean 0.9977。q28 相对下限的余量只有 0.0002，而一次质量失败的代价是两次整片编码再加一次搁置，所以那多出来的 15 个百分点不值得换这条尾部风险。 - -另外两条值得记下来的量测，它们回答的是「到底是 QSV 的问题还是文件的问题」： - -- **跨编码器对照。** 一支 720p、节奏不均匀的源，QSV q24 量到 mean 0.9154、QSV q28 量到 0.9146、x265 crf26 量到 0.9164——引擎的对齐搜索同时报 `no alignment explains the mismatch`。这条拒绝属于源档，不属于 Quick Sync。 -- **并发。** 引擎拒绝在共享 work root 上起第二个编码器：`another transcode is already running (pid 7, /work/run-qsv.lock, 3 s old)`。换成各自独立的 work root，两个作业能跑——但墙钟 201 秒对单人 107 秒。两倍时间，零吞吐收益。QSV 一次只跑一个作业。 - -## 四个坑，每个都花掉一轮 - -1. **永远不要把共享片库挂到容器的 `/lib`。** 我把 SMB 共享挂进了容器的 `/lib`——而那是容器自己的动态加载器目录。八个作业全在 1 秒内死掉，报 `exec /bin/bash: no such file or directory`，读起来像镜像坏了，而不是挂载点错了。挂 `/library` 就正常。 -2. **合成源会给出与事实相反的 ICQ 结论。** `hevc_qsv` 的码率控制对 `rgb24` 输入会被驱动拒绝（`Selected ratecontrol mode is unsupported`），所以用 `testsrc`/`mandelbrot` 自检会得出相反的结论。要么用真文件，要么加 `format=nv12`。 -3. **已经是 HEVC 的源根本没法标定。** 闸门会回 `SKIP already HEVC hvc1`，`--overwrite` 也不解除。这个片库里肥档 1080p H.264 原件早就转完了，剩下还在肥档的都是 HEVC——所以那一档已经没有可量的素材了。这是缺输入，不是漏做任务。 -4. **读首次尝试，不读重试**（见上面那道悬崖）。 - -## 如果重来一次 - -1. **速度要在你真正会跑的那个产物上量**，不要在它的一片切片上量。整件事里我唯一报错的数字，就是信了摘录的墙钟。 -2. **先让机器证明它有没有这块硬件，再排测试计划。** 六分钟的 `lspci` 加 `virsh dumpxml`，就把「为什么 QSV 在 VM 里失败」这个悬了几周的问题结掉了。 -3. **把闸门留在回路里。** 因为每一次运行都由引擎自己的校验判收，上面这些表格是决策，不是观点——将来在生产里会拒掉坏文件的，正是这个把我的 ICQ 30 拒掉的闸门。 -4. **把诚实的适用范围写进文档。** 这些数字是 UHD 770 上 Linux iHD 26.3.5 + oneVPL 2.17 的数；生产目标是同一块硅片上的 Windows MSDK 路径，我不能声称两者一致。这里站得住的是：命令在真硬件上跑得通、质量↔体积曲线的位置、以及锁的行为——留给将来的，只是「在 Windows+Intel 上按同一张表复测一次」，而不是又一个下午。 - -## 结果 - -QSV 从「存在于代码里、永远选不到」变成了一个带公开数字的已标定档，安全规则也跟着改了：原来那条钉住「**出货默认值必须代表未标定**」的测试，现在钉的是「**出货默认值必须是量过的值**」，同时保留「空值 = 硬排除」的全部断言。在没装 Intel 显卡的机器上，doctor 那行也从信息级的「等硬件」变成了警告——这本身是另一个小教训：一个已标定的编码器自检失败该长什么样，以及为什么机器上不存在的硬件应该进一行汇总，而不是进警告。 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Calibrating Intel Quick Sync HEVC: A Cliff at 28, and a Preset That Does Nothing" -description: "I measured QSV HEVC on real 1080p files: ICQ 18–22 inflate thin sources, the quality cliff sits between 28 and 30, and fast equals medium byte for byte." -pubDate: 2026-10-14 -updatedDate: 2026-10-16 -category: tutorials -tags: ["ffmpeg", "hevc", "qsv", "transcoding", "video"] -ogImage: /og/calibrating-intel-quick-sync-hevc-quality.jpg -banner: /banners/calibrating-intel-quick-sync-hevc-quality.jpg -draft: false --- -## The question I could not answer: which number do I ship? + -My transcoding queue decides, per file, which encoder does the work: `copy` -when the source is already efficient, NVENC on the GPU, x265 on the CPU — -and now Intel Quick Sync. Each tier has two knobs that decide quality, and -every tier except QSV had numbers I had measured on my own library. +
+
+ 🎵 Track Quick Switcher & Jumper + 60 Full-Length Tracks · Lazy Loaded +
+
+ + + + + + + +
+
+ + +
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
-QSV had none. And the code refused to guess: a shipping default of -`qsv_quality = 0` / `qsv_preset = ""` means *nobody has calibrated this*, -and the engine treats that as a hard exclusion — QSV can exist in the code -and never be selected on any machine, including the one that has the -hardware. +## The Problem: The 240-Second Wall -So the task was not "turn on QSV". It was "produce measured numbers for two -knobs, so the encoder becomes selectable honestly". +In earlier batches, songs generated via standard ComfyUI text-to-music nodes frequently +exhibited one of two failure modes: -## Problem one: the hardware is not in the machine where the app runs +1. **The Chopped Outro**: Setting `max_duration = 240.0s` while feeding a full lyric + sheet resulted in the track hitting the final timestamp while the acoustic model was + still executing the chorus. All 12 initial test runs ended at exactly `240.000s`—an + unusable hard cliff. +2. **The Panic Accelerando**: If you prompt the model to fit too many lines into a tight + window, the auto-regressive acoustic token predictor compresses the vocal pacing, + turning an emotional Celtic ballad into an unnatural rap cadence. -The queue runs on Windows 11 in a VM. That VM has an RTX 3060 passed through -and no Intel graphics of any kind — which is exactly why QSV had never been -measured: the obvious move is to run one test locally and read the numbers -off. +The fundamental mistake was treating `max_duration` as the *desired song length*. +In generative auto-regressive audio, **max duration is not your target length; it is your +acoustic safety wall.** -The iGPU is on the host instead (unRaid, i9-13900K), idle and unclaimed: +## The Three Architecture Fixes -```text -00:02.0 VGA compatible controller: Intel Raptor Lake-S GT1 [UHD Graphics 770] - driver: i915 refcount: 0 -/dev/dri/card0 /dev/dri/renderD128 (present, world-accessible) -no running container declares a dri device -virsh dumpxml Win11: hostdevs = 01:00.0, 01:00.1 (the 3060 + its HDMI audio) -``` +### 1. The 295.0s Headroom Formula -Two conclusions, both useful. The VM's `Error creating a MFX session` is -**expected** and not a bug to chase — the VM can never see that iGPU. And -the only way to measure QSV is on the Linux side, in a container, on the -host. +To produce songs that naturally conclude with a 4-minute runtime, you must structure +the lyric density for **210s to 260s**, but provision the generator with **295.0s** +of ceiling: -## Measure with the engine's own gate, not with ffmpeg +$$\text{Headroom Buffer} = \text{Max Duration (295s)} - \text{Expected Lyric Duration (220s)} \approx 75\text{s}$$ -This is the part that makes the numbers worth anything. I did not write -ffmpeg commands and compute SSIM myself — the calibration runs **the engine -that will do the work**, with its own verification and its own accept/reject -gate: +This ~75-second buffer provides the acoustic model with the token budget required to: +- Complete the final verse without panic compression; +- Introduce a dedicated, melodic instrumental outro; +- Execute natural acoustic decay, room reverb dissipation, and final silence. + +### 2. Automated Hard-Cut Quarantine Gate + +Any take whose generated length measures within 0.1s of the hard ceiling (`>= 294.9s`) +is treated as an unresolved truncation failure and automatically quarantined. Only takes +that resolve naturally before the ceiling (`duration < 294.0s`) are accepted into the +production archive. Across our 60-song run, all 60 finalized masters ended between +**129.9s and 285.6s** with zero abrupt cutoffs. + +### 3. VRAM Stabilization on a 12GB RTX 3060 + +Running YuE2-3B in BF16 on consumer hardware requires tight memory lifecycle management: ```bash -python3 library_run.py --work-dir /work --tier qsv --force-tier qsv \ - --qsv-quality Q --qsv-preset P --path --keep-local --overwrite +# Launch ComfyUI standalone with minimal VRAM overhead +cd /d/app/ComfyUI +./python_embeded/python.exe -s ComfyUI/main.py \ + --windows-standalone-build \ + --port 8188 \ + --disable-pinned-memory \ + --disable-nvml-pressure \ + --cache-classic ``` -`--keep-local` means encode, verify, run the gate, and then *do not* land -the file — the verdict is what I am collecting, not a new library. Sources -came in read-only over an existing SMB mount, and I hashed each source -before and after: unchanged. +By enforcing immediate model unloading via `/free` after batch execution and maintaining +zero pinned memory overhead, generation of 4.5-minute sequences remains rock-solid without +triggering CUDA out-of-memory errors. -The image is `linuxserver/ffmpeg` (ffmpeg 9.0.2, iHD 26.3.5, oneVPL 2.17) -plus `python3` — plus **numpy**, which is not optional. Without it the gate -prints `flat-frame scan skipped: No module named 'numpy'` and quietly -changes how near-flat frames are judged, which changes verdicts. +--- -`-global_quality` maps to `RateControlMethod: ICQ` on `hevc_qsv`, which is -the knob being calibrated. +## The 60 Full-Length Song Archive -## The ladder: on a thin source, low ICQ makes the file *bigger* +All 60 tracks below were generated locally with YuE2 BF16, tagged with clean ID3v2.3 +metadata, paired with custom 1024×1024 Gemini-rendered full-bleed cover art (converted to +WebP), and hosted on CDN with HTTP 206 range-seeking support. -First run, on a real 1080p25 source at 951 kbps (0.0184 bits per pixel — a -"thin" file, the kind that is already squeezed): +*Tip: Use the interactive selector above to jump directly to any track.* -| ICQ | engine verdict | mean SSIM | video = source | -|---|---|---|---| -| 18 | **FAIL size** — 176.8% | — | — | -| 20 | **FAIL size** — 141.8% | — | — | -| 22 | **FAIL size** — 107.6% | — | — | -| 24 | ok | 0.9967 | 81.1% | -| 26 | ok | 0.9959 | 61.0% | -| 28 | ok | 0.9951 | 46.4% | -| 30 | **FAIL quality** | 0.9942 | — | -| 32 | **FAIL quality** | 0.9942 | — | +
+
+ The Thimble Under the Moss cover art +
+
+
+ 01. The Thimble Under the Moss + MP3 + 3分11秒 (191.1s) · Celtic Folk · 清脆稚气女童声（类似 Cécile Corbel《Arrietty's Song》）｜中速轻快（约 105 BPM） +
+ + +
+
-The counterintuitive one: **low ICQ does not mean safe.** ICQ 18–22 produce -files *larger* than the source, so the engine's size ceiling (100% of the -source's video bitrate) rejects them. QSV is not alone in this — NVENC at a -low `cq` does the same thing on thin inputs — but the boundary sits higher -here: the usable range starts at 24, not 18. +
+
+ Mist of Skellig Michael cover art +
+
+
+ 02. Mist of Skellig Michael + MP3 + 4分26秒 (266.2s) · Celtic Folk · 空灵圣洁的小女孩独唱（类似 Enya + 幼态音色）｜慢速悠长（约 72 BPM，字少音长） +
+ + +
+
-Third, the floor is where the choice gets made: with a **0.995 mean-SSIM -floor**, q28 passes with 0.0001 of margin and q26 with 0.0010. +
+
+ The Rose of Connemara cover art +
+
+
+ 03. The Rose of Connemara + MP3 + 4分43秒 (283.1s) · Celtic Folk · 纯净稚气少女音（类似 Cara Dillon 早期清亮少女感）｜中慢速叙事（约 82 BPM） +
+ + +
+
-Excerpts (180-second lossless slices) are how you rank ten candidates -without ten full encodes. They are only legitimate if they agree with -full-file runs, so I ran the real files end to end: +
+
+ Lanterns on the Moor cover art +
+
+
+ 04. Lanterns on the Moor + MP3 + 3分31秒 (211.2s) · Celtic Folk · 俏皮精灵般的女童声（类似 Cécile Corbel 欢快舞曲风）｜快速三拍子（约 128 BPM） +
+ + +
+
-The 180-second excerpt of the first file measured mean 0.9959; the full file -measured 0.9960. That is close enough to trust the shortcut **for quality -ranking** — and I now have the evidence to say so rather than hope so. +
+
+ The Weaver of Raglan Lane cover art +
+
+
+ 05. The Weaver of Raglan Lane + MP3 + 4分39秒 (279.3s) · Celtic Folk · 醇厚温暖的凯尔特民谣男声（类似 Luke Kelly，调剂男声）｜中慢速（约 80 BPM） +
+ + +
+
-The excerpt runs said preset only changes speed: veryfast 30 s / 61.6%, fast -30 s / 61.0%, medium 60 s / 61.0%, slow and veryslow 60 s. Clean story: pick -veryfast, save 30 seconds per file. +
+
+ 硝子の花束 cover art +
+
+
+ 06. 硝子の花束 + MP3 + 3分49秒 (230.0s) · Japanese Healing · 稚气通透、略带脆弱感的动漫少女音（祈妹风）｜中慢速钢琴弦乐（约 78 BPM） +
+ + +
+
-```text -fast: 168,435,266 B enc 360 s mean SSIM 0.9960 -medium: 168,435,266 B enc 360 s mean SSIM 0.9960 -``` +
+
+ 銀の時計塔と宵の歌 cover art +
+
+
+ 07. 銀の時計塔と宵の歌 + MP3 + 4分34秒 (274.8s) · Japanese Healing · 清亮圣洁少女音 + 梶浦语吟唱｜中速华尔兹三拍子（约 110 BPM） +
+ + +
+
-So: **excerpts rank quality, never speed.** Speed has to be measured on full -files, where the fixed overhead disappears. +
+
+ 森の郵便屋さん cover art +
+
+
+ 08. 森の郵便屋さん + MP3 + 3分44秒 (225.0s) · Japanese Healing · 极度轻柔、如耳语般的稚气女童声｜慢速木吉他与竖琴（约 68 BPM） +
+ + +
+
-`qsv_quality = 26`, `qsv_preset = medium` — 61.0% of the source's video -bitrate at mean SSIM 0.9960 (min 0.9938) at roughly 5.6× realtime, and 50.4% -at mean 0.9977 on the thinnest file in the library. q28's margin over the -floor is 0.0002, and the cost of a quality failure is two full encodes plus -a park, so the extra 15 percentage points of savings are not worth the tail -risk. +
+
+ 白鳥の祈り cover art +
+
+
+ 09. 白鳥の祈り + MP3 + 4分06秒 (246.3s) · Japanese Healing · 空灵通透的高音少女/女童声，尾音带空灵共鸣｜慢速大气（约 70 BPM） +
+ + +
+
-- **Cross-encoder control.** One 720p source with non-uniform cadence scored - QSV q24 mean 0.9154, QSV q28 mean 0.9146, and x265 crf26 mean 0.9164 — and - the engine's alignment search reported - `no alignment explains the mismatch`. The rejection belongs to the source, - not to Quick Sync. -- **Concurrency.** The engine refuses a second encoder on a shared work - root: - `another transcode is already running (pid 7, /work/run-qsv.lock, 3 s old)`. - With separate work roots, two jobs run — and take 201 s where one takes - 107 s. Twice the wall clock, zero throughput gain. One QSV job at a time. +
+
+ 星屑のオルゴール cover art +
+
+
+ 10. 星屑のオルゴール + MP3 + 3分31秒 (211.8s) · Japanese Healing · 清甜稚气的少女声｜中慢速轻盈感（约 88 BPM） +
+ + +
+
-1. **Never mount a shared library at `/lib`.** I mounted the SMB share into - the container at `/lib` — which is where the container's own dynamic - loader lives. All eight jobs died in under a second with - `exec /bin/bash: no such file or directory`, which reads like a broken - image, not a bad mount. `/library` works. -2. **Synthetic sources lie about ICQ.** `hevc_qsv`'s rate control is - rejected by the driver for `rgb24` input - (`Selected ratecontrol mode is unsupported`) — so a - `testsrc`/`mandelbrot` self-test concludes the opposite of the truth. Use - a real file, or add `format=nv12`. -3. **Already-HEVC sources cannot be calibrated at all.** The gate answers - `SKIP already HEVC hvc1` and `--overwrite` does not lift it. The fat - 1080p H.264 originals in this library had all been converted long ago; - the remaining fat files are HEVC — so that tier has nothing left to - measure on. That is a missing input, not a skipped task. -4. **Read the first attempt, not the retry** (see the cliff above). +
+
+ ひまわり行進曲 cover art +
+
+
+ 11. ひまわり行進曲 + MP3 + 3分18秒 (198.0s) · Japanese Energetic · 元气满满的小女孩独唱/齐唱感｜中快速欢快节奏（约 118 BPM） +
+ + +
+
-1. **Measure speed on the artifact you will actually run**, not on a slice - of it. My one wrong number in this whole exercise came from trusting an - excerpt's wall clock. -2. **Make the machine prove it has the hardware before you plan the test.** - Six minutes of `lspci` and `virsh dumpxml` retired the "why does QSV fail - in the VM" question that had been open for weeks. -3. **Keep the gate in the loop.** Because the engine's own verification - judged every run, the tables above are decisions, not opinions — the same - gate that will reject a bad file in production rejected my ICQ 30. -4. **Write the honest scope line into the notes.** These numbers are Linux - iHD 26.3.5 + oneVPL 2.17 on a UHD 770; the production target is the - Windows MSDK path on the same silicon, and I cannot claim they agree. - What this establishes is that the command runs on real hardware, where - the quality↔size curve sits, and how the lock behaves — leaving a - re-measurement on Windows+Intel as one table, instead of another - afternoon. +
+
+ プリンは世界を救うのだ！ cover art +
+
+
+ 12. プリンは世界を救うのだ！ + MP3 + 3分23秒 (203.4s) · Japanese Energetic · 软萌夸张的稚气萝莉音（高音、语速快、字密）｜快速电音流行（约 145 BPM） +
+ + +
+
-QSV went from "present in the code and never selectable" to a calibrated -tier with published numbers, and the safety rule changed with it: the test -that used to pin *shipping defaults must mean uncalibrated* now pins -*shipping defaults must be measured values*, while keeping every assertion -that an empty value is a hard exclusion. The doctor output on a machine with -no Intel GPU moved from an informational "waiting for hardware" line to a -warning — which is its own small lesson about what a calibrated encoder -failing self-check should look like, and why hardware the machine does not -have belongs in a summary row rather than a warning. -```` +
+
+ こねこ探偵団のテーマ cover art +
+
+
+ 13. こねこ探偵団のテーマ + MP3 + 2分16秒 (136.1s) · Japanese Energetic · 奶声奶气的女童声（6~8岁童音感）｜中速轻快爵士童谣（约 115 BPM） +
+ + +
+
-## 文章 16 / 16：转成 HEVC 到底会不会让文件变小？我实测了 -- **Slug**: `does-hevc-actually-shrink-your-files` -- **英文标题**: Does Re-encoding to HEVC Actually Shrink Your Files? I Measured It. -- **分类**: `tutorials` | **发布日期**: 2026-10-08 -- **线上地址**: https://blog.hoelee.com/posts/does-hevc-actually-shrink-your-files/ (EN) | https://blog.hoelee.com/posts/zh/does-hevc-actually-shrink-your-files/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】：[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】： -【人设与用词卫生】： -【CRO与转化建议】： -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "转成 HEVC 到底会不会让文件变小？我实测了" -description: "在真实文件上对 HEVC 编码器做的实测对比：NVENC cq27 让三个测试文件中的两个变得更大；同画质下软件 x265 比 NVENC 小 10–22%；同样是 1080p，一个源缩了 49%，另一个只缩了 17%。" -pubDate: 2026-10-08 -category: tutorials -tags: [ffmpeg, hevc, x265, nvenc, video] -ogImage: /og/zh-does-hevc-actually-shrink-your-files.jpg -banner: /banners/does-hevc-actually-shrink-your-files.jpg -draft: false ---- +
+
+ 夏祭りのラムネ星 cover art +
+
+
+ 14. 夏祭りのラムネ星 + MP3 + 3分14秒 (195.0s) · Japanese Energetic · 清亮少年男声 + 稚气少女声（调剂少年音）｜中快速夏日流行（约 122 BPM） +
+ + +
+
-"转成 HEVC，文件就变小了。"这句话被重复得太多，以至于没人去实测它。我不得不测：我手上有一个 582 个课程视频的库，之前那轮重编码在 54 个文件里有 **50 个把文件变大了**，平均产出是原来的 111%。 +
+
+ ほうきに乗って宅配便 cover art +
+
+
+ 15. ほうきに乗って宅配便 + MP3 + 3分16秒 (196.1s) · Japanese Energetic · 清脆阳光的小女孩声音｜中速轻快复古流行（约 116 BPM） +
+ + +
+
-简短版本： +
+
+ 白露·竹间灯 cover art +
+
+
+ 16. 白露·竹间灯 + MP3 + 3分42秒 (222.6s) · Chinese Guofeng · 清甜稚气的少女/女童声｜中速轻快电子国风（约 108 BPM） +
+ + +
+
-## 方法：要么在同画质下比，要么别比 +
+
+ 小鲤跃龙门 cover art +
+
+
+ 17. 小鲤跃龙门 + MP3 + 2分09秒 (129.9s) · Chinese Guofeng · 活泼软萌的稚气少女音（语速偏快、吐字灵动）｜中快速（约 120 BPM） +
+ + +
+
-我把每个源的 60 秒窗口用多种设置编码，然后对每一份输出： +
+
+ 云海归鲲 cover art +
+
+
+ 18. 云海归鲲 + MP3 + 3分56秒 (236.9s) · Chinese Guofeng · 空灵仙气的少年假声/美声男声（调剂男声）｜慢速悠远（约 68 BPM） +
+ + +
+
-有一个前提：这套度量工具必须自洽。把一个文件和**它自己**走同一条代码路径比对，必须得到 SSIM 1.0000。当我的工具做不到时——它在一个和自己的比对中给出了 0.87，因为其中一侧是从 open GOP 源的中间 seek 解码的——我本可以"测出"任何结论并信以为真。 +
+
+ 纸伞巷 cover art +
+
+
+ 19. 纸伞巷 + MP3 + 3分34秒 (214.6s) · Chinese Guofeng · 清亮温婉的少女声（带一点点江南小调韵味）｜中慢速（约 80 BPM） +
+ + +
+
-## 结果 +
+
+ 山间小道士 cover art +
+
+
+ 20. 山间小道士 + MP3 + 3分07秒 (187.6s) · Chinese Guofeng · 纯真稚气的小女孩/少女声音｜中速轻快民谣古风（约 105 BPM） +
+ + +
+
-| 设置 | 1080p 音乐（2720 kbps，bpp 0.0437） | 720p（1208 kbps，bpp 0.0437） | 1080×1920 竖屏（4490 kbps，bpp 0.0722） | -|---|---|---|---| -| `hevc_nvenc` cq27 | 72.4%（min SSIM 0.9963） | **115.2%**（0.9935） | **119.3%**（0.9930） | -| `hevc_nvenc` cq30 | 51.0%（0.9945） | 82.6%（0.9907） | 83.8%（0.9901） | -| `hevc_nvenc` cq31 | 43.0%（0.9941） | 72.9%（0.9894） | 74.3%（0.9890） | -| `hevc_nvenc` cq30 + AQ/lookahead | 55.8%（0.9958） | 88.1%（0.9914） | 84.4%（0.9935） | -| `libx265` crf26 | **33.7%**（0.9927） | **56.7%**（0.9879） | **67.0%**（0.9913） | -| `libx265` crf28 | 26.4%（0.9910） | 45.0%（0.9852） | 53.2%（0.9888） | +
+
+ 橘子汽水的夏天 cover art +
+
+
+ 21. 橘子汽水的夏天 + MP3 + 2分48秒 (169.0s) · Gentle Acoustic · 清澈稚气的少女声｜中慢速木吉他民谣（约 82 BPM） +
+ + +
+
-**1. 那个"著名默认值"是错的默认值。** 网上说"HEVC，高质量"时，很多人会落到 cq27 —— 而在三个文件里的两个上，它产出的视频流**比它要替换掉的源还大**。这并不是说 HEVC 比 H.264 差，而是说：一个在你自己的测试片段上校准出来的固定质量旋钮，对别人编好的文件毫无意义。 +
+
+ 外婆的小院 cover art +
+
+
+ 22. 外婆的小院 + MP3 + 3分32秒 (212.4s) · Gentle Acoustic · 纯真软糯的小女孩声音｜慢速温情（约 72 BPM） +
+ + +
+
-**3. 同画质下软件 x265 比 NVENC 更小。** 比较 SSIM 接近的那两行：cq31 vs crf26（在 720p 文件上 min SSIM 0.9894 vs 0.9927）——72.9% vs 56.7%，也就是说**文件小了 22%，而量出来的画质更高**。整体看，多出来的节省大约是 **10–22%**。这不是说 NVENC 编得差，而是它为吞吐量做了优化。约束是磁盘就花 CPU，约束是时间就花 GPU。 +
+
+ 开往北方的绿皮车 cover art +
+
+
+ 23. 开往北方的绿皮车 + MP3 + 3分36秒 (216.3s) · Gentle Acoustic · 低沉温暖的民谣男声（调剂男声）｜慢速叙事（约 75 BPM） +
+ + +
+
-## 这对 VP9/AV1 源意味着什么 +
+
+ 写给流浪猫的信 cover art +
+
+
+ 24. 写给流浪猫的信 + MP3 + 2分09秒 (129.9s) · Gentle Acoustic · 俏皮温柔的稚气少女音｜中速轻快尤克里里/木吉他（约 98 BPM） +
+ + +
+
-在一个由 YouTube VP9/AV1 组成的素材库上，用硬件编码器以 cq27 重编码，产出是原文件的 **105–112%**——明显不如不动它。这种情况下正确的动作是 `-c copy`：让视频流逐位不变，只重编音频容器（MP4 里的 MP3 在 iOS 上放不了，AAC 可以）。 +
+
+ 平凡的傍晚 cover art +
+
+
+ 25. 平凡的傍晚 + MP3 + 3分41秒 (221.0s) · Gentle Acoustic · 温柔治愈的青年男声（调剂男声）｜慢速钢琴与木吉他（约 74 BPM） +
+ + +
+
-1. **动手之前先按 bpp 分类。** 每像素每秒位数（`视频码率 / (宽 × 高 × 帧率)`）。一次 ffprobe 调用，它区分"有余量"和"别碰它"的能力，比分辨率或文件大小都强。 -2. **已经很高效率的源绝不重编码。** 复制视频流、需要的话修音频，然后走开。重编码变不出本来不存在的余量。 -3. **设质量，不设码率——然后对着源验证。** 逐帧比对的度量，加上一个证明工具本身可用的自检。 -4. **按瓶颈选编码器。** 磁盘 → x265（同画质小 10–22%）。时间 → NVENC。 -5. **给输出加上限。** 无论怎么做，都要断言结果不比源更大。就这一条断言，本可以在那 54 个文件被碰之前就抓住整个 111% 的问题。 -6. **别忘了音频。** 视频被压狠之后，音轨可能是文件里剩下的最大部分。 +
+
+ Light of the Eventide cover art +
+
+
+ 26. Light of the Eventide + MP3 + 4分09秒 (249.0s) · Angelic Choir · 天籁女童高音领唱 + 纯净童声合唱｜极慢速庄严（约 66 BPM） +
+ + +
+
-在那个 582 个文件的库上，用上面的分档和我另文写的回读校验： +
+
+ Bells of the Winter Star cover art +
+
+
+ 27. Bells of the Winter Star + MP3 + 2分47秒 (167.6s) · Angelic Choir · 清脆女童多声部齐唱與轮唱｜中快速灵动节奏（约 125 BPM） +
+ + +
+
-这就是"HEVC 会不会让我的文件变小"的诚实答案：取决于你在重编码什么，而唯一知道你那个数字的办法，就是在你自己的文件上、把画质固定住去测一遍。 -```` +
+
+ Paper Planes in the Wind cover art +
+
+
+ 28. Paper Planes in the Wind + MP3 + 3分25秒 (205.5s) · Angelic Choir · 清亮纯真的女童领唱 + 学院风童声合唱｜中速温暖（约 96 BPM） +
+ + +
+
+ +
+
+ Lullaby of the Little Star cover art +
+
+
+ 29. Lullaby of the Little Star + MP3 + 3分47秒 (227.8s) · Angelic Choir · 稚气纯净的小女孩独唱（突出童真独唱感）｜慢速温柔（约 70 BPM） +
+ + +
+
+ +
+
+ Hymn of the Morning Meadow cover art +
+
+
+ 30. Hymn of the Morning Meadow + MP3 + 4分19秒 (259.9s) · Angelic Choir · 神圣清澈的女童领唱 + 宏大童声和声（帕赫贝尔卡农式递进）｜慢速治愈（约 72 BPM） +
+ + +
+
+ +
+
+ 地板下的借物小人 cover art +
+
+
+ 31. 地板下的借物小人 + MP3 + 3分07秒 (187.7s) · Celtic Folk (ZH) · 清脆稚气女童声｜中速轻快凯尔特竖琴与锡哨（约 105 BPM） +
+ + +
+
+ +
+
+ 迷雾孤岛的竖琴 cover art +
+
+
+ 32. 迷雾孤岛的竖琴 + MP3 + 3分24秒 (204.4s) · Celtic Folk (ZH) · 空灵圣洁的小女孩独唱｜慢速悠长（约 70 BPM，字少音长） +
+ + +
+
+ +
+
+ 石楠花开的高地 cover art +
+
+
+ 33. 石楠花开的高地 + MP3 + 3分36秒 (216.9s) · Celtic Folk (ZH) · 纯净稚气少女音｜中慢速木吉他与凯尔特小提琴（约 82 BPM） +
+ + +
+
+ +
+
+ 橡树下的萤火舞会 cover art +
+
+
+ 34. 橡树下的萤火舞会 + MP3 + 3分43秒 (223.1s) · Celtic Folk (ZH) · 俏皮精灵般的女童声（语速快、节奏感强）｜快速三拍子（约 128 BPM） +
+ + +
+
+ +
+
+ 老航海家的绿宝石 cover art +
+
+
+ 35. 老航海家的绿宝石 + MP3 + 3分49秒 (229.8s) · Celtic Folk (ZH) · 醇厚温暖的凯尔特民谣男声（调剂男声）｜中慢速（约 80 BPM） +
+ + +
+
+ +
+
+ 玻璃温室的告别 cover art +
+
+
+ 36. 玻璃温室的告别 + MP3 + 3分37秒 (217.0s) · Japanese Healing (ZH) · 稚气通透、略带脆弱感的动漫少女音｜中慢速钢琴与弦乐（约 78 BPM） +
+ + +
+
+ +
+
+ 银色钟塔与暮光咏叹 cover art +
+
+
+ 37. 银色钟塔与暮光咏叹 + MP3 + 4分01秒 (241.4s) · Japanese Healing (ZH) · 清亮圣洁少女音 + 造语吟唱｜中速古典哥特华尔兹（约 110 BPM） +
+ + +
+
+ +
+
+ 森林邮差的蒲公英 cover art +
+
+
+ 38. 森林邮差的蒲公英 + MP3 + 3分26秒 (206.7s) · Japanese Healing (ZH) · 极度轻柔、如耳语般的稚气女童声｜慢速木吉他与钢片琴（约 68 BPM） +
+ + +
+
+ +
+
+ 白鸟与雪原的祈愿 cover art +
+
+
+ 39. 白鸟与雪原的祈愿 + MP3 + 3分39秒 (220.0s) · Japanese Healing (ZH) · 空灵通透的高音女童/少女声｜慢速大气钢琴与管弦（约 70 BPM） +
+ + +
+
+ +
+
+ 发条星尘八音盒 cover art +
+
+
+ 40. 发条星尘八音盒 + MP3 + 3分18秒 (198.2s) · Japanese Healing (ZH) · 清甜稚气的少女声｜中慢速八音盒与木吉他（约 88 BPM） +
+ + +
+
+ +
+
+ 向日葵小队出发 cover art +
+
+
+ 41. 向日葵小队出发 + MP3 + 2分39秒 (159.4s) · Japanese Energetic (ZH) · 元气满满的小女孩领唱 + 童声合唱感｜中快速欢快节奏（约 118 BPM） +
+ + +
+
+ +
+
+ 焦糖布丁保卫战！ cover art +
+
+
+ 42. 焦糖布丁保卫战！ + MP3 + 3分38秒 (218.9s) · Japanese Energetic (ZH) · 软萌夸张的稚气萝莉音（高音、语速快、字密）｜快速电音流行（约 145 BPM） +
+ + +
+
+ +
+
+ 猫咪侦探事务所 cover art +
+
+
+ 43. 猫咪侦探事务所 + MP3 + 2分31秒 (151.4s) · Japanese Energetic (ZH) · 奶声奶气的女童声（6～8岁童音感）｜中速轻快木琴与铜管（约 115 BPM） +
+ + +
+
+ +
+
+ 夏夜烟火与波子汽水 cover art +
+
+
+ 44. 夏夜烟火与波子汽水 + MP3 + 3分07秒 (187.8s) · Japanese Energetic (ZH) · 清亮少年男声 + 稚气少女声（调剂少年音）｜中快速夏日流行（约 122 BPM） +
+ + +
+
+ +
+
+ 骑上扫帚送快递 cover art +
+
+
+ 45. 骑上扫帚送快递 + MP3 + 3分08秒 (188.5s) · Japanese Energetic (ZH) · 清脆阳光的小女孩声音｜中速轻快木吉他与手鼓（约 116 BPM） +
+ + +
+
+ +
+
+ 小满·青梅煮雨 cover art +
+
+
+ 46. 小满·青梅煮雨 + MP3 + 3分34秒 (214.9s) · Chinese Guofeng New · 清甜稚气的少女/女童声｜中速轻快电子国风（约 106 BPM） +
+ + +
+
+ +
+
+ 糖画摊前的小狐仙 cover art +
+
+
+ 47. 糖画摊前的小狐仙 + MP3 + 3分12秒 (192.2s) · Chinese Guofeng New · 活泼软萌的稚气少女音（字密语速快）｜中快速（约 124 BPM） +
+ + +
+
+ +
+
+ 鲛人泪·沧海月 cover art +
+
+
+ 48. 鲛人泪·沧海月 + MP3 + 3分59秒 (239.8s) · Chinese Guofeng New · 空灵仙气的少年假声/美声男声（调剂男声）｜慢速深邃（约 68 BPM） +
+ + +
+
+ +
+
+ 兰亭旧墨 cover art +
+
+
+ 49. 兰亭旧墨 + MP3 + 3分49秒 (229.3s) · Chinese Guofeng New · 清亮温婉的少女声｜中慢速古筝与二胡（约 80 BPM） +
+ + +
+
+ +
+
+ 采药小童下山记 cover art +
+
+
+ 50. 采药小童下山记 + MP3 + 3分24秒 (204.4s) · Chinese Guofeng New · 纯真稚气的小女孩声音｜中速轻快木吉他与竹笛（约 104 BPM） +
+ + +
+
+ +
+
+ 晒太阳的旧毛衣 cover art +
+
+
+ 51. 晒太阳的旧毛衣 + MP3 + 3分04秒 (184.9s) · Gentle Acoustic New · 清澈稚气的少女声｜中慢速木吉他民谣（约 80 BPM） +
+ + +
+
+ +
+
+ 奶奶的桂花糕 cover art +
+
+
+ 52. 奶奶的桂花糕 + MP3 + 3分42秒 (222.4s) · Gentle Acoustic New · 纯真软糯的小女孩声音｜慢速温情口琴与吉他（约 72 BPM） +
+ + +
+
+ +
+
+ 北方的末班渡轮 cover art +
+
+
+ 53. 北方的末班渡轮 + MP3 + 3分49秒 (229.6s) · Gentle Acoustic New · 低沉温暖的民谣男声（调剂男声）｜慢速深沉（约 74 BPM） +
+ + +
+
+ +
+
+ 云朵收集指南 cover art +
+
+
+ 54. 云朵收集指南 + MP3 + 2分56秒 (176.4s) · Gentle Acoustic New · 俏皮温柔的稚气少女音｜中速轻快（约 98 BPM） +
+ + +
+
+ +
+
+ 街角二十四小时书店 cover art +
+
+
+ 55. 街角二十四小时书店 + MP3 + 4分13秒 (253.4s) · Gentle Acoustic New · 温柔治愈的青年男声（调剂男声）｜慢速钢琴与木吉他（约 74 BPM） +
+ + +
+
+ +
+
+ 暮光归途 cover art +
+
+
+ 56. 暮光归途 + MP3 + 4分02秒 (242.6s) · Angelic Choir (ZH) · 天籁女童高音领唱 + 纯净童声合唱｜极慢速庄严管风琴与弦乐（约 66 BPM） +
+ + +
+
+ +
+
+ 冬夜银铃 cover art +
+
+
+ 57. 冬夜银铃 + MP3 + 2分59秒 (179.0s) · Angelic Choir (ZH) · 清脆女童多声部齐唱与轮唱｜中快速灵动三拍子（约 125 BPM） +
+ + +
+
+ +
+
+ 迎风飞翔的纸飞机 cover art +
+
+
+ 58. 迎风飞翔的纸飞机 + MP3 + 3分53秒 (233.2s) · Angelic Choir (ZH) · 清亮纯真的女童领唱 + 学院风童声合唱｜中速温暖（约 96 BPM） +
+ + +
+
+ +
+
+ 小星星的晚安曲 cover art +
+
+
+ 59. 小星星的晚安曲 + MP3 + 3分51秒 (232.0s) · Angelic Choir (ZH) · 稚气纯净的小女孩独唱｜慢速竖琴与小提琴（约 70 BPM） +
+ + +
+
+ +
+
+ 晨曦草地的赞美诗 cover art +
+
+
+ 60. 晨曦草地的赞美诗 + MP3 + 4分26秒 (266.0s) · Angelic Choir (ZH) · 神圣清澈的女童领唱 + 宏大童声和声（卡农式递进）｜慢速治愈（约 72 BPM） +
+ + +
+
+ + -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Does Re-encoding to HEVC Actually Shrink Your Files? I Measured It." -description: "A measured comparison of HEVC encoders on real files: NVENC at cq27 made two of three files bigger, software x265 landed 10–22% smaller at matched quality, and one 1080p source shrank 49% while another shrank 17%." -pubDate: 2026-10-08 -category: tutorials -tags: [ffmpeg, hevc, x265, nvenc, video] -ogImage: /og/does-hevc-actually-shrink-your-files.jpg -banner: /banners/does-hevc-actually-shrink-your-files.jpg -draft: false --- -## The assumption worth testing +## Want This For Your Business? -"Convert it to HEVC, the files get smaller." It is repeated often enough -that nobody measures it. I had a reason to: a library of 582 course videos -where re-encoding had made files **bigger** in 50 of 54 cases, with the -average output at 111% of the original. +I build and operate custom self-hosted AI media pipelines, automated video & audio +generation backends, and full-stack web platforms for small and medium enterprises. -So I measured. Real files, one variable at a time, quality checked against -the source rather than assumed. - -The short version: - -- Re-encoding a **H.264** source to HEVC at matched quality shrank files by -**33–66%** — but the spread is huge and depends on the source, not just its -bitrate. -- A common "good quality" hardware setting (`hevc_nvenc` at cq27) made **two -of three** test files *larger* than the source. -- At matched quality, **software x265 is 10–22% smaller than NVENC**. Hardware -encoding buys time, not size. -- Re-encoding an already-efficient source (VP9/AV1 from YouTube) buys -nothing. It grows. - -## Method: measure at equal quality, or don't bother - -Comparing "output bitrate" alone is meaningless — any encoder can hit a low -bitrate by throwing quality away. The measurement has to hold quality fixed. - -I encode a 60-second window of each source at several settings, and for each -output: - -1. measure the output's **video bitrate** as a percentage of the source's -video bitrate (not the file size — audio is excluded so the two are -comparable), and 2. measure **SSIM against the source**, frame by frame, -downscaled to 480×270 for a stable metric, reporting min and mean. - -One prerequisite: the harness must be self-consistent. Comparing a file with -*itself* through the same code path must return SSIM 1.0000. When mine did not -— it returned 0.87 on a file compared with itself, because one side was -decoded from a mid-file seek on an open-GOP source — I could have "measured" -anything and believed it. - -```bash -# self-test: this must print 1.000000, or your comparison is lying to you -ffmpeg -i any.mp4 -f rawvideo -pix_fmt yuv420p a.yuv -ffmpeg -i any.mp4 -f rawvideo -pix_fmt yuv420p b.yuv -ffmpeg -i a.yuv -i b.yuv -lavfi ssim -f null - -``` - -## Results - -Three real sources from the library I was working on, all 25–30 fps H.264, -video bitrate as a percentage of the source's video bitrate: - -| Setting | 1080p music (2720 kbps, bpp 0.0437) | 720p (1208 kbps, bpp 0.0437) | 1080×1920 vertical (4490 kbps, bpp 0.0722) | -|---|---|---|---| -| `hevc_nvenc` cq27 | 72.4% (min SSIM 0.9963) | **115.2%** (0.9935) | **119.3%** (0.9930) | -| `hevc_nvenc` cq30 | 51.0% (0.9945) | 82.6% (0.9907) | 83.8% (0.9901) | -| `hevc_nvenc` cq31 | 43.0% (0.9941) | 72.9% (0.9894) | 74.3% (0.9890) | -| `hevc_nvenc` cq30 + AQ/lookahead | 55.8% (0.9958) | 88.1% (0.9914) | 84.4% (0.9935) | -| `libx265` crf26 | **33.7%** (0.9927) | **56.7%** (0.9879) | **67.0%** (0.9913) | -| `libx265` crf28 | 26.4% (0.9910) | 45.0% (0.9852) | 53.2% (0.9888) | - -Four things fall out of that table. - -**1. The famous default is the wrong default.** cq27 is where a lot of the -internet lands when it says "HEVC, high quality" — and on two of three files -it produced a *bigger* video stream than the source it was replacing. That -is not an argument that HEVC is worse than H.264. It is an argument that a -fixed quality knob calibrated on your own test clip means nothing on someone -else's encode. - -**2. The shrink depends on the source's efficiency, not its bitrate.** Samples -1 and 2 have identical bpp (bits per pixel per second) — 0.0437 — and one -shrank 49% where the other shrank 17%. The 720p file was already a decent -encode; the 1080p one was a fat one. Same bpp, different headroom. bpp is -still the cheapest first filter (it told me which *files* to re-encode), but -it does not predict the ratio. - -**3. Software x265 beats NVENC on size, at equal quality.** Compare the rows -at similar SSIM: cq31 vs crf26 (min SSIM 0.9894 vs 0.9927 on the 720p file) -— 72.9% vs 56.7%. That is a 22% smaller file at a *higher* measured quality. -Across the set, analysis gives roughly **10–22%** extra saving. NVENC is not -worse at encoding; it is optimized for throughput. If your constraint is -disk, spend CPU; if it is time, spend GPU. - -**4. Audio becomes the floor.** Sample 1's video stream dropped to 33.7% of -the source's video bitrate — but the *file* only reached 53.1% of the -original size on the full-length test, because after that drop the AAC audio -(matched to the source's audio bitrate, as iOS requires AAC in MP4) was -about 43% of the output's total bitrate. Once the video is this compressed, -the audio track is the next lever — and it is a much smaller one. - -## What this means for VP9/AV1 sources - -Everything above is about **H.264** sources. The modern YouTube formats are -a different story, and I measured that too: 14 sampled AVC1 sources had a -median bpp of 0.053 (the "compatibility ladder" YouTube serves to devices -that cannot do VP9/AV1 is generous), while the VP9/AV1 renditions of the -same material sit far lower. Re-encodable headroom: plenty on H.264, none on -VP9/AV1. - -On a library of YouTube-sourced VP9/AV1 files, re-encoding with a hardware -encoder at cq27 produced files at **105–112%** of the originals — measurably -worse than leaving them alone. The correct action there is `-c copy`: keep -the video stream bit-identical and re-encode only the audio container (MP3 -in MP4 is not playable on iOS; AAC is). - -## The decision rules I actually use now - -1. **Classify by bpp before encoding anything.** Bits per pixel per second -(`video_bitrate / (width × height × fps)`). It is one ffprobe call and it -separates "there is headroom here" from "leave it alone" better than -resolution or filesize does. 2. **Never re-encode a source that is already -efficient.** Copy the video stream, fix the audio if it needs it, and move -on. A re-encode cannot create headroom that is not there. 3. **Set quality, -not bitrate — then verify against the source.** A metric compared frame by -frame, with a self-test proving your harness works. 4. **Choose the encoder -by your bottleneck.** Disk → x265 (10–22% smaller at equal quality). Time → -NVENC. 5. **Cap the output.** Whatever you do, assert the result is not -bigger than the source. That single assertion would have caught the entire -111% problem before it touched 54 files. 6. **Remember the audio.** After -aggressive video compression, the audio track can be the largest remaining -component of the file. - -## The measured outcome - -On the 582-file library, with the tiering above and the read-back -verification I described elsewhere: - -- **copy tier** (efficient sources, video bit-identical): ~100% of the -original — by design, quality preserved exactly. -- **mid tier** (NVENC, cq31): landed around **72%**. -- **fat tier** (x265, crf26): landed **33–44%**, median **39%**. - -That is the honest answer to "does HEVC shrink my files": it depends on what -you are re-encoding, and the only way to know your number is to measure it -on your own files with quality held fixed. +- **WhatsApp**: [012-7972969](https://wa.me/60127972969) +- **Email**: [me@hoelee.com](mailto:me@hoelee.com?subject=Inquiry%20regarding%20AI%20Audio%20Pipeline) +- **Website**: [hoelee.com](https://hoelee.com) ```` --- diff --git a/docs/consultant-review/batches/batch-05.md b/docs/consultant-review/batches/batch-05.md index 30daa0a..3d4df29 100644 --- a/docs/consultant-review/batches/batch-05.md +++ b/docs/consultant-review/batches/batch-05.md @@ -1,10 +1,10 @@ # 博客文章顾问审阅资料包 — Batch-05 > **批次编号**：`Batch-05` -> **包含文章数**：1 篇 -> **本批次总字符量**：128,681 字（严格限制 < 200,000 字） +> **包含文章数**：5 篇 +> **本批次总字符量**：159,503 字（严格限制 < 200,000 字） > **模式**：BILINGUAL -> **生成时间**：2026-10-27 04:23:31 +> **生成时间**：2026-10-27 05:23:38 # 外部顾问审阅前置须知与工程背景规范 @@ -39,15 +39,19 @@ ## 本批次文章目录索引 -1. [tutorials] **用 YuE2 BF16 生成完整长歌：突破时长限制与音频硬截断** (`generating-full-length-songs-with-yue2-comfyui-zh`) — ~128,681 字符 +1. [tutorials] **给没有时间戳的歌曲自动生成字幕** (`generating-subtitles-for-songs-without-timestamps`) — ~88,620 字符 +2. [tutorials] **购买主机前，如何核实一家服务商的真实情况** (`how-to-verify-a-hosting-provider-before-you-buy`) — ~9,649 字符 +3. [web3] **一个基于 Chainlink VRF v2.5 与 Automation 的自动化彩票合约** (`chainlink-vrf-v2-lottery-contract`) — ~25,244 字符 +4. [web3] **全链上 SVG NFT：把艺术作品放进合约里** (`fully-on-chain-svg-nfts`) — ~24,005 字符 +5. [web3] **为什么在 Windows 上克隆后，我的链上 NFT 图像变了** (`why-my-on-chain-nft-art-changed-on-windows`) — ~11,985 字符 --- -## 文章 01 / 01：用 YuE2 BF16 生成完整长歌：突破时长限制与音频硬截断 -- **Slug**: `generating-full-length-songs-with-yue2-comfyui` -- **英文标题**: Generating Full-Length Songs with YuE2 BF16: Overcoming Duration Limits and Audio Truncation -- **分类**: `tutorials` | **发布日期**: 2026-10-26 -- **线上地址**: https://blog.hoelee.com/posts/generating-full-length-songs-with-yue2-comfyui/ (EN) | https://blog.hoelee.com/posts/zh/generating-full-length-songs-with-yue2-comfyui/ (ZH) +## 文章 01 / 05：给没有时间戳的歌曲自动生成字幕 +- **Slug**: `generating-subtitles-for-songs-without-timestamps` +- **英文标题**: Generating Subtitles for Songs That Ship Without Timestamps +- **分类**: `tutorials` | **发布日期**: 2026-10-20 +- **线上地址**: https://blog.hoelee.com/posts/generating-subtitles-for-songs-without-timestamps/ (EN) | https://blog.hoelee.com/posts/zh/generating-subtitles-for-songs-without-timestamps/ (ZH) - **本篇审核状态**: ⏳ 待审核 (Pending) ### ✍️ 顾问审阅意见区 (Review Feedback) @@ -61,63 +65,175 @@ ### 🇨🇳 中文版原文 (Chinese Markdown Twin) ````markdown --- -title: "用 YuE2 BF16 生成完整长歌：突破时长限制与音频硬截断" -description: "AI 音乐工具常在歌词中途戛然而止。本文详解如何利用 YuE2 BF16、295秒余量预留公式与本地 ComfyUI，成功批量跑出 60 首自然收尾的完整长歌全集。" -pubDate: 2026-10-26 +title: "给没有时间戳的歌曲自动生成字幕" +description: "AI 音乐工具只给你音频 —— 没有逐行时间，没有卡拉 OK。这是我给 18 首歌生成 LRC 与 SRT 字幕的完整流程，为什么单纯转写不行，以及让歌词框滚到底的两个 bug。" +pubDate: 2026-10-20 category: tutorials blueprint: ai-agent-automation -blueprintLayer: L4-aigc-media-pipelines +blueprintLayer: L3-local-gpu-inference stream: creative-lab -tags: ["ai-music", "yue2", "comfyui", "audio-engineering", "subtitles", "python"] -ogImage: "/og/zh-generating-full-length-songs-with-yue2-comfyui.jpg" -banner: "/banners/generating-full-length-songs-with-yue2-comfyui.jpg" +tags: ["subtitles", "lrc", "srt", "whisper", "forced-alignment", "ai-music", "python"] +ogImage: "/og/zh-generating-subtitles-for-songs-without-timestamps.jpg" +banner: "/banners/generating-subtitles-for-songs-without-timestamps.jpg" draft: false --- +我试过的每一个 AI 音乐工具，交出来的东西形状都一样：一个音频文件。没有一个会告诉你**每一句是什么时候唱的**。只想听的话没问题，但只要你想把歌词放到屏幕上 —— 卡拉 OK 的那一行、视频字幕、播放器下面的滚动框 —— 就完全不够用。 -在当前几乎所有的 AI 音乐生成方案中，最普遍且让人抓狂的缺陷莫过于： -**歌曲唱到一半，突然被硬生生切断**。 +时间轴才是全部难点。音频反而是简单的部分。 -不管是各大商用平台还是开源权重，受限于固定的声学 Token 训练窗口，只要我们尝试塞入完整的 -叙事性歌词结构——即标准的 9 阶段曲式 `[Intro] -> [Verse 1] -> [Chorus] -> [Interlude] -> [Verse 2] -> [Chorus] -> [Bridge] -> [Chorus] -> [Outro]`——生成引擎就极易在进入副歌高潮或尾奏时撞上时长硬天花板，在歌手吐字的中途突兀静音。 +这是我最后跑通的一套流程，用在本地生成的 18 首英文歌上，外加它在浏览器里崩掉的两个地方。下面全部可复现：它唯一需要你提供的东西，是**你已经写好的歌词**。 -在单张 RTX 3060 12GB 显卡上经历数百次实跑与测试后，我摸索出了一套能够稳定产出 -**4 分钟以上、具备自然淡出收尾、绝不截断**的完整长歌工程链路。 +## 你真正需要的两样东西 -本文将拆解导致音频硬截断的底层成因、295 秒余量预留公式，并完整交付 **60 首高质量长歌全集**（每首均配套 1024×1024 独立 WebP 封面与逐行同步滚动歌词）。 +两种字幕格式，共用同一份数据结构： ---- +| 格式 | 用在哪 | 长什么样 | +|---|---|---| +| **LRC** | 音乐播放器、卡拉 OK 块、`