diff --git a/docs/consultant-review/AUDIT-TRACKER.md b/docs/consultant-review/AUDIT-TRACKER.md index 53522ff..2b9007b 100644 --- a/docs/consultant-review/AUDIT-TRACKER.md +++ b/docs/consultant-review/AUDIT-TRACKER.md @@ -1,6 +1,6 @@ # 全站文章顾问审阅状态总控台 (Audit Tracker) -> **最后同步时间**:2026-10-27 05:32:31 +> **最后同步时间**:2026-10-27 05:36:05 > **全站总文章数**:91 篇 | **已审批完成**:74 篇 (81.3%) | **待审阅**:17 篇 ## 一、操作命令速查 @@ -70,17 +70,17 @@ | 56 | `devops` | `when-smart-says-healthy-but-your-raid-is-corrupting-data` | SMART 显示健康,RAID 却在悄悄损坏数据 | ✅ 已通过 | Batch-02 | 2026-10-27 05:21 | 外部顾问审阅完成,已采纳并落地全量修改意见(含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理) | | 57 | `devops` | `why-chrome-forgets-its-tabs-in-a-container` | 为什么容器里的 Chrome 会忘记标签页 —— 以及我是怎么修好的 | ✅ 已通过 | Batch-02 | 2026-10-27 05:21 | 外部顾问审阅完成,已采纳并落地全量修改意见(含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理) | | 58 | `devops` | `why-telegram-bot-notifications-die` | Telegram 机器人通知为什么发不出去:IPv6、DNS 和一个追查了一整天的 400 | ✅ 已通过 | Batch-02 | 2026-10-27 05:23 | 外部顾问审阅完成,已采纳并落地全量修改意见(含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理) | -| 59 | `devops` | `why-your-cdp-clicks-silently-fail` | 为什么你的 CDP 点击会静默失效:平滑滚动、最小化窗口与 JS 派发事件 | ⏳ 待审阅 | Batch-02 | - | | -| 60 | `devops` | `why-your-grafana-dashboard-shows-no-data` | Prometheus 明明有数据,Grafana 面板却全是 No data 的原因 | ⏳ 待审阅 | Batch-02 | - | | -| 61 | `devops` | `your-disk-full-alert-is-lying` | 你的“磁盘将满”告警在说谎:别再按百分比告警 | ⏳ 待审阅 | Batch-02 | - | | -| 62 | `engineering` | `adding-english-mode-to-a-chinese-only-web-app` | 浏览器翻译没有 API:我给一个纯中文网页应用加上了英文模式 | ⏳ 待审阅 | Batch-02 | - | | -| 63 | `engineering` | `debugging-remotion-video-jitter` | 排查 Remotion 视频抽搐:为什么在背景图挂载音频低频 Scale 会毁了转场 | ⏳ 待审阅 | Batch-02 | - | | -| 64 | `engineering` | `embedding-fonts-so-a-pdf-looks-the-same-everywhere` | 把字体嵌进 PDF,让它在任何机器上都长一样 | ⏳ 待审阅 | Batch-02 | - | | -| 65 | `engineering` | `jpa-version-field-lost-update` | 那个把静默丢失更新变成 409 的 JPA 字段:@Version 乐观锁 | ⏳ 待审阅 | Batch-02 | - | | -| 66 | `engineering` | `migrating-codeigniter-iis-to-openlitespeed` | 把 CodeIgniter 4 应用从 IIS 迁移到 OpenLiteSpeed | ⏳ 待审阅 | Batch-02 | - | | -| 67 | `engineering` | `scraping-bot-walled-marketplace-warm-browser-session` | 用「热浏览器会话」爬取有反爬墙的电商网站 | ⏳ 待审阅 | Batch-02 | - | | -| 68 | `engineering` | `two-ways-a-javascript-file-stops-working-and-looks-fine` | JavaScript 文件静默失效的两种方式:而页面看上去一切正常 | ⏳ 待审阅 | Batch-02 | - | | -| 69 | `engineering` | `verifying-a-pdf-report-page-by-page` | 逐页验证一份 19 页的 PDF 报告 | ⏳ 待审阅 | Batch-02 | - | | +| 59 | `devops` | `why-your-cdp-clicks-silently-fail` | 为什么你的 CDP 点击会静默失效:平滑滚动、最小化窗口与 JS 派发事件 | ⏳ 待审阅 | Batch-01 | - | | +| 60 | `devops` | `why-your-grafana-dashboard-shows-no-data` | Prometheus 明明有数据,Grafana 面板却全是 No data 的原因 | ⏳ 待审阅 | Batch-01 | - | | +| 61 | `devops` | `your-disk-full-alert-is-lying` | 你的“磁盘将满”告警在说谎:别再按百分比告警 | ⏳ 待审阅 | Batch-01 | - | | +| 62 | `engineering` | `adding-english-mode-to-a-chinese-only-web-app` | 浏览器翻译没有 API:我给一个纯中文网页应用加上了英文模式 | ⏳ 待审阅 | Batch-01 | - | | +| 63 | `engineering` | `debugging-remotion-video-jitter` | 排查 Remotion 视频抽搐:为什么在背景图挂载音频低频 Scale 会毁了转场 | ⏳ 待审阅 | Batch-01 | - | | +| 64 | `engineering` | `embedding-fonts-so-a-pdf-looks-the-same-everywhere` | 把字体嵌进 PDF,让它在任何机器上都长一样 | ⏳ 待审阅 | Batch-01 | - | | +| 65 | `engineering` | `jpa-version-field-lost-update` | 那个把静默丢失更新变成 409 的 JPA 字段:@Version 乐观锁 | ⏳ 待审阅 | Batch-01 | - | | +| 66 | `engineering` | `migrating-codeigniter-iis-to-openlitespeed` | 把 CodeIgniter 4 应用从 IIS 迁移到 OpenLiteSpeed | ⏳ 待审阅 | Batch-01 | - | | +| 67 | `engineering` | `scraping-bot-walled-marketplace-warm-browser-session` | 用「热浏览器会话」爬取有反爬墙的电商网站 | ⏳ 待审阅 | Batch-01 | - | | +| 68 | `engineering` | `two-ways-a-javascript-file-stops-working-and-looks-fine` | JavaScript 文件静默失效的两种方式:而页面看上去一切正常 | ⏳ 待审阅 | Batch-01 | - | | +| 69 | `engineering` | `verifying-a-pdf-report-page-by-page` | 逐页验证一份 19 页的 PDF 报告 | ⏳ 待审阅 | Batch-01 | - | | | 70 | `engineering` | `why-500000-lines-of-commit-churn-only-left-230000-in-production` | 30 天提交 51.8 万行,生产环境只留下 23.5 万行 | ✅ 已通过 | Batch-04 | 2026-10-27 05:21 | 外部顾问审阅完成,已采纳并落地全量修改意见(含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理) | | 71 | `notes` | `adding-a-domain-to-someone-elses-mail-server` | 给别人家的邮件服务器加一个域名 | ✅ 已通过 | Batch-04 | 2026-10-27 05:21 | 外部顾问审阅完成,已采纳并落地全量修改意见(含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理) | | 72 | `notes` | `hello-world` | 你好,世界 —— 关于这个博客 | ✅ 已通过 | Batch-04 | 2026-10-27 05:21 | 外部顾问审阅完成,已采纳并落地全量修改意见(含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理) | @@ -97,9 +97,9 @@ | 83 | `notes` | `why-your-windows-mcp-calls-silently-fail` | 为什么你的 Windows 桌面 MCP 调用会静默失败:攻破 10 秒空闲门禁与避坑实录 | ✅ 已通过 | Batch-04 | 2026-10-27 05:21 | 外部顾问审阅完成,已采纳并落地全量修改意见(含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理) | | 84 | `tutorials` | `calibrating-intel-quick-sync-hevc-quality` | 校准 Intel Quick Sync HEVC:28 与 30 之间的悬崖,和一个毫无作用的 preset | ✅ 已通过 | Batch-04 | 2026-10-27 05:21 | 外部顾问审阅完成,已采纳并落地全量修改意见(含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理) | | 85 | `tutorials` | `does-hevc-actually-shrink-your-files` | 转成 HEVC 到底会不会让文件变小?我实测了 | ✅ 已通过 | Batch-04 | 2026-10-27 05:21 | 外部顾问审阅完成,已采纳并落地全量修改意见(含数字矛盾/OPSEC脱敏/双语CTA/排版折叠/禁用词清理) | -| 86 | `tutorials` | `generating-full-length-songs-with-yue2-comfyui` | 用 YuE2 BF16 生成完整长歌:突破时长限制与音频硬截断 | ⏳ 待审阅 | Batch-04 | - | | -| 87 | `tutorials` | `generating-subtitles-for-songs-without-timestamps` | 给没有时间戳的歌曲自动生成字幕 | ⏳ 待审阅 | Batch-05 | - | | -| 88 | `tutorials` | `how-to-verify-a-hosting-provider-before-you-buy` | 购买主机前,如何核实一家服务商的真实情况 | ⏳ 待审阅 | Batch-05 | - | | -| 89 | `web3` | `chainlink-vrf-v2-lottery-contract` | 一个基于 Chainlink VRF v2.5 与 Automation 的自动化彩票合约 | ⏳ 待审阅 | Batch-05 | - | | -| 90 | `web3` | `fully-on-chain-svg-nfts` | 全链上 SVG NFT:把艺术作品放进合约里 | ⏳ 待审阅 | Batch-05 | - | | -| 91 | `web3` | `why-my-on-chain-nft-art-changed-on-windows` | 为什么在 Windows 上克隆后,我的链上 NFT 图像变了 | ⏳ 待审阅 | Batch-05 | - | | +| 86 | `tutorials` | `generating-full-length-songs-with-yue2-comfyui` | 用 YuE2 BF16 生成完整长歌:突破时长限制与音频硬截断 | ⏳ 待审阅 | Batch-03 | - | | +| 87 | `tutorials` | `generating-subtitles-for-songs-without-timestamps` | 给没有时间戳的歌曲自动生成字幕 | ⏳ 待审阅 | Batch-04 | - | | +| 88 | `tutorials` | `how-to-verify-a-hosting-provider-before-you-buy` | 购买主机前,如何核实一家服务商的真实情况 | ⏳ 待审阅 | Batch-04 | - | | +| 89 | `web3` | `chainlink-vrf-v2-lottery-contract` | 一个基于 Chainlink VRF v2.5 与 Automation 的自动化彩票合约 | ⏳ 待审阅 | Batch-04 | - | | +| 90 | `web3` | `fully-on-chain-svg-nfts` | 全链上 SVG NFT:把艺术作品放进合约里 | ⏳ 待审阅 | Batch-04 | - | | +| 91 | `web3` | `why-my-on-chain-nft-art-changed-on-windows` | 为什么在 Windows 上克隆后,我的链上 NFT 图像变了 | ⏳ 待审阅 | Batch-04 | - | | diff --git a/docs/consultant-review/audit-tracker.json b/docs/consultant-review/audit-tracker.json index 97774bd..b87a131 100644 --- a/docs/consultant-review/audit-tracker.json +++ b/docs/consultant-review/audit-tracker.json @@ -19,7 +19,7 @@ "category": "engineering", "pub_date": "2025-06-11", "status": "pending", - "batch_id": "Batch-02", + "batch_id": "Batch-01", "approved_at": null, "notes": "", "en_chars": 11859, @@ -123,7 +123,7 @@ "category": "web3", "pub_date": "2024-12-10", "status": "pending", - "batch_id": "Batch-05", + "batch_id": "Batch-04", "approved_at": null, "notes": "", "en_chars": 15801, @@ -136,7 +136,7 @@ "category": "engineering", "pub_date": "2025-11-26", "status": "pending", - "batch_id": "Batch-02", + "batch_id": "Batch-01", "approved_at": null, "notes": "", "en_chars": 9584, @@ -175,7 +175,7 @@ "category": "engineering", "pub_date": "2025-01-22", "status": "pending", - "batch_id": "Batch-02", + "batch_id": "Batch-01", "approved_at": null, "notes": "", "en_chars": 8033, @@ -227,7 +227,7 @@ "category": "web3", "pub_date": "2024-10-08", "status": "pending", - "batch_id": "Batch-05", + "batch_id": "Batch-04", "approved_at": null, "notes": "", "en_chars": 14598, @@ -240,7 +240,7 @@ "category": "tutorials", "pub_date": "2026-10-26", "status": "pending", - "batch_id": "Batch-04", + "batch_id": "Batch-03", "approved_at": null, "notes": "", "en_chars": 131667, @@ -253,7 +253,7 @@ "category": "tutorials", "pub_date": "2026-10-20", "status": "pending", - "batch_id": "Batch-05", + "batch_id": "Batch-04", "approved_at": null, "notes": "", "en_chars": 46399, @@ -370,7 +370,7 @@ "category": "tutorials", "pub_date": "2025-11-26", "status": "pending", - "batch_id": "Batch-05", + "batch_id": "Batch-04", "approved_at": null, "notes": "", "en_chars": 6743, @@ -409,7 +409,7 @@ "category": "engineering", "pub_date": "2026-08-19", "status": "pending", - "batch_id": "Batch-02", + "batch_id": "Batch-01", "approved_at": null, "notes": "", "en_chars": 12217, @@ -435,7 +435,7 @@ "category": "engineering", "pub_date": "2026-09-27", "status": "pending", - "batch_id": "Batch-02", + "batch_id": "Batch-01", "approved_at": null, "notes": "", "en_chars": 14943, @@ -656,7 +656,7 @@ "category": "engineering", "pub_date": "2026-09-13", "status": "pending", - "batch_id": "Batch-02", + "batch_id": "Batch-01", "approved_at": null, "notes": "", "en_chars": 11245, @@ -890,7 +890,7 @@ "category": "engineering", "pub_date": "2025-02-20", "status": "pending", - "batch_id": "Batch-02", + "batch_id": "Batch-01", "approved_at": null, "notes": "", "en_chars": 7750, @@ -955,7 +955,7 @@ "category": "engineering", "pub_date": "2026-09-28", "status": "pending", - "batch_id": "Batch-02", + "batch_id": "Batch-01", "approved_at": null, "notes": "", "en_chars": 12525, @@ -1085,7 +1085,7 @@ "category": "web3", "pub_date": "2026-08-19", "status": "pending", - "batch_id": "Batch-05", + "batch_id": "Batch-04", "approved_at": null, "notes": "", "en_chars": 7725, @@ -1111,7 +1111,7 @@ "category": "devops", "pub_date": "2025-03-19", "status": "pending", - "batch_id": "Batch-02", + "batch_id": "Batch-01", "approved_at": null, "notes": "", "en_chars": 8010, @@ -1124,7 +1124,7 @@ "category": "devops", "pub_date": "2026-05-17", "status": "pending", - "batch_id": "Batch-02", + "batch_id": "Batch-01", "approved_at": null, "notes": "", "en_chars": 7347, @@ -1163,7 +1163,7 @@ "category": "devops", "pub_date": "2026-06-24", "status": "pending", - "batch_id": "Batch-02", + "batch_id": "Batch-01", "approved_at": null, "notes": "", "en_chars": 5930, diff --git a/docs/consultant-review/batches/batch-01.md b/docs/consultant-review/batches/batch-01.md index d76d8c5..1230356 100644 --- a/docs/consultant-review/batches/batch-01.md +++ b/docs/consultant-review/batches/batch-01.md @@ -1,10 +1,10 @@ # 博客文章顾问审阅资料包 — Batch-01 > **批次编号**:`Batch-01` -> **包含文章数**:8 篇 -> **本批次总字符量**:174,959 字(严格限制 < 200,000 字) +> **包含文章数**:11 篇 +> **本批次总字符量**:167,652 字(严格限制 < 200,000 字) > **模式**:BILINGUAL -> **生成时间**:2026-10-27 05:23:38 +> **生成时间**:2026-10-27 05:36:05 # 外部顾问审阅前置须知与工程背景规范 @@ -39,22 +39,25 @@ ## 本批次文章目录索引 -1. [ai] **Claude 和 GPT 明明更强,我为什么还是买了 GPU 跑本地 AI** (`why-i-still-bought-a-local-gpu`) — ~16,735 字符 -2. [ai] **为什么我的 AI 歌曲总是丢结尾:在 12GB 显卡上跑 YuE2** (`yue2-on-a-12gb-gpu-songs-that-lost-their-endings`) — ~53,369 字符 -3. [case-studies] **我如何用 FLUX.1 Kontext 打造 AI 家具合成演示** (`ai-furniture-compositing-with-flux-kontext`) — ~19,749 字符 -4. [case-studies] **从一首歌到 4 分钟完整 MV:我如何用 Gemini、AI 生成与 Filmora 剪辑串起全流程实操** (`from-audio-to-a-4-minute-ai-music-video`) — ~16,195 字符 -5. [case-studies] **我是如何用 Remotion 与 Docker CDP 打造全自动 MV 生成与多语种油管发布流的** (`how-i-built-an-automated-music-video-pipeline`) — ~14,196 字符 -6. [case-studies] **我是如何构建 DigiKedai Telegram AI 客服机器人的** (`how-i-built-the-digikedai-telegram-bot`) — ~15,990 字符 -7. [case-studies] **我是如何托管这个博客的:Astro、Gitea Actions 与自托管 CI/CD** (`how-i-host-this-blog`) — ~10,053 字符 -8. [case-studies] **我用 Suno AI 做出自己的歌** (`how-i-made-my-own-songs-with-suno-ai`) — ~28,672 字符 +1. [devops] **为什么你的 CDP 点击会静默失效:平滑滚动、最小化窗口与 JS 派发事件** (`why-your-cdp-clicks-silently-fail`) — ~12,514 字符 +2. [devops] **Prometheus 明明有数据,Grafana 面板却全是 No data 的原因** (`why-your-grafana-dashboard-shows-no-data`) — ~11,212 字符 +3. [devops] **你的“磁盘将满”告警在说谎:别再按百分比告警** (`your-disk-full-alert-is-lying`) — ~8,912 字符 +4. [engineering] **浏览器翻译没有 API:我给一个纯中文网页应用加上了英文模式** (`adding-english-mode-to-a-chinese-only-web-app`) — ~18,070 字符 +5. [engineering] **排查 Remotion 视频抽搐:为什么在背景图挂载音频低频 Scale 会毁了转场** (`debugging-remotion-video-jitter`) — ~15,111 字符 +6. [engineering] **把字体嵌进 PDF,让它在任何机器上都长一样** (`embedding-fonts-so-a-pdf-looks-the-same-everywhere`) — ~11,465 字符 +7. [engineering] **那个把静默丢失更新变成 409 的 JPA 字段:@Version 乐观锁** (`jpa-version-field-lost-update`) — ~19,575 字符 +8. [engineering] **把 CodeIgniter 4 应用从 IIS 迁移到 OpenLiteSpeed** (`migrating-codeigniter-iis-to-openlitespeed`) — ~23,361 字符 +9. [engineering] **用「热浏览器会话」爬取有反爬墙的电商网站** (`scraping-bot-walled-marketplace-warm-browser-session`) — ~16,928 字符 +10. [engineering] **JavaScript 文件静默失效的两种方式:而页面看上去一切正常** (`two-ways-a-javascript-file-stops-working-and-looks-fine`) — ~11,294 字符 +11. [engineering] **逐页验证一份 19 页的 PDF 报告** (`verifying-a-pdf-report-page-by-page`) — ~19,210 字符 --- -## 文章 01 / 08:Claude 和 GPT 明明更强,我为什么还是买了 GPU 跑本地 AI -- **Slug**: `why-i-still-bought-a-local-gpu` -- **英文标题**: Why I Still Bought a GPU to Run AI When Claude and GPT Are Obviously Stronger -- **分类**: `ai` | **发布日期**: 2026-09-21 -- **线上地址**: https://blog.hoelee.com/posts/why-i-still-bought-a-local-gpu/ (EN) | https://blog.hoelee.com/posts/zh/why-i-still-bought-a-local-gpu/ (ZH) +## 文章 01 / 11:为什么你的 CDP 点击会静默失效:平滑滚动、最小化窗口与 JS 派发事件 +- **Slug**: `why-your-cdp-clicks-silently-fail` +- **英文标题**: Why Your CDP Clicks Silently Fail: Smooth Scroll, Minimised Windows, and JS-Dispatched Events +- **分类**: `devops` | **发布日期**: 2025-03-19 +- **线上地址**: https://blog.hoelee.com/posts/why-your-cdp-clicks-silently-fail/ (EN) | https://blog.hoelee.com/posts/zh/why-your-cdp-clicks-silently-fail/ (ZH) - **本篇审核状态**: ⏳ 待审核 (Pending) ### ✍️ 顾问审阅意见区 (Review Feedback) @@ -68,451 +71,262 @@ ### 🇨🇳 中文版原文 (Chinese Markdown Twin) ````markdown --- -title: "Claude 和 GPT 明明更强,我为什么还是买了 GPU 跑本地 AI" -description: "Claude Opus 和 GPT 确实比我能在本地跑的任何模型都强。但我还是买了 GPU——这篇讲成本、控制权与可用性,附实际的判断标准。" -pubDate: 2026-09-21 -category: ai -blueprint: ai-agent-automation -blueprintLayer: L3-local-gpu-inference -tags: [local-llm, gpu, llama-cpp, lm-studio, agents, self-hosting, cost] -ogImage: /og/zh-why-i-still-bought-a-local-gpu.jpg -banner: /banners/why-i-still-bought-a-local-gpu.jpg +title: "为什么你的 CDP 点击会静默失效:平滑滚动、最小化窗口与 JS 派发事件" +description: "Chrome DevTools Protocol 点击报告成功却毫无反应的三个原因:平滑滚动导致的过期坐标、最小化窗口被丢弃的坐标输入、以及页面收到了点击但拒绝了它。" +pubDate: 2025-03-19 +updatedDate: 2026-09-29 +category: devops +tags: [cdp, chrome, react, browser-automation, debugging, python] +ogImage: /og/zh-why-your-cdp-clicks-silently-fail.jpg +banner: /banners/why-your-cdp-clicks-silently-fail.jpg draft: false -dynamicCta: true --- -在我管理的自托管生产节点(Self-hosted production infrastructure)上,常年运行着全套 Docker 容器栈、NAS 存储、Cloudflare 隧道与自动化调度任务。但我照样在付顶级模型的钱。 +如果你用 Chrome DevTools Protocol 驱动网页应用,你一定见过这个失败:脚本报告 `clicked @591,361`,元素明明在,点击也确实发出去了 —— 然后什么都没发生。没有异常,没有控制台报错,页面毫无变化。 -所以每次别人知道我也在跑本地模型,问的都是同一句话:*Claude 和 GPT 就是 -比较强,本地模型又麻烦、又慢、效果还比较差,那你到底为什么要花钱买 GPU?* +我在自动化某平台卖家后台时整整耗了一个晚上在这上面。找到的三个原因值得写下来,因为**它们对「报告点击成功」的那段代码全都是隐形的**。 -如果只比模型能力,我完全同意。今天要处理一个全新的问题、做复杂推理、规划 -产品架构、或者跑一个很长的 Agent 任务,我大部分时间还是先用 Claude、GPT。 -我部署本地模型,从来不是因为我觉得它已经打赢闭源模型。 +## 原因一:坐标过期,因为页面是平滑滚动的 -而是因为 AI 越来越好用之后,出现了另一个问题:我做的每一件事,都开始建立 -在一个我自己完全不能控制的服务上。 +最直观的点击辅助函数大概长这样: ---- +```python +r = self.ev("""(()=>{const e=%s; e.scrollIntoView({block:'center'}); + const b=e.getBoundingClientRect(); + return JSON.stringify({x:b.left+b.width/2, y:b.top+b.height/2});})()""") +self.click_xy(json.loads(r)["x"], json.loads(r)["y"]) +``` -## 真正的问题不是能力,是依赖 +滚动到可视区、读取元素矩形、点它的中心。逻辑没错 —— 除非页面有 `scroll-behavior: smooth`(现代 UI 组件库里非常常见)。那时 `scrollIntoView()` 启动的是**动画**,而同一个 tick 里的 `getBoundingClientRect()` 返回的是元素**即将离开**的位置。你的点击落在元素**原来**的地方,落在当时在那儿的别的东西上,而日志忠实地告诉你:它点击了你要求的坐标。 -今天模型可以用多少、方案怎么收费、什么地区能不能用、版本什么时候换掉, -最后都不是使用者决定的。Anthropic 本来就有自己的地区限制,有些市场不能 -直接买 Claude 的商业服务;OpenAI 的方案、额度跟模型也一直在调。这些是商业 -服务,变动本来就可以预期。 +修法是两行 —— 关掉平滑滚动,然后重新读矩形: -以前 ChatGPT 只是偶尔拿来问问题,今天不能用,我明天再问没什么差别。 +```python +self.ev("""(()=>{document.documentElement.style.scrollBehavior='auto'; + if(document.body) document.body.style.scrollBehavior='auto'; return 'ok';})()""") +moved = self.ev("""(()=>{const e=%s; if(!e) return 'no'; + e.scrollIntoView({block:'center', behavior:'instant'}); return 'ok';})()""" % sel) +time.sleep(0.35) # 等布局稳定 +# 然后再读矩形(元素晚一帧才渲染的话,外面再套一层重试) +``` -但当我开始拿 AI 写程式、跑 Agent、分析文件、处理客服、整理公司资料,甚至 -把它接进每天都在跑的工作流程之后,问题就变了。当 AI 从聊天工具变成工作里 -的基础设施,你要考虑的就不只是「哪一个模型比较强」,而是: +两个细节很关键:`scrollIntoView` 上的 `behavior:'instant'` 会覆盖 CSS;而读取前那一小段等待,才是让坐标可信的原因。没有重试循环的话,这个辅助函数在晚一帧渲染的元素上仍会偶发失败 —— **偶发失败是最糟的一类,因为它会让你怀疑自己的判断。** -> 如果我今天很依赖这个模型,但有一天它的使用条件改掉,我原本的工作还能不能继续? +## 原因二:窗口最小化时,坐标点击被丢弃 -这就是为什么我觉得,闭源模型越强,本地模型的重要性会越来越高。因为闭源 -模型太好用,大家就会把越来越多事情交给它;依赖越深的同时,也会越被这些 -闭源模型控制。 +修好之后,我的点击辅助函数在一次性探针里能用,一放进真正的脚本就再也没成功过。同样的选择器、同样的坐标、同样的页面。差别最后落在可见性上: -还有一件事很值得讲清楚。每一家大模型公司都在谈 AI 安全、模型风险,甚至 -限制某些内容、地区或使用方式,但同时每一家又都还在拼命把模型做得更强—— -因为模型能力本身就是竞争力,你不做,别人还是会做。 +``` +document.visibilityState → "hidden" +document.hasFocus() → true +``` -所以最后真正被控制的,往往不是 AI 要不要继续发展,而是谁能用、怎么用、 -用多少,以及模型公司愿意提供到什么程度。对一般人来说,这可能只是服务 -条款;对企业来说,这就是**供应商风险**。假设一间公司的核心流程,加上后面 -所有 Agent、工具、资料处理,全部绑在同一个供应商身上,那它其实不只是在买 -Token——它是在把一部分工作的主控权交出去。 +当目标标签页不是可见窗口的活动标签页时,CDP 合成的坐标级鼠标事件是不可靠的 —— 浏览器可能接受、可能限流、也可能直接不往任何地方送。注意 `hasFocus()` **在文档隐藏时依然返回 `true`**,所以它不是有效的判断依据;要看的是 `document.hidden` / `visibilityState`。 ---- +两个修法,我都用: -## 「可是本地模型就是比较差啊」——对,但这是错误的比较方式 +```python +def activate(self): # 把标签页提到窗口最前 + self.cdp("Page.bringToFront") + time.sleep(0.2) -这是最诚实的反对意见,而且它是事实。拿一个能放进 24GB、32GB 消费级显卡的 -模型,直接跟 Claude Opus 全面比较,很多复杂任务就是会输。这没什么好辩的。 +def click_js(self, sel): # 在页面内部派发事件来点击 + return self.ev("""(()=>{const e=%s; if(!e) return 'no'; + ['mousedown','mouseup','click'].forEach(k=> + e.dispatchEvent(new MouseEvent(k,{bubbles:true,cancelable:true,view:window}))); + return 'js-clicked';})()""" % sel) +``` -但大家很容易用了错误的比较方式。真正该问的不是「Qwen 能不能打赢 Claude」, -而是: +`Page.bringToFront` 解决了常见情况。但真正从不出错的是 JS 派发的点击:它不依赖窗口是否可见,而且 React 挂在根节点上的事件监听器会收到它,和收到真实点击的方式完全一样 —— 因为事件冒泡到的是同一个根。一个用坐标点击五次才能打开一次的日期选择器,用这种方式**每次都开**。 -> **这个能力差距,到底有没有大到值得你永远用最贵的模型来做这件事?** +需要知道的取舍:派发事件会跳过 hit-testing,所以它也会「点到」用户根本够不到的元素(被遮住、滚出屏幕、透明度为零)。在这里这是优点 —— 如果你用它去掩盖布局 bug,那就是坑 —— 所以凡是可视化的东西我都用坐标点击,而遇到自定义控件时才伸手去拿 `click_js`。 -因为这个交换并不对称。二三十 B 的模型确实落后顶级模型一截,但成本低一到 -两个数量级,或者直接跑在你已经买的硬体上。比例不同,决策就不同。 +## 原因三:页面收到了点击,然后拒绝了它 -### 我把模型能力看成分界线,不是排行榜 +这个最浪费时间,因为它的症状和前两个一模一样:点击发出去了,什么都没变。 -如果一个任务低于某个能力门槛,那就是做不了,当然用强模型。但一旦跨过 -那条线,后面真正影响我使用方式的,就不再是 benchmark 分数,而是成本、 -速度、能不能长时间运行,以及我能不能控制这套系统。 +区分它们只需要注入一个监听器: -以我的实际分法: +```js +window.__ev = []; +['mousedown','mouseup','click'].forEach(k => + document.addEventListener(k, e => window.__ev.push(k + ':' + e.target.tagName), true)); +``` -| 任务 | 我用什么 | -|---|---| -| 问题还很模糊、连方向都不知道 | 顶级闭源模型——前面判断错了,后面做再多都没意义 | -| 产品架构、资料格式、验收条件 | 顶级闭源模型 | -| 照著已定规格建专案骨架、改档案、跑测试、看到错误再修 | 本地模型 | -| 长时间 Agent、重试、批次工作、大量内容产出 | 本地模型 | +- 有事件记录 → 输入链路是通的,问题在应用侧(原因三),或者在你读取结果的方式上(见下)。 +- 一条都没有 → 是原因一或原因二。 -关键是第二块。一旦方向确定、Agent 只是在执行规格,一个任务可能会呼叫模型 -二十次、五十次。这五十次真的每一次都需要市面上最贵的模型吗?通常不需要。 -Agent 已经不是在帮我决定产品要怎么做,它是在按照前面已经写好的规格执行。 -只要本地模型可以正确理解这些规格、知道工具怎么用,它就已经跨过可以工作的 -门槛。 +挂上监听器后我点了一次 `Confirm`,缓冲区满了 —— 说明点击没问题,是应用在说「不」。理由在一段我从没想过要看的文字里: ---- +> “Please create a new shop welcome voucher after the existing one is expired.” -## Agent 会把小小的价差一直放大 +表单拒绝创建同类型的第二张券。没有字段级红字、头 500 毫秒里没有提示条、URL 也没变。**拒绝是真的,点击是无辜的。** -用聊天,是问一次、回答一次。Claude 一次比本地模型贵很多,你也不会有特别 -感受。 +这一类 bug 还有两个工具: -但 Agent 不是一次。一个任务可能呼叫模型二十次、五十次,失败之后还会重新 -跑。模型单次价格的差距,放进这种连续任务里就会被一直放大。 +```js +document.elementFromPoint(x, y) // 那个坐标上到底是什么? +document.querySelectorAll('button').length // 你的选择器到底命中了几个? +``` -而真正做过这些流程的人都知道,最花资源的往往还不是最后正式执行,而是前面 -测试。你会改 Prompt、换工具、重新切资料、调整工作流程,同一个东西跑十次、 -二十次很正常。 +第二个抓到过一处很细的问题:门户页面在 DOM 里留了一个**隐藏的** `Confirm` 按钮(来自收起状态的日期面板),于是 `.pop()` 返回的是那个看不见的副本,点击自然落到了无人的地方。按可见性和尺寸一起过滤: -每次都是 API,我自己就会很明显感觉到:又在烧钱。用自己那张卡,当然还是有 -电费跟硬体成本,但我今天让一个 Agent 跑五次还是五十次,不会每跑一次就多 -一笔帐单。这对我来说很重要,因为它会直接影响我敢不敢一直连续测试。很多 -真正稳定的 AI workflow,不是第一次写完就成功,而是让它失败很多次之后, -才慢慢找到什么地方容易出问题。 +```python +"[...document.querySelectorAll('button')].filter(b=>/^Confirm$/.test(b.innerText.trim())" +" && b.offsetParent!==null && b.getBoundingClientRect().width>0).pop()" +``` -本地算力其中一个很大的价值,就是你可以把这些大量试错搬回自己的固定成本 -里。这跟单纯比较「一百万 Token 多少钱」其实是两件事。 +## 第四种「失败」,只有回读才发现 -算一笔很实在的工程账:一个包含工具调用的典型 Agent 循环,跑完一次完整的代码测试或文件重构,通常要经历 30–50 轮迭代。由于每轮都需要把历史上下文和工具执行输出重新带入,平均单轮上下文在 15k–25k tokens 左右——单次 Agent 运行就要消耗 600k–1.2M tokens。如果你在调试一套新的复杂工作流,一天反复测试优化 20 次,就是 1,500 万到 2,400 万 tokens。按照顶级 API(如 Claude 3.5 Sonnet 的 $3/$15 每百万 token)计算,这一天的调试账单就在 $50 到 $150 美元;若是 Claude Opus,直接突破数百美元。 +还有一类「失败」其实不是失败:**写入成功了,而你的读取是旧的。** 我因为列表页没有立刻显示,就断定创建被静默拒绝,又点了一次;后来才发现那张券一直都存在 —— 列表只是延迟了大约十分钟。这个教训可以推广:一次写入之后,唯一可信的结论是在**等待足够时间之后**回读得到的,而「看不到变化」意味着**未知**,不是失败。 -而在我这台配有 12GB 显存的自托管 GPU 节点上(通过 LM Studio / llama.cpp 本地运行量化版 Qwen 2.5),整机满载功耗不过 170W 左右。一天跑 20 次还是跑 100 次,新增的边际成本只有几毛钱电费。这笔账的本质不是简单的 token 单价对比,而是决定了你在工程探索时,敢不敢给 Agent 足够的试错空间。 +## 检查清单 ---- +1. 关掉平滑滚动;用 `behavior:'instant'` 滚动;等一会儿再读矩形。 +2. 点击前调用 `Page.bringToFront`;判断依据用 `document.hidden`,不是 `hasFocus()`。 +3. 自定义控件用 JS 派发 `mousedown`/`mouseup`/`click`,别点坐标。 +4. 先挂事件监听器,证明输入是否到达,再去怪应用。 +5. 数一数选择器命中几个 —— 组件库里隐藏的重复元素很常见。 +6. 写入之后先等待再回读;永远不要重复点击一个还没验证过的提交按钮。 -## 开放模型已经跨过可用的线,而这条线还在移动 +这些都不算冷门知识,但它们中的每一条,在一行 `clicked @591,361` 面前都是隐形的。如果你正在对着一个自己无法控制的后台做自动化:点击是整条链路里最不可靠的一环 —— 也是最容易被最后才怀疑的一环。 -三年前,要在消费级显卡上跑一个真的可以帮你工作的模型,限制很多。现在 -二三十 B 的模型经过量化之后,24GB、32GB 这种显卡已经可以处理不少工作; -再往更大的记忆体走,语言、图片,甚至部分影片模型都开始可以自己部署。 - -真正重要的不是「开源终于打赢闭源了」,而是原本只有闭源模型做得到的工作, -正在一个一个掉进本地可以处理、而且可以被接受的范围。假设三年前十个工作 -只有两个能放本地,现在已经有四五个可以做——这个比例本身就已经改变了 -本地部署的价值。 - -而且我买 GPU 的时候,不是在买 Qwen 这一颗模型。模型可以一直换,也会一直 -出现新的:如果之后出现一个更适合 Coding 的开放模型,我直接换权重;再过 -半年有更好的量化方式,同一张卡可能又可以装进原本跑不动的模型。我真正买 -下来的,是**一块自己可以控制的运算资源**,后面的模型可以一直替换。 - -这就是本地跟闭源服务最不一样的地方。闭源模型能力通常比较强,但你用的是 -某家公司现在愿意提供给你的版本;本地硬体能力可能没那么高,但只要硬体 -撑得住,我可以自己决定今天跑什么模型、换什么版本、要跑多久。 - ---- - -## 固定算力 vs. 每次租算力 - -如果 AI 永远只是拿来问问题,我完全不觉得大家都需要买 GPU。但现在发展方向 -很明显不是这样:Agent 会长时间执行工作,图片跟影片模型会需要大量生成, -公司也会把更多固定流程交给 AI。我们不是单纯使用模型的次数变多,而是模型 -开始变成一个持续工作的运算单位。 - -这时候,「固定拥有一部分算力」跟「每一次执行都向外面租」,就会开始变成 -两种不同的成本结构。使用量少,租一定比较方便;使用量够大、够固定之后, -自己有算力就能更好控制与估算成本。 - -它还多了一个闭源服务给不了的东西:主控权。当机器是我的,我要跑内部资料、 -接自己的工具、长时间执行 Agent,都不需要每次先确认供应商的方案、额度或者 -限制有没有改。尤其公司真的开始把 AI 接进内部流程之后,这件事就不只是省 -Token——有些资料本来就不适合离开自己的环境,有些 Agent 权限也不可能直接 -全部交给外部服务。当模型跑在自己的环境里,资料、工具、权限跟执行方式可以 -一起设计。这才是「控制权」真正指的东西,不只是把权重下载下来而已。 - ---- - -## 我现在实际怎么判断 - -我不会再用「它比 Claude 笨,所以没有价值」这种方式判断一个本地模型。我会 -依序看四件事: - -1. **这颗模型能不能跨过我这个工作的门槛?** -2. **这个任务会执行多少次?** -3. **是不是需要大量试错,或长时间的 Agent?** -4. **这些资料跟系统,到底要不要留在自己的环境?** - -这几个条件一叠起来,本地模型即使不是最强,还是可能比顶级闭源模型更适合 -那一段工作。而且随著开放模型越来越强,这个范围只会一直变大。 - -所以我真正看好的,不是某一颗模型,也不是某一张 GPU,而是这个趋势: -**可以在自己硬体上完成的 AI 工作正在变多。** 以前买一张消费级显示卡, -可能只能拿来跑一些语言模型;现在开始可以接 Agent、图片、影片,后面模型 -跟推理框架继续优化,同样的硬体又可能得到新的用途。 - -硬体当然会折旧,新卡也可能越来越贵,但至少模型不是绑死的。只要这台机器 -还有足够的运算能力,我就可以一直换新的开放模型。 - ---- - -## 一句话版本 - -我现在已经不会只看哪个模型比较聪明。我比较的是:**这件工作到底需要多强的 -模型、它要被执行多少次,以及我愿意把多少控制权留在外部服务手上。** - -需要顶级推理的地方,我照样花钱用最强的模型。但当工作开始变成大量、连续、 -固定,而且本地模型的能力已经跨过可使用门槛时,我就没有理由每一个步骤都 -继续用最贵的闭源模型。 - -浓缩成一句: - -> 用顶级模型决定要做什么,用本地算力把它做上一千次。 - -这就是我买 GPU 的原因。 +我平时就在做这类浏览器自动化与排障 —— 自托管服务、Docker 与 Traefik 栈,偶尔还有被 bot 墙挡住的市场页面。如果你有条工作流总因为「UI 有自己的想法」而断掉,跟我说说:[WhatsApp](https://wa.me/60127972969) · [me@hoelee.com](mailto:me@hoelee.com?subject=Browser%20automation%20debugging) · [hoelee.com](https://hoelee.com)。 ```` ### 🇬🇧 英文版原文 (English Markdown Primary) ````markdown --- -title: "Why I Still Bought a GPU to Run AI When Claude and GPT Are Obviously Stronger" -description: "Claude Opus and GPT are stronger than anything I can run locally. Here is the cost, control, and uptime argument for owning a GPU anyway — with real numbers." -pubDate: 2026-09-21 -category: ai -blueprint: ai-agent-automation -blueprintLayer: L3-local-gpu-inference -tags: [local-llm, gpu, llama-cpp, lm-studio, agents, self-hosting, cost] -ogImage: /og/why-i-still-bought-a-local-gpu.jpg -banner: /banners/why-i-still-bought-a-local-gpu.jpg +title: "Why Your CDP Clicks Silently Fail: Smooth Scroll, Minimised Windows, and JS-Dispatched Events" +description: "Three reasons a Chrome DevTools Protocol click reports success and does nothing: a stale rect from smooth scrolling, a minimised window that drops coordinate input, and a page that received the click but rejected it." +pubDate: 2025-03-19 +updatedDate: 2026-09-29 +category: devops +tags: [cdp, chrome, react, browser-automation, debugging, python] +ogImage: /og/why-your-cdp-clicks-silently-fail.jpg +banner: /banners/why-your-cdp-clicks-silently-fail.jpg draft: false -dynamicCta: true --- -Across the self-hosted production infrastructure I manage — Docker stacks, NAS -storage, tunnels, and automated cron pipelines — I still pay for frontier models. +If you drive a web app with the Chrome DevTools Protocol, you have met this failure: your script reports `clicked @591,361`, the element is there, the click landed — and nothing happens. No exception, no console error, no change on the page. -So when people find out I run local models too, the question is always the same: -*Claude and GPT are stronger. Local models are a pain to set up, slower, and -often just worse. So why did you spend money on a GPU?* +I spent an evening on exactly this while automating a marketplace's seller admin panel, and the three causes I found are worth writing down, because they are all invisible to the code that reports the click. -If you are only comparing raw model capability, I agree with you completely. -When I hit a genuinely new problem, do complex reasoning, plan a product -architecture, or kick off a long agent run, I still reach for Claude or GPT -first. I did not deploy local models because I think they have beaten the -closed frontier. +## Cause 1: a stale rect, because the page scrolls smoothly -I did it because of a different problem that only showed up *after* AI got good -enough to depend on. Everything I built started sitting on a service I do not -control. +The obvious click helper looks like this: ---- +```python +r = self.ev("""(()=>{const e=%s; e.scrollIntoView({block:'center'}); + const b=e.getBoundingClientRect(); + return JSON.stringify({x:b.left+b.width/2, y:b.top+b.height/2});})()""") +self.click_xy(json.loads(r)["x"], json.loads(r)["y"]) +``` -## The problem is not capability — it is dependency +Scroll into view, read the element's rectangle, click its centre. Correct — unless the page has `scroll-behavior: smooth` (very common in modern UI kits). Then `scrollIntoView()` starts an **animation**, and `getBoundingClientRect()` in the same tick returns the position the element is *about to leave*. Your click goes to where the element was, lands on whatever is there, and the log tells you it clicked the coordinates you asked for. -How many tokens you get, what a plan costs, which regions can buy the commercial -service, when the model version changes under you — none of that is the user's -decision. Anthropic has regional restrictions of its own; some markets cannot -buy Claude's commercial service directly. OpenAI's plans, quotas and model -lineup keep shifting. These are commercial services, so of course they change. +The fix is two lines — turn smooth scrolling off, then re-read the rect: -That was fine when ChatGPT was something I opened occasionally to ask a -question. If it was unavailable today, I would ask tomorrow and lose nothing. +```python +self.ev("""(()=>{document.documentElement.style.scrollBehavior='auto'; + if(document.body) document.body.style.scrollBehavior='auto'; return 'ok';})()""") +moved = self.ev("""(()=>{const e=%s; if(!e) return 'no'; + e.scrollIntoView({block:'center', behavior:'instant'}); return 'ok';})()""" % sel) +time.sleep(0.35) # let layout settle +# then read the rect (and retry a couple of times if the element is not there yet) +``` -It stopped being fine once I started using AI to write code, run agents, -analyze documents, handle support, organize company data, and wire models into -workflows that run every day. When AI moves from *chat tool* to *infrastructure*, -the question changes. It is no longer "which model is smarter?" It becomes: +Two details matter. `behavior:'instant'` on `scrollIntoView` overrides the CSS; and the small wait before reading is what makes the rect trustworthy. Without the retry loop the helper still fails intermittently on elements that render a frame late — intermittent failures are the worst kind, because they make you doubt your own diagnosis. -> "If I depend on this model today, and its terms change tomorrow, does my work -> still run?" +## Cause 2: the window is minimised, and coordinate clicks are dropped -That is why I think the stronger the closed models get, the more valuable a -local model becomes. Precisely *because* they are so good, people hand them more -and more work. And the deeper the dependency, the more of your operation someone -else controls. +After the fix, my click helper worked in a one-off probe and then never again inside the real script. Same selector, same coordinates, same page. The difference turned out to be visibility: -There is a second-order version of this that rarely gets said plainly. Every big -lab talks about AI safety, model risk, and restricting certain content, regions -or use cases — while all of them keep racing to make models stronger, because -capability *is* the competitive advantage. Nobody stops, because if you stop, -someone else does not. +``` +document.visibilityState → "hidden" +document.hasFocus() → true +``` -So what actually gets controlled is rarely whether AI keeps developing. It is -who may use it, how, how much, and how far the vendor is willing to go. For an -individual, that reads like a terms-of-service page. For a business, it is -**supplier risk**. If your core process and every agent, tool and data pipeline -around it are pinned to one vendor, you are not just buying tokens — you are -handing over part of the steering wheel. +CDP-synthesised mouse events at coordinates are unreliable when the target tab is not the active tab of a visible window — the browser may accept them, throttle them, or route them nowhere. Note that `hasFocus()` returns `true` even when the document is hidden, so it is not a useful check; `document.hidden`/`visibilityState` is. ---- +Two fixes, and I use both: -## "But local models are worse" — yes, and that is the wrong comparison +```python +def activate(self): # bring the tab to the front of its window + self.cdp("Page.bringToFront") + time.sleep(0.2) -This is the most honest objection, and it is true. Put a model that fits on a -24GB or 32GB consumer card against Claude Opus and it loses on complex tasks. No -argument. +def click_js(self, sel): # click by dispatching events from inside the page + return self.ev("""(()=>{const e=%s; if(!e) return 'no'; + ['mousedown','mouseup','click'].forEach(k=> + e.dispatchEvent(new MouseEvent(k,{bubbles:true,cancelable:true,view:window}))); + return 'js-clicked';})()""" % sel) +``` -But the comparison most people run is wrong. The question is not *can Qwen beat -Claude*. It is: +`Page.bringToFront` fixes the common case. But the JS-dispatched click is the one that never fails: it does not depend on the window being visible, and React's root-level event listener receives it exactly as it receives a real click, because the events bubble to the same root. A date-picker input that opened maybe one time in five with coordinate clicks opens **every** time this way. -> **Is the capability gap large enough to justify using the most expensive model -> for this task, forever?** +The trade-off to know: dispatching events skips hit-testing, so it will also “click” an element a user could not reach (covered, scrolled off, zero-opacity). That is a feature here — and a trap if you use it to paper over a layout bug — so I keep coordinate clicking for anything visual and reach for `click_js` when the widget is a custom control. -Because the trade is not symmetric. A 20–30B model is meaningfully behind the -frontier — and costs one to two orders of magnitude less per token, or runs on -hardware you already own. Different ratio, different decision. +## Cause 3: the page received the click and rejected it -### I treat capability as a threshold, not a ranking +This is the one that wasted the most time, because the symptom is identical to causes 1 and 2: click fires, nothing changes. -If a task is below some capability line, it cannot be done at all — use the -strong model. But once a model crosses that line, the things that decide how I -actually work stop being benchmark scores and become cost, speed, whether it can -run for hours, and whether I control the system. +Distinguishing them takes one injected listener: -Concretely, here is how it splits for me: +```js +window.__ev = []; +['mousedown','mouseup','click'].forEach(k => + document.addEventListener(k, e => window.__ev.push(k + ':' + e.target.tagName), true)); +``` -| Task | What I reach for | -|---|---| -| Vague problem, no clear direction yet | Frontier model — a wrong early call invalidates everything downstream | -| Product architecture, data model, acceptance criteria | Frontier model | -| Scaffolding a project to a fixed spec, editing files, running tests, fixing failures | Local model | -| Long-running agents, retries, batch jobs, bulk content | Local model | +- Events recorded → the input path works. The problem is in the app (cause 3), or in how you read the result (see below). +- Nothing recorded → causes 1 or 2. -The important shift is the second block. Once direction is settled and the agent -is *executing a spec*, a single task may call a model 20–50 times. Do all fifty -calls need the most expensive model on the market? Usually not. The agent is not -deciding what the product should be — it is following specifications that are -already written. If a local model understands those specs and knows how to use -the tools, it has crossed the working threshold. +With the listener armed I clicked `Confirm` and the buffer filled up — so the click was fine, and the app was saying no. The reason was in text I had not thought to look at: ---- +> “Please create a new shop welcome voucher after the existing one is expired.” -## Agents turn a small price difference into a large one +The form refuses a second voucher of that type. No field-level red text, no toast in the first 500 ms, the URL unchanged. **The rejection was real and the click was innocent.** -With chat, you ask once and get one answer. If Claude costs many times more than -a local model, you barely notice. +Two more tools for this class of bug: -Agents are not one call. A task can invoke the model twenty times, fifty times; -it fails and reruns. The per-token price gap gets multiplied by every call. +```js +document.elementFromPoint(x, y) // what is actually on top at that coordinate? +document.querySelectorAll('button').length // how many match your selector? +``` -And anyone who has actually built these pipelines knows the expensive part is -not the final run — it is the testing before it. You rewrite the prompt, swap -tooling, re-chunk the data, adjust the workflow, and run the same thing ten or -twenty times. That is normal. +The second one caught a subtle one: the portal leaves a **hidden** `Confirm` button in the DOM from the collapsed date-panel, so `.pop()` returned the invisible duplicate and the click went nowhere visible. Filter by visibility *and* size: -On metered API calls, I feel it: *I am burning money right now.* On my own card, -there is still electricity and hardware cost, but running an agent five times or -fifty times does not add a line to an invoice. That difference changes my -behavior — it decides whether I dare keep testing. Most stable AI workflows are -not written correctly the first time. They get there after failing enough times -that you find out where they break. +```python +"[...document.querySelectorAll('button')].filter(b=>/^Confirm$/.test(b.innerText.trim())" +" && b.offsetParent!==null && b.getBoundingClientRect().width>0).pop()" +``` -The value of local compute is that you move that trial-and-error into a fixed -cost. That is a different question from "how much does a million tokens cost." +## And the fourth failure I only found by re-reading -Here is the math on a real engineering workload: a typical agent loop with tool-use iterating through tests and refactoring takes 30–50 turns. Because each turn carries previous tool outputs and conversation history, an average turn easily loads 15k–25k tokens of context — consuming 600k to 1.2M tokens in a single agent run. If you are tuning a new autonomous workflow and test it 20 times a day, you burn 15M to 24M tokens. On frontier APIs like Claude 3.5 Sonnet ($3/$15 per million tokens), that single day of debugging costs $50–$150; on Opus, it crosses several hundred dollars. +There is a class of “failure” that is not one: **the write succeeded and your read is stale.** I concluded a create had been silently refused because the list page did not show it immediately, clicked again, and later discovered the voucher had existed the whole time — the list just lagged about ten minutes. The lesson generalises: after a write, the only trustworthy verdict is a read-back *after* a real waiting period, and “no visible change” means *unknown*, not *failed*. -On my own self-hosted GPU node (a dedicated 12 GB VRAM card running quantized Qwen 2.5 via LM Studio / llama.cpp), the entire rig pulls under 200W at full load. Running 20 iterations or 100 iterations adds cents in electricity and $0 in marginal API bills. The real math is not just "cost per million tokens" — it is whether you can afford to let an agent fail enough times to become reliable. +## The checklist ---- +1. Turn off smooth scrolling; scroll with `behavior:'instant'`; wait, then read the rect. +2. Call `Page.bringToFront` before clicking; treat `document.hidden` as the signal, not `hasFocus()`. +3. For custom widgets, dispatch `mousedown`/`mouseup`/`click` from JS rather than clicking coordinates. +4. Arm an event listener to prove whether input arrived before you blame the app. +5. Count your selectors — hidden duplicates are common in component libraries. +6. After a write, wait and re-read; never re-click a submit you have not verified. -## Open models crossed the useful line, and the line keeps moving +None of this is exotic, but every item is invisible in a log line that says `clicked @591,361`. If you are building automation against an admin panel you do not control, the click is the least reliable part of the pipeline — and the one most likely to be blamed last. -Three years ago, running a model on a consumer card that could genuinely do work -for you was a research project. Today, 20–30B models — quantized — handle real -work on 24GB and 32GB cards. Go up in VRAM and language, image, and now portions -of video generation become self-hostable too. - -The important part is not "open source finally won." It is that work which used -to be closed-model-only keeps falling inside the range that local hardware can -do acceptably. If three years ago two of ten jobs could run locally, and today -four or five can, that ratio alone has changed the value of owning the hardware. - -And when I buy a GPU, I am not buying a specific model. Models change. If a -better coding model ships next month, I swap weights. If a better quantization -scheme lands in six months, the same card suddenly fits a model it could not run -before. What I actually bought is **compute I control**; the models on top of it -are disposable. - -That is the structural difference. A closed model is stronger — and also exactly -the version one company currently chooses to offer you, at terms it currently -chooses. Local hardware is weaker — but as long as the card holds up, I decide -which model runs today, which version I switch to, and how long it runs. - ---- - -## Fixed compute vs. renting compute - -If AI stays a question-and-answer tool, I do not think most people need a GPU at -all. But that is clearly not the direction. Agents run for long stretches. Image -and video models want bulk generation. Companies hand more fixed processes to -AI. We are not just *using models more often* — models are becoming a continuously -working compute unit. - -At that point, owning part of the capacity and renting every execution become -**two different cost structures**. Low usage: renting is obviously more -convenient. High, steady usage: owning lets you control and predict cost. - -It also buys something no API can sell you — control. When the machine is mine, -I can point it at internal data, connect my own tools, and run agents for hours -without checking whether a vendor's plan, quota or limits changed overnight. -Once AI is inside real company processes, this stops being about saving tokens. -Some data should not leave the environment it lives in. Some agent permissions -should not be handed wholesale to an external service. If the model runs on my -own hardware, data, tools, permissions and execution can be designed together. -*That* is what I mean by control — not merely having the weights on disk. - ---- - -## What I actually evaluate now - -I no longer judge a local model by "it is dumber than Claude, therefore -worthless." Four questions, in order: - -1. **Does this model clear my working threshold** for this specific job? -2. **How many times will this task execute?** -3. **Does it need heavy trial-and-error, or long agent runs?** -4. **Does the data or system need to stay in my own environment?** - -When those stack up, a local model — even one that is not the strongest — can be -the better tool for that slice of work. And as open models improve, that slice -keeps getting bigger. - -So what I am bullish on is not one model or one GPU. It is the trend: **more AI -work becomes doable on hardware you own.** A consumer card that used to run only -language models now runs agents, image generation, video. Better models and -better inference frameworks keep giving the same metal new jobs. - -Hardware depreciates, new cards get more expensive — that is real. But the model -is not locked in. As long as the machine has enough compute, I can keep swapping -in newer open models. - ---- - -## The short version - -I no longer compare "which model is smarter." I compare **how much capability -this job needs, how many times it will run, and how much control I am willing to -leave in someone else's hands.** - -Where I need frontier reasoning, I pay for the frontier. But when work becomes -high-volume, continuous and predictable — and a local model has crossed the -usable threshold — there is no reason to keep paying frontier prices for every -single step. - -In one line: - -> Use the frontier model to decide what to build. Use local compute to build it -> a thousand times. - -That is why I bought the GPU. +I build and debug this kind of browser automation for a living — self-hosted services, Docker and Traefik stacks, and the odd scraped-but-bot-walled marketplace. If you have a workflow that keeps breaking because a UI has other ideas, tell me about it: [WhatsApp](https://wa.me/60127972969) · [me@hoelee.com](mailto:me@hoelee.com?subject=Browser%20automation%20debugging) · [hoelee.com](https://hoelee.com). ```` --- -## 文章 02 / 08:为什么我的 AI 歌曲总是丢结尾:在 12GB 显卡上跑 YuE2 -- **Slug**: `yue2-on-a-12gb-gpu-songs-that-lost-their-endings` -- **英文标题**: Why My AI Songs Kept Losing Their Endings: Running YuE2 on a 12 GB GPU -- **分类**: `ai` | **发布日期**: 2026-10-20 -- **线上地址**: https://blog.hoelee.com/posts/yue2-on-a-12gb-gpu-songs-that-lost-their-endings/ (EN) | https://blog.hoelee.com/posts/zh/yue2-on-a-12gb-gpu-songs-that-lost-their-endings/ (ZH) +## 文章 02 / 11:Prometheus 明明有数据,Grafana 面板却全是 No data 的原因 +- **Slug**: `why-your-grafana-dashboard-shows-no-data` +- **英文标题**: Why Your Grafana Dashboard Shows No Data (When Prometheus Is Fine) +- **分类**: `devops` | **发布日期**: 2026-05-17 +- **线上地址**: https://blog.hoelee.com/posts/why-your-grafana-dashboard-shows-no-data/ (EN) | https://blog.hoelee.com/posts/zh/why-your-grafana-dashboard-shows-no-data/ (ZH) - **本篇审核状态**: ⏳ 待审核 (Pending) ### ✍️ 顾问审阅意见区 (Review Feedback) @@ -526,2406 +340,290 @@ That is why I bought the GPU. ### 🇨🇳 中文版原文 (Chinese Markdown Twin) ````markdown --- -title: "为什么我的 AI 歌曲总是丢结尾:在 12GB 显卡上跑 YuE2" -description: "12 首中文歌时长全都恰好 240.000 秒,尾奏不见了。原因不在歌词 —— 是一道硬上限、一堵显存天花板,和一个不看 BPM 的规划器。" -pubDate: 2026-10-20 -category: ai -blueprint: ai-agent-automation -blueprintLayer: L4-aigc-media-pipelines -stream: creative-lab -tags: ["yue2", "comfyui", "ai-music", "local-ai", "gpu", "vram", "mandarin"] -ogImage: "/og/zh-yue2-on-a-12gb-gpu-songs-that-lost-their-endings.jpg" -banner: "/banners/yue2-on-a-12gb-gpu-songs-that-lost-their-endings.jpg" +title: "Prometheus 明明有数据,Grafana 面板却全是 No data 的原因" +description: "所有 target 都是 up、数据就在 Prometheus 里、面板表达式也没错——41 个面板却全部 No data。原因是 Grafana 变量过滤了自己,而我的验证方式恰好掩盖了它。" +pubDate: 2026-05-17 +updatedDate: 2026-09-29 +category: devops +blueprint: self-hosted-infrastructure +blueprintLayer: L3-observability-self-healing +tags: [grafana, prometheus, dashboards, monitoring, promql, observability] +ogImage: /og/zh-why-your-grafana-dashboard-shows-no-data.jpg +banner: /banners/why-your-grafana-dashboard-shows-no-data.jpg draft: false -dynamicCta: true --- -YuE2 是一个 30 亿参数的开源音乐模型,能按风格提示词和歌词唱中文。我把它放在自己那台 12GB 显存的 RTX 3060 上用 ComfyUI 本地跑,一次排了 12 首原创歌 —— 结果第一批出来,**每一首的时长都恰好是 240.000 秒**。这个数字不是巧合,而且是个坏消息。 -## 每首都恰好 240.000 秒 +打开每天在看的面板,结果是空的。不是\"某一格坏了\",而是**从上到下每一格**都写着 *No data*。去查 Prometheus,一切健康。这篇讲我踩到的那个具体且不直观的原因,以及为什么我自己的验证会同时告诉我\"面板没问题\"。 -12 个文件,时长精确到毫秒完全一致: +## 最省时间的排查顺序 -``` -01_A1_..._糖画小仙.mp3 duration=240.000000 -02_A2_..._溪边捉云记.mp3 duration=240.000000 -03_A3_..._寒露拾桂子.mp3 duration=240.000000 -... -``` - -我设的是 `max_duration=240`,因为参考指南说 12 段结构大约会落在三分半左右。结果它精确落在 240 秒 —— 这意味着模型从来没有"决定结束",它是把时间用光了。 - -## 整数是一种自白 - -YuE2 用 ABC 规划器来编排:读取歌词,给每一段分配小节数,生成一份符号化的曲谱草稿。音乐生成器照着这份草稿唱,**草稿结束它就结束**。 - -所以一首真正唱完的歌,输出时长一定是**小于**你设的上限的 —— 它自己停下来了。 - -判断方法只要一条命令: +常见原因(数据源选错、时间范围不对、`rate()` 用在会重置的计数器上、target 没被抓取)都不适用。真正有效的检查顺序是: ```bash -d=$(ffprobe -v error -show_entries format=duration -of csv=p=0 take.mp3) -python -c "print(abs(float('$d') - 240.0) <= 0.005)" # True = 被硬切 +# 1. target 真被抓取了吗? +curl -s http://prometheus:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job, health}' + +# 2. 指标现在存在吗?(完全绕过 Grafana) +curl -s -G http://prometheus:9090/api/v1/query \ + --data-urlencode 'query=node_uname_info{job="unraid-host"}' | jq '.data.result[0].metric' + +# 3. 面板表达式本身对吗? +curl -s -G http://prometheus:9090/api/v1/query \ + --data-urlencode 'query=count(node_cpu_seconds_total{mode="idle",job="unraid-host"})' ``` -如果时长和 `max_duration` 相差不到几毫秒,这次渲染就是被截断的。歌词需要的时间超过了上限允许的范围,切点那一句被拦腰砍断。 +三项都过了:7 个 target `up`、2.5 万条序列在采集、手动跑面板表达式也有数据。所以问题不在指标、不在抓取、也不在 PromQL——而在**它们之间的变量层**。 -## 确认尾奏真的丢了 +## 真正的原因:变量过滤了自己 -光看时长只能知道"被切了",不知道"丢了什么"。所以我把每首的尾巴切出来,用本地语音模型(SenseVoice,跑在 CPU 上,GPU 渲染时不用停)转写: +面板上有一个主机选择器,两个变量串联:`$nodename`(哪台机器)和 `$node`(它的 `instance` 标签)。第一个的定义是: + +```promql +# BROKEN — 变量用自身的值来过滤自己 +label_values(node_uname_info{job="unraid-host", nodename=~"$nodename"}, nodename) +``` + +首次加载时 `$nodename` 还没有值,选择器就变成 `nodename=~""`——**什么都匹配不到**。查询返回零行的变量没有任何选项,于是它保持空;而 `$node` 又是基于 `$nodename` 定义的: + +```promql +label_values(node_uname_info{job="unraid-host", nodename="$nodename"}, instance) +``` + +于是 `$node` 也是空,所有以 `$node` 为过滤条件的面板自然匹配不到序列。Grafana 说 *No data* 并没有骗人:面板查询真的没有数据,因为给它限定范围的变量解析成了空。 + +修法是去掉自引用,并给每个变量存一个明确的默认值: + +```promql +# FIXED — 不自引用,每个变量只跳一层 +# $nodename +label_values(node_uname_info{job="unraid-host"}, nodename) +# $node +label_values(node_uname_info{job="unraid-host", nodename="$nodename"}, instance) +``` + +## 为什么我的验证没抓到 + +这才是值得抄的部分。我\"验证\"面板的方式是:**手动**把每个面板表达式里的变量替换成我知道正确的值(`$nodename=unRaid`、`$node=host.docker.internal:9100`),然后断言查询有数据点。13 项检查全部通过——面板依然是空的,因为 bug 在被替换的那个值**上游**:Grafana 自己对变量的解析是空的。 + +**一个替换掉\"可疑值\"的验证,永远发现不了那个值解析失败。** 要抓它,得读 Grafana 真实存下来的值: ```bash -ffmpeg -v error -y -i take.mp3 -ss 232 -t 20 -ar 16000 -ac 1 -c:a pcm_s16le t.wav -curl -s -X POST http://127.0.0.1:20129/v1/audio/transcriptions \ - -F "file=@t.wav" -F "language=zh" +curl -s -u admin:"$PW" http://grafana:4010/api/dashboards/uid/rYdddlPWk \ + | jq '.dashboard.templating.list[] | {name, current: .current.value}' ``` -(对于连末遍副歌都没唱到的曲目,我将切片窗口向前扩展到了最后 30 秒即 `-ss 210`,以准确定位截断发生时的唱段。)结果没有歧义: - -| 歌 | 唱到哪 | 缺了什么 | -|---|---|---| -| 糖画小仙 | 末遍副歌最后一句 | **整段 `[Outro]`** | -| 溪边捉云记 | 215 秒还停在 `[Bridge]` | Solo + 末遍副歌 + Outro | -| 寒露拾桂子 | 215 秒还停在 `[Bridge]` | 末遍副歌 + Outro | -| 纸鸢与旧木偶 | 末遍副歌第 5 句 | 副歌后半 + Outro | - -所以不是混音问题、也不是淡出 —— 是结构不完整。 - -## 试过但没用的三件事,和半有用的一件 - -### 1. 提高 BPM —— 完全无效 - -直觉上:节奏更快、每秒塞进更多字、同样的窗口能装更多歌词。我在整批上验过:117 BPM 的国风组和 140 BPM 的 DJ 燃曲组,**吞吐几乎一样**。 - -YuE2 的时长来自 ABC 规划器分配的小节数,不看你风格提示词里的 BPM 标签。BPM 只影响"感觉",买不到时间。别在它身上调时长。 - -### 2. 同 seed 重跑 —— 静悄悄地什么都没做 - -ComfyUI 是用 `--cache-classic` 启动的,所以重新提交一个逐字节相同的任务(同样的风格、同样的歌词、同样的 seed)会直接命中节点缓存。任务"完成"于 10 秒左右,还你一份一模一样的截断音频,文件名加个 `_00002`。 - -看起来像一次又快又成功的重试。实际上是命中了一个已知有问题 take 的缓存。 - -### 3. 换 seed —— 四次里能救三次 - -规划器分配小节的方式依赖 seed,所以换个 seed 确实可能给出更短、能唱完的版本。这一招救回了我四首里的三首: - -| 歌 | 原始 | 新 seed | 结果 | -|---|---|---|---| -| 溪边捉云记 | 卡在 281 秒 | 20002 | 279.7 秒,完整 | -| 外婆的绿搪瓷碗 | 卡在 300 秒 | 20007 | 251.9 秒,完整 | -| 北溟有小鱼 | 卡在 300 秒 | 20005、30005 | 再次撞顶 | - -## 预算是"每秒几个字",而这个数还在变 - -我试过把它做成算术:数字数、除以吞吐、得出时长。实测不配合。 - -| 歌 | 汉字数 | 上限 | 实际 | 装下了吗 | -|---|---|---|---|---| -| 糖画小仙 | 556 | 272 秒 | 258.7 秒 | ✅ | -| 纸鸢与旧木偶 | 572 | 300 秒 | 279.9 秒 | ✅ | -| 溪边捉云记 | 578 | 281 秒 | **281.0 秒** | ❌ 被切 | -| 小楼檐下听雨眠 | 580 | 300 秒 | 256.3 秒 | ✅ | -| 画卷小锦鲤 | 634 | 300 秒 | 285.9 秒 | ✅ | -| 三楼窗台的蒲公英 | 693 | 329 秒 | 247.4 秒 | ✅ | -| 弄堂口的竹蜻蜓 | 692 | 300 秒 | 250.0 秒 | ✅ | - -把最后两行和第三行对比着看:**693 个字在 247 秒内唱完了,578 个字却撑爆了 281 秒。** 字数换算时长的公式在这里根本不存在 —— 实测吞吐落在每秒 2.0 到 2.8 个汉字之间,取决于规划器怎么切段落。 - -所以字数只适合用来**预警预算**,不适合用来掐时间。 - -## 真正卡住的瓶颈是显存 - -一旦接受"给它足够空间、让它自己收尾",上限自然想设大一点。但在 12GB 卡上设不大。 - -渲染过程中每 5 秒采一次 `nvidia-smi` 得到的峰值显存: - -| `max_duration` | 峰值显存 | -|---|---| -| 240 秒 | 10.3 GB | -| 272 秒 | 11.65 GB | -| 278 秒 | 11.07 GB | -| 281 秒 | 12.06 GB | -| 300 秒 | 12.03 GB | - -显存总量为 12,288 MiB(12 GB)。**300 秒就是实际天花板** —— 再往上,你离"渲染到第六分钟时因为一个浏览器标签页而 OOM 崩掉"只差一步。 - -## 解法 - -三处改动,都很便宜: - -1. **把上限设到 300 秒,让规划器自己收尾。** 不要追目标时长;给足余量,它会在音乐结束的地方结束。 -2. **被切的 take 归档,不要混进成品。** 时长等于上限的 take 移到 `未自然结尾\` 目录。成品目录里就只有完整的歌,而且重跑永远不会覆盖掉一个可用的 take。 -3. **换 seed 最多试三次。** 换 seed 有用,但如果一首歌的歌词真的超过显存能撑的时长,它会永远失败。三次不成就动歌词 —— 别烧一小时 GPU 假装下一个 seed 会中。 - -## 那首始终装不下的歌 - -12 首里有一首没做成。**北溟有小鱼,593 字,失败了五次:** - -| 尝试 | 参数 | 结果 | -|---|---|---| -| 1 | seed 10005,上限 287 秒 | 撞 287 秒 | -| 2 | seed 10005,上限 300 秒 | 撞 300 秒 | -| 3 | seed 20005,上限 300 秒 | 撞 300 秒 | -| 4 | seed 30005,上限 300 秒 | 撞 300 秒 | -| 5 | seed 40005,删掉两段纯器乐 | 撞 300 秒 | - -我甚至把 `[Interlude]` 和 `[Instrumental Solo]` 两个标记删掉 —— 从 12 段降到 10 段,一个唱词都没动 —— 它还是溢出。它的规划器要的时间超过了这张卡能给的上限。诚实的解法是砍掉大约 40 个字再重跑;我宁可告诉你这件事,也不想交一首结尾被咬掉一块的第十二首歌。 - -## 在这之前踩到的一个坑 - -值得单独警告:**YuE2 会把你歌词里括号内的中文制作备注唱出来。** - -如果你贴的歌词长这样(大多数歌词稿就是这么写的): - -``` -[Intro] -(重鼓点起,急促琵琶轮指,唢呐破空,合成器铺底) -嘿!哈!山海震! -``` - -那么那行括号内容就成了一句歌词。模型不知道那是舞台说明。提交前先清掉: - -```python -clean = "\n".join(l for l in raw.split("\n") - if not re.match(r"^\s*[((].*[))]\s*$", l)) -``` - -配器和动态写进风格提示词。歌词框里只留段落标签和要唱的字。 - -## 我会怎么改 - -- **预警做在歌词阶段,不在渲染之后。** 先数字数。12GB 卡上超过约 600 字就要有心理准备,超过 550 字就该准备删点什么。 -- **永远不要把 `max_duration` 设成目标时长。** 设得比估算宽松,让它自己收尾。上限是安全护栏,不是旋钮。 -- **把"时长 == 上限"当成流水线里的硬失败**,而不是三天后觉得副歌怪怪的才发现。 -- **先换 seed,再动歌词** —— 免费,而且大多数时候有效。然后三次为止。 - -## 十一首歌 - -下面这些是自然收尾的成品,女童声、中文、跑在一张 12GB 显卡上,成本比一年的云端音乐额度还低。每个播放器都带同步歌词 —— 因为模型不输出时间戳,时间轴是用转写结果和歌词原文对齐反推出来的。 - - - -### 1. 糖画小仙 — Sugar-Painting Fairy - -*Guofeng electronic* · 556 characters · 259 s - -
- - -
- -### 2. 溪边捉云记 — Catching Clouds by the Stream - -*Guofeng electronic* · 578 characters · 280 s - -
- - -
- -### 3. 寒露拾桂子 — Picking Osmanthus at Cold Dew - -*Guofeng electronic* · 578 characters · 290 s - -
- - -
- -### 4. 纸鸢与旧木偶 — The Kite and the Old Puppet - -*Guofeng ballad* · 572 characters · 280 s - -
- - -
- -### 5. 小楼檐下听雨眠 — Listening to Rain Under the Eaves - -*Guofeng ballad* · 580 characters · 256 s - -
- - -
- -### 6. 外婆的绿搪瓷碗 — Grandma's Green Enamel Bowl - -*Urban folk* · 679 characters · 252 s - -
- - -
- -### 7. 弄堂口的竹蜻蜓 — The Bamboo Dragonfly at the Alley Mouth - -*Urban folk* · 692 characters · 250 s - -
- - -
- -### 8. 三楼窗台的蒲公英 — Dandelion on the Third-Floor Windowsill - -*Urban folk* · 693 characters · 247 s - -
- - -
- -### 9. 小哪吒借红绫 — Little Nezha Borrows the Red Sash - -*Guofeng DJ remix* · 612 characters · 271 s - -
- - -
- -### 10. 画卷小锦鲤 — The Koi That Left the Painting - -*Guofeng DJ remix* · 634 characters · 286 s - -
- - -
- -### 11. 山海小当家 — Little Boss of the Classic of Mountains and Seas - -*Guofeng DJ remix* · 647 characters · 237 s - -
- - -
- - -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Why My AI Songs Kept Losing Their Endings: Running YuE2 on a 12 GB GPU" -description: "Twelve Mandarin songs came out exactly 240.000 seconds long with the outros missing. The cause wasn't the lyrics — it was a hard cap, a VRAM ceiling, and a planner that ignores BPM." -pubDate: 2026-10-20 -category: ai -blueprint: ai-agent-automation -blueprintLayer: L4-aigc-media-pipelines -stream: creative-lab -tags: ["yue2", "comfyui", "ai-music", "local-ai", "gpu", "vram", "mandarin"] -ogImage: "/og/yue2-on-a-12gb-gpu-songs-that-lost-their-endings.jpg" -banner: "/banners/yue2-on-a-12gb-gpu-songs-that-lost-their-endings.jpg" -draft: false -dynamicCta: true ---- -YuE2 is a 3B open-weights music model that sings Mandarin from a style prompt -and a lyric sheet. I ran it locally on a 12 GB RTX 3060 through ComfyUI — -twelve original songs in one serial queue — and the first batch came back with -**every single file exactly 240.000 seconds long**. That number is not a -coincidence, and it is not good news. - -## Every track was exactly 240.000 seconds - -Twelve files, twelve identical durations to the millisecond: - -``` -01_A1_..._糖画小仙.mp3 duration=240.000000 -02_A2_..._溪边捉云记.mp3 duration=240.000000 -03_A3_..._寒露拾桂子.mp3 duration=240.000000 -... -``` - -I had asked for `max_duration=240` because the style guide I was working from -said a 12-section structure would land around three and a half minutes. It -landed on 240 seconds exactly — which means the model never decided to stop. -It ran out of room. - -## A round number is a confession - -YuE2 plans a song with an ABC planner: it reads the lyric sheet, decides how -many bars each section gets, and emits a symbolic sketch. The music generator -then sings that sketch and **ends when the sketch ends**. A finished song -therefore comes out *shorter* than the cap you set — it stops on its own. - -Here is the tell, and it costs one shell command: +更直接的检查对象是变量查询本身——跑 Grafana 会跑的那条: ```bash -d=$(ffprobe -v error -show_entries format=duration -of csv=p=0 take.mp3) -python -c "print(abs(float('$d') - 240.0) <= 0.005)" # True = hard cut +# $nodename 的 label_values() 到底返回什么? +curl -s -G http://prometheus:9090/api/v1/query \ + --data-urlencode 'query=count by (nodename) (node_uname_info{job="unraid-host"})' | jq '.data.result' ``` -If the duration equals `max_duration` to within a few milliseconds, the render -was truncated. The lyric sheet wanted more time than the cap allowed, and the -last thing being sung at the cutoff point got sliced off mid-phrase. +那里是空,面板就不可能渲染出来,无论数据多健康。 -## Confirming the outros were actually missing +## 第二个坑:`$__all` 不是 `.*` -Duration alone tells you *that* it was cut, not *what* was lost. So I pulled -the tail of each take and transcribed it with a local speech model (SenseVoice, -CPU — it stays running while the GPU renders): +做自动化面板检查时,值为\"All\"的变量,其 current 是哨兵字符串 `$__all`——不是正则。把 `name=~"$__all"` 原样展开会匹配不到任何东西,于是完全健康的容器面板\"全军覆没\"。展开前要把 `$__all` 解析成该变量的 `allValue`(通常是 `.*`),否则你追的是一个只存在于自己检查脚本里的 bug。 -```bash -ffmpeg -v error -y -i take.mp3 -ss 232 -t 20 -ar 16000 -ac 1 -c:a pcm_s16le t.wav -curl -s -X POST http://127.0.0.1:20129/v1/audio/transcriptions \ - -F "file=@t.wav" -F "language=zh" -``` +## 不是每个空面板都是 bug -(For tracks that had not even reached the final chorus, I widened the slice window back to the final 30 seconds with `-ss 210` to locate the active section when truncation hit.) The results were unambiguous: +修完之后 25 个面板有 21 个出数。剩下 4 个是**设计上就不可能**有数据的,最好在面板说明里写清楚: -| Song | Sung up to | What was missing | -|---|---|---| -| 糖画小仙 | final chorus, last line | the entire `[Outro]` | -| 溪边捉云记 | `[Bridge]` at 215 s | solo + final chorus + outro | -| 寒露拾桂子 | `[Bridge]` at 215 s | final chorus + outro | -| 纸鸢与旧木偶 | final chorus, line 5 | rest of chorus + outro | +- **PSI 面板**需要 `/proc/pressure`,而这台 NAS 的内核不暴露它——指标在这台机器上无法存在;同样的面板在会导出 `node_pressure_*` 的主机上正常显示。 +- **根文件系统面板**的表达式带 `fstype!="rootfs"`,而某台主机的 `/` 是**内存盘**(unRaid 就是),所以永远匹配不到。 -So: not a mixing artefact, not a fade-out. The songs were structurally -incomplete. +区分\"因为 bug 空\"和\"因为这台机器不可能有这个指标而空\",是真修好和瞎忙一下午的分界线。 -## Three fixes that did nothing, and one that half-worked +## 我下次会怎么做 -### 1. Raising the BPM — no effect - -The obvious reasoning: faster tempo, more syllables per second, more lyrics -fit in the same window. I tested this across the batch: the 117 BPM guofeng -tracks and the 140 BPM DJ remixes had **effectively identical throughput**. - -YuE2's duration comes from the ABC planner's bar count, not from the BPM token -in your style prompt. The BPM tag shapes the *feel*; it does not buy you room. -Stop tuning it for duration. - -### 2. Re-running with the same seed — silently does nothing - -ComfyUI was started with `--cache-classic`, so resubmitting a byte-identical -prompt — same style, same lyrics, same seed — hits the node cache. The job -"completes" in about ten seconds and hands you back the same truncated audio -with a `_00002` filename. - -It looks like a fast, successful retry. It is a cache hit on a known-bad take. - -### 3. Rolling new seeds — works about 3 times out of 4 - -The planner's bar allocation depends on the seed, so a different seed really can -produce a shorter, complete song from the same lyrics. This rescued three of my -four clipped tracks: - -| Song | Original | New seed | Result | -|---|---|---|---| -| 溪边捉云记 | clipped at 281 s | 20002 | 279.7 s, complete | -| 外婆的绿搪瓷碗 | clipped at 300 s | 20007 | 251.9 s, complete | -| 北溟有小鱼 | clipped at 300 s | 20005, 30005 | clipped again | - -## The budget is characters per second — and it isn't constant - -I tried to turn this into arithmetic: count the Chinese characters, divide by a -throughput figure, get the duration. The measurements refused to cooperate. - -| Song | Characters | Cap | Actual | Fits? | -|---|---|---|---|---| -| 糖画小仙 | 556 | 272 s | 258.7 s | ✅ | -| 纸鸢与旧木偶 | 572 | 300 s | 279.9 s | ✅ | -| 溪边捉云记 | 578 | 281 s | **281.0 s** | ❌ cut | -| 小楼檐下听雨眠 | 580 | 300 s | 256.3 s | ✅ | -| 画卷小锦鲤 | 634 | 300 s | 285.9 s | ✅ | -| 三楼窗台的蒲公英 | 693 | 329 s | 247.4 s | ✅ | -| 弄堂口的竹蜻蜓 | 692 | 300 s | 250.0 s | ✅ | - -Read the last two rows against the third. **693 characters finished in 247 -seconds, while 578 characters overflowed 281 seconds.** A lyrics-to-duration -formula simply does not exist here — the observed throughput ranges from about -2.0 to 2.8 Chinese characters per second, depending on how the planner chops the -sections. - -So characters are useful for a *budget warning*, not for setting a timer. - -## The limit that actually binds: VRAM - -Once you accept "give it enough room and let it close naturally", the cap wants -to be large. On a 12 GB card it cannot be. - -Measured peak VRAM while rendering, sampled every five seconds with -`nvidia-smi` during the run: - -| `max_duration` | Peak VRAM | -|---|---| -| 240 s | 10.3 GB | -| 272 s | 11.65 GB | -| 278 s | 11.07 GB | -| 281 s | 12.06 GB | -| 300 s | 12.03 GB | - -The card has 12,288 MiB (12 GB) of VRAM. **300 seconds is the practical ceiling** — past that -you are one desktop browser tab away from an out-of-memory abort in the middle -of a six-minute render. - -## The fix - -Three changes, all cheap: - -1. **Set the cap to 300 s and let the planner close the song itself.** Do not - chase a target length; give it headroom and it ends where the music ends. -2. **Archive clipped takes instead of mixing them in.** A take whose duration - equals the cap goes to `未自然结尾\` ("did not end naturally"). The output - folder then contains only complete songs, and re-renders never destroy a - usable take. -3. **Cap the seed roulette at three attempts.** Different seeds help, but a song - whose lyrics genuinely need more than the VRAM ceiling allows will fail - forever. Three strikes and you trim the lyric sheet — don't burn an hour of - GPU pretending the next seed will be the one. - -## The song that never fit - -One of the twelve never made it. **北溟有小鱼 — 593 characters — failed five -times:** - -| Attempt | Parameters | Result | -|---|---|---| -| 1 | seed 10005, cap 287 s | hit 287 s | -| 2 | seed 10005, cap 300 s | hit 300 s | -| 3 | seed 20005, cap 300 s | hit 300 s | -| 4 | seed 30005, cap 300 s | hit 300 s | -| 5 | seed 40005, both instrumental sections removed | hit 300 s | - -I even deleted the `[Interlude]` and `[Instrumental Solo]` markers — 12 sections -down to 10, not one sung word changed — and it still overflowed. Its planner -wants more time than a 12 GB card can give. The honest fix is to cut about 40 -characters of lyric and re-render, and I would rather tell you that than ship a -twelfth track with a bitten-off ending. - -## A trap I hit before any of this - -Worth its own warning: **YuE2 will sing the Chinese production notes in your -lyric sheet.** - -If you paste lyrics that look like this — which is how most lyric sheets are -written — - -``` -[Intro] -(重鼓点起,急促琵琶轮指,唢呐破空,合成器铺底) -嘿!哈!山海震! -``` - -…then that parenthetical line becomes a lyric. The model does not know it is a -stage direction. Strip them before submitting: - -```python -clean = "\n".join(l for l in raw.split("\n") - if not re.match(r"^\s*[((].*[))]\s*$", l)) -``` - -Arrangement and dynamics belong in the style prompt. The lyric box gets section -tags and sung words, nothing else. - -## What I'd do differently - -- **Pre-flight the lyric sheet, not the render.** Count characters first. Over - roughly 600 characters on a 12 GB card, expect trouble; over 550, expect to - trim something. -- **Never set `max_duration` to a target length.** Set it generously above the - estimate and let the planner finish. The cap is a safety rail, not a dial. -- **Treat "duration == cap" as a hard failure in the pipeline**, not something - you notice three days later when a chorus sounds wrong. -- **Re-run with a new seed before touching the lyrics** — it is free and works - most of the time. Then stop at three. - -## Eleven tracks - -These are the ones that closed naturally, sung by a child voice, in Mandarin, -on a 12 GB card that cost less than a year of cloud music credits. Each player -below has synced lyrics — the timing was recovered by transcription and aligned -against the written lyrics, since the model emits no timestamps. - - - -### 1. 糖画小仙 — Sugar-Painting Fairy - -*Guofeng electronic* · 556 characters · 259 s - -
- - -
- -### 2. 溪边捉云记 — Catching Clouds by the Stream - -*Guofeng electronic* · 578 characters · 280 s - -
- - -
- -### 3. 寒露拾桂子 — Picking Osmanthus at Cold Dew - -*Guofeng electronic* · 578 characters · 290 s - -
- - -
- -### 4. 纸鸢与旧木偶 — The Kite and the Old Puppet - -*Guofeng ballad* · 572 characters · 280 s - -
- - -
- -### 5. 小楼檐下听雨眠 — Listening to Rain Under the Eaves - -*Guofeng ballad* · 580 characters · 256 s - -
- - -
- -### 6. 外婆的绿搪瓷碗 — Grandma's Green Enamel Bowl - -*Urban folk* · 679 characters · 252 s - -
- - -
- -### 7. 弄堂口的竹蜻蜓 — The Bamboo Dragonfly at the Alley Mouth - -*Urban folk* · 692 characters · 250 s - -
- - -
- -### 8. 三楼窗台的蒲公英 — Dandelion on the Third-Floor Windowsill - -*Urban folk* · 693 characters · 247 s - -
- - -
- -### 9. 小哪吒借红绫 — Little Nezha Borrows the Red Sash - -*Guofeng DJ remix* · 612 characters · 271 s - -
- - -
- -### 10. 画卷小锦鲤 — The Koi That Left the Painting - -*Guofeng DJ remix* · 634 characters · 286 s - -
- - -
- -### 11. 山海小当家 — Little Boss of the Classic of Mountains and Seas - -*Guofeng DJ remix* · 647 characters · 237 s - -
- - -
- - -```` - ---- - -## 文章 03 / 08:我如何用 FLUX.1 Kontext 打造 AI 家具合成演示 -- **Slug**: `ai-furniture-compositing-with-flux-kontext` -- **英文标题**: How I Built an AI Furniture Compositing Demo With FLUX.1 Kontext -- **分类**: `case-studies` | **发布日期**: 2026-09-11 -- **线上地址**: https://blog.hoelee.com/posts/ai-furniture-compositing-with-flux-kontext/ (EN) | https://blog.hoelee.com/posts/zh/ai-furniture-compositing-with-flux-kontext/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】:[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】: -【人设与用词卫生】: -【CRO与转化建议】: -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "我如何用 FLUX.1 Kontext 打造 AI 家具合成演示" -description: "用 fal.ai FLUX.1 Kontext 把家具产品照合成到风格化房间场景——多图端点、tainted canvas 的修复,以及为什么必须关掉 prompt enhancement。" -pubDate: 2026-09-11 -category: case-studies -blueprint: sme-digital-suite -blueprintLayer: L1-high-conversion-web -tags: [fal-ai, flux, ai-image, javascript, canvas, ecommerce] -ogImage: /og/zh-ai-furniture-compositing-with-flux-kontext.jpg -banner: /banners/ai-furniture-compositing-with-flux-kontext.jpg ---- - -一位家具客户带着一个每个小电商卖家最终都会撞上的问题来找我:他们的产品图都是**孤立的产品照**——一张白底桌子、一张白底椅子——但顾客不会从一片白茫茫的虚无里买家具,他们买的是*房间*。给每个产品、每种室内风格都拍一次实体棚拍,对一个人的生意来说根本不可能。 - -需求是:拿两张产品照(一张餐桌、一把椅子),把它们一起放进一个可信的、高级感的室内场景——生成出来的,而不是拍出来的。这篇文章讲的就是我为验证这件事可行而做的概念验证:我选的模型,以及三个差点把它搞砸的 bug。 - -## 为什么选 FLUX.1 Kontext——以及那个关键的端点 - -第一反应自然是通用文生图模型:*「一张餐桌和一把椅子放在现代客厅里」*。可当客户说*「不——是我的桌子、我的椅子,就是我产品页上那一款」*时,这个思路就崩了。生成一个相似品毫无价值;重点是要让**真实产品原封不动**,只改变它周围的房间。 - -这正是 **FLUX.1 Kontext**(通过 [fal.ai](https://fal.ai))专门解决的问题:以图生图,参考图*锚定*产品,提示词描述场景。但有个坑让我耗掉一个下午:这里有**两个**端点。 - -| 端点 | 参考图输入 | 用途 | -|---|---|---| -| `fal-ai/flux-pro/kontext` | `image_url`(单张) | 编辑 / 重新语境化一张图 | -| `fal-ai/flux-pro/kontext/multi` | `image_urls`(数组) | **合并多张参考图** | - -我需要把*两个独立产品融合进一个场景*,所以普通 `kontext` 不够——它只吃一张参考图。`multi` 变体接受参考图数组,这才是「桌子 + 椅子 → 一个房间」能成立的关键。它被标记为实验版,但这件事上它是唯一的选择。 - -## 用一个纯 HTML 文件调用 fal.ai——不用 SDK,不用服务器 - -做概念验证我不想搭后端。fal.ai 暴露了一个纯 REST 队列 API,所以一个 `index.html` 配 vanilla JavaScript 就能搞定全部。但这是三步,不是一步: - -
-index.html — fal.ai REST queue submission & polling (32 lines) - -```js -// 1. 提交——返回 request_id + 轮询 URL -const submit = await fetch( - "https://queue.fal.run/fal-ai/flux-pro/kontext/multi", - { - method: "POST", - headers: { - Authorization: "Key " + FAL_KEY, - "Content-Type": "application/json", - }, - body: JSON.stringify({ - prompt: "...", - image_urls: [tableDataUrl, chairDataUrl], - enhance_prompt: false, - }), - } -); -const { request_id, status_url, response_url } = await submit.json(); - -// 2. 轮询状态直到 COMPLETED(约 5–30 秒) -let status; -do { - await new Promise((r) => setTimeout(r, 2000)); - status = (await (await fetch(status_url, { - headers: { Authorization: "Key " + FAL_KEY }, - })).json()).status; -} while (status === "IN_QUEUE" || status === "IN_PROGRESS"); - -// 3. 取结果 -const result = await (await fetch(response_url, { - headers: { Authorization: "Key " + FAL_KEY }, -})).json(); -// result.images[0].url 就是生成的图片 -``` - -
- -有两件事我很早就验证了,因为它们决定了「单文件」方案成不成立: - -1. **CORS 是开放的。** `queue.fal.run` 返回宽松的 `access-control-allow-origin`,并允许 `authorization` 头,所以浏览器可以直接调用,无需代理。我在写任何 UI 之前就用 preflight 确认了这一点。 -2. **本地图以 base64 data URI 传入。** fal.ai 的 `image_urls` 接受 `data:` URI,所以我根本不需要先把客户的照片传到存储桶——demo 直接读文件、压缩,然后塞进请求体。 - -## Canvas 压缩层 - -手机照片有 5–10 MB。两张这样、再 base64 编码,请求体会膨胀到无法使用,生成也会变慢。所以提交前,demo 会把每张图过一遍 `` 来缩放并重新编码: - -```js -async function imgToDataURI(file, maxSize = 1024, quality = 0.85) { - const img = await createImageBitmap(file); // 或 new Image() - const scale = Math.min(1, maxSize / Math.max(img.width, img.height)); - const c = document.createElement("canvas"); - c.width = Math.round(img.width * scale); - c.height = Math.round(img.height * scale); - c.getContext("2d").drawImage(img, 0, 0, c.width, c.height); - return c.toDataURL("image/jpeg", quality); // 每张约 200–400 KB -} -``` - -一张 7 MB 的手机照变成约 300 KB 的 JPEG。上传快,模型处理也快。 - -## Bug #1 ——「Tainted canvases may not be exported」 - -demo 在我脑子里能跑,在浏览器里却不行。我一点「生成」就收到: - -> `Failed to execute 'toDataURL' on 'HTMLCanvasElement': Tainted canvases may not be exported.` - -这是浏览器的安全边界,不是 fal.ai 的问题。当你用 `file://` 打开页面、并用 `` 加载一张*本地*图片时,那张图属于**不透明源**。它一被画到 canvas 上,canvas 就变成 *tainted*,`toDataURL()` 拒绝导出——浏览器不会让网页读回一张它无法证明有权读取的文件的像素。 - -有两个修复,我在不同阶段都用过: - -1. **把默认图以 base64 data URI 内嵌进 HTML 源码。** data URI 不会污染 canvas,所以即使从磁盘双击打开也能跑。 -2. **走 HTTP、用同源相对路径**(`src="chair.jpg"`)。一旦页面和图片同源,canvas 就保持干净。这是真实部署的正确答案——demo 现在跑在一个 URL 上,而不是双击打开的文件。 - -通用规则:**只要任何不透明源的图片碰到 canvas,`toDataURL` 立刻报错。** 如果你要加载本地文件,要么内嵌,要么走 HTTP。不改浏览器安全设置,没有第三条路。 - -## Bug #2 —— API key 就在 HTML 里 - -对 demo 来说这可以容忍;对任何公开的东西来说这是个洞。客户端 `fetch` 里的 `Key` 头意味着 key 就在页面源码里,任何人按 F12 就能读到。在交付客户内部预览期间,该静态页面仅搭配了极低余额的限额测试 Key 并在受控私密链接下访问;而在下一阶段的正式 WordPress 插件中,Key 将彻底移至服务端 PHP。 - -真正 WordPress 插件(下一阶段)的计划是把 key 挪到**服务端 PHP**:插件的端点去做带鉴权的 fal.ai 调用并返回图片,浏览器只跟插件自己的路由对话。demo 证明了管线能跑通;插件会把秘密放回秘密该待的地方。 - -## Bug #3 —— 输出看起来「像第二张图」 - -这是让我怀疑 `multi` 到底能不能用的那个。我换了一张新桌子图、生成,结果看起来跟椅子参考图几乎一模一样——好像模型直接忽略了桌子、复制了椅子。 - -在怪模型之前,我先查了一件可证伪的事:**`multi` 到底是在合成,还是只是在回显某一张输入?** 我用感知哈希和平均像素差对比了生成图和两张源图: - -| 对比 | 平均像素差(0 = 相同) | dHash 距离 | -|---|---|---| -| 结果 vs 桌子 | 55.7 | 31 | -| 结果 vs 椅子 | 55.3 | 29 | -| 桌子 vs 椅子 | 69.3 | 30 | - -结果离两张输入都*很远*——端点是真在合成新场景,不是复制。所以「像椅子」的问题不在 API,而在**输入和提示词**。 - -三个叠加的原因: - -1. **参考图不一致。** 最初的 `chair.jpg` 是一张完整的「桌+椅」*场景*照,不是干净的单椅——于是模型看到一张图就已经满足「桌+椅同框」,就倾向照着它来。 -2. **按位置指代不可靠。** Kontext 按图片*内容*匹配,而不是数组顺序。说「第一张图」/「第二张图」并不能可靠地映射到「桌子」/「椅子」。修复是直接*命名物体*——*「the dining table」*和*「the chair」*——让模型自己去匹配正确的参考图。 -3. **测试初期开启了 prompt enhancement。** fal.ai 的 `enhance_prompt` 参数一旦开启,会在后台把提示词改写成更丰富的审美描述——当目标是*「绝不要重新解读产品」*时,这正好帮了倒忙。我将其显式锁定为 `enhance_prompt: false`,强制模型严格执行字面约束,而不是添油加醋。 - -## 最终起作用的提示词 - -客户的核心要求很严格:**原样保留家具**,只去发明房间。这需要一段把大部分篇幅花在*阻止*重新解读、而不是描述风格上的提示词: - -> Create a photorealistic interior photograph using the exact dining table and -> exact chair from the reference images. Preserve both furniture pieces exactly -> as shown — do not redesign, recolor, repaint, restyle, replace, or reinterpret -> either product. Keep their original geometry, proportions, construction, -> material, finish, texture, and color exactly unchanged. -> -> Place the chair naturally beside the dining table in a realistic dining -> position, slightly pulled under the table and properly aligned with it. -> Maintain realistic scale, perspective, and physical contact with the floor. -> The chair and table must look photographed together in the same real room, -> with natural contact shadows — not composited or pasted together. -> -> Preserve the true original color. Do not allow the room lighting, wall or -> floor colors, or grading to alter the furniture. -> -> Set the scene in a spacious modern living room with floor-to-ceiling windows -> and subtle warm afternoon daylight. Use restrained, neutral interior colors. -> -> Photorealistic commercial furniture photography, realistic camera -> perspective, natural proportions, high detail. - -demo 里内置了六段这样的提示词,每种场景风格一段(现代客厅、温馨餐厅、北欧风、侘寂风、以及干净的影棚),每一段只改「set the scene」那一节。 +- **永远不要定义过滤自己的模板变量。** 一个变量只跳一层。 +- **给面板依赖的变量存明确的 `current` 值**,别让全新加载依赖下拉框被选中。 +- **用 Grafana 存下来的值做验证**,而不是我以为的值。 +- **故意留空的缺口写进面板说明**,否则未来的你会花一个下午去\"修\"一个本来就没打算工作的面板。 ## 结果 -一个 `index.html`,没有后端、没有 SDK——两张产品照进,一个布置好的房间出,**每张图 $0.04**,生成约 15–20 秒。客户现在有了一个能拿给*他们*顾客看的可用预览,带场景选择器和可编辑提示词,然后再决定是否投入完整的 WordPress 插件。 +三台主机现在跑同一套布局——每台一个 41 面板的主机面板 + 一个 10 面板的容器面板,分别有 21/23/25 个面板出数,两类结构性空缺写在面板里,而不是留给下一个打开它的人去猜。 -这个概念验证回答了一个它生来就要回答的问题:**是的,你可以保留真实产品、只生成它周围的房间**——只要你尊重那三条真正要紧的约束:每个产品一张干净的参考图、按物体命名(而非按位置)的提示词、以及 `enhance_prompt: false`。 +## 需要为你的业务做这个吗? -## 我会做不同的地方 +如果你有一堆没人信任的 Grafana 面板,或者服务器和 NAS 完全没有监控,我可以帮你搭自托管的 Prometheus + Grafana(主机与容器指标、合理的告警规则、通知推到 Telegram 和邮件),也会把你那些悄悄\"不出数\"的面板修好。 -- **第一天就要求干净的单品参考图。** 下游的每一种失败模式——「像某一张图」的问题、比例漂移——都能追溯到不一致的源照片。我会在碰任何模型之前,先发一份给客户的简短说明(「每张照片一个产品、纯色背景、没有其他家具」)。 -- **先显式关掉 `enhance_prompt`。** 我浪费了一次生成,就因为 enhancement 把我精心写的「别重新解读」改写成恰恰相反的东西。 -- **在写 UI 之前先用真实 preflight 验证 CORS。** 两行 `curl` 的事,却能给整个架构决策祛除风险。 +**WhatsApp:[+60 12-797 2969](https://wa.me/60127972969)** · **邮箱:[me@hoelee.com](mailto:me@hoelee.com?subject=Grafana%20monitoring)** · **[hoelee.com](https://hoelee.com)** ---- - -## 想为你的产品目录做这个吗? - -我为电商企业构建 AI 图像管线、WordPress 插件,以及自托管基础设施。如果你想把产品摆进风格化的房间场景——或者想雇我做类似的 AI 集成——我非常乐意交流: - -- 📱 **WhatsApp:** [+60 12-797 2969](https://wa.me/60127972969) -- 📧 **邮箱:** [me@hoelee.com](mailto:me@hoelee.com?subject=AI%20product%20image%20compositing) -- 🌐 **网站:** [hoelee.com](https://hoelee.com) +网站设计与开发是我的主业;服务器加固与自托管基础设施是它的另一半。 ```` ### 🇬🇧 英文版原文 (English Markdown Primary) ````markdown --- -title: "How I Built an AI Furniture Compositing Demo With FLUX.1 Kontext" -description: "Compositing furniture photos into styled room scenes with fal.ai FLUX.1 Kontext — the multi-image endpoint, the tainted-canvas fix, and why prompt enhancement had to go." -pubDate: 2026-09-11 -category: case-studies -blueprint: sme-digital-suite -blueprintLayer: L1-high-conversion-web -tags: [fal-ai, flux, ai-image, javascript, canvas, ecommerce] -ogImage: /og/ai-furniture-compositing-with-flux-kontext.jpg -banner: /banners/ai-furniture-compositing-with-flux-kontext.jpg +title: "Why Your Grafana Dashboard Shows No Data (When Prometheus Is Fine)" +description: "Every target up, data in Prometheus, panel expressions correct — and all 41 panels reading No data. The culprit was a Grafana variable filtering on itself, plus the verification habit that hid it." +pubDate: 2026-05-17 +updatedDate: 2026-09-29 +category: devops +blueprint: self-hosted-infrastructure +blueprintLayer: L3-observability-self-healing +tags: [grafana, prometheus, dashboards, monitoring, promql, observability] +ogImage: /og/why-your-grafana-dashboard-shows-no-data.jpg +banner: /banners/why-your-grafana-dashboard-shows-no-data.jpg +draft: false --- -A furniture client came to me with a problem that every small e-commerce seller -eventually hits: their catalogue photos are **isolated product shots** — a -table on white, a chair on white — but customers don't buy furniture from a -white void. They buy the *room*. Staging a real photoshoot for every product, -in every interior style, is out of the question for a one-person business. +You open the dashboard you use every day and it's empty. Not "one panel broke" empty — +**every panel**, top to bottom, says *No data*. You check Prometheus and everything is +healthy. This post is the specific, non-obvious cause I hit, and the way my own +verification convinced me the dashboard was fine while it wasn't. -The ask: take two product photos (a dining table, a chair) and place them -together into a believable, high-end interior scene — generated, not -photographed. This post is the story of the proof-of-concept I built to prove -that's possible, the model I chose, and the three bugs that tried to eat it. +## The searchable question -## Why FLUX.1 Kontext — and the endpoint that matters +*My Grafana panels show No data but the data is in Prometheus — why?* -The obvious first instinct is a general text-to-image model: *"a dining table -and a chair in a modern living room."* That fails the moment the client says -*"no — MY table, MY chair, the exact one on my product page."* Generating a -lookalike product is worthless; the whole point is to keep the **real product -unchanged** and only change the room around it. +The usual answers (wrong datasource, wrong time range, `rate()` on a counter that +resets, missing scrape target) did not apply. Here is the checklist that narrowed it +down, in the order that costs the least time: -That's the specific problem **FLUX.1 Kontext** (via [fal.ai](https://fal.ai)) -is built for: image-conditioned generation, where the reference image *anchors* -the product and the prompt describes the scene. But there's a subtlety that cost -me an afternoon: there are **two** endpoints. +```bash +# 1. is the target actually being scraped? +curl -s http://prometheus:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job, health}' -| Endpoint | Reference input | Use case | -|---|---|---| -| `fal-ai/flux-pro/kontext` | `image_url` (single) | edit / re-context one image | -| `fal-ai/flux-pro/kontext/multi` | `image_urls` (array) | **combine multiple reference images** | +# 2. does the metric exist right now? (bypasses Grafana entirely) +curl -s -G http://prometheus:9090/api/v1/query \ + --data-urlencode 'query=node_uname_info{job="unraid-host"}' | jq '.data.result[0].metric' -I needed to *fuse two separate products into one scene*, so plain `kontext` -wasn't enough — it only takes one reference image. The `multi` variant accepts -an array of reference images and is what makes "table + chair → one room" -possible. It's flagged experimental, but it's the only game in town for this. - -## Calling fal.ai from a plain HTML file — no SDK, no server - -For a proof-of-concept I didn't want to stand up a backend. fal.ai exposes a -plain REST queue API, so a single `index.html` with vanilla JavaScript can do -the whole job. There are three steps, not one: - -
-index.html — fal.ai REST queue submission & polling (32 lines) - -```js -// 1. Submit — returns request_id + polling URLs -const submit = await fetch( - "https://queue.fal.run/fal-ai/flux-pro/kontext/multi", - { - method: "POST", - headers: { - Authorization: "Key " + FAL_KEY, - "Content-Type": "application/json", - }, - body: JSON.stringify({ - prompt: "...", - image_urls: [tableDataUrl, chairDataUrl], - enhance_prompt: false, - }), - } -); -const { request_id, status_url, response_url } = await submit.json(); - -// 2. Poll status until COMPLETED (takes ~5–30s) -let status; -do { - await new Promise((r) => setTimeout(r, 2000)); - status = (await (await fetch(status_url, { - headers: { Authorization: "Key " + FAL_KEY }, - })).json()).status; -} while (status === "IN_QUEUE" || status === "IN_PROGRESS"); - -// 3. Fetch the result -const result = await (await fetch(response_url, { - headers: { Authorization: "Key " + FAL_KEY }, -})).json(); -// result.images[0].url is the generated image +# 3. is the panel's expression sound? +curl -s -G http://prometheus:9090/api/v1/query \ + --data-urlencode 'query=count(node_cpu_seconds_total{mode="idle",job="unraid-host"})' ``` -
+All three were green: 7 targets `up`, 25k series ingested, and the panel's raw +expression returned data when I ran it by hand. So the problem was not the metrics, +the scrape, or the PromQL — it was **the variable layer between them**. -Two things I verified early, because they make or break the "single file" -approach: +## The actual cause: a variable that filtered on itself -1. **CORS is open.** `queue.fal.run` returns a permissive - `access-control-allow-origin` and allows the `authorization` header, so a - browser can call it directly with no proxy. I confirmed this with a - preflight before writing any UI. -2. **Local images go in as base64 data URIs.** fal.ai accepts `data:` URIs in - `image_urls`, so I never had to upload the client's photos to a storage - bucket first — the demo reads a file, compresses it, and ships it straight - in the request body. +The dashboard had a host picker. Two variables chained: `$nodename` (which machine) +and `$node` (its `instance` label). The definition of the first one was: -## The canvas compression layer - -Phone photos are 5–10 MB. Two of those, base64-encoded, bloats the request to -unusable size and slows generation. So before submitting, the demo runs each -image through a `` to resize and re-encode it: - -```js -async function imgToDataURI(file, maxSize = 1024, quality = 0.85) { - const img = await createImageBitmap(file); // or new Image() - const scale = Math.min(1, maxSize / Math.max(img.width, img.height)); - const c = document.createElement("canvas"); - c.width = Math.round(img.width * scale); - c.height = Math.round(img.height * scale); - c.getContext("2d").drawImage(img, 0, 0, c.width, c.height); - return c.toDataURL("image/jpeg", quality); // ~200–400 KB each -} +```promql +# BROKEN — the variable filters on its own value +label_values(node_uname_info{job="unraid-host", nodename=~"$nodename"}, nodename) ``` -A 7 MB phone shot becomes a ~300 KB JPEG. Fast to upload, fast for the model to -process. +On first load `$nodename` has no value, so the selector becomes +`nodename=~""` — which matches **nothing**. A variable whose query returns no rows has +no options, so it stays empty. `$node` is then defined against `$nodename`: -## Bug #1 — "Tainted canvases may not be exported" +```promql +label_values(node_uname_info{job="unraid-host", nodename="$nodename"}, instance) +``` -The demo worked in my head. Then it didn't work in a browser. The moment I hit -"generate" I got: +…which is also empty, so every panel filtering on `$node` matches no series. Grafana +isn't lying when it says *No data*: the panel query genuinely has no data, because the +variable that scopes it resolved to nothing. -> `Failed to execute 'toDataURL' on 'HTMLCanvasElement': Tainted canvases may not be exported.` +The fix is to stop the self-reference and give each variable an explicit saved default: -This is a browser security boundary, not a fal.ai problem. When you open the -page over `file://` and load a *local* image with ``, -that image is an **opaque origin**. The moment it's drawn onto a canvas, the -canvas becomes *tainted*, and `toDataURL()` refuses to export it — the browser -won't let a webpage read back the pixels of a file it can't prove it's allowed -to read. +```promql +# FIXED — no self-reference, one hop per variable +# $nodename +label_values(node_uname_info{job="unraid-host"}, nodename) +# $node +label_values(node_uname_info{job="unraid-host", nodename="$nodename"}, instance) +``` -Two fixes, and I used both at different points: +Then save `current` values for both so the dashboard opens populated instead of +depending on a click. (This is also why the same 41-panel layout can be cloned to +three hosts and just work.) -1. **Inline the default images as base64 data URIs** directly in the HTML - source. Data URIs don't taint the canvas, so the demo works even when - double-clicked from disk. -2. **Serve over HTTP with same-origin relative paths** (`src="chair.jpg"`). - Once the page and images share an origin, the canvas stays clean. This is - the right answer for the real deployment — the demo now lives at a URL, not - a double-clicked file. +## Why my verification missed it -The general rule: **`toDataURL` fails the instant any opaque-origin image touches -the canvas.** If you're loading local files, either inline them or serve over -HTTP. There is no third way without changing browser security settings. +This is the part worth stealing. I "verified" the dashboard by expanding each panel's +expression **by hand**, substituting the variable values I knew were correct +(`$nodename=unRaid`, `$node=host.docker.internal:9100`), and asserting the query +returned points. Thirteen of those checks passed. The dashboard was still blank, +because the bug was upstream of the substitution: Grafana's *own* resolution of the +variables was empty. -## Bug #2 — the API key is in the HTML +**A verification that substitutes the very value under suspicion cannot detect that +value failing to resolve.** To catch it, read the values Grafana actually saved and +expand with those: -For a demo this is tolerable; for anything public it's a hole. A `Key` header in -a client-side `fetch` means the key is in the page source, readable by anyone -who presses F12. During the client internal review, this demo ran strictly with -a low-balance restricted test key over a gated link; for production, client-side -keys are a non-starter. +```bash +curl -s -u admin:"$PW" http://grafana:4010/api/dashboards/uid/rYdddlPWk \ + | jq '.dashboard.templating.list[] | {name, current: .current.value}' +``` -The plan for the real WordPress plugin (the next phase) is to move the key to -**server-side PHP**: the plugin's endpoint does the authenticated fal.ai call -and returns the image, while the browser only talks to the plugin's own route. -The demo proved the pipeline works; the plugin will put the secret where -secrets belong. +The correct check is on the variable query itself — run what Grafana runs: -## Bug #3 — the output looked like "the second image" +```bash +# what does $nodename's label_values() actually return? +curl -s -G http://prometheus:9090/api/v1/query \ + --data-urlencode 'query=count by (nodename) (node_uname_info{job="unraid-host"})' | jq '.data.result' +``` -This was the one that made me question whether `multi` even worked. I swapped in -a new table photo, generated, and the result looked almost identical to the -chair reference — as if the model had just ignored the table and copied the -chair. +Empty there means the dashboard cannot possibly render, no matter how healthy the data is. -Before blaming the model, I checked something falsifiable: **is `multi` even -compositing, or just echoing one input?** I compared the generated image -against both source images with a perceptual hash and mean pixel difference: +## The second trap: `$__all` is not `.*` -| Comparison | Mean pixel diff (0 = identical) | dHash distance | -|---|---|---| -| result vs table | 55.7 | 31 | -| result vs chair | 55.3 | 29 | -| table vs chair | 69.3 | 30 | +While automating panel checks, a variable set to *All* reports its current value as the +sentinel string `$__all` — not a regex. Expanding panel expressions naively turns +`name=~"$__all"` into a literal that matches nothing, so a perfectly healthy +container dashboard "fails" every panel. Resolve `$__all` to the variable's `allValue` +(usually `.*`) before expanding, or you'll chase a bug that only exists in your checker. -The result was *far* from both inputs — the endpoint genuinely composites a new -scene, not a copy. So the "looks like the chair" problem wasn't the API; it was -the **inputs and the prompt**. +## Not every empty panel is a bug -Three compounding causes: +After the fix, 21 of 25 panels had data. The remaining four were empty **by design** and +worth naming honestly in the panel description: -1. **The reference photos were inconsistent.** The original `chair.jpg` was a - photo of a full table-and-chair *scene*, not a clean single chair — so the - model already saw "table + chair" satisfied by one image and leaned on it. -2. **Positional prompting is unreliable.** Kontext matches images by *content*, - not array order. Saying "the first image" / "the second image" doesn't - reliably map to "the table" / "the chair". The fix is to name the objects — - *"the dining table"* and *"the chair"* — and let the model match them to the - right reference. -3. **Prompt enhancement was enabled in early tests.** When turned on, fal.ai's - `enhance_prompt` rewrites your prompt into a richer aesthetic description — - exactly wrong when the goal is *"do not reinterpret the product."* I explicitly - pinned `enhance_prompt: false` so the model obeys the literal constraints instead - of embellishing them. +- **PSI panels** (`Pressure`, `Pressure Stall Information`) need `/proc/pressure`, which + the NAS kernel doesn't expose. The metric cannot exist there — the same panels on a + host that *does* export `node_pressure_*` render fine. +- **Root filesystem panels** whose expression carries `fstype!="rootfs"` can never match + on a host whose `/` is a **ramdisk**, which is what unRaid's root is. -## The prompt that finally worked - -The client's core requirement was strict: **preserve the furniture exactly** and -only invent the room. That needs a prompt that spends most of its length -*preventing* reinterpretation, not describing style: - -> Create a photorealistic interior photograph using the exact dining table and -> exact chair from the reference images. Preserve both furniture pieces exactly -> as shown — do not redesign, recolor, repaint, restyle, replace, or reinterpret -> either product. Keep their original geometry, proportions, construction, -> material, finish, texture, and color exactly unchanged. -> -> Place the chair naturally beside the dining table in a realistic dining -> position, slightly pulled under the table and properly aligned with it. -> Maintain realistic scale, perspective, and physical contact with the floor. -> The chair and table must look photographed together in the same real room, -> with natural contact shadows — not composited or pasted together. -> -> Preserve the true original color. Do not allow the room lighting, wall or -> floor colors, or grading to alter the furniture. -> -> Set the scene in a spacious modern living room with floor-to-ceiling windows -> and subtle warm afternoon daylight. Use restrained, neutral interior colors. -> -> Photorealistic commercial furniture photography, realistic camera -> perspective, natural proportions, high detail. - -The demo ships six of these, one per scene style (modern living room, cozy -dining room, Scandinavian, wabi-sabi, and a clean studio), each differing only -in the "set the scene" paragraph. - -## The result - -One `index.html`, no backend, no SDK — two product photos in, a staged room -out, at **$0.04 per image** and roughly 15–20 seconds of generation. The client -now has a working preview they can show *their* customers, with a scene selector -and an editable prompt, before committing to the full WordPress plugin. - -The proof-of-concept answered the question it was built to answer: **yes, you -can keep the real product and only generate the room around it** — as long as -you respect the three constraints that actually matter: a clean reference image -per product, object-named (not positional) prompts, and `enhance_prompt: false`. +Distinguishing "broken because of a bug" from "empty because the metric cannot exist on +this host" is the difference between a real fix and a wild goose chase. ## What I'd do differently -- **Require clean, single-product reference images on day one.** Every failure - mode downstream — the "looks like one image" problem, the proportion drift — - traces back to inconsistent source photos. I'd ship a tiny client-facing - guide ("one product per photo, plain background, no other furniture") before - touching any model. -- **Turn `enhance_prompt` off explicitly, first.** I lost a generation to the - enhancement rewriting my careful "do not reinterpret" instructions into - exactly the opposite. -- **Validate CORS with a real preflight before building UI.** It's a two-line - `curl` and it de-risks the entire architecture decision. +- **Never define a template variable that filters on itself.** One hop per variable. +- **Always save explicit `current` values** for variables a dashboard depends on, so a + fresh load doesn't rely on a populated dropdown. +- **Verify with Grafana's saved values, not my idea of them.** The masking is invisible + in the test result and obvious in the UI. +- **Describe intentional gaps in the panel description.** Future-you will otherwise + spend an afternoon "fixing" a panel that was never meant to work there. ---- +## The result -## Want this for your product catalogue? +All three hosts now run the same instrumented layout — a 41-panel host dashboard and a +10-panel container dashboard each — with 21/23/25 panels returning data respectively, +and the two structurally-empty classes documented in place instead of silently +confusing whoever opens them next. -I build AI image pipelines, WordPress plugins, and self-hosted infrastructure -for e-commerce businesses. If you want to stage your products in styled room -scenes — or hire me for a similar AI integration — I'd love to talk: +## Want this for your business? -- 📱 **WhatsApp:** [+60 12-797 2969](https://wa.me/60127972969) -- 📧 **Email:** [me@hoelee.com](mailto:me@hoelee.com?subject=AI%20product%20image%20compositing) -- 🌐 **Website:** [hoelee.com](https://hoelee.com) +If you have Grafana dashboards that nobody trusts — or servers and a NAS with no +monitoring at all — I build self-hosted Prometheus + Grafana stacks (host and container +metrics, sensible alert rules, alerts to Telegram and email) and I'll fix the ones that +have quietly stopped showing data. + +**WhatsApp: [+60 12-797 2969](https://wa.me/60127972969)** · **Email: [me@hoelee.com](mailto:me@hoelee.com?subject=Grafana%20monitoring)** · **[hoelee.com](https://hoelee.com)** + +Website design and development is my main line of work; server hardening and +self-hosted infrastructure is the other half of it. ```` --- -## 文章 04 / 08:从一首歌到 4 分钟完整 MV:我如何用 Gemini、AI 生成与 Filmora 剪辑串起全流程实操 -- **Slug**: `from-audio-to-a-4-minute-ai-music-video` -- **英文标题**: From Audio to a 4-Minute Music Video: Orchestrating Gemini, AI Diffusion, and Filmora Editing -- **分类**: `case-studies` | **发布日期**: 2026-10-25 -- **线上地址**: https://blog.hoelee.com/posts/from-audio-to-a-4-minute-ai-music-video/ (EN) | https://blog.hoelee.com/posts/zh/from-audio-to-a-4-minute-ai-music-video/ (ZH) +## 文章 03 / 11:你的“磁盘将满”告警在说谎:别再按百分比告警 +- **Slug**: `your-disk-full-alert-is-lying` +- **英文标题**: Your Disk-Full Alert Is Lying: Stop Alerting on Percentages +- **分类**: `devops` | **发布日期**: 2026-06-24 +- **线上地址**: https://blog.hoelee.com/posts/your-disk-full-alert-is-lying/ (EN) | https://blog.hoelee.com/posts/zh/your-disk-full-alert-is-lying/ (ZH) - **本篇审核状态**: ⏳ 待审核 (Pending) ### ✍️ 顾问审阅意见区 (Review Feedback) @@ -2939,294 +637,234 @@ scenes — or hire me for a similar AI integration — I'd love to talk: ### 🇨🇳 中文版原文 (Chinese Markdown Twin) ````markdown --- -title: "从一首歌到 4 分钟完整 MV:我如何用 Gemini、AI 生成与 Filmora 剪辑串起全流程实操" -description: "基于《月牙湾》1080P 高清动漫 OP 制作实录:详解 AI 音乐重构、Gemini 提示词分镜规划、文字生图与图生视频多轮采样筛选调优,以及 Filmora 剪辑与特效合成的真实生产实践。" -pubDate: 2026-10-25 -category: case-studies -blueprint: ai-agent-automation -blueprintLayer: L4-aigc-media-pipelines -stream: creative-lab -tags: ["ai-video", "music-video", "gemini", "filmora", "face-swapping", "ai-music", "creative-workflows"] -ogImage: "/og/zh-from-audio-to-a-4-minute-ai-music-video.jpg" -banner: "/banners/from-audio-to-a-4-minute-ai-music-video.jpg" +title: "你的“磁盘将满”告警在说谎:别再按百分比告警" +description: "\"剩余 3%\"听着像紧急事故,直到你发现那是 500 GB。在多 TB 的卷上,百分比阈值会在没事的时候乱叫,也会在真出事的时候沉默。" +pubDate: 2026-06-24 +updatedDate: 2026-09-29 +category: devops +blueprint: self-hosted-infrastructure +blueprintLayer: L3-observability-self-healing +tags: [monitoring, alerting, prometheus, grafana, storage, capacity, observability] +ogImage: /og/zh-your-disk-full-alert-is-lying.jpg +banner: /banners/your-disk-full-alert-is-lying.jpg draft: false --- -绝大多数 AI 视频演示都停留在 5 秒的单镜头片段:一个女生转头、一段雨水打在水坑里的特写。发在社交媒体上很惊艳,但要真正用 AI 从零做出一整首长达 4 分钟(240 秒)、包含音乐生成与 20+ 个连续镜头的完整音乐视频(MV),完全是另一套现实的工程与创作实操。 +我搭好监控、写了一套看着合理的规则,然后用一个下午删掉了其中大半阈值。它们全是百分比,而且都以同一种方式错了。 -真实做过的人都知道,**AI 根本做不到“一键完美直出”**: -* 让 Gemini 生成分镜提示词,需要细致引导才不偷工减料; -* 文字生图经常出现五官崩坏、服装变色; -* 图生视频跑出来的微动作,往往需要反复生成多次(重抽)才能挑出一段自然的; -* 最后要把这些零散的素材剪成一首节奏对齐、转场自然的完整 MV,必须依赖 **Filmora** 这样的专业剪辑软件做精细卡点、特效叠加与平滑过渡。 +## 三个教我道理的告警 -为了验证整套流水线能否真正跑通,我重构改编了经典华语曲目**《月牙湾》(Crescent Bay)**,将其打造为日系热血奇幻动漫 OP 风(Oriental Fantasy J-Rock),并完成了整整 4 分钟 1080P 高清成片。 +**1.「文件系统使用率超过 90%」**——它在一个还剩约 500 GB 的 17 TB 卷上触发。没事发生,也不会马上有事发生。这种体量的卷,一次大备份就能让百分比摆动几个点:比例是噪声,而\"还有 500 GB 可用\"才是世界的真实状态。 -以下是完整的最终成品(可直接全屏播放): +**2.「内存使用率超过 90%」**——在一台**可用**内存 4.8 GB 的主机上反复触发。Linux 会用空闲内存做 page cache,需要时立刻归还;`MemTotal - MemFree` 描述的是文件缓存,不是你的风险。真正预示 OOM 的是 `MemAvailable`。 -
- -
+**3.「CPU steal 超过 25%」**——一台把邮件栈跑得完全正常的 VPS,steal 长期在 41%。steal 说明宿主机忙,它是**容量**信号而不是**故障**信号;7 vCPU 的机器 41% steal 就是这台机器的成本。这条规则产生的告警永远为真、也永远无用。 ---- +共同点:**我在对比例告警,而比例不知道东西有多大。** -## 1. 音乐生成与风格重构:日系热血动漫 OP 的诞生 +## 对\"后果\"告警,而不是对比例 -视觉的灵魂是节奏。制作 MV 的第一步,首先是产生符合设想的音乐母带。 +我现在对每个阈值都问一句:*如果这个状态持续下去,实际会发生什么?* 答案几乎总能用绝对单位表达。 -原版的《月牙湾》带有浓厚的西域民谣与流行底蕴。为了赋予它更强烈的视觉冲击力和画面推动感,我通过 AI 音乐生成工具将整首歌重构为 **Oriental Fantasy J-Rock / AniSong(东方奇幻日系动漫 OP 风格)**: -* **鼓点与贝斯**:换上高速双踩双底鼓(Double Bass Drum)与持续滚动的失真电贝斯,让副歌的推进感拉满; -* **吉他与乐器交织**:前奏保留了西域传统风情的空灵音色,但在主歌切入后迅速叠加大失真主音电吉他; -* **人声质感**:要求高燃、清亮且穿透力极强的热血女声,在副歌瞬间爆发。 +| 不要这样 | 改成 | 原因 | +|---|---|---| +| 文件系统 > 90% | 剩余空间 < N GB | \"我还能不能写进去\"才是问题 | +| 内存使用 > 90% | `MemAvailable` < 512 MB | 逼近 OOM,而不是\"缓存很暖和\" | +| CPU steal > 25% | steal > 50% | 容量成本 vs 真的被饿死 | +| 负载 > N | `load1 / 核数 > 2` | 不谈核数的负载没有意义 | -有了这首 4 分钟的完整音频后,我将全曲按情绪起伏拆解成 24 个左右的时钟分段(前奏、主歌 A/B、副歌爆发、间奏吉他 Solo、尾奏),为后续的分镜画面打好节拍地基。 +PromQL 的前后对比: ---- +```promql +# BEFORE — 17 TB 卷的百分比;还剩半 TB 就开叫 +(1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 90 -## 2. Gemini 分镜设计:两阶段提示词规划 - -很多人用大模型生成分镜时,习惯把整首歌歌词丢给它,说“帮我生成 24 个生图提示词”。这样做的结果往往惨不忍睹:前 3 个镜头写得很详细,到了第 10 镜以后提示词就只剩两三句话,甚至连主角长相都丢了。 - -在 Gemini 中,我采用**分阶段结构化输出**: - -1. **第一阶段:锁定视觉核心 DNA(Visual Constants)** - * **角色固定锚点**:固定主角发型(黑长直双环髻、红丝带发簪)、服装(青白双层汉元素织物)、外貌气质与年龄; - * **色彩与摄影基调**:统一色温(4500K 冷青色调)、电影镜头感(ARRI 50mm 镜头虚化、胶片微颗粒感); - * **留白区域**:明确每张图采用三分法构图,人物位于画面一侧,另一侧预留干净的景深虚化负空间(Copy Space)。 - -2. **第二阶段:生成 24 镜生图(T2I)与生视频(I2V)提示词** - * 让 Gemini 逐个格子输出:该镜头的**生图提示词(Text-to-Image Prompt)**以及对应的**图生视频运动提示词(Image-to-Video Motion Prompt)**; - * 人物镜头与纯意象空镜头(大漠风沙、屋檐雨滴、古琴特写)穿插交替,避免连续正脸特写造成视觉疲劳。 - ---- - -## 3. 生图与图生视频的现实:多轮采样与筛选调优 - -AI 视频制作中耗费最多时间的,就是**“生图 -> 挑图 -> 垫图生视频 -> 重新采样挑选”**的反复迭代过程。 - -### ① 文字生图(T2I):解决画风与角色漂移 -即使有详细的提示词,生图模型也常常会出现五官比例怪异、衣服颜色擅自更改、或在空镜头里莫名其妙画一张人脸的情况。 -* 单个镜头通常需要生成 3~5 次,挑选出光影、人物站位和留白最符合分镜的一张; -* 挑选出的首帧图片,必须保证主角的标志性服饰和发型在视觉上具备高度辨识度。 - -### ② 图生视频(I2V):控制微动作,避免身体畸变 -拿挑选好的静态图送入图生视频模型生成 5~10 秒的动态视频,是崩坏率最高的环节: -* **严禁否定词**:在正向提示词中写“不要变形、不要乱动”,模型反而容易产生肢体扭曲;必须使用正面稳定词(如微风吹拂发丝、胸口自然呼吸起伏、镜头缓慢推进); -* **动作幅度控制**:AI 生成长镜头时,动作幅度越小越逼真。大步奔跑或大幅度转身很容易引发身体部位撕裂; -* **多次生成挑选最佳片段**:一个 10 秒的视频镜头,常常需要生成 3 次以上。有时候前 4 秒很自然、第 5 秒开始崩坏,这没关系——只要有几秒是完美的,就可以保留下来放到剪辑软件中裁剪使用。 - -### ③ 面部一致性锁颜:本地 FaceFusion 深度模型解决“多镜头变脸” -即便在提示词中固定了发型与服饰,文生图与图生视频在 20 多个连续镜头中,人物五官骨相依然会出现微小的“角色漂移”,更无法直接保证封面海报与成片女主拥有完全一致的面孔。 -* **本地算力部署**:利用本机显卡(RTX 3060 12GB)离线运行 **FaceFusion** 神经网络换脸流水线; -* **拒绝 2D 贴纸感**:放弃容易产生贴纸感和边缘闪烁的通用 128 像素简易算法,选用训练精度更高的 **320 规格深度神经网络模型(DFM)**,让光影与 3D 轮廓深度融合; -* **高清面部修复**:配合 **SCRFD** 灵敏人脸检测与 **GFPGAN** 高清修复,将精挑细选的高辨识度东方少女面容(与封面海报保持 100% 同款面庞)无缝置换入主角核心特写镜头中,彻底消除“一转镜头换了个人”的违和感。 - ---- - -## 4. Filmora 剪辑与后期合成:真正让素材变成 MV 的关键一步 - -AI 生成出来的只是一个个孤立的视频切片,**把这些碎片变成一首真正有呼吸感、有节奏感的音乐视频,核心功力全在剪辑软件里**。 - -我使用 **Filmora** 完成了最后的重度后期与合成: - -``` -[原始音频轨道] ────────────────────────────────────────────── (对齐卡点基准) -[视频轨 1 / 2] [ 镜头1 (淡出) ] ── (平滑交叉叠化) ── [ 镜头2 (淡入) ] -[特效与转场层] [ 光晕 / 风沙粒子 / 局部调色 LUT ] -[歌词与排版层] [ 双语艺术字幕 / 动态节拍对齐 ] +# AFTER — 真正装数据的卷上,还剩多少容量 +min by (instance, mountpoint) ( + node_filesystem_avail_bytes{mountpoint=~"/volume[0-9]+|/mnt/ssd|/mnt/disk[0-9]+"} +) < 25e9 ``` -1. **精准对位音乐节拍(Beats & Cuts)**: - * 将音频波形导入 Filmora,标记出吉他切入点、重低音鼓点与高潮副歌爆发点; - * 将 AI 生成的 24 段视频按节奏修剪,只截取每段生成最平稳、动作最自然的 6~10 秒精华部分,让镜头的切换严丝合缝地踩在鼓点上。 -2. **消灭黑屏跳闪,打造平滑过渡**: - * 绝不能用粗暴的黑场淡入淡出(Dip to Black),否则整首 MV 每 10 秒就会像眨眼一样暗下去一次; - * 在 Filmora 中使用两轨交叉叠化(Cross Dissolve)与柔光过渡,让前一个画面的残影与后一个画面的光影自然交融,呈现出电影般的无缝穿梭感。 -3. **叠加氛围特效与统一色调**: - * 针对沙漠敦煌与动漫热血主题,叠加了微弱的风沙粒子动效与镜头光晕(Lens Flare); - * 通过全局 LUT 统一 20 多个不同镜头的明暗与冷暖色阶,消除不同生成批次之间的色差。 -4. **歌词排版与呈现**: - * 结合大号中文艺术字体与下方的英文字幕,做好边框与阴影,防止歌词在动态背景中被背景颜色吞噬。 +```promql +# BEFORE — 把 page cache 当成压力 +(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90 ---- +# AFTER — 真正的 OOM 前兆 +node_memory_MemAvailable_bytes < 512 * 1024 * 1024 +``` -## 5. 经验总结:个人创作者的 AI 生产力闭环 +两个让规则能活下来的细节: -完成这一首 4 分钟的《月牙湾》MV,给我最大的感受是: +**把比较放进阈值条件里,不要塞进表达式。** 查询就写 `min by (mountpoint) (node_filesystem_avail_bytes{...})`,让告警规则去和 `25000000000` 比。若你在 PromQL 里写死 `< 25e9`、规则里又留着继承来的 `> 90`,规则照样会评估——但界面显示的阈值是错的,下一个读它的人得从两个互相矛盾的条件下反推你的意图。 -1. **AI 放大的是制作上限,但不能取代审美判断**:AI 可以在短时间内产出海量的画面与动作构想,但“哪一张图最好、哪一段动作能用、在哪个鼓点切镜头”,依然需要创作者的主观审美与导演直觉。 -2. **多工具协同才是最优解**: - * **Gemini** 负责文字创意、世界观定调与分镜拆解; - * **AI 绘图与视频扩散** 负责提供高质感的视听资产; - * **FaceFusion (本地 RTX 3060)** 负责锁定女主角核心特写镜头与封面海报的面部一致性; - * **Filmora** 负责精准的时间轴控制、镜头拼接、声画对齐与视觉特效。 +**排除那些\"只是另一个挂载点的视图\"。** 一条\"剩余空间\"规则有多诚实,取决于它的选择器。在我自己的环境里,同一个 NAS 卷出现三次(`/volume1`、`/opt`、以及在另一台主机上以 CIFS 再次挂载),而 unRaid 的 shfs 联合挂载报告 `avail=0`——不加排除的规则不是永远误报,就是把容量算成两倍。一个写明白的选择器胜过聪明的通用写法。 -这种“AI 生成资产 + 人工剪辑雕琢”的工作流,让个人或小型团队在过去需要整个制作公司才能完成的 4 分钟完整 MV 制作,变成了一个人在工作站前几天内即可交付的现实。 +## 噪声不是无害的 ---- +坏阈值的代价不在告警本身,而在**训练**:每条\"没事乱叫\"的告警都在教你先扫一眼、然后忽略;等你真正该看的那条到来时,它出现在一个你早已不再读的频道里。我的规则总数**减少了**,覆盖率却上升了——更少,但每条都对应一个明确的后果。 -*如果您对 AI 音视频流水线搭建、数字内容自动化生成或商业多媒体交付感兴趣,欢迎交流探讨:* +## 阈值是私人的,形状是通用的 -- 📱 **WhatsApp:** [+60 12-797 2969](https://wa.me/60127972969) -- 📧 **邮件:** [me@hoelee.com](mailto:me@hoelee.com?subject=%E5%92%A8%E8%AF%A2%20AI%20MV%20%E5%88%9B%E4%BD%9C%E4%B8%8E%E6%B5%81%E6%B0%B4%E7%BA%BF) -- 🌐 **网站:** [hoelee.com](https://hoelee.com) +上面的数字是我的:NAS 卷剩不到 25 GB 我会在意、可用内存低于 512 MB 我才管、VPS steal 超过 50% 才算被饿。你的数字随硬件和容忍度而变。 + +能通用的是形状: + +- 用绝对单位而非比例; +- 把后果写进告警的摘要里; +- 选择器明确声明它在替哪些挂载点说话; +- 规则数量少到你还能读完每一条。 + +## 结果 + +同一套栈,从\"三条永久无用报警\"变成 16 条在机房健康时保持安静的规则——其中一条立刻暴露了一个只剩 21 GB 的 NAS 卷,而那正是百分比规则一直藏在\"99% 满的巨物\"背后、被你学会忽略的东西。 + +## 需要为你的业务做这个吗? + +如果你的监控在发你已经学会忽略的告警,或者你根本没有监控、宁愿从一条规则而不是从一次失败的备份里得知磁盘满了,我可以帮你搭自托管的监控与告警(Prometheus + Grafana,通知推 Telegram 和邮件),阈值按\"什么会真的坏\"来定。 + +**WhatsApp:[+60 12-797 2969](https://wa.me/60127972969)** · **邮箱:[me@hoelee.com](mailto:me@hoelee.com?subject=Monitoring%20and%20alerting)** · **[hoelee.com](https://hoelee.com)** + +网站设计与开发是我的主业;服务器加固与自托管基础设施是它的另一半。 ```` ### 🇬🇧 英文版原文 (English Markdown Primary) ````markdown --- -title: "From Audio to a 4-Minute Music Video: Orchestrating Gemini, AI Diffusion, and Filmora Editing" -description: "A complete production breakdown of building a 4-minute 1080p anime OP music video (Crescent Bay): AI music generation, Gemini storyboard prompting, iterative image-to-video rerolls, and Filmora VFX editing." -pubDate: 2026-10-25 -category: case-studies -blueprint: ai-agent-automation -blueprintLayer: L4-aigc-media-pipelines -stream: creative-lab -tags: ["ai-video", "music-video", "gemini", "filmora", "face-swapping", "ai-music", "creative-workflows"] -ogImage: "/og/from-audio-to-a-4-minute-ai-music-video.jpg" -banner: "/banners/from-audio-to-a-4-minute-ai-music-video.jpg" +title: "Your Disk-Full Alert Is Lying: Stop Alerting on Percentages" +description: "3% free sounds like an emergency until you notice it is 500 GB. On multi-terabyte volumes, percentage thresholds fire when nothing is wrong — and miss when something is." +pubDate: 2026-06-24 +updatedDate: 2026-09-29 +category: devops +blueprint: self-hosted-infrastructure +blueprintLayer: L3-observability-self-healing +tags: [monitoring, alerting, prometheus, grafana, storage, capacity, observability] +ogImage: /og/your-disk-full-alert-is-lying.jpg +banner: /banners/your-disk-full-alert-is-lying.jpg draft: false --- -Most AI video demonstrations end after five seconds: a stunning single-shot clip of a character turning their head or raindrops hitting a puddle. It looks cinematic in a tweet, but assembling a complete four-minute music video (240 seconds) with music generation and 20+ continuous scenes is a vastly different engineering and creative challenge. +I built a monitoring stack, wrote a reasonable set of rules, and then spent an +afternoon deleting most of my own thresholds. Every one of them was a percentage, and +every one of them was wrong in the same specific way. -Anyone who has actually produced long-form AI content knows the blunt truth: **AI never outputs a perfect 4-minute video in one click**. -* Gemini requires careful multi-stage prompt engineering, otherwise it starts cutting corners halfway through; -* Text-to-Image models frequently distort hands or swap character clothes; -* Image-to-Video models suffer from high deformation rates on 10-second clips, demanding multiple rerolls before landing a usable segment; -* Finally, turning these isolated raw clips into a coherent, rhythmic music video requires heavy post-production in professional NLE software like **Wondershare Filmora** for beat-cutting, visual effects, and seamless cross-dissolves. +## The three alerts that taught me -To test how viable this end-to-end production workflow is, I remade the classic track **Crescent Bay (月牙湾)** into a high-energy J-Rock anime opening (Oriental Fantasy J-Rock) and produced a full 4-minute 1080p music video. +**1. "Filesystem above 90% used."** It fired on a 17 TB volume that had ~500 GB free. +Nothing was wrong. Nothing was *about* to be wrong. On a volume that size, a single +large backup can swing the percentage by several points — the ratio is noise, and the +fact that 500 GB is still available is the actual state of the world. -Here is the finished music video: +**2. "Memory above 90% used."** Fired constantly on a host with 4.8 GB *available*. +Linux uses free RAM for page cache and gives it back on demand; `MemTotal - MemFree` +describes your filesystem cache, not your risk. The number that predicts an OOM is +`MemAvailable`. -
- -
+**3. "CPU steal above 25%."** Fired on a VPS that had been running its mail stack +perfectly at 41% steal. Steal means the hypervisor is busy — it is a *capacity* signal, +not a *failure* signal, and 41% steal on a 7-vCPU box was simply what that host costs. +The rule produced alerts that were always true and never actionable. ---- +The common thread: **I was alerting on a ratio, and the ratio doesn't know how big the +thing is.** -## 1. Music Generation & Re-Arrangement: Birth of an Anime Opening +## Alert on the consequence, not the ratio -The soul of an impactful video is its musical tempo and momentum. Before drafting a single visual, the first step was creating the master audio. +The test I now apply to every threshold: *if this condition persists, what will actually +happen?* That answer is almost always expressible in absolute units. -The original *Crescent Bay* is a classic Chinese pop track with traditional Silk Road instrumentation. To give it the visual driving force required for an anime opening, I reimagined the arrangement through AI music generation tools into an **Oriental Fantasy J-Rock / AniSong**: -* **Drums & Bass:** Introduced rapid double-bass drum rhythms and a sustained, driving distorted electric bass to build intense forward momentum into the chorus; -* **Guitars & Exotic Textures:** Kept the evocative traditional desert flute and acoustic strums during the intro, then slammed into overdriven lead electric guitars as the verses kick in; -* **Vocal Character:** Tuned for a soaring, resonant, high-energy female vocal that cuts through the heavy rock mix during the chorus. +| Instead of | Alert on | Because | +|---|---|---| +| Filesystem > 90% | Free space < N GB | "Can I still write?" is the question | +| Memory > 90% used | `MemAvailable` < 512 MB | about to OOM, not "cache is warm" | +| CPU steal > 25% | Steal > 50% | capacity cost vs. starvation | +| Load average > N | `load1 / cores > 2` | load is meaningless without core count | -With the completed 4-minute audio ready, I divided the track into 24 distinct 10-second timeline cells (Intro, Verse A/B, Pre-Chorus, Chorus explosions, Guitar Solo, Outro) to serve as the structural anchor for all visuals. +In PromQL, before and after: ---- +```promql +# BEFORE — percentage of a 17 TB pool; fires with half a terabyte still free +(1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 90 -## 2. Gemini Storyboard Architecture: Multi-Stage Prompting - -A common failure mode when using LLMs for storyboarding is dumping the entire lyric sheet into a single prompt and asking for "24 scene prompts." The model writes detailed paragraphs for the first three scenes, but by Scene 12 it burns through its context budget, dropping character descriptions and producing generic single-sentence prompts. - -Using Gemini, I decoupled the workflow into two structured stages: - -1. **Stage 1: Locking Visual DNA & Constants** - * **Character Anchors:** Fixed hairstyle (straight jet-black hair in twin-looped buns with a trailing crimson ribbon), dual-layer clothing (pale celadon-cyan organza outer robe over matte ivory silk), and age/silhouette; - * **Cinematographic Constants:** Enforced a consistent 4500K cool-jade color palette and optical traits (`ARRI Alexa 65, Panavision Primo 50mm anamorphic lens, shallow depth of field`); - * **Rule-of-Thirds Composition:** Positioned subjects strictly on one third of the frame, demanding clean, creamy out-of-focus negative space on the opposing side for typography. - -2. **Stage 2: 24 Atomic Scene Prompts** - * Tasked Gemini to generate both the **Text-to-Image (T2I) visual prompt** and the corresponding **Image-to-Video (I2V) motion prompt** for each 10-second grid cell; - * Alternated between character portraits, environmental wide shots, and macro B-roll (wind chimes, rain ripples on stone, close-up of zither strings) to prevent visual fatigue from continuous close-up portraits. - ---- - -## 3. The Reality of Video Diffusion: Iterative Rerolls & Quality Control - -In real-world generative video production, the bulk of production time is spent on **iterative rerolls and quality filtering**. - -### ① Text-to-Image (T2I): Fighting Drift -Even with explicit prompt anchors, image models occasionally render strange facial proportions, hallucinate western clothing, or insert unwanted faces into macro landscape shots. -* Each shot required generating 3 to 5 candidates to find the one with optimal lighting, correct silhouette, and clean negative space; -* Selected base frames served as strict visual anchors for the video diffusion stage. - -### ② Image-to-Video (I2V): Controlling Micro-Motions -Passing still images into video diffusion models (Veo, Kling, Runway) for 5 to 10 seconds is where most deformations occur: -* **Never Use Negative Words in Positive Prompts:** Prompting `no morphing` or `don't distort` frequently backfires in text encoders, triggering limb anomalies. Instead, use positive stabilizing anchors: `consistent anatomy, temporal stability, subtle breathing, gentle wind blowing through hair`; -* **Micro-Motions Over Large Gestures:** In diffusion models, smaller motions look far more realistic. Complex full-body turns or rapid running almost always cause torso tearing; -* **Multiple Rerolls to Harvest Usable Takes:** Producing a 10-second clip often takes 3 or more attempts. Even if a take begins to morph at second 6, the first 5 flawless seconds can be harvested and trimmed during editing. - -### ③ Locking Facial Identity via Local FaceFusion -Even when hairstyle and clothing descriptors are nailed down in prompts, text-to-image and video diffusion models exhibit subtle bone-structure and facial drift across 20+ distinct scenes. Crucially, raw diffusion cannot guarantee that the protagonist in the final video matches the hero cover art. -* **Offline Local GPU Pipeline:** Deployed **FaceFusion** locally on an NVIDIA RTX 3060 12GB; -* **Beyond 2D Affine Warping:** Bypassed basic 128px insightface warping (which causes plastic edge shimmering) in favor of high-resolution **320 DFM deep neural models**, achieving seamless 3D contour integration and natural skin lighting; -* **Restoration & Detection:** Paired with **SCRFD** face detection and **GFPGAN** neural enhancement to lock in a striking, consistent East Asian heroine face across both key close-up video cuts and the official 2.7K cover artwork. - ---- - -## 4. Filmora Post-Production: Turning Raw Clips into a Music Video - -AI models only produce disconnected video fragments. **Transforming these raw assets into a cohesive, breathing music video happens entirely in the editing suite**. - -I assembled and finished the entire video in **Wondershare Filmora**: - -``` -[Master Audio Track] ────────────────────────────────────────────── (Beat & Rhythm Guide) -[Video Tracks 1 & 2] [ Clip 1 (Fade Out) ] ── (Cross Dissolve) ── [ Clip 2 (Fade In) ] -[VFX & Adjustment] [ Lens Flare / Dust Particles / Master Cinematic LUT ] -[Typography Layer] [ Bilingual Synchronized Artistic Subtitles ] +# AFTER — capacity left, on the volumes that hold data +min by (instance, mountpoint) ( + node_filesystem_avail_bytes{mountpoint=~"/volume[0-9]+|/mnt/ssd|/mnt/disk[0-9]+"} +) < 25e9 ``` -1. **Precision Beat-Cutting**: - * Imported the master audio and marked downbeats, guitar riffs, and dynamic chorus drops; - * Trimmed each AI video clip to its cleanest, most stable 5–9 second segment, aligning cuts precisely with musical transients. -2. **Eliminating Black-Screen Dips with Cross-Dissolves**: - * Hard fade-to-black transitions (`Dip to Black`) make a 4-minute video feel like an annoying strobe light; - * Used multi-track overlapping cross-dissolves with gentle easing, allowing light and motion from the preceding clip to blend smoothly into the next. -3. **Atmospheric VFX & Unified Color Grading**: - * Added subtle desert dust particle overlays and delicate anamorphic lens flares to enhance the fantasy anime atmosphere; - * Applied a master color-grading LUT across all 24 clips to harmonize luminance and color temperature variations between different generation batches. -4. **Bilingual Subtitle Styling**: - * Styled large, expressive Chinese calligraphy font lyrics paired with clear English translations beneath, backed by delicate shadows to maintain legibility against bright desert and sky backdrops. +```promql +# BEFORE — page cache counted as "pressure" +(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90 ---- +# AFTER — the actual OOM precursor +node_memory_MemAvailable_bytes < 512 * 1024 * 1024 +``` -## 5. Key Takeaways for Solo Creators +Two details that make these rules survive contact with a real dashboard: -Producing this 4-minute *Crescent Bay* music video clarified what modern AI content creation actually looks like: +**Keep the comparison in the threshold condition, not inside the expression.** Query +`min by (mountpoint) (node_filesystem_avail_bytes{...})`, then let the alert rule +compare it to `25000000000`. If you bake `< 25e9` into the PromQL *and* leave an +inherited `> 90` in the rule, the rule still evaluates — but it displays a nonsense +threshold, and the next person to read it has to reconstruct your intent from two +contradictory conditions. -1. **AI Expands Production Bandwidth, But Human Taste Directs the Cut:** Generative models can spit out endless visual variations, but selecting the right take, controlling motion amplitude, and knowing exactly when to cut on a beat remains the domain of human creative direction. -2. **Tool Synergy Wins:** - * **Gemini** handles conceptualization, world-building, and structured storyboard prompting; - * **AI Image & Video Diffusion** generate high-fidelity raw audio-visual assets; - * **FaceFusion (Local RTX 3060)** locks protagonist facial identity and consistency across key close-ups and hero cover art; - * **Filmora** provides precise timeline control, clip assembly, rhythm synchronization, and post-processing VFX. +**Exclude the mounts that are views of other mounts.** A "free space" rule is only as +honest as its selector. On my own setup, one NAS volume appears three times (as +`/volume1`, as `/opt`, and again on another host as a CIFS re-mount), and unRaid's shfs +union mount reports `avail=0` — a rule without exclusions would either fire forever or +double-count capacity. One explicit selector beats a clever generic one: -This collaborative workflow makes it possible for a solo creator to deliver an entire 4-minute anime music video—a project that previously required a dedicated production team—in a matter of days from a single workstation. +```promql +node_filesystem_avail_bytes{mountpoint=~"/volume[0-9]+|/mnt/ssd|/mnt/disk[0-9]+"} +``` ---- +## Noise is not harmless -*Interested in custom AI video pipelines, automated digital media workflows, or high-throughput creative production for your business? Let's connect:* +The cost of a bad threshold isn't the alert itself — it's the *training*. Every alert +that fires while nothing is wrong teaches you to skim, then to ignore, and then the one +that matters arrives in a channel you've stopped reading. My rule count went **down** +while coverage went up: fewer, each tied to a named consequence. -- 📱 **WhatsApp:** [+60 12-797 2969](https://wa.me/60127972969) -- 📧 **Email:** [me@hoelee.com](mailto:me@hoelee.com?subject=AI%20Music%20Video%20Production) -- 🌐 **Website:** [hoelee.com](https://hoelee.com) +## Thresholds are personal, shapes are not + +The specific numbers above are mine — 25 GB free before a NAS volume bothers me, 512 MB +available before I care about memory, 50% steal before a VPS is being starved. Yours +will differ with your hardware and your tolerance. + +What transfers is the shape: + +- absolute units over ratios, +- the consequence written into the alert's summary, +- a selector that states exactly which mounts it speaks for, +- and a rule count small enough that you still read every message. + +## The result + +The same stack that was producing three permanent, uninformative warnings now runs 16 +rules that stay silent when the lab is healthy — and one of them immediately surfaced a +NAS volume sitting at 21 GB free, which is the kind of thing a percentage rule had been +hiding behind a 99%-full giant that I'd learned to ignore. + +## Want this for your business? + +If your monitoring sends you alerts you've learned to ignore, or you have none and would +rather find out about a full disk from a rule than from a failed backup, I set up +self-hosted monitoring and alerting (Prometheus + Grafana, alerts to Telegram and email) +with thresholds tied to what actually breaks. + +**WhatsApp: [+60 12-797 2969](https://wa.me/60127972969)** · **Email: [me@hoelee.com](mailto:me@hoelee.com?subject=Monitoring%20and%20alerting)** · **[hoelee.com](https://hoelee.com)** + +Website design and development is my main line of work; server hardening and +self-hosted infrastructure is the other half of it. ```` --- -## 文章 05 / 08:我是如何用 Remotion 与 Docker CDP 打造全自动 MV 生成与多语种油管发布流的 -- **Slug**: `how-i-built-an-automated-music-video-pipeline` -- **英文标题**: How I Built an Automated Music Video Pipeline with Remotion and Dockerized YouTube CDP -- **分类**: `case-studies` | **发布日期**: 2025-04-18 -- **线上地址**: https://blog.hoelee.com/posts/how-i-built-an-automated-music-video-pipeline/ (EN) | https://blog.hoelee.com/posts/zh/how-i-built-an-automated-music-video-pipeline/ (ZH) +## 文章 04 / 11:浏览器翻译没有 API:我给一个纯中文网页应用加上了英文模式 +- **Slug**: `adding-english-mode-to-a-chinese-only-web-app` +- **英文标题**: No API for Browser Translation: Adding an English Mode to a Chinese-Only Web App +- **分类**: `engineering` | **发布日期**: 2025-06-11 +- **线上地址**: https://blog.hoelee.com/posts/adding-english-mode-to-a-chinese-only-web-app/ (EN) | https://blog.hoelee.com/posts/zh/adding-english-mode-to-a-chinese-only-web-app/ (ZH) - **本篇审核状态**: ⏳ 待审核 (Pending) ### ✍️ 顾问审阅意见区 (Review Feedback) @@ -3240,310 +878,2461 @@ This collaborative workflow makes it possible for a solo creator to deliver an e ### 🇨🇳 中文版原文 (Chinese Markdown Twin) ````markdown --- -title: "我是如何用 Remotion 与 Docker CDP 打造全自动 MV 生成与多语种油管发布流的" -description: '基于 Remotion、Whisper、Docker 与 Chrome DevTools Protocol 的全自动音乐视频流水线:从歌词声学对齐、1080P 渲染到 YouTube 自动发布与多语种外挂字幕。' -pubDate: 2025-04-18 -updatedDate: 2026-10-22 -category: case-studies -blueprint: ai-agent-automation -blueprintLayer: L4-aigc-media-pipelines -stream: creative-lab -tags: ["remotion", "docker", "cdp", "youtube", "automation", "video-generation", "whisper"] -ogImage: "/og/zh-how-i-built-an-automated-music-video-pipeline.jpg" -banner: "/banners/how-i-built-an-automated-music-video-pipeline.jpg" +title: "浏览器翻译没有 API:我给一个纯中文网页应用加上了英文模式" +description: "浏览器不提供触发翻译的 JavaScript API,而我的应用明明是中文却声明自己是英文。于是我改用反向代理,把一个英文模式注入进去。" +pubDate: 2025-06-11 +updatedDate: 2026-09-29 +category: engineering +blueprint: sme-digital-suite +blueprintLayer: L1-high-conversion-web +tags: ["i18n", "nginx", "vue", "javascript", "self-hosting"] +ogImage: "/og/zh-adding-english-mode-to-a-chinese-only-web-app.jpg" +banner: "/banners/adding-english-mode-to-a-chinese-only-web-app.jpg" draft: false --- -绝大多数 AI 音乐制作流程在生成完音频文件后都会陷入停滞。你手里有一份 MP3 文件和一段纯文本歌词。如果想把它做成能在 YouTube、Bilibili 或各大社交平台上发布的成品视频,通常需要经历漫长而繁琐的手工流程:打开剪辑软件、裁剪背景画面、逐行拉字幕调关键帧、导出几个 G 的庞大文件、守着浏览器上传、手动填写封面与标签、再手动复制粘贴字幕文件。 +「能不能用 JavaScript 触发浏览器自带的翻译?」答案是**不能**。接受这一点花了我一个下午 —— 但更有意思的发现出现在前面:我的应用其实一直在破坏浏览器的翻译功能,而且只用一个属性就做到了。 -做一首歌可能要耗费大半天,如果有几十首歌曲库,这种手工做法完全无法规模化。 +## 我在做什么 -我需要一条纯自动化的工业级管线:只需丢入音频文件与歌词,离开工位。几分钟后,一段 1080P 高清 MV 视频即渲染完成,自带跟随音乐动态律动的音频动效环、高精度三行滚动歌词、智能分镜人物避让切换,并全自动通过远程 CDP 发布到 YouTube,挂载自定义封面与 8 种语言的外挂同步字幕。 +我维护着一个私人的邀请制阅读平台 `book.hoelee.com`:自建的阅读器(Kotlin/Spring 后端 + Vue 单页应用),装着我自己的藏书,并给受邀读者开了账号。界面是纯中文 —— 不是「以中文为主」,而是**只有中文**。我在相信这一点之前把每个前端 bundle 都翻了一遍:没有 i18n 库、没有语言包、没有语言切换,只有一个简繁转换表和一堆英文 TTS 语音名。 -以下是整套系统的架构设计、核心踩坑点与线上实战方案。 +后来我开始把邀请卡发给只说英文的读者。他们能注册,然后迎面撞上:书架、浏览书仓、确定、设置 —— 每一个控件都在他们读不懂的语言里。这不是「体验不够精致」的问题。对我邀请的一半人来说,这个产品是不可用的。 -> **线上成品展示**:由本管线全自动渲染并发布上线的真实曲目为:[《糖画小仙》1080p 官方 MV(YouTube)](https://youtu.be/vZbyKeCCSRA),内置精准声学对齐与 8 语同步外挂字幕(英、日、韩、西、法、德、马、繁中)。 +有一条约束决定了后面所有做法:**我不改这个应用。** 它的上游仓库已经归档,界面是压缩过的 Vue bundle —— 任何我打进镜像里的补丁,都会在下一次换镜像时消失。所以我做的东西必须待在**外面一层**:我已经架在应用前面的那个 nginx 网关。 -> **TL;DR** —— 四段互不耦合的工序:Whisper 词级对齐 → Remotion 1080P 渲染 → Docker 归档落盘 → CDP 发布到 YouTube Studio。每一段各自失败、也各自可单独排查;最难缠的两段单独写了长文:[把大文件送进浏览器容器](/posts/zh/uploading-large-videos-to-youtube-via-cdp/) 与 [消除渲染抖动](/posts/zh/debugging-remotion-video-jitter/)。 +## 发现一:应用声明了自己错误的语言,而这会关掉翻译 ---- +写代码之前,我先看看浏览器到底看到了什么: -## 1. 端到端执行流程与耗时分解 - -为了诚实呈现系统的实际吞吐能力,而不是笼统承诺“三分钟搞定一切”,以下是针对 4 分 21 秒(261 秒)1080P 30fps 曲目的实测端到端耗时: - -| 阶段 | 处理流程与工具 | 实测耗时 | 是否真正无人值守? | -|---|---|---|---| -| **1. 歌词声学对齐** | `faster-whisper`(词级时间戳)+ 能量起始峰值锁定 | 14.2 秒 | ✅ 100% 自动 | -| **2. 构图与留白核验** | 4x2 分镜拼图生成 + 左右留白侧校验(`sides.json`) | 2.1 秒 | ⚠️ 建议 1 键人工核对 | -| **3. Remotion 1080P 渲染** | Remotion CLI(30 fps,H.264,7830 帧,i9-13900K + RTX 3060) | 218.3 秒 | ✅ 100% 自动 | -| **4. Docker 容器落盘** | Portainer Docker Archive API tar 流写入(128.85 MB 视频 + 封面) | 1.89 秒 | ✅ 100% 自动 | -| **5. YouTube 自动化发布** | Chrome CDP 自动化(注入文件、填充简介、勾选公开) | 32.4 秒 | ✅ 100% 自动 | -| **6. 多语种字幕处理** | 大模型批量翻译(8 语 × 56 句)+ CDP 自动上架发布 | 48.6 秒 | ✅ 100% 自动 | -| **全流程总计** | **从音频文件到油管 8 语公开发布** | **约 5.3 分钟** | **98% 自动化** | - -4 分钟视频的渲染阶段在我们的工作站上用时 **218.3 秒**(约 3.6 分钟),随后的入库、发布与 8 语字幕处理约 1 分多钟。在分镜留白侧确认无误后,后续全链路无需任何人工交互。 - ---- - -## 2. 整体系统架构 - -整个自动化系统分为四个完全解耦的层级: - -``` -[原始 MP3 音频 + 文本歌词] - │ - ▼ -1. 声学起唱对齐(Whisper 词级时间戳 + 声学起唱校准 -> LRC + SRT) - │ - ▼ -2. 动态 React 视频合成(Remotion 1080P 30fps) - ├─ 动态环形音频动效(320px 基础半径 -> 380px 低频弹性外扩 + 72 根跳动频谱柱) - ├─ 三行平滑滚动歌词(留白侧智能滑动与人物避让) - └─ 纯净透明度叠化背景(1.8s 贝塞尔曲线过渡,彻底杜绝抖动) - │ - ▼ -3. 容器免鉴权落盘(Portainer Docker Archive API -> 容器 /config/Downloads) - │ - ▼ -4. YouTube 自动化发布(远程 Chrome DevTools Protocol / Playwright) - ├─ 自动挂载视频与封面 cover.jpg - ├─ 自动填充标题、简介、SEO 标签与官网/业务联络方式 - ├─ 穿透发布向导,选择公开(Public)发布 - └─ 调度大模型翻译并自动挂载发布 8 国外挂字幕(英、日、韩、西、法、德、马、繁中) +```bash +$ curl -s https://book.hoelee.com/index.html | grep -o ']*>' + ``` ---- +外壳声明英文,界面 100% 中文。 -## 3. 解决歌唱声学字幕对齐难题 +这件事比看上去严重。Chrome、Edge、Safari 决定「要不要**主动提示**翻译」,很大程度取决于页面声明的语言。一个声明自己是英文、却渲染中文内容的页面,不会得到那个「要翻译此页面吗?」的提示 —— 对我的新读者来说最有用的一项无障碍功能,就这么被悄悄关掉了;而「右键 → 翻译成中文/英文」是大多数人从来不知道的操作。 -AI 歌唱音频中普遍存在拖音、转音、颤音和节奏弹性拉伸。如果直接用常规语音转写工具识别,伴奏强烈的鼓点和乐器声往往会导致整句唱词被漏掉。 +修复只要一行,而且是作用在**响应出站时**,不是打进应用里: -为了让咬字对齐落在**渲染帧**这一级,管线采用了**基于已知歌词的声学起唱对齐(Forced Acoustic Onset Alignment)**: -1. 以人工确认的标准歌词作为基准锚点; -2. 调用本地 `faster-whisper` 并开启词级时间戳(`word_timestamps=True`); -3. 针对每行歌词,锁定首字真实发音起振的能量起跑点,校准字幕起止点。 - -这样产出的 `.lrc` 与 `.srt` 即可精准对齐到**物理渲染帧**:在 30 fps 帧率下,单帧时长为 33.3 ms,这才是视频渲染真正起决定性作用的时间分辨率。当歌手唱出乐句第一个辅音的瞬间,高亮状态即在对应渲染帧切换。词级时间戳并非“微秒级精确”,这套管线也绝不作此夸大宣称——它将每行歌词对齐到声学起振点,并交由 33.3 ms 的物理帧网格做最终裁决。 - ---- - -## 4. Remotion 留白侧人物避让算法 - -在采用插画背景的 MV 中,一个常见痛点是歌词遮挡画面主角。如果某一镜小仙女画在画面右侧,歌词依然居中或靠右就会直接盖在人物脸上。 - -为了从根本上规避遮挡,管线引入了 `sides.json` 留白侧定义文件: - -```json -["left", "right", "left", "right", "left", "right", "left", "right"] +```nginx +location = /index.html { + # sub_filter 不能改写 gzip 后的内容 -> 让上游返回未压缩的 HTML + proxy_set_header Accept-Encoding ""; + sub_filter_once on; + # 应用声明 lang="en" 而界面是中文 -> 浏览器因此永远不提示中译英。 + # 说出事实。 + sub_filter '' ''; + proxy_pass http://hectorqin-reader:8080; +} ``` -在 Remotion 合成组件(`MusicVideo.tsx`)中,音频环与歌词容器由统一的动态坐标驱动: +用发现它的方式验证:`curl -s https://book.hoelee.com/index.html | grep -o ' { - const side = sides[idx] || (idx % 2 === 0 ? 'left' : 'right'); - return side === 'left' ? 540 : 1380; -}; +## 发现二:浏览器翻译没有 API + +页面老实之后,下一个问题自然是:我能不能自己提供一个按钮,做浏览器菜单里那件事?**没有这样的 API。** 浏览器翻译是 UI 层功能,网页无法调用它,也没有任何暴露给网页的接口可以脚本化它。 + +于是我逐个排除了替代方案: + +| 方案 | 为什么放弃 | +|---|---| +| 改写应用的 JS bundle 来替换文案 | 把自己绑死在压缩后的内部实现上,下次换镜像就坏。也违背了「不改应用」的原则。 | +| 用 `translate.goog` 代理整站 | 那是**另一个源(origin)**。应用的登录令牌存在 `localStorage` 里,而它是按源隔离的 —— 读者会变成未登录,Service Worker / PWA 也会坏掉;而且每个 API 调用都要绕道 Google。 | +| 教读者右键 → 翻译 | 对**书籍正文**有用,但它是一次一浏览器的手动仪式,对根本不知道有这功能的人毫无帮助。对一个我自己掌控全栈的产品来说不够。 | +| Fork 应用,正经加 i18n | 在一个已归档的代码库上做几周,而且上游一变我就得重译一遍。 | + +剩下的就是最不花哨的那条路:自己用一个词典在浏览器里翻译界面 —— 再把它包进一个只在应用内出现的按钮。 + +## 做法:词典 + 按钮,在网关层注入 + +网关本来就在提供我的前台页面,那它再多提供一个静态文件、只往应用外壳里注入一个 script 标签就行: + +```nginx +location = /gate-en.js { + root /usr/share/nginx/html; + add_header Cache-Control "public, max-age=300" always; +} + +location = /index.html { + # ... 上面的 lang 改写 ... + sub_filter '' ''; + proxy_pass http://hectorqin-reader:8080; +} ``` -在每张分镜结束前的 1.8 秒内,歌词与音频环使用贝塞尔缓动平滑横向滑移。在下一张图片淡入之前,文字已平稳就位在留白侧,右侧人物与景物 100% 毫无遮挡。 +整个部署就是这样:一个 40 KB 的脚本,只在 `/index.html` 加载,所以按钮只存在于应用内,别处都没有。脚本做三件事。 ---- +### 1. 从应用自己的 bundle 里「收割」词汇表 -## 5. 远程 YouTube CDP 上传通道 - -视频在本地 Windows 工作站上渲染,而已登录 YouTube 账号的浏览器运行在 DSM NAS 上的独立 Docker 容器(`chrome2`)中。 - -为了无需人工扫码或重新授权即可完成上架: -1. **Docker 归档落盘**:130MB 的 MP4 视频与封面图片通过 Portainer REST API 打包成 tar 流直接 PUT 写入容器文件系统(`/config/Downloads`),2 秒内完成落盘,彻底规避 CDP 协议的内存上限; -2. **CDP 节点注入**:通过 Playwright 接入 CDP(`ws://192.168.1.1:9242`),直接向 DOM 的文件输入节点注入容器路径: +界面文案本来就在压缩后的 JavaScript 里,以字符串字面量的形式存在。把所有含中日韩字符的字符串抓出来并计数: ```python -cdp.send("DOM.setFileInputFiles", { - "files": ["/config/Downloads/国风电子 糖画小仙.mp4"], - "nodeId": video_node["nodeId"], -}) +import re, collections +CJK = re.compile(r'[\u4e00-\u9fff]') +count = collections.Counter() +for bundle in bundles: + for m in re.finditer(r'"([^"\\\n]{2,40})"', bundle): + if CJK.search(m.group(1)): + count[m.group(1)] += 1 +print(len(count)) # 840 条唯一字符串,按出现频率排序 ``` -3. **多语种字幕调度**:字幕翻译自动化脚本将基准 SRT 提取为纯文本行数组,通过本地 OmniRoute 大模型网关翻译为 8 种语言,并调用 CDP 遍历 YouTube Studio 后台,自动创建语言并发布所有外挂字幕轨。 +跑出 840 条唯一字符串:菜单(书架、书源管理)、确认框(确认要删除所选择的书籍吗?)、设置项(段落行距)、错误信息(本地书籍源文件不存在)。词典最终是 **871 条** —— 收割来的,加上我在英文模式下实际使用应用、看还有什么没翻而补上的。 + +### 2. 按 DOM 的真实样子匹配,而不是你希望的样子 + +朴素的子串替换会毁掉中文词组:书架 就藏在 加入书架 里面,单字更糟(页 藏在 页面 里)。三条规则解决了它: + +- **先做整标签匹配,基于去掉空白后的文本。** 应用里的标签是 `" 书架 "` 这种带空格的,所以词典里的 `设置` 必须能匹配 DOM 里的 `" 设置 "`。 +- **然后是最长优先的子串替换**,只允许长度 ≥ 2 的键参与。 +- **单字键只做整标签匹配**(章、页、条),这样它们永远不可能破坏更长的词。 + +```js +function tr(s) { + if (!s || !CJK.test(s)) return s; + var hit = EXACT[s.replace(/[\s\u3000]/g, '')]; // 整标签匹配,空白归一化 + if (hit) return keepWS(s, hit); + var t = s; + for (var i = 0; i < KEYS.length; i++) { // KEYS:最长优先,长度 >= 2 + if (t.indexOf(KEYS[i]) >= 0) t = t.split(KEYS[i]).join(DICT[KEYS[i]]); + } + // 收拾数量短语:共58个可用书源 -> "58 usable sources" + return t.replace(/共\s*(\d+)/g, '$1').replace(/(\d+)\s*个/g, '$1').replace(/(\d)([A-Za-z])/g, '$1 $2'); +} +``` + +### 3. 跟上框架的重绘 + +它是 Vue 应用,DOM 一直在被重写 —— 只跑一次的翻译,一秒后就有一半是错的。一个带 150 ms 防抖的 `MutationObserver`,在任何变化之后重跑一遍遍历。 + +这一遍是幂等的:标签一旦变成英文就不再含中日韩字符,于是不会被再写一次,也不会形成写入循环。文本节点之外,`placeholder`、`title`、`aria-label`、`alt` 和输入框的 `value` 也会翻译 —— 所以提示气泡和搜索框也一起变英文了。 + +UI 是右下角的一个小胶囊:关闭时显示 `EN`,开启后显示 `中文`,选择记在 `localStorage`,还提供「隐藏此按钮」给更愿意用自己翻译器的读者。 + +## 它**刻意**不翻译什么 + +**书籍正文。** 阅读器把 EPUB 内容渲染在 iframe 里,我的遍历器按设计跳过 iframe —— 而且我无论如何也不会去机器替换一本小说的正文。 + +同样保持原样的还有:书名与作者、书源名称、我自己建的分组名(梯子、精品、正版),以及应用那句诗。这些是**数据**,不是界面。翻译界面是一种体贴;悄悄改写用户自己的内容是另一回事,而且糟糕得多。 + +书籍正文仍然交给浏览器自带的翻译 —— 而且因为「发现一」,它现在真的会被提示出来。按钮的弹窗里就是这么写的。 + +## 要量,不要用眼睛看 + +「截图看起来基本是英文了」不算结果。我在开关按钮的前后,数了实时 DOM 里含中日韩字符的文本节点: + +```js +let cjk = 0, total = 0; // 遍历器跳过注入的 UI 与 script/style +while ((n = walker.nextNode())) { + const t = n.nodeValue.trim(); + if (!t) continue; + total++; + if (/[\u4e00-\u9fff]/.test(t)) cjk++; +} +``` + +| 界面 | 之前 | 之后 | 已翻译 | +|---|---|---|---| +| 登录 / 注册对话框 | 80 | 12 | 85% | +| 书架 + 设置 | 259 | 60 | 77% | +| 阅读页界面 | 214 | 31 | 88% | + +那些界面里仍被计为中文的,全部是数据 —— 书名、作者、书源名、我自己的分组名、那句诗 —— 正是应该保持中文的部分。 + +## 我会怎么做得不一样 + +**不要用手工重写一份 871 条的词典。** 我重构过一次,静默丢了 25 个键。症状看起来像逻辑 bug:设置 还是中文,而它的每个兄弟标签都翻译正常。匹配器没问题 —— 是**条目**不见了。在你相信一次重写之前,用程序对比键集合: + +```bash +python -c "print(sorted(set(old_keys) - set(new_keys)))" +``` + +**自动化 UI 检查要写短。** 我的无头运行器每次调用硬超时 60 秒。一个遍历六本书的循环返回 HTTP 408 和空 body,而我的 JSON 解析器报的是「解析错误」而不是「超时」—— 我一度在查错的东西。 + +**调试之前先搞清楚 SPA 的真实路由。** `#/reader/1` 会渲染出一个空白外壳,让人以为应用坏了;真正的路由是 `#/reader?bookUrl=`。 + +**注意你自己的跳转逻辑。** 我的网关会把任何刷新送回前台页,除非设置了 `sessionStorage` 的通行标记。把界面切**回**中文会重载页面 —— 所以按钮会先设好同一个标记,否则读者会因为改了个语言设置被踢出应用。 + +## 结果 + +一个只懂英文的读者,现在打开平台、点一下按钮,就能用英文读界面。应用本身没有被改动:整个功能只是一个由 sidecar nginx 容器提供的 40 KB 脚本,所以下一次换镜像不可能弄坏它;而浏览器自带的翻译,终于有一个会声明自己真实语言的页面可以配合。 + +我留下的经验:**当平台不给你 API 时,往外看一层。** 反向代理是添加「应用里加不了的行为」的正当位置 —— 而且通常是唯一能在升级中存活的位置。 --- -## 6. 经验与教训 - -- **渲染与发布必须解耦**:早期版本尝试过一边渲染一边推流或直接向浏览器传 buffer,编码器稍有波动就会导致浏览器连接重置。先在磁盘写出完整 MP4 再通过归档传输,两道环节互不牵连,容错率极高。 -- **合成前必须进行九宫格审查**:正式渲染全片前,先生成一张全分镜缩略拼图快速校验,肉眼确认每一张图的留白侧标记完全吻合,避免全片渲染数分钟后才发现有一处文字挡了人物。 - ---- - -## 想为你的业务打造类似的系统吗? - -我专注于自动化音视频制作管线、AI 音乐与多媒体创作流,以及基于 Docker / NAS 的自托管自动化基础设施搭建。 - -无论你是需要自动化媒体制作系统,还是需要定制全栈软件工具: -- **WhatsApp**: [直接在 WhatsApp 上联系我](https://wa.me/60127972969) -- **邮箱**: [me@hoelee.com](mailto:me@hoelee.com?subject=自动化视频管线合作咨询) -- **官网**: [hoelee.com](https://www.hoelee.com/) +*我为马来西亚的中小企业搭建并自托管这类平台 —— 网页应用、私人书库、内部工具。有需要可以找我:[WhatsApp +60 12-797 2969](https://wa.me/60127972969)、[me@hoelee.com](mailto:me@hoelee.com?subject=Self-hosted%20platform),或 [hoelee.com](https://www.hoelee.com)。* ```` ### 🇬🇧 英文版原文 (English Markdown Primary) ````markdown --- -title: "How I Built an Automated Music Video Pipeline with Remotion and Dockerized YouTube CDP" -description: 'Explore an automated music video workflow using Remotion, Whisper, Docker and Chrome DevTools Protocol, from lyric alignment and rendering to YouTube publishing and multilingual subtitles.' -pubDate: 2025-04-18 -updatedDate: 2026-10-22 -category: case-studies -blueprint: ai-agent-automation -blueprintLayer: L4-aigc-media-pipelines -stream: creative-lab -tags: ["remotion", "docker", "cdp", "youtube", "automation", "video-generation", "whisper"] -ogImage: "/og/how-i-built-an-automated-music-video-pipeline.jpg" -banner: "/banners/how-i-built-an-automated-music-video-pipeline.jpg" +title: "No API for Browser Translation: Adding an English Mode to a Chinese-Only Web App" +description: "Browsers expose no JavaScript API to trigger translation — and my app declared English while being Chinese. How I injected an English mode at the reverse proxy." +pubDate: 2025-06-11 +updatedDate: 2026-09-29 +category: engineering +blueprint: sme-digital-suite +blueprintLayer: L1-high-conversion-web +tags: ["i18n", "nginx", "vue", "javascript", "self-hosting"] +ogImage: "/og/adding-english-mode-to-a-chinese-only-web-app.jpg" +banner: "/banners/adding-english-mode-to-a-chinese-only-web-app.jpg" draft: false --- -Most AI music workflows hit a brick wall right after the track is generated. You have an MP3 file and maybe a text block of lyrics. If you want to turn that into a finished YouTube video, you typically enter a tedious manual loop: opening a video editor, cutting background footage, adjusting rolling captions line by line, setting keyframes, exporting a multi-gigabyte render, waiting for YouTube Studio to upload, typing descriptions, and pasting subtitle tracks. +Can I trigger the browser's translate feature from JavaScript? No. That answer cost me an afternoon to accept — but the more interesting discovery came first: my app was already sabotaging the browser's translation, and it did it with a single attribute. -Doing that once takes an afternoon. Doing that across dozens of tracks is unsustainable. +## What I was working on -I wanted a single automated pipeline: give it an audio file and lyrics, and walk away. A few minutes later, a 1080p music video is rendered with dynamic audio-reactive visuals, synchronized three-line rolling lyrics, smart scene cut alternation, and automatically published to YouTube with 8-language external subtitles. +I run a private, invite-only reading platform at `book.hoelee.com`: a self-hosted reader (Kotlin/Spring backend, Vue single-page app) holding my library, with accounts for invited readers. The interface is Chinese-only. Not "Chinese-first" — *only*. I grepped every served bundle before believing it: no i18n library, no locale files, no language switcher. Just a Simplified/Traditional character converter and a list of English TTS voice names. -Here is the exact architecture, the failure points I had to solve, and the code running this in production. +Then I started handing out invite cards to English-speaking readers. They could register, and then they met 书架, 浏览书仓, 确定, 设置 — every control in a language they don't read. That isn't a polish problem. For half the people I invited, the product was unusable. -> **Live Production Proof**: The full benchmark artifact rendered and published by this exact pipeline is live on YouTube: [《糖画小仙》 (Sugar Art Fairy) 1080p MV](https://youtu.be/vZbyKeCCSRA), featuring 8-language CC subtitle tracks (EN, JA, KO, ES, FR, DE, MS, ZH-Hant). +One constraint shaped everything that follows: **I don't modify the app.** Its upstream repository is archived and the UI is a minified Vue bundle, so anything I patch inside the image dies the next time I swap the image. Whatever I build has to live one layer out — in the nginx gateway I already run in front of the app. -> **TL;DR** — Four decoupled stages: Whisper word-level alignment → Remotion 1080p render → Docker Archive staging → CDP upload to YouTube Studio. Each fails on its own, so each can be debugged on its own; the two that fought back hardest have their own write-ups: [staging large files into the browser container](/posts/uploading-large-videos-to-youtube-via-cdp/) and [killing render jitter](/posts/debugging-remotion-video-jitter/). +## Finding 1: the app declares the wrong language, and that disables translation ---- +Before writing a single line of code, I looked at what the browser sees: -## 1. End-to-End Execution & Timing Breakdown - -To be completely honest about performance rather than promising a vague "three minutes": here is the measured benchmark run for our 4-minute 21-second (261s) 1080p 30fps song: - -| Stage | Process & Tools | Measured Time | Unattended? | -|---|---|---|---| -| **1. Audio-Lyric Alignment** | `faster-whisper` (word timestamps) + acoustic onset peak extraction | 14.2 s | ✅ 100% Autonomous | -| **2. Layout Verification** | 4x2 montage thumbnail generation + negative space side audit | 2.1 s | ⚠️ 1-click human check | -| **3. Remotion 1080p Render** | Remotion CLI (30 fps, H.264, 7,830 frames, i9-13900K + RTX 3060) | 218.3 s | ✅ 100% Autonomous | -| **4. Docker Staging** | Portainer Docker Archive API tar stream (128.85 MB MP4 + cover) | 1.89 s | ✅ 100% Autonomous | -| **5. YouTube Publication** | Headless Chrome CDP (Studio upload dialog, public visibility) | 32.4 s | ✅ 100% Autonomous | -| **6. Multilingual Subtitles** | LLM translation (8 languages × 56 cues) + CDP Studio upload | 48.6 s | ✅ 100% Autonomous | -| **Total Pipeline Cycle** | **From raw audio to public YouTube video with 8 languages** | **~5.3 minutes** | **98% Autonomous** | - -The actual rendering of a 4-minute video takes **218.3 seconds** (under 3.7 minutes) on our workstation, and the upload/subtitle publication pipeline finishes in another minute. Once the storyboard layout is confirmed, the entire sequence runs with zero human intervention. - ---- - -## 2. System Architecture - -The pipeline consists of four decoupled layers: - -``` -[Audio MP3 + Lyrics] - │ - ▼ -1. Acoustic Onset Alignment (Whisper word-level timestamping -> LRC + SRT) - │ - ▼ -2. Dynamic React Video Synthesis (Remotion 1080p 30fps) - ├─ Dynamic Circular Visualizer (320px base -> 380px bass expansion + 72 radial bars) - ├─ 3-Line Smooth Scrolling Lyrics (Negative space side-tracking) - └─ Pure-Opacity Image Slideshow (no per-frame scaling · 1.8s Bézier crossfade) - │ - ▼ -3. Headless Docker Staging (Portainer Docker Archive API -> Container /config/Downloads) - │ - ▼ -4. Autonomous YouTube Publication (Remote Chrome DevTools Protocol / Playwright) - ├─ Attach video & custom thumbnail - ├─ Fill SEO metadata, biography, & service contact links - ├─ Step through upload wizard & set visibility to Public - └─ Upload & publish 8 multilingual subtitle tracks (EN, JA, KO, ES, FR, DE, MS, ZH-TW) +```bash +$ curl -s https://book.hoelee.com/index.html | grep -o ']*>' + ``` ---- +The shell says English. The interface is 100% Chinese. -## 3. Solving Acoustic Lyric Alignment +That matters more than it looks. Chrome, Edge and Safari decide whether to *offer* translation largely from the page's declared language. A page that declares English while rendering Chinese text doesn't get the "Translate this page?" prompt — the single most useful accessibility feature for my new readers was silently switched off, and right-click → *Translate to English* is a ritual most people never learn. -AI music generators stretch vowels, add melodic vibrato, and change tempo dynamically. Standard text-to-speech alignment or plain transcription drops entire verses when instruments overpower vocals. +The fix is one line, applied to the response on the way out instead of inside the app: -Instead of blind speech recognition, I use **forced acoustic onset alignment**: -1. Take the ground-truth lyrics written beforehand. -2. Run `faster-whisper` on the vocal track with word-level timestamps (`word_timestamps=True`). -3. For each lyric line, find the closest acoustic voice activity onset for the starting character and lock the timestamp. - -That yields a `.lrc` and `.srt` pair aligned to the **rendered frame**: at 30 fps one frame is 33.3 ms, so a frame is the resolution that actually matters here. When the singer lands the first consonant of a phrase, the highlight switches on that frame. Word-level timestamps are not microsecond-accurate, and the pipeline does not claim to be — it lines each line up with the acoustic onset and lets the frame grid decide. - ---- - -## 4. Negative-Space Side Alternation in Remotion - -In music videos with illustrated backgrounds, a central problem is character obstruction. If a slide shows a character standing on the right, centering the lyrics or locking them to the right covers the artwork. - -To solve this deterministically, the pipeline extracts a `sides.json` layout manifest: - -```json -["left", "right", "left", "right", "left", "right", "left", "right"] +```nginx +location = /index.html { + # sub_filter cannot rewrite a gzipped body -> ask upstream for plain HTML + proxy_set_header Accept-Encoding ""; + sub_filter_once on; + # the app declares lang="en" while its UI is Chinese -> browsers then never + # offer Chinese -> English translation. Declare the truth. + sub_filter '' ''; + proxy_pass http://hectorqin-reader:8080; +} ``` -In Remotion (`MusicVideo.tsx`), the visualizer ring and lyric stack share a single animated X coordinate: +Verify the same way you found it: `curl -s https://book.hoelee.com/index.html | grep -o ' { - const side = sides[idx] || (idx % 2 === 0 ? 'left' : 'right'); - return side === 'left' ? 540 : 1380; -}; +## Finding 2: there is no browser-translate API + +With the page honest again, the obvious next question was whether I could *offer* translation myself — a button that does what the browser's menu item does. There is no such API. Browser translation is a UI-level feature; a page cannot invoke it, and no web-exposed surface exists to script it. + +So I worked through the alternatives and rejected them one by one: + +| Approach | Why I dropped it | +|---|---| +| Rewrite the app's JS bundles to swap strings | Couples me to minified internals; the next image breaks it. Also violates the no-modify rule in spirit. | +| Proxy the site through `translate.goog` | It's a **different origin**. The app keeps its auth token in `localStorage`, which is per-origin — readers would land logged out, and the service worker/PWA breaks. Every API call would also round-trip through Google. | +| Tell readers to right-click → Translate | Works for *book text*, but it's a per-visit, per-browser ritual that does nothing for someone who doesn't know it exists. Not good enough for a product where I control the whole stack. | +| Fork the app and add i18n properly | Weeks of work on an archived codebase, for an interface I'd have to re-translate on every upstream change. | + +What's left is the unglamorous one: translate the interface myself, in the browser, with a dictionary — and wrap it in a button that only exists inside the app. + +## The fix: a dictionary and a button, injected at the gateway + +The gateway already serves my front page, so it can serve one more static file and inject a script tag into the app shell only: + +```nginx +location = /gate-en.js { + root /usr/share/nginx/html; + add_header Cache-Control "public, max-age=300" always; +} + +location = /index.html { + # ... the lang rewrite above ... + sub_filter '' ''; + proxy_pass http://hectorqin-reader:8080; +} ``` -During the last 1.8 seconds of a scene, the HUD coordinates glide across the canvas using a smooth cubic Bézier curve (`Easing.bezier(0.42, 0, 0.58, 1)`). The lyrics drift gracefully into the empty negative space before the new illustration fades in, guaranteeing zero overlap with focal characters. +That's the whole deployment: a 40 KB script, loaded only on `/index.html`, so the button exists inside the app and nowhere else. The script does three things. ---- +### 1. Harvest the vocabulary from the app's own bundles -## 5. The Remote YouTube Upload Bridge - -Rendering happens on a Windows workstation, while my authenticated YouTube Studio session runs inside an isolated Docker container (`chrome2`) on a Synology NAS. - -To upload without browser authentication prompts: -1. **Docker Archive Staging**: The ~129 MB MP4 video and cover image package is tar-streamed directly into the container filesystem via Portainer's REST API (`PUT /api/endpoints/2/docker/containers/{id}/archive?path=/config/Downloads`). It bypasses CDP byte limits and writes in under 2 seconds. The full pattern — code, tar caveats, and what it does not solve — is in [Uploading 100MB+ Videos to YouTube via CDP](/posts/uploading-large-videos-to-youtube-via-cdp/). -2. **CDP File Injection**: Using Playwright over CDP (`ws://192.168.1.1:9242`), the upload file input receives the local container path: +The UI strings are already in the minified JavaScript as quoted literals. Extract every one that contains a CJK character and count them: ```python -cdp.send("DOM.setFileInputFiles", { - "files": ["/config/Downloads/国风电子 糖画小仙.mp4"], - "nodeId": video_node["nodeId"], -}) +import re, collections +CJK = re.compile(r'[\u4e00-\u9fff]') +count = collections.Counter() +for bundle in bundles: + for m in re.finditer(r'"([^"\\\n]{2,40})"', bundle): + if CJK.search(m.group(1)): + count[m.group(1)] += 1 +print(len(count)) # 840 unique strings, most frequent first ``` -3. **Multilingual Subtitle Dispatch**: An automated worker splits the calibrated SRT into an array of lines, translates them into 8 languages (English, Japanese, Korean, Spanish, French, German, Malay, Traditional Chinese) via our local OmniRoute LLM gateway, and uploads each `.srt` file through the Studio Translations interface. +840 unique strings came out: menus (书架, 书源管理), confirmations (确认要删除所选择的书籍吗?), settings labels (段落行距), error messages (本地书籍源文件不存在). The dictionary ended up at **871 entries** — the harvest plus everything I found by running the app in English mode and watching what stayed Chinese. + +### 2. Match like the DOM actually is, not like you wish it were + +Naive substring replacement mangles Chinese compounds: 书架 sits inside 加入书架, and single characters are worse (页 inside 页面). Three rules fixed it: + +- **Whole-label match first, on whitespace-stripped text.** The app stores labels like `" 书架 "` with padding, so a dictionary key of `设置` has to match `" 设置 "` in the DOM. +- **Then longest-first substring replacement**, restricted to keys of two or more characters. +- **One-character keys are whole-label only** (章, 页, 条), so they can never corrupt a longer word. + +```js +function tr(s) { + if (!s || !CJK.test(s)) return s; + var hit = EXACT[s.replace(/[\s\u3000]/g, '')]; // whole label, whitespace-normalised + if (hit) return keepWS(s, hit); + var t = s; + for (var i = 0; i < KEYS.length; i++) { // KEYS: longest-first, length >= 2 + if (t.indexOf(KEYS[i]) >= 0) t = t.split(KEYS[i]).join(DICT[KEYS[i]]); + } + // tidy count phrases: 共58个可用书源 -> "58 usable sources" + return t.replace(/共\s*(\d+)/g, '$1').replace(/(\d+)\s*个/g, '$1').replace(/(\d)([A-Za-z])/g, '$1 $2'); +} +``` + +### 3. Keep up with the framework + +It's a Vue app, so the DOM is rewritten constantly — a translation pass that runs once is a translation pass that's half wrong a second later. A `MutationObserver` with a 150 ms debounce re-runs the walk after any change. + +The pass is idempotent: once a label is English it contains no CJK, so nothing is rewritten and there is no write loop. Text nodes plus `placeholder`, `title`, `aria-label`, `alt` and input `value` attributes all get translated, which is why tooltips and the search box come along too. + +The UI is a small pill in the bottom-right corner: `EN` when off, `中文` when on, remembered in `localStorage`, with a "Hide this button" link for readers who prefer their own translator. + +## What it deliberately does not translate + +**Book text.** The reader renders EPUB content inside an iframe; my walker skips iframes by design, and I would never machine-swap a novel's prose anyway. + +Also left alone: book titles and authors, book-source names, and the group names I created myself (梯子, 精品, 正版), plus the app's poem tagline. Those are *data*, not interface. Translating the chrome is a courtesy; silently rewriting a user's own content is a different and much worse idea. + +For book text the browser's own translation is still the right tool — and it now actually gets offered, thanks to Finding 1. The button's popup says exactly that. + +## Measure it, don't eyeball it + +"Screenshots look mostly English" is not a result. I counted CJK-containing text nodes in the live DOM before and after toggling the button: + +```js +let cjk = 0, total = 0; // walker skips the injected UI + script/style +while ((n = walker.nextNode())) { + const t = n.nodeValue.trim(); + if (!t) continue; + total++; + if (/[\u4e00-\u9fff]/.test(t)) cjk++; +} +``` + +| Screen | Before | After | Translated | +|---|---|---|---| +| Login / register dialog | 80 | 12 | 85% | +| Bookshelf + settings | 259 | 60 | 77% | +| Reading page chrome | 214 | 31 | 88% | + +Everything still counted as Chinese on those screens is data — titles, authors, source names, my own group names, the tagline — which is exactly what should stay Chinese. + +## What I'd do differently + +**Don't rewrite an 871-entry dictionary by hand.** I restructured mine once and silently dropped 25 keys. The symptom looked like a logic bug: 设置 stayed Chinese while every sibling label translated correctly. The matcher was fine — the *entry* was gone. Diff the key sets programmatically before you trust a rewrite: + +```bash +python -c "print(sorted(set(old_keys) - set(new_keys)))" +``` + +**Keep automated UI checks short.** My headless runner hard-times-out at 60 seconds per call. A six-book loop came back HTTP 408 with an empty body, and my JSON parser reported a parse error rather than a timeout — I debugged the wrong thing for a while. + +**Learn the SPA's real routes before debugging it.** `#/reader/1` renders a blank shell and makes you think the app is broken; the actual route is `#/reader?bookUrl=`. + +**Mind your own bounce logic.** My gateway sends any reload back to the front page unless a `sessionStorage` pass flag is set. Switching the interface *back* to Chinese reloads the page — so the button sets that same flag first, or the reader gets thrown out of the app for changing a language setting. + +## Result + +An English-only reader now opens the platform, taps one button, and reads the interface in English. The app itself is untouched: the whole feature is a 40 KB script served by a sidecar nginx container, so the next image update cannot break it, and the browser's own translation has a page that declares its real language to work with. + +The lesson I'm keeping: **when the platform won't give you an API, look one layer out.** A reverse proxy is a legitimate place to add behaviour you can't add in the app — and it's usually the only place that survives upgrades. --- -## 6. What I Would Do Differently +*I build and self-host platforms like this one — web apps, private libraries, internal tools for small businesses in Malaysia. If you need something similar: [WhatsApp +60 12-797 2969](https://wa.me/60127972969), [me@hoelee.com](mailto:me@hoelee.com?subject=Self-hosted%20platform), or [hoelee.com](https://www.hoelee.com).* +```` -- **Decouple rendering from upload**: Early revisions tried to pipe ffmpeg buffers directly into a browser tab. Encoding hiccups would freeze the browser socket. Saving the MP4 locally first and staging via Docker Archive completely isolated the two failure domains. -- **Strict layout verification**: Always generate a 4x2 montage thumbnail of all storyboard scenes before starting full rendering. Visually asserting the negative space of each slide prevents re-rendering four-minute videos due to a single misplaced side flag. +--- + +## 文章 05 / 11:排查 Remotion 视频抽搐:为什么在背景图挂载音频低频 Scale 会毁了转场 +- **Slug**: `debugging-remotion-video-jitter` +- **英文标题**: Debugging Remotion Video Jitter: Why Bass Scaling Ruins Background Transitions +- **分类**: `engineering` | **发布日期**: 2025-11-26 +- **线上地址**: https://blog.hoelee.com/posts/debugging-remotion-video-jitter/ (EN) | https://blog.hoelee.com/posts/zh/debugging-remotion-video-jitter/ (ZH) +- **本篇审核状态**: ⏳ 待审核 (Pending) + +### ✍️ 顾问审阅意见区 (Review Feedback) +```text +【审阅结论】:[ ] 通过 [ ] 需微调 [ ] 需重构 +【事实与数字核对】: +【人设与用词卫生】: +【CRO与转化建议】: +``` + +### 🇨🇳 中文版原文 (Chinese Markdown Twin) +````markdown +--- +title: "排查 Remotion 视频抽搐:为什么在背景图挂载音频低频 Scale 会毁了转场" +description: "深入排查 Remotion 音频可视化中背景画面剧烈抽搐与切镜暗闪的根本原因:解耦 HUD 与背景图层,以及纯净单向叠化与硬件基准实测对比。" +pubDate: 2025-11-26 +updatedDate: 2026-10-22 +category: engineering +stream: creative-lab +tags: ["remotion", "react", "audio-visualizer", "css-animation", "debugging"] +ogImage: "/og/zh-debugging-remotion-video-jitter.jpg" +banner: "/banners/debugging-remotion-video-jitter.jpg" +draft: false +--- + +使用 React 和 Remotion 编写自动化视频赋予了开发者对每一帧画面的极致掌控。你可以直接将 Web Audio 的频域分析数据映射到 SVG 属性、CSS 变换矩阵以及排版样式上,这种灵活性是传统非线性剪辑软件很难比拟的。 + +然而,将视频帧当作常规网页来渲染时,极易踩入微妙的视觉陷阱。在我们的自动化音乐视频生成管线中,一个在开发组件预览时看似完全正常的设计,在全量导出为 1080P 30fps 视频后,却出现了极其刺眼、让人目眩的“地震式剧烈抽搐”。 + +本文记录这次视觉 BUG 的深度排查过程:为什么对音频动效的朴素直觉会毁了整部视频,以及如何重构 React 视频图层架构以实现丝滑稳定的影视级转场。 + +> **线上成品验证**:修复后的渲染流水线已正式交付并发布至 YouTube:[《糖画小仙》1080p 官方 MV](https://youtu.be/vZbyKeCCSRA),全片背景稳固、转场平滑。 + +> **TL;DR** —— 让整幅背景图跟着低频段缩放,等于每一帧都重新做图像插值采样,肉眼读到的就是剧烈抖动与震颤;再加上两层半透明图层对称交叉淡化,在时间中点必然透出底层的黑色容器。修复方案:背景画面绝对静态,将所有音频动效彻底隔离在顶层 HUD 频谱层,转场采用「100% 实体底图 + 顶层单向淡入」。渲染耗时实测**缩短 38.0%(从 352 秒降至 218 秒,吞吐速度提升 1.61 倍)**。这一篇是 [全自动 MV 生成流水线](/posts/zh/how-i-built-an-automated-music-video-pipeline/) 的渲染篇;文件落盘与发布那一半见 [Docker 容器大文件上传](/posts/zh/uploading-large-videos-to-youtube-via-cdp/)。 + +--- + +## 1. 故障现象:全画面的“高频地震” + +该视频合成组件最初的视觉设想非常清晰: +1. 底层是一组 1080P 高清插画分镜,在不同章节间慢速淡入淡出(Crossfade); +2. 画面上方浮动一个跟随音乐重音律动的环形音频频谱仪; +3. 三行书法歌词居中显示在音频环内,并根据每张插画的主体位置在画面左右两侧平滑滑移避让。 + +在开发工具里单帧点按检查时,画面一切正常。但在以 30fps 正常播放渲染出的成片时,肉眼观感极度难受:每当强烈的电子重音鼓点响起,整张背景图每秒剧烈颤抖震荡数十次;不仅如此,在两张图片交替切镜的 1.8 秒窗口内,画面中央还会突然闪现明显的“暗黑闪烁”,亮度瞬间下沉。 + +--- + +## 2. 根因分析一:低频 Scale 能量污染背景图 + +翻阅最初的 `BackgroundSlideshow.tsx` 组件,我们发现了一段看似合情合理的动效代码: + +```tsx +// 埋下灾难的罪魁祸首代码 +const bassScale = interpolate(bassLevel, [0, 1], [1.0, 1.025], { + extrapolateRight: 'clamp', +}); + +// 应用在全屏背景图片上: + +``` + +在直觉中,重低音响起时让背景图随着节奏轻微放大 2.5%,听起来像是在模拟动感的镜头呼吸感。然而在真实物理渲染中: +- **电音鼓点的瞬态脉冲只有 30~60 毫秒**:低频能量在连续 2 到 3 帧之内瞬间从零冲上顶峰,又在下一帧骤跌回原点; +- **1920x1080 的巨幅背景图放大 2.5%,边缘物理像素瞬间位移多达 25 到 30 像素**; +- 这种在相邻帧之间急剧拉扯的剧烈像素位移,人眼接收到的根本不是“音乐律动”,而是被视网膜本能判定为**强烈的镜头颠簸、抽动与丢帧残影**; +- 更致命的是,在 1.8 秒的交叉淡入淡出期间,两张重叠的图片各自具有不同的锚点和缩放系数,两层图像以不同的相位同时高频抖动,产生了极其混乱的重影眩晕感。 + +--- + +## 3. 根因分析二:对称线性透明度重叠时的“黑闪”陷阱 + +第二个问题是切镜时的亮度下陷黑闪。在传统 Web CSS 中,人们通常很自然地写出: + +```css +/* 常见的渐变叠化陷阱 */ +.current-slide { opacity: 1 -> 0; } +.next-slide { opacity: 0 -> 1; } +``` + +当两张图片都以线性渐变进行透明度变换时,在正中间那颗时间点($t = 0.5$): +- 当前图片透明度 = 0.5 +- 下一张图片透明度 = 0.5 + +来看浏览器和 Remotion 的合成器是如何将这几层叠在底层的纯黑容器之上的: + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Layer 2: 下一张图片(Next Slide,透明度 = 0.5) │ +├─────────────────────────────────────────────────────────────┤ +│ Layer 1: 当前图片(Current Slide,透明度 = 0.5) │ +├─────────────────────────────────────────────────────────────┤ +│ Layer 0: 根容器底色(Root Container,背景色 = #000000 纯黑) │ +└─────────────────────────────────────────────────────────────┘ +``` + +由于多层半透明通道是依次向下进行乘法 Alpha 混合: +$$\text{有效综合不透明度} = 1 - (1 - 0.5) \times (1 - 0.5) = 1 - 0.25 = 0.75$$ + +合成后的总有效透明度只有 **0.75**,这意味着多达 **25% 的底层纯黑背景**被暴露了出来!这不是错觉,而是纯粹的数学光度衰减。每一次转场切镜,画面整体照度都会凭空跌掉四分之一,造成非常明显的瞬间暗淡闪烁。 + +--- + +## 4. 重构方案:稳固基座 + 纯净单向叠化 + +找到症结后,我们对组件进行了彻底重构: + +### 1. 彻底剥离背景图层的所有音频缩放 +背景插画必须稳如磐石。所有的音频跳动、瞬态光斑与能量爆发,**只能且必须限定在顶层的 HUD 频谱仪中**。背景层坚决不承载任何高频振荡属性。 + +### 2. 底层 100% 实体 + 顶层单向平滑淡入 +坚决不要让底图淡出。**当前图片始终保持 100% 不透明**,下一张图片直接覆盖在其上方,慢速执行 `0 -> 1.0` 的单向淡入: + +```tsx +return ( +
+ {/* 1. 底层图片:坚固不透明,彻底消除暗黑闪烁 */} + + + {/* 2. 顶层图片:在 1.8 秒内平滑淡入 (0 -> 1.0) */} + {isTransitioning && ( + + )} +
+); +``` + +### 3. HUD 与背景物理层级绝对隔离 +歌词弹幕与音频环容器设置在独立的顶层图层中,并赋予 `pointerEvents: 'none'`。歌词在留白侧之间的横向位移完全根据当前时间戳自主计算,与底层的图片渐变完全物理隔离,互不干涉。 + +--- + +## 5. 改造后的成效与基准复现 + +重构落地后,对比极其显著: + +1. **彻底消除画面抖动**:无论音乐鼓点多么狂暴密集,背景画面在所有测试分镜中纹丝不动,人物与细节清晰稳定; +2. **转场不再发生黑闪**:1.8 秒的单向贝塞尔叠化始终保持 100% 连续亮度,杜绝了中间态透出底层黑色的缺陷; +3. **渲染吞吐速度大幅提升**:移除了对超大尺寸背景图的高频矩阵重算后,渲染性能获得了实测提升。 + +### 基准实测与硬件复现条件 + +| 评估指标 | 优化前版本 | 优化后版本 | 变化幅度与提升 | +|---|---|---|---| +| **渲染总耗时** | 352.0 秒 | 218.3 秒 | **耗时减少 38.0% (-133.7s)** | +| **平均渲染帧率** | 22.24 fps | 35.87 fps | **吞吐速度提升 1.61 倍** | +| **总计处理帧数** | 7,830 帧 | 7,830 帧 | 完全一致(261 秒 / 4 分 21 秒) | +| **输出视频规格** | 1920 × 1080 (1080p @ 30fps) | 1920 × 1080 (1080p @ 30fps) | 完全一致 | +| **编码器设置** | H.264 (CRF 18, preset medium) | H.264 (CRF 18, preset medium) | 完全一致 | + +**基准测试宿主环境:** +- **CPU**:Intel Core i9-13900K(24 核 / 32 线程,最高睿频 5.8 GHz) +- **GPU**:NVIDIA GeForce RTX 3060(12GB GDDR6 显存,驱动版本 551.xx) +- **内存**:64GB DDR5 5600 MHz +- **系统**:Windows 11 专业版 64 位 +- **软件栈**:Node.js v20.x,Remotion v4.0.x,开启 Chromium 无头 GPU 加速 + +两次基准测试均在同一工作站、连续以完全相同的音频频谱输入和插画素材进行。渲染耗时缩短 38.0%,即吞吐速度达到优化前的 **1.615 倍**,使原本慢于真实播放时长的渲染变为高于实时帧率的高效输出。 + +--- + +## 本专题关联阅读 +- **系统总架构篇**:[我是如何用 Remotion 与 Docker CDP 打造全自动 MV 生成与多语种油管发布流的](/posts/zh/how-i-built-an-automated-music-video-pipeline/) +- **大文件上传篇**:[借助 Docker Archive API 模式通过 CDP 自动化上传 100MB+ 视频到 YouTube](/posts/zh/uploading-large-videos-to-youtube-via-cdp/) +- **线上实成片**:[前往 YouTube 观看《糖画小仙》1080P 官方完整成片](https://youtu.be/vZbyKeCCSRA) +```` + +### 🇬🇧 英文版原文 (English Markdown Primary) +````markdown +--- +title: "Debugging Remotion Video Jitter: Why Bass Scaling Ruins Background Transitions" +description: "Diagnose audio-reactive background jitter and crossfade flicker in Remotion. See how separating the HUD from background layers can improve visual stability and rendering performance." +pubDate: 2025-11-26 +updatedDate: 2026-10-22 +category: engineering +stream: creative-lab +tags: ["remotion", "react", "audio-visualizer", "css-animation", "debugging"] +ogImage: "/og/debugging-remotion-video-jitter.jpg" +banner: "/banners/debugging-remotion-video-jitter.jpg" +draft: false +--- + +Building programmatic videos with React and Remotion gives you complete DOM control over every frame. You can map web audio frequencies directly to SVG attributes, CSS transforms, and typography in ways standard video editors can barely touch. + +However, treating video frames like standard web pages introduces subtle visual bugs. In our automated music video pipeline, an export that looked plausible in unit tests developed a jarring, violent shudder when rendered into a 1080p MP4. + +Here is what went wrong, why our initial intuition about audio reactivity failed, and how to structure React video layers so audio-driven HUDs never pollute background transitions. + +> **Live Reference**: The repaired rendering pipeline produced the final production MV for [《糖画小仙》 (Sugar Art Fairy) on YouTube](https://youtu.be/vZbyKeCCSRA), running rock-solid at 1080p 30fps. + +> **TL;DR** — Scaling a full-bleed background image with the bass band forces an interpolation resampling on every frame, which the human eye perceives as violent shudder. Furthermore, two stacked semi-transparent layers crossfading symmetrically dip through the black root container at the midpoint. Fixes: keep the background canvas completely static, decouple all audio-reactive pulsing into isolated foreground HUD layers, and fade in the incoming slide on top of a 100% solid base. Render time dropped by **38.0% (from 352s to 218s, a 1.61× throughput speedup)**. This is the rendering chapter of our [music-video pipeline](/posts/how-i-built-an-automated-music-video-pipeline/); the file-staging half is covered in [uploading large videos via CDP](/posts/uploading-large-videos-to-youtube-via-cdp/). + +--- + +## 1. The Symptom: A Whole-Frame Earthquake + +The visual effect was supposed to be simple: +1. An illustrated 1080p background slowly crossfading between slides. +2. A pulsing circular spectrum visualizer in the foreground. +3. Three lines of Chinese calligraphy lyrics that slide between left and right depending on where the character in the artwork is standing. + +In preview scrubbing, everything looked acceptable. But during full-speed 30fps playback of the rendered 1080p file, the viewer felt instant visual fatigue. Every time the electronic drum pattern kicked in, the entire scene appeared to vibrate violently up to 30 times per second. Furthermore, during slide transitions, the screen experienced micro-flashes where the background brightness suddenly dipped. + +--- + +## 2. Root Cause 1: Low-Frequency Scale Pollution + +Looking at the initial `BackgroundSlideshow.tsx`, we had attached an audio reactivity hook: + +```tsx +// The culprit in BackgroundSlideshow.tsx +const bassScale = interpolate(bassLevel, [0, 1], [1.0, 1.025], { + extrapolateRight: 'clamp', +}); + +// Applied to the background image: + +``` + +On paper, expanding the image by 2.5% on bass hits sounds like an engaging dynamic pulse. In reality: +- **Electronic audio beats hit in 30–60ms bursts**: The bass amplitude oscillates rapidly between zero and peak within 2 to 3 consecutive frames. +- **A 1920x1080 image scaling by 2.5% shifts edge pixels by nearly 25 to 30 physical pixels in a single frame**. +- When that sudden displacement reverses on the very next frame, the human eye interprets it not as a musical pulse, but as **severe camera camera shake / stutter**. +- To make matters worse, during the 1.8-second slide transition, two overlapping images with different transform origins were simultaneously vibrating at conflicting scales, creating a hallucinatory double-vision jitter. + +--- + +## 3. Root Cause 2: Symmetrical Opacity Crossfade DIP + +The second issue was the crossfade flash. In standard web CSS, people often do: + +```css +/* Web crossfade trap */ +.current-slide { opacity: 1 -> 0; } +.next-slide { opacity: 0 -> 1; } +``` + +When two images crossfade with linear opacity, consider what happens at the exact midpoint ($t = 0.5$): +- $\text{Current Slide Opacity} = 0.5$ +- $\text{Next Slide Opacity} = 0.5$ + +Here is how the browser/Remotion compositing pipeline stacks these layers against the root black container: + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Layer 2: Next Slide (opacity = 0.5) │ +├─────────────────────────────────────────────────────────────┤ +│ Layer 1: Current Slide (opacity = 0.5) │ +├─────────────────────────────────────────────────────────────┤ +│ Layer 0: Root Container (background = #000000 black) │ +└─────────────────────────────────────────────────────────────┘ +``` + +Because alpha blending operates multiplicatively down the stack: +$$\text{Composite Alpha} = 1 - (1 - 0.5) \times (1 - 0.5) = 1 - 0.25 = 0.75$$ + +The total combined alpha drops to **0.75**, exposing **25% of the black background**! This is not an optical illusion; it is mathematical light loss. Every time a transition peaked, the frame darkened by a quarter of its luminance, creating an unmistakable flicker. + +--- + +## 4. The Solution: Rock-Solid Base + Pure Single-Direction Opacity + +To fix both issues permanently: + +### 1. Remove All Scaling from Backgrounds +Background artwork must remain rock-solid. All audio reactivity belongs exclusively in the HUD layer (the radial spectrum bars and glowing rings). The background canvas should only ever receive smooth, non-oscillating transformations. + +### 2. Base Solid + Top Layer Fade-In +Instead of fading out the current image, **keep the current image at solid 100% opacity**. Only animate the incoming image on top: + +```tsx +return ( +
+ {/* Base Slide: Stays 100% solid, eliminating any drop to black */} + + + {/* Next Slide: Smoothly fades in on top over 1.8s */} + {isTransitioning && ( + + )} +
+); +``` + +### 3. Physical Decoupling of HUD and Canvas +The lyric box and circular spectrum are mounted inside a completely separate container layer with `pointerEvents: 'none'`. As the HUD glides across the canvas from `centerX: 540` to `centerX: 1380`, it calculates its position strictly from the timeline clock. It has zero knowledge of and zero impact on the background's opacity crossfade. + +--- + +## 5. The Outcome & Benchmark Reproduction + +After removing the background `bassScale` and switching to one-way opacity stacking: + +1. **Eliminated visible jitter**: Background illustrations remain crisp and visually anchored through rapid 140 BPM electronic drum drops across all test sequences. +2. **Eliminated transition light dips**: The 1.8s Bezier crossfade maintains 100% luminance continuity without dipping through the dark container base. +3. **Significant throughput speedup**: Removing per-frame matrix interpolation and scaling calculations on high-resolution background assets delivered a measurable performance boost. + +### Reproduction Benchmark + +| Metric | Before Optimization | After Optimization | Delta | +|---|---|---|---| +| **Render Duration** | 352.0 seconds | 218.3 seconds | **-38.0% (-133.7s)** | +| **Render Throughput** | 22.24 fps | 35.87 fps | **+1.61× speedup** | +| **Total Frames** | 7,830 frames | 7,830 frames | Identical (261s / 4m 21s) | +| **Output Resolution** | 1920 × 1080 (1080p @ 30fps) | 1920 × 1080 (1080p @ 30fps) | Identical | +| **Codec & Settings** | H.264 (CRF 18, preset medium) | H.264 (CRF 18, preset medium) | Identical | + +**Test Environment Specifications:** +- **CPU**: Intel Core i9-13900K (24 cores / 32 threads, boost up to 5.8 GHz) +- **GPU**: NVIDIA GeForce RTX 3060 (12GB GDDR6 VRAM, Driver 551.xx) +- **RAM**: 64GB DDR5 5600 MHz +- **OS**: Windows 11 Pro 64-bit +- **Software**: Node.js v20.x, Remotion v4.0.x with Chromium headless GPU rendering enabled + +Both benchmark runs were executed consecutively using identical media assets, audio spectrogram inputs, and composition parameters. The 38.0% reduction in render time directly translates to a **1.61× throughput speedup**, turning a process that was previously slower than realtime into an accelerated 35.8 fps export. + +--- + +## Related Guides in this Series +- **Pipeline Architecture**: [How I Built an Automated Music Video Pipeline with Remotion and Docker](/posts/how-i-built-an-automated-music-video-pipeline/) +- **Remote Publishing**: [Uploading 100MB+ Videos to YouTube via CDP: The Docker Archive Pattern](/posts/uploading-large-videos-to-youtube-via-cdp/) +- **Live Output**: [Watch the finished MV on YouTube](https://youtu.be/vZbyKeCCSRA) +```` + +--- + +## 文章 06 / 11:把字体嵌进 PDF,让它在任何机器上都长一样 +- **Slug**: `embedding-fonts-so-a-pdf-looks-the-same-everywhere` +- **英文标题**: Embedding Fonts So a PDF Looks the Same on Every Machine +- **分类**: `engineering` | **发布日期**: 2025-01-22 +- **线上地址**: https://blog.hoelee.com/posts/embedding-fonts-so-a-pdf-looks-the-same-everywhere/ (EN) | https://blog.hoelee.com/posts/zh/embedding-fonts-so-a-pdf-looks-the-same-everywhere/ (ZH) +- **本篇审核状态**: ⏳ 待审核 (Pending) + +### ✍️ 顾问审阅意见区 (Review Feedback) +```text +【审阅结论】:[ ] 通过 [ ] 需微调 [ ] 需重构 +【事实与数字核对】: +【人设与用词卫生】: +【CRO与转化建议】: +``` + +### 🇨🇳 中文版原文 (Chinese Markdown Twin) +````markdown +--- +title: "把字体嵌进 PDF,让它在任何机器上都长一样" +description: "四个字族、八个字面、两个渲染引擎,外加一条 CSS 怪癖:为了不让生成的报告依赖服务器上恰好装了什么字体,我做了什么。" +pubDate: 2025-01-22 +updatedDate: 2026-10-16 +category: engineering +tags: ["pdf", "fonts", "mpdf", "typography", "php"] +ogImage: /og/zh-embedding-fonts-so-a-pdf-looks-the-same-everywhere.jpg +banner: /banners/embedding-fonts-so-a-pdf-looks-the-same-everywhere.jpg +draft: false +--- + +## 需求只有一句话 + +一个报告生成器必须产出在任何地方都长得一样的文档:中文标题用宋体、中文正文用黑体、英文标题与封面数字用一款展示衬线体、英文正文和表格用 Inter。字重也要对——强调用 medium、标题用 semibold——而且是三语、两个 PDF 渲染引擎都要对。 + +同时,**字体必须跟着文档走**。不是「服务器装过」,也不是「读者大概有」,而是**嵌进文件里**,这样另一个国家的印厂看到的页面是一样的。 + +老管线在最后这句上栽得很隐蔽:它依赖服务器上用包管理器装过的字体,而且**只嵌入了一款中文字体**。第二档字重根本不存在,于是「粗体」是渲染器的合成近似;而想发现这件事,唯一的办法是打开 PDF 去看里面到底装了什么。 + +## 规则一:从成品 PDF 里把字体读回来 + +「已嵌入」是一个关于**输出**的断言,所以我量的是输出。同一份文档,改前与改后: + +| | 改前 | 改后 | +|---|---|---| +| zh 页数 | 23 | **23** | +| en 页数 | 28 | **29** | +| bm 页数 | 30 | **30** | +| 真正嵌进去的字体 | 一款中文字体 + Roboto | 思源宋 Medium + SemiBold · 思源黑 Regular + Medium · Cormorant Garamond Medium + SemiBold · Inter Regular + SemiBold + Medium · 拉丁报告里的汉字走思源黑 | +| 渲染时间 / 峰值内存 | 基线 | 同一量级(±10%,属噪声) | +| 文件体积 | 基线 | 约 +3% | + +每个字面的字形数是收据:中文报告里 15,988 个字形落在正文黑体、2,247 个落在它的 medium 档、634 个落在标题宋体的 medium、222 个落在它的 semibold。这就是「字体在服务器上」与「字体在文件里」的区别——另一个国家的读者看到的是同样四个字面,而代价是某一种语言正好多一页(拉丁字体的度量比它替换掉的那个更宽)。 + +## 规则二:只出静态实例,理由写在解析器里 + +现代做法看起来该是「每个字族发一个可变字体」,但在这里是错的,有两个实测理由: + +1. **PDF 引擎的字体解析器只读 `glyf`/`loca`。** 喂它一个 PostScript/CFF 轮廓的 OpenType 文件,它直接拒绝:`Fonts with postscript outlines are not supported`。问题不是「可变」,是 CFF 轮廓。 +2. **它完全不读 `fvar` 表**,所以可变字体会**静默地**解析成它的**默认实例**——而中文字族的默认实例恰恰是最细的那一档(`name` 表里写得清清楚楚:Thin 100 与 ExtraLight 200)。整份报告会变成发丝体,而整条管线里不会报任何一个错。 + +所以构建步骤拿上游的可变字体**实例化成静态切面**:跑实例化器,然后把 `name`、`OS/2.usWeightClass` 和 `head.macStyle` 改写成它现在真实的字重(源文件的名字里还写着 Thin)。产物旁边有一份 manifest,逐文件记下 family、字重、sha256、字形数与码位数;许可证也随文件一起进仓库。 + +## 规则三:引擎只会说两种字重,那就映射到两个槽位 + +渲染引擎的 CSS 只认 `normal` 和 `bold`——它的源码自己就这么写("Does not support: bolder, lighter, 100..900")。所以数字字重对它毫无意义;而同一条管线里的无头 Chromium 又完全认它们。 + +可用的映射是:每个字族声明一个 **R 槽**和一个 **B 槽**(常规一档、更重一档),`font-weight: 500` 落到目标字族的 R 槽,`600` **只在 PDF 那一侧被改写成 `bold`**,落到 B 槽。Chromium 侧照写 `600`。两条测试钉住它:B 槽必须比 R 槽重(读 `usWeightClass`,不是假设),且 `font-weight: 600` 绝不允许出现在交给 PDF 引擎的样式表里。 + +有一处细节值得单写一段:在这个管线里,**medium 不是强调手段,而是一个字族成员**。中文标题用宋体的 medium 当常规、用 semibold 做强调;英文标题用展示衬线体做同样的事。这里搞错,屏幕上完全看不出来,而印出来一目了然。 + +## 规则四:不要按你控制不了输入的字集做子集化 + +第一版把中文字体按两个国家字集的并集做了子集,产物从约 50 MB 降到约 29 MB。然后我跑了姓名探针,发现这些字被裁掉了: + +```text +犇 骉 頔 祎 廸 玧 祾 赟 堃 +``` + +每一个都是真实的人名用字。报告会把客户姓名印在封面上,而姓名是自由输入——所以一套基于子集的构建,迟早会在产品最醒目的那一页上放一个豆腐块。全字形覆盖是约 31k 码位、约 50 MB 的嵌入字体;为了永远不给客户看一个空方块,这个价钱很便宜。 + +## 规则五:把「缺字」变成构建失败,而不是惊喜 + +闸门跑三项检查,任何一项不过就非零退出: + +- manifest 里声明的每个字体文件都在,且 sha256 与磁盘上的文件一致; +- 内容库(三语全部)+ 品牌字面 + 各类探针(简体、繁体、CJK 标点、全角、中英混排、数字、拉丁扩展、符号)+ 姓名用字,**每一个字在「它实际会被哪一档字体印出来」的那一档里都有字形**; +- 查字形用的是 PDF 引擎自己的解析器,而不是第三方库,所以这条检查不可能和渲染器互相矛盾。 + +已知边界写在明处,不藏起来:✅ ✔ 这类符号在这几款正文字体里都没有字形,而引擎也画不了彩色 emoji,所以它们不出现在报告正文;拉丁标题那款展示衬线体只有 974 个字形,不画 ① ★ ℃,这也是为什么内容库的检查覆盖**所有**槽位而不只是正文;而拉丁报告里的汉字必须走专门的 CJK 槽位——两条测试防止两个渲染器在这点上产生分歧。 + +## 如果重来一次 + +1. **在动样式之前先定字族与字重。** 版式做完之后再去加第二档字重,意味着每一页页数都要手工复核一遍。 +2. **选字体格式之前先读渲染器的源码。** 可变字体看着是审美问题,实际上是解析器问题——两个文件、十分钟就定了。 +3. **把要嵌进文档的一切当作许可证产物。** 字体文件连同许可证一起进仓库,否则就得进别人的法务审查。 +4. **量「到底嵌进去了什么」。** 页数、每个字面的字形数、文件体积、渲染时间:四项都动了,其中三项的方向和我猜的相反。 + +## 结果 + +改前:一款中文字体加一款界面字体、靠合成粗体、还取决于服务器装过什么。改后:四个字族、八个静态字面真正嵌在文件里,字重是各个字族真实的成员,页数与上一版相差不超过一页,而且有一条 lint 闸门:某个字没有字形时直接让构建失败——于是失败形态从「客户看到一个方块」变成了「CI 拒绝发版」。这跟这条管线里其它地方的取舍是同一个:**让错误发不出去,而不是让错误容易被看见。** +```` + +### 🇬🇧 英文版原文 (English Markdown Primary) +````markdown +--- +title: "Embedding Fonts So a PDF Looks the Same on Every Machine" +description: "Four families, eight faces, two renderers and one CSS quirk: what it took to stop a generated report depending on whatever fonts the server happened to have." +pubDate: 2025-01-22 +updatedDate: 2026-10-16 +category: engineering +tags: ["pdf", "fonts", "mpdf", "typography", "php"] +ogImage: /og/embedding-fonts-so-a-pdf-looks-the-same-everywhere.jpg +banner: /banners/embedding-fonts-so-a-pdf-looks-the-same-everywhere.jpg +draft: false +--- + +## The requirement was one sentence + +A report generator has to produce a document that looks the same everywhere: +Chinese headings in a serif, Chinese body text in a sans, Latin headings and +cover numerals in a display serif, Latin body text and tables in Inter. +Weights matter — medium for emphasis, semibold for headings — across three +languages and two PDF renderers. + +And the fonts must travel with the document. Not "the server has them +installed", not "the reader probably has them" — **embedded**, so a print +shop in another country sees the same pages. + +The old pipeline failed on the last clause in a way that was invisible: it +depended on fonts installed on the server by a package manager, and it only +ever embedded a single CJK face. The second weight did not exist, so "bold" +was the renderer's synthetic approximation, and the only way to notice was +to open the PDF and look at what was actually inside it. + +## Rule one: read the fonts back out of the finished PDF + +"Embedded" is a claim about the output, so I measured the output. The same +document, rendered before and after: + +| | before | after | +|---|---|---| +| zh page count | 23 | **23** | +| en page count | 28 | **29** | +| bm page count | 30 | **30** | +| fonts actually embedded | one CJK face + Roboto | Source Han Serif Medium + SemiBold · Source Han Sans Regular + Medium · Cormorant Garamond Medium + SemiBold · Inter Regular + SemiBold + Medium · Source Han Sans for the CJK runs inside Latin reports | +| render time / peak memory | baseline | same order (±10%, noise) | +| file size | baseline | roughly +3% | + +The glyph counts per face are the receipt: in the Chinese report, 15,988 +glyphs land in the body sans, 2,247 in its medium weight, 634 in the heading +serif medium, 222 in its semibold. That is the difference between "the font +is on the server" and "the font is in the file" — a reader in another +country sees the same four faces, and a page count that moved by exactly one +page in one language (the Latin metrics are wider than what they replaced) +is the honest cost. + +## Rule two: static instances only, and the reason is in the parser + +The obvious modern choice — ship one variable font per family — is wrong +here, for two measured reasons: + +1. **The PDF engine's font parser reads `glyf`/`loca` only.** Feed it an + OpenType file with PostScript/CFF outlines and it refuses: + `Fonts with postscript outlines are not supported`. A variable font is + not the problem; CFF outlines are. +2. **It does not read the `fvar` table at all**, so a variable font silently + resolves to its *default instance* — and the default instances of the CJK + families are the thinnest ones (Thin 100 and ExtraLight 200, visible in + the `name` table). Every page would render in a hairline weight, with no + error anywhere in the pipeline. + +So the build step takes the upstream variable fonts and **instantiates +static cuts**: it runs the instancer, then rewrites `name`, +`OS/2.usWeightClass` and `head.macStyle` so the file identifies itself as +the weight it now is (the source's own name still says "Thin"). A manifest +records family, weight class, sha256, glyph count and code-point count for +every file, and the licence files ship next to them. + +## Rule three: the engine speaks two weights, so map onto two slots + +The renderer's CSS handling honours exactly `normal` and `bold` — its source +says so in as many words ("Does not support: bolder, lighter, 100..900"). +Numeric weights are therefore meaningless to it, while the headless-Chromium +renderer in the same pipeline honours them properly. + +The mapping that works: each family declares an **R slot** and a **B slot** +(regular-ish and heavier), `font-weight: 500` lands on the R slot of the +intended family, and `600` is **rewritten to `bold`** on the PDF side only, +which lands on the B slot. Chromium keeps `600` as written. Two tests pin +it: the B slot must be heavier than the R slot (read from `usWeightClass`, +not assumed), and `font-weight: 600` must never appear in the stylesheet +handed to the PDF engine. + +One subtlety worth the paragraph: a "medium" weight is not an emphasis +mechanism in this pipeline, it is a *family member*. Chinese headings use +the serif's medium as their regular and its semibold for emphasis; Latin +headings do the same with the display serif. Getting that wrong is invisible +on a screen and obvious in print. + +## Rule four: do not subset against a character set you do not control + +The first attempt subset the CJK fonts to the union of two national +character sets and produced a ~29 MB bundle instead of ~50 MB. Then I ran +the name probe and found these were cut: + +```text +犇 骉 頔 祎 廸 玧 祾 赟 堃 +``` + +Every one of them is a real given name character. The report prints a +customer's name on the cover, and the name is free input — so a subset-based +build will eventually put a tofu box on the most prominent page of the +product. Full coverage is ~31k code points and roughly 50 MB of embedded +fonts; that is a fine price for never showing a customer an empty rectangle. + +## Rule five: make missing glyphs a build failure, not a surprise + +The gate runs three checks and exits non-zero on any of them: + +- every font file declared in the manifest is present, and its sha256 + matches the file on disk; +- every character in the content library (all three languages) plus the + brand strings, plus probe classes — simplified, traditional, CJK + punctuation, full-width forms, mixed-script, digits, Latin extended, + symbols — plus the name characters, has a glyph **in the face that will + actually print it**; +- the lookup uses the PDF engine's own font parser rather than a third-party + library, so the check cannot disagree with the renderer. + +The known edges are written down instead of hidden. Dingbats and emoji (✅ ✔) +exist in none of these text fonts and the engine cannot draw colour emoji, +so they are kept out of the report body. The display serif has only 974 +glyphs and does not draw symbols like ① ★ ℃, which is why the +content-library check covers *all* slots rather than just the body. And CJK +characters inside a Latin report must go through the dedicated CJK slot — +two tests keep the renderers from disagreeing about that. + +## What I would do differently + +1. **Choose faces and weights before styling anything.** Retrofitting a + second weight after the layout exists means re-checking every page count + by hand. +2. **Read the renderer's source before choosing a font format.** The + variable-font decision looked like a taste question and was actually a + parser question — two files settled it in ten minutes. +3. **Treat everything you ship inside a document as a licence artifact.** + Font files enter the repo with their licences, or they enter someone + else's legal review. +4. **Measure what got embedded.** Page count, glyph-per-face counts, file + size, render time: all four moved, and three of them moved in ways I + would have guessed wrong. + +## The result + +Before: one CJK face and a UI font, with synthetic bold, depending on what +the server had installed. After: four families and eight static faces +embedded in the file, weights that are real members of their families, page +counts within one page of the previous layout, and a lint gate that fails +the build when a character has no glyph — so the failure mode moved from "a +customer sees a box" to "CI refuses the release". Which is the same trade as +everywhere else in this pipeline: make the mistake impossible to ship rather +than easy to notice. +```` + +--- + +## 文章 07 / 11:那个把静默丢失更新变成 409 的 JPA 字段:@Version 乐观锁 +- **Slug**: `jpa-version-field-lost-update` +- **英文标题**: The JPA Field That Turns a Silent Lost Update Into a 409 +- **分类**: `engineering` | **发布日期**: 2026-08-19 +- **线上地址**: https://blog.hoelee.com/posts/jpa-version-field-lost-update/ (EN) | https://blog.hoelee.com/posts/zh/jpa-version-field-lost-update/ (ZH) +- **本篇审核状态**: ⏳ 待审核 (Pending) + +### ✍️ 顾问审阅意见区 (Review Feedback) +```text +【审阅结论】:[ ] 通过 [ ] 需微调 [ ] 需重构 +【事实与数字核对】: +【人设与用词卫生】: +【CRO与转化建议】: +``` + +### 🇨🇳 中文版原文 (Chinese Markdown Twin) +````markdown +--- +title: "那个把静默丢失更新变成 409 的 JPA 字段:@Version 乐观锁" +description: "在 REST + JPA 的读-改-写流程里,两个编辑同时保存同一条记录时,后保存者会静默覆盖先保存者的修改,而 API 两次都返回成功——JPA 的 @Version 乐观锁把这个静默丢失更新变成 HTTP 409 冲突:请求携带期望版本号,版本不符就拒绝写入,并提示客户端重新读取再试。" +pubDate: 2026-08-19 +category: engineering +tags: [java, spring, jpa, hibernate, rest, testing] +ogImage: /og/zh-jpa-version-field-lost-update.jpg +banner: /banners/jpa-version-field-lost-update.jpg +draft: false +--- + +两个人同时打开同一篇文章,两个人都改了,两个人都保存。其中一次 +保存悄无声息地消失了——没有报错,没有警告,而且 API 对两次请求 +都返回了 HTTP 200。 + +这就是丢失更新(lost update)问题,是读-改-写(read-modify-write) +API 里最安静的数据丢失 bug。我在做 Spring Boot 作品集 demo 时撞上 +了它——这是一个 Spring Boot 3.5.16(Java 21)的 REST API,编辑器 +从浏览器页面创建和更新文章。修法最终落在三处:JPA 实体上的一个 +`@Version` 字段、请求契约里的一个字段、以及 API 层把它映射成 +HTTP 409 Conflict 的一个异常。这篇文章就沿着真实代码走一遍这条路。 + +## 问题:一条编辑消失了,却谁都怪不上 + +为什么两个人保存同一条记录时,其中一个人的修改会消失? + +跟着时间线走。编辑器 A 和编辑器 B 都拉取了同一篇文章。A 先保存: +行被更新,API 返回 200。B 稍后保存:行被*再次*更新,这次覆盖了 +A 的文字,API 又返回 200。每个请求都成功;每个响应都告诉它的调用 +方「你的保存就是这条记录的当前状态」——对其中一个人来说这是假话。 +A 的编辑就这么没了,两个客户端都没有任何办法知道。 + +失败是静默的,因为它就是读-改-写流程的正常行为,不是异常。这正是 +它在真实系统里存活的理由,也是为什么在要展示工程判断力的东西里, +值得把它修得干净。 + +## 我先试的方案:不带版本号的读-改-写,以及它为什么失败 + +我对更新流程的第一版草图是最直白的那种:`GET` 记录、编辑、 +`PUT` 回去,只带新的标题和正文,什么都不带。 + +```text +GET /api/posts/{id} # 读取记录 +PUT /api/posts/{id} # 写回新的标题和正文 +``` + +服务端加载行、套用更新、返回 200。它无从知道在这一行数据上,客户端 +在 `GET` 和 `PUT` 之间已经被别人动过——读取和写入是两个互不相干的 +请求,契约里没有任何东西把它们连起来。最后写入的人赢,而两个写入 +者都被告知成功。 + +在这里,200 比报错更糟。报错至少告诉输掉的那个客户端「出事了」, +给他们一个去查的理由。200 则告诉两个客户端「你们的保存就是记录 +当前的状态」——对其中一个是撒谎。输掉的那次编辑现在哪儿都不存在: +屏幕上没有,数据库里也没有;而因为客户端相信自己保存成功了, +没有人会去找这份数据。 + +## 修法:一个 @Version 字段、一份契约、一个冲突 + +修法一共五小步,全部都在 demo 的源码里。 + +### 1. 实体随身携带一个版本列 + +```java +@Entity +@Table(name = "posts", indexes = @Index(name = "idx_posts_title", columnList = "title")) +@EntityListeners(AuditingEntityListener.class) +public class Post { + + @Id + @GeneratedValue + @UuidGenerator + private UUID id; + + @Column(nullable = false, length = 160) + private String title; + + @Column(nullable = false, length = 10_000) + private String body; + + @Version + private long version; + // ... +} +``` + +`@Version` 告诉 Hibernate 把 `version` 当作乐观锁。它生成的每一条 +`UPDATE` 都变成有条件的: + +```sql +UPDATE posts +SET title = ?, body = ?, version = version + 1, ... +WHERE id = ? AND version = ? +``` + +如果这行数据的版本号已经和语句构造时的不一致,受影响行数为零, +Hibernate 就会拒绝这次 flush,而不是悄悄覆盖。实体从不用手自增 +计数器——更新方法只碰 `title` 和 `body`;版本号跟着写入一起走。 + +### 2. 版本号进入 API 契约 + +锁如果客户端无法参与,就没有用,所以版本号在两个方向上都穿过 +API。响应的 record 暴露它: + +```java +public record PostResponse(UUID id, long authorId, String title, String body, + long version, Instant createdAt, Instant updatedAt) { +} +``` + +更新请求则要求把它带回来: + +```java +public record UpdatePostRequest( + @NotBlank @Size(max = 160) String title, + @NotBlank @Size(max = 10_000) String body, + @NotNull @Min(0) Long version) { +} +``` + +`UpdatePostRequest` 旁边的学习注记说得很直白: +「the expected version is part of the update +contract, making optimistic locking visible to clients.」 +(期望的版本号是更新契约的一部分,让乐观锁对客户端可见。) + +### 3. 在拥有写事务的 service 里检查版本 + +```java +@CachePut(cacheNames = "posts", key = "#postId") +@Transactional +public PostResponse updatePost(UUID postId, UpdatePostRequest request) { + Post post = postRepository.findById(postId) + .orElseThrow(() -> new PostNotFoundException(postId)); + if (post.getVersion() != request.version()) { + throw new PostVersionConflictException(postId); + } + post.update(request.title().trim(), request.body().trim()); + return PostResponse.from(postRepository.saveAndFlush(post)); +} +``` + +读取、比较、保存全部发生在一个 `@Transactional` 方法里——这正是让 +检查诚实的那部分:版本号是和*写入那一刻*的行状态比较,而不是和 +早前某个请求留下的快照比较。如果读取和写入分属不同事务,检查到的 +就是这行数据已经离开的旧版本,整套机制就成了表演。`PostService` +的学习注记点名了这条规则:「service methods +centralize transaction boundaries, cache coherence, and +optimistic-locking rules.」(service 方法集中管理事务边界、缓存 +一致性与乐观锁规则。)demo 还开着 +`spring.jpa.open-in-view: false`,所以不会有残留的会话跨请求地 +供给过期实体——行是在写入事务内部重新读取的。 + +显式的检查会确定性地挡掉过期请求。`@Version` 列在底层仍然有意义: +如果检查与 flush 之间恰好插进一次别的提交,那条有条件的 `UPDATE` +影响零行,Hibernate 照样拒绝这次写入。 + +### 4. 异常处理器把它映射成 409 + +异常本身带着客户端能直接照做的信息: + +```java +public class PostVersionConflictException extends RuntimeException { + + public PostVersionConflictException(UUID postId) { + super("Post %s has changed; fetch it again before retrying".formatted(postId)); + } +} +``` + +错误边界把它转成正确的 HTTP 应答: + +```java +@ExceptionHandler(PostVersionConflictException.class) +ProblemDetail handleConflict(PostVersionConflictException exception) { + return problem(HttpStatus.CONFLICT, "POST_VERSION_CONFLICT", exception.getMessage()); +} +``` + +`HttpStatus.CONFLICT` 就是 HTTP 409,响应是一份 problem +document——一个 `type` URI、一个稳定的机器码 +(`POST_VERSION_CONFLICT`)和一段人类可读的说明。demo 在配置里 +开启了框架的 problem-details 支持 +(`spring.mvc.problemdetails.enabled: true`),所以校验错误和所有 +其他错误都共用同一种响应形状。 + +### 5. 整条链路,从头到尾 + +```text +过期的 PUT /api/posts/{id} + -> PostService.updatePost 在事务内部重新读取这一行 + -> post.getVersion() != request.version() + -> PostVersionConflictException: "fetch it again before retrying" + -> ApiExceptionHandler.handleConflict + -> HTTP 409, ProblemDetail with code POST_VERSION_CONFLICT + -> 客户端知道自己的前提过期了 +``` + +一个 `@Version` 字段、请求契约里的一个 `version`、一个异常、一个 +handler 方法。Controller 什么都没改——`updatePost` 看起来还是普通 +的 `PUT`。 + +## 为什么答案是冲突,而不是重试 + +当两个编辑器产出同一份记录的两种不同版本时,服务端无法知道哪一方 +的意图才是被需要的。从各自作者的角度看,两次保存都正当;重放那条 +过期的写入,只是重放同一次数据丢失。 + +诚实的结论就是 HTTP 对 409 的定义:请求与资源的当前状态冲突, +并且客户端被明确告知如何解决——「fetch it again before retrying」 +(重新取一次再试)。输掉的客户端重新读取记录,看到另一位作者的 +修改,由人来决定保留什么。静默覆盖变成了一处看得见的决策点——这 +正是这件事的全部意义。 + +## 这个 demo 的其余部分证明了什么 + +版本字段只是一个小系统里的一层,而其余部分都能用同样的方式核实 +——下面每一行都是仓库里的一个类或配置文件,不是一句口号: + +| 层 | 做了什么 | 在哪 | +|---|---|---| +| 缓存 | 单篇文章读取使用有界 Caffeine 缓存(上限 500,TTL 10 分钟,都来自配置);读取用 `@Cacheable`、更新用 `@CachePut`、删除用 `@CacheEvict`;`GET /api/showcase/cache` 暴露请求数、命中数、未命中数和命中率 | `CacheConfig`、`CacheProperties`、`ShowcaseMetricsController` | +| 安全 | 无状态 HTTP Basic + BCrypt;页面与读 API 公开,所有写操作需要 `EDITOR` 角色;本地默认是专用的非机密 `demo-editor`/`changeit` 账户,可用环境变量覆盖 | `SecurityConfig`、`application.yml` | +| 错误 | 请求 record 在边界处校验;一个 `@RestControllerAdvice` 返回一致的 problem document 和机器码——`POST_NOT_FOUND`(404)、`POST_VERSION_CONFLICT`(409)、带字段错误表的 `VALIDATION_FAILED`(400) | `CreatePostRequest`、`UpdatePostRequest`、`ApiExceptionHandler` | +| 配置 | 本地 profile 使用内存 H2 数据库(PostgreSQL 模式),`ddl-auto: update` 并预置示例文章;`prod` 选择 PostgreSQL,所有凭据来自 `APP_*` 环境变量,`ddl-auto: validate` | `application.yml`、`application-prod.yml` | +| 测试 | 一个 `@SpringBootTest` 套件断言:渲染的 Thymeleaf 页面、未认证 401 对编辑器 201、`VALIDATION_FAILED` 问题响应、公开搜索,以及重复读取后的缓存命中 | `DemoApplicationIntegrationTest` | + +## 如果做真东西,我会怎么改 + +一旦这不再是本地 demo,有两件事立刻要变。项目 README 里两件都写了, +值得作为我自己的判断再重复一遍。 + +第一,schema 管理。我会在任何时候把 `ddl-auto` 设为 `validate` 之前, +先加 Flyway 或 Liquibase 迁移。demo 默认 profile 用的是 `update`, +在临时数据库上很方便,养成习惯就很危险——生产 schema 应该是 +有版本管理的代码,而不是启动时的副作用。 + +第二,密钥。本地编辑器账户是刻意的非机密 demo 账户,带默认凭据。 +做真东西的话,我会把凭据放进托管的密钥存储(managed secret store), +拿不到凭据就拒绝启动,而不是回退到默认值。`prod` profile 已经做到 +所有凭据来自环境变量、不提交任何密钥——这正是我想要的形态, +只是不要那些回退值。 + +## 结果 + +过期的更新现在返回 HTTP 409,机器码 `POST_VERSION_CONFLICT`,而不是 +一个静静丢掉另一位作者修改的 200。丢失不再可能悄悄发生:失败是 +响亮的、可操作的——「fetch it again before retrying」——并且集成 +测试套件端到端地断言了访问与校验行为。 + +整个修法就是一个字段上的一个注解,加上让版本号成为契约一部分的 +管道代码。如果一条记录可能同时被两个人读取,这个字段就是「编辑 +丢失」与「双方都看得见的冲突」之间的分界线。 + +我平时就做 Java/Spring 后端和全栈 Web 应用——从浏览器页面到数据库 +的 REST API:JPA、乐观锁、缓存、校验和一整套集成测试。如果你的表单 +或字段里也有「编辑悄悄消失」的问题,跟我说说: +[WhatsApp](https://wa.me/60127972969) · +[me@hoelee.com](mailto:me@hoelee.com?subject=Spring%20Boot%20help) · +[hoelee.com](https://hoelee.com)。 +```` + +### 🇬🇧 英文版原文 (English Markdown Primary) +````markdown +--- +title: "The JPA Field That Turns a Silent Lost Update Into a 409" +description: "How a JPA @Version field turns a silent lost update into a 409 conflict — optimistic locking makes a stale REST write fail loudly instead of overwriting." +pubDate: 2026-08-19 +category: engineering +tags: [java, spring, jpa, hibernate, rest, testing] +ogImage: /og/jpa-version-field-lost-update.jpg +banner: /banners/jpa-version-field-lost-update.jpg +draft: false +--- + +Two people open the same post. Both edit it. Both save. One of the saves +disappears without a trace — no error, no warning, and the API answered +HTTP 200 for both of them. + +That is the lost update problem, and it is the quietest data-loss bug in +read-modify-write APIs. I hit it while building my Spring Boot portfolio +demo — a Spring Boot 3.5.16 (Java 21) REST API where editors create and +update posts from a browser page. The fix turned out to be one `@Version` +field on the JPA entity, one field on the request contract, and one +exception that the API layer maps to HTTP 409 Conflict. This post traces +that path through the actual code. + +## The problem: an edit that vanishes with no one to blame + +Why does one editor's change disappear when two people save the same +record? + +Follow the timeline. Editor A and editor B both fetch the same post. +A saves first: the row is updated and the API answers 200. B saves a +moment later: the row is updated *again*, this time over A's text, and +the API answers 200 again. Every request succeeded; every response told +its caller that their save is the state of the record. A's edit is +simply gone, and neither client has any way to know. + +The failure is silent because it is normal read-modify-write behaviour, +not an anomaly. That is exactly why the bug survives in real systems, and +why it is worth fixing properly in something meant to demonstrate +engineering judgement. + +## What I tried first, and why it failed + +My first sketch of the update flow was the obvious one: `GET` the record, +edit it, `PUT` it back with the new title and body and nothing else. + +```text +GET /api/posts/{id} # read the record +PUT /api/posts/{id} # write it back with a new title and body +``` + +The server loads the row, applies the update, and returns 200. It has no +way to notice that the row moved underneath the client between the `GET` +and the `PUT` — the read and the write are two unrelated requests, and +nothing in the contract connects them. The last write wins, and both +writers are told they succeeded. + +A 200 is worse than an error here. An error at least tells the losing +client that something happened and gives them a reason to look. A 200 +tells both clients that their save is the current state of the record, +which is a lie for one of them. The losing edit now exists nowhere — not +on screen, not in the database — and because the client believes it +saved, nobody goes looking for the data. + +## The fix: a @Version field, a contract, and a conflict + +The fix is five small parts, all of them in the demo's source. + +### 1. The entity carries a version column + +```java +@Entity +@Table(name = "posts", indexes = @Index(name = "idx_posts_title", columnList = "title")) +@EntityListeners(AuditingEntityListener.class) +public class Post { + + @Id + @GeneratedValue + @UuidGenerator + private UUID id; + + @Column(nullable = false, length = 160) + private String title; + + @Column(nullable = false, length = 10_000) + private String body; + + @Version + private long version; + // ... +} +``` + +`@Version` tells Hibernate to treat `version` as an optimistic lock. Every +`UPDATE` it generates becomes conditional: + +```sql +UPDATE posts +SET title = ?, body = ?, version = version + 1, ... +WHERE id = ? AND version = ? +``` + +If the row's version no longer matches the value the statement was built +with, zero rows change and Hibernate rejects the flush instead of quietly +overwriting. The entity never bumps the counter by hand — the update +method only touches `title` and `body`; the version moves with the write +itself. + +### 2. The version is part of the API contract + +A lock is useless if the client cannot take part in it, so the version +travels through the API in both directions. The response record exposes +it: + +```java +public record PostResponse(UUID id, long authorId, String title, String body, + long version, Instant createdAt, Instant updatedAt) { +} +``` + +and the update request requires it back: + +```java +public record UpdatePostRequest( + @NotBlank @Size(max = 160) String title, + @NotBlank @Size(max = 10_000) String body, + @NotNull @Min(0) Long version) { +} +``` + +The learning note next to `UpdatePostRequest` says it plainly: "the +expected version is part of the update contract, making optimistic +locking visible to clients." + +### 3. The service checks it inside the transaction that owns the write + +```java +@CachePut(cacheNames = "posts", key = "#postId") +@Transactional +public PostResponse updatePost(UUID postId, UpdatePostRequest request) { + Post post = postRepository.findById(postId) + .orElseThrow(() -> new PostNotFoundException(postId)); + if (post.getVersion() != request.version()) { + throw new PostVersionConflictException(postId); + } + post.update(request.title().trim(), request.body().trim()); + return PostResponse.from(postRepository.saveAndFlush(post)); +} +``` + +The load, the comparison and the save all happen inside one +`@Transactional` method, and that is the part that makes the check +honest: the version is compared against the row as it stands *at write +time*, not against a snapshot from an earlier request. If the read and +the write lived in different transactions, the check would compare +against a version the row has already left behind, and the whole scheme +would be theatre. The learning note on `PostService` names the rule: +"service methods centralize transaction boundaries, cache coherence, and +optimistic-locking rules." The demo also runs with +`spring.jpa.open-in-view: false`, so no lingering session serves a stale +entity across requests — the row is re-read inside the writing +transaction. + +The explicit check catches stale requests deterministically. The +`@Version` column still matters underneath: if another commit slips in +between the check and the flush, the conditional `UPDATE` affects zero +rows and Hibernate refuses the write anyway. + +### 4. The handler maps the exception to a 409 + +The exception itself carries the message a client can act on: + +```java +public class PostVersionConflictException extends RuntimeException { + + public PostVersionConflictException(UUID postId) { + super("Post %s has changed; fetch it again before retrying".formatted(postId)); + } +} +``` + +and the error boundary turns it into the right HTTP answer: + +```java +@ExceptionHandler(PostVersionConflictException.class) +ProblemDetail handleConflict(PostVersionConflictException exception) { + return problem(HttpStatus.CONFLICT, "POST_VERSION_CONFLICT", exception.getMessage()); +} +``` + +`HttpStatus.CONFLICT` is HTTP 409, and the response is a problem document +— a `type` URI, a stable machine code (`POST_VERSION_CONFLICT`) and a +human-readable detail. The demo enables the framework's problem-details +support in its configuration (`spring.mvc.problemdetails.enabled: true`), +so validation and every other error ride the same shape. + +### 5. The whole path, end to end + +```text +stale PUT /api/posts/{id} + -> PostService.updatePost re-reads the row inside its transaction + -> post.getVersion() != request.version() + -> PostVersionConflictException: "fetch it again before retrying" + -> ApiExceptionHandler.handleConflict + -> HTTP 409, ProblemDetail with code POST_VERSION_CONFLICT + -> the client knows its premise was stale +``` + +One `@Version` field, one `version` in the request contract, one +exception, one handler method. Nothing on the controller changed — +`updatePost` still looks like an ordinary `PUT`. + +## Why a conflict is the correct answer, not a retry + +When two editors produce two different versions of the same record, the +server cannot know which intent is the one meant. Both saves are +legitimate from their authors' point of view, and replaying the stale +write just replays the same data loss. + +The honest verdict is HTTP's own definition of 409: the request conflicts +with the current state of the resource, and the client is told exactly +how to resolve it — "fetch it again before retrying". The losing client +re-reads the record, sees the other writer's change, and a human decides +what to keep. The silent overwrite becomes a visible decision point, +which is the entire purpose of the exercise. + +## What the rest of the demo proves + +The version field is one layer of a small system, and the rest of it is +verifiable the same way — every row below is a class or a config file in +the repository, not a claim: + +| Layer | What it does | Where it lives | +|---|---|---| +| Caching | Bounded Caffeine cache on single-post reads (max 500, TTL 10 min, both from config); `@Cacheable` on read, `@CachePut` on update, `@CacheEvict` on delete; `GET /api/showcase/cache` exposes requests, hits, misses and hit rate | `CacheConfig`, `CacheProperties`, `ShowcaseMetricsController` | +| Security | Stateless HTTP Basic with BCrypt; pages and read APIs are public, every mutation needs the `EDITOR` role; local defaults are a throwaway `demo-editor`/`changeit` account overridable via env vars | `SecurityConfig`, `application.yml` | +| Errors | Request records validate at the boundary; one `@RestControllerAdvice` returns consistent problem documents with machine codes — `POST_NOT_FOUND` (404), `POST_VERSION_CONFLICT` (409), `VALIDATION_FAILED` (400) with a field-error map | `CreatePostRequest`, `UpdatePostRequest`, `ApiExceptionHandler` | +| Configuration | Local profile runs an in-memory H2 database in PostgreSQL mode with `ddl-auto: update` and seeded sample posts; `prod` selects PostgreSQL, takes every credential from `APP_*` env vars and sets `ddl-auto: validate` | `application.yml`, `application-prod.yml` | +| Tests | One `@SpringBootTest` suite asserts the rendered Thymeleaf pages, unauthenticated 401s versus editor 201s, the `VALIDATION_FAILED` problem response, public search, and cache hits after repeated reads | `DemoApplicationIntegrationTest` | + +## What I would do differently for anything real + +Two things change the moment this stops being a local demo. The project +README says both, and they are worth repeating as my own judgement. + +First, schema management. I would add Flyway or Liquibase migrations +before ever setting `ddl-auto` to `validate`. The demo's default profile +uses `update`, which is convenient on a scratch database and dangerous as +a habit — production schema should be versioned code, not a side effect +of startup. + +Second, secrets. The local editor account is intentionally a non-secret +demo account with default credentials. For anything real, I would keep +credentials in a managed secret store and refuse to boot without them, +instead of falling back to a default. The `prod` profile already takes +every credential from environment variables with no committed secrets — +that is the shape I would keep, minus the fallbacks. + +## The result + +A stale update now returns HTTP 409 with code `POST_VERSION_CONFLICT` +instead of a 200 that silently discards the other writer's change. The +loss can no longer happen quietly: the failure is loud, actionable — +"fetch it again before retrying" — and the integration test suite asserts +the access and validation behaviour end to end. + +The whole fix is one annotation on one field, plus plumbing that makes +the version part of the contract. If a record can be read by two people +at the same time, that field is the difference between a lost edit and a +conflict both clients can see. + +I build Java/Spring back ends and full-stack web applications — REST APIs +with JPA, optimistic locking, caching, validation and integration tests, +from the browser page down to the database. If you have a form or a field +where edits keep quietly disappearing, tell me about it: +[WhatsApp](https://wa.me/60127972969) · +[me@hoelee.com](mailto:me@hoelee.com?subject=Spring%20Boot%20help) · +[hoelee.com](https://hoelee.com). +```` + +--- + +## 文章 08 / 11:把 CodeIgniter 4 应用从 IIS 迁移到 OpenLiteSpeed +- **Slug**: `migrating-codeigniter-iis-to-openlitespeed` +- **英文标题**: Migrating a CodeIgniter 4 App From IIS to OpenLiteSpeed +- **分类**: `engineering` | **发布日期**: 2026-09-27 +- **线上地址**: https://blog.hoelee.com/posts/migrating-codeigniter-iis-to-openlitespeed/ (EN) | https://blog.hoelee.com/posts/zh/migrating-codeigniter-iis-to-openlitespeed/ (ZH) +- **本篇审核状态**: ⏳ 待审核 (Pending) + +### ✍️ 顾问审阅意见区 (Review Feedback) +```text +【审阅结论】:[ ] 通过 [ ] 需微调 [ ] 需重构 +【事实与数字核对】: +【人设与用词卫生】: +【CRO与转化建议】: +``` + +### 🇨🇳 中文版原文 (Chinese Markdown Twin) +````markdown +--- +title: "把 CodeIgniter 4 应用从 IIS 迁移到 OpenLiteSpeed" +description: "我把一个 17,000 行的 CodeIgniter 4 应用从 Windows IIS 搬到了 CyberPanel 上的 OpenLiteSpeed。迁移后冒出两个致命错误——而 IIS 一直靠 SSO 把它们藏着。" +pubDate: 2026-09-27 +category: engineering +blueprint: sme-digital-suite +blueprintLayer: L1-high-conversion-web +tags: [codeigniter, php, openspeedway, cyberpanel, iis, migration, litespeed, linux] +ogImage: /og/zh-migrating-codeigniter-iis-to-openlitespeed.jpg +banner: /banners/migrating-codeigniter-iis-to-openlitespeed.jpg +draft: false +--- + +我维护着一个数字命理报告生成器:一个 CodeIgniter 4 应用,接收出生日期和姓名,跑一遍命理引擎,渲染出 19 页 A4 报告。它在 Windows Server + IIS 上跑了很多年。2026 年 9 月,我把它迁到 CyberPanel 上的 OpenLiteSpeed,跑在 Linux 上,换了个域名。 + +迁移本身没什么好说的——拷文件、把 docroot 指向 `public/`、装依赖。真正值得写下来的是:**新主机上第一个未认证请求,就在两条路由上直接致命崩溃——而这两条路由在 IIS 上"一直好好的"。** + +两个 bug 都是真的。两个从一开始就存在于 IIS 上。两个都看不见,因为 IIS 在出问题的路由前面恰好挡了一层 authentik SSO。 + +## 为什么这件事重要 + +如果你的自托管应用前面挂了一层认证网关,那这层网关做的事不止是保护数据——它同时**在帮你藏 bug**。任何"只对未认证用户出错"的东西永远不会以未认证身份运行,所以永远不会报错。而它会在最糟的时刻暴露:你迁移的时候、你拆掉网关的时候、你把路由暴露到公网的时候。 + +我这个应用,就"健康"了整整几年。我把新域名指过去的那一刻,两条路由返 HTTP 500。代码一行没改。唯一变的是网关没了。 + +两个修复加起来大概二十分钟。找到它们花了两个小时——因为真正的错误信息被 CodeIgniter 的 CLI 错误渲染器挡在了后面。 + +## 环境 + +这是一个相当普通的 CI4 项目,但有一个不太常见的架构细节:报告引擎不是库,而是**同一台主机上的 HTTP 端点**。控制器拼一个请求,`curl` 回自己域名的 `/api/single`,引擎返回算好的 JSON 数值,控制器再把它渲染成可打印的报告。 + +正是这个"自调用"设计,让 IIS 上必须有 `CONST_IIS_INTERNAL_BASE`。这点我后面会回来说——它才是真正有意思的部分。 + +| | 迁移前 | 迁移后 | +|---|---|---| +| 操作系统 | Windows Server | Ubuntu (CyberPanel) | +| Web 服务器 | IIS | OpenLiteSpeed 1.9.0 | +| PHP | 8.4 (Windows 版) | 8.4.25 (lsphp84) | +| Docroot | `C:\inetpub\calc.hoelee.com` | `/home/<域名>/public_html/public` | +| 认证网关 | authentik SSO 挡 `/lifecode`、`/api/*` | 无 | + +## 第一步:docroot 指向 `public/`,不是项目根目录 + +CI4 是两段式目录结构。`app/`、`vendor/`、`writable/` 和 `.env` 都在项目根目录,只有 `public/` 本该对 Web 可见。 + +CyberPanel 建的站点根目录叫 `public_html`。我的第一反应是把整个项目解压进 `public_html`,docroot 不动。 + +**那样会把 `app/`、`vendor/`、`.env` 文件和 `writable/` 的 session 数据全部暴露在 HTTP 之下。** 这个项目的 `.env` 里有一个 webhook 凭证。任何人猜到 `/../.env`——或者干脆直接请求 `/.env`,因为这文件就躺在被服务的目录下面——就拿到了。 + +正确做法是解压进 `public_html`,然后把 vhost 的 docroot 往下一层,改成 `public_html/public`: + +```bash +# 解压成 app/、vendor/、.env 位于 public_html 之下 +cd /home/<域名>/public_html +tar -xzf /tmp/deploy.tar.gz + +# docroot 必须是 public_html/public,不是 public_html +sudo sed -i 's#/home/<域名>/public_html\$#/home/<域名>/public_html/public#' \ + /usr/local/lsws/conf/vhosts/<域名>/vhost.conf + +sudo /usr/local/lsws/bin/lshttpd -t +sudo systemctl restart lsws +``` + +这是整个迁移里最重要的一步,也是大多数教程跳过的一步。其他什么都不做,这一步也得做。 + +## 第二步:IIS 藏起来的两个致命错误 + +### 致命错误一 —— `env()` 调用得太早 + +第一个 500 完全没有响应体。这不太寻常——CI4 正常会渲染点东西出来。一个空白的 500 通常意味着 PHP 在框架的错误处理器装上之前就死了。 + +CI4 的启动流程会在 `Boot::bootWeb()` 里很早加载 `app/Config/Constants.php`(通过 `Boot::loadConstants()`)。而这件事发生在 **`Common.php` 助手文件加载之前**——`env()` 就定义在那个文件里。 + +所以这一行: + +```php +// app/Config/Constants.php —— 错误写法 +define('CONST_fullBase', env('hoelee.fullBase')); +``` + +会失败于: + +``` +Fatal error: Uncaught Error: Call to undefined function env() +in app/Config/Constants.php:96 +``` + +规则是绝对的:**`Constants.php` 里只能放普通常量。** 不能有 `env()`、不能有 `getenv()`、不能有配置助手。如果你真需要依赖环境的值,就定义到启动流程更靠后的位置——或者把这个常量降级成 fallback,真正的值后面再读。 + +我做的正是后者。`CONST_fullBase` 变回普通字符串,消费它的助手函数改为**先**查框架自己的 `app.baseURL`(那个才是 `.env` 驱动的): + +```php +// app/Helpers/hoelee_helper.php +function getFullBase(bool $selfCall = false): string +{ + // .env 驱动的 baseURL 优先;CONST_fullBase 只是 fallback + if ($selfCall && defined('CONST_IIS_INTERNAL_BASE') && CONST_IIS_INTERNAL_BASE) { + return rtrim(CONST_IIS_INTERNAL_BASE, '/'); + } + $appBase = config('App')->baseURL; + if ($appBase) return rtrim($appBase, '/'); + if (defined('CONST_fullBase') && CONST_fullBase) return rtrim(CONST_fullBase, '/'); + return ''; +} +``` + +**这个是我自己搞出来的。** 同一周做凭证清理重构时引入的——我把一个硬编码 URL 挪进 `.env`,然后在 `Constants.php` 里用 `env()` 读它。在我机器上是通的,因为本地 `.env` 走的是另一条路径被读到。干净启动下它从来没通过。我是在**部署测试**后一小时内抓到的;如果只是代码审查,很可能就这么放过去了。 + +### 致命错误二 —— CI4 控制器里的 `parent::__construct()` + +第二个失败在 `/lifecode`。这个是应用里**原本就存在**的 bug,不是我引入的。 + +绕过 CLI 渲染器之后,真正的报错是: + +``` +Error: Cannot call constructor +``` + +CI4 的基础类 `CodeIgniter\Controller` **根本没有构造函数**。它实现的是 `initController()`,由框架把 request、response、logger 三个对象传进去。标准写法是: + +```php +// CI4 的正确写法 +public function initController( + RequestInterface $request, + ResponseInterface $response, + LoggerInterface $logger +) { + parent::initController($request, $response, $logger); + // 你的初始化代码 +} +``` + +但 `Lifecode.php` 和 `ApiEn.php` 用的是传统构造函数,还在调 `parent::__construct()`: + +```php +// 错误 —— CodeIgniter\Controller 没有 __construct() +public function __construct() +{ + parent::__construct(); + // ... +} +``` + +对一个没有定义 `__construct()` 的父类调 `parent::__construct()`,在 PHP 8 里是致命错误。把两个控制器改成 `initController()` 就修好了——同时要为新签名补上三个 interface 的 `use` 语句。 + +我没有盲信报错信息,而是先确认父类**确实**没有构造函数: + +```bash +grep -n 'function __construct\|function initController' \ + vendor/codeigniter4/framework/system/Controller.php +``` + +只出来 `initController()`。这一步值得做——"Cannot call constructor" 在父类**有**构造函数但内部报错时也会出现,重写签名之前你得先搞清楚自己是哪种情况。 + +## 为什么这两个 bug 在 IIS 上都不出现 + +这是让我改变部署观念的部分。 + +在 IIS 主机上,`/lifecode` 和 `/api/*` 挡在 authentik SSO 后面。对这两条路由的未认证请求返回 **HTTP 302 跳转到 `auth.hoelee.com`**——压根不会碰到控制器。那个坏掉的构造函数从来没被执行过。这条路由大概从写出来那天就是坏的,而它从来没被要求真正处理过一次请求。 + +我直接对比了两台主机来确认: + +```bash +curl -sI https://calc.hoelee.com/lifecode | head -1 +# HTTP/2 302 <- authentik 跳转;控制器根本没运行 + +curl -sI http://<新主机>/lifecode | head -1 +# HTTP/1.1 500 <- 没有网关;控制器运行并致命崩溃 +``` + +那个 302 就是这个 bug 活下来的原因。应用看起来健康,只是因为它不健康的部分根本触不到。 + +**教训说白了就是:一条路由前面挂了认证网关,就意味着这条路由自己的代码没有任何有效的测试覆盖。** 如果你要迁移或者拆网关,请专门留出时间,用未认证身份把每一条曾被网关挡着的路由都打一遍,再宣布迁移完成。小应用上这只是十行 `curl` 循环。在我这里,它本来几秒钟就能找出这两个 bug,而不是两个小时: + +```bash +for p in / /read/single /read/partner /lifecode /api/date /api/single; do + printf '%-16s %s\n' "$p" "$(curl -s -o /dev/null -w '%{http_code}' https://<主机>$p)" +done +``` + +## 第三步:那个不再是 loopback 的自调用 + +现在是这个架构里真正有意思的后果。 + +报告引擎是靠控制器 `curl` 自己的主机来调用的。在 IIS 上,`CONST_IIS_INTERNAL_BASE` 把这个调用指向 `http://localhost:7296`——loopback 接口,请求不出机器,也不碰 DNS 和 TLS。 + +在 LiteSpeed 上这个常量被**故意不定义**,于是 `getFullBase()` 落到 `app.baseURL`。这意味着每一次生成报告,现在都是**一个真实的对外 HTTPS 请求,打到应用自己的公网 URL,再从 Cloudflare 绕回来**。 + +那问题就来了:这到底能不能跑? + +答案是能,但值得专门测一次,因为它的失败模式很容易误导人。这是确切的测法——**在服务器上跑**,不是在你的笔记本上: + +```bash +curl -s -o /dev/null -w 'HTTPS self-call: %{http_code}\n' \ + -X POST 'https://<域名>/api/single' -d 'nameCn=test&dob=1990-01-01' +``` + +两个要注意的地方: + +- **在服务器上跑。** 我从 Windows 机器上打同一个 URL,拿到的是 404 和 500——那是我这台机器 DNS 解析和 Cloudflare 机器人防护的产物,不是服务器的问题。在错误的主机上测试,会得出很有把握的错误结论。 +- **看 JSON,不只看状态码。** 一个 200 配一个被截断的 body,比干脆失败更糟。我在信任之前先确认了响应能被解析。 + +这里藏着一个真实的取舍,我还没完全解决:走公网 URL 的自调用意味着生成报告依赖 Cloudflare 在线、每次报告多一次 TLS 握手、并且在整个过程中占用两个 PHP worker。在 LiteSpeed 上 worker 池不大的情况下,并发报告一多就可能死锁——每个请求都在等另一个请求,而后者拿不到空闲 worker。以目前的流量没问题。但在规模上去之前,这个自调用应该换成绕过 CDN 的内部路径。 + +## 第四步:验证真实产出,不是状态码 + +`/` 返 200 只证明落地页能渲染。它不证明报告引擎能用,而引擎才是整个产品。所以最后一步是端到端生成一份真报告: + +```bash +curl -s -X POST 'https://<域名>/read/single' \ + -d 'nameCn=test&dob=1990-01-01&gender=m' \ + -o report.html -w 'HTTP:%{http_code} bytes:%{size_download}\n' +``` + +结果:**HTTP 200、99,189 字节、19 页 A4。** 我又用中文字符输入跑了一遍,把引擎的 UTF-8 路径也走通——HTTP 200、99,205 字节、19 页、姓名渲染正常、输出里零 PHP 警告。 + +真正有意义的检查,是新旧主机之间页数可比。它证明字体、DPI 和分页逻辑都完整地熬过了这次搬迁。 + +关于那个 UTF-8 测试的插曲:我第一次跑的时候返回了 500,差点就一头扎进去查 bug。原因是我自己的 shell——中文字符在进 `curl` 的路上被终端编码搞坏了,应用收到的是非法字节。把同一个请求写成服务器上的脚本再跑,编码在我控制之内,就返回了 200。**在你调试编码故障之前,先确认服务器真正收到的字节就是你打算发出去的字节。** + +## 如果重来一次,我会怎么改 + +**迁移前就测未认证路由,而不是迁移后。** 那两个小时的排查,用一个遍历路由列表的 `curl` 循环就能完全避免。我现在把"列出所有路由、以未认证身份逐个请求、记录状态码"当成任何迁移的第零步。 + +**不要在 `Constants.php` 里调 `env()`。** 我在文件里留了注释专门写这件事,因为下一个人——多半是六个月后的我自己——一定会想这么干。 + +**迁移前先查框架版本对应的 project-space 配置。** 同一周我把 CI4 从 4.6.3 升到 4.7.4,它在两个升级指南里根本没提的属性上致命崩溃了两次:`Config\App::$permittedURIChars`(4.7 的 Router 要求)和 `Config\Format::$jsonEncodeDepth`(JSONFormatter 要求,而第二个正好打断了引擎的 JSON 自调用)。Composer 会更新 `vendor/`,但**永远**不会合并 `app/Config/*.php`,因为那些属于 project-space。我最后往 14 个配置文件里手工合并了新属性。那是另一篇文章,但迁移的教训是同一个形状:**框架会告诉你 `vendor/` 里改了什么;没有任何东西会告诉你 `app/` 里改了什么。** + +**先确认 docroot,再动别的。** 如果我解压进 `public_html` 就收手,应用是能跑的——同时静静地把 `.env` 通过 HTTP 服务出去。能跑的迁移,不等于安全的迁移。 + +## 结果 + +一个晚上。六条路由全部验证 200,中英文输入的报告产出在字节层面可比,零 PHP 警告,另外还从代码库里清掉了两个长期潜伏的 bug——而这两个 bug 一直被 IIS 的认证网关遮着。 + +应用现在跑在 OpenLiteSpeed 上,docroot 正确分离,主机可自动化管理,凭证放在 `.env` 而不是写死在常量里。 + +--- + +**想让你的应用也做这套迁移?** 我可以在 IIS、Apache、nginx 和 LiteSpeed 之间迁移 PHP 应用——包括那些麻烦的部分:自调用架构、藏 bug 的 SSO 网关,以及会碰到 project-space 配置的框架升级。 + +[WhatsApp +60 12-797 2969](https://wa.me/60127972969) · [me@hoelee.com](mailto:me@hoelee.com?subject=CodeIgniter%20迁移) · [hoelee.com](https://hoelee.com) +```` + +### 🇬🇧 英文版原文 (English Markdown Primary) +````markdown +--- +title: "Migrating a CodeIgniter 4 App From IIS to OpenLiteSpeed" +description: "I moved a 17,000-line CodeIgniter 4 app from Windows IIS to OpenLiteSpeed on CyberPanel. Two fatal errors appeared that IIS had been hiding behind SSO." +pubDate: 2026-09-27 +category: engineering +blueprint: sme-digital-suite +blueprintLayer: L1-high-conversion-web +tags: [codeigniter, php, openspeedway, cyberpanel, iis, migration, litespeed, linux] +ogImage: /og/migrating-codeigniter-iis-to-openlitespeed.jpg +banner: /banners/migrating-codeigniter-iis-to-openlitespeed.jpg +draft: false +--- + +I maintain a numerology report generator: a CodeIgniter 4 application that takes a birth date and a name, runs them through a numerology engine, and renders a 19-page A4 report. It had run on Windows Server with IIS for years. In September 2026 I moved it to OpenLiteSpeed on CyberPanel, on Linux, on a different domain. + +The migration itself was unremarkable — copy the files, point the docroot at `public/`, install dependencies. What made it worth writing about is that **the first unauthenticated request to the new host fatally crashed on two routes that had been working fine on IIS for years.** + +Both bugs were real. Both were present on IIS the whole time. Neither was visible, because IIS had an authentik SSO gate in front of the exact routes that were broken. + +## Why this matters + +If you run a self-hosted app behind an authentication gateway, your auth layer is doing more than protecting data — it is also **hiding your bugs**. Anything that only breaks for unauthenticated users never runs unauthenticated, so it never fails. The failure surfaces at the worst possible moment: when you migrate, remove the gate, or expose the route to the public internet. + +In my case the app had been reporting "healthy" for as long as I'd owned it. The moment I pointed a new domain at it, two routes returned HTTP 500. Nothing had changed in the code. The only thing that changed was that the gate was gone. + +The fix for both took about twenty minutes. Finding them took two hours, because the actual error messages were hidden behind CodeIgniter's CLI error renderer. + +## The setup + +The app is a fairly ordinary CI4 project with an unusual architecture detail: the report engine is not a library, it is a **set of HTTP endpoints on the same host**. The controllers generate a request, `curl` it back to `/api/single` on their own domain, and the engine returns JSON with the computed numbers. The controller then renders that into the printable report view. + +That self-call design is what made `CONST_IIS_INTERNAL_BASE` necessary on IIS. Let me come back to it — it turns out to be the interesting part. + +| | Before | After | +|---|---|---| +| OS | Windows Server | Ubuntu (CyberPanel) | +| Web server | IIS | OpenLiteSpeed 1.9.0 | +| PHP | 8.4 (Windows build) | 8.4.25 (lsphp84) | +| Docroot | `C:\inetpub\calc.hoelee.com` | `/home//public_html/public` | +| Auth gate | authentik SSO on `/lifecode`, `/api/*` | none | + +## Step 1: Point the docroot at `public/`, not the project root + +CI4 ships with a two-folder layout. `app/`, `vendor/`, `writable/` and `.env` live in the project root. Only `public/` is meant to be web-accessible. + +CyberPanel creates the document root as `public_html`. My first instinct was to extract the whole project into `public_html` and leave the docroot alone. + +**That would have exposed `app/`, `vendor/`, the `.env` file, and `writable/` session data over HTTP.** The `.env` in this project contains a webhook credential. Anyone who guessed `/../.env` — or just fetched `/.env`, since the file sits directly under a served directory — gets it. + +The correct layout is to extract into `public_html` and then repoint the vhost docroot one level deeper, at `public_html/public`: + +```bash +# extract the project so that app/, vendor/ and .env sit UNDER public_html +cd /home//public_html +tar -xzf /tmp/deploy.tar.gz + +# the docroot must be public_html/public, NOT public_html +sudo sed -i 's#/home//public_html\$#/home//public_html/public#' \ + /usr/local/lsws/conf/vhosts//vhost.conf + +sudo /usr/local/lsws/bin/lshttpd -t +sudo systemctl restart lsws +``` + +This is the single most important step in the migration, and it is the one most guides skip. If you do nothing else, do this. + +## Step 2: The two fatals IIS was hiding + +### Fatal 1 — `env()` called too early + +The first 500 had no body at all. That is unusual — CI4 normally renders something. An empty 500 usually means PHP died before the framework's error handler was installed. + +CI4's bootstrap loads `app/Config/Constants.php` very early in `Boot::bootWeb()`, via `Boot::loadConstants()`. That happens **before** the `Common.php` helper file is loaded, which is where `env()` is defined. + +So this line: + +```php +// app/Config/Constants.php — BROKEN +define('CONST_fullBase', env('hoelee.fullBase')); +``` + +fails with: + +``` +Fatal error: Uncaught Error: Call to undefined function env() +in app/Config/Constants.php:96 +``` + +The rule is absolute: **`Constants.php` may only contain plain constants.** No `env()`, no `getenv()`, no config helper. If you need environment-dependent values there, define them further along the bootstrap — or make the constant a fallback and read the real value later. + +That is exactly what I did. `CONST_fullBase` became a plain string, and the helper that consumes it checks the framework's own `app.baseURL` (which *is* `.env`-driven) first: + +```php +// app/Helpers/hoelee_helper.php +function getFullBase(bool $selfCall = false): string +{ + // the .env-driven baseURL wins; CONST_fullBase is only a fallback + if ($selfCall && defined('CONST_IIS_INTERNAL_BASE') && CONST_IIS_INTERNAL_BASE) { + return rtrim(CONST_IIS_INTERNAL_BASE, '/'); + } + $appBase = config('App')->baseURL; + if ($appBase) return rtrim($appBase, '/'); + if (defined('CONST_fullBase') && CONST_fullBase) return rtrim(CONST_fullBase, '/'); + return ''; +} +``` + +**This one was my own doing.** I had introduced it during a secrets-cleanup refactor in the same week — I moved a hardcoded URL into `.env` and called `env()` in `Constants.php` to read it. It worked on my machine because the local `.env` was being read through a different path. It never worked on a clean boot. I caught it within an hour *only because* I deployed and tested; a code review would plausibly have waved it through. + +### Fatal 2 — `parent::__construct()` in a CI4 controller + +The second failure was on `/lifecode`, and this one is a genuine pre-existing bug in the application — not something I introduced. + +The error, once I got past the CLI renderer, was: + +``` +Error: Cannot call constructor +``` + +CI4's base `CodeIgniter\Controller` class **has no constructor**. It implements `initController()`, which the framework calls with the request, response, and logger objects. The standard pattern is: + +```php +// correct CI4 pattern +public function initController( + RequestInterface $request, + ResponseInterface $response, + LoggerInterface $logger +) { + parent::initController($request, $response, $logger); + // your setup here +} +``` + +But `Lifecode.php` and `ApiEn.php` declared a classic constructor and called `parent::__construct()`: + +```php +// BROKEN — CodeIgniter\Controller has no __construct() +public function __construct() +{ + parent::__construct(); + // ... +} +``` + +`parent::__construct()` on a parent class that does not define `__construct()` is a fatal in PHP 8. Converting both controllers to `initController()` fixed it — and required adding the three `use` statements for the interface types in the new signature. + +I checked whether the base class *really* had no constructor rather than trusting the error message: + +```bash +grep -n 'function __construct\|function initController' \ + vendor/codeigniter4/framework/system/Controller.php +``` + +Only `initController()` came back. Worth doing — "Cannot call constructor" also fires when a parent *has* a constructor that errors internally, and you want to know which case you are in before you rewrite the signature. + +## Why neither bug showed up on IIS + +This is the part that changed how I think about the deployment. + +On the IIS host, `/lifecode` and `/api/*` sat behind authentik SSO. An unauthenticated request to either returned **HTTP 302 to `auth.hoelee.com`** — it never reached the controller at all. The broken constructor was never executed. The route had presumably been broken since it was written, and it had never once been asked to serve a request. + +I confirmed this by comparing the two hosts directly: + +```bash +curl -sI https://calc.hoelee.com/lifecode | head -1 +# HTTP/2 302 <- authentik redirect; controller never runs + +curl -sI http:///lifecode | head -1 +# HTTP/1.1 500 <- no gate; controller runs and fatals +``` + +The 302 is why the bug survived. The app looked healthy because the unhealthy parts were unreachable. + +**The lesson, stated plainly: an auth gate in front of a route means that route has no working test coverage for its own code.** If you migrate or remove the gate, budget time to hit every previously-gated route unauthenticated before you call the migration done. On a small app that is a ten-line `curl` loop. Here it would have found both bugs in seconds instead of two hours: + +```bash +for p in / /read/single /read/partner /lifecode /api/date /api/single; do + printf '%-16s %s\n' "$p" "$(curl -s -o /dev/null -w '%{http_code}' https://$p)" +done +``` + +## Step 3: The self-call that stops being a loopback + +Now the interesting architectural consequence. + +The report engine is reached by the controller `curl`-ing its own host. On IIS, `CONST_IIS_INTERNAL_BASE` pointed that call at `http://localhost:7296` — the loopback interface — so the request never left the machine and never touched TLS or DNS. + +On LiteSpeed that constant is deliberately left **undefined**, so `getFullBase()` falls through to `app.baseURL`. Which means every report generation now makes a **real outbound HTTPS request to the app's own public URL** and comes back in through Cloudflare. + +Which raises the obvious question: does that work at all? + +The answer is yes, but it is worth testing explicitly, because the failure mode is confusing. Here is the exact test — run it **on the server**, not from your laptop: + +```bash +curl -s -o /dev/null -w 'HTTPS self-call: %{http_code}\n' \ + -X POST 'https:///api/single' -d 'nameCn=test&dob=1990-01-01' +``` + +Two things to watch for: + +- **Run it from the server.** From my Windows machine the same URL returned 404 and 500 — an artefact of DNS resolution and Cloudflare bot protection on my caller, not a server problem. Testing from the wrong host produces confident, wrong conclusions. +- **Watch the JSON, not just the status code.** A 200 with a truncated body is worse than a clean failure. I checked the response was parseable before trusting it. + +There is a real trade-off hiding here, and I have not fully resolved it: a public-URL self-call means report generation depends on Cloudflare being up, costs a TLS handshake per report, and occupies two PHP workers for the duration. On LiteSpeed with a small worker pool, a burst of concurrent reports could deadlock — each request waiting on another request that has no free worker. For the current traffic level it is fine. Before this scales, the self-call should move to an internal path that bypasses the CDN. + +## Step 4: Verify the actual output, not the status code + +A 200 on `/` proves the landing page renders. It does not prove the report engine works, and the engine is the entire product. So the last step was generating a real report end to end: + +```bash +curl -s -X POST 'https:///read/single' \ + -d 'nameCn=test&dob=1990-01-01&gender=m' \ + -o report.html -w 'HTTP:%{http_code} bytes:%{size_download}\n' +``` + +The result: **HTTP 200, 99,189 bytes, 19 A4 pages.** I ran it again with Chinese-character input to exercise the UTF-8 path through the engine — HTTP 200, 99,205 bytes, 19 pages, name rendering correctly, zero PHP warnings in the output. + +Byte-comparable page counts between the old and new host is the check that matters. It proves the fonts, the DPI, and the pagination logic all survived the move. + +A side note on the UTF-8 test: my first attempt at it returned a 500 and I nearly went bug-hunting. The cause was my own shell — the Chinese characters were being mangled by the terminal encoding on the way into `curl`, so the app received invalid bytes. Running the same request from a script on the server, where the encoding is under my control, returned 200. **Before you debug an encoding failure, confirm the bytes actually arriving at the server are the bytes you meant to send.** + +## What I'd do differently + +**Test unauthenticated routes before migrating, not after.** The whole two-hour debugging session was avoidable with a `curl` loop over the route list. I now treat "list every route, hit it unauthenticated, record the status" as step zero of any migration. + +**Do not call `env()` in `Constants.php`.** I have left a comment in the file itself saying so, because the next person — probably me in six months — will be tempted. + +**Check the framework version's project-space config before the move.** The same week, I upgraded CI4 from 4.6.3 to 4.7.4, and it fataled twice on properties the upgrade guide did not mention: `Config\App::$permittedURIChars`, required by the 4.7 Router, and `Config\Format::$jsonEncodeDepth`, required by the JSONFormatter — the second of which broke the engine's JSON self-call specifically. Composer updates `vendor/` but never merges `app/Config/*.php`, because those are project-space files. I ended up merging new properties into 14 config files. That is a separate post, but the migration lesson is the same shape: **the framework tells you what changed in `vendor/`; nothing tells you what changed in `app/`.** + +**Confirm the docroot before anything else.** If I had extracted into `public_html` and stopped, the app would have worked — and quietly served `.env` over HTTP. A migration that works is not the same as a migration that is safe. + +## The result + +One evening. Six routes verified 200, byte-comparable report output on both Chinese and ASCII input, zero PHP warnings, and two long-standing latent bugs removed from the codebase that IIS had been concealing behind an auth gate. + +The app is now running on OpenLiteSpeed with a properly separated docroot, on a host I can automate, with the credential in `.env` rather than a constant. + +--- + +**Want this done for your own application?** I migrate PHP applications between IIS, Apache, nginx and LiteSpeed — including the awkward parts: self-calling architectures, SSO gates that hide bugs, and framework upgrades that touch project-space config. + +[WhatsApp +60 12-797 2969](https://wa.me/60127972969) · [me@hoelee.com](mailto:me@hoelee.com?subject=CodeIgniter%20migration) · [hoelee.com](https://hoelee.com) +```` + +--- + +## 文章 09 / 11:用「热浏览器会话」爬取有反爬墙的电商网站 +- **Slug**: `scraping-bot-walled-marketplace-warm-browser-session` +- **英文标题**: Scraping a Bot-Walled Marketplace With a Warm Browser Session +- **分类**: `engineering` | **发布日期**: 2026-09-13 +- **线上地址**: https://blog.hoelee.com/posts/scraping-bot-walled-marketplace-warm-browser-session/ (EN) | https://blog.hoelee.com/posts/zh/scraping-bot-walled-marketplace-warm-browser-session/ (ZH) +- **本篇审核状态**: ⏳ 待审核 (Pending) + +### ✍️ 顾问审阅意见区 (Review Feedback) +```text +【审阅结论】:[ ] 通过 [ ] 需微调 [ ] 需重构 +【事实与数字核对】: +【人设与用词卫生】: +【CRO与转化建议】: +``` + +### 🇨🇳 中文版原文 (Chinese Markdown Twin) +````markdown +--- +title: "用「热浏览器会话」爬取有反爬墙的电商网站" +description: "为一位做二手手机生意的客户:爬取有反爬墙的电商商品列表——热 CDP 会话、失败即止的健康检查、分段等待渲染。" +pubDate: 2026-09-13 +category: engineering +tags: ["scraping", "cdp", "headless-chrome", "anti-bot", "shopee", "python"] +ogImage: "/og/zh-scraping-bot-walled-marketplace-warm-browser-session.jpg" +banner: "/banners/scraping-bot-walled-marketplace-warm-browser-session.jpg" +draft: false +--- + +一位客户请我做一个听起来很简单的东西:每天把一个大电商平台上的二手手机列出来,按价格排序,好让他的进货决策靠数据而不是靠感觉。这位客户靠倒卖二手电子产品吃饭——一台手机差几百马币就是他的利润。 + +结果完全不是一段干净的 `requests` + BeautifulSoup 脚本。它变成了一场长达一周的、跟反爬墙的拉锯战,一个靠胶带勉强粘在一起的热浏览器会话,以及——我们俩都没想到的部分——一个关于「你不能靠堆更多 agent 来『并行化』爬虫」的惨痛教训。 + +这就是那个故事,按照我经历的顺序来讲:问题、我试过却失败的每一件事、最终奏效的办法,以及下次我会怎么做。 + +## 问题,用别人真正会搜索的方式表述 + +> "如何从一个拦截无头浏览器的网站爬取商品列表?" + +或者更诚实地说:**当一个电商网站已经把「防爬」当成它唯一的本职工作,你要怎么爬它?** Shopee 不会礼貌地把 HTML 喂给 `curl`。它会检测自动化、抛出一个 `/verify` 验证码,然后静默地给你一个「看起来正常但里面什么都没有」的页面。 + +客户的需求清单很短: + +- 搜索二手手机,只筛选「二手」成色,存储 128 GB 及以上。 +- 提取每个商品的所有存储规格和价格。 +- 按价格升序排序。 +- 不搞一百个 headless 实例的 `chromedriver` 农场——倒卖的利润养不起一百个住宅 IP,我的也一样。 + +从这里开始,全部都是调试故事。 + +## 我试过的,以及为什么失败 + +### 尝试 1:通过 CDP 用冷的无头 Chrome——「直接把 Chrome 远程调试开起来」 + +我用 `--headless --remote-debugging-port=9222` 启动了 Chrome,用 Chrome DevTools Protocol 连上去,然后导航到搜索 URL。大约十秒钟里我觉得自己很聪明。 + +**失败:** 页面加载了,但每个商品卡片都是空的。DOM 是个空壳。Shopee 指纹识别了这个无头浏览器(没有真正的 GPU、user-agent 指纹不一致、`navigator.webdriver` 标志),把我重定向进了一个我永远过不去的 `/verify` 验证码循环。冷的无头 profile 一上来就撞墙。 + +教训很无聊但很重要:**如果你的浏览器本身在「身份」上撒谎,选择器写得再好也没用。** + +### 尝试 2:并行子 agent——「直接开 N 个 worker」 + +这是我最想让它成功的方案。我有一套多 agent 的架构。本能反应就是:开五个 agent,每个爬一个不同的搜索词,收集结果,合并。 + +**失败:** 它们全都连到**同一个**浏览器会话上。它们不会各自拿到一个浏览器——只会各自拿到共享 Chrome 实例里的一个标签页,或者更糟:全都挤在同一个标签页上。每一次 `Page.navigate` 都会把上一个 agent 的上下文抹掉。而且关键在于: + +1. **热会话是共享资源。** 只有一个没被标记的 Canary profile。五个 agent 交错访问同一个标签页,在 Shopee 眼里,跟一个浏览器疯狂快速导航没有区别。而这恰恰是**触发**反爬墙的行为。 +2. **反爬的「节奏」是累积的。** 我的可用方案依赖导航之间约 7 秒的间隔来显得像真人。五个 agent 并发导航,就等于五个导航以零间隔打向同一个会话。 + +所以「横向扩展」的本能——加更多 worker——对我赖以存活的那件事(看起来像真人)是有害的。这里的并发上限不是 CPU 也不是内存,而是**一个热浏览器,一次一个标签页。** + +### 尝试 3:靠「Buy With Voucher」按钮来拿价格 + +一旦列表能渲染出来了,我假设购买按钮的文案里带着价格。大多数电商网站确实如此。但这个渲染出来的按钮写的是 `Add To Cart` / `Buy Now`。我用来匹配「Buy With Voucher … RM…」的正则什么都匹配不到,我在原地打转了好一会儿,才真正去读了 DOM。 + +**失败:** Shopee 商品页的价格不在按钮里。它被渲染成**拆分的 span**——`RM` 在一个元素里、`850` 在另一个里、`00` 又在另一个里——所以朴素的叶子文本匹配只会抓到碎片,或者当商品有多个规格时,抓到一个**区间**(`RM850.00 - RM1150.00`)。 + +### 尝试 4:点击规格选项来拿精确价格 + +对于单轴规格选择器(比如一个存储下拉:128GB / 256GB / 512GB),点击一个选项会把价格收敛成单个值,我的提取方法能正常工作。 + +**失败:** 对于**双轴**选择器(品牌 × 存储),点击第一个轴(某个品牌)会锁定一个价格,但加上第二个轴(某个存储规格)又把它**解散**回商品全区间。这个面板渲染的是「所有可购买组合」的区间,而不是我选中的那个具体组合;而且「已选中」状态是用一个「同时匹配其他所有选项框」的 class 来标记的。我没法可靠地知道我到底锁定了哪个组合。 + +我在这个上面花了实打实的时间,最后才接受这个诚实的答案:**对于真正的双轴商品,正确的数据是价格区间,而不是一个瞎猜的「每个组合」数字。** 发布一个「有时候能用」的脆弱覆盖方案,比老实报告一个区间更糟糕。 + +## 修复——真正奏效的东西 + +三块,都很小,都是吃了亏才学会的。 + +### 1. 热的人性化节奏的浏览器会话(地基) + +别跟墙硬刚。用一个已经登录过、正常浏览过的浏览器 profile——一个「热」profile。用真实的 7 秒间隔来导航。它更慢,但它是唯一能稳定活下来的方式。 + +```python +# 不是真正的代码——是它的「形状」。先导航,然后像真人一样等待。 +goto_url(target) +time.sleep(7) # 真人节奏,不可妥协 +wait_for_selector(".shopee-search-item-result__item") +``` + +### 2. 失败即止,别傻等超时 + +在任何爬取之前,先打 CDP 的 HTTP 端点,检查会话状态。如果 Canary 挂掉了、或标签页停在 `/verify` 上,就在约 1 秒内中止,而不是在「怎么还不渲染」的超时里浪费 40 秒。 + +```python +import json, urllib.request + +def session_healthy(): + try: + tabs = json.load(urllib.request.urlopen("http://127.0.0.1:9222/json")) + except Exception: + return False + for t in tabs: + if t.get("type") == "page" and "/verify" in t.get("url", ""): + return False # 撞到验证码墙——重新热会话,别爬了 + return True +``` + +就这么一个改动,把「静默的 40 秒死胡同」变成了「即时、可执行的失败」。 + +### 3. 正确的价格选择器:最深的纯 RM 节点 + +购买按钮是个谎言。价格是**那个文本内容纯粹是一个 RM 值的、最深的元素**——在选中规格之前是一个区间(`RM850.00 - RM1150.00`),之后是单个值(`RM850.00`)。遍历 DOM,取文本能匹配上的最深元素: + +```python +import re + +RM = re.compile(r"^RM\s?[\d,.]+(\s?[-–]\s?RM\s?[\d,.]+)?$") + +def find_price(root): + best, depth = None, -1 + for el in root.querySelectorAll("*"): + txt = (el.textContent or "").strip() + if txt and RM.match(txt) and " " not in txt.split("RM")[0]: + d = depth_of(el) + if d > depth: + best, depth = el, d + return best.textContent.strip() if best else None +``` + +这就是全部的诀窍。没有脆弱的 class 名。没有瞎猜。一个「整个文本就是一个价格」的节点,就是价格本身。 + +## 结果——一个可量化的成果 + +从单个热会话、一次串行遍历中,我提取了 **20+ 个二手手机商品及其各规格价格**,筛选到 128 GB 及以上,按价格升序排序。最便宜的真实目标——一台 128GB 机型——以 **RM299** 的价格浮出水面,另外还有几台 128/256GB 的选项在 RM450–880 区间。现在一条命令(`sweep.py ...`)就能重新检查整份候选清单并输出 JSONLines;客户每天进货前跑一次。 + +但我最庆幸的数字是这个:**零验证码。** 「热会话 + 人性化节奏 + 失败即止」这套组合,在整次遍历中一次都没触发过 `/verify`。 + +## 下次我会怎么做 + +1. **我会一开始就写「失败即止」的健康检查**,在任何爬取代码之前。三十秒的活,换来未来每次失败省四十秒。 +2. **我会从第一天就抵抗并行化的冲动。** 「爬虫太慢」的答案是把更多工作打包进一次串行遍历(批量搜索、对多个 URL 跑 `sweep.py`),而不是「再加 worker」。共享的浏览器状态让并发变得更糟,而不是更好。 +3. **我会立刻接受双轴的限制**,老实报告区间,而不是在一个根本没法可靠化的选择器上烧一个下午。 +4. **我会别再猜 DOM 结构,先去探测它。** 几乎每一个错误假设(URL 路径里 `/i.` vs `-i.`、把 `Buy With Voucher` 当价格锚点、「过滤器是个真正的 `` 复选框」)都来自**没有先读真实的 DOM**。 + +## 可复用的部分 + +所有这一切被做成了一个小的独立工具集——每个脚本都是一次自包含的 `python x.py ...` 调用: + +| 脚本 | 作用 | +|---|---| +| `search.py` | 粗粒度搜索 → 商品卡片 | +| `used_search.py` | 点击「二手」条件复选框,校验 URL 里的 `USED_ITEM`,再提取 | +| `detail.py` | 从商品页提取各规格价格(最深 RM 节点法 + 分段等待) | +| `sweep.py` | 一次会话内对 N 个 URL 做批量详情提取,按 item-id 去重,写 JSONLines,打印表格 | +| `cdp_common.py` | 共享的健康检查 + 导航辅助 | + +真正可复用的教训不是那些 Shopee 专属的选择器——而是**姿态**:**假设页面在 DOM 上撒谎,失败即止,用批量化而不是并行 worker 来扩展。** + +--- + +*这是一次真实的调试过程,不是事后补写的教程。这个爬虫工具集、以及这里的每一次失败,都发生在为一个做二手电子产品倒卖的客户构建它的过程中——我保留了能泛化的部分,砍掉了那些只对某个马来西亚电商网站、某个时间点有效的部分。* + +--- + +## 想为你的业务做这个吗? + +我做定制爬虫和数据提取管道——热会话 CDP 爬虫、反爬墙感知的采集器、批量价格监控扫描——也做全栈 Web 应用(PHP/CodeIgniter、Java/Spring、React/TypeScript)和自托管自动化(n8n、Telegram 机器人、NAS 上的 Docker)。 + +如果你需要从某个「会反抗」的网站拿商品或价格数据,直接点进来,告诉我你想提取什么: + +- 💬 [WhatsApp](https://wa.me/60127972969) — 直接私信我 +- ✉️ [Email](mailto:me@hoelee.com?subject=Custom%20scraping%20pipeline) — 告诉我网站和数据 +- 🌐 [hoelee.com](https://hoelee.com) — 看看我还做了些什么 +```` + +### 🇬🇧 英文版原文 (English Markdown Primary) +````markdown +--- +title: "Scraping a Bot-Walled Marketplace With a Warm Browser Session" +description: "Scraping bot-walled marketplace listings for a client's used-electronics business: warm CDP session, fail-fast health checks, split-wait rendering." +pubDate: 2026-09-13 +category: engineering +tags: ["scraping", "cdp", "headless-chrome", "anti-bot", "shopee", "python"] +ogImage: "/og/scraping-bot-walled-marketplace-warm-browser-session.jpg" +banner: "/banners/scraping-bot-walled-marketplace-warm-browser-session.jpg" +draft: false +--- + +A client asked me to build something that sounds simple: every day, list the second-hand phones on a big marketplace, sorted by price, so their restocking decisions come from data instead of gut feel. The client resells used electronics for a living — a few hundred ringgit either way on a phone is the whole margin. + +The result was nothing like a clean `requests` + BeautifulSoup script. It was a week-long fight against an anti-bot wall, a warm browser session held together with duct tape, and — the part neither of us saw coming — a hard lesson about why you cannot just "parallelize" a scraper by throwing more agents at it. + +This is that story, told in the order I lived it: the problem, everything I tried that failed, the thing that finally worked, and what I'd do differently. + +## The problem, phrased the way someone would actually search it + +> "How do I scrape product listings from a site that blocks headless browsers?" + +Or, more honestly: **how do you scrape a marketplace that has already made scraping its sole job description?** Shopee doesn't politely serve HTML to `curl`. It detects automation, throws a `/verify` captcha, and silently serves you a page that looks fine but contains nothing. + +The client's requirement list was small: + +- Search for used phones, filter to "used condition only", storage 128 GB and up. +- Extract every listing's storage variants and prices. +- Sort by price ascending. +- Do it without a `chromedriver` farm of a hundred headless instances — a reseller's margins don't pay for a hundred residential IPs, and neither do mine. + +Everything after this point is the debugging story. + +## What I tried, and why it failed + +### Attempt 1: cold headless Chrome via CDP — "just point Chrome's remote debugger at it" + +I launched Chrome with `--headless --remote-debugging-port=9222`, connected with the Chrome DevTools Protocol, and navigated to a search URL. It felt clever for about ten seconds. + +**Failure:** the page loaded, but every product card was empty. The DOM was a shell. Shopee had fingerprinted the headless browser (no real GPU, no real user-agent fingerprint consistency, `navigator.webdriver` flag) and redirected me into a `/verify` captcha loop I never got past. Cold headless profiles hit the wall immediately. + +The lesson is boring but important: **anti-bot systems don't care how good your selectors are if the browser itself is lying about what it is.** + +### Attempt 2: parallel subagents — "just spawn N workers" + +This was the one I really wanted to work. I have a multi-agent setup. The natural instinct is: spawn five agents, each scraping a different search query, gather the results, merge. + +**Failure:** all of them attach to the *same* browser session. They don't get their own browser — they get their own *tab* in one shared Chrome instance, or worse, they all fight over the same single tab. Every `Page.navigate` un-does the previous agent's context. And critically: + +1. **The warm session is a shared resource.** There is one Canary profile that isn't flagged. Five agents navigated the same tab in an interleaved order, which is indistinguishable — to Shopee — from one browser doing rapid-fire navigation. That's exactly the behavior that *triggers* the wall. +2. **Anti-bot pacing is cumulative.** My working setup depends on ~7-second gaps between navigations to look human. Five agents navigating concurrently means five navigations hitting the same session with effectively zero gap. + +So the "horizontal scaling" instinct — more workers — is actively hostile to the one thing keeping me alive: looking human. The concurrency ceiling here isn't CPU or memory. It's **one warm browser, one tab at a time.** + +### Attempt 3: trusting the "Buy With Voucher" button for the price + +Once I had listings rendering, I assumed the buy button's label contained the price. On most marketplaces it does. On this render, the button said `Add To Cart` / `Buy Now`. My regex for "Buy With Voucher … RM…" matched nothing, and I went around in circles for a while before actually reading the DOM. + +**Failure:** the price on Shopee product pages isn't in the button. It's rendered as **split spans** — `RM` in one element, `850` in another, `00` in a third — so a naive leaf-text match returns fragments, or a **range** (`RM850.00 - RM1150.00`) when the listing has multiple variants. + +### Attempt 4: clicking variant options to get an exact price + +For a single-axis variant selector (e.g. one dropdown of storage options — 128GB / 256GB / 512GB), clicking one option collapses the price to a single value, and my extraction worked. + +**Failure:** for a **two-axis** selector (brand × storage), clicking the first axis (a brand) locks a price, but adding the second axis (a storage size) *un-collapses* it back to the product-wide range. The panel renders the range of all purchasable combinations, not the exact combination I selected, and the "selected" state is marked by a class that also matches every other option box. There is no reliable way to know which combination I actually locked. + +I spent real time on this before accepting the honest answer: **for genuinely two-axis listings, the correct data is the price range, not a guessed per-combination number.** Shipping a fragile override that "sometimes works" would be worse than reporting an honest range. + +## The fix — what actually worked + +Three pieces, all small, all learned the hard way. + +### 1. A warm, human-paced browser session (the foundation) + +Don't fight the wall. Use a browser profile that has already logged in and browsed normally — a "warm" profile. Navigate with real 7-second gaps. It is slower and it is the only thing that reliably survives. + +```python +# not real code — the *shape* of it. Navigate, then wait like a human. +goto_url(target) +time.sleep(7) # human pacing, non-negotiable +wait_for_selector(".shopee-search-item-result__item") +``` + +### 2. Fail fast, don't time out + +Before any scrape, hit the CDP HTTP endpoint and check the session state. If Canary is down or the tab is sitting on `/verify`, abort in ~1 second instead of wasting 40 seconds on a "did not render" timeout. + +```python +import json, urllib.request + +def session_healthy(): + try: + tabs = json.load(urllib.request.urlopen("http://127.0.0.1:9222/json")) + except Exception: + return False + for t in tabs: + if t.get("type") == "page" and "/verify" in t.get("url", ""): + return False # captcha wall — re-warm the session, don't scrape + return True +``` + +This one change turned "silent 40-second dead-ends" into "instant, actionable failure." + +### 3. The right price selector: deepest pure-RM node + +The buy button is a lie. The price is the **deepest element whose text is a pure RM value** — a range (`RM850.00 - RM1150.00`) before a variant is selected, a single value (`RM850.00`) after. Walk the DOM and take the deepest element whose text matches: + +```python +import re + +RM = re.compile(r"^RM\s?[\d,.]+(\s?[-–]\s?RM\s?[\d,.]+)?$") + +def find_price(root): + best, depth = None, -1 + for el in root.querySelectorAll("*"): + txt = (el.textContent or "").strip() + if txt and RM.match(txt) and " " not in txt.split("RM")[0]: + d = depth_of(el) + if d > depth: + best, depth = el, d + return best.textContent.strip() if best else None +``` + +That's the whole trick. No brittle class names. No guessing. A node whose entire text *is* a price is the price. + +## The result — one quantified outcome + +From a single warm session, in one serial pass, I extracted **20+ used-phone listings with per-variant prices**, filtered to 128 GB and up, sorted by price. The cheapest real target — a 128GB model — surfaced at **RM299**, alongside several 128/256GB options in the RM450–880 range. One command (`sweep.py ...`) now re-checks the whole shortlist and dumps JSONLines; the client runs it once a day before restocking. + +But the number I'm most glad about is this: **zero captchas.** The warm-session + human-pacing + fail-fast combo survived the entire sweep without tripping `/verify` once. + +## What I'd do differently + +1. **I'd write the fail-fast health check first**, before a single scraping line. It's thirty seconds of work that saves forty seconds per future failure. +2. **I'd resist the parallelization instinct day one.** The answer to "the scraper is slow" is "bundle more work into one serial pass" (batch search, `sweep.py` over many URLs), not "more workers." Shared browser state makes concurrency worse, not better. +3. **I'd accept the two-axis limitation immediately** and report ranges, instead of burning an afternoon on a selector that can't be made reliable. +4. **I'd stop guessing DOM structure and probe it.** Almost every mistaken assumption (`/i.` vs `-i.` in the URL path, `Buy With Voucher` as the price anchor, "the filter is a real `` checkbox") came from *not reading the actual DOM first*. + +## The reusable pieces + +All of this became a small, standalone toolkit — each script is one self-contained `python x.py ...` call: + +| Script | What it does | +|---|---| +| `search.py` | Coarse search → listing cards | +| `used_search.py` | Clicks the "Used" condition checkbox, verifies `USED_ITEM` in the URL, extracts | +| `detail.py` | Pulls per-variant prices from a product page (deepest-RM-node method + split-wait) | +| `sweep.py` | Batch detail over N URLs in one session, dedupes by item-id, writes JSONLines, prints a table | +| `cdp_common.py` | Shared health check + navigation helpers | + +The genuinely reusable lesson isn't the Shopee-specific selectors — it's the *posture*: **assume the page is lying about its DOM, fail fast, and scale by batching, not by parallel workers.** + +--- + +*This is a real debugging session, not a tutorial written after the fact. The scraping toolkit and every one of these failures happened while building it for a client who resells used electronics — I kept the parts that generalize and cut the parts that only matter to one Malaysian marketplace at one point in time.* --- ## Want this for your business? -I build automated video generation pipelines, custom music production workflows, and self-hosted infrastructure for creators, educators, and brands. +I build custom scraping and data-extraction pipelines — warm-session CDP scrapers, bot-wall-aware crawlers, and batch price-monitoring sweeps — as well as full-stack web apps (PHP/CodeIgniter, Java/Spring, React/TypeScript) and self-hosted automation (n8n, Telegram bots, Docker on NAS). -If you want to automate your media workflow or need custom full-stack tools: -- **WhatsApp**: [Chat directly on WhatsApp](https://wa.me/60127972969) -- **Email**: [me@hoelee.com](mailto:me@hoelee.com?subject=Automated%20Video%20Pipeline%20Inquiry) -- **Website**: [hoelee.com](https://www.hoelee.com/) +If you need product or price data off a site that fights back, tap through and tell me what you're trying to extract: + +- 💬 [WhatsApp](https://wa.me/60127972969) — message me directly +- ✉️ [Email](mailto:me@hoelee.com?subject=Custom%20scraping%20pipeline) — tell me the site + the data +- 🌐 [hoelee.com](https://hoelee.com) — see what else I build ```` --- -## 文章 06 / 08:我是如何构建 DigiKedai Telegram AI 客服机器人的 -- **Slug**: `how-i-built-the-digikedai-telegram-bot` -- **英文标题**: How I Built the DigiKedai Telegram AI Bot -- **分类**: `case-studies` | **发布日期**: 2026-09-06 -- **线上地址**: https://blog.hoelee.com/posts/how-i-built-the-digikedai-telegram-bot/ (EN) | https://blog.hoelee.com/posts/zh/how-i-built-the-digikedai-telegram-bot/ (ZH) +## 文章 10 / 11:JavaScript 文件静默失效的两种方式:而页面看上去一切正常 +- **Slug**: `two-ways-a-javascript-file-stops-working-and-looks-fine` +- **英文标题**: Two Ways a JavaScript File Stops Working and Looks Fine +- **分类**: `engineering` | **发布日期**: 2025-02-20 +- **线上地址**: https://blog.hoelee.com/posts/two-ways-a-javascript-file-stops-working-and-looks-fine/ (EN) | https://blog.hoelee.com/posts/zh/two-ways-a-javascript-file-stops-working-and-looks-fine/ (ZH) - **本篇审核状态**: ⏳ 待审核 (Pending) ### ✍️ 顾问审阅意见区 (Review Feedback) @@ -3557,1502 +3346,795 @@ If you want to automate your media workflow or need custom full-stack tools: ### 🇨🇳 中文版原文 (Chinese Markdown Twin) ````markdown --- -title: "我是如何构建 DigiKedai Telegram AI 客服机器人的" -description: "一个跨 Telegram 与 WhatsApp 解答疑问并全自动开通免费试用账号的 AI 客服机器人——TypeScript、grammY、Cloudflare 隧道 webhook,以及自建 LiteLLM 网关。" -pubDate: 2026-09-06 -category: case-studies -blueprint: ai-agent-automation -blueprintLayer: L1-deterministic-orchestration -tags: [telegram, docker, cloudflare, litellm, typescript, ai] -ogImage: /og/zh-how-i-built-the-digikedai-telegram-bot.jpg -banner: /banners/how-i-built-the-digikedai-telegram-bot.jpg -summary: - audience: "企业主与产品/运营负责人" - painPoint: "一人数字商品电商面临大量重复售前咨询(试用申请、账号开通、选品),夜间与周末漏单严重,人工客服成本高。" - impact: - - metric: "24/7 秒级响应" - detail: "全天候接管 Telegram 与 WhatsApp 双端咨询,零人工介入自动发放免费试用账号" - - metric: "0 价格幻觉" - detail: "三层确定性 SKU 检索 + 状态机拦截购买意图,杜绝大模型胡报价或泄漏内部路径" - - metric: "~175 项单测护航" - detail: "单容器稳定运行于 NAS,通过自建 LiteLLM 网关实现模型高可用切换" - stack: ["TypeScript", "grammY", "LiteLLM", "NocoDB", "Cloudflare Tunnel", "PostgreSQL"] - jumpLinks: - business: "#why-a-business-needs-a-bot-like-this" - architecture: "#the-architecture" - result: "#the-result" -dynamicCta: true +title: "JavaScript 文件静默失效的两种方式:而页面看上去一切正常" +description: "一个被参数名遮住的函数名,加一个声明在首次调用之后的 let,让整段脚本停摆——而页面渲染得完美好看,因为屏幕上什么都没变。" +pubDate: 2025-02-20 +updatedDate: 2026-10-16 +category: engineering +tags: ["javascript", "i18n", "debugging", "frontend"] +ogImage: /og/zh-two-ways-a-javascript-file-stops-working-and-looks-fine.jpg +banner: /banners/two-ways-a-javascript-file-stops-working-and-looks-fine.jpg +draft: false --- -Digi Kedai 销售数字产品——在线课程、电子书、模板——每一单都会带来一连串同样的顾客问题:*「这门课有免费试看吗?」「我怎么拿到账号?」「我该买哪个套餐?」*。对一个人的生意来说,人工回复根本忙不过来。于是我打造了一个 AI 客服机器人,全天候解答这些问题,甚至能在无人介入的情况下自动发放免费试用账号。 +## 一次「什么都没发生」的改动 -这篇讲的是它是怎么搭起来的——架构、LLM 接线,以及那些各自吃掉我一整个下午的 bug。 +我给一个自己用的媒体转码队列看板加了语言切换:三份语言包、一个 `t(key, vars)` 函数、界面 chrome 上的 `data-i18n`、顶栏一个 `` in the top bar. -Two reasons, one personal and one practical. +After deploying it, the page looked exactly as it had before. Not broken — +*unchanged*. Same layout, same Chinese labels, no missing panels, no error +card, no console banner. The switcher was simply not there, and neither was +anything else the new code was supposed to do. -**Personally:** I like experimenting with creative AI. A song is a tight, -self-contained project — lyrics, style, structure, performance — that -exercises the same prompt-crafting muscles as building a bot or wiring a -pipeline. And the output is shareable in a way a config file never is. +That is the shape of bug I want to write down, because the page was not +"mostly working". **The script had stopped executing partway through — +twice, for two unrelated one-line reasons — and the page rendered perfectly +anyway.** -**Practically:** this is now a client skill. Small businesses need jingles, -background music for videos, event music, and branded audio — without a -studio budget. If I can deliver a usable track from a one-page brief, -that's a service, not a party trick. Suno's current **v6** models (released -Sep 9, 2026) made this dramatically more controllable than the V3-era -version I started with in April 2024. +## Why nothing looked broken -## The current standard: what Suno's models look like in 2026 +Two design decisions I would make again, and both of them hid the failure. -Version confusion is the first thing to clear up, because old guides -online reference models that don't exist anymore: +The first is that the default language lives in the markup. The Chinese text +is written inside the elements; `data-i18n` is only a hook that *overrides* +it when you switch. That is deliberate — the page needs no fetch to render, +and if a language pack goes missing you get Chinese instead of a row of +`top.theme_to_dark` keys. It also means a completely dead i18n layer +produces a page that is, visually, correct. -| Model | When | Who gets it | -|---|---|---| -| **v6 / v6-wild** | Sep 9, 2026 | Paid plans (flagship + experimental) | -| **v6-mini** | Sep 9, 2026 | Everyone — fast, efficient, great for drafts | -| **v5.5** | Mar 26, 2026 | Paid — added Voices, Custom Models, My Taste | -| **v4.5-all** | Oct 21, 2025 | Free plan (no commercial rights, attribution required) | -| V4 / V4.5 / V5 | 2024–2026 | Paid plans, superseded | +The second is that the buttons are wired with inline handlers +(`onclick="rtFoo()"`), which after a script abort point at functions that +were never defined. Nothing warns you. The page only reveals the truth when +a human clicks something. -The v6 family added the control features that changed how I work: +So the symptom was "my change had no effect", which is the least informative +symptom a web page can give you. -- **Plain-language section edits** — "change the chorus so it's sung by a - gospel choir" edits one section without regenerating the song. -- **Single lyric swaps** — "change 'love' to 'light'" updates one line. -- **Mashups** — combine vocals from one song, drums from another, new - lyrics, in one request. -- **Sample → isolate → rebuild** — pull a riff at 0:45, isolate the - guitar, build a beat around it. -- **Multimodal input** — start from text, audio, an image, or even a video. +## Failure one: a parameter that swallowed a function -My own tracks below were made with the earlier V4-era workflow (April -2025). The fundamentals — style field, lyric metatags, iteration — are -unchanged, and I note where v6 would have saved me steps. +The file has a theme block and an i18n block. The theme block had this +signature: -## My workflow: lyrics first, style second - -The order matters more than people think. Most beginners open Suno, paste -a genre into Simple Mode, and get a generic song. I do the opposite: - -1. **Write the song as a document first** — title, structure, lyrics. The - emotion and the story come from me; Suno supplies the performance. -2. **Craft the style prompt** — genre + mood + era + instruments + vocal - persona + production, often 200+ characters, describing the *journey* - of the song, not just its genre. -3. **Add metatags to the lyrics** — `[Verse]`, `[Chorus]`, `[Bridge]`, - `[Whispered]`, `[Belted]`. Without these, Suno defaults to a flat - verse-chorus-verse with no emotional arc. -4. **Generate 3–5 takes** per song and keep the best. Repair via - Extend/Reuse Prompt, and restate the genre in every extension because - style drifts. -5. **Iterate on pronunciation** — AI singers read phonetically, so I - respell ("through" → "thru", hyphenate unusual syllables) and always - test proper nouns in a short clip first. - -The worked example below shows exactly what this looks like. - -## The songs - -Seven tracks, hosted on my own media subdomain (`content.hoelee.com` via -Publitio) and embedded with plain HTML5 audio. Press play: - - - -### 1. 轮回的渡船 — The Ferry of Reincarnation - -Chinese folk ballad (古风民谣). The ferryman of the River of Forgetfulness, -Meng Po's soup gone cold, a soul that refuses to reboard the wheel of -rebirth. Written as a mythic love story about choosing someone across -lifetimes. Starts hushed over sparse strings and builds to a full, -anguished chorus. - -
- - -
- -### 2. Happy Way to School - -English children's song — the original prompt called for a "fizzy, -orange-soda indie folk" sound: acoustic guitar, playful whistling, and -bright kid-friendly vocals. It's the walk-to-school ritual turned into a -tiny anthem: ponytails bouncing, kittens at the café, friends in step. - -
- - -
- -### 3. 新靓 — Heart Sutra, Meditative Beats - -The 般若波罗蜜多心经 (Heart Sutra) chanted over rhythmic, meditative -production — the brief was "meditation depth + groove energy". 观自在菩萨 -opening, the full 色不異空 passage, ending on 揭諦揭諦. An attempt to make -ancient scripture something you can actually move to. - -
- - -
- -### 4. 新劲 — Fresh Energy - -A harder-edged companion to 新靓 — same meditative lineage, more percussive -drive. No lyric sheet on this one; it lives as a vibe piece. - -
- -
- -### 5. 泡泡星球漫游记 — Bubble Planet Adventure - -Kids' indie folk, and the one with the most documented backstory — the -full original brief, style prompt, and lyric sheet are in the worked -example below. Bubble planets, candy fountains, a galactic playground. - -
- - -
- -### 6. 日漫心经 — Heart Sutra, Anime Style - -The same classical text, reimagined with an anime-inspired sound palette — -big chorus energy, dramatic key lifts, the emotional grammar of -contemporary J-pop openings. - -
- -
- -### 7. 未接来电 Unread Heartbeats (v2) - -A mandopop ballad about that one missed call: the dial tone as a heartbeat, -the read receipts that stay unread. v2 is the final master — tighter -arrangement, cleaner vocal mix. (The original take stays in my archive.) - -
- -
- - - -## Worked example: Bubble Planet Adventure, from brief to track - -This is the full chain for 泡泡星球漫游记 — the only one I kept every -scrap of paperwork for. It shows the workflow above in concrete form. - -**Step 1 — the concept brief (my notes, in Chinese):** - -> 独立民谣(Indie Folk):木吉他+口哨声组合,适合突出童趣感 -> 人声选择「精灵少女」音色库,自带俏皮气声 -> 打击乐:玻璃瓶敲击音效呼应「橘子汽水」意象 -> 副歌加入反向混响(reverse reverb)增强魔法感 - -**Step 2 — the English style prompt (what actually went into Suno):** - -```text -Indie folk vibe with acoustic guitar and playful whistles, fizzy like -orange soda, featuring sprite-girl vocals and clinking glass bottle -beats, reverse reverb brings dreamlike magic +```js +function applyTheme(t) { // t = the theme + label.textContent = t('top.theme_to_dark'); // TypeError: t is not a function +} ``` -Four decisions compressed into one line: the *instrument* (acoustic -guitar + whistles), the *feel* (fizzy like orange soda), the *voice* -(sprite-girl with airy delivery), and the *production trick* (reverse -reverb on the chorus for the magic feel). +`applyTheme` was written before the i18n layer existed, and `t` was a +perfectly reasonable name for "theme". When the i18n layer arrived, `t` in +that scope stopped meaning the theme and started meaning *the function I +wanted to call* — except the parameter shadows it, so the call became +"invoke a string". -**Step 3 — the lyric sheet with metatags:** +A `TypeError` at the top level of a classic script does not just fail one +line. Everything after it never runs: the language-pack loader, the +`data-i18n` sweep, the switcher rendering. One wrong letter removed an +entire feature, silently. -
-Bubble Planet Odyssey — Metatagged lyric sheet (38 lines) +The fix is one character — rename the parameter to `th` — plus a comment +where the next person will look: -```text -[Intro][Whistling][Indie][Girl] -彩云兜着阳光转圈圈 -贝壳装满彩虹的碎片 -跳进橘子汽水的夏天 -海鸥掠过浪花的琴键 - -[Chorus][Indie Folk][Reverse Reverb][Whistling] -泡泡载着梦飞向屋檐(飞呀飞呀) -尾巴挂着星星的秋千(晃呀晃呀) -鲸鱼喷出糖果的喷泉(甜到脚尖) -魔法地图画满冒险线(转个圈圈) - -[Verse][Indie][Girl] -橡皮艇划开银河水面 -萤火虫点亮薄荷月圆 -棉花糖云朵蓬松柔软 -流星滑梯通向我窗前 - -[Chorus]… (repeat) - -[Interlude] - -[Bridge][Fantasy Pop][Giggles] -水晶风筝追着蝴蝶结 -跳跳糖在舌尖开派对 -彩虹滑板穿过梧桐叶 -月亮船摇着光的湖水 - -[Outro][Indie Folk][Girl] -每个气泡都是新世界 -装得下所有奇妙遇见 -在泡泡星球蹦跳向前 -明天又是崭新的乐园 - -[Fade to End] +```js +function applyTheme(th) { // NOT `t`: t() is the i18n function in this file ``` -
+## Failure two: a declaration that arrived too late -Note what the metatags enforce that a bare lyric sheet wouldn't: -section *labels* (`[Bridge]`, `[Outro]`), *delivery* (`[Whistling]`, -`[Giggles]`), a *production change* mid-song (`[Reverse Reverb]` only on -choruses), and a stylistic shift in the bridge (`[Fantasy Pop]`). The -fill-in parentheticals — 飞呀飞呀, 晃呀晃呀 — tell the singer exactly -how to ornament each line. +The second cause was in the same file, and it is a JavaScript rule people +know and still trip over. The theme block runs at load and calls `t('…')`. +The i18n block, further down the file, began with: -**Step 4 — generate, listen, repeat.** 3–5 takes per section, keep the -best, extend the chorus that landed. The final track plays above. +```js +let LANG = localStorage.getItem('rt_lang') || 'zh-CN'; +``` -## What I'd do differently with v6 +`let` and `const` are hoisted but **not initialised**: between the start of +the scope and that line they live in the temporal dead zone, and any access +throws `Cannot access 'LANG' before initialization`. Because the theme block +sits *above* the i18n block, the failure was not a race — it was guaranteed +on every single load. -If I made these today, four things change: +Note the distinction from `var`, which would have quietly given me +`undefined` and a working page. The TDZ is the language protecting me from +exactly the bug I was about to write; the only reason it looked like +"nothing happened" is the markup fallback again. -1. **Section edits instead of full rerolls.** 轮回的渡船's bridge took six - generations to nail. With v6 I'd say *"make the bridge slower and - sparser, just voice and a plucked guzheng"* and keep the rest intact. -2. **Single-line lyric swaps.** The 忘川 couplet went through four wordings; - v6 edits one line without rebuilding the song. -3. **Voices / custom models.** v5.5's Voices would let me keep one - consistent singer across all seven tracks instead of seven different - AI vocalists. -4. **Stems and Studio.** On the Premier plan, Studio (a browser DAW with - multitrack + MIDI export) means fixing a vocal artifact instead of - regenerating around it. +The fix is ordering, and ordering here is a *dependency*, not a style +choice: if a block calls `t()`, the i18n declarations have to be above it. +The file had been organised by topic ("theme, then i18n") because that is +how it grew. -The craft layer — decent lyrics, an emotional arc, honest metatags — -carries straight across model versions. The models get more controllable; -the songwriting stays the job. +## Why my checks did not catch either one + +I had tests. They checked intent, not effect: + +- the three language packs exist and parse; +- every key used in the markup exists in every pack; +- no pack contains a dead key (a typo detector); +- the `{var}` placeholder sets in the three packs match. + +All true. All green. None of them executes the page, so none of them can see +a script that aborts on line 12 of 900. Static checks are structurally +unable to catch "the file is valid and the program is dead". + +## Assert on effects, in a browser + +What found both bugs was asking the live page questions that only a running +program can answer: + +```js +typeof t === 'function' // is the entry point still callable? +document.querySelectorAll('[data-i18n]').length // hooks present +rtCheckHandlers() // do the inline handlers still resolve? +``` + +That last one is worth more than it looks. The page carries **219 inline +`onclick` attributes naming 217 distinct functions**; the check enumerates +them and asserts each name is defined on `window`. It turns a hand-clicked +smoke test into a two-second automated one, and it is the check that turns +"the page looks right" into a measurable property. + +The second tool is cheaper and nastier: a **pseudo-language**. +`?lang=pseudo` takes every string in the pack, prefixes it with `[~~` and +stretches it 1.6×. Then assert two things — no horizontal overflow at +1440px, and no element in the nav still lacks the `[~~` prefix. The first +catches layout that only breaks in a longer language; the second catches +nodes that never got a translation hook at all. It found a real one in the +same run. + +Honest scope note: a pseudo-locale does not prove your Malay translation is +*good*. It proves the *plumbing* is complete — every string flows through +the mechanism, and the layout survives length. Judgement about the words +stays with a human. + +## What I would do differently + +1. **Load the page in a real browser and assert `typeof` on the entry points + before believing a client-side refactor worked.** A dead script is + invisible to every static check you own. +2. **Never let a parameter name shadow a function the same file calls.** One + letter bought me an entire afternoon; the comment is the part that stops + it recurring. +3. **Treat declaration order as a dependency, not aesthetics.** If a block + calls a function or reads a constant, that declaration goes above it — + especially in one long script. +4. **Generate the language packs from one table.** Then "the keys match" is + a test rather than a hope, and drift cannot be merged. +5. **When a change does nothing, suspect the script before the change.** + "Nothing happened" is the default failure mode of a rendered page, not + evidence that your code was inert. ## The result -Seven original songs, written and produced with Suno AI, hosted on my own -infrastructure (`content.hoelee.com`), embedded here with plain HTML5 audio -and synced lyrics — no music-studio budget, no booking a vocalist, no -session musician. The whole catalog lives in one folder on my NAS, -versioned like any other project. +Two one-line causes, both in the same file, both found by browser-side +assertions rather than by reading code again. The dashboard switches +language at runtime now, the default render is byte-for-byte what it was +before the change, and the three packs come from a single generated table — +so a missing key shows up as a red test instead of as a word that quietly +stays Chinese. -If you're curious about AI music, start with your own lyrics and an honest -style sentence — genre, feel, voice, one production trick — then iterate -until one take surprises you. That's the whole craft. +The generalisation is the part I keep: **a crash tells you where to look. A +silent stop tells you nothing, and looks like success.** If a feature can +fail without moving a single pixel, then "the page still looks right" is not +a check. +```` --- -## Want original music — or AI-powered anything — for your business? +## 文章 11 / 11:逐页验证一份 19 页的 PDF 报告 +- **Slug**: `verifying-a-pdf-report-page-by-page` +- **英文标题**: Verifying a 19-Page PDF Report Page by Page +- **分类**: `engineering` | **发布日期**: 2026-09-28 +- **线上地址**: https://blog.hoelee.com/posts/verifying-a-pdf-report-page-by-page/ (EN) | https://blog.hoelee.com/posts/zh/verifying-a-pdf-report-page-by-page/ (ZH) +- **本篇审核状态**: ⏳ 待审核 (Pending) -I build websites, Telegram/WhatsApp bots, and self-hosted infrastructure, -and I produce original AI-assisted music for brands that want a jingle, -background tracks, or event audio without a studio budget. There's no -session musician to book and no licence maze — you get an original track -you can actually use. If that sounds useful, I'd love to talk: +### ✍️ 顾问审阅意见区 (Review Feedback) +```text +【审阅结论】:[ ] 通过 [ ] 需微调 [ ] 需重构 +【事实与数字核对】: +【人设与用词卫生】: +【CRO与转化建议】: +``` -- 📱 **WhatsApp:** [+60 12-797 2969](https://wa.me/60127972969) -- 📧 **Email:** [me@hoelee.com](mailto:me@hoelee.com?subject=Music%20for%20my%20business) -- 🌐 **Website:** [hoelee.com](https://hoelee.com) +### 🇨🇳 中文版原文 (Chinese Markdown Twin) +````markdown +--- +title: "逐页验证一份 19 页的 PDF 报告" +description: "如何逐页验证生成的 PDF 报告:用浏览器打印同一份 HTML 作基线,逐页比对几何尺寸,把每一处偏差都压到 2pt 以内。" +pubDate: 2026-09-28 +updatedDate: 2026-09-29 +category: engineering +blueprint: sme-digital-suite +blueprintLayer: L4-managed-ops-handover +tags: [php, codeigniter, pdf, css, print, testing] +ogImage: /og/zh-verifying-a-pdf-report-page-by-page.jpg +banner: /banners/verifying-a-pdf-report-page-by-page.jpg +draft: false +--- + +你怎么知道一份生成出来的 PDF 每一页都真的正确——不只是第一页? +我维护着一个 CodeIgniter 4 应用:输入一个人的出生信息,生成一份 +中文命理/生命密码报告——十九张密排的 A4 sheet,封面、数字图表、 +方向九宫格、「一到九」的个人特征页。以前这份报告按网站原本的设计 +交付:操作员在浏览器里按 Ctrl+P,手工另存成 PDF。我把它换成了 +服务端渲染(mPDF),一夜之间,产品的质量取决于一个我看不见 +如何排版的引擎。 + +这就是这个问题可被搜索的版本:如何把一份多页 PDF 报告逐页地、 +用数字而不是肉眼,去和浏览器自己的打印输出对版?这个过程抓到 +了真实的 bug——包括一行被渲染成 2.5pt、从第二页起几乎看不见的 +页眉——最后全部十九页与浏览器基线的偏差都压进了 2pt。 + +## 为什么「看起来没问题」不算测试 + +没有人能用肉眼验证一份 19 页的报告。过去的流程是:打开 PDF, +扫一眼封面,交付。第 17 页标题漂移、页脚插图压住版权行、页眉 +小到读不出来——快速翻一遍永远发现不了,而花钱买报告的顾客 +看到的是全尺寸。 + +天真的做法会失败,是结构性的原因。服务端 PDF 渲染器不会像 +浏览器那样排你的 HTML:分页、页边距、基线、字距全都会漂移—— +而且不存在一个「看起来没问题」的测试,能让你在下周有人改了 +CSS 之后再跑一次。我第一次单趟渲染出来是 27 页,而设计预期 +是 19 页:mPDF 没有 CSS 裁剪,而 sheet 靠 `overflow: hidden` +藏住溢出。报告的 `@page` 规则更糟:mPDF 为这条规则本身 +翻了一页,19 张 sheet 炸成 12,789 页。页数都不稳定,所以 +「扫一眼第一页」不是验证——是碰运气。 + +## 基线:让浏览器去打印你的 HTML + +操作员的 Ctrl+P,就是 Chrome 用打印 CSS 打印报告自己的 HTML。 +所以参照标准不是「我觉得它应该长什么样」,而是浏览器自己的 +打印输出——在同一台机器、同一套网页字体下生成。做法:把报告 +目录用本地静态服务起起来(字体必须同源,否则 webfont 拒绝 +加载),再用 headless Chrome 打印。报告 CSS 声明了 +`@page { margin: 0 }`,所以浏览器输出是无边距满幅,再配合 +`-webkit-print-color-adjust: exact` 保住背景图——等价于操作员 +勾选「背景图形」: + +```bash +python -m http.server 8123 --bind 127.0.0.1 --directory + +"C:\Program Files\Google\Chrome\Application\chrome.exe" --headless=new --disable-gpu \ + --no-pdf-header-footer --user-data-dir=%TEMP%\chromeprofile --virtual-time-budget=30000 \ + --print-to-pdf=chrome-win.pdf "http://127.0.0.1:8123/report-local.html" +``` + +然后在 VM 上,用同一份 HTML 出服务端版本: + +```bash +php tools/pdf-render.php /tmp/report-v10.html /tmp/mpdf.pdf +``` + +## 逐页比对,而不是整体对比 + +以浏览器的 PDF 为基准,我在 Windows 这边用 +`uv run --with pymupdf` 逐页对比两份文档:页面尺寸、文字条数、 +图片包围盒(x0/y0/宽/高)、附图说明文字的 y 坐标。判定阈值: +**偏差 ≤2pt(0.7mm)算对齐**;超过 5pt 就要查根因。而且只比 +「同一元素在两份 PDF 里的差」——永远不要比绝对页数,因为页数 +一致是前提,不是测试本身。 + +## 逐页比对抓到了什么 + +下面每个 bug 都有可测量的前后数字,而且都修在库或模板里—— +不是用管道糊过去。 + +**看不见的页眉行。** 从第二页起每页顶部有一行小字,用户直接 +反馈说小到读不出来。页眉是一个 `font-size: 10px` 的 div, +里面套一张 auto 宽的 table,第一格写着 `width: 100%`。mPDF +把这解读成「表格超宽」,于是连同字号一起把整行缩到三分之一: +**2.5pt,而 Chrome 是 7.5pt**。修法(`fixPageHeaderTables()`): +给表格显式宽度 + 显式字号(px→pt),并去掉第一格的 +`width: 100%`: + +```php +$pt = round((float) $m[2] * 0.75, 2); // 10px = 7.5pt +``` + +**匹配过宽的外边距规则。** `.sheet { margin: 5mm auto }` 是为 +屏幕预览准备的(sheet 之间的阴影缝隙)。mPDF 当真了,把 +296mm 的 sheet 推到 301mm——越过 297mm 的页面——于是 sheet +在页边被切开,auto-fit 把整页缩小 3%,带白边。最直观的修法、 +在样式表末尾追加一条 `.sheet { margin: 0; }`,完全没用:mPDF +对同名选择器的两条规则只认「先出现的赢」。所以库要原地改写 +这条规则(`stripSheetMargins()`),而匹配器必须小心什么才算 +「sheet 规则」。护栏是一个负向回顾断言: + +```php +'~(?` 标签 +居中,而 mPDF 8 的 `Center` 标签处理器是空类——标签整个被 +丢弃,所有居中的标题和表格全部左对齐(「前言」实测 x=30, +Chrome 是 x=280)。`expandCenterTags()` 把 `
` 改写成 +`
`,并给居中块里的表格补上 +`align="center"`——因为父级的 `text-align` 传不到表格。 +修复后:x=281,Chrome 280。 + +**行距比浏览器高 15%。** 模板的 normalize.css 声明了 +`html { line-height: 1.15 }`;mPDF 不继承它,退回自己的字体 +度量(1.33)。这累积成每页下半部 20–40pt 的漂移——sheet 3 +甚至溢出到第二页,触发整页缩小。把 `useFixedNormalLineHeight` +设成模板自己的值之后,正文行距 15.5pt,Chrome 是 15.7。 +表格还要显式加 `td, th { padding: 1px }`——mPDF 默认的 +单元格内边距比浏览器大 2px。 + +**字体是错的。** mPDF 读不了网页用的 `.woff`,文字回退到 +自带的 Sun-ExtA;而正文栈里的「微软雅黑」在 Linux 服务器上 +根本不存在。我注册了三套真 TTF——标题手写体 MaShanZheng、 +拉丁与数字 Roboto、正文 CJK 用 wqy-microhei——并把模板的 +字体栈映射过去。一个坑:必须关掉 mPDF 的自动按脚本选字, +否则它挑「第一个支持中文的已注册字体」——整页正文都变成 +手写体。 + +**还有一个与 mPDF 无关的 bug。** 第 7 页的主插图在**两份** +PDF 里都是破图——比对显示 Chrome 和 mPDF 里是同一个破损 +占位块。模板硬编码了 `https://cdn.hoelee.com/...`,这个域名 +已经不再解析(NXDOMAIN),所以每个引擎都抓不到图。把模板 +改回应用自己的 base URL,并让 `localiseAssets()` 把任何 +host 的 `/static/` 路径都映射到本地文件,两个引擎里的插图 +都恢复了。只有并排比对才能暴露这一类 bug——单独看哪个引擎 +都「正常」。 + +## 一张 sheet,一页 + +渲染器这么设计是有意的:库把 HTML 按 `
` +切开,每张 sheet 单独渲染成一份单页文档,合并时只取第 1 页—— +物理上保证「一张 sheet = 正好一页 A4」,永远不会跨页断裂。 +mPDF 量中文宽度和 Chrome 略有不同,个别 sheet 会高出几毫米。 +与其丢内容,库按一把缩放梯子——1.0, 1.005, 1.01, 1.02, 1.03, +1.06, 1.10, 1.15, 1.22——取第一个能落进一页的比例,缩小整张 +sheet 而不是裁内容。隐藏溢出是浏览器打印做的事 +(`overflow: hidden`);一份收费产品如果 PDF 里静默丢了页内 +内容,就是无声的交付事故,所以设计选择是绝不丢内容。sheet 3 +现在需要 x1.005——0.5%,肉眼不可见——以前是 x1.03。 + +**「19/7 页」是什么意思。** 报告模板永远按固定顺序渲染十九张 +sheet;「版本」是这些 sheet 上的一个过滤器,不是第二份模板。 +完整版是 **19 页**;RM49 的精华版是**这 19 张里的 7 张**, +页码重编,每张入选的 sheet 都登记了一个文字标记——模板被 +改动/重排导致取错页时会大声失败,而不是把错误的章节交给 +顾客。两个版本走同一条管线,也用同一种方式验收: +`tools/pdf-verify.php --expect=19` 和 +`--expect=7` 双双 PASS +——页数加逐页 ink 检查(ghostscript 50dpi)证明没有任何 +空白页。 + +## 为什么收据是独立文档 + +收据不是报告裁剪出来的。它是自己的一份单页 A4 文档:真 +16mm 页边距(报告刻意做满幅无边距)、三语、单趟渲染—— +因为里面没有 sheet。它在发「已收款」邮件时才懒生成:付款 +回调必须毫秒级应答,0.3–1 秒的 PDF 渲染不属于回调。它落进 +同一个交付存储,文件名带 `-receipt` 后缀,永远不会和报告 +文件撞名,而且幂等:重试、重寄、顾客自己来拿,拿到的都是 +同一份。做它的过程从另一个方向印证了同一个论点——连单页 +文档都和浏览器不一样:`` 上的 `display: block` +不生效,左右并排的两张表把右列的数值裁出页面右边界, +合计行必须写在明细表内部,否则标签会浮在半空。 + +## 我会怎么做不一样 + +对版方法现在躺在项目笔记里,是一份文档化的流程,不是仓库 +里的脚本——这就是差距。仓库里自动化的验收工具只证明页数和 +逐页 ink,永远抓不到 2.5pt 的页眉或 15% 的行距漂移。我会把 +浏览器基线比对做成仓库验证工具链里的一个脚本:用同一份 +HTML 分别喂 Chrome 和库,diff 几何,任何超过 5pt 的偏差 +直接失败。这样,一次让打印布局悄悄回退的 CSS 改动会在构建 +时炸掉,而不是送到顾客手里。 + +我还会在写任何 mPDF 补偿代码之前就先出 Chrome 基线。 +「用操作员用的同一个引擎打印,然后测量」才是解锁点;之后 +每个修复都是机械活。还有两件诚实的遗留工作:伴侣合盘与 +家庭套餐还没跑过这套逐页对版,19 页的 PDF(背景图加嵌入 +字体约 20 MB)也还需要在交付前压缩。 + +## 结果 + +修复后的完整版验收: + +```text +$ php tools/pdf-verify.php /tmp/report-v10.html --expect=19 +out : 20,225,818 bytes, 19 pages, 10.4s, peak 188 MB +per-page ink check: all pages have content | report pages=19 +expected 19 pages => MATCH +RESULT: PASS +``` + +- **19 页 vs Chrome 基线的 19 页 — MATCH**,一张 sheet 一页, + 无跨页断裂。 +- 每一项实测偏差都**在 2pt(0.7mm)以内**:第 6 页偏了 + 213.5pt 的页脚插图、从 2.5pt 恢复到浏览器同款 7.5pt 的 + 页眉、归位的居中标题、15.5pt vs Chrome 15.7 的正文行距、 + 与网页字体一致的嵌入字体。 +- ink 检查确认**没有任何空白页**,PDF 文字可提取——封面能 + 读出真实文字,这对一份顾客要复制内容的报告很重要。 +- 精华版:**7 页 PASS**,4.99 MB,2.8 秒。 + +这就是「第一页看起来没问题」和「全部十九页都在浏览器两个点 +以内」的区别。前者是肉眼扫一遍就交付的结果;后者是拿浏览器 +自己当测试得到的结果。 + +--- + +我平时就做这类 Web 应用与打印/PDF 报告管线,也做网站设计与 +开发。如果你有一份服务器渲染的文档——报告、收据、发票—— +想在它送到顾客手里之前确认每一页都正确,跟我说说: +[WhatsApp](https://wa.me/60127972969) · [me@hoelee.com](mailto:me@hoelee.com?subject=PDF%20report%20pipeline) · +[hoelee.com](https://hoelee.com)。 +```` + +### 🇬🇧 英文版原文 (English Markdown Primary) +````markdown +--- +title: "Verifying a 19-Page PDF Report Page by Page" +description: "Verify a PDF report page by page: print the same HTML from your browser as a baseline, compare the per-page geometry, and keep every deviation under 2pt." +pubDate: 2026-09-28 +updatedDate: 2026-09-29 +category: engineering +blueprint: sme-digital-suite +blueprintLayer: L4-managed-ops-handover +tags: [php, codeigniter, pdf, css, print, testing] +ogImage: /og/verifying-a-pdf-report-page-by-page.jpg +banner: /banners/verifying-a-pdf-report-page-by-page.jpg +draft: false +--- + +How do you know a generated PDF actually looks right on every page — not just +page one? I maintain a CodeIgniter 4 app that turns birth details into a +Chinese numerology report. Until recently the operator printed it with Ctrl+P +and saved a PDF by hand. I replaced it with a server-side renderer (mPDF), and +overnight the product's quality depended on a layout engine I could not see. + +This post is the searchable version of that problem: verifying a multi-page +PDF report page by page against the browser's own print output, with numbers +instead of eyeballs. It caught real bugs — a header line rendered at 2.5pt, +effectively invisible on every page after the first — and ended with all +nineteen pages within 2pt of the browser. + +## Why "looks fine" is not a test + +Nobody can verify a 19-page report by eye. The old ritual was: open the PDF, +skim the cover, ship it. A drifting heading on page 17, a footer illustration +over the copyright line, an unreadable header — a quick flick never catches +those, and the customer who paid sees them at full size. + +Naive approaches fail structurally. A server-side PDF renderer does not lay +out HTML the way a browser does, so page breaks, margins, baselines and +spacing all drift — and there is no "looks fine" test you can re-run after +someone edits the CSS. My first single-pass render came out as 27 pages where +the design expects 19 — mPDF has no CSS clipping and the sheets hide overflow +with `overflow: hidden`. The report's own `@page` rule was worse: mPDF emitted +a page break for the rule itself, and 19 sheets exploded into 12,789 pages. +Page counts were not even stable, so eyeballing page one was not verification; +it was hope. + +## The baseline: make the browser print your HTML + +The operator's Ctrl+P is Chrome printing the report's own HTML with its print +CSS — same machine, same web fonts. So the reference is not "what I think it +should look like"; it is the browser's own print output. I served the report +directory locally (fonts must be same-origin or webfonts refuse to load) and +printed with headless Chrome. The report CSS declares `@page { margin: 0 }`, +so the output is borderless full-bleed, and `-webkit-print-color-adjust: +exact` keeps the background graphics the operator ticks: + +```bash +python -m http.server 8123 --bind 127.0.0.1 --directory + +"C:\Program Files\Google\Chrome\Application\chrome.exe" --headless=new --disable-gpu \ + --no-pdf-header-footer --user-data-dir=%TEMP%\chromeprofile --virtual-time-budget=30000 \ + --print-to-pdf=chrome-win.pdf "http://127.0.0.1:8123/report-local.html" +``` + +Then the server-side output, from the same HTML, on the VM: + +```bash +php tools/pdf-render.php /tmp/report-v10.html /tmp/mpdf.pdf +``` + +## Compare page by page, not overall + +I compared the two PDFs page by page with `uv run --with pymupdf`: page size, +text-block count, image bounding boxes (x0/y0/width/height), and caption text +y. Threshold: **within 2pt (0.7mm) counts as aligned**; over 5pt gets a root +cause. Compare the same element across the two PDFs — never absolute page +counts; page parity is the prerequisite, not the test. + +## What the comparison caught + +Each bug below had a measurable before/after and a fix in the library or +template. + +**The invisible header line.** A user reported the small header line, present +on every page after the cover, was unreadable. It is a `font-size: 10px` div +wrapping an auto-width table whose first cell declares `width: 100%` — mPDF +read that as "table too wide" and scaled the whole line, font included, to a +third of its size: **2.5pt vs Chrome's 7.5pt**. `fixPageHeaderTables()` gives +the table an explicit width and font size in points (px → pt) and drops the +offending cell: + +```php +$pt = round((float) $m[2] * 0.75, 2); // 10px = 7.5pt +``` + +**The margin rule that matched too much.** `.sheet { margin: 5mm auto }` +spaces the sheets on screen. mPDF took it literally and pushed the 296mm sheet +to 301mm — past the 297mm page — so the sheet was cut at the page edge and +auto-fit shrank the whole page by 3%, white border included. Appending `.sheet +{ margin: 0; }` does nothing — mPDF honours the *first* rule for a duplicated +selector. So the library rewrites the rule in place (`stripSheetMargins()`), +and the matcher must not over-match — the guard is a negative lookbehind: + +```php +'~(?` tag; mPDF 8's `Center` handler is an empty class, so the tag is +dropped and every centred heading and table came out left-aligned (the preface +heading measured x=30 against Chrome's x=280). `expandCenterTags()` rewrites +`
` to `
` and adds `align="center"` to +tables inside centred blocks, because parent `text-align` does not reach +tables. After the fix: x=281, Chrome 280. + +**Line spacing 15% taller than the browser's.** The template's normalize.css +declares `html { line-height: 1.15 }`; mPDF does not inherit it and fell back +to its own font metrics (1.33) — 20–40pt of drift accumulated on the lower +half of every page, and sheet 3 overflowed onto a second page, triggering a +whole-page shrink. Setting `useFixedNormalLineHeight` to the template's own +value brought body leading to 15.5pt against Chrome's 15.7. Tables needed `td, +th { padding: 1px }` — mPDF's default cell padding is 2px larger than a +browser's. + +**The fonts were wrong.** mPDF cannot read the `.woff` files the web views +use, so text fell back to its built-in Sun-ExtA, and "Microsoft YaHei" does +not exist on the Linux server. I registered the real TTFs — MaShanZheng for +headings, Roboto for Latin, wqy-microhei for CJK — and mapped the template's +stacks onto them. One trap: mPDF's automatic script-to-font selection must be +off, or it picks the first registered font that supports Chinese — the entire +body came out in handwriting. + +**And one bug that had nothing to do with mPDF.** Page 7's main illustration +was broken in *both* PDFs — the same broken stub in Chrome and mPDF. The +template hardcoded `https://cdn.hoelee.com/...`, a domain that no longer +resolves (NXDOMAIN), so every engine fetched nothing. Fixing the template to +use the app's own base URL and mapping any host's `/static/` path to local +files restored it in both engines. Only a side-by-side comparison surfaces +this — each engine alone looks "fine". + +## One sheet, one page + +The library splits the HTML into its `
` blocks and +renders each sheet as its own one-page document, importing page 1 and merging +— a physical guarantee that one sheet is exactly one A4 page, never split +across pages. mPDF measures CJK text widths slightly differently from Chrome, +so some sheets come out a few millimetres too tall. Rather than lose content, +the library re-renders the sheet at the smallest scale factor from a ladder — +1.0, 1.005, 1.01, 1.02, 1.03, 1.06, 1.10, 1.15, 1.22 — that fits, shrinking +the whole sheet instead of clipping. The browser's print clips with `overflow: +hidden`; a PDF that silently dropped in-sheet content would be a delivery +accident, so the design never drops content. Sheet 3 needs x1.005 today (0.5%, +invisible) where it used to need x1.03. + +**What "19/7 pages" means.** The report template always renders nineteen +sheets in a fixed order; the "edition" is a filter over those sheets, not a +second template. The full report is **19 pages**; the RM49 essence edition is +**7 of those 19 sheets**, renumbered, each selected sheet carrying a marker so +a rearranged template fails loudly instead of shipping the wrong chapters. +Both editions run the same pipeline and verify the same way: +`tools/pdf-verify.php --expect=19` and `--expect=7` both PASS — page count +plus a per-page ink check (ghostscript at 50dpi) proving no blank pages. + +## Why the invoice is a separate document + +The receipt is not a cut-down report. It is its own one-page A4 document: real +16mm page margins (the report is deliberately full-bleed), three languages, +and a single-pass render because there are no sheets. It is generated lazily +at email time — the payment callback must answer in milliseconds, and a 0.3–1s +render does not belong in it. It lands in the same delivery store under a +`-receipt` filename, never colliding with the report file, and it is +idempotent: retries get the same file. Even one page differed from the +browser: `display: block` on `` was ignored, side-by-side tables +clipped the right column's values off the page edge, and the total row had to +live inside the items table or its label floated in mid-air. + +## What I'd do differently + +The method lives in the project notes as a documented ritual, not a committed +script — that is the gap. The repo's automated acceptance tool proves page +count and per-page ink, which would never catch a 2.5pt header or a 15% +leading drift. I would turn the browser-baseline comparison into a script in +the repo's verification tooling: render the same HTML in Chrome and in the +library, diff the geometry, fail on any deviation over 5pt. Then a CSS change +that silently regresses the print layout fails the build instead of reaching a +customer. + +I would also have generated the Chrome baseline before writing any mPDF +compensation code — "print with the same engine the operator uses, then +measure" was the unlock. Two honest items remain: the partner and family +reports have not had this page-by-page pass yet, and the 19-page PDF (20 MB +with backgrounds and embedded fonts) still needs compression before delivery. + +## The result + +Full report verification, after the fixes: + +```text +$ php tools/pdf-verify.php /tmp/report-v10.html --expect=19 +out : 20,225,818 bytes, 19 pages, 10.4s, peak 188 MB +per-page ink check: all pages have content | report pages=19 +expected 19 pages => MATCH +RESULT: PASS +``` + +- **19 pages vs the Chrome baseline's 19 — MATCH**, one A4 page per sheet, + no splits. +- Every measured deviation is **within 2pt (0.7mm)** of the browser's print + output: footer art 213.5pt off on page 6 aligned, header restored to the + browser's 7.5pt, centred headings back, leading 15.5pt vs 15.7, embedded + fonts matching the web fonts. +- The ink check confirms **no blank pages**, and the PDF text is extractable — + the cover reads back as real text, which matters for a report customers + copy from. +- Essence edition: **7 pages PASS**, 4.99 MB, 2.8 seconds. + +That is the difference between "the first page looks fine" and "all nineteen +pages within two points of the browser": the first ships when you eyeball a +PDF; the second is what you get when the browser itself is the test. + +--- + +I build web applications and print/PDF report pipelines like this one, and I +do website design and development. If you have a document your server renders +— a report, a receipt, an invoice — and you want to be sure it is right on +every page before it reaches a customer, tell me about it: +[WhatsApp](https://wa.me/60127972969) · +[me@hoelee.com](mailto:me@hoelee.com?subject=PDF%20report%20pipeline) · +[hoelee.com](https://hoelee.com). ```` --- diff --git a/docs/consultant-review/batches/batch-02.md b/docs/consultant-review/batches/batch-02.md index 5126112..c357527 100644 --- a/docs/consultant-review/batches/batch-02.md +++ b/docs/consultant-review/batches/batch-02.md @@ -1,10 +1,10 @@ # 博客文章顾问审阅资料包 — Batch-02 > **批次编号**:`Batch-02` -> **包含文章数**:11 篇 -> **本批次总字符量**:167,652 字(严格限制 < 200,000 字) +> **包含文章数**:1 篇 +> **本批次总字符量**:128,681 字(严格限制 < 200,000 字) > **模式**:BILINGUAL -> **生成时间**:2026-10-27 05:23:38 +> **生成时间**:2026-10-27 05:36:05 # 外部顾问审阅前置须知与工程背景规范 @@ -39,25 +39,15 @@ ## 本批次文章目录索引 -1. [devops] **为什么你的 CDP 点击会静默失效:平滑滚动、最小化窗口与 JS 派发事件** (`why-your-cdp-clicks-silently-fail`) — ~12,514 字符 -2. [devops] **Prometheus 明明有数据,Grafana 面板却全是 No data 的原因** (`why-your-grafana-dashboard-shows-no-data`) — ~11,212 字符 -3. [devops] **你的“磁盘将满”告警在说谎:别再按百分比告警** (`your-disk-full-alert-is-lying`) — ~8,912 字符 -4. [engineering] **浏览器翻译没有 API:我给一个纯中文网页应用加上了英文模式** (`adding-english-mode-to-a-chinese-only-web-app`) — ~18,070 字符 -5. [engineering] **排查 Remotion 视频抽搐:为什么在背景图挂载音频低频 Scale 会毁了转场** (`debugging-remotion-video-jitter`) — ~15,111 字符 -6. [engineering] **把字体嵌进 PDF,让它在任何机器上都长一样** (`embedding-fonts-so-a-pdf-looks-the-same-everywhere`) — ~11,465 字符 -7. [engineering] **那个把静默丢失更新变成 409 的 JPA 字段:@Version 乐观锁** (`jpa-version-field-lost-update`) — ~19,575 字符 -8. [engineering] **把 CodeIgniter 4 应用从 IIS 迁移到 OpenLiteSpeed** (`migrating-codeigniter-iis-to-openlitespeed`) — ~23,361 字符 -9. [engineering] **用「热浏览器会话」爬取有反爬墙的电商网站** (`scraping-bot-walled-marketplace-warm-browser-session`) — ~16,928 字符 -10. [engineering] **JavaScript 文件静默失效的两种方式:而页面看上去一切正常** (`two-ways-a-javascript-file-stops-working-and-looks-fine`) — ~11,294 字符 -11. [engineering] **逐页验证一份 19 页的 PDF 报告** (`verifying-a-pdf-report-page-by-page`) — ~19,210 字符 +1. [tutorials] **用 YuE2 BF16 生成完整长歌:突破时长限制与音频硬截断** (`generating-full-length-songs-with-yue2-comfyui-zh`) — ~128,681 字符 --- -## 文章 01 / 11:为什么你的 CDP 点击会静默失效:平滑滚动、最小化窗口与 JS 派发事件 -- **Slug**: `why-your-cdp-clicks-silently-fail` -- **英文标题**: Why Your CDP Clicks Silently Fail: Smooth Scroll, Minimised Windows, and JS-Dispatched Events -- **分类**: `devops` | **发布日期**: 2025-03-19 -- **线上地址**: https://blog.hoelee.com/posts/why-your-cdp-clicks-silently-fail/ (EN) | https://blog.hoelee.com/posts/zh/why-your-cdp-clicks-silently-fail/ (ZH) +## 文章 01 / 01:用 YuE2 BF16 生成完整长歌:突破时长限制与音频硬截断 +- **Slug**: `generating-full-length-songs-with-yue2-comfyui` +- **英文标题**: Generating Full-Length Songs with YuE2 BF16: Overcoming Duration Limits and Audio Truncation +- **分类**: `tutorials` | **发布日期**: 2026-10-26 +- **线上地址**: https://blog.hoelee.com/posts/generating-full-length-songs-with-yue2-comfyui/ (EN) | https://blog.hoelee.com/posts/zh/generating-full-length-songs-with-yue2-comfyui/ (ZH) - **本篇审核状态**: ⏳ 待审核 (Pending) ### ✍️ 顾问审阅意见区 (Review Feedback) @@ -71,4070 +61,3268 @@ ### 🇨🇳 中文版原文 (Chinese Markdown Twin) ````markdown --- -title: "为什么你的 CDP 点击会静默失效:平滑滚动、最小化窗口与 JS 派发事件" -description: "Chrome DevTools Protocol 点击报告成功却毫无反应的三个原因:平滑滚动导致的过期坐标、最小化窗口被丢弃的坐标输入、以及页面收到了点击但拒绝了它。" -pubDate: 2025-03-19 -updatedDate: 2026-09-29 -category: devops -tags: [cdp, chrome, react, browser-automation, debugging, python] -ogImage: /og/zh-why-your-cdp-clicks-silently-fail.jpg -banner: /banners/why-your-cdp-clicks-silently-fail.jpg -draft: false ---- - -如果你用 Chrome DevTools Protocol 驱动网页应用,你一定见过这个失败:脚本报告 `clicked @591,361`,元素明明在,点击也确实发出去了 —— 然后什么都没发生。没有异常,没有控制台报错,页面毫无变化。 - -我在自动化某平台卖家后台时整整耗了一个晚上在这上面。找到的三个原因值得写下来,因为**它们对「报告点击成功」的那段代码全都是隐形的**。 - -## 原因一:坐标过期,因为页面是平滑滚动的 - -最直观的点击辅助函数大概长这样: - -```python -r = self.ev("""(()=>{const e=%s; e.scrollIntoView({block:'center'}); - const b=e.getBoundingClientRect(); - return JSON.stringify({x:b.left+b.width/2, y:b.top+b.height/2});})()""") -self.click_xy(json.loads(r)["x"], json.loads(r)["y"]) -``` - -滚动到可视区、读取元素矩形、点它的中心。逻辑没错 —— 除非页面有 `scroll-behavior: smooth`(现代 UI 组件库里非常常见)。那时 `scrollIntoView()` 启动的是**动画**,而同一个 tick 里的 `getBoundingClientRect()` 返回的是元素**即将离开**的位置。你的点击落在元素**原来**的地方,落在当时在那儿的别的东西上,而日志忠实地告诉你:它点击了你要求的坐标。 - -修法是两行 —— 关掉平滑滚动,然后重新读矩形: - -```python -self.ev("""(()=>{document.documentElement.style.scrollBehavior='auto'; - if(document.body) document.body.style.scrollBehavior='auto'; return 'ok';})()""") -moved = self.ev("""(()=>{const e=%s; if(!e) return 'no'; - e.scrollIntoView({block:'center', behavior:'instant'}); return 'ok';})()""" % sel) -time.sleep(0.35) # 等布局稳定 -# 然后再读矩形(元素晚一帧才渲染的话,外面再套一层重试) -``` - -两个细节很关键:`scrollIntoView` 上的 `behavior:'instant'` 会覆盖 CSS;而读取前那一小段等待,才是让坐标可信的原因。没有重试循环的话,这个辅助函数在晚一帧渲染的元素上仍会偶发失败 —— **偶发失败是最糟的一类,因为它会让你怀疑自己的判断。** - -## 原因二:窗口最小化时,坐标点击被丢弃 - -修好之后,我的点击辅助函数在一次性探针里能用,一放进真正的脚本就再也没成功过。同样的选择器、同样的坐标、同样的页面。差别最后落在可见性上: - -``` -document.visibilityState → "hidden" -document.hasFocus() → true -``` - -当目标标签页不是可见窗口的活动标签页时,CDP 合成的坐标级鼠标事件是不可靠的 —— 浏览器可能接受、可能限流、也可能直接不往任何地方送。注意 `hasFocus()` **在文档隐藏时依然返回 `true`**,所以它不是有效的判断依据;要看的是 `document.hidden` / `visibilityState`。 - -两个修法,我都用: - -```python -def activate(self): # 把标签页提到窗口最前 - self.cdp("Page.bringToFront") - time.sleep(0.2) - -def click_js(self, sel): # 在页面内部派发事件来点击 - return self.ev("""(()=>{const e=%s; if(!e) return 'no'; - ['mousedown','mouseup','click'].forEach(k=> - e.dispatchEvent(new MouseEvent(k,{bubbles:true,cancelable:true,view:window}))); - return 'js-clicked';})()""" % sel) -``` - -`Page.bringToFront` 解决了常见情况。但真正从不出错的是 JS 派发的点击:它不依赖窗口是否可见,而且 React 挂在根节点上的事件监听器会收到它,和收到真实点击的方式完全一样 —— 因为事件冒泡到的是同一个根。一个用坐标点击五次才能打开一次的日期选择器,用这种方式**每次都开**。 - -需要知道的取舍:派发事件会跳过 hit-testing,所以它也会「点到」用户根本够不到的元素(被遮住、滚出屏幕、透明度为零)。在这里这是优点 —— 如果你用它去掩盖布局 bug,那就是坑 —— 所以凡是可视化的东西我都用坐标点击,而遇到自定义控件时才伸手去拿 `click_js`。 - -## 原因三:页面收到了点击,然后拒绝了它 - -这个最浪费时间,因为它的症状和前两个一模一样:点击发出去了,什么都没变。 - -区分它们只需要注入一个监听器: - -```js -window.__ev = []; -['mousedown','mouseup','click'].forEach(k => - document.addEventListener(k, e => window.__ev.push(k + ':' + e.target.tagName), true)); -``` - -- 有事件记录 → 输入链路是通的,问题在应用侧(原因三),或者在你读取结果的方式上(见下)。 -- 一条都没有 → 是原因一或原因二。 - -挂上监听器后我点了一次 `Confirm`,缓冲区满了 —— 说明点击没问题,是应用在说「不」。理由在一段我从没想过要看的文字里: - -> “Please create a new shop welcome voucher after the existing one is expired.” - -表单拒绝创建同类型的第二张券。没有字段级红字、头 500 毫秒里没有提示条、URL 也没变。**拒绝是真的,点击是无辜的。** - -这一类 bug 还有两个工具: - -```js -document.elementFromPoint(x, y) // 那个坐标上到底是什么? -document.querySelectorAll('button').length // 你的选择器到底命中了几个? -``` - -第二个抓到过一处很细的问题:门户页面在 DOM 里留了一个**隐藏的** `Confirm` 按钮(来自收起状态的日期面板),于是 `.pop()` 返回的是那个看不见的副本,点击自然落到了无人的地方。按可见性和尺寸一起过滤: - -```python -"[...document.querySelectorAll('button')].filter(b=>/^Confirm$/.test(b.innerText.trim())" -" && b.offsetParent!==null && b.getBoundingClientRect().width>0).pop()" -``` - -## 第四种「失败」,只有回读才发现 - -还有一类「失败」其实不是失败:**写入成功了,而你的读取是旧的。** 我因为列表页没有立刻显示,就断定创建被静默拒绝,又点了一次;后来才发现那张券一直都存在 —— 列表只是延迟了大约十分钟。这个教训可以推广:一次写入之后,唯一可信的结论是在**等待足够时间之后**回读得到的,而「看不到变化」意味着**未知**,不是失败。 - -## 检查清单 - -1. 关掉平滑滚动;用 `behavior:'instant'` 滚动;等一会儿再读矩形。 -2. 点击前调用 `Page.bringToFront`;判断依据用 `document.hidden`,不是 `hasFocus()`。 -3. 自定义控件用 JS 派发 `mousedown`/`mouseup`/`click`,别点坐标。 -4. 先挂事件监听器,证明输入是否到达,再去怪应用。 -5. 数一数选择器命中几个 —— 组件库里隐藏的重复元素很常见。 -6. 写入之后先等待再回读;永远不要重复点击一个还没验证过的提交按钮。 - -这些都不算冷门知识,但它们中的每一条,在一行 `clicked @591,361` 面前都是隐形的。如果你正在对着一个自己无法控制的后台做自动化:点击是整条链路里最不可靠的一环 —— 也是最容易被最后才怀疑的一环。 - -我平时就在做这类浏览器自动化与排障 —— 自托管服务、Docker 与 Traefik 栈,偶尔还有被 bot 墙挡住的市场页面。如果你有条工作流总因为「UI 有自己的想法」而断掉,跟我说说:[WhatsApp](https://wa.me/60127972969) · [me@hoelee.com](mailto:me@hoelee.com?subject=Browser%20automation%20debugging) · [hoelee.com](https://hoelee.com)。 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Why Your CDP Clicks Silently Fail: Smooth Scroll, Minimised Windows, and JS-Dispatched Events" -description: "Three reasons a Chrome DevTools Protocol click reports success and does nothing: a stale rect from smooth scrolling, a minimised window that drops coordinate input, and a page that received the click but rejected it." -pubDate: 2025-03-19 -updatedDate: 2026-09-29 -category: devops -tags: [cdp, chrome, react, browser-automation, debugging, python] -ogImage: /og/why-your-cdp-clicks-silently-fail.jpg -banner: /banners/why-your-cdp-clicks-silently-fail.jpg -draft: false ---- - -If you drive a web app with the Chrome DevTools Protocol, you have met this failure: your script reports `clicked @591,361`, the element is there, the click landed — and nothing happens. No exception, no console error, no change on the page. - -I spent an evening on exactly this while automating a marketplace's seller admin panel, and the three causes I found are worth writing down, because they are all invisible to the code that reports the click. - -## Cause 1: a stale rect, because the page scrolls smoothly - -The obvious click helper looks like this: - -```python -r = self.ev("""(()=>{const e=%s; e.scrollIntoView({block:'center'}); - const b=e.getBoundingClientRect(); - return JSON.stringify({x:b.left+b.width/2, y:b.top+b.height/2});})()""") -self.click_xy(json.loads(r)["x"], json.loads(r)["y"]) -``` - -Scroll into view, read the element's rectangle, click its centre. Correct — unless the page has `scroll-behavior: smooth` (very common in modern UI kits). Then `scrollIntoView()` starts an **animation**, and `getBoundingClientRect()` in the same tick returns the position the element is *about to leave*. Your click goes to where the element was, lands on whatever is there, and the log tells you it clicked the coordinates you asked for. - -The fix is two lines — turn smooth scrolling off, then re-read the rect: - -```python -self.ev("""(()=>{document.documentElement.style.scrollBehavior='auto'; - if(document.body) document.body.style.scrollBehavior='auto'; return 'ok';})()""") -moved = self.ev("""(()=>{const e=%s; if(!e) return 'no'; - e.scrollIntoView({block:'center', behavior:'instant'}); return 'ok';})()""" % sel) -time.sleep(0.35) # let layout settle -# then read the rect (and retry a couple of times if the element is not there yet) -``` - -Two details matter. `behavior:'instant'` on `scrollIntoView` overrides the CSS; and the small wait before reading is what makes the rect trustworthy. Without the retry loop the helper still fails intermittently on elements that render a frame late — intermittent failures are the worst kind, because they make you doubt your own diagnosis. - -## Cause 2: the window is minimised, and coordinate clicks are dropped - -After the fix, my click helper worked in a one-off probe and then never again inside the real script. Same selector, same coordinates, same page. The difference turned out to be visibility: - -``` -document.visibilityState → "hidden" -document.hasFocus() → true -``` - -CDP-synthesised mouse events at coordinates are unreliable when the target tab is not the active tab of a visible window — the browser may accept them, throttle them, or route them nowhere. Note that `hasFocus()` returns `true` even when the document is hidden, so it is not a useful check; `document.hidden`/`visibilityState` is. - -Two fixes, and I use both: - -```python -def activate(self): # bring the tab to the front of its window - self.cdp("Page.bringToFront") - time.sleep(0.2) - -def click_js(self, sel): # click by dispatching events from inside the page - return self.ev("""(()=>{const e=%s; if(!e) return 'no'; - ['mousedown','mouseup','click'].forEach(k=> - e.dispatchEvent(new MouseEvent(k,{bubbles:true,cancelable:true,view:window}))); - return 'js-clicked';})()""" % sel) -``` - -`Page.bringToFront` fixes the common case. But the JS-dispatched click is the one that never fails: it does not depend on the window being visible, and React's root-level event listener receives it exactly as it receives a real click, because the events bubble to the same root. A date-picker input that opened maybe one time in five with coordinate clicks opens **every** time this way. - -The trade-off to know: dispatching events skips hit-testing, so it will also “click” an element a user could not reach (covered, scrolled off, zero-opacity). That is a feature here — and a trap if you use it to paper over a layout bug — so I keep coordinate clicking for anything visual and reach for `click_js` when the widget is a custom control. - -## Cause 3: the page received the click and rejected it - -This is the one that wasted the most time, because the symptom is identical to causes 1 and 2: click fires, nothing changes. - -Distinguishing them takes one injected listener: - -```js -window.__ev = []; -['mousedown','mouseup','click'].forEach(k => - document.addEventListener(k, e => window.__ev.push(k + ':' + e.target.tagName), true)); -``` - -- Events recorded → the input path works. The problem is in the app (cause 3), or in how you read the result (see below). -- Nothing recorded → causes 1 or 2. - -With the listener armed I clicked `Confirm` and the buffer filled up — so the click was fine, and the app was saying no. The reason was in text I had not thought to look at: - -> “Please create a new shop welcome voucher after the existing one is expired.” - -The form refuses a second voucher of that type. No field-level red text, no toast in the first 500 ms, the URL unchanged. **The rejection was real and the click was innocent.** - -Two more tools for this class of bug: - -```js -document.elementFromPoint(x, y) // what is actually on top at that coordinate? -document.querySelectorAll('button').length // how many match your selector? -``` - -The second one caught a subtle one: the portal leaves a **hidden** `Confirm` button in the DOM from the collapsed date-panel, so `.pop()` returned the invisible duplicate and the click went nowhere visible. Filter by visibility *and* size: - -```python -"[...document.querySelectorAll('button')].filter(b=>/^Confirm$/.test(b.innerText.trim())" -" && b.offsetParent!==null && b.getBoundingClientRect().width>0).pop()" -``` - -## And the fourth failure I only found by re-reading - -There is a class of “failure” that is not one: **the write succeeded and your read is stale.** I concluded a create had been silently refused because the list page did not show it immediately, clicked again, and later discovered the voucher had existed the whole time — the list just lagged about ten minutes. The lesson generalises: after a write, the only trustworthy verdict is a read-back *after* a real waiting period, and “no visible change” means *unknown*, not *failed*. - -## The checklist - -1. Turn off smooth scrolling; scroll with `behavior:'instant'`; wait, then read the rect. -2. Call `Page.bringToFront` before clicking; treat `document.hidden` as the signal, not `hasFocus()`. -3. For custom widgets, dispatch `mousedown`/`mouseup`/`click` from JS rather than clicking coordinates. -4. Arm an event listener to prove whether input arrived before you blame the app. -5. Count your selectors — hidden duplicates are common in component libraries. -6. After a write, wait and re-read; never re-click a submit you have not verified. - -None of this is exotic, but every item is invisible in a log line that says `clicked @591,361`. If you are building automation against an admin panel you do not control, the click is the least reliable part of the pipeline — and the one most likely to be blamed last. - -I build and debug this kind of browser automation for a living — self-hosted services, Docker and Traefik stacks, and the odd scraped-but-bot-walled marketplace. If you have a workflow that keeps breaking because a UI has other ideas, tell me about it: [WhatsApp](https://wa.me/60127972969) · [me@hoelee.com](mailto:me@hoelee.com?subject=Browser%20automation%20debugging) · [hoelee.com](https://hoelee.com). -```` - ---- - -## 文章 02 / 11:Prometheus 明明有数据,Grafana 面板却全是 No data 的原因 -- **Slug**: `why-your-grafana-dashboard-shows-no-data` -- **英文标题**: Why Your Grafana Dashboard Shows No Data (When Prometheus Is Fine) -- **分类**: `devops` | **发布日期**: 2026-05-17 -- **线上地址**: https://blog.hoelee.com/posts/why-your-grafana-dashboard-shows-no-data/ (EN) | https://blog.hoelee.com/posts/zh/why-your-grafana-dashboard-shows-no-data/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】:[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】: -【人设与用词卫生】: -【CRO与转化建议】: -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "Prometheus 明明有数据,Grafana 面板却全是 No data 的原因" -description: "所有 target 都是 up、数据就在 Prometheus 里、面板表达式也没错——41 个面板却全部 No data。原因是 Grafana 变量过滤了自己,而我的验证方式恰好掩盖了它。" -pubDate: 2026-05-17 -updatedDate: 2026-09-29 -category: devops -blueprint: self-hosted-infrastructure -blueprintLayer: L3-observability-self-healing -tags: [grafana, prometheus, dashboards, monitoring, promql, observability] -ogImage: /og/zh-why-your-grafana-dashboard-shows-no-data.jpg -banner: /banners/why-your-grafana-dashboard-shows-no-data.jpg -draft: false ---- - -打开每天在看的面板,结果是空的。不是\"某一格坏了\",而是**从上到下每一格**都写着 *No data*。去查 Prometheus,一切健康。这篇讲我踩到的那个具体且不直观的原因,以及为什么我自己的验证会同时告诉我\"面板没问题\"。 - -## 最省时间的排查顺序 - -常见原因(数据源选错、时间范围不对、`rate()` 用在会重置的计数器上、target 没被抓取)都不适用。真正有效的检查顺序是: - -```bash -# 1. target 真被抓取了吗? -curl -s http://prometheus:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job, health}' - -# 2. 指标现在存在吗?(完全绕过 Grafana) -curl -s -G http://prometheus:9090/api/v1/query \ - --data-urlencode 'query=node_uname_info{job="unraid-host"}' | jq '.data.result[0].metric' - -# 3. 面板表达式本身对吗? -curl -s -G http://prometheus:9090/api/v1/query \ - --data-urlencode 'query=count(node_cpu_seconds_total{mode="idle",job="unraid-host"})' -``` - -三项都过了:7 个 target `up`、2.5 万条序列在采集、手动跑面板表达式也有数据。所以问题不在指标、不在抓取、也不在 PromQL——而在**它们之间的变量层**。 - -## 真正的原因:变量过滤了自己 - -面板上有一个主机选择器,两个变量串联:`$nodename`(哪台机器)和 `$node`(它的 `instance` 标签)。第一个的定义是: - -```promql -# BROKEN — 变量用自身的值来过滤自己 -label_values(node_uname_info{job="unraid-host", nodename=~"$nodename"}, nodename) -``` - -首次加载时 `$nodename` 还没有值,选择器就变成 `nodename=~""`——**什么都匹配不到**。查询返回零行的变量没有任何选项,于是它保持空;而 `$node` 又是基于 `$nodename` 定义的: - -```promql -label_values(node_uname_info{job="unraid-host", nodename="$nodename"}, instance) -``` - -于是 `$node` 也是空,所有以 `$node` 为过滤条件的面板自然匹配不到序列。Grafana 说 *No data* 并没有骗人:面板查询真的没有数据,因为给它限定范围的变量解析成了空。 - -修法是去掉自引用,并给每个变量存一个明确的默认值: - -```promql -# FIXED — 不自引用,每个变量只跳一层 -# $nodename -label_values(node_uname_info{job="unraid-host"}, nodename) -# $node -label_values(node_uname_info{job="unraid-host", nodename="$nodename"}, instance) -``` - -## 为什么我的验证没抓到 - -这才是值得抄的部分。我\"验证\"面板的方式是:**手动**把每个面板表达式里的变量替换成我知道正确的值(`$nodename=unRaid`、`$node=host.docker.internal:9100`),然后断言查询有数据点。13 项检查全部通过——面板依然是空的,因为 bug 在被替换的那个值**上游**:Grafana 自己对变量的解析是空的。 - -**一个替换掉\"可疑值\"的验证,永远发现不了那个值解析失败。** 要抓它,得读 Grafana 真实存下来的值: - -```bash -curl -s -u admin:"$PW" http://grafana:4010/api/dashboards/uid/rYdddlPWk \ - | jq '.dashboard.templating.list[] | {name, current: .current.value}' -``` - -更直接的检查对象是变量查询本身——跑 Grafana 会跑的那条: - -```bash -# $nodename 的 label_values() 到底返回什么? -curl -s -G http://prometheus:9090/api/v1/query \ - --data-urlencode 'query=count by (nodename) (node_uname_info{job="unraid-host"})' | jq '.data.result' -``` - -那里是空,面板就不可能渲染出来,无论数据多健康。 - -## 第二个坑:`$__all` 不是 `.*` - -做自动化面板检查时,值为\"All\"的变量,其 current 是哨兵字符串 `$__all`——不是正则。把 `name=~"$__all"` 原样展开会匹配不到任何东西,于是完全健康的容器面板\"全军覆没\"。展开前要把 `$__all` 解析成该变量的 `allValue`(通常是 `.*`),否则你追的是一个只存在于自己检查脚本里的 bug。 - -## 不是每个空面板都是 bug - -修完之后 25 个面板有 21 个出数。剩下 4 个是**设计上就不可能**有数据的,最好在面板说明里写清楚: - -- **PSI 面板**需要 `/proc/pressure`,而这台 NAS 的内核不暴露它——指标在这台机器上无法存在;同样的面板在会导出 `node_pressure_*` 的主机上正常显示。 -- **根文件系统面板**的表达式带 `fstype!="rootfs"`,而某台主机的 `/` 是**内存盘**(unRaid 就是),所以永远匹配不到。 - -区分\"因为 bug 空\"和\"因为这台机器不可能有这个指标而空\",是真修好和瞎忙一下午的分界线。 - -## 我下次会怎么做 - -- **永远不要定义过滤自己的模板变量。** 一个变量只跳一层。 -- **给面板依赖的变量存明确的 `current` 值**,别让全新加载依赖下拉框被选中。 -- **用 Grafana 存下来的值做验证**,而不是我以为的值。 -- **故意留空的缺口写进面板说明**,否则未来的你会花一个下午去\"修\"一个本来就没打算工作的面板。 - -## 结果 - -三台主机现在跑同一套布局——每台一个 41 面板的主机面板 + 一个 10 面板的容器面板,分别有 21/23/25 个面板出数,两类结构性空缺写在面板里,而不是留给下一个打开它的人去猜。 - -## 需要为你的业务做这个吗? - -如果你有一堆没人信任的 Grafana 面板,或者服务器和 NAS 完全没有监控,我可以帮你搭自托管的 Prometheus + Grafana(主机与容器指标、合理的告警规则、通知推到 Telegram 和邮件),也会把你那些悄悄\"不出数\"的面板修好。 - -**WhatsApp:[+60 12-797 2969](https://wa.me/60127972969)** · **邮箱:[me@hoelee.com](mailto:me@hoelee.com?subject=Grafana%20monitoring)** · **[hoelee.com](https://hoelee.com)** - -网站设计与开发是我的主业;服务器加固与自托管基础设施是它的另一半。 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Why Your Grafana Dashboard Shows No Data (When Prometheus Is Fine)" -description: "Every target up, data in Prometheus, panel expressions correct — and all 41 panels reading No data. The culprit was a Grafana variable filtering on itself, plus the verification habit that hid it." -pubDate: 2026-05-17 -updatedDate: 2026-09-29 -category: devops -blueprint: self-hosted-infrastructure -blueprintLayer: L3-observability-self-healing -tags: [grafana, prometheus, dashboards, monitoring, promql, observability] -ogImage: /og/why-your-grafana-dashboard-shows-no-data.jpg -banner: /banners/why-your-grafana-dashboard-shows-no-data.jpg -draft: false ---- - -You open the dashboard you use every day and it's empty. Not "one panel broke" empty — -**every panel**, top to bottom, says *No data*. You check Prometheus and everything is -healthy. This post is the specific, non-obvious cause I hit, and the way my own -verification convinced me the dashboard was fine while it wasn't. - -## The searchable question - -*My Grafana panels show No data but the data is in Prometheus — why?* - -The usual answers (wrong datasource, wrong time range, `rate()` on a counter that -resets, missing scrape target) did not apply. Here is the checklist that narrowed it -down, in the order that costs the least time: - -```bash -# 1. is the target actually being scraped? -curl -s http://prometheus:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job, health}' - -# 2. does the metric exist right now? (bypasses Grafana entirely) -curl -s -G http://prometheus:9090/api/v1/query \ - --data-urlencode 'query=node_uname_info{job="unraid-host"}' | jq '.data.result[0].metric' - -# 3. is the panel's expression sound? -curl -s -G http://prometheus:9090/api/v1/query \ - --data-urlencode 'query=count(node_cpu_seconds_total{mode="idle",job="unraid-host"})' -``` - -All three were green: 7 targets `up`, 25k series ingested, and the panel's raw -expression returned data when I ran it by hand. So the problem was not the metrics, -the scrape, or the PromQL — it was **the variable layer between them**. - -## The actual cause: a variable that filtered on itself - -The dashboard had a host picker. Two variables chained: `$nodename` (which machine) -and `$node` (its `instance` label). The definition of the first one was: - -```promql -# BROKEN — the variable filters on its own value -label_values(node_uname_info{job="unraid-host", nodename=~"$nodename"}, nodename) -``` - -On first load `$nodename` has no value, so the selector becomes -`nodename=~""` — which matches **nothing**. A variable whose query returns no rows has -no options, so it stays empty. `$node` is then defined against `$nodename`: - -```promql -label_values(node_uname_info{job="unraid-host", nodename="$nodename"}, instance) -``` - -…which is also empty, so every panel filtering on `$node` matches no series. Grafana -isn't lying when it says *No data*: the panel query genuinely has no data, because the -variable that scopes it resolved to nothing. - -The fix is to stop the self-reference and give each variable an explicit saved default: - -```promql -# FIXED — no self-reference, one hop per variable -# $nodename -label_values(node_uname_info{job="unraid-host"}, nodename) -# $node -label_values(node_uname_info{job="unraid-host", nodename="$nodename"}, instance) -``` - -Then save `current` values for both so the dashboard opens populated instead of -depending on a click. (This is also why the same 41-panel layout can be cloned to -three hosts and just work.) - -## Why my verification missed it - -This is the part worth stealing. I "verified" the dashboard by expanding each panel's -expression **by hand**, substituting the variable values I knew were correct -(`$nodename=unRaid`, `$node=host.docker.internal:9100`), and asserting the query -returned points. Thirteen of those checks passed. The dashboard was still blank, -because the bug was upstream of the substitution: Grafana's *own* resolution of the -variables was empty. - -**A verification that substitutes the very value under suspicion cannot detect that -value failing to resolve.** To catch it, read the values Grafana actually saved and -expand with those: - -```bash -curl -s -u admin:"$PW" http://grafana:4010/api/dashboards/uid/rYdddlPWk \ - | jq '.dashboard.templating.list[] | {name, current: .current.value}' -``` - -The correct check is on the variable query itself — run what Grafana runs: - -```bash -# what does $nodename's label_values() actually return? -curl -s -G http://prometheus:9090/api/v1/query \ - --data-urlencode 'query=count by (nodename) (node_uname_info{job="unraid-host"})' | jq '.data.result' -``` - -Empty there means the dashboard cannot possibly render, no matter how healthy the data is. - -## The second trap: `$__all` is not `.*` - -While automating panel checks, a variable set to *All* reports its current value as the -sentinel string `$__all` — not a regex. Expanding panel expressions naively turns -`name=~"$__all"` into a literal that matches nothing, so a perfectly healthy -container dashboard "fails" every panel. Resolve `$__all` to the variable's `allValue` -(usually `.*`) before expanding, or you'll chase a bug that only exists in your checker. - -## Not every empty panel is a bug - -After the fix, 21 of 25 panels had data. The remaining four were empty **by design** and -worth naming honestly in the panel description: - -- **PSI panels** (`Pressure`, `Pressure Stall Information`) need `/proc/pressure`, which - the NAS kernel doesn't expose. The metric cannot exist there — the same panels on a - host that *does* export `node_pressure_*` render fine. -- **Root filesystem panels** whose expression carries `fstype!="rootfs"` can never match - on a host whose `/` is a **ramdisk**, which is what unRaid's root is. - -Distinguishing "broken because of a bug" from "empty because the metric cannot exist on -this host" is the difference between a real fix and a wild goose chase. - -## What I'd do differently - -- **Never define a template variable that filters on itself.** One hop per variable. -- **Always save explicit `current` values** for variables a dashboard depends on, so a - fresh load doesn't rely on a populated dropdown. -- **Verify with Grafana's saved values, not my idea of them.** The masking is invisible - in the test result and obvious in the UI. -- **Describe intentional gaps in the panel description.** Future-you will otherwise - spend an afternoon "fixing" a panel that was never meant to work there. - -## The result - -All three hosts now run the same instrumented layout — a 41-panel host dashboard and a -10-panel container dashboard each — with 21/23/25 panels returning data respectively, -and the two structurally-empty classes documented in place instead of silently -confusing whoever opens them next. - -## Want this for your business? - -If you have Grafana dashboards that nobody trusts — or servers and a NAS with no -monitoring at all — I build self-hosted Prometheus + Grafana stacks (host and container -metrics, sensible alert rules, alerts to Telegram and email) and I'll fix the ones that -have quietly stopped showing data. - -**WhatsApp: [+60 12-797 2969](https://wa.me/60127972969)** · **Email: [me@hoelee.com](mailto:me@hoelee.com?subject=Grafana%20monitoring)** · **[hoelee.com](https://hoelee.com)** - -Website design and development is my main line of work; server hardening and -self-hosted infrastructure is the other half of it. -```` - ---- - -## 文章 03 / 11:你的“磁盘将满”告警在说谎:别再按百分比告警 -- **Slug**: `your-disk-full-alert-is-lying` -- **英文标题**: Your Disk-Full Alert Is Lying: Stop Alerting on Percentages -- **分类**: `devops` | **发布日期**: 2026-06-24 -- **线上地址**: https://blog.hoelee.com/posts/your-disk-full-alert-is-lying/ (EN) | https://blog.hoelee.com/posts/zh/your-disk-full-alert-is-lying/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】:[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】: -【人设与用词卫生】: -【CRO与转化建议】: -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "你的“磁盘将满”告警在说谎:别再按百分比告警" -description: "\"剩余 3%\"听着像紧急事故,直到你发现那是 500 GB。在多 TB 的卷上,百分比阈值会在没事的时候乱叫,也会在真出事的时候沉默。" -pubDate: 2026-06-24 -updatedDate: 2026-09-29 -category: devops -blueprint: self-hosted-infrastructure -blueprintLayer: L3-observability-self-healing -tags: [monitoring, alerting, prometheus, grafana, storage, capacity, observability] -ogImage: /og/zh-your-disk-full-alert-is-lying.jpg -banner: /banners/your-disk-full-alert-is-lying.jpg -draft: false ---- - -我搭好监控、写了一套看着合理的规则,然后用一个下午删掉了其中大半阈值。它们全是百分比,而且都以同一种方式错了。 - -## 三个教我道理的告警 - -**1.「文件系统使用率超过 90%」**——它在一个还剩约 500 GB 的 17 TB 卷上触发。没事发生,也不会马上有事发生。这种体量的卷,一次大备份就能让百分比摆动几个点:比例是噪声,而\"还有 500 GB 可用\"才是世界的真实状态。 - -**2.「内存使用率超过 90%」**——在一台**可用**内存 4.8 GB 的主机上反复触发。Linux 会用空闲内存做 page cache,需要时立刻归还;`MemTotal - MemFree` 描述的是文件缓存,不是你的风险。真正预示 OOM 的是 `MemAvailable`。 - -**3.「CPU steal 超过 25%」**——一台把邮件栈跑得完全正常的 VPS,steal 长期在 41%。steal 说明宿主机忙,它是**容量**信号而不是**故障**信号;7 vCPU 的机器 41% steal 就是这台机器的成本。这条规则产生的告警永远为真、也永远无用。 - -共同点:**我在对比例告警,而比例不知道东西有多大。** - -## 对\"后果\"告警,而不是对比例 - -我现在对每个阈值都问一句:*如果这个状态持续下去,实际会发生什么?* 答案几乎总能用绝对单位表达。 - -| 不要这样 | 改成 | 原因 | -|---|---|---| -| 文件系统 > 90% | 剩余空间 < N GB | \"我还能不能写进去\"才是问题 | -| 内存使用 > 90% | `MemAvailable` < 512 MB | 逼近 OOM,而不是\"缓存很暖和\" | -| CPU steal > 25% | steal > 50% | 容量成本 vs 真的被饿死 | -| 负载 > N | `load1 / 核数 > 2` | 不谈核数的负载没有意义 | - -PromQL 的前后对比: - -```promql -# BEFORE — 17 TB 卷的百分比;还剩半 TB 就开叫 -(1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 90 - -# AFTER — 真正装数据的卷上,还剩多少容量 -min by (instance, mountpoint) ( - node_filesystem_avail_bytes{mountpoint=~"/volume[0-9]+|/mnt/ssd|/mnt/disk[0-9]+"} -) < 25e9 -``` - -```promql -# BEFORE — 把 page cache 当成压力 -(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90 - -# AFTER — 真正的 OOM 前兆 -node_memory_MemAvailable_bytes < 512 * 1024 * 1024 -``` - -两个让规则能活下来的细节: - -**把比较放进阈值条件里,不要塞进表达式。** 查询就写 `min by (mountpoint) (node_filesystem_avail_bytes{...})`,让告警规则去和 `25000000000` 比。若你在 PromQL 里写死 `< 25e9`、规则里又留着继承来的 `> 90`,规则照样会评估——但界面显示的阈值是错的,下一个读它的人得从两个互相矛盾的条件下反推你的意图。 - -**排除那些\"只是另一个挂载点的视图\"。** 一条\"剩余空间\"规则有多诚实,取决于它的选择器。在我自己的环境里,同一个 NAS 卷出现三次(`/volume1`、`/opt`、以及在另一台主机上以 CIFS 再次挂载),而 unRaid 的 shfs 联合挂载报告 `avail=0`——不加排除的规则不是永远误报,就是把容量算成两倍。一个写明白的选择器胜过聪明的通用写法。 - -## 噪声不是无害的 - -坏阈值的代价不在告警本身,而在**训练**:每条\"没事乱叫\"的告警都在教你先扫一眼、然后忽略;等你真正该看的那条到来时,它出现在一个你早已不再读的频道里。我的规则总数**减少了**,覆盖率却上升了——更少,但每条都对应一个明确的后果。 - -## 阈值是私人的,形状是通用的 - -上面的数字是我的:NAS 卷剩不到 25 GB 我会在意、可用内存低于 512 MB 我才管、VPS steal 超过 50% 才算被饿。你的数字随硬件和容忍度而变。 - -能通用的是形状: - -- 用绝对单位而非比例; -- 把后果写进告警的摘要里; -- 选择器明确声明它在替哪些挂载点说话; -- 规则数量少到你还能读完每一条。 - -## 结果 - -同一套栈,从\"三条永久无用报警\"变成 16 条在机房健康时保持安静的规则——其中一条立刻暴露了一个只剩 21 GB 的 NAS 卷,而那正是百分比规则一直藏在\"99% 满的巨物\"背后、被你学会忽略的东西。 - -## 需要为你的业务做这个吗? - -如果你的监控在发你已经学会忽略的告警,或者你根本没有监控、宁愿从一条规则而不是从一次失败的备份里得知磁盘满了,我可以帮你搭自托管的监控与告警(Prometheus + Grafana,通知推 Telegram 和邮件),阈值按\"什么会真的坏\"来定。 - -**WhatsApp:[+60 12-797 2969](https://wa.me/60127972969)** · **邮箱:[me@hoelee.com](mailto:me@hoelee.com?subject=Monitoring%20and%20alerting)** · **[hoelee.com](https://hoelee.com)** - -网站设计与开发是我的主业;服务器加固与自托管基础设施是它的另一半。 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Your Disk-Full Alert Is Lying: Stop Alerting on Percentages" -description: "3% free sounds like an emergency until you notice it is 500 GB. On multi-terabyte volumes, percentage thresholds fire when nothing is wrong — and miss when something is." -pubDate: 2026-06-24 -updatedDate: 2026-09-29 -category: devops -blueprint: self-hosted-infrastructure -blueprintLayer: L3-observability-self-healing -tags: [monitoring, alerting, prometheus, grafana, storage, capacity, observability] -ogImage: /og/your-disk-full-alert-is-lying.jpg -banner: /banners/your-disk-full-alert-is-lying.jpg -draft: false ---- - -I built a monitoring stack, wrote a reasonable set of rules, and then spent an -afternoon deleting most of my own thresholds. Every one of them was a percentage, and -every one of them was wrong in the same specific way. - -## The three alerts that taught me - -**1. "Filesystem above 90% used."** It fired on a 17 TB volume that had ~500 GB free. -Nothing was wrong. Nothing was *about* to be wrong. On a volume that size, a single -large backup can swing the percentage by several points — the ratio is noise, and the -fact that 500 GB is still available is the actual state of the world. - -**2. "Memory above 90% used."** Fired constantly on a host with 4.8 GB *available*. -Linux uses free RAM for page cache and gives it back on demand; `MemTotal - MemFree` -describes your filesystem cache, not your risk. The number that predicts an OOM is -`MemAvailable`. - -**3. "CPU steal above 25%."** Fired on a VPS that had been running its mail stack -perfectly at 41% steal. Steal means the hypervisor is busy — it is a *capacity* signal, -not a *failure* signal, and 41% steal on a 7-vCPU box was simply what that host costs. -The rule produced alerts that were always true and never actionable. - -The common thread: **I was alerting on a ratio, and the ratio doesn't know how big the -thing is.** - -## Alert on the consequence, not the ratio - -The test I now apply to every threshold: *if this condition persists, what will actually -happen?* That answer is almost always expressible in absolute units. - -| Instead of | Alert on | Because | -|---|---|---| -| Filesystem > 90% | Free space < N GB | "Can I still write?" is the question | -| Memory > 90% used | `MemAvailable` < 512 MB | about to OOM, not "cache is warm" | -| CPU steal > 25% | Steal > 50% | capacity cost vs. starvation | -| Load average > N | `load1 / cores > 2` | load is meaningless without core count | - -In PromQL, before and after: - -```promql -# BEFORE — percentage of a 17 TB pool; fires with half a terabyte still free -(1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 90 - -# AFTER — capacity left, on the volumes that hold data -min by (instance, mountpoint) ( - node_filesystem_avail_bytes{mountpoint=~"/volume[0-9]+|/mnt/ssd|/mnt/disk[0-9]+"} -) < 25e9 -``` - -```promql -# BEFORE — page cache counted as "pressure" -(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90 - -# AFTER — the actual OOM precursor -node_memory_MemAvailable_bytes < 512 * 1024 * 1024 -``` - -Two details that make these rules survive contact with a real dashboard: - -**Keep the comparison in the threshold condition, not inside the expression.** Query -`min by (mountpoint) (node_filesystem_avail_bytes{...})`, then let the alert rule -compare it to `25000000000`. If you bake `< 25e9` into the PromQL *and* leave an -inherited `> 90` in the rule, the rule still evaluates — but it displays a nonsense -threshold, and the next person to read it has to reconstruct your intent from two -contradictory conditions. - -**Exclude the mounts that are views of other mounts.** A "free space" rule is only as -honest as its selector. On my own setup, one NAS volume appears three times (as -`/volume1`, as `/opt`, and again on another host as a CIFS re-mount), and unRaid's shfs -union mount reports `avail=0` — a rule without exclusions would either fire forever or -double-count capacity. One explicit selector beats a clever generic one: - -```promql -node_filesystem_avail_bytes{mountpoint=~"/volume[0-9]+|/mnt/ssd|/mnt/disk[0-9]+"} -``` - -## Noise is not harmless - -The cost of a bad threshold isn't the alert itself — it's the *training*. Every alert -that fires while nothing is wrong teaches you to skim, then to ignore, and then the one -that matters arrives in a channel you've stopped reading. My rule count went **down** -while coverage went up: fewer, each tied to a named consequence. - -## Thresholds are personal, shapes are not - -The specific numbers above are mine — 25 GB free before a NAS volume bothers me, 512 MB -available before I care about memory, 50% steal before a VPS is being starved. Yours -will differ with your hardware and your tolerance. - -What transfers is the shape: - -- absolute units over ratios, -- the consequence written into the alert's summary, -- a selector that states exactly which mounts it speaks for, -- and a rule count small enough that you still read every message. - -## The result - -The same stack that was producing three permanent, uninformative warnings now runs 16 -rules that stay silent when the lab is healthy — and one of them immediately surfaced a -NAS volume sitting at 21 GB free, which is the kind of thing a percentage rule had been -hiding behind a 99%-full giant that I'd learned to ignore. - -## Want this for your business? - -If your monitoring sends you alerts you've learned to ignore, or you have none and would -rather find out about a full disk from a rule than from a failed backup, I set up -self-hosted monitoring and alerting (Prometheus + Grafana, alerts to Telegram and email) -with thresholds tied to what actually breaks. - -**WhatsApp: [+60 12-797 2969](https://wa.me/60127972969)** · **Email: [me@hoelee.com](mailto:me@hoelee.com?subject=Monitoring%20and%20alerting)** · **[hoelee.com](https://hoelee.com)** - -Website design and development is my main line of work; server hardening and -self-hosted infrastructure is the other half of it. -```` - ---- - -## 文章 04 / 11:浏览器翻译没有 API:我给一个纯中文网页应用加上了英文模式 -- **Slug**: `adding-english-mode-to-a-chinese-only-web-app` -- **英文标题**: No API for Browser Translation: Adding an English Mode to a Chinese-Only Web App -- **分类**: `engineering` | **发布日期**: 2025-06-11 -- **线上地址**: https://blog.hoelee.com/posts/adding-english-mode-to-a-chinese-only-web-app/ (EN) | https://blog.hoelee.com/posts/zh/adding-english-mode-to-a-chinese-only-web-app/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】:[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】: -【人设与用词卫生】: -【CRO与转化建议】: -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "浏览器翻译没有 API:我给一个纯中文网页应用加上了英文模式" -description: "浏览器不提供触发翻译的 JavaScript API,而我的应用明明是中文却声明自己是英文。于是我改用反向代理,把一个英文模式注入进去。" -pubDate: 2025-06-11 -updatedDate: 2026-09-29 -category: engineering -blueprint: sme-digital-suite -blueprintLayer: L1-high-conversion-web -tags: ["i18n", "nginx", "vue", "javascript", "self-hosting"] -ogImage: "/og/zh-adding-english-mode-to-a-chinese-only-web-app.jpg" -banner: "/banners/adding-english-mode-to-a-chinese-only-web-app.jpg" -draft: false ---- - -「能不能用 JavaScript 触发浏览器自带的翻译?」答案是**不能**。接受这一点花了我一个下午 —— 但更有意思的发现出现在前面:我的应用其实一直在破坏浏览器的翻译功能,而且只用一个属性就做到了。 - -## 我在做什么 - -我维护着一个私人的邀请制阅读平台 `book.hoelee.com`:自建的阅读器(Kotlin/Spring 后端 + Vue 单页应用),装着我自己的藏书,并给受邀读者开了账号。界面是纯中文 —— 不是「以中文为主」,而是**只有中文**。我在相信这一点之前把每个前端 bundle 都翻了一遍:没有 i18n 库、没有语言包、没有语言切换,只有一个简繁转换表和一堆英文 TTS 语音名。 - -后来我开始把邀请卡发给只说英文的读者。他们能注册,然后迎面撞上:书架、浏览书仓、确定、设置 —— 每一个控件都在他们读不懂的语言里。这不是「体验不够精致」的问题。对我邀请的一半人来说,这个产品是不可用的。 - -有一条约束决定了后面所有做法:**我不改这个应用。** 它的上游仓库已经归档,界面是压缩过的 Vue bundle —— 任何我打进镜像里的补丁,都会在下一次换镜像时消失。所以我做的东西必须待在**外面一层**:我已经架在应用前面的那个 nginx 网关。 - -## 发现一:应用声明了自己错误的语言,而这会关掉翻译 - -写代码之前,我先看看浏览器到底看到了什么: - -```bash -$ curl -s https://book.hoelee.com/index.html | grep -o ']*>' - -``` - -外壳声明英文,界面 100% 中文。 - -这件事比看上去严重。Chrome、Edge、Safari 决定「要不要**主动提示**翻译」,很大程度取决于页面声明的语言。一个声明自己是英文、却渲染中文内容的页面,不会得到那个「要翻译此页面吗?」的提示 —— 对我的新读者来说最有用的一项无障碍功能,就这么被悄悄关掉了;而「右键 → 翻译成中文/英文」是大多数人从来不知道的操作。 - -修复只要一行,而且是作用在**响应出站时**,不是打进应用里: - -```nginx -location = /index.html { - # sub_filter 不能改写 gzip 后的内容 -> 让上游返回未压缩的 HTML - proxy_set_header Accept-Encoding ""; - sub_filter_once on; - # 应用声明 lang="en" 而界面是中文 -> 浏览器因此永远不提示中译英。 - # 说出事实。 - sub_filter '' ''; - proxy_pass http://hectorqin-reader:8080; -} -``` - -用发现它的方式验证:`curl -s https://book.hoelee.com/index.html | grep -o '' ''; - proxy_pass http://hectorqin-reader:8080; -} -``` - -整个部署就是这样:一个 40 KB 的脚本,只在 `/index.html` 加载,所以按钮只存在于应用内,别处都没有。脚本做三件事。 - -### 1. 从应用自己的 bundle 里「收割」词汇表 - -界面文案本来就在压缩后的 JavaScript 里,以字符串字面量的形式存在。把所有含中日韩字符的字符串抓出来并计数: - -```python -import re, collections -CJK = re.compile(r'[\u4e00-\u9fff]') -count = collections.Counter() -for bundle in bundles: - for m in re.finditer(r'"([^"\\\n]{2,40})"', bundle): - if CJK.search(m.group(1)): - count[m.group(1)] += 1 -print(len(count)) # 840 条唯一字符串,按出现频率排序 -``` - -跑出 840 条唯一字符串:菜单(书架、书源管理)、确认框(确认要删除所选择的书籍吗?)、设置项(段落行距)、错误信息(本地书籍源文件不存在)。词典最终是 **871 条** —— 收割来的,加上我在英文模式下实际使用应用、看还有什么没翻而补上的。 - -### 2. 按 DOM 的真实样子匹配,而不是你希望的样子 - -朴素的子串替换会毁掉中文词组:书架 就藏在 加入书架 里面,单字更糟(页 藏在 页面 里)。三条规则解决了它: - -- **先做整标签匹配,基于去掉空白后的文本。** 应用里的标签是 `" 书架 "` 这种带空格的,所以词典里的 `设置` 必须能匹配 DOM 里的 `" 设置 "`。 -- **然后是最长优先的子串替换**,只允许长度 ≥ 2 的键参与。 -- **单字键只做整标签匹配**(章、页、条),这样它们永远不可能破坏更长的词。 - -```js -function tr(s) { - if (!s || !CJK.test(s)) return s; - var hit = EXACT[s.replace(/[\s\u3000]/g, '')]; // 整标签匹配,空白归一化 - if (hit) return keepWS(s, hit); - var t = s; - for (var i = 0; i < KEYS.length; i++) { // KEYS:最长优先,长度 >= 2 - if (t.indexOf(KEYS[i]) >= 0) t = t.split(KEYS[i]).join(DICT[KEYS[i]]); - } - // 收拾数量短语:共58个可用书源 -> "58 usable sources" - return t.replace(/共\s*(\d+)/g, '$1').replace(/(\d+)\s*个/g, '$1').replace(/(\d)([A-Za-z])/g, '$1 $2'); -} -``` - -### 3. 跟上框架的重绘 - -它是 Vue 应用,DOM 一直在被重写 —— 只跑一次的翻译,一秒后就有一半是错的。一个带 150 ms 防抖的 `MutationObserver`,在任何变化之后重跑一遍遍历。 - -这一遍是幂等的:标签一旦变成英文就不再含中日韩字符,于是不会被再写一次,也不会形成写入循环。文本节点之外,`placeholder`、`title`、`aria-label`、`alt` 和输入框的 `value` 也会翻译 —— 所以提示气泡和搜索框也一起变英文了。 - -UI 是右下角的一个小胶囊:关闭时显示 `EN`,开启后显示 `中文`,选择记在 `localStorage`,还提供「隐藏此按钮」给更愿意用自己翻译器的读者。 - -## 它**刻意**不翻译什么 - -**书籍正文。** 阅读器把 EPUB 内容渲染在 iframe 里,我的遍历器按设计跳过 iframe —— 而且我无论如何也不会去机器替换一本小说的正文。 - -同样保持原样的还有:书名与作者、书源名称、我自己建的分组名(梯子、精品、正版),以及应用那句诗。这些是**数据**,不是界面。翻译界面是一种体贴;悄悄改写用户自己的内容是另一回事,而且糟糕得多。 - -书籍正文仍然交给浏览器自带的翻译 —— 而且因为「发现一」,它现在真的会被提示出来。按钮的弹窗里就是这么写的。 - -## 要量,不要用眼睛看 - -「截图看起来基本是英文了」不算结果。我在开关按钮的前后,数了实时 DOM 里含中日韩字符的文本节点: - -```js -let cjk = 0, total = 0; // 遍历器跳过注入的 UI 与 script/style -while ((n = walker.nextNode())) { - const t = n.nodeValue.trim(); - if (!t) continue; - total++; - if (/[\u4e00-\u9fff]/.test(t)) cjk++; -} -``` - -| 界面 | 之前 | 之后 | 已翻译 | -|---|---|---|---| -| 登录 / 注册对话框 | 80 | 12 | 85% | -| 书架 + 设置 | 259 | 60 | 77% | -| 阅读页界面 | 214 | 31 | 88% | - -那些界面里仍被计为中文的,全部是数据 —— 书名、作者、书源名、我自己的分组名、那句诗 —— 正是应该保持中文的部分。 - -## 我会怎么做得不一样 - -**不要用手工重写一份 871 条的词典。** 我重构过一次,静默丢了 25 个键。症状看起来像逻辑 bug:设置 还是中文,而它的每个兄弟标签都翻译正常。匹配器没问题 —— 是**条目**不见了。在你相信一次重写之前,用程序对比键集合: - -```bash -python -c "print(sorted(set(old_keys) - set(new_keys)))" -``` - -**自动化 UI 检查要写短。** 我的无头运行器每次调用硬超时 60 秒。一个遍历六本书的循环返回 HTTP 408 和空 body,而我的 JSON 解析器报的是「解析错误」而不是「超时」—— 我一度在查错的东西。 - -**调试之前先搞清楚 SPA 的真实路由。** `#/reader/1` 会渲染出一个空白外壳,让人以为应用坏了;真正的路由是 `#/reader?bookUrl=`。 - -**注意你自己的跳转逻辑。** 我的网关会把任何刷新送回前台页,除非设置了 `sessionStorage` 的通行标记。把界面切**回**中文会重载页面 —— 所以按钮会先设好同一个标记,否则读者会因为改了个语言设置被踢出应用。 - -## 结果 - -一个只懂英文的读者,现在打开平台、点一下按钮,就能用英文读界面。应用本身没有被改动:整个功能只是一个由 sidecar nginx 容器提供的 40 KB 脚本,所以下一次换镜像不可能弄坏它;而浏览器自带的翻译,终于有一个会声明自己真实语言的页面可以配合。 - -我留下的经验:**当平台不给你 API 时,往外看一层。** 反向代理是添加「应用里加不了的行为」的正当位置 —— 而且通常是唯一能在升级中存活的位置。 - ---- - -*我为马来西亚的中小企业搭建并自托管这类平台 —— 网页应用、私人书库、内部工具。有需要可以找我:[WhatsApp +60 12-797 2969](https://wa.me/60127972969)、[me@hoelee.com](mailto:me@hoelee.com?subject=Self-hosted%20platform),或 [hoelee.com](https://www.hoelee.com)。* -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "No API for Browser Translation: Adding an English Mode to a Chinese-Only Web App" -description: "Browsers expose no JavaScript API to trigger translation — and my app declared English while being Chinese. How I injected an English mode at the reverse proxy." -pubDate: 2025-06-11 -updatedDate: 2026-09-29 -category: engineering -blueprint: sme-digital-suite -blueprintLayer: L1-high-conversion-web -tags: ["i18n", "nginx", "vue", "javascript", "self-hosting"] -ogImage: "/og/adding-english-mode-to-a-chinese-only-web-app.jpg" -banner: "/banners/adding-english-mode-to-a-chinese-only-web-app.jpg" -draft: false ---- - -Can I trigger the browser's translate feature from JavaScript? No. That answer cost me an afternoon to accept — but the more interesting discovery came first: my app was already sabotaging the browser's translation, and it did it with a single attribute. - -## What I was working on - -I run a private, invite-only reading platform at `book.hoelee.com`: a self-hosted reader (Kotlin/Spring backend, Vue single-page app) holding my library, with accounts for invited readers. The interface is Chinese-only. Not "Chinese-first" — *only*. I grepped every served bundle before believing it: no i18n library, no locale files, no language switcher. Just a Simplified/Traditional character converter and a list of English TTS voice names. - -Then I started handing out invite cards to English-speaking readers. They could register, and then they met 书架, 浏览书仓, 确定, 设置 — every control in a language they don't read. That isn't a polish problem. For half the people I invited, the product was unusable. - -One constraint shaped everything that follows: **I don't modify the app.** Its upstream repository is archived and the UI is a minified Vue bundle, so anything I patch inside the image dies the next time I swap the image. Whatever I build has to live one layer out — in the nginx gateway I already run in front of the app. - -## Finding 1: the app declares the wrong language, and that disables translation - -Before writing a single line of code, I looked at what the browser sees: - -```bash -$ curl -s https://book.hoelee.com/index.html | grep -o ']*>' - -``` - -The shell says English. The interface is 100% Chinese. - -That matters more than it looks. Chrome, Edge and Safari decide whether to *offer* translation largely from the page's declared language. A page that declares English while rendering Chinese text doesn't get the "Translate this page?" prompt — the single most useful accessibility feature for my new readers was silently switched off, and right-click → *Translate to English* is a ritual most people never learn. - -The fix is one line, applied to the response on the way out instead of inside the app: - -```nginx -location = /index.html { - # sub_filter cannot rewrite a gzipped body -> ask upstream for plain HTML - proxy_set_header Accept-Encoding ""; - sub_filter_once on; - # the app declares lang="en" while its UI is Chinese -> browsers then never - # offer Chinese -> English translation. Declare the truth. - sub_filter '' ''; - proxy_pass http://hectorqin-reader:8080; -} -``` - -Verify the same way you found it: `curl -s https://book.hoelee.com/index.html | grep -o '' ''; - proxy_pass http://hectorqin-reader:8080; -} -``` - -That's the whole deployment: a 40 KB script, loaded only on `/index.html`, so the button exists inside the app and nowhere else. The script does three things. - -### 1. Harvest the vocabulary from the app's own bundles - -The UI strings are already in the minified JavaScript as quoted literals. Extract every one that contains a CJK character and count them: - -```python -import re, collections -CJK = re.compile(r'[\u4e00-\u9fff]') -count = collections.Counter() -for bundle in bundles: - for m in re.finditer(r'"([^"\\\n]{2,40})"', bundle): - if CJK.search(m.group(1)): - count[m.group(1)] += 1 -print(len(count)) # 840 unique strings, most frequent first -``` - -840 unique strings came out: menus (书架, 书源管理), confirmations (确认要删除所选择的书籍吗?), settings labels (段落行距), error messages (本地书籍源文件不存在). The dictionary ended up at **871 entries** — the harvest plus everything I found by running the app in English mode and watching what stayed Chinese. - -### 2. Match like the DOM actually is, not like you wish it were - -Naive substring replacement mangles Chinese compounds: 书架 sits inside 加入书架, and single characters are worse (页 inside 页面). Three rules fixed it: - -- **Whole-label match first, on whitespace-stripped text.** The app stores labels like `" 书架 "` with padding, so a dictionary key of `设置` has to match `" 设置 "` in the DOM. -- **Then longest-first substring replacement**, restricted to keys of two or more characters. -- **One-character keys are whole-label only** (章, 页, 条), so they can never corrupt a longer word. - -```js -function tr(s) { - if (!s || !CJK.test(s)) return s; - var hit = EXACT[s.replace(/[\s\u3000]/g, '')]; // whole label, whitespace-normalised - if (hit) return keepWS(s, hit); - var t = s; - for (var i = 0; i < KEYS.length; i++) { // KEYS: longest-first, length >= 2 - if (t.indexOf(KEYS[i]) >= 0) t = t.split(KEYS[i]).join(DICT[KEYS[i]]); - } - // tidy count phrases: 共58个可用书源 -> "58 usable sources" - return t.replace(/共\s*(\d+)/g, '$1').replace(/(\d+)\s*个/g, '$1').replace(/(\d)([A-Za-z])/g, '$1 $2'); -} -``` - -### 3. Keep up with the framework - -It's a Vue app, so the DOM is rewritten constantly — a translation pass that runs once is a translation pass that's half wrong a second later. A `MutationObserver` with a 150 ms debounce re-runs the walk after any change. - -The pass is idempotent: once a label is English it contains no CJK, so nothing is rewritten and there is no write loop. Text nodes plus `placeholder`, `title`, `aria-label`, `alt` and input `value` attributes all get translated, which is why tooltips and the search box come along too. - -The UI is a small pill in the bottom-right corner: `EN` when off, `中文` when on, remembered in `localStorage`, with a "Hide this button" link for readers who prefer their own translator. - -## What it deliberately does not translate - -**Book text.** The reader renders EPUB content inside an iframe; my walker skips iframes by design, and I would never machine-swap a novel's prose anyway. - -Also left alone: book titles and authors, book-source names, and the group names I created myself (梯子, 精品, 正版), plus the app's poem tagline. Those are *data*, not interface. Translating the chrome is a courtesy; silently rewriting a user's own content is a different and much worse idea. - -For book text the browser's own translation is still the right tool — and it now actually gets offered, thanks to Finding 1. The button's popup says exactly that. - -## Measure it, don't eyeball it - -"Screenshots look mostly English" is not a result. I counted CJK-containing text nodes in the live DOM before and after toggling the button: - -```js -let cjk = 0, total = 0; // walker skips the injected UI + script/style -while ((n = walker.nextNode())) { - const t = n.nodeValue.trim(); - if (!t) continue; - total++; - if (/[\u4e00-\u9fff]/.test(t)) cjk++; -} -``` - -| Screen | Before | After | Translated | -|---|---|---|---| -| Login / register dialog | 80 | 12 | 85% | -| Bookshelf + settings | 259 | 60 | 77% | -| Reading page chrome | 214 | 31 | 88% | - -Everything still counted as Chinese on those screens is data — titles, authors, source names, my own group names, the tagline — which is exactly what should stay Chinese. - -## What I'd do differently - -**Don't rewrite an 871-entry dictionary by hand.** I restructured mine once and silently dropped 25 keys. The symptom looked like a logic bug: 设置 stayed Chinese while every sibling label translated correctly. The matcher was fine — the *entry* was gone. Diff the key sets programmatically before you trust a rewrite: - -```bash -python -c "print(sorted(set(old_keys) - set(new_keys)))" -``` - -**Keep automated UI checks short.** My headless runner hard-times-out at 60 seconds per call. A six-book loop came back HTTP 408 with an empty body, and my JSON parser reported a parse error rather than a timeout — I debugged the wrong thing for a while. - -**Learn the SPA's real routes before debugging it.** `#/reader/1` renders a blank shell and makes you think the app is broken; the actual route is `#/reader?bookUrl=`. - -**Mind your own bounce logic.** My gateway sends any reload back to the front page unless a `sessionStorage` pass flag is set. Switching the interface *back* to Chinese reloads the page — so the button sets that same flag first, or the reader gets thrown out of the app for changing a language setting. - -## Result - -An English-only reader now opens the platform, taps one button, and reads the interface in English. The app itself is untouched: the whole feature is a 40 KB script served by a sidecar nginx container, so the next image update cannot break it, and the browser's own translation has a page that declares its real language to work with. - -The lesson I'm keeping: **when the platform won't give you an API, look one layer out.** A reverse proxy is a legitimate place to add behaviour you can't add in the app — and it's usually the only place that survives upgrades. - ---- - -*I build and self-host platforms like this one — web apps, private libraries, internal tools for small businesses in Malaysia. If you need something similar: [WhatsApp +60 12-797 2969](https://wa.me/60127972969), [me@hoelee.com](mailto:me@hoelee.com?subject=Self-hosted%20platform), or [hoelee.com](https://www.hoelee.com).* -```` - ---- - -## 文章 05 / 11:排查 Remotion 视频抽搐:为什么在背景图挂载音频低频 Scale 会毁了转场 -- **Slug**: `debugging-remotion-video-jitter` -- **英文标题**: Debugging Remotion Video Jitter: Why Bass Scaling Ruins Background Transitions -- **分类**: `engineering` | **发布日期**: 2025-11-26 -- **线上地址**: https://blog.hoelee.com/posts/debugging-remotion-video-jitter/ (EN) | https://blog.hoelee.com/posts/zh/debugging-remotion-video-jitter/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】:[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】: -【人设与用词卫生】: -【CRO与转化建议】: -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "排查 Remotion 视频抽搐:为什么在背景图挂载音频低频 Scale 会毁了转场" -description: "深入排查 Remotion 音频可视化中背景画面剧烈抽搐与切镜暗闪的根本原因:解耦 HUD 与背景图层,以及纯净单向叠化与硬件基准实测对比。" -pubDate: 2025-11-26 -updatedDate: 2026-10-22 -category: engineering +title: "用 YuE2 BF16 生成完整长歌:突破时长限制与音频硬截断" +description: "AI 音乐工具常在歌词中途戛然而止。本文详解如何利用 YuE2 BF16、295秒余量预留公式与本地 ComfyUI,成功批量跑出 60 首自然收尾的完整长歌全集。" +pubDate: 2026-10-26 +category: tutorials +blueprint: ai-agent-automation +blueprintLayer: L4-aigc-media-pipelines stream: creative-lab -tags: ["remotion", "react", "audio-visualizer", "css-animation", "debugging"] -ogImage: "/og/zh-debugging-remotion-video-jitter.jpg" -banner: "/banners/debugging-remotion-video-jitter.jpg" +tags: ["ai-music", "yue2", "comfyui", "audio-engineering", "subtitles", "python"] +ogImage: "/og/zh-generating-full-length-songs-with-yue2-comfyui.jpg" +banner: "/banners/generating-full-length-songs-with-yue2-comfyui.jpg" draft: false --- -使用 React 和 Remotion 编写自动化视频赋予了开发者对每一帧画面的极致掌控。你可以直接将 Web Audio 的频域分析数据映射到 SVG 属性、CSS 变换矩阵以及排版样式上,这种灵活性是传统非线性剪辑软件很难比拟的。 +在当前几乎所有的 AI 音乐生成方案中,最普遍且让人抓狂的缺陷莫过于: +**歌曲唱到一半,突然被硬生生切断**。 -然而,将视频帧当作常规网页来渲染时,极易踩入微妙的视觉陷阱。在我们的自动化音乐视频生成管线中,一个在开发组件预览时看似完全正常的设计,在全量导出为 1080P 30fps 视频后,却出现了极其刺眼、让人目眩的“地震式剧烈抽搐”。 +不管是各大商用平台还是开源权重,受限于固定的声学 Token 训练窗口,只要我们尝试塞入完整的 +叙事性歌词结构——即标准的 9 阶段曲式 `[Intro] -> [Verse 1] -> [Chorus] -> [Interlude] -> [Verse 2] -> [Chorus] -> [Bridge] -> [Chorus] -> [Outro]`——生成引擎就极易在进入副歌高潮或尾奏时撞上时长硬天花板,在歌手吐字的中途突兀静音。 -本文记录这次视觉 BUG 的深度排查过程:为什么对音频动效的朴素直觉会毁了整部视频,以及如何重构 React 视频图层架构以实现丝滑稳定的影视级转场。 +在单张 RTX 3060 12GB 显卡上经历数百次实跑与测试后,我摸索出了一套能够稳定产出 +**4 分钟以上、具备自然淡出收尾、绝不截断**的完整长歌工程链路。 -> **线上成品验证**:修复后的渲染流水线已正式交付并发布至 YouTube:[《糖画小仙》1080p 官方 MV](https://youtu.be/vZbyKeCCSRA),全片背景稳固、转场平滑。 - -> **TL;DR** —— 让整幅背景图跟着低频段缩放,等于每一帧都重新做图像插值采样,肉眼读到的就是剧烈抖动与震颤;再加上两层半透明图层对称交叉淡化,在时间中点必然透出底层的黑色容器。修复方案:背景画面绝对静态,将所有音频动效彻底隔离在顶层 HUD 频谱层,转场采用「100% 实体底图 + 顶层单向淡入」。渲染耗时实测**缩短 38.0%(从 352 秒降至 218 秒,吞吐速度提升 1.61 倍)**。这一篇是 [全自动 MV 生成流水线](/posts/zh/how-i-built-an-automated-music-video-pipeline/) 的渲染篇;文件落盘与发布那一半见 [Docker 容器大文件上传](/posts/zh/uploading-large-videos-to-youtube-via-cdp/)。 +本文将拆解导致音频硬截断的底层成因、295 秒余量预留公式,并完整交付 **60 首高质量长歌全集**(每首均配套 1024×1024 独立 WebP 封面与逐行同步滚动歌词)。 --- -## 1. 故障现象:全画面的“高频地震” + -该视频合成组件最初的视觉设想非常清晰: -1. 底层是一组 1080P 高清插画分镜,在不同章节间慢速淡入淡出(Crossfade); -2. 画面上方浮动一个跟随音乐重音律动的环形音频频谱仪; -3. 三行书法歌词居中显示在音频环内,并根据每张插画的主体位置在画面左右两侧平滑滑移避让。 - -在开发工具里单帧点按检查时,画面一切正常。但在以 30fps 正常播放渲染出的成片时,肉眼观感极度难受:每当强烈的电子重音鼓点响起,整张背景图每秒剧烈颤抖震荡数十次;不仅如此,在两张图片交替切镜的 1.8 秒窗口内,画面中央还会突然闪现明显的“暗黑闪烁”,亮度瞬间下沉。 - ---- - -## 2. 根因分析一:低频 Scale 能量污染背景图 - -翻阅最初的 `BackgroundSlideshow.tsx` 组件,我们发现了一段看似合情合理的动效代码: - -```tsx -// 埋下灾难的罪魁祸首代码 -const bassScale = interpolate(bassLevel, [0, 1], [1.0, 1.025], { - extrapolateRight: 'clamp', -}); - -// 应用在全屏背景图片上: - -``` - -在直觉中,重低音响起时让背景图随着节奏轻微放大 2.5%,听起来像是在模拟动感的镜头呼吸感。然而在真实物理渲染中: -- **电音鼓点的瞬态脉冲只有 30~60 毫秒**:低频能量在连续 2 到 3 帧之内瞬间从零冲上顶峰,又在下一帧骤跌回原点; -- **1920x1080 的巨幅背景图放大 2.5%,边缘物理像素瞬间位移多达 25 到 30 像素**; -- 这种在相邻帧之间急剧拉扯的剧烈像素位移,人眼接收到的根本不是“音乐律动”,而是被视网膜本能判定为**强烈的镜头颠簸、抽动与丢帧残影**; -- 更致命的是,在 1.8 秒的交叉淡入淡出期间,两张重叠的图片各自具有不同的锚点和缩放系数,两层图像以不同的相位同时高频抖动,产生了极其混乱的重影眩晕感。 - ---- - -## 3. 根因分析二:对称线性透明度重叠时的“黑闪”陷阱 - -第二个问题是切镜时的亮度下陷黑闪。在传统 Web CSS 中,人们通常很自然地写出: - -```css -/* 常见的渐变叠化陷阱 */ -.current-slide { opacity: 1 -> 0; } -.next-slide { opacity: 0 -> 1; } -``` - -当两张图片都以线性渐变进行透明度变换时,在正中间那颗时间点($t = 0.5$): -- 当前图片透明度 = 0.5 -- 下一张图片透明度 = 0.5 - -来看浏览器和 Remotion 的合成器是如何将这几层叠在底层的纯黑容器之上的: - -``` -┌─────────────────────────────────────────────────────────────┐ -│ Layer 2: 下一张图片(Next Slide,透明度 = 0.5) │ -├─────────────────────────────────────────────────────────────┤ -│ Layer 1: 当前图片(Current Slide,透明度 = 0.5) │ -├─────────────────────────────────────────────────────────────┤ -│ Layer 0: 根容器底色(Root Container,背景色 = #000000 纯黑) │ -└─────────────────────────────────────────────────────────────┘ -``` - -由于多层半透明通道是依次向下进行乘法 Alpha 混合: -$$\text{有效综合不透明度} = 1 - (1 - 0.5) \times (1 - 0.5) = 1 - 0.25 = 0.75$$ - -合成后的总有效透明度只有 **0.75**,这意味着多达 **25% 的底层纯黑背景**被暴露了出来!这不是错觉,而是纯粹的数学光度衰减。每一次转场切镜,画面整体照度都会凭空跌掉四分之一,造成非常明显的瞬间暗淡闪烁。 - ---- - -## 4. 重构方案:稳固基座 + 纯净单向叠化 - -找到症结后,我们对组件进行了彻底重构: - -### 1. 彻底剥离背景图层的所有音频缩放 -背景插画必须稳如磐石。所有的音频跳动、瞬态光斑与能量爆发,**只能且必须限定在顶层的 HUD 频谱仪中**。背景层坚决不承载任何高频振荡属性。 - -### 2. 底层 100% 实体 + 顶层单向平滑淡入 -坚决不要让底图淡出。**当前图片始终保持 100% 不透明**,下一张图片直接覆盖在其上方,慢速执行 `0 -> 1.0` 的单向淡入: - -```tsx -return ( -
- {/* 1. 底层图片:坚固不透明,彻底消除暗黑闪烁 */} - - - {/* 2. 顶层图片:在 1.8 秒内平滑淡入 (0 -> 1.0) */} - {isTransitioning && ( - - )} +
+ -); -``` - -### 3. HUD 与背景物理层级绝对隔离 -歌词弹幕与音频环容器设置在独立的顶层图层中,并赋予 `pointerEvents: 'none'`。歌词在留白侧之间的横向位移完全根据当前时间戳自主计算,与底层的图片渐变完全物理隔离,互不干涉。 - ---- - -## 5. 改造后的成效与基准复现 - -重构落地后,对比极其显著: - -1. **彻底消除画面抖动**:无论音乐鼓点多么狂暴密集,背景画面在所有测试分镜中纹丝不动,人物与细节清晰稳定; -2. **转场不再发生黑闪**:1.8 秒的单向贝塞尔叠化始终保持 100% 连续亮度,杜绝了中间态透出底层黑色的缺陷; -3. **渲染吞吐速度大幅提升**:移除了对超大尺寸背景图的高频矩阵重算后,渲染性能获得了实测提升。 - -### 基准实测与硬件复现条件 - -| 评估指标 | 优化前版本 | 优化后版本 | 变化幅度与提升 | -|---|---|---|---| -| **渲染总耗时** | 352.0 秒 | 218.3 秒 | **耗时减少 38.0% (-133.7s)** | -| **平均渲染帧率** | 22.24 fps | 35.87 fps | **吞吐速度提升 1.61 倍** | -| **总计处理帧数** | 7,830 帧 | 7,830 帧 | 完全一致(261 秒 / 4 分 21 秒) | -| **输出视频规格** | 1920 × 1080 (1080p @ 30fps) | 1920 × 1080 (1080p @ 30fps) | 完全一致 | -| **编码器设置** | H.264 (CRF 18, preset medium) | H.264 (CRF 18, preset medium) | 完全一致 | - -**基准测试宿主环境:** -- **CPU**:Intel Core i9-13900K(24 核 / 32 线程,最高睿频 5.8 GHz) -- **GPU**:NVIDIA GeForce RTX 3060(12GB GDDR6 显存,驱动版本 551.xx) -- **内存**:64GB DDR5 5600 MHz -- **系统**:Windows 11 专业版 64 位 -- **软件栈**:Node.js v20.x,Remotion v4.0.x,开启 Chromium 无头 GPU 加速 - -两次基准测试均在同一工作站、连续以完全相同的音频频谱输入和插画素材进行。渲染耗时缩短 38.0%,即吞吐速度达到优化前的 **1.615 倍**,使原本慢于真实播放时长的渲染变为高于实时帧率的高效输出。 - ---- - -## 本专题关联阅读 -- **系统总架构篇**:[我是如何用 Remotion 与 Docker CDP 打造全自动 MV 生成与多语种油管发布流的](/posts/zh/how-i-built-an-automated-music-video-pipeline/) -- **大文件上传篇**:[借助 Docker Archive API 模式通过 CDP 自动化上传 100MB+ 视频到 YouTube](/posts/zh/uploading-large-videos-to-youtube-via-cdp/) -- **线上实成片**:[前往 YouTube 观看《糖画小仙》1080P 官方完整成片](https://youtu.be/vZbyKeCCSRA) -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Debugging Remotion Video Jitter: Why Bass Scaling Ruins Background Transitions" -description: "Diagnose audio-reactive background jitter and crossfade flicker in Remotion. See how separating the HUD from background layers can improve visual stability and rendering performance." -pubDate: 2025-11-26 -updatedDate: 2026-10-22 -category: engineering -stream: creative-lab -tags: ["remotion", "react", "audio-visualizer", "css-animation", "debugging"] -ogImage: "/og/debugging-remotion-video-jitter.jpg" -banner: "/banners/debugging-remotion-video-jitter.jpg" -draft: false ---- - -Building programmatic videos with React and Remotion gives you complete DOM control over every frame. You can map web audio frequencies directly to SVG attributes, CSS transforms, and typography in ways standard video editors can barely touch. - -However, treating video frames like standard web pages introduces subtle visual bugs. In our automated music video pipeline, an export that looked plausible in unit tests developed a jarring, violent shudder when rendered into a 1080p MP4. - -Here is what went wrong, why our initial intuition about audio reactivity failed, and how to structure React video layers so audio-driven HUDs never pollute background transitions. - -> **Live Reference**: The repaired rendering pipeline produced the final production MV for [《糖画小仙》 (Sugar Art Fairy) on YouTube](https://youtu.be/vZbyKeCCSRA), running rock-solid at 1080p 30fps. - -> **TL;DR** — Scaling a full-bleed background image with the bass band forces an interpolation resampling on every frame, which the human eye perceives as violent shudder. Furthermore, two stacked semi-transparent layers crossfading symmetrically dip through the black root container at the midpoint. Fixes: keep the background canvas completely static, decouple all audio-reactive pulsing into isolated foreground HUD layers, and fade in the incoming slide on top of a 100% solid base. Render time dropped by **38.0% (from 352s to 218s, a 1.61× throughput speedup)**. This is the rendering chapter of our [music-video pipeline](/posts/how-i-built-an-automated-music-video-pipeline/); the file-staging half is covered in [uploading large videos via CDP](/posts/uploading-large-videos-to-youtube-via-cdp/). - ---- - -## 1. The Symptom: A Whole-Frame Earthquake - -The visual effect was supposed to be simple: -1. An illustrated 1080p background slowly crossfading between slides. -2. A pulsing circular spectrum visualizer in the foreground. -3. Three lines of Chinese calligraphy lyrics that slide between left and right depending on where the character in the artwork is standing. - -In preview scrubbing, everything looked acceptable. But during full-speed 30fps playback of the rendered 1080p file, the viewer felt instant visual fatigue. Every time the electronic drum pattern kicked in, the entire scene appeared to vibrate violently up to 30 times per second. Furthermore, during slide transitions, the screen experienced micro-flashes where the background brightness suddenly dipped. - ---- - -## 2. Root Cause 1: Low-Frequency Scale Pollution - -Looking at the initial `BackgroundSlideshow.tsx`, we had attached an audio reactivity hook: - -```tsx -// The culprit in BackgroundSlideshow.tsx -const bassScale = interpolate(bassLevel, [0, 1], [1.0, 1.025], { - extrapolateRight: 'clamp', -}); - -// Applied to the background image: - -``` - -On paper, expanding the image by 2.5% on bass hits sounds like an engaging dynamic pulse. In reality: -- **Electronic audio beats hit in 30–60ms bursts**: The bass amplitude oscillates rapidly between zero and peak within 2 to 3 consecutive frames. -- **A 1920x1080 image scaling by 2.5% shifts edge pixels by nearly 25 to 30 physical pixels in a single frame**. -- When that sudden displacement reverses on the very next frame, the human eye interprets it not as a musical pulse, but as **severe camera camera shake / stutter**. -- To make matters worse, during the 1.8-second slide transition, two overlapping images with different transform origins were simultaneously vibrating at conflicting scales, creating a hallucinatory double-vision jitter. - ---- - -## 3. Root Cause 2: Symmetrical Opacity Crossfade DIP - -The second issue was the crossfade flash. In standard web CSS, people often do: - -```css -/* Web crossfade trap */ -.current-slide { opacity: 1 -> 0; } -.next-slide { opacity: 0 -> 1; } -``` - -When two images crossfade with linear opacity, consider what happens at the exact midpoint ($t = 0.5$): -- $\text{Current Slide Opacity} = 0.5$ -- $\text{Next Slide Opacity} = 0.5$ - -Here is how the browser/Remotion compositing pipeline stacks these layers against the root black container: - -``` -┌─────────────────────────────────────────────────────────────┐ -│ Layer 2: Next Slide (opacity = 0.5) │ -├─────────────────────────────────────────────────────────────┤ -│ Layer 1: Current Slide (opacity = 0.5) │ -├─────────────────────────────────────────────────────────────┤ -│ Layer 0: Root Container (background = #000000 black) │ -└─────────────────────────────────────────────────────────────┘ -``` - -Because alpha blending operates multiplicatively down the stack: -$$\text{Composite Alpha} = 1 - (1 - 0.5) \times (1 - 0.5) = 1 - 0.25 = 0.75$$ - -The total combined alpha drops to **0.75**, exposing **25% of the black background**! This is not an optical illusion; it is mathematical light loss. Every time a transition peaked, the frame darkened by a quarter of its luminance, creating an unmistakable flicker. - ---- - -## 4. The Solution: Rock-Solid Base + Pure Single-Direction Opacity - -To fix both issues permanently: - -### 1. Remove All Scaling from Backgrounds -Background artwork must remain rock-solid. All audio reactivity belongs exclusively in the HUD layer (the radial spectrum bars and glowing rings). The background canvas should only ever receive smooth, non-oscillating transformations. - -### 2. Base Solid + Top Layer Fade-In -Instead of fading out the current image, **keep the current image at solid 100% opacity**. Only animate the incoming image on top: - -```tsx -return ( -
- {/* Base Slide: Stays 100% solid, eliminating any drop to black */} - - - {/* Next Slide: Smoothly fades in on top over 1.8s */} - {isTransitioning && ( - - )} +
+ + + + + + +
-); +
+ + +
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+ +## 核心痛点:240 秒的时长硬墙 + +在初期的实验中,使用默认配置调用 ComfyUI 生成音频时,经常陷入两个误区: + +1. **尾奏被腰斩(硬截断)**:若将 `max_duration` 设为 `240.0s` 并填满长歌词,模型在唱到最后一段副歌时时间刚好用尽。此前跑出的 12 首样本无一例外精确停在 `240.000s`,尾部毫无渐隐,沦为残次品。 +2. **语速失控(慌乱压缩)**:如果试图在有限的窗口内通过提示词催促模型“唱快点”,自回归声学模型会自动压缩发音时长,导致原本舒缓的民谣变成含混不清的“急口令”。 + +根本原因在于:**不能把最大时长(Max Duration)当成目标歌曲时长来设置。在自回归音频生成中,最大时长必须作为声学安全余量缓冲槽(Headroom Buffer)。** + +## 解决架构:三项核心工程策略 + +### 1. 295.0 秒 Headroom 留白公式 + +要得到一首体感约为 4 分钟的完整长歌,必须将**歌词词量规划在 210秒~260秒区间**,而底层生成参数直接赋予 **295.0 秒**的宽松上限: + +$$\text{Headroom 留白余量} = \text{参数上限 (295s)} - \text{预估词量时长 (约 220s)} \approx 75\text{秒}$$ + +这 75 秒看似多余的余量,是自回归声学模型实现“体面收尾”的命脉所在: +- 保证最后一节副歌完整舒展唱完,不产生急促抢拍; +- 赋予模型充足的 Token 生成纯乐器 Outro 尾奏; +- 让尾音伴随混响自然衰减至真正的分贝零点。 + +### 2. 自动化硬切隔离门禁(Quarantine Gate) + +在批量跑批自动化脚本中,加入严格的质量门禁:凡是实际音频长度逼近上限(`duration >= 294.9s`)的文件,判定为发生硬截断,自动归档进 `未自然结尾/` 隔离目录;仅有在上限之前自然休止收音(`duration < 294.0s`)的成品才予以放行。 + +本次实跑的 60 首曲目最终时长均匀分布在 **129.9s 至 285.6s**,100% 实现了自然淡出。 + +### 3. 12GB 显存的高效自持 + +在 12GB 显存下运行 YuE2-3B BF16,必须严格执行显存释放策略: + +```bash +# 启动 ComfyUI(关键参数防 OOM) +cd /d/app/ComfyUI +./python_embeded/python.exe -s ComfyUI/main.py \ + --windows-standalone-build \ + --port 8188 \ + --disable-pinned-memory \ + --disable-nvml-pressure \ + --cache-classic ``` -### 3. Physical Decoupling of HUD and Canvas -The lyric box and circular spectrum are mounted inside a completely separate container layer with `pointerEvents: 'none'`. As the HUD glides across the canvas from `centerX: 540` to `centerX: 1380`, it calculates its position strictly from the timeline clock. It has zero knowledge of and zero impact on the background's opacity crossfade. +在批处理完成后及时调用 `/free` 卸载显存,避免连续生成多首长音频时产生显存碎片积压。 --- -## 5. The Outcome & Benchmark Reproduction - -After removing the background `bassScale` and switching to one-way opacity stacking: - -1. **Eliminated visible jitter**: Background illustrations remain crisp and visually anchored through rapid 140 BPM electronic drum drops across all test sequences. -2. **Eliminated transition light dips**: The 1.8s Bezier crossfade maintains 100% luminance continuity without dipping through the dark container base. -3. **Significant throughput speedup**: Removing per-frame matrix interpolation and scaling calculations on high-resolution background assets delivered a measurable performance boost. - -### Reproduction Benchmark - -| Metric | Before Optimization | After Optimization | Delta | -|---|---|---|---| -| **Render Duration** | 352.0 seconds | 218.3 seconds | **-38.0% (-133.7s)** | -| **Render Throughput** | 22.24 fps | 35.87 fps | **+1.61× speedup** | -| **Total Frames** | 7,830 frames | 7,830 frames | Identical (261s / 4m 21s) | -| **Output Resolution** | 1920 × 1080 (1080p @ 30fps) | 1920 × 1080 (1080p @ 30fps) | Identical | -| **Codec & Settings** | H.264 (CRF 18, preset medium) | H.264 (CRF 18, preset medium) | Identical | - -**Test Environment Specifications:** -- **CPU**: Intel Core i9-13900K (24 cores / 32 threads, boost up to 5.8 GHz) -- **GPU**: NVIDIA GeForce RTX 3060 (12GB GDDR6 VRAM, Driver 551.xx) -- **RAM**: 64GB DDR5 5600 MHz -- **OS**: Windows 11 Pro 64-bit -- **Software**: Node.js v20.x, Remotion v4.0.x with Chromium headless GPU rendering enabled - -Both benchmark runs were executed consecutively using identical media assets, audio spectrogram inputs, and composition parameters. The 38.0% reduction in render time directly translates to a **1.61× throughput speedup**, turning a process that was previously slower than realtime into an accelerated 35.8 fps export. - ---- - -## Related Guides in this Series -- **Pipeline Architecture**: [How I Built an Automated Music Video Pipeline with Remotion and Docker](/posts/how-i-built-an-automated-music-video-pipeline/) -- **Remote Publishing**: [Uploading 100MB+ Videos to YouTube via CDP: The Docker Archive Pattern](/posts/uploading-large-videos-to-youtube-via-cdp/) -- **Live Output**: [Watch the finished MV on YouTube](https://youtu.be/vZbyKeCCSRA) -```` - ---- - -## 文章 06 / 11:把字体嵌进 PDF,让它在任何机器上都长一样 -- **Slug**: `embedding-fonts-so-a-pdf-looks-the-same-everywhere` -- **英文标题**: Embedding Fonts So a PDF Looks the Same on Every Machine -- **分类**: `engineering` | **发布日期**: 2025-01-22 -- **线上地址**: https://blog.hoelee.com/posts/embedding-fonts-so-a-pdf-looks-the-same-everywhere/ (EN) | https://blog.hoelee.com/posts/zh/embedding-fonts-so-a-pdf-looks-the-same-everywhere/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】:[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】: -【人设与用词卫生】: -【CRO与转化建议】: -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "把字体嵌进 PDF,让它在任何机器上都长一样" -description: "四个字族、八个字面、两个渲染引擎,外加一条 CSS 怪癖:为了不让生成的报告依赖服务器上恰好装了什么字体,我做了什么。" -pubDate: 2025-01-22 -updatedDate: 2026-10-16 -category: engineering -tags: ["pdf", "fonts", "mpdf", "typography", "php"] -ogImage: /og/zh-embedding-fonts-so-a-pdf-looks-the-same-everywhere.jpg -banner: /banners/embedding-fonts-so-a-pdf-looks-the-same-everywhere.jpg -draft: false ---- - -## 需求只有一句话 - -一个报告生成器必须产出在任何地方都长得一样的文档:中文标题用宋体、中文正文用黑体、英文标题与封面数字用一款展示衬线体、英文正文和表格用 Inter。字重也要对——强调用 medium、标题用 semibold——而且是三语、两个 PDF 渲染引擎都要对。 - -同时,**字体必须跟着文档走**。不是「服务器装过」,也不是「读者大概有」,而是**嵌进文件里**,这样另一个国家的印厂看到的页面是一样的。 - -老管线在最后这句上栽得很隐蔽:它依赖服务器上用包管理器装过的字体,而且**只嵌入了一款中文字体**。第二档字重根本不存在,于是「粗体」是渲染器的合成近似;而想发现这件事,唯一的办法是打开 PDF 去看里面到底装了什么。 - -## 规则一:从成品 PDF 里把字体读回来 - -「已嵌入」是一个关于**输出**的断言,所以我量的是输出。同一份文档,改前与改后: - -| | 改前 | 改后 | -|---|---|---| -| zh 页数 | 23 | **23** | -| en 页数 | 28 | **29** | -| bm 页数 | 30 | **30** | -| 真正嵌进去的字体 | 一款中文字体 + Roboto | 思源宋 Medium + SemiBold · 思源黑 Regular + Medium · Cormorant Garamond Medium + SemiBold · Inter Regular + SemiBold + Medium · 拉丁报告里的汉字走思源黑 | -| 渲染时间 / 峰值内存 | 基线 | 同一量级(±10%,属噪声) | -| 文件体积 | 基线 | 约 +3% | - -每个字面的字形数是收据:中文报告里 15,988 个字形落在正文黑体、2,247 个落在它的 medium 档、634 个落在标题宋体的 medium、222 个落在它的 semibold。这就是「字体在服务器上」与「字体在文件里」的区别——另一个国家的读者看到的是同样四个字面,而代价是某一种语言正好多一页(拉丁字体的度量比它替换掉的那个更宽)。 - -## 规则二:只出静态实例,理由写在解析器里 - -现代做法看起来该是「每个字族发一个可变字体」,但在这里是错的,有两个实测理由: - -1. **PDF 引擎的字体解析器只读 `glyf`/`loca`。** 喂它一个 PostScript/CFF 轮廓的 OpenType 文件,它直接拒绝:`Fonts with postscript outlines are not supported`。问题不是「可变」,是 CFF 轮廓。 -2. **它完全不读 `fvar` 表**,所以可变字体会**静默地**解析成它的**默认实例**——而中文字族的默认实例恰恰是最细的那一档(`name` 表里写得清清楚楚:Thin 100 与 ExtraLight 200)。整份报告会变成发丝体,而整条管线里不会报任何一个错。 - -所以构建步骤拿上游的可变字体**实例化成静态切面**:跑实例化器,然后把 `name`、`OS/2.usWeightClass` 和 `head.macStyle` 改写成它现在真实的字重(源文件的名字里还写着 Thin)。产物旁边有一份 manifest,逐文件记下 family、字重、sha256、字形数与码位数;许可证也随文件一起进仓库。 - -## 规则三:引擎只会说两种字重,那就映射到两个槽位 - -渲染引擎的 CSS 只认 `normal` 和 `bold`——它的源码自己就这么写("Does not support: bolder, lighter, 100..900")。所以数字字重对它毫无意义;而同一条管线里的无头 Chromium 又完全认它们。 - -可用的映射是:每个字族声明一个 **R 槽**和一个 **B 槽**(常规一档、更重一档),`font-weight: 500` 落到目标字族的 R 槽,`600` **只在 PDF 那一侧被改写成 `bold`**,落到 B 槽。Chromium 侧照写 `600`。两条测试钉住它:B 槽必须比 R 槽重(读 `usWeightClass`,不是假设),且 `font-weight: 600` 绝不允许出现在交给 PDF 引擎的样式表里。 - -有一处细节值得单写一段:在这个管线里,**medium 不是强调手段,而是一个字族成员**。中文标题用宋体的 medium 当常规、用 semibold 做强调;英文标题用展示衬线体做同样的事。这里搞错,屏幕上完全看不出来,而印出来一目了然。 - -## 规则四:不要按你控制不了输入的字集做子集化 - -第一版把中文字体按两个国家字集的并集做了子集,产物从约 50 MB 降到约 29 MB。然后我跑了姓名探针,发现这些字被裁掉了: - -```text -犇 骉 頔 祎 廸 玧 祾 赟 堃 -``` - -每一个都是真实的人名用字。报告会把客户姓名印在封面上,而姓名是自由输入——所以一套基于子集的构建,迟早会在产品最醒目的那一页上放一个豆腐块。全字形覆盖是约 31k 码位、约 50 MB 的嵌入字体;为了永远不给客户看一个空方块,这个价钱很便宜。 - -## 规则五:把「缺字」变成构建失败,而不是惊喜 - -闸门跑三项检查,任何一项不过就非零退出: - -- manifest 里声明的每个字体文件都在,且 sha256 与磁盘上的文件一致; -- 内容库(三语全部)+ 品牌字面 + 各类探针(简体、繁体、CJK 标点、全角、中英混排、数字、拉丁扩展、符号)+ 姓名用字,**每一个字在「它实际会被哪一档字体印出来」的那一档里都有字形**; -- 查字形用的是 PDF 引擎自己的解析器,而不是第三方库,所以这条检查不可能和渲染器互相矛盾。 - -已知边界写在明处,不藏起来:✅ ✔ 这类符号在这几款正文字体里都没有字形,而引擎也画不了彩色 emoji,所以它们不出现在报告正文;拉丁标题那款展示衬线体只有 974 个字形,不画 ① ★ ℃,这也是为什么内容库的检查覆盖**所有**槽位而不只是正文;而拉丁报告里的汉字必须走专门的 CJK 槽位——两条测试防止两个渲染器在这点上产生分歧。 - -## 如果重来一次 - -1. **在动样式之前先定字族与字重。** 版式做完之后再去加第二档字重,意味着每一页页数都要手工复核一遍。 -2. **选字体格式之前先读渲染器的源码。** 可变字体看着是审美问题,实际上是解析器问题——两个文件、十分钟就定了。 -3. **把要嵌进文档的一切当作许可证产物。** 字体文件连同许可证一起进仓库,否则就得进别人的法务审查。 -4. **量「到底嵌进去了什么」。** 页数、每个字面的字形数、文件体积、渲染时间:四项都动了,其中三项的方向和我猜的相反。 - -## 结果 - -改前:一款中文字体加一款界面字体、靠合成粗体、还取决于服务器装过什么。改后:四个字族、八个静态字面真正嵌在文件里,字重是各个字族真实的成员,页数与上一版相差不超过一页,而且有一条 lint 闸门:某个字没有字形时直接让构建失败——于是失败形态从「客户看到一个方块」变成了「CI 拒绝发版」。这跟这条管线里其它地方的取舍是同一个:**让错误发不出去,而不是让错误容易被看见。** -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Embedding Fonts So a PDF Looks the Same on Every Machine" -description: "Four families, eight faces, two renderers and one CSS quirk: what it took to stop a generated report depending on whatever fonts the server happened to have." -pubDate: 2025-01-22 -updatedDate: 2026-10-16 -category: engineering -tags: ["pdf", "fonts", "mpdf", "typography", "php"] -ogImage: /og/embedding-fonts-so-a-pdf-looks-the-same-everywhere.jpg -banner: /banners/embedding-fonts-so-a-pdf-looks-the-same-everywhere.jpg -draft: false ---- - -## The requirement was one sentence - -A report generator has to produce a document that looks the same everywhere: -Chinese headings in a serif, Chinese body text in a sans, Latin headings and -cover numerals in a display serif, Latin body text and tables in Inter. -Weights matter — medium for emphasis, semibold for headings — across three -languages and two PDF renderers. - -And the fonts must travel with the document. Not "the server has them -installed", not "the reader probably has them" — **embedded**, so a print -shop in another country sees the same pages. - -The old pipeline failed on the last clause in a way that was invisible: it -depended on fonts installed on the server by a package manager, and it only -ever embedded a single CJK face. The second weight did not exist, so "bold" -was the renderer's synthetic approximation, and the only way to notice was -to open the PDF and look at what was actually inside it. - -## Rule one: read the fonts back out of the finished PDF - -"Embedded" is a claim about the output, so I measured the output. The same -document, rendered before and after: - -| | before | after | -|---|---|---| -| zh page count | 23 | **23** | -| en page count | 28 | **29** | -| bm page count | 30 | **30** | -| fonts actually embedded | one CJK face + Roboto | Source Han Serif Medium + SemiBold · Source Han Sans Regular + Medium · Cormorant Garamond Medium + SemiBold · Inter Regular + SemiBold + Medium · Source Han Sans for the CJK runs inside Latin reports | -| render time / peak memory | baseline | same order (±10%, noise) | -| file size | baseline | roughly +3% | - -The glyph counts per face are the receipt: in the Chinese report, 15,988 -glyphs land in the body sans, 2,247 in its medium weight, 634 in the heading -serif medium, 222 in its semibold. That is the difference between "the font -is on the server" and "the font is in the file" — a reader in another -country sees the same four faces, and a page count that moved by exactly one -page in one language (the Latin metrics are wider than what they replaced) -is the honest cost. - -## Rule two: static instances only, and the reason is in the parser - -The obvious modern choice — ship one variable font per family — is wrong -here, for two measured reasons: - -1. **The PDF engine's font parser reads `glyf`/`loca` only.** Feed it an - OpenType file with PostScript/CFF outlines and it refuses: - `Fonts with postscript outlines are not supported`. A variable font is - not the problem; CFF outlines are. -2. **It does not read the `fvar` table at all**, so a variable font silently - resolves to its *default instance* — and the default instances of the CJK - families are the thinnest ones (Thin 100 and ExtraLight 200, visible in - the `name` table). Every page would render in a hairline weight, with no - error anywhere in the pipeline. - -So the build step takes the upstream variable fonts and **instantiates -static cuts**: it runs the instancer, then rewrites `name`, -`OS/2.usWeightClass` and `head.macStyle` so the file identifies itself as -the weight it now is (the source's own name still says "Thin"). A manifest -records family, weight class, sha256, glyph count and code-point count for -every file, and the licence files ship next to them. - -## Rule three: the engine speaks two weights, so map onto two slots - -The renderer's CSS handling honours exactly `normal` and `bold` — its source -says so in as many words ("Does not support: bolder, lighter, 100..900"). -Numeric weights are therefore meaningless to it, while the headless-Chromium -renderer in the same pipeline honours them properly. - -The mapping that works: each family declares an **R slot** and a **B slot** -(regular-ish and heavier), `font-weight: 500` lands on the R slot of the -intended family, and `600` is **rewritten to `bold`** on the PDF side only, -which lands on the B slot. Chromium keeps `600` as written. Two tests pin -it: the B slot must be heavier than the R slot (read from `usWeightClass`, -not assumed), and `font-weight: 600` must never appear in the stylesheet -handed to the PDF engine. - -One subtlety worth the paragraph: a "medium" weight is not an emphasis -mechanism in this pipeline, it is a *family member*. Chinese headings use -the serif's medium as their regular and its semibold for emphasis; Latin -headings do the same with the display serif. Getting that wrong is invisible -on a screen and obvious in print. - -## Rule four: do not subset against a character set you do not control - -The first attempt subset the CJK fonts to the union of two national -character sets and produced a ~29 MB bundle instead of ~50 MB. Then I ran -the name probe and found these were cut: - -```text -犇 骉 頔 祎 廸 玧 祾 赟 堃 -``` - -Every one of them is a real given name character. The report prints a -customer's name on the cover, and the name is free input — so a subset-based -build will eventually put a tofu box on the most prominent page of the -product. Full coverage is ~31k code points and roughly 50 MB of embedded -fonts; that is a fine price for never showing a customer an empty rectangle. - -## Rule five: make missing glyphs a build failure, not a surprise - -The gate runs three checks and exits non-zero on any of them: - -- every font file declared in the manifest is present, and its sha256 - matches the file on disk; -- every character in the content library (all three languages) plus the - brand strings, plus probe classes — simplified, traditional, CJK - punctuation, full-width forms, mixed-script, digits, Latin extended, - symbols — plus the name characters, has a glyph **in the face that will - actually print it**; -- the lookup uses the PDF engine's own font parser rather than a third-party - library, so the check cannot disagree with the renderer. - -The known edges are written down instead of hidden. Dingbats and emoji (✅ ✔) -exist in none of these text fonts and the engine cannot draw colour emoji, -so they are kept out of the report body. The display serif has only 974 -glyphs and does not draw symbols like ① ★ ℃, which is why the -content-library check covers *all* slots rather than just the body. And CJK -characters inside a Latin report must go through the dedicated CJK slot — -two tests keep the renderers from disagreeing about that. - -## What I would do differently - -1. **Choose faces and weights before styling anything.** Retrofitting a - second weight after the layout exists means re-checking every page count - by hand. -2. **Read the renderer's source before choosing a font format.** The - variable-font decision looked like a taste question and was actually a - parser question — two files settled it in ten minutes. -3. **Treat everything you ship inside a document as a licence artifact.** - Font files enter the repo with their licences, or they enter someone - else's legal review. -4. **Measure what got embedded.** Page count, glyph-per-face counts, file - size, render time: all four moved, and three of them moved in ways I - would have guessed wrong. - -## The result - -Before: one CJK face and a UI font, with synthetic bold, depending on what -the server had installed. After: four families and eight static faces -embedded in the file, weights that are real members of their families, page -counts within one page of the previous layout, and a lint gate that fails -the build when a character has no glyph — so the failure mode moved from "a -customer sees a box" to "CI refuses the release". Which is the same trade as -everywhere else in this pipeline: make the mistake impossible to ship rather -than easy to notice. -```` - ---- - -## 文章 07 / 11:那个把静默丢失更新变成 409 的 JPA 字段:@Version 乐观锁 -- **Slug**: `jpa-version-field-lost-update` -- **英文标题**: The JPA Field That Turns a Silent Lost Update Into a 409 -- **分类**: `engineering` | **发布日期**: 2026-08-19 -- **线上地址**: https://blog.hoelee.com/posts/jpa-version-field-lost-update/ (EN) | https://blog.hoelee.com/posts/zh/jpa-version-field-lost-update/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】:[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】: -【人设与用词卫生】: -【CRO与转化建议】: -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "那个把静默丢失更新变成 409 的 JPA 字段:@Version 乐观锁" -description: "在 REST + JPA 的读-改-写流程里,两个编辑同时保存同一条记录时,后保存者会静默覆盖先保存者的修改,而 API 两次都返回成功——JPA 的 @Version 乐观锁把这个静默丢失更新变成 HTTP 409 冲突:请求携带期望版本号,版本不符就拒绝写入,并提示客户端重新读取再试。" -pubDate: 2026-08-19 -category: engineering -tags: [java, spring, jpa, hibernate, rest, testing] -ogImage: /og/zh-jpa-version-field-lost-update.jpg -banner: /banners/jpa-version-field-lost-update.jpg -draft: false ---- - -两个人同时打开同一篇文章,两个人都改了,两个人都保存。其中一次 -保存悄无声息地消失了——没有报错,没有警告,而且 API 对两次请求 -都返回了 HTTP 200。 - -这就是丢失更新(lost update)问题,是读-改-写(read-modify-write) -API 里最安静的数据丢失 bug。我在做 Spring Boot 作品集 demo 时撞上 -了它——这是一个 Spring Boot 3.5.16(Java 21)的 REST API,编辑器 -从浏览器页面创建和更新文章。修法最终落在三处:JPA 实体上的一个 -`@Version` 字段、请求契约里的一个字段、以及 API 层把它映射成 -HTTP 409 Conflict 的一个异常。这篇文章就沿着真实代码走一遍这条路。 - -## 问题:一条编辑消失了,却谁都怪不上 - -为什么两个人保存同一条记录时,其中一个人的修改会消失? - -跟着时间线走。编辑器 A 和编辑器 B 都拉取了同一篇文章。A 先保存: -行被更新,API 返回 200。B 稍后保存:行被*再次*更新,这次覆盖了 -A 的文字,API 又返回 200。每个请求都成功;每个响应都告诉它的调用 -方「你的保存就是这条记录的当前状态」——对其中一个人来说这是假话。 -A 的编辑就这么没了,两个客户端都没有任何办法知道。 - -失败是静默的,因为它就是读-改-写流程的正常行为,不是异常。这正是 -它在真实系统里存活的理由,也是为什么在要展示工程判断力的东西里, -值得把它修得干净。 - -## 我先试的方案:不带版本号的读-改-写,以及它为什么失败 - -我对更新流程的第一版草图是最直白的那种:`GET` 记录、编辑、 -`PUT` 回去,只带新的标题和正文,什么都不带。 - -```text -GET /api/posts/{id} # 读取记录 -PUT /api/posts/{id} # 写回新的标题和正文 -``` - -服务端加载行、套用更新、返回 200。它无从知道在这一行数据上,客户端 -在 `GET` 和 `PUT` 之间已经被别人动过——读取和写入是两个互不相干的 -请求,契约里没有任何东西把它们连起来。最后写入的人赢,而两个写入 -者都被告知成功。 - -在这里,200 比报错更糟。报错至少告诉输掉的那个客户端「出事了」, -给他们一个去查的理由。200 则告诉两个客户端「你们的保存就是记录 -当前的状态」——对其中一个是撒谎。输掉的那次编辑现在哪儿都不存在: -屏幕上没有,数据库里也没有;而因为客户端相信自己保存成功了, -没有人会去找这份数据。 - -## 修法:一个 @Version 字段、一份契约、一个冲突 - -修法一共五小步,全部都在 demo 的源码里。 - -### 1. 实体随身携带一个版本列 - -```java -@Entity -@Table(name = "posts", indexes = @Index(name = "idx_posts_title", columnList = "title")) -@EntityListeners(AuditingEntityListener.class) -public class Post { - - @Id - @GeneratedValue - @UuidGenerator - private UUID id; - - @Column(nullable = false, length = 160) - private String title; - - @Column(nullable = false, length = 10_000) - private String body; - - @Version - private long version; - // ... -} -``` - -`@Version` 告诉 Hibernate 把 `version` 当作乐观锁。它生成的每一条 -`UPDATE` 都变成有条件的: - -```sql -UPDATE posts -SET title = ?, body = ?, version = version + 1, ... -WHERE id = ? AND version = ? -``` - -如果这行数据的版本号已经和语句构造时的不一致,受影响行数为零, -Hibernate 就会拒绝这次 flush,而不是悄悄覆盖。实体从不用手自增 -计数器——更新方法只碰 `title` 和 `body`;版本号跟着写入一起走。 - -### 2. 版本号进入 API 契约 - -锁如果客户端无法参与,就没有用,所以版本号在两个方向上都穿过 -API。响应的 record 暴露它: - -```java -public record PostResponse(UUID id, long authorId, String title, String body, - long version, Instant createdAt, Instant updatedAt) { -} -``` - -更新请求则要求把它带回来: - -```java -public record UpdatePostRequest( - @NotBlank @Size(max = 160) String title, - @NotBlank @Size(max = 10_000) String body, - @NotNull @Min(0) Long version) { -} -``` - -`UpdatePostRequest` 旁边的学习注记说得很直白: -「the expected version is part of the update -contract, making optimistic locking visible to clients.」 -(期望的版本号是更新契约的一部分,让乐观锁对客户端可见。) - -### 3. 在拥有写事务的 service 里检查版本 - -```java -@CachePut(cacheNames = "posts", key = "#postId") -@Transactional -public PostResponse updatePost(UUID postId, UpdatePostRequest request) { - Post post = postRepository.findById(postId) - .orElseThrow(() -> new PostNotFoundException(postId)); - if (post.getVersion() != request.version()) { - throw new PostVersionConflictException(postId); +## 60 首长歌全集在线鉴赏 + +以下 60 首作品均已上传至 Publitio 媒体存储,支持 HTTP 206 自由拖拽定位。封面全量采用 1024×1024 独立生成的无字 WebP 艺术插画,并内嵌同步 LRC 歌词。 + +*操作提示:可直接利用上方导航面板进行分类筛选,或从下拉菜单中快速直达指定曲目。* + +
+
+ The Thimble Under the Moss cover art +
+
+
+ 01. The Thimble Under the Moss + MP3 + 3分11秒 (191.1s) · 凯尔特竖琴民谣 · 清脆稚气女童声(类似 Cécile Corbel《Arrietty's Song》)|中速轻快(约 105 BPM) +
+ + +
+
+ +
+
+ Mist of Skellig Michael cover art +
+
+
+ 02. Mist of Skellig Michael + MP3 + 4分26秒 (266.2s) · 凯尔特竖琴民谣 · 空灵圣洁的小女孩独唱(类似 Enya + 幼态音色)|慢速悠长(约 72 BPM,字少音长) +
+ + +
+
+ +
+
+ The Rose of Connemara cover art +
+
+
+ 03. The Rose of Connemara + MP3 + 4分43秒 (283.1s) · 凯尔特竖琴民谣 · 纯净稚气少女音(类似 Cara Dillon 早期清亮少女感)|中慢速叙事(约 82 BPM) +
+ + +
+
+ +
+
+ Lanterns on the Moor cover art +
+
+
+ 04. Lanterns on the Moor + MP3 + 3分31秒 (211.2s) · 凯尔特竖琴民谣 · 俏皮精灵般的女童声(类似 Cécile Corbel 欢快舞曲风)|快速三拍子(约 128 BPM) +
+ + +
+
+ +
+
+ The Weaver of Raglan Lane cover art +
+
+
+ 05. The Weaver of Raglan Lane + MP3 + 4分39秒 (279.3s) · 凯尔特竖琴民谣 · 醇厚温暖的凯尔特民谣男声(类似 Luke Kelly,调剂男声)|中慢速(约 80 BPM) +
+ + +
+
+ +
+
+ 硝子の花束 cover art +
+
+
+ 06. 硝子の花束 + MP3 + 3分49秒 (230.0s) · 日系治愈动漫 · 稚气通透、略带脆弱感的动漫少女音(祈妹风)|中慢速钢琴弦乐(约 78 BPM) +
+ + +
+
+ +
+
+ 銀の時計塔と宵の歌 cover art +
+
+
+ 07. 銀の時計塔と宵の歌 + MP3 + 4分34秒 (274.8s) · 日系治愈动漫 · 清亮圣洁少女音 + 梶浦语吟唱|中速华尔兹三拍子(约 110 BPM) +
+ + +
+
+ +
+
+ 森の郵便屋さん cover art +
+
+
+ 08. 森の郵便屋さん + MP3 + 3分44秒 (225.0s) · 日系治愈动漫 · 极度轻柔、如耳语般的稚气女童声|慢速木吉他与竖琴(约 68 BPM) +
+ + +
+
+ +
+
+ 白鳥の祈り cover art +
+
+
+ 09. 白鳥の祈り + MP3 + 4分06秒 (246.3s) · 日系治愈动漫 · 空灵通透的高音少女/女童声,尾音带空灵共鸣|慢速大气(约 70 BPM) +
+ + +
+
+ +
+
+ 星屑のオルゴール cover art +
+
+
+ 10. 星屑のオルゴール + MP3 + 3分31秒 (211.8s) · 日系治愈动漫 · 清甜稚气的少女声|中慢速轻盈感(约 88 BPM) +
+ + +
+
+ +
+
+ ひまわり行進曲 cover art +
+
+
+ 11. ひまわり行進曲 + MP3 + 3分18秒 (198.0s) · 日系元气童趣 · 元气满满的小女孩独唱/齐唱感|中快速欢快节奏(约 118 BPM) +
+ + +
+
+ +
+
+ プリンは世界を救うのだ! cover art +
+
+
+ 12. プリンは世界を救うのだ! + MP3 + 3分23秒 (203.4s) · 日系元气童趣 · 软萌夸张的稚气萝莉音(高音、语速快、字密)|快速电音流行(约 145 BPM) +
+ + +
+
+ +
+
+ こねこ探偵団のテーマ cover art +
+
+
+ 13. こねこ探偵団のテーマ + MP3 + 2分16秒 (136.1s) · 日系元气童趣 · 奶声奶气的女童声(6~8岁童音感)|中速轻快爵士童谣(约 115 BPM) +
+ + +
+
+ +
+
+ 夏祭りのラムネ星 cover art +
+
+
+ 14. 夏祭りのラムネ星 + MP3 + 3分14秒 (195.0s) · 日系元气童趣 · 清亮少年男声 + 稚气少女声(调剂少年音)|中快速夏日流行(约 122 BPM) +
+ + +
+
+ +
+
+ ほうきに乗って宅配便 cover art +
+
+
+ 15. ほうきに乗って宅配便 + MP3 + 3分16秒 (196.1s) · 日系元气童趣 · 清脆阳光的小女孩声音|中速轻快复古流行(约 116 BPM) +
+ + +
+
+ +
+
+ 白露·竹间灯 cover art +
+
+
+ 16. 白露·竹间灯 + MP3 + 3分42秒 (222.6s) · 中文国风古意 · 清甜稚气的少女/女童声|中速轻快电子国风(约 108 BPM) +
+ + +
+
+ +
+
+ 小鲤跃龙门 cover art +
+
+
+ 17. 小鲤跃龙门 + MP3 + 2分09秒 (129.9s) · 中文国风古意 · 活泼软萌的稚气少女音(语速偏快、吐字灵动)|中快速(约 120 BPM) +
+ + +
+
+ +
+
+ 云海归鲲 cover art +
+
+
+ 18. 云海归鲲 + MP3 + 3分56秒 (236.9s) · 中文国风古意 · 空灵仙气的少年假声/美声男声(调剂男声)|慢速悠远(约 68 BPM) +
+ + +
+
+ +
+
+ 纸伞巷 cover art +
+
+
+ 19. 纸伞巷 + MP3 + 3分34秒 (214.6s) · 中文国风古意 · 清亮温婉的少女声(带一点点江南小调韵味)|中慢速(约 80 BPM) +
+ + +
+
+ +
+
+ 山间小道士 cover art +
+
+
+ 20. 山间小道士 + MP3 + 3分07秒 (187.6s) · 中文国风古意 · 纯真稚气的小女孩/少女声音|中速轻快民谣古风(约 105 BPM) +
+ + +
+
+ +
+
+ 橘子汽水的夏天 cover art +
+
+
+ 21. 橘子汽水的夏天 + MP3 + 2分48秒 (169.0s) · 温暖民谣治愈 · 清澈稚气的少女声|中慢速木吉他民谣(约 82 BPM) +
+ + +
+
+ +
+
+ 外婆的小院 cover art +
+
+
+ 22. 外婆的小院 + MP3 + 3分32秒 (212.4s) · 温暖民谣治愈 · 纯真软糯的小女孩声音|慢速温情(约 72 BPM) +
+ + +
+
+ +
+
+ 开往北方的绿皮车 cover art +
+
+
+ 23. 开往北方的绿皮车 + MP3 + 3分36秒 (216.3s) · 温暖民谣治愈 · 低沉温暖的民谣男声(调剂男声)|慢速叙事(约 75 BPM) +
+ + +
+
+ +
+
+ 写给流浪猫的信 cover art +
+
+
+ 24. 写给流浪猫的信 + MP3 + 2分09秒 (129.9s) · 温暖民谣治愈 · 俏皮温柔的稚气少女音|中速轻快尤克里里/木吉他(约 98 BPM) +
+ + +
+
+ +
+
+ 平凡的傍晚 cover art +
+
+
+ 25. 平凡的傍晚 + MP3 + 3分41秒 (221.0s) · 温暖民谣治愈 · 温柔治愈的青年男声(调剂男声)|慢速钢琴与木吉他(约 74 BPM) +
+ + +
+
+ +
+
+ Light of the Eventide cover art +
+
+
+ 26. Light of the Eventide + MP3 + 4分09秒 (249.0s) · 空灵天籁合唱 · 天籁女童高音领唱 + 纯净童声合唱|极慢速庄严(约 66 BPM) +
+ + +
+
+ +
+
+ Bells of the Winter Star cover art +
+
+
+ 27. Bells of the Winter Star + MP3 + 2分47秒 (167.6s) · 空灵天籁合唱 · 清脆女童多声部齐唱與轮唱|中快速灵动节奏(约 125 BPM) +
+ + +
+
+ +
+
+ Paper Planes in the Wind cover art +
+
+
+ 28. Paper Planes in the Wind + MP3 + 3分25秒 (205.5s) · 空灵天籁合唱 · 清亮纯真的女童领唱 + 学院风童声合唱|中速温暖(约 96 BPM) +
+ + +
+
+ +
+
+ Lullaby of the Little Star cover art +
+
+
+ 29. Lullaby of the Little Star + MP3 + 3分47秒 (227.8s) · 空灵天籁合唱 · 稚气纯净的小女孩独唱(突出童真独唱感)|慢速温柔(约 70 BPM) +
+ + +
+
+ +
+
+ Hymn of the Morning Meadow cover art +
+
+
+ 30. Hymn of the Morning Meadow + MP3 + 4分19秒 (259.9s) · 空灵天籁合唱 · 神圣清澈的女童领唱 + 宏大童声和声(帕赫贝尔卡农式递进)|慢速治愈(约 72 BPM) +
+ + +
+
+ +
+
+ 地板下的借物小人 cover art +
+
+
+ 31. 地板下的借物小人 + MP3 + 3分07秒 (187.7s) · 凯尔特童话民谣 (中文) · 清脆稚气女童声|中速轻快凯尔特竖琴与锡哨(约 105 BPM) +
+ + +
+
+ +
+
+ 迷雾孤岛的竖琴 cover art +
+
+
+ 32. 迷雾孤岛的竖琴 + MP3 + 3分24秒 (204.4s) · 凯尔特童话民谣 (中文) · 空灵圣洁的小女孩独唱|慢速悠长(约 70 BPM,字少音长) +
+ + +
+
+ +
+
+ 石楠花开的高地 cover art +
+
+
+ 33. 石楠花开的高地 + MP3 + 3分36秒 (216.9s) · 凯尔特童话民谣 (中文) · 纯净稚气少女音|中慢速木吉他与凯尔特小提琴(约 82 BPM) +
+ + +
+
+ +
+
+ 橡树下的萤火舞会 cover art +
+
+
+ 34. 橡树下的萤火舞会 + MP3 + 3分43秒 (223.1s) · 凯尔特童话民谣 (中文) · 俏皮精灵般的女童声(语速快、节奏感强)|快速三拍子(约 128 BPM) +
+ + +
+
+ +
+
+ 老航海家的绿宝石 cover art +
+
+
+ 35. 老航海家的绿宝石 + MP3 + 3分49秒 (229.8s) · 凯尔特童话民谣 (中文) · 醇厚温暖的凯尔特民谣男声(调剂男声)|中慢速(约 80 BPM) +
+ + +
+
+ +
+
+ 玻璃温室的告别 cover art +
+
+
+ 36. 玻璃温室的告别 + MP3 + 3分37秒 (217.0s) · 日系治愈物语 (中文) · 稚气通透、略带脆弱感的动漫少女音|中慢速钢琴与弦乐(约 78 BPM) +
+ + +
+
+ +
+
+ 银色钟塔与暮光咏叹 cover art +
+
+
+ 37. 银色钟塔与暮光咏叹 + MP3 + 4分01秒 (241.4s) · 日系治愈物语 (中文) · 清亮圣洁少女音 + 造语吟唱|中速古典哥特华尔兹(约 110 BPM) +
+ + +
+
+ +
+
+ 森林邮差的蒲公英 cover art +
+
+
+ 38. 森林邮差的蒲公英 + MP3 + 3分26秒 (206.7s) · 日系治愈物语 (中文) · 极度轻柔、如耳语般的稚气女童声|慢速木吉他与钢片琴(约 68 BPM) +
+ + +
+
+ +
+
+ 白鸟与雪原的祈愿 cover art +
+
+
+ 39. 白鸟与雪原的祈愿 + MP3 + 3分39秒 (220.0s) · 日系治愈物语 (中文) · 空灵通透的高音女童/少女声|慢速大气钢琴与管弦(约 70 BPM) +
+ + +
+
+ +
+
+ 发条星尘八音盒 cover art +
+
+
+ 40. 发条星尘八音盒 + MP3 + 3分18秒 (198.2s) · 日系治愈物语 (中文) · 清甜稚气的少女声|中慢速八音盒与木吉他(约 88 BPM) +
+ + +
+
+ +
+
+ 向日葵小队出发 cover art +
+
+
+ 41. 向日葵小队出发 + MP3 + 2分39秒 (159.4s) · 元气童趣流行 (中文) · 元气满满的小女孩领唱 + 童声合唱感|中快速欢快节奏(约 118 BPM) +
+ + +
+
+ +
+
+ 焦糖布丁保卫战! cover art +
+
+
+ 42. 焦糖布丁保卫战! + MP3 + 3分38秒 (218.9s) · 元气童趣流行 (中文) · 软萌夸张的稚气萝莉音(高音、语速快、字密)|快速电音流行(约 145 BPM) +
+ + +
+
+ +
+
+ 猫咪侦探事务所 cover art +
+
+
+ 43. 猫咪侦探事务所 + MP3 + 2分31秒 (151.4s) · 元气童趣流行 (中文) · 奶声奶气的女童声(6~8岁童音感)|中速轻快木琴与铜管(约 115 BPM) +
+ + +
+
+ +
+
+ 夏夜烟火与波子汽水 cover art +
+
+
+ 44. 夏夜烟火与波子汽水 + MP3 + 3分07秒 (187.8s) · 元气童趣流行 (中文) · 清亮少年男声 + 稚气少女声(调剂少年音)|中快速夏日流行(约 122 BPM) +
+ + +
+
+ +
+
+ 骑上扫帚送快递 cover art +
+
+
+ 45. 骑上扫帚送快递 + MP3 + 3分08秒 (188.5s) · 元气童趣流行 (中文) · 清脆阳光的小女孩声音|中速轻快木吉他与手鼓(约 116 BPM) +
+ + +
+
+ +
+
+ 小满·青梅煮雨 cover art +
+
+
+ 46. 小满·青梅煮雨 + MP3 + 3分34秒 (214.9s) · 国风新篇古韵 (中文) · 清甜稚气的少女/女童声|中速轻快电子国风(约 106 BPM) +
+ + +
+
+ +
+
+ 糖画摊前的小狐仙 cover art +
+
+
+ 47. 糖画摊前的小狐仙 + MP3 + 3分12秒 (192.2s) · 国风新篇古韵 (中文) · 活泼软萌的稚气少女音(字密语速快)|中快速(约 124 BPM) +
+ + +
+
+ +
+
+ 鲛人泪·沧海月 cover art +
+
+
+ 48. 鲛人泪·沧海月 + MP3 + 3分59秒 (239.8s) · 国风新篇古韵 (中文) · 空灵仙气的少年假声/美声男声(调剂男声)|慢速深邃(约 68 BPM) +
+ + +
+
+ +
+
+ 兰亭旧墨 cover art +
+
+
+ 49. 兰亭旧墨 + MP3 + 3分49秒 (229.3s) · 国风新篇古韵 (中文) · 清亮温婉的少女声|中慢速古筝与二胡(约 80 BPM) +
+ + +
+
+ +
+
+ 采药小童下山记 cover art +
+
+
+ 50. 采药小童下山记 + MP3 + 3分24秒 (204.4s) · 国风新篇古韵 (中文) · 纯真稚气的小女孩声音|中速轻快木吉他与竹笛(约 104 BPM) +
+ + +
+
+ +
+
+ 晒太阳的旧毛衣 cover art +
+
+
+ 51. 晒太阳的旧毛衣 + MP3 + 3分04秒 (184.9s) · 温暖民谣新篇 (中文) · 清澈稚气的少女声|中慢速木吉他民谣(约 80 BPM) +
+ + +
+
+ +
+
+ 奶奶的桂花糕 cover art +
+
+
+ 52. 奶奶的桂花糕 + MP3 + 3分42秒 (222.4s) · 温暖民谣新篇 (中文) · 纯真软糯的小女孩声音|慢速温情口琴与吉他(约 72 BPM) +
+ + +
+
+ +
+
+ 北方的末班渡轮 cover art +
+
+
+ 53. 北方的末班渡轮 + MP3 + 3分49秒 (229.6s) · 温暖民谣新篇 (中文) · 低沉温暖的民谣男声(调剂男声)|慢速深沉(约 74 BPM) +
+ + +
+
+ +
+
+ 云朵收集指南 cover art +
+
+
+ 54. 云朵收集指南 + MP3 + 2分56秒 (176.4s) · 温暖民谣新篇 (中文) · 俏皮温柔的稚气少女音|中速轻快(约 98 BPM) +
+ + +
+
+ +
+
+ 街角二十四小时书店 cover art +
+
+
+ 55. 街角二十四小时书店 + MP3 + 4分13秒 (253.4s) · 温暖民谣新篇 (中文) · 温柔治愈的青年男声(调剂男声)|慢速钢琴与木吉他(约 74 BPM) +
+ + +
+
+ +
+
+ 暮光归途 cover art +
+
+
+ 56. 暮光归途 + MP3 + 4分02秒 (242.6s) · 天籁童声圣咏 (中文) · 天籁女童高音领唱 + 纯净童声合唱|极慢速庄严管风琴与弦乐(约 66 BPM) +
+ + +
+
+ +
+
+ 冬夜银铃 cover art +
+
+
+ 57. 冬夜银铃 + MP3 + 2分59秒 (179.0s) · 天籁童声圣咏 (中文) · 清脆女童多声部齐唱与轮唱|中快速灵动三拍子(约 125 BPM) +
+ + +
+
+ +
+
+ 迎风飞翔的纸飞机 cover art +
+
+
+ 58. 迎风飞翔的纸飞机 + MP3 + 3分53秒 (233.2s) · 天籁童声圣咏 (中文) · 清亮纯真的女童领唱 + 学院风童声合唱|中速温暖(约 96 BPM) +
+ + +
+
+ +
+
+ 小星星的晚安曲 cover art +
+
+
+ 59. 小星星的晚安曲 + MP3 + 3分51秒 (232.0s) · 天籁童声圣咏 (中文) · 稚气纯净的小女孩独唱|慢速竖琴与小提琴(约 70 BPM) +
+ + +
+
+ +
+
+ 晨曦草地的赞美诗 cover art +
+
+
+ 60. 晨曦草地的赞美诗 + MP3 + 4分26秒 (266.0s) · 天籁童声圣咏 (中文) · 神圣清澈的女童领唱 + 宏大童声和声(卡农式递进)|慢速治愈(约 72 BPM) +
+ + +
+
+ + -### 4. 异常处理器把它映射成 409 - -异常本身带着客户端能直接照做的信息: - -```java -public class PostVersionConflictException extends RuntimeException { - - public PostVersionConflictException(UUID postId) { - super("Post %s has changed; fetch it again before retrying".formatted(postId)); - } -} -``` - -错误边界把它转成正确的 HTTP 应答: - -```java -@ExceptionHandler(PostVersionConflictException.class) -ProblemDetail handleConflict(PostVersionConflictException exception) { - return problem(HttpStatus.CONFLICT, "POST_VERSION_CONFLICT", exception.getMessage()); -} -``` - -`HttpStatus.CONFLICT` 就是 HTTP 409,响应是一份 problem -document——一个 `type` URI、一个稳定的机器码 -(`POST_VERSION_CONFLICT`)和一段人类可读的说明。demo 在配置里 -开启了框架的 problem-details 支持 -(`spring.mvc.problemdetails.enabled: true`),所以校验错误和所有 -其他错误都共用同一种响应形状。 - -### 5. 整条链路,从头到尾 - -```text -过期的 PUT /api/posts/{id} - -> PostService.updatePost 在事务内部重新读取这一行 - -> post.getVersion() != request.version() - -> PostVersionConflictException: "fetch it again before retrying" - -> ApiExceptionHandler.handleConflict - -> HTTP 409, ProblemDetail with code POST_VERSION_CONFLICT - -> 客户端知道自己的前提过期了 -``` - -一个 `@Version` 字段、请求契约里的一个 `version`、一个异常、一个 -handler 方法。Controller 什么都没改——`updatePost` 看起来还是普通 -的 `PUT`。 - -## 为什么答案是冲突,而不是重试 - -当两个编辑器产出同一份记录的两种不同版本时,服务端无法知道哪一方 -的意图才是被需要的。从各自作者的角度看,两次保存都正当;重放那条 -过期的写入,只是重放同一次数据丢失。 - -诚实的结论就是 HTTP 对 409 的定义:请求与资源的当前状态冲突, -并且客户端被明确告知如何解决——「fetch it again before retrying」 -(重新取一次再试)。输掉的客户端重新读取记录,看到另一位作者的 -修改,由人来决定保留什么。静默覆盖变成了一处看得见的决策点——这 -正是这件事的全部意义。 - -## 这个 demo 的其余部分证明了什么 - -版本字段只是一个小系统里的一层,而其余部分都能用同样的方式核实 -——下面每一行都是仓库里的一个类或配置文件,不是一句口号: - -| 层 | 做了什么 | 在哪 | -|---|---|---| -| 缓存 | 单篇文章读取使用有界 Caffeine 缓存(上限 500,TTL 10 分钟,都来自配置);读取用 `@Cacheable`、更新用 `@CachePut`、删除用 `@CacheEvict`;`GET /api/showcase/cache` 暴露请求数、命中数、未命中数和命中率 | `CacheConfig`、`CacheProperties`、`ShowcaseMetricsController` | -| 安全 | 无状态 HTTP Basic + BCrypt;页面与读 API 公开,所有写操作需要 `EDITOR` 角色;本地默认是专用的非机密 `demo-editor`/`changeit` 账户,可用环境变量覆盖 | `SecurityConfig`、`application.yml` | -| 错误 | 请求 record 在边界处校验;一个 `@RestControllerAdvice` 返回一致的 problem document 和机器码——`POST_NOT_FOUND`(404)、`POST_VERSION_CONFLICT`(409)、带字段错误表的 `VALIDATION_FAILED`(400) | `CreatePostRequest`、`UpdatePostRequest`、`ApiExceptionHandler` | -| 配置 | 本地 profile 使用内存 H2 数据库(PostgreSQL 模式),`ddl-auto: update` 并预置示例文章;`prod` 选择 PostgreSQL,所有凭据来自 `APP_*` 环境变量,`ddl-auto: validate` | `application.yml`、`application-prod.yml` | -| 测试 | 一个 `@SpringBootTest` 套件断言:渲染的 Thymeleaf 页面、未认证 401 对编辑器 201、`VALIDATION_FAILED` 问题响应、公开搜索,以及重复读取后的缓存命中 | `DemoApplicationIntegrationTest` | - -## 如果做真东西,我会怎么改 - -一旦这不再是本地 demo,有两件事立刻要变。项目 README 里两件都写了, -值得作为我自己的判断再重复一遍。 - -第一,schema 管理。我会在任何时候把 `ddl-auto` 设为 `validate` 之前, -先加 Flyway 或 Liquibase 迁移。demo 默认 profile 用的是 `update`, -在临时数据库上很方便,养成习惯就很危险——生产 schema 应该是 -有版本管理的代码,而不是启动时的副作用。 - -第二,密钥。本地编辑器账户是刻意的非机密 demo 账户,带默认凭据。 -做真东西的话,我会把凭据放进托管的密钥存储(managed secret store), -拿不到凭据就拒绝启动,而不是回退到默认值。`prod` profile 已经做到 -所有凭据来自环境变量、不提交任何密钥——这正是我想要的形态, -只是不要那些回退值。 - -## 结果 - -过期的更新现在返回 HTTP 409,机器码 `POST_VERSION_CONFLICT`,而不是 -一个静静丢掉另一位作者修改的 200。丢失不再可能悄悄发生:失败是 -响亮的、可操作的——「fetch it again before retrying」——并且集成 -测试套件端到端地断言了访问与校验行为。 - -整个修法就是一个字段上的一个注解,加上让版本号成为契约一部分的 -管道代码。如果一条记录可能同时被两个人读取,这个字段就是「编辑 -丢失」与「双方都看得见的冲突」之间的分界线。 - -我平时就做 Java/Spring 后端和全栈 Web 应用——从浏览器页面到数据库 -的 REST API:JPA、乐观锁、缓存、校验和一整套集成测试。如果你的表单 -或字段里也有「编辑悄悄消失」的问题,跟我说说: -[WhatsApp](https://wa.me/60127972969) · -[me@hoelee.com](mailto:me@hoelee.com?subject=Spring%20Boot%20help) · -[hoelee.com](https://hoelee.com)。 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "The JPA Field That Turns a Silent Lost Update Into a 409" -description: "How a JPA @Version field turns a silent lost update into a 409 conflict — optimistic locking makes a stale REST write fail loudly instead of overwriting." -pubDate: 2026-08-19 -category: engineering -tags: [java, spring, jpa, hibernate, rest, testing] -ogImage: /og/jpa-version-field-lost-update.jpg -banner: /banners/jpa-version-field-lost-update.jpg -draft: false --- -Two people open the same post. Both edit it. Both save. One of the saves -disappears without a trace — no error, no warning, and the API answered -HTTP 200 for both of them. +## 需要为你的业务构建自动化 AI 媒体流水线? -That is the lost update problem, and it is the quietest data-loss bug in -read-modify-write APIs. I hit it while building my Spring Boot portfolio -demo — a Spring Boot 3.5.16 (Java 21) REST API where editors create and -update posts from a browser page. The fix turned out to be one `@Version` -field on the JPA entity, one field on the request contract, and one -exception that the API layer maps to HTTP 409 Conflict. This post traces -that path through the actual code. +我为马来西亚及海外中小企业提供定制化自托管 AI 系统部署、自动化音视频内容生产管线以及全栈网站开发服务。 -## The problem: an edit that vanishes with no one to blame - -Why does one editor's change disappear when two people save the same -record? - -Follow the timeline. Editor A and editor B both fetch the same post. -A saves first: the row is updated and the API answers 200. B saves a -moment later: the row is updated *again*, this time over A's text, and -the API answers 200 again. Every request succeeded; every response told -its caller that their save is the state of the record. A's edit is -simply gone, and neither client has any way to know. - -The failure is silent because it is normal read-modify-write behaviour, -not an anomaly. That is exactly why the bug survives in real systems, and -why it is worth fixing properly in something meant to demonstrate -engineering judgement. - -## What I tried first, and why it failed - -My first sketch of the update flow was the obvious one: `GET` the record, -edit it, `PUT` it back with the new title and body and nothing else. - -```text -GET /api/posts/{id} # read the record -PUT /api/posts/{id} # write it back with a new title and body -``` - -The server loads the row, applies the update, and returns 200. It has no -way to notice that the row moved underneath the client between the `GET` -and the `PUT` — the read and the write are two unrelated requests, and -nothing in the contract connects them. The last write wins, and both -writers are told they succeeded. - -A 200 is worse than an error here. An error at least tells the losing -client that something happened and gives them a reason to look. A 200 -tells both clients that their save is the current state of the record, -which is a lie for one of them. The losing edit now exists nowhere — not -on screen, not in the database — and because the client believes it -saved, nobody goes looking for the data. - -## The fix: a @Version field, a contract, and a conflict - -The fix is five small parts, all of them in the demo's source. - -### 1. The entity carries a version column - -```java -@Entity -@Table(name = "posts", indexes = @Index(name = "idx_posts_title", columnList = "title")) -@EntityListeners(AuditingEntityListener.class) -public class Post { - - @Id - @GeneratedValue - @UuidGenerator - private UUID id; - - @Column(nullable = false, length = 160) - private String title; - - @Column(nullable = false, length = 10_000) - private String body; - - @Version - private long version; - // ... -} -``` - -`@Version` tells Hibernate to treat `version` as an optimistic lock. Every -`UPDATE` it generates becomes conditional: - -```sql -UPDATE posts -SET title = ?, body = ?, version = version + 1, ... -WHERE id = ? AND version = ? -``` - -If the row's version no longer matches the value the statement was built -with, zero rows change and Hibernate rejects the flush instead of quietly -overwriting. The entity never bumps the counter by hand — the update -method only touches `title` and `body`; the version moves with the write -itself. - -### 2. The version is part of the API contract - -A lock is useless if the client cannot take part in it, so the version -travels through the API in both directions. The response record exposes -it: - -```java -public record PostResponse(UUID id, long authorId, String title, String body, - long version, Instant createdAt, Instant updatedAt) { -} -``` - -and the update request requires it back: - -```java -public record UpdatePostRequest( - @NotBlank @Size(max = 160) String title, - @NotBlank @Size(max = 10_000) String body, - @NotNull @Min(0) Long version) { -} -``` - -The learning note next to `UpdatePostRequest` says it plainly: "the -expected version is part of the update contract, making optimistic -locking visible to clients." - -### 3. The service checks it inside the transaction that owns the write - -```java -@CachePut(cacheNames = "posts", key = "#postId") -@Transactional -public PostResponse updatePost(UUID postId, UpdatePostRequest request) { - Post post = postRepository.findById(postId) - .orElseThrow(() -> new PostNotFoundException(postId)); - if (post.getVersion() != request.version()) { - throw new PostVersionConflictException(postId); - } - post.update(request.title().trim(), request.body().trim()); - return PostResponse.from(postRepository.saveAndFlush(post)); -} -``` - -The load, the comparison and the save all happen inside one -`@Transactional` method, and that is the part that makes the check -honest: the version is compared against the row as it stands *at write -time*, not against a snapshot from an earlier request. If the read and -the write lived in different transactions, the check would compare -against a version the row has already left behind, and the whole scheme -would be theatre. The learning note on `PostService` names the rule: -"service methods centralize transaction boundaries, cache coherence, and -optimistic-locking rules." The demo also runs with -`spring.jpa.open-in-view: false`, so no lingering session serves a stale -entity across requests — the row is re-read inside the writing -transaction. - -The explicit check catches stale requests deterministically. The -`@Version` column still matters underneath: if another commit slips in -between the check and the flush, the conditional `UPDATE` affects zero -rows and Hibernate refuses the write anyway. - -### 4. The handler maps the exception to a 409 - -The exception itself carries the message a client can act on: - -```java -public class PostVersionConflictException extends RuntimeException { - - public PostVersionConflictException(UUID postId) { - super("Post %s has changed; fetch it again before retrying".formatted(postId)); - } -} -``` - -and the error boundary turns it into the right HTTP answer: - -```java -@ExceptionHandler(PostVersionConflictException.class) -ProblemDetail handleConflict(PostVersionConflictException exception) { - return problem(HttpStatus.CONFLICT, "POST_VERSION_CONFLICT", exception.getMessage()); -} -``` - -`HttpStatus.CONFLICT` is HTTP 409, and the response is a problem document -— a `type` URI, a stable machine code (`POST_VERSION_CONFLICT`) and a -human-readable detail. The demo enables the framework's problem-details -support in its configuration (`spring.mvc.problemdetails.enabled: true`), -so validation and every other error ride the same shape. - -### 5. The whole path, end to end - -```text -stale PUT /api/posts/{id} - -> PostService.updatePost re-reads the row inside its transaction - -> post.getVersion() != request.version() - -> PostVersionConflictException: "fetch it again before retrying" - -> ApiExceptionHandler.handleConflict - -> HTTP 409, ProblemDetail with code POST_VERSION_CONFLICT - -> the client knows its premise was stale -``` - -One `@Version` field, one `version` in the request contract, one -exception, one handler method. Nothing on the controller changed — -`updatePost` still looks like an ordinary `PUT`. - -## Why a conflict is the correct answer, not a retry - -When two editors produce two different versions of the same record, the -server cannot know which intent is the one meant. Both saves are -legitimate from their authors' point of view, and replaying the stale -write just replays the same data loss. - -The honest verdict is HTTP's own definition of 409: the request conflicts -with the current state of the resource, and the client is told exactly -how to resolve it — "fetch it again before retrying". The losing client -re-reads the record, sees the other writer's change, and a human decides -what to keep. The silent overwrite becomes a visible decision point, -which is the entire purpose of the exercise. - -## What the rest of the demo proves - -The version field is one layer of a small system, and the rest of it is -verifiable the same way — every row below is a class or a config file in -the repository, not a claim: - -| Layer | What it does | Where it lives | -|---|---|---| -| Caching | Bounded Caffeine cache on single-post reads (max 500, TTL 10 min, both from config); `@Cacheable` on read, `@CachePut` on update, `@CacheEvict` on delete; `GET /api/showcase/cache` exposes requests, hits, misses and hit rate | `CacheConfig`, `CacheProperties`, `ShowcaseMetricsController` | -| Security | Stateless HTTP Basic with BCrypt; pages and read APIs are public, every mutation needs the `EDITOR` role; local defaults are a throwaway `demo-editor`/`changeit` account overridable via env vars | `SecurityConfig`, `application.yml` | -| Errors | Request records validate at the boundary; one `@RestControllerAdvice` returns consistent problem documents with machine codes — `POST_NOT_FOUND` (404), `POST_VERSION_CONFLICT` (409), `VALIDATION_FAILED` (400) with a field-error map | `CreatePostRequest`, `UpdatePostRequest`, `ApiExceptionHandler` | -| Configuration | Local profile runs an in-memory H2 database in PostgreSQL mode with `ddl-auto: update` and seeded sample posts; `prod` selects PostgreSQL, takes every credential from `APP_*` env vars and sets `ddl-auto: validate` | `application.yml`, `application-prod.yml` | -| Tests | One `@SpringBootTest` suite asserts the rendered Thymeleaf pages, unauthenticated 401s versus editor 201s, the `VALIDATION_FAILED` problem response, public search, and cache hits after repeated reads | `DemoApplicationIntegrationTest` | - -## What I would do differently for anything real - -Two things change the moment this stops being a local demo. The project -README says both, and they are worth repeating as my own judgement. - -First, schema management. I would add Flyway or Liquibase migrations -before ever setting `ddl-auto` to `validate`. The demo's default profile -uses `update`, which is convenient on a scratch database and dangerous as -a habit — production schema should be versioned code, not a side effect -of startup. - -Second, secrets. The local editor account is intentionally a non-secret -demo account with default credentials. For anything real, I would keep -credentials in a managed secret store and refuse to boot without them, -instead of falling back to a default. The `prod` profile already takes -every credential from environment variables with no committed secrets — -that is the shape I would keep, minus the fallbacks. - -## The result - -A stale update now returns HTTP 409 with code `POST_VERSION_CONFLICT` -instead of a 200 that silently discards the other writer's change. The -loss can no longer happen quietly: the failure is loud, actionable — -"fetch it again before retrying" — and the integration test suite asserts -the access and validation behaviour end to end. - -The whole fix is one annotation on one field, plus plumbing that makes -the version part of the contract. If a record can be read by two people -at the same time, that field is the difference between a lost edit and a -conflict both clients can see. - -I build Java/Spring back ends and full-stack web applications — REST APIs -with JPA, optimistic locking, caching, validation and integration tests, -from the browser page down to the database. If you have a form or a field -where edits keep quietly disappearing, tell me about it: -[WhatsApp](https://wa.me/60127972969) · -[me@hoelee.com](mailto:me@hoelee.com?subject=Spring%20Boot%20help) · -[hoelee.com](https://hoelee.com). -```` - ---- - -## 文章 08 / 11:把 CodeIgniter 4 应用从 IIS 迁移到 OpenLiteSpeed -- **Slug**: `migrating-codeigniter-iis-to-openlitespeed` -- **英文标题**: Migrating a CodeIgniter 4 App From IIS to OpenLiteSpeed -- **分类**: `engineering` | **发布日期**: 2026-09-27 -- **线上地址**: https://blog.hoelee.com/posts/migrating-codeigniter-iis-to-openlitespeed/ (EN) | https://blog.hoelee.com/posts/zh/migrating-codeigniter-iis-to-openlitespeed/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】:[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】: -【人设与用词卫生】: -【CRO与转化建议】: -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "把 CodeIgniter 4 应用从 IIS 迁移到 OpenLiteSpeed" -description: "我把一个 17,000 行的 CodeIgniter 4 应用从 Windows IIS 搬到了 CyberPanel 上的 OpenLiteSpeed。迁移后冒出两个致命错误——而 IIS 一直靠 SSO 把它们藏着。" -pubDate: 2026-09-27 -category: engineering -blueprint: sme-digital-suite -blueprintLayer: L1-high-conversion-web -tags: [codeigniter, php, openspeedway, cyberpanel, iis, migration, litespeed, linux] -ogImage: /og/zh-migrating-codeigniter-iis-to-openlitespeed.jpg -banner: /banners/migrating-codeigniter-iis-to-openlitespeed.jpg -draft: false ---- - -我维护着一个数字命理报告生成器:一个 CodeIgniter 4 应用,接收出生日期和姓名,跑一遍命理引擎,渲染出 19 页 A4 报告。它在 Windows Server + IIS 上跑了很多年。2026 年 9 月,我把它迁到 CyberPanel 上的 OpenLiteSpeed,跑在 Linux 上,换了个域名。 - -迁移本身没什么好说的——拷文件、把 docroot 指向 `public/`、装依赖。真正值得写下来的是:**新主机上第一个未认证请求,就在两条路由上直接致命崩溃——而这两条路由在 IIS 上"一直好好的"。** - -两个 bug 都是真的。两个从一开始就存在于 IIS 上。两个都看不见,因为 IIS 在出问题的路由前面恰好挡了一层 authentik SSO。 - -## 为什么这件事重要 - -如果你的自托管应用前面挂了一层认证网关,那这层网关做的事不止是保护数据——它同时**在帮你藏 bug**。任何"只对未认证用户出错"的东西永远不会以未认证身份运行,所以永远不会报错。而它会在最糟的时刻暴露:你迁移的时候、你拆掉网关的时候、你把路由暴露到公网的时候。 - -我这个应用,就"健康"了整整几年。我把新域名指过去的那一刻,两条路由返 HTTP 500。代码一行没改。唯一变的是网关没了。 - -两个修复加起来大概二十分钟。找到它们花了两个小时——因为真正的错误信息被 CodeIgniter 的 CLI 错误渲染器挡在了后面。 - -## 环境 - -这是一个相当普通的 CI4 项目,但有一个不太常见的架构细节:报告引擎不是库,而是**同一台主机上的 HTTP 端点**。控制器拼一个请求,`curl` 回自己域名的 `/api/single`,引擎返回算好的 JSON 数值,控制器再把它渲染成可打印的报告。 - -正是这个"自调用"设计,让 IIS 上必须有 `CONST_IIS_INTERNAL_BASE`。这点我后面会回来说——它才是真正有意思的部分。 - -| | 迁移前 | 迁移后 | -|---|---|---| -| 操作系统 | Windows Server | Ubuntu (CyberPanel) | -| Web 服务器 | IIS | OpenLiteSpeed 1.9.0 | -| PHP | 8.4 (Windows 版) | 8.4.25 (lsphp84) | -| Docroot | `C:\inetpub\calc.hoelee.com` | `/home/<域名>/public_html/public` | -| 认证网关 | authentik SSO 挡 `/lifecode`、`/api/*` | 无 | - -## 第一步:docroot 指向 `public/`,不是项目根目录 - -CI4 是两段式目录结构。`app/`、`vendor/`、`writable/` 和 `.env` 都在项目根目录,只有 `public/` 本该对 Web 可见。 - -CyberPanel 建的站点根目录叫 `public_html`。我的第一反应是把整个项目解压进 `public_html`,docroot 不动。 - -**那样会把 `app/`、`vendor/`、`.env` 文件和 `writable/` 的 session 数据全部暴露在 HTTP 之下。** 这个项目的 `.env` 里有一个 webhook 凭证。任何人猜到 `/../.env`——或者干脆直接请求 `/.env`,因为这文件就躺在被服务的目录下面——就拿到了。 - -正确做法是解压进 `public_html`,然后把 vhost 的 docroot 往下一层,改成 `public_html/public`: - -```bash -# 解压成 app/、vendor/、.env 位于 public_html 之下 -cd /home/<域名>/public_html -tar -xzf /tmp/deploy.tar.gz - -# docroot 必须是 public_html/public,不是 public_html -sudo sed -i 's#/home/<域名>/public_html\$#/home/<域名>/public_html/public#' \ - /usr/local/lsws/conf/vhosts/<域名>/vhost.conf - -sudo /usr/local/lsws/bin/lshttpd -t -sudo systemctl restart lsws -``` - -这是整个迁移里最重要的一步,也是大多数教程跳过的一步。其他什么都不做,这一步也得做。 - -## 第二步:IIS 藏起来的两个致命错误 - -### 致命错误一 —— `env()` 调用得太早 - -第一个 500 完全没有响应体。这不太寻常——CI4 正常会渲染点东西出来。一个空白的 500 通常意味着 PHP 在框架的错误处理器装上之前就死了。 - -CI4 的启动流程会在 `Boot::bootWeb()` 里很早加载 `app/Config/Constants.php`(通过 `Boot::loadConstants()`)。而这件事发生在 **`Common.php` 助手文件加载之前**——`env()` 就定义在那个文件里。 - -所以这一行: - -```php -// app/Config/Constants.php —— 错误写法 -define('CONST_fullBase', env('hoelee.fullBase')); -``` - -会失败于: - -``` -Fatal error: Uncaught Error: Call to undefined function env() -in app/Config/Constants.php:96 -``` - -规则是绝对的:**`Constants.php` 里只能放普通常量。** 不能有 `env()`、不能有 `getenv()`、不能有配置助手。如果你真需要依赖环境的值,就定义到启动流程更靠后的位置——或者把这个常量降级成 fallback,真正的值后面再读。 - -我做的正是后者。`CONST_fullBase` 变回普通字符串,消费它的助手函数改为**先**查框架自己的 `app.baseURL`(那个才是 `.env` 驱动的): - -```php -// app/Helpers/hoelee_helper.php -function getFullBase(bool $selfCall = false): string -{ - // .env 驱动的 baseURL 优先;CONST_fullBase 只是 fallback - if ($selfCall && defined('CONST_IIS_INTERNAL_BASE') && CONST_IIS_INTERNAL_BASE) { - return rtrim(CONST_IIS_INTERNAL_BASE, '/'); - } - $appBase = config('App')->baseURL; - if ($appBase) return rtrim($appBase, '/'); - if (defined('CONST_fullBase') && CONST_fullBase) return rtrim(CONST_fullBase, '/'); - return ''; -} -``` - -**这个是我自己搞出来的。** 同一周做凭证清理重构时引入的——我把一个硬编码 URL 挪进 `.env`,然后在 `Constants.php` 里用 `env()` 读它。在我机器上是通的,因为本地 `.env` 走的是另一条路径被读到。干净启动下它从来没通过。我是在**部署测试**后一小时内抓到的;如果只是代码审查,很可能就这么放过去了。 - -### 致命错误二 —— CI4 控制器里的 `parent::__construct()` - -第二个失败在 `/lifecode`。这个是应用里**原本就存在**的 bug,不是我引入的。 - -绕过 CLI 渲染器之后,真正的报错是: - -``` -Error: Cannot call constructor -``` - -CI4 的基础类 `CodeIgniter\Controller` **根本没有构造函数**。它实现的是 `initController()`,由框架把 request、response、logger 三个对象传进去。标准写法是: - -```php -// CI4 的正确写法 -public function initController( - RequestInterface $request, - ResponseInterface $response, - LoggerInterface $logger -) { - parent::initController($request, $response, $logger); - // 你的初始化代码 -} -``` - -但 `Lifecode.php` 和 `ApiEn.php` 用的是传统构造函数,还在调 `parent::__construct()`: - -```php -// 错误 —— CodeIgniter\Controller 没有 __construct() -public function __construct() -{ - parent::__construct(); - // ... -} -``` - -对一个没有定义 `__construct()` 的父类调 `parent::__construct()`,在 PHP 8 里是致命错误。把两个控制器改成 `initController()` 就修好了——同时要为新签名补上三个 interface 的 `use` 语句。 - -我没有盲信报错信息,而是先确认父类**确实**没有构造函数: - -```bash -grep -n 'function __construct\|function initController' \ - vendor/codeigniter4/framework/system/Controller.php -``` - -只出来 `initController()`。这一步值得做——"Cannot call constructor" 在父类**有**构造函数但内部报错时也会出现,重写签名之前你得先搞清楚自己是哪种情况。 - -## 为什么这两个 bug 在 IIS 上都不出现 - -这是让我改变部署观念的部分。 - -在 IIS 主机上,`/lifecode` 和 `/api/*` 挡在 authentik SSO 后面。对这两条路由的未认证请求返回 **HTTP 302 跳转到 `auth.hoelee.com`**——压根不会碰到控制器。那个坏掉的构造函数从来没被执行过。这条路由大概从写出来那天就是坏的,而它从来没被要求真正处理过一次请求。 - -我直接对比了两台主机来确认: - -```bash -curl -sI https://calc.hoelee.com/lifecode | head -1 -# HTTP/2 302 <- authentik 跳转;控制器根本没运行 - -curl -sI http://<新主机>/lifecode | head -1 -# HTTP/1.1 500 <- 没有网关;控制器运行并致命崩溃 -``` - -那个 302 就是这个 bug 活下来的原因。应用看起来健康,只是因为它不健康的部分根本触不到。 - -**教训说白了就是:一条路由前面挂了认证网关,就意味着这条路由自己的代码没有任何有效的测试覆盖。** 如果你要迁移或者拆网关,请专门留出时间,用未认证身份把每一条曾被网关挡着的路由都打一遍,再宣布迁移完成。小应用上这只是十行 `curl` 循环。在我这里,它本来几秒钟就能找出这两个 bug,而不是两个小时: - -```bash -for p in / /read/single /read/partner /lifecode /api/date /api/single; do - printf '%-16s %s\n' "$p" "$(curl -s -o /dev/null -w '%{http_code}' https://<主机>$p)" -done -``` - -## 第三步:那个不再是 loopback 的自调用 - -现在是这个架构里真正有意思的后果。 - -报告引擎是靠控制器 `curl` 自己的主机来调用的。在 IIS 上,`CONST_IIS_INTERNAL_BASE` 把这个调用指向 `http://localhost:7296`——loopback 接口,请求不出机器,也不碰 DNS 和 TLS。 - -在 LiteSpeed 上这个常量被**故意不定义**,于是 `getFullBase()` 落到 `app.baseURL`。这意味着每一次生成报告,现在都是**一个真实的对外 HTTPS 请求,打到应用自己的公网 URL,再从 Cloudflare 绕回来**。 - -那问题就来了:这到底能不能跑? - -答案是能,但值得专门测一次,因为它的失败模式很容易误导人。这是确切的测法——**在服务器上跑**,不是在你的笔记本上: - -```bash -curl -s -o /dev/null -w 'HTTPS self-call: %{http_code}\n' \ - -X POST 'https://<域名>/api/single' -d 'nameCn=test&dob=1990-01-01' -``` - -两个要注意的地方: - -- **在服务器上跑。** 我从 Windows 机器上打同一个 URL,拿到的是 404 和 500——那是我这台机器 DNS 解析和 Cloudflare 机器人防护的产物,不是服务器的问题。在错误的主机上测试,会得出很有把握的错误结论。 -- **看 JSON,不只看状态码。** 一个 200 配一个被截断的 body,比干脆失败更糟。我在信任之前先确认了响应能被解析。 - -这里藏着一个真实的取舍,我还没完全解决:走公网 URL 的自调用意味着生成报告依赖 Cloudflare 在线、每次报告多一次 TLS 握手、并且在整个过程中占用两个 PHP worker。在 LiteSpeed 上 worker 池不大的情况下,并发报告一多就可能死锁——每个请求都在等另一个请求,而后者拿不到空闲 worker。以目前的流量没问题。但在规模上去之前,这个自调用应该换成绕过 CDN 的内部路径。 - -## 第四步:验证真实产出,不是状态码 - -`/` 返 200 只证明落地页能渲染。它不证明报告引擎能用,而引擎才是整个产品。所以最后一步是端到端生成一份真报告: - -```bash -curl -s -X POST 'https://<域名>/read/single' \ - -d 'nameCn=test&dob=1990-01-01&gender=m' \ - -o report.html -w 'HTTP:%{http_code} bytes:%{size_download}\n' -``` - -结果:**HTTP 200、99,189 字节、19 页 A4。** 我又用中文字符输入跑了一遍,把引擎的 UTF-8 路径也走通——HTTP 200、99,205 字节、19 页、姓名渲染正常、输出里零 PHP 警告。 - -真正有意义的检查,是新旧主机之间页数可比。它证明字体、DPI 和分页逻辑都完整地熬过了这次搬迁。 - -关于那个 UTF-8 测试的插曲:我第一次跑的时候返回了 500,差点就一头扎进去查 bug。原因是我自己的 shell——中文字符在进 `curl` 的路上被终端编码搞坏了,应用收到的是非法字节。把同一个请求写成服务器上的脚本再跑,编码在我控制之内,就返回了 200。**在你调试编码故障之前,先确认服务器真正收到的字节就是你打算发出去的字节。** - -## 如果重来一次,我会怎么改 - -**迁移前就测未认证路由,而不是迁移后。** 那两个小时的排查,用一个遍历路由列表的 `curl` 循环就能完全避免。我现在把"列出所有路由、以未认证身份逐个请求、记录状态码"当成任何迁移的第零步。 - -**不要在 `Constants.php` 里调 `env()`。** 我在文件里留了注释专门写这件事,因为下一个人——多半是六个月后的我自己——一定会想这么干。 - -**迁移前先查框架版本对应的 project-space 配置。** 同一周我把 CI4 从 4.6.3 升到 4.7.4,它在两个升级指南里根本没提的属性上致命崩溃了两次:`Config\App::$permittedURIChars`(4.7 的 Router 要求)和 `Config\Format::$jsonEncodeDepth`(JSONFormatter 要求,而第二个正好打断了引擎的 JSON 自调用)。Composer 会更新 `vendor/`,但**永远**不会合并 `app/Config/*.php`,因为那些属于 project-space。我最后往 14 个配置文件里手工合并了新属性。那是另一篇文章,但迁移的教训是同一个形状:**框架会告诉你 `vendor/` 里改了什么;没有任何东西会告诉你 `app/` 里改了什么。** - -**先确认 docroot,再动别的。** 如果我解压进 `public_html` 就收手,应用是能跑的——同时静静地把 `.env` 通过 HTTP 服务出去。能跑的迁移,不等于安全的迁移。 - -## 结果 - -一个晚上。六条路由全部验证 200,中英文输入的报告产出在字节层面可比,零 PHP 警告,另外还从代码库里清掉了两个长期潜伏的 bug——而这两个 bug 一直被 IIS 的认证网关遮着。 - -应用现在跑在 OpenLiteSpeed 上,docroot 正确分离,主机可自动化管理,凭证放在 `.env` 而不是写死在常量里。 - ---- - -**想让你的应用也做这套迁移?** 我可以在 IIS、Apache、nginx 和 LiteSpeed 之间迁移 PHP 应用——包括那些麻烦的部分:自调用架构、藏 bug 的 SSO 网关,以及会碰到 project-space 配置的框架升级。 - -[WhatsApp +60 12-797 2969](https://wa.me/60127972969) · [me@hoelee.com](mailto:me@hoelee.com?subject=CodeIgniter%20迁移) · [hoelee.com](https://hoelee.com) -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Migrating a CodeIgniter 4 App From IIS to OpenLiteSpeed" -description: "I moved a 17,000-line CodeIgniter 4 app from Windows IIS to OpenLiteSpeed on CyberPanel. Two fatal errors appeared that IIS had been hiding behind SSO." -pubDate: 2026-09-27 -category: engineering -blueprint: sme-digital-suite -blueprintLayer: L1-high-conversion-web -tags: [codeigniter, php, openspeedway, cyberpanel, iis, migration, litespeed, linux] -ogImage: /og/migrating-codeigniter-iis-to-openlitespeed.jpg -banner: /banners/migrating-codeigniter-iis-to-openlitespeed.jpg -draft: false ---- - -I maintain a numerology report generator: a CodeIgniter 4 application that takes a birth date and a name, runs them through a numerology engine, and renders a 19-page A4 report. It had run on Windows Server with IIS for years. In September 2026 I moved it to OpenLiteSpeed on CyberPanel, on Linux, on a different domain. - -The migration itself was unremarkable — copy the files, point the docroot at `public/`, install dependencies. What made it worth writing about is that **the first unauthenticated request to the new host fatally crashed on two routes that had been working fine on IIS for years.** - -Both bugs were real. Both were present on IIS the whole time. Neither was visible, because IIS had an authentik SSO gate in front of the exact routes that were broken. - -## Why this matters - -If you run a self-hosted app behind an authentication gateway, your auth layer is doing more than protecting data — it is also **hiding your bugs**. Anything that only breaks for unauthenticated users never runs unauthenticated, so it never fails. The failure surfaces at the worst possible moment: when you migrate, remove the gate, or expose the route to the public internet. - -In my case the app had been reporting "healthy" for as long as I'd owned it. The moment I pointed a new domain at it, two routes returned HTTP 500. Nothing had changed in the code. The only thing that changed was that the gate was gone. - -The fix for both took about twenty minutes. Finding them took two hours, because the actual error messages were hidden behind CodeIgniter's CLI error renderer. - -## The setup - -The app is a fairly ordinary CI4 project with an unusual architecture detail: the report engine is not a library, it is a **set of HTTP endpoints on the same host**. The controllers generate a request, `curl` it back to `/api/single` on their own domain, and the engine returns JSON with the computed numbers. The controller then renders that into the printable report view. - -That self-call design is what made `CONST_IIS_INTERNAL_BASE` necessary on IIS. Let me come back to it — it turns out to be the interesting part. - -| | Before | After | -|---|---|---| -| OS | Windows Server | Ubuntu (CyberPanel) | -| Web server | IIS | OpenLiteSpeed 1.9.0 | -| PHP | 8.4 (Windows build) | 8.4.25 (lsphp84) | -| Docroot | `C:\inetpub\calc.hoelee.com` | `/home//public_html/public` | -| Auth gate | authentik SSO on `/lifecode`, `/api/*` | none | - -## Step 1: Point the docroot at `public/`, not the project root - -CI4 ships with a two-folder layout. `app/`, `vendor/`, `writable/` and `.env` live in the project root. Only `public/` is meant to be web-accessible. - -CyberPanel creates the document root as `public_html`. My first instinct was to extract the whole project into `public_html` and leave the docroot alone. - -**That would have exposed `app/`, `vendor/`, the `.env` file, and `writable/` session data over HTTP.** The `.env` in this project contains a webhook credential. Anyone who guessed `/../.env` — or just fetched `/.env`, since the file sits directly under a served directory — gets it. - -The correct layout is to extract into `public_html` and then repoint the vhost docroot one level deeper, at `public_html/public`: - -```bash -# extract the project so that app/, vendor/ and .env sit UNDER public_html -cd /home//public_html -tar -xzf /tmp/deploy.tar.gz - -# the docroot must be public_html/public, NOT public_html -sudo sed -i 's#/home//public_html\$#/home//public_html/public#' \ - /usr/local/lsws/conf/vhosts//vhost.conf - -sudo /usr/local/lsws/bin/lshttpd -t -sudo systemctl restart lsws -``` - -This is the single most important step in the migration, and it is the one most guides skip. If you do nothing else, do this. - -## Step 2: The two fatals IIS was hiding - -### Fatal 1 — `env()` called too early - -The first 500 had no body at all. That is unusual — CI4 normally renders something. An empty 500 usually means PHP died before the framework's error handler was installed. - -CI4's bootstrap loads `app/Config/Constants.php` very early in `Boot::bootWeb()`, via `Boot::loadConstants()`. That happens **before** the `Common.php` helper file is loaded, which is where `env()` is defined. - -So this line: - -```php -// app/Config/Constants.php — BROKEN -define('CONST_fullBase', env('hoelee.fullBase')); -``` - -fails with: - -``` -Fatal error: Uncaught Error: Call to undefined function env() -in app/Config/Constants.php:96 -``` - -The rule is absolute: **`Constants.php` may only contain plain constants.** No `env()`, no `getenv()`, no config helper. If you need environment-dependent values there, define them further along the bootstrap — or make the constant a fallback and read the real value later. - -That is exactly what I did. `CONST_fullBase` became a plain string, and the helper that consumes it checks the framework's own `app.baseURL` (which *is* `.env`-driven) first: - -```php -// app/Helpers/hoelee_helper.php -function getFullBase(bool $selfCall = false): string -{ - // the .env-driven baseURL wins; CONST_fullBase is only a fallback - if ($selfCall && defined('CONST_IIS_INTERNAL_BASE') && CONST_IIS_INTERNAL_BASE) { - return rtrim(CONST_IIS_INTERNAL_BASE, '/'); - } - $appBase = config('App')->baseURL; - if ($appBase) return rtrim($appBase, '/'); - if (defined('CONST_fullBase') && CONST_fullBase) return rtrim(CONST_fullBase, '/'); - return ''; -} -``` - -**This one was my own doing.** I had introduced it during a secrets-cleanup refactor in the same week — I moved a hardcoded URL into `.env` and called `env()` in `Constants.php` to read it. It worked on my machine because the local `.env` was being read through a different path. It never worked on a clean boot. I caught it within an hour *only because* I deployed and tested; a code review would plausibly have waved it through. - -### Fatal 2 — `parent::__construct()` in a CI4 controller - -The second failure was on `/lifecode`, and this one is a genuine pre-existing bug in the application — not something I introduced. - -The error, once I got past the CLI renderer, was: - -``` -Error: Cannot call constructor -``` - -CI4's base `CodeIgniter\Controller` class **has no constructor**. It implements `initController()`, which the framework calls with the request, response, and logger objects. The standard pattern is: - -```php -// correct CI4 pattern -public function initController( - RequestInterface $request, - ResponseInterface $response, - LoggerInterface $logger -) { - parent::initController($request, $response, $logger); - // your setup here -} -``` - -But `Lifecode.php` and `ApiEn.php` declared a classic constructor and called `parent::__construct()`: - -```php -// BROKEN — CodeIgniter\Controller has no __construct() -public function __construct() -{ - parent::__construct(); - // ... -} -``` - -`parent::__construct()` on a parent class that does not define `__construct()` is a fatal in PHP 8. Converting both controllers to `initController()` fixed it — and required adding the three `use` statements for the interface types in the new signature. - -I checked whether the base class *really* had no constructor rather than trusting the error message: - -```bash -grep -n 'function __construct\|function initController' \ - vendor/codeigniter4/framework/system/Controller.php -``` - -Only `initController()` came back. Worth doing — "Cannot call constructor" also fires when a parent *has* a constructor that errors internally, and you want to know which case you are in before you rewrite the signature. - -## Why neither bug showed up on IIS - -This is the part that changed how I think about the deployment. - -On the IIS host, `/lifecode` and `/api/*` sat behind authentik SSO. An unauthenticated request to either returned **HTTP 302 to `auth.hoelee.com`** — it never reached the controller at all. The broken constructor was never executed. The route had presumably been broken since it was written, and it had never once been asked to serve a request. - -I confirmed this by comparing the two hosts directly: - -```bash -curl -sI https://calc.hoelee.com/lifecode | head -1 -# HTTP/2 302 <- authentik redirect; controller never runs - -curl -sI http:///lifecode | head -1 -# HTTP/1.1 500 <- no gate; controller runs and fatals -``` - -The 302 is why the bug survived. The app looked healthy because the unhealthy parts were unreachable. - -**The lesson, stated plainly: an auth gate in front of a route means that route has no working test coverage for its own code.** If you migrate or remove the gate, budget time to hit every previously-gated route unauthenticated before you call the migration done. On a small app that is a ten-line `curl` loop. Here it would have found both bugs in seconds instead of two hours: - -```bash -for p in / /read/single /read/partner /lifecode /api/date /api/single; do - printf '%-16s %s\n' "$p" "$(curl -s -o /dev/null -w '%{http_code}' https://$p)" -done -``` - -## Step 3: The self-call that stops being a loopback - -Now the interesting architectural consequence. - -The report engine is reached by the controller `curl`-ing its own host. On IIS, `CONST_IIS_INTERNAL_BASE` pointed that call at `http://localhost:7296` — the loopback interface — so the request never left the machine and never touched TLS or DNS. - -On LiteSpeed that constant is deliberately left **undefined**, so `getFullBase()` falls through to `app.baseURL`. Which means every report generation now makes a **real outbound HTTPS request to the app's own public URL** and comes back in through Cloudflare. - -Which raises the obvious question: does that work at all? - -The answer is yes, but it is worth testing explicitly, because the failure mode is confusing. Here is the exact test — run it **on the server**, not from your laptop: - -```bash -curl -s -o /dev/null -w 'HTTPS self-call: %{http_code}\n' \ - -X POST 'https:///api/single' -d 'nameCn=test&dob=1990-01-01' -``` - -Two things to watch for: - -- **Run it from the server.** From my Windows machine the same URL returned 404 and 500 — an artefact of DNS resolution and Cloudflare bot protection on my caller, not a server problem. Testing from the wrong host produces confident, wrong conclusions. -- **Watch the JSON, not just the status code.** A 200 with a truncated body is worse than a clean failure. I checked the response was parseable before trusting it. - -There is a real trade-off hiding here, and I have not fully resolved it: a public-URL self-call means report generation depends on Cloudflare being up, costs a TLS handshake per report, and occupies two PHP workers for the duration. On LiteSpeed with a small worker pool, a burst of concurrent reports could deadlock — each request waiting on another request that has no free worker. For the current traffic level it is fine. Before this scales, the self-call should move to an internal path that bypasses the CDN. - -## Step 4: Verify the actual output, not the status code - -A 200 on `/` proves the landing page renders. It does not prove the report engine works, and the engine is the entire product. So the last step was generating a real report end to end: - -```bash -curl -s -X POST 'https:///read/single' \ - -d 'nameCn=test&dob=1990-01-01&gender=m' \ - -o report.html -w 'HTTP:%{http_code} bytes:%{size_download}\n' -``` - -The result: **HTTP 200, 99,189 bytes, 19 A4 pages.** I ran it again with Chinese-character input to exercise the UTF-8 path through the engine — HTTP 200, 99,205 bytes, 19 pages, name rendering correctly, zero PHP warnings in the output. - -Byte-comparable page counts between the old and new host is the check that matters. It proves the fonts, the DPI, and the pagination logic all survived the move. - -A side note on the UTF-8 test: my first attempt at it returned a 500 and I nearly went bug-hunting. The cause was my own shell — the Chinese characters were being mangled by the terminal encoding on the way into `curl`, so the app received invalid bytes. Running the same request from a script on the server, where the encoding is under my control, returned 200. **Before you debug an encoding failure, confirm the bytes actually arriving at the server are the bytes you meant to send.** - -## What I'd do differently - -**Test unauthenticated routes before migrating, not after.** The whole two-hour debugging session was avoidable with a `curl` loop over the route list. I now treat "list every route, hit it unauthenticated, record the status" as step zero of any migration. - -**Do not call `env()` in `Constants.php`.** I have left a comment in the file itself saying so, because the next person — probably me in six months — will be tempted. - -**Check the framework version's project-space config before the move.** The same week, I upgraded CI4 from 4.6.3 to 4.7.4, and it fataled twice on properties the upgrade guide did not mention: `Config\App::$permittedURIChars`, required by the 4.7 Router, and `Config\Format::$jsonEncodeDepth`, required by the JSONFormatter — the second of which broke the engine's JSON self-call specifically. Composer updates `vendor/` but never merges `app/Config/*.php`, because those are project-space files. I ended up merging new properties into 14 config files. That is a separate post, but the migration lesson is the same shape: **the framework tells you what changed in `vendor/`; nothing tells you what changed in `app/`.** - -**Confirm the docroot before anything else.** If I had extracted into `public_html` and stopped, the app would have worked — and quietly served `.env` over HTTP. A migration that works is not the same as a migration that is safe. - -## The result - -One evening. Six routes verified 200, byte-comparable report output on both Chinese and ASCII input, zero PHP warnings, and two long-standing latent bugs removed from the codebase that IIS had been concealing behind an auth gate. - -The app is now running on OpenLiteSpeed with a properly separated docroot, on a host I can automate, with the credential in `.env` rather than a constant. - ---- - -**Want this done for your own application?** I migrate PHP applications between IIS, Apache, nginx and LiteSpeed — including the awkward parts: self-calling architectures, SSO gates that hide bugs, and framework upgrades that touch project-space config. - -[WhatsApp +60 12-797 2969](https://wa.me/60127972969) · [me@hoelee.com](mailto:me@hoelee.com?subject=CodeIgniter%20migration) · [hoelee.com](https://hoelee.com) -```` - ---- - -## 文章 09 / 11:用「热浏览器会话」爬取有反爬墙的电商网站 -- **Slug**: `scraping-bot-walled-marketplace-warm-browser-session` -- **英文标题**: Scraping a Bot-Walled Marketplace With a Warm Browser Session -- **分类**: `engineering` | **发布日期**: 2026-09-13 -- **线上地址**: https://blog.hoelee.com/posts/scraping-bot-walled-marketplace-warm-browser-session/ (EN) | https://blog.hoelee.com/posts/zh/scraping-bot-walled-marketplace-warm-browser-session/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】:[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】: -【人设与用词卫生】: -【CRO与转化建议】: -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "用「热浏览器会话」爬取有反爬墙的电商网站" -description: "为一位做二手手机生意的客户:爬取有反爬墙的电商商品列表——热 CDP 会话、失败即止的健康检查、分段等待渲染。" -pubDate: 2026-09-13 -category: engineering -tags: ["scraping", "cdp", "headless-chrome", "anti-bot", "shopee", "python"] -ogImage: "/og/zh-scraping-bot-walled-marketplace-warm-browser-session.jpg" -banner: "/banners/scraping-bot-walled-marketplace-warm-browser-session.jpg" -draft: false ---- - -一位客户请我做一个听起来很简单的东西:每天把一个大电商平台上的二手手机列出来,按价格排序,好让他的进货决策靠数据而不是靠感觉。这位客户靠倒卖二手电子产品吃饭——一台手机差几百马币就是他的利润。 - -结果完全不是一段干净的 `requests` + BeautifulSoup 脚本。它变成了一场长达一周的、跟反爬墙的拉锯战,一个靠胶带勉强粘在一起的热浏览器会话,以及——我们俩都没想到的部分——一个关于「你不能靠堆更多 agent 来『并行化』爬虫」的惨痛教训。 - -这就是那个故事,按照我经历的顺序来讲:问题、我试过却失败的每一件事、最终奏效的办法,以及下次我会怎么做。 - -## 问题,用别人真正会搜索的方式表述 - -> "如何从一个拦截无头浏览器的网站爬取商品列表?" - -或者更诚实地说:**当一个电商网站已经把「防爬」当成它唯一的本职工作,你要怎么爬它?** Shopee 不会礼貌地把 HTML 喂给 `curl`。它会检测自动化、抛出一个 `/verify` 验证码,然后静默地给你一个「看起来正常但里面什么都没有」的页面。 - -客户的需求清单很短: - -- 搜索二手手机,只筛选「二手」成色,存储 128 GB 及以上。 -- 提取每个商品的所有存储规格和价格。 -- 按价格升序排序。 -- 不搞一百个 headless 实例的 `chromedriver` 农场——倒卖的利润养不起一百个住宅 IP,我的也一样。 - -从这里开始,全部都是调试故事。 - -## 我试过的,以及为什么失败 - -### 尝试 1:通过 CDP 用冷的无头 Chrome——「直接把 Chrome 远程调试开起来」 - -我用 `--headless --remote-debugging-port=9222` 启动了 Chrome,用 Chrome DevTools Protocol 连上去,然后导航到搜索 URL。大约十秒钟里我觉得自己很聪明。 - -**失败:** 页面加载了,但每个商品卡片都是空的。DOM 是个空壳。Shopee 指纹识别了这个无头浏览器(没有真正的 GPU、user-agent 指纹不一致、`navigator.webdriver` 标志),把我重定向进了一个我永远过不去的 `/verify` 验证码循环。冷的无头 profile 一上来就撞墙。 - -教训很无聊但很重要:**如果你的浏览器本身在「身份」上撒谎,选择器写得再好也没用。** - -### 尝试 2:并行子 agent——「直接开 N 个 worker」 - -这是我最想让它成功的方案。我有一套多 agent 的架构。本能反应就是:开五个 agent,每个爬一个不同的搜索词,收集结果,合并。 - -**失败:** 它们全都连到**同一个**浏览器会话上。它们不会各自拿到一个浏览器——只会各自拿到共享 Chrome 实例里的一个标签页,或者更糟:全都挤在同一个标签页上。每一次 `Page.navigate` 都会把上一个 agent 的上下文抹掉。而且关键在于: - -1. **热会话是共享资源。** 只有一个没被标记的 Canary profile。五个 agent 交错访问同一个标签页,在 Shopee 眼里,跟一个浏览器疯狂快速导航没有区别。而这恰恰是**触发**反爬墙的行为。 -2. **反爬的「节奏」是累积的。** 我的可用方案依赖导航之间约 7 秒的间隔来显得像真人。五个 agent 并发导航,就等于五个导航以零间隔打向同一个会话。 - -所以「横向扩展」的本能——加更多 worker——对我赖以存活的那件事(看起来像真人)是有害的。这里的并发上限不是 CPU 也不是内存,而是**一个热浏览器,一次一个标签页。** - -### 尝试 3:靠「Buy With Voucher」按钮来拿价格 - -一旦列表能渲染出来了,我假设购买按钮的文案里带着价格。大多数电商网站确实如此。但这个渲染出来的按钮写的是 `Add To Cart` / `Buy Now`。我用来匹配「Buy With Voucher … RM…」的正则什么都匹配不到,我在原地打转了好一会儿,才真正去读了 DOM。 - -**失败:** Shopee 商品页的价格不在按钮里。它被渲染成**拆分的 span**——`RM` 在一个元素里、`850` 在另一个里、`00` 又在另一个里——所以朴素的叶子文本匹配只会抓到碎片,或者当商品有多个规格时,抓到一个**区间**(`RM850.00 - RM1150.00`)。 - -### 尝试 4:点击规格选项来拿精确价格 - -对于单轴规格选择器(比如一个存储下拉:128GB / 256GB / 512GB),点击一个选项会把价格收敛成单个值,我的提取方法能正常工作。 - -**失败:** 对于**双轴**选择器(品牌 × 存储),点击第一个轴(某个品牌)会锁定一个价格,但加上第二个轴(某个存储规格)又把它**解散**回商品全区间。这个面板渲染的是「所有可购买组合」的区间,而不是我选中的那个具体组合;而且「已选中」状态是用一个「同时匹配其他所有选项框」的 class 来标记的。我没法可靠地知道我到底锁定了哪个组合。 - -我在这个上面花了实打实的时间,最后才接受这个诚实的答案:**对于真正的双轴商品,正确的数据是价格区间,而不是一个瞎猜的「每个组合」数字。** 发布一个「有时候能用」的脆弱覆盖方案,比老实报告一个区间更糟糕。 - -## 修复——真正奏效的东西 - -三块,都很小,都是吃了亏才学会的。 - -### 1. 热的人性化节奏的浏览器会话(地基) - -别跟墙硬刚。用一个已经登录过、正常浏览过的浏览器 profile——一个「热」profile。用真实的 7 秒间隔来导航。它更慢,但它是唯一能稳定活下来的方式。 - -```python -# 不是真正的代码——是它的「形状」。先导航,然后像真人一样等待。 -goto_url(target) -time.sleep(7) # 真人节奏,不可妥协 -wait_for_selector(".shopee-search-item-result__item") -``` - -### 2. 失败即止,别傻等超时 - -在任何爬取之前,先打 CDP 的 HTTP 端点,检查会话状态。如果 Canary 挂掉了、或标签页停在 `/verify` 上,就在约 1 秒内中止,而不是在「怎么还不渲染」的超时里浪费 40 秒。 - -```python -import json, urllib.request - -def session_healthy(): - try: - tabs = json.load(urllib.request.urlopen("http://127.0.0.1:9222/json")) - except Exception: - return False - for t in tabs: - if t.get("type") == "page" and "/verify" in t.get("url", ""): - return False # 撞到验证码墙——重新热会话,别爬了 - return True -``` - -就这么一个改动,把「静默的 40 秒死胡同」变成了「即时、可执行的失败」。 - -### 3. 正确的价格选择器:最深的纯 RM 节点 - -购买按钮是个谎言。价格是**那个文本内容纯粹是一个 RM 值的、最深的元素**——在选中规格之前是一个区间(`RM850.00 - RM1150.00`),之后是单个值(`RM850.00`)。遍历 DOM,取文本能匹配上的最深元素: - -```python -import re - -RM = re.compile(r"^RM\s?[\d,.]+(\s?[-–]\s?RM\s?[\d,.]+)?$") - -def find_price(root): - best, depth = None, -1 - for el in root.querySelectorAll("*"): - txt = (el.textContent or "").strip() - if txt and RM.match(txt) and " " not in txt.split("RM")[0]: - d = depth_of(el) - if d > depth: - best, depth = el, d - return best.textContent.strip() if best else None -``` - -这就是全部的诀窍。没有脆弱的 class 名。没有瞎猜。一个「整个文本就是一个价格」的节点,就是价格本身。 - -## 结果——一个可量化的成果 - -从单个热会话、一次串行遍历中,我提取了 **20+ 个二手手机商品及其各规格价格**,筛选到 128 GB 及以上,按价格升序排序。最便宜的真实目标——一台 128GB 机型——以 **RM299** 的价格浮出水面,另外还有几台 128/256GB 的选项在 RM450–880 区间。现在一条命令(`sweep.py ...`)就能重新检查整份候选清单并输出 JSONLines;客户每天进货前跑一次。 - -但我最庆幸的数字是这个:**零验证码。** 「热会话 + 人性化节奏 + 失败即止」这套组合,在整次遍历中一次都没触发过 `/verify`。 - -## 下次我会怎么做 - -1. **我会一开始就写「失败即止」的健康检查**,在任何爬取代码之前。三十秒的活,换来未来每次失败省四十秒。 -2. **我会从第一天就抵抗并行化的冲动。** 「爬虫太慢」的答案是把更多工作打包进一次串行遍历(批量搜索、对多个 URL 跑 `sweep.py`),而不是「再加 worker」。共享的浏览器状态让并发变得更糟,而不是更好。 -3. **我会立刻接受双轴的限制**,老实报告区间,而不是在一个根本没法可靠化的选择器上烧一个下午。 -4. **我会别再猜 DOM 结构,先去探测它。** 几乎每一个错误假设(URL 路径里 `/i.` vs `-i.`、把 `Buy With Voucher` 当价格锚点、「过滤器是个真正的 `` 复选框」)都来自**没有先读真实的 DOM**。 - -## 可复用的部分 - -所有这一切被做成了一个小的独立工具集——每个脚本都是一次自包含的 `python x.py ...` 调用: - -| 脚本 | 作用 | -|---|---| -| `search.py` | 粗粒度搜索 → 商品卡片 | -| `used_search.py` | 点击「二手」条件复选框,校验 URL 里的 `USED_ITEM`,再提取 | -| `detail.py` | 从商品页提取各规格价格(最深 RM 节点法 + 分段等待) | -| `sweep.py` | 一次会话内对 N 个 URL 做批量详情提取,按 item-id 去重,写 JSONLines,打印表格 | -| `cdp_common.py` | 共享的健康检查 + 导航辅助 | - -真正可复用的教训不是那些 Shopee 专属的选择器——而是**姿态**:**假设页面在 DOM 上撒谎,失败即止,用批量化而不是并行 worker 来扩展。** - ---- - -*这是一次真实的调试过程,不是事后补写的教程。这个爬虫工具集、以及这里的每一次失败,都发生在为一个做二手电子产品倒卖的客户构建它的过程中——我保留了能泛化的部分,砍掉了那些只对某个马来西亚电商网站、某个时间点有效的部分。* - ---- - -## 想为你的业务做这个吗? - -我做定制爬虫和数据提取管道——热会话 CDP 爬虫、反爬墙感知的采集器、批量价格监控扫描——也做全栈 Web 应用(PHP/CodeIgniter、Java/Spring、React/TypeScript)和自托管自动化(n8n、Telegram 机器人、NAS 上的 Docker)。 - -如果你需要从某个「会反抗」的网站拿商品或价格数据,直接点进来,告诉我你想提取什么: - -- 💬 [WhatsApp](https://wa.me/60127972969) — 直接私信我 -- ✉️ [Email](mailto:me@hoelee.com?subject=Custom%20scraping%20pipeline) — 告诉我网站和数据 -- 🌐 [hoelee.com](https://hoelee.com) — 看看我还做了些什么 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Scraping a Bot-Walled Marketplace With a Warm Browser Session" -description: "Scraping bot-walled marketplace listings for a client's used-electronics business: warm CDP session, fail-fast health checks, split-wait rendering." -pubDate: 2026-09-13 -category: engineering -tags: ["scraping", "cdp", "headless-chrome", "anti-bot", "shopee", "python"] -ogImage: "/og/scraping-bot-walled-marketplace-warm-browser-session.jpg" -banner: "/banners/scraping-bot-walled-marketplace-warm-browser-session.jpg" -draft: false ---- - -A client asked me to build something that sounds simple: every day, list the second-hand phones on a big marketplace, sorted by price, so their restocking decisions come from data instead of gut feel. The client resells used electronics for a living — a few hundred ringgit either way on a phone is the whole margin. - -The result was nothing like a clean `requests` + BeautifulSoup script. It was a week-long fight against an anti-bot wall, a warm browser session held together with duct tape, and — the part neither of us saw coming — a hard lesson about why you cannot just "parallelize" a scraper by throwing more agents at it. - -This is that story, told in the order I lived it: the problem, everything I tried that failed, the thing that finally worked, and what I'd do differently. - -## The problem, phrased the way someone would actually search it - -> "How do I scrape product listings from a site that blocks headless browsers?" - -Or, more honestly: **how do you scrape a marketplace that has already made scraping its sole job description?** Shopee doesn't politely serve HTML to `curl`. It detects automation, throws a `/verify` captcha, and silently serves you a page that looks fine but contains nothing. - -The client's requirement list was small: - -- Search for used phones, filter to "used condition only", storage 128 GB and up. -- Extract every listing's storage variants and prices. -- Sort by price ascending. -- Do it without a `chromedriver` farm of a hundred headless instances — a reseller's margins don't pay for a hundred residential IPs, and neither do mine. - -Everything after this point is the debugging story. - -## What I tried, and why it failed - -### Attempt 1: cold headless Chrome via CDP — "just point Chrome's remote debugger at it" - -I launched Chrome with `--headless --remote-debugging-port=9222`, connected with the Chrome DevTools Protocol, and navigated to a search URL. It felt clever for about ten seconds. - -**Failure:** the page loaded, but every product card was empty. The DOM was a shell. Shopee had fingerprinted the headless browser (no real GPU, no real user-agent fingerprint consistency, `navigator.webdriver` flag) and redirected me into a `/verify` captcha loop I never got past. Cold headless profiles hit the wall immediately. - -The lesson is boring but important: **anti-bot systems don't care how good your selectors are if the browser itself is lying about what it is.** - -### Attempt 2: parallel subagents — "just spawn N workers" - -This was the one I really wanted to work. I have a multi-agent setup. The natural instinct is: spawn five agents, each scraping a different search query, gather the results, merge. - -**Failure:** all of them attach to the *same* browser session. They don't get their own browser — they get their own *tab* in one shared Chrome instance, or worse, they all fight over the same single tab. Every `Page.navigate` un-does the previous agent's context. And critically: - -1. **The warm session is a shared resource.** There is one Canary profile that isn't flagged. Five agents navigated the same tab in an interleaved order, which is indistinguishable — to Shopee — from one browser doing rapid-fire navigation. That's exactly the behavior that *triggers* the wall. -2. **Anti-bot pacing is cumulative.** My working setup depends on ~7-second gaps between navigations to look human. Five agents navigating concurrently means five navigations hitting the same session with effectively zero gap. - -So the "horizontal scaling" instinct — more workers — is actively hostile to the one thing keeping me alive: looking human. The concurrency ceiling here isn't CPU or memory. It's **one warm browser, one tab at a time.** - -### Attempt 3: trusting the "Buy With Voucher" button for the price - -Once I had listings rendering, I assumed the buy button's label contained the price. On most marketplaces it does. On this render, the button said `Add To Cart` / `Buy Now`. My regex for "Buy With Voucher … RM…" matched nothing, and I went around in circles for a while before actually reading the DOM. - -**Failure:** the price on Shopee product pages isn't in the button. It's rendered as **split spans** — `RM` in one element, `850` in another, `00` in a third — so a naive leaf-text match returns fragments, or a **range** (`RM850.00 - RM1150.00`) when the listing has multiple variants. - -### Attempt 4: clicking variant options to get an exact price - -For a single-axis variant selector (e.g. one dropdown of storage options — 128GB / 256GB / 512GB), clicking one option collapses the price to a single value, and my extraction worked. - -**Failure:** for a **two-axis** selector (brand × storage), clicking the first axis (a brand) locks a price, but adding the second axis (a storage size) *un-collapses* it back to the product-wide range. The panel renders the range of all purchasable combinations, not the exact combination I selected, and the "selected" state is marked by a class that also matches every other option box. There is no reliable way to know which combination I actually locked. - -I spent real time on this before accepting the honest answer: **for genuinely two-axis listings, the correct data is the price range, not a guessed per-combination number.** Shipping a fragile override that "sometimes works" would be worse than reporting an honest range. - -## The fix — what actually worked - -Three pieces, all small, all learned the hard way. - -### 1. A warm, human-paced browser session (the foundation) - -Don't fight the wall. Use a browser profile that has already logged in and browsed normally — a "warm" profile. Navigate with real 7-second gaps. It is slower and it is the only thing that reliably survives. - -```python -# not real code — the *shape* of it. Navigate, then wait like a human. -goto_url(target) -time.sleep(7) # human pacing, non-negotiable -wait_for_selector(".shopee-search-item-result__item") -``` - -### 2. Fail fast, don't time out - -Before any scrape, hit the CDP HTTP endpoint and check the session state. If Canary is down or the tab is sitting on `/verify`, abort in ~1 second instead of wasting 40 seconds on a "did not render" timeout. - -```python -import json, urllib.request - -def session_healthy(): - try: - tabs = json.load(urllib.request.urlopen("http://127.0.0.1:9222/json")) - except Exception: - return False - for t in tabs: - if t.get("type") == "page" and "/verify" in t.get("url", ""): - return False # captcha wall — re-warm the session, don't scrape - return True -``` - -This one change turned "silent 40-second dead-ends" into "instant, actionable failure." - -### 3. The right price selector: deepest pure-RM node - -The buy button is a lie. The price is the **deepest element whose text is a pure RM value** — a range (`RM850.00 - RM1150.00`) before a variant is selected, a single value (`RM850.00`) after. Walk the DOM and take the deepest element whose text matches: - -```python -import re - -RM = re.compile(r"^RM\s?[\d,.]+(\s?[-–]\s?RM\s?[\d,.]+)?$") - -def find_price(root): - best, depth = None, -1 - for el in root.querySelectorAll("*"): - txt = (el.textContent or "").strip() - if txt and RM.match(txt) and " " not in txt.split("RM")[0]: - d = depth_of(el) - if d > depth: - best, depth = el, d - return best.textContent.strip() if best else None -``` - -That's the whole trick. No brittle class names. No guessing. A node whose entire text *is* a price is the price. - -## The result — one quantified outcome - -From a single warm session, in one serial pass, I extracted **20+ used-phone listings with per-variant prices**, filtered to 128 GB and up, sorted by price. The cheapest real target — a 128GB model — surfaced at **RM299**, alongside several 128/256GB options in the RM450–880 range. One command (`sweep.py ...`) now re-checks the whole shortlist and dumps JSONLines; the client runs it once a day before restocking. - -But the number I'm most glad about is this: **zero captchas.** The warm-session + human-pacing + fail-fast combo survived the entire sweep without tripping `/verify` once. - -## What I'd do differently - -1. **I'd write the fail-fast health check first**, before a single scraping line. It's thirty seconds of work that saves forty seconds per future failure. -2. **I'd resist the parallelization instinct day one.** The answer to "the scraper is slow" is "bundle more work into one serial pass" (batch search, `sweep.py` over many URLs), not "more workers." Shared browser state makes concurrency worse, not better. -3. **I'd accept the two-axis limitation immediately** and report ranges, instead of burning an afternoon on a selector that can't be made reliable. -4. **I'd stop guessing DOM structure and probe it.** Almost every mistaken assumption (`/i.` vs `-i.` in the URL path, `Buy With Voucher` as the price anchor, "the filter is a real `` checkbox") came from *not reading the actual DOM first*. - -## The reusable pieces - -All of this became a small, standalone toolkit — each script is one self-contained `python x.py ...` call: - -| Script | What it does | -|---|---| -| `search.py` | Coarse search → listing cards | -| `used_search.py` | Clicks the "Used" condition checkbox, verifies `USED_ITEM` in the URL, extracts | -| `detail.py` | Pulls per-variant prices from a product page (deepest-RM-node method + split-wait) | -| `sweep.py` | Batch detail over N URLs in one session, dedupes by item-id, writes JSONLines, prints a table | -| `cdp_common.py` | Shared health check + navigation helpers | - -The genuinely reusable lesson isn't the Shopee-specific selectors — it's the *posture*: **assume the page is lying about its DOM, fail fast, and scale by batching, not by parallel workers.** - ---- - -*This is a real debugging session, not a tutorial written after the fact. The scraping toolkit and every one of these failures happened while building it for a client who resells used electronics — I kept the parts that generalize and cut the parts that only matter to one Malaysian marketplace at one point in time.* - ---- - -## Want this for your business? - -I build custom scraping and data-extraction pipelines — warm-session CDP scrapers, bot-wall-aware crawlers, and batch price-monitoring sweeps — as well as full-stack web apps (PHP/CodeIgniter, Java/Spring, React/TypeScript) and self-hosted automation (n8n, Telegram bots, Docker on NAS). - -If you need product or price data off a site that fights back, tap through and tell me what you're trying to extract: - -- 💬 [WhatsApp](https://wa.me/60127972969) — message me directly -- ✉️ [Email](mailto:me@hoelee.com?subject=Custom%20scraping%20pipeline) — tell me the site + the data -- 🌐 [hoelee.com](https://hoelee.com) — see what else I build -```` - ---- - -## 文章 10 / 11:JavaScript 文件静默失效的两种方式:而页面看上去一切正常 -- **Slug**: `two-ways-a-javascript-file-stops-working-and-looks-fine` -- **英文标题**: Two Ways a JavaScript File Stops Working and Looks Fine -- **分类**: `engineering` | **发布日期**: 2025-02-20 -- **线上地址**: https://blog.hoelee.com/posts/two-ways-a-javascript-file-stops-working-and-looks-fine/ (EN) | https://blog.hoelee.com/posts/zh/two-ways-a-javascript-file-stops-working-and-looks-fine/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】:[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】: -【人设与用词卫生】: -【CRO与转化建议】: -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "JavaScript 文件静默失效的两种方式:而页面看上去一切正常" -description: "一个被参数名遮住的函数名,加一个声明在首次调用之后的 let,让整段脚本停摆——而页面渲染得完美好看,因为屏幕上什么都没变。" -pubDate: 2025-02-20 -updatedDate: 2026-10-16 -category: engineering -tags: ["javascript", "i18n", "debugging", "frontend"] -ogImage: /og/zh-two-ways-a-javascript-file-stops-working-and-looks-fine.jpg -banner: /banners/two-ways-a-javascript-file-stops-working-and-looks-fine.jpg -draft: false ---- - -## 一次「什么都没发生」的改动 - -我给一个自己用的媒体转码队列看板加了语言切换:三份语言包、一个 `t(key, vars)` 函数、界面 chrome 上的 `data-i18n`、顶栏一个 `` in the top bar. - -After deploying it, the page looked exactly as it had before. Not broken — -*unchanged*. Same layout, same Chinese labels, no missing panels, no error -card, no console banner. The switcher was simply not there, and neither was -anything else the new code was supposed to do. - -That is the shape of bug I want to write down, because the page was not -"mostly working". **The script had stopped executing partway through — -twice, for two unrelated one-line reasons — and the page rendered perfectly -anyway.** - -## Why nothing looked broken - -Two design decisions I would make again, and both of them hid the failure. - -The first is that the default language lives in the markup. The Chinese text -is written inside the elements; `data-i18n` is only a hook that *overrides* -it when you switch. That is deliberate — the page needs no fetch to render, -and if a language pack goes missing you get Chinese instead of a row of -`top.theme_to_dark` keys. It also means a completely dead i18n layer -produces a page that is, visually, correct. - -The second is that the buttons are wired with inline handlers -(`onclick="rtFoo()"`), which after a script abort point at functions that -were never defined. Nothing warns you. The page only reveals the truth when -a human clicks something. - -So the symptom was "my change had no effect", which is the least informative -symptom a web page can give you. - -## Failure one: a parameter that swallowed a function - -The file has a theme block and an i18n block. The theme block had this -signature: - -```js -function applyTheme(t) { // t = the theme - label.textContent = t('top.theme_to_dark'); // TypeError: t is not a function -} -``` - -`applyTheme` was written before the i18n layer existed, and `t` was a -perfectly reasonable name for "theme". When the i18n layer arrived, `t` in -that scope stopped meaning the theme and started meaning *the function I -wanted to call* — except the parameter shadows it, so the call became -"invoke a string". - -A `TypeError` at the top level of a classic script does not just fail one -line. Everything after it never runs: the language-pack loader, the -`data-i18n` sweep, the switcher rendering. One wrong letter removed an -entire feature, silently. - -The fix is one character — rename the parameter to `th` — plus a comment -where the next person will look: - -```js -function applyTheme(th) { // NOT `t`: t() is the i18n function in this file -``` - -## Failure two: a declaration that arrived too late - -The second cause was in the same file, and it is a JavaScript rule people -know and still trip over. The theme block runs at load and calls `t('…')`. -The i18n block, further down the file, began with: - -```js -let LANG = localStorage.getItem('rt_lang') || 'zh-CN'; -``` - -`let` and `const` are hoisted but **not initialised**: between the start of -the scope and that line they live in the temporal dead zone, and any access -throws `Cannot access 'LANG' before initialization`. Because the theme block -sits *above* the i18n block, the failure was not a race — it was guaranteed -on every single load. - -Note the distinction from `var`, which would have quietly given me -`undefined` and a working page. The TDZ is the language protecting me from -exactly the bug I was about to write; the only reason it looked like -"nothing happened" is the markup fallback again. - -The fix is ordering, and ordering here is a *dependency*, not a style -choice: if a block calls `t()`, the i18n declarations have to be above it. -The file had been organised by topic ("theme, then i18n") because that is -how it grew. - -## Why my checks did not catch either one - -I had tests. They checked intent, not effect: - -- the three language packs exist and parse; -- every key used in the markup exists in every pack; -- no pack contains a dead key (a typo detector); -- the `{var}` placeholder sets in the three packs match. - -All true. All green. None of them executes the page, so none of them can see -a script that aborts on line 12 of 900. Static checks are structurally -unable to catch "the file is valid and the program is dead". - -## Assert on effects, in a browser - -What found both bugs was asking the live page questions that only a running -program can answer: - -```js -typeof t === 'function' // is the entry point still callable? -document.querySelectorAll('[data-i18n]').length // hooks present -rtCheckHandlers() // do the inline handlers still resolve? -``` - -That last one is worth more than it looks. The page carries **219 inline -`onclick` attributes naming 217 distinct functions**; the check enumerates -them and asserts each name is defined on `window`. It turns a hand-clicked -smoke test into a two-second automated one, and it is the check that turns -"the page looks right" into a measurable property. - -The second tool is cheaper and nastier: a **pseudo-language**. -`?lang=pseudo` takes every string in the pack, prefixes it with `[~~` and -stretches it 1.6×. Then assert two things — no horizontal overflow at -1440px, and no element in the nav still lacks the `[~~` prefix. The first -catches layout that only breaks in a longer language; the second catches -nodes that never got a translation hook at all. It found a real one in the -same run. - -Honest scope note: a pseudo-locale does not prove your Malay translation is -*good*. It proves the *plumbing* is complete — every string flows through -the mechanism, and the layout survives length. Judgement about the words -stays with a human. - -## What I would do differently - -1. **Load the page in a real browser and assert `typeof` on the entry points - before believing a client-side refactor worked.** A dead script is - invisible to every static check you own. -2. **Never let a parameter name shadow a function the same file calls.** One - letter bought me an entire afternoon; the comment is the part that stops - it recurring. -3. **Treat declaration order as a dependency, not aesthetics.** If a block - calls a function or reads a constant, that declaration goes above it — - especially in one long script. -4. **Generate the language packs from one table.** Then "the keys match" is - a test rather than a hope, and drift cannot be merged. -5. **When a change does nothing, suspect the script before the change.** - "Nothing happened" is the default failure mode of a rendered page, not - evidence that your code was inert. - -## The result - -Two one-line causes, both in the same file, both found by browser-side -assertions rather than by reading code again. The dashboard switches -language at runtime now, the default render is byte-for-byte what it was -before the change, and the three packs come from a single generated table — -so a missing key shows up as a red test instead of as a word that quietly -stays Chinese. - -The generalisation is the part I keep: **a crash tells you where to look. A -silent stop tells you nothing, and looks like success.** If a feature can -fail without moving a single pixel, then "the page still looks right" is not -a check. -```` - ---- - -## 文章 11 / 11:逐页验证一份 19 页的 PDF 报告 -- **Slug**: `verifying-a-pdf-report-page-by-page` -- **英文标题**: Verifying a 19-Page PDF Report Page by Page -- **分类**: `engineering` | **发布日期**: 2026-09-28 -- **线上地址**: https://blog.hoelee.com/posts/verifying-a-pdf-report-page-by-page/ (EN) | https://blog.hoelee.com/posts/zh/verifying-a-pdf-report-page-by-page/ (ZH) -- **本篇审核状态**: ⏳ 待审核 (Pending) - -### ✍️ 顾问审阅意见区 (Review Feedback) -```text -【审阅结论】:[ ] 通过 [ ] 需微调 [ ] 需重构 -【事实与数字核对】: -【人设与用词卫生】: -【CRO与转化建议】: -``` - -### 🇨🇳 中文版原文 (Chinese Markdown Twin) -````markdown ---- -title: "逐页验证一份 19 页的 PDF 报告" -description: "如何逐页验证生成的 PDF 报告:用浏览器打印同一份 HTML 作基线,逐页比对几何尺寸,把每一处偏差都压到 2pt 以内。" -pubDate: 2026-09-28 -updatedDate: 2026-09-29 -category: engineering -blueprint: sme-digital-suite -blueprintLayer: L4-managed-ops-handover -tags: [php, codeigniter, pdf, css, print, testing] -ogImage: /og/zh-verifying-a-pdf-report-page-by-page.jpg -banner: /banners/verifying-a-pdf-report-page-by-page.jpg -draft: false ---- - -你怎么知道一份生成出来的 PDF 每一页都真的正确——不只是第一页? -我维护着一个 CodeIgniter 4 应用:输入一个人的出生信息,生成一份 -中文命理/生命密码报告——十九张密排的 A4 sheet,封面、数字图表、 -方向九宫格、「一到九」的个人特征页。以前这份报告按网站原本的设计 -交付:操作员在浏览器里按 Ctrl+P,手工另存成 PDF。我把它换成了 -服务端渲染(mPDF),一夜之间,产品的质量取决于一个我看不见 -如何排版的引擎。 - -这就是这个问题可被搜索的版本:如何把一份多页 PDF 报告逐页地、 -用数字而不是肉眼,去和浏览器自己的打印输出对版?这个过程抓到 -了真实的 bug——包括一行被渲染成 2.5pt、从第二页起几乎看不见的 -页眉——最后全部十九页与浏览器基线的偏差都压进了 2pt。 - -## 为什么「看起来没问题」不算测试 - -没有人能用肉眼验证一份 19 页的报告。过去的流程是:打开 PDF, -扫一眼封面,交付。第 17 页标题漂移、页脚插图压住版权行、页眉 -小到读不出来——快速翻一遍永远发现不了,而花钱买报告的顾客 -看到的是全尺寸。 - -天真的做法会失败,是结构性的原因。服务端 PDF 渲染器不会像 -浏览器那样排你的 HTML:分页、页边距、基线、字距全都会漂移—— -而且不存在一个「看起来没问题」的测试,能让你在下周有人改了 -CSS 之后再跑一次。我第一次单趟渲染出来是 27 页,而设计预期 -是 19 页:mPDF 没有 CSS 裁剪,而 sheet 靠 `overflow: hidden` -藏住溢出。报告的 `@page` 规则更糟:mPDF 为这条规则本身 -翻了一页,19 张 sheet 炸成 12,789 页。页数都不稳定,所以 -「扫一眼第一页」不是验证——是碰运气。 - -## 基线:让浏览器去打印你的 HTML - -操作员的 Ctrl+P,就是 Chrome 用打印 CSS 打印报告自己的 HTML。 -所以参照标准不是「我觉得它应该长什么样」,而是浏览器自己的 -打印输出——在同一台机器、同一套网页字体下生成。做法:把报告 -目录用本地静态服务起起来(字体必须同源,否则 webfont 拒绝 -加载),再用 headless Chrome 打印。报告 CSS 声明了 -`@page { margin: 0 }`,所以浏览器输出是无边距满幅,再配合 -`-webkit-print-color-adjust: exact` 保住背景图——等价于操作员 -勾选「背景图形」: - -```bash -python -m http.server 8123 --bind 127.0.0.1 --directory - -"C:\Program Files\Google\Chrome\Application\chrome.exe" --headless=new --disable-gpu \ - --no-pdf-header-footer --user-data-dir=%TEMP%\chromeprofile --virtual-time-budget=30000 \ - --print-to-pdf=chrome-win.pdf "http://127.0.0.1:8123/report-local.html" -``` - -然后在 VM 上,用同一份 HTML 出服务端版本: - -```bash -php tools/pdf-render.php /tmp/report-v10.html /tmp/mpdf.pdf -``` - -## 逐页比对,而不是整体对比 - -以浏览器的 PDF 为基准,我在 Windows 这边用 -`uv run --with pymupdf` 逐页对比两份文档:页面尺寸、文字条数、 -图片包围盒(x0/y0/宽/高)、附图说明文字的 y 坐标。判定阈值: -**偏差 ≤2pt(0.7mm)算对齐**;超过 5pt 就要查根因。而且只比 -「同一元素在两份 PDF 里的差」——永远不要比绝对页数,因为页数 -一致是前提,不是测试本身。 - -## 逐页比对抓到了什么 - -下面每个 bug 都有可测量的前后数字,而且都修在库或模板里—— -不是用管道糊过去。 - -**看不见的页眉行。** 从第二页起每页顶部有一行小字,用户直接 -反馈说小到读不出来。页眉是一个 `font-size: 10px` 的 div, -里面套一张 auto 宽的 table,第一格写着 `width: 100%`。mPDF -把这解读成「表格超宽」,于是连同字号一起把整行缩到三分之一: -**2.5pt,而 Chrome 是 7.5pt**。修法(`fixPageHeaderTables()`): -给表格显式宽度 + 显式字号(px→pt),并去掉第一格的 -`width: 100%`: - -```php -$pt = round((float) $m[2] * 0.75, 2); // 10px = 7.5pt -``` - -**匹配过宽的外边距规则。** `.sheet { margin: 5mm auto }` 是为 -屏幕预览准备的(sheet 之间的阴影缝隙)。mPDF 当真了,把 -296mm 的 sheet 推到 301mm——越过 297mm 的页面——于是 sheet -在页边被切开,auto-fit 把整页缩小 3%,带白边。最直观的修法、 -在样式表末尾追加一条 `.sheet { margin: 0; }`,完全没用:mPDF -对同名选择器的两条规则只认「先出现的赢」。所以库要原地改写 -这条规则(`stripSheetMargins()`),而匹配器必须小心什么才算 -「sheet 规则」。护栏是一个负向回顾断言: - -```php -'~(?` 标签 -居中,而 mPDF 8 的 `Center` 标签处理器是空类——标签整个被 -丢弃,所有居中的标题和表格全部左对齐(「前言」实测 x=30, -Chrome 是 x=280)。`expandCenterTags()` 把 `
` 改写成 -`
`,并给居中块里的表格补上 -`align="center"`——因为父级的 `text-align` 传不到表格。 -修复后:x=281,Chrome 280。 - -**行距比浏览器高 15%。** 模板的 normalize.css 声明了 -`html { line-height: 1.15 }`;mPDF 不继承它,退回自己的字体 -度量(1.33)。这累积成每页下半部 20–40pt 的漂移——sheet 3 -甚至溢出到第二页,触发整页缩小。把 `useFixedNormalLineHeight` -设成模板自己的值之后,正文行距 15.5pt,Chrome 是 15.7。 -表格还要显式加 `td, th { padding: 1px }`——mPDF 默认的 -单元格内边距比浏览器大 2px。 - -**字体是错的。** mPDF 读不了网页用的 `.woff`,文字回退到 -自带的 Sun-ExtA;而正文栈里的「微软雅黑」在 Linux 服务器上 -根本不存在。我注册了三套真 TTF——标题手写体 MaShanZheng、 -拉丁与数字 Roboto、正文 CJK 用 wqy-microhei——并把模板的 -字体栈映射过去。一个坑:必须关掉 mPDF 的自动按脚本选字, -否则它挑「第一个支持中文的已注册字体」——整页正文都变成 -手写体。 - -**还有一个与 mPDF 无关的 bug。** 第 7 页的主插图在**两份** -PDF 里都是破图——比对显示 Chrome 和 mPDF 里是同一个破损 -占位块。模板硬编码了 `https://cdn.hoelee.com/...`,这个域名 -已经不再解析(NXDOMAIN),所以每个引擎都抓不到图。把模板 -改回应用自己的 base URL,并让 `localiseAssets()` 把任何 -host 的 `/static/` 路径都映射到本地文件,两个引擎里的插图 -都恢复了。只有并排比对才能暴露这一类 bug——单独看哪个引擎 -都「正常」。 - -## 一张 sheet,一页 - -渲染器这么设计是有意的:库把 HTML 按 `
` -切开,每张 sheet 单独渲染成一份单页文档,合并时只取第 1 页—— -物理上保证「一张 sheet = 正好一页 A4」,永远不会跨页断裂。 -mPDF 量中文宽度和 Chrome 略有不同,个别 sheet 会高出几毫米。 -与其丢内容,库按一把缩放梯子——1.0, 1.005, 1.01, 1.02, 1.03, -1.06, 1.10, 1.15, 1.22——取第一个能落进一页的比例,缩小整张 -sheet 而不是裁内容。隐藏溢出是浏览器打印做的事 -(`overflow: hidden`);一份收费产品如果 PDF 里静默丢了页内 -内容,就是无声的交付事故,所以设计选择是绝不丢内容。sheet 3 -现在需要 x1.005——0.5%,肉眼不可见——以前是 x1.03。 - -**「19/7 页」是什么意思。** 报告模板永远按固定顺序渲染十九张 -sheet;「版本」是这些 sheet 上的一个过滤器,不是第二份模板。 -完整版是 **19 页**;RM49 的精华版是**这 19 张里的 7 张**, -页码重编,每张入选的 sheet 都登记了一个文字标记——模板被 -改动/重排导致取错页时会大声失败,而不是把错误的章节交给 -顾客。两个版本走同一条管线,也用同一种方式验收: -`tools/pdf-verify.php --expect=19` 和 -`--expect=7` 双双 PASS -——页数加逐页 ink 检查(ghostscript 50dpi)证明没有任何 -空白页。 - -## 为什么收据是独立文档 - -收据不是报告裁剪出来的。它是自己的一份单页 A4 文档:真 -16mm 页边距(报告刻意做满幅无边距)、三语、单趟渲染—— -因为里面没有 sheet。它在发「已收款」邮件时才懒生成:付款 -回调必须毫秒级应答,0.3–1 秒的 PDF 渲染不属于回调。它落进 -同一个交付存储,文件名带 `-receipt` 后缀,永远不会和报告 -文件撞名,而且幂等:重试、重寄、顾客自己来拿,拿到的都是 -同一份。做它的过程从另一个方向印证了同一个论点——连单页 -文档都和浏览器不一样:`` 上的 `display: block` -不生效,左右并排的两张表把右列的数值裁出页面右边界, -合计行必须写在明细表内部,否则标签会浮在半空。 - -## 我会怎么做不一样 - -对版方法现在躺在项目笔记里,是一份文档化的流程,不是仓库 -里的脚本——这就是差距。仓库里自动化的验收工具只证明页数和 -逐页 ink,永远抓不到 2.5pt 的页眉或 15% 的行距漂移。我会把 -浏览器基线比对做成仓库验证工具链里的一个脚本:用同一份 -HTML 分别喂 Chrome 和库,diff 几何,任何超过 5pt 的偏差 -直接失败。这样,一次让打印布局悄悄回退的 CSS 改动会在构建 -时炸掉,而不是送到顾客手里。 - -我还会在写任何 mPDF 补偿代码之前就先出 Chrome 基线。 -「用操作员用的同一个引擎打印,然后测量」才是解锁点;之后 -每个修复都是机械活。还有两件诚实的遗留工作:伴侣合盘与 -家庭套餐还没跑过这套逐页对版,19 页的 PDF(背景图加嵌入 -字体约 20 MB)也还需要在交付前压缩。 - -## 结果 - -修复后的完整版验收: - -```text -$ php tools/pdf-verify.php /tmp/report-v10.html --expect=19 -out : 20,225,818 bytes, 19 pages, 10.4s, peak 188 MB -per-page ink check: all pages have content | report pages=19 -expected 19 pages => MATCH -RESULT: PASS -``` - -- **19 页 vs Chrome 基线的 19 页 — MATCH**,一张 sheet 一页, - 无跨页断裂。 -- 每一项实测偏差都**在 2pt(0.7mm)以内**:第 6 页偏了 - 213.5pt 的页脚插图、从 2.5pt 恢复到浏览器同款 7.5pt 的 - 页眉、归位的居中标题、15.5pt vs Chrome 15.7 的正文行距、 - 与网页字体一致的嵌入字体。 -- ink 检查确认**没有任何空白页**,PDF 文字可提取——封面能 - 读出真实文字,这对一份顾客要复制内容的报告很重要。 -- 精华版:**7 页 PASS**,4.99 MB,2.8 秒。 - -这就是「第一页看起来没问题」和「全部十九页都在浏览器两个点 -以内」的区别。前者是肉眼扫一遍就交付的结果;后者是拿浏览器 -自己当测试得到的结果。 - ---- - -我平时就做这类 Web 应用与打印/PDF 报告管线,也做网站设计与 -开发。如果你有一份服务器渲染的文档——报告、收据、发票—— -想在它送到顾客手里之前确认每一页都正确,跟我说说: -[WhatsApp](https://wa.me/60127972969) · [me@hoelee.com](mailto:me@hoelee.com?subject=PDF%20report%20pipeline) · -[hoelee.com](https://hoelee.com)。 -```` - -### 🇬🇧 英文版原文 (English Markdown Primary) -````markdown ---- -title: "Verifying a 19-Page PDF Report Page by Page" -description: "Verify a PDF report page by page: print the same HTML from your browser as a baseline, compare the per-page geometry, and keep every deviation under 2pt." -pubDate: 2026-09-28 -updatedDate: 2026-09-29 -category: engineering -blueprint: sme-digital-suite -blueprintLayer: L4-managed-ops-handover -tags: [php, codeigniter, pdf, css, print, testing] -ogImage: /og/verifying-a-pdf-report-page-by-page.jpg -banner: /banners/verifying-a-pdf-report-page-by-page.jpg -draft: false ---- - -How do you know a generated PDF actually looks right on every page — not just -page one? I maintain a CodeIgniter 4 app that turns birth details into a -Chinese numerology report. Until recently the operator printed it with Ctrl+P -and saved a PDF by hand. I replaced it with a server-side renderer (mPDF), and -overnight the product's quality depended on a layout engine I could not see. - -This post is the searchable version of that problem: verifying a multi-page -PDF report page by page against the browser's own print output, with numbers -instead of eyeballs. It caught real bugs — a header line rendered at 2.5pt, -effectively invisible on every page after the first — and ended with all -nineteen pages within 2pt of the browser. - -## Why "looks fine" is not a test - -Nobody can verify a 19-page report by eye. The old ritual was: open the PDF, -skim the cover, ship it. A drifting heading on page 17, a footer illustration -over the copyright line, an unreadable header — a quick flick never catches -those, and the customer who paid sees them at full size. - -Naive approaches fail structurally. A server-side PDF renderer does not lay -out HTML the way a browser does, so page breaks, margins, baselines and -spacing all drift — and there is no "looks fine" test you can re-run after -someone edits the CSS. My first single-pass render came out as 27 pages where -the design expects 19 — mPDF has no CSS clipping and the sheets hide overflow -with `overflow: hidden`. The report's own `@page` rule was worse: mPDF emitted -a page break for the rule itself, and 19 sheets exploded into 12,789 pages. -Page counts were not even stable, so eyeballing page one was not verification; -it was hope. - -## The baseline: make the browser print your HTML - -The operator's Ctrl+P is Chrome printing the report's own HTML with its print -CSS — same machine, same web fonts. So the reference is not "what I think it -should look like"; it is the browser's own print output. I served the report -directory locally (fonts must be same-origin or webfonts refuse to load) and -printed with headless Chrome. The report CSS declares `@page { margin: 0 }`, -so the output is borderless full-bleed, and `-webkit-print-color-adjust: -exact` keeps the background graphics the operator ticks: - -```bash -python -m http.server 8123 --bind 127.0.0.1 --directory - -"C:\Program Files\Google\Chrome\Application\chrome.exe" --headless=new --disable-gpu \ - --no-pdf-header-footer --user-data-dir=%TEMP%\chromeprofile --virtual-time-budget=30000 \ - --print-to-pdf=chrome-win.pdf "http://127.0.0.1:8123/report-local.html" -``` - -Then the server-side output, from the same HTML, on the VM: - -```bash -php tools/pdf-render.php /tmp/report-v10.html /tmp/mpdf.pdf -``` - -## Compare page by page, not overall - -I compared the two PDFs page by page with `uv run --with pymupdf`: page size, -text-block count, image bounding boxes (x0/y0/width/height), and caption text -y. Threshold: **within 2pt (0.7mm) counts as aligned**; over 5pt gets a root -cause. Compare the same element across the two PDFs — never absolute page -counts; page parity is the prerequisite, not the test. - -## What the comparison caught - -Each bug below had a measurable before/after and a fix in the library or -template. - -**The invisible header line.** A user reported the small header line, present -on every page after the cover, was unreadable. It is a `font-size: 10px` div -wrapping an auto-width table whose first cell declares `width: 100%` — mPDF -read that as "table too wide" and scaled the whole line, font included, to a -third of its size: **2.5pt vs Chrome's 7.5pt**. `fixPageHeaderTables()` gives -the table an explicit width and font size in points (px → pt) and drops the -offending cell: - -```php -$pt = round((float) $m[2] * 0.75, 2); // 10px = 7.5pt -``` - -**The margin rule that matched too much.** `.sheet { margin: 5mm auto }` -spaces the sheets on screen. mPDF took it literally and pushed the 296mm sheet -to 301mm — past the 297mm page — so the sheet was cut at the page edge and -auto-fit shrank the whole page by 3%, white border included. Appending `.sheet -{ margin: 0; }` does nothing — mPDF honours the *first* rule for a duplicated -selector. So the library rewrites the rule in place (`stripSheetMargins()`), -and the matcher must not over-match — the guard is a negative lookbehind: - -```php -'~(?` tag; mPDF 8's `Center` handler is an empty class, so the tag is -dropped and every centred heading and table came out left-aligned (the preface -heading measured x=30 against Chrome's x=280). `expandCenterTags()` rewrites -`
` to `
` and adds `align="center"` to -tables inside centred blocks, because parent `text-align` does not reach -tables. After the fix: x=281, Chrome 280. - -**Line spacing 15% taller than the browser's.** The template's normalize.css -declares `html { line-height: 1.15 }`; mPDF does not inherit it and fell back -to its own font metrics (1.33) — 20–40pt of drift accumulated on the lower -half of every page, and sheet 3 overflowed onto a second page, triggering a -whole-page shrink. Setting `useFixedNormalLineHeight` to the template's own -value brought body leading to 15.5pt against Chrome's 15.7. Tables needed `td, -th { padding: 1px }` — mPDF's default cell padding is 2px larger than a -browser's. - -**The fonts were wrong.** mPDF cannot read the `.woff` files the web views -use, so text fell back to its built-in Sun-ExtA, and "Microsoft YaHei" does -not exist on the Linux server. I registered the real TTFs — MaShanZheng for -headings, Roboto for Latin, wqy-microhei for CJK — and mapped the template's -stacks onto them. One trap: mPDF's automatic script-to-font selection must be -off, or it picks the first registered font that supports Chinese — the entire -body came out in handwriting. - -**And one bug that had nothing to do with mPDF.** Page 7's main illustration -was broken in *both* PDFs — the same broken stub in Chrome and mPDF. The -template hardcoded `https://cdn.hoelee.com/...`, a domain that no longer -resolves (NXDOMAIN), so every engine fetched nothing. Fixing the template to -use the app's own base URL and mapping any host's `/static/` path to local -files restored it in both engines. Only a side-by-side comparison surfaces -this — each engine alone looks "fine". - -## One sheet, one page - -The library splits the HTML into its `
` blocks and -renders each sheet as its own one-page document, importing page 1 and merging -— a physical guarantee that one sheet is exactly one A4 page, never split -across pages. mPDF measures CJK text widths slightly differently from Chrome, -so some sheets come out a few millimetres too tall. Rather than lose content, -the library re-renders the sheet at the smallest scale factor from a ladder — -1.0, 1.005, 1.01, 1.02, 1.03, 1.06, 1.10, 1.15, 1.22 — that fits, shrinking -the whole sheet instead of clipping. The browser's print clips with `overflow: -hidden`; a PDF that silently dropped in-sheet content would be a delivery -accident, so the design never drops content. Sheet 3 needs x1.005 today (0.5%, -invisible) where it used to need x1.03. - -**What "19/7 pages" means.** The report template always renders nineteen -sheets in a fixed order; the "edition" is a filter over those sheets, not a -second template. The full report is **19 pages**; the RM49 essence edition is -**7 of those 19 sheets**, renumbered, each selected sheet carrying a marker so -a rearranged template fails loudly instead of shipping the wrong chapters. -Both editions run the same pipeline and verify the same way: -`tools/pdf-verify.php --expect=19` and `--expect=7` both PASS — page count -plus a per-page ink check (ghostscript at 50dpi) proving no blank pages. - -## Why the invoice is a separate document - -The receipt is not a cut-down report. It is its own one-page A4 document: real -16mm page margins (the report is deliberately full-bleed), three languages, -and a single-pass render because there are no sheets. It is generated lazily -at email time — the payment callback must answer in milliseconds, and a 0.3–1s -render does not belong in it. It lands in the same delivery store under a -`-receipt` filename, never colliding with the report file, and it is -idempotent: retries get the same file. Even one page differed from the -browser: `display: block` on `` was ignored, side-by-side tables -clipped the right column's values off the page edge, and the total row had to -live inside the items table or its label floated in mid-air. - -## What I'd do differently - -The method lives in the project notes as a documented ritual, not a committed -script — that is the gap. The repo's automated acceptance tool proves page -count and per-page ink, which would never catch a 2.5pt header or a 15% -leading drift. I would turn the browser-baseline comparison into a script in -the repo's verification tooling: render the same HTML in Chrome and in the -library, diff the geometry, fail on any deviation over 5pt. Then a CSS change -that silently regresses the print layout fails the build instead of reaching a -customer. - -I would also have generated the Chrome baseline before writing any mPDF -compensation code — "print with the same engine the operator uses, then -measure" was the unlock. Two honest items remain: the partner and family -reports have not had this page-by-page pass yet, and the 19-page PDF (20 MB -with backgrounds and embedded fonts) still needs compression before delivery. - -## The result - -Full report verification, after the fixes: - -```text -$ php tools/pdf-verify.php /tmp/report-v10.html --expect=19 -out : 20,225,818 bytes, 19 pages, 10.4s, peak 188 MB -per-page ink check: all pages have content | report pages=19 -expected 19 pages => MATCH -RESULT: PASS -``` - -- **19 pages vs the Chrome baseline's 19 — MATCH**, one A4 page per sheet, - no splits. -- Every measured deviation is **within 2pt (0.7mm)** of the browser's print - output: footer art 213.5pt off on page 6 aligned, header restored to the - browser's 7.5pt, centred headings back, leading 15.5pt vs 15.7, embedded - fonts matching the web fonts. -- The ink check confirms **no blank pages**, and the PDF text is extractable — - the cover reads back as real text, which matters for a report customers - copy from. -- Essence edition: **7 pages PASS**, 4.99 MB, 2.8 seconds. - -That is the difference between "the first page looks fine" and "all nineteen -pages within two points of the browser": the first ships when you eyeball a -PDF; the second is what you get when the browser itself is the test. - ---- - -I build web applications and print/PDF report pipelines like this one, and I -do website design and development. If you have a document your server renders -— a report, a receipt, an invoice — and you want to be sure it is right on -every page before it reaches a customer, tell me about it: -[WhatsApp](https://wa.me/60127972969) · -[me@hoelee.com](mailto:me@hoelee.com?subject=PDF%20report%20pipeline) · -[hoelee.com](https://hoelee.com). +- **WhatsApp**: [012-7972969](https://wa.me/60127972969) +- **Email**: [me@hoelee.com](mailto:me@hoelee.com?subject=AI%20Audio%20Pipeline%20Inquiry) +- **个人网站**: [hoelee.com](https://hoelee.com) ```` --- diff --git a/docs/consultant-review/batches/batch-03.md b/docs/consultant-review/batches/batch-03.md index a10666e..f5bffd7 100644 --- a/docs/consultant-review/batches/batch-03.md +++ b/docs/consultant-review/batches/batch-03.md @@ -2,9 +2,9 @@ > **批次编号**:`Batch-03` > **包含文章数**:1 篇 -> **本批次总字符量**:128,681 字(严格限制 < 200,000 字) +> **本批次总字符量**:131,667 字(严格限制 < 200,000 字) > **模式**:BILINGUAL -> **生成时间**:2026-10-27 05:23:38 +> **生成时间**:2026-10-27 05:36:05 # 外部顾问审阅前置须知与工程背景规范 @@ -39,7 +39,7 @@ ## 本批次文章目录索引 -1. [tutorials] **用 YuE2 BF16 生成完整长歌:突破时长限制与音频硬截断** (`generating-full-length-songs-with-yue2-comfyui-zh`) — ~128,681 字符 +1. [tutorials] **用 YuE2 BF16 生成完整长歌:突破时长限制与音频硬截断** (`generating-full-length-songs-with-yue2-comfyui-en`) — ~131,667 字符 --- @@ -58,32 +58,38 @@ 【CRO与转化建议】: ``` -### 🇨🇳 中文版原文 (Chinese Markdown Twin) +### 🇬🇧 英文版原文 (English Markdown Primary) ````markdown --- -title: "用 YuE2 BF16 生成完整长歌:突破时长限制与音频硬截断" -description: "AI 音乐工具常在歌词中途戛然而止。本文详解如何利用 YuE2 BF16、295秒余量预留公式与本地 ComfyUI,成功批量跑出 60 首自然收尾的完整长歌全集。" +title: "Generating Full-Length Songs with YuE2 BF16: Overcoming Duration Limits and Audio Truncation" +description: "AI music tools cut off complex tracks mid-sentence. Here is how I solved the 240s duration wall using YuE2 BF16, 295s headroom budgeting, and a 60-song pipeline." pubDate: 2026-10-26 category: tutorials blueprint: ai-agent-automation blueprintLayer: L4-aigc-media-pipelines stream: creative-lab tags: ["ai-music", "yue2", "comfyui", "audio-engineering", "subtitles", "python"] -ogImage: "/og/zh-generating-full-length-songs-with-yue2-comfyui.jpg" +ogImage: "/og/generating-full-length-songs-with-yue2-comfyui.jpg" banner: "/banners/generating-full-length-songs-with-yue2-comfyui.jpg" draft: false --- -在当前几乎所有的 AI 音乐生成方案中,最普遍且让人抓狂的缺陷莫过于: -**歌曲唱到一半,突然被硬生生切断**。 +Every generative music engine I have benchmarked suffers from the same fatal flaw: +**the sudden, unceremonious cutoff**. -不管是各大商用平台还是开源权重,受限于固定的声学 Token 训练窗口,只要我们尝试塞入完整的 -叙事性歌词结构——即标准的 9 阶段曲式 `[Intro] -> [Verse 1] -> [Chorus] -> [Interlude] -> [Verse 2] -> [Chorus] -> [Bridge] -> [Chorus] -> [Outro]`——生成引擎就极易在进入副歌高潮或尾奏时撞上时长硬天花板,在歌手吐字的中途突兀静音。 +Whether you use cloud APIs or local weights, models trained on fixed acoustic token +windows struggle the moment you give them a complete, narrative song structure. A +9-part progression—`[Intro] -> [Verse 1] -> [Chorus] -> [Interlude] -> [Verse 2] -> [Chorus] -> [Bridge] -> [Chorus] -> [Outro]`—inevitably +collides with the hard ceiling of the generation buffer. The vocalist is mid-word in +the climax, and the audio abruptly flatlines into digital silence. -在单张 RTX 3060 12GB 显卡上经历数百次实跑与测试后,我摸索出了一套能够稳定产出 -**4 分钟以上、具备自然淡出收尾、绝不截断**的完整长歌工程链路。 +After iterating across hundreds of takes on an RTX 3060 12GB rig, I built an end-to-end +pipeline that generates genuine 4-minute full-length songs with natural melodic resolution +and zero mid-phrase amputations. -本文将拆解导致音频硬截断的底层成因、295 秒余量预留公式,并完整交付 **60 首高质量长歌全集**(每首均配套 1024×1024 独立 WebP 封面与逐行同步滚动歌词)。 +Below is the breakdown of why standard generation fails, the 295-second headroom formula +that fixes it, and the full collection of **60 completed full-length songs** (with +individual 1024×1024 cover art and synchronized scrolling lyrics). --- @@ -129,22 +135,22 @@ draft: false
- - - - - - - + + + + + + +
- + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + +
+
+ Mist Over Skellig album cover
-
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
+
Mist Over Skellig · Celtic folk · 18 lines · 193 s.srt
+ +
-
+
-## The Problem: The 240-Second Wall +
+
+ Willow by the Silver Stream album cover +
+
+
Willow by the Silver Stream · Celtic folk · 19 lines · 197 s.srt
+ + +
+
-In earlier batches, songs generated via standard ComfyUI text-to-music nodes frequently -exhibited one of two failure modes: +
+
+ Clockwork Starlight album cover +
+
+
Clockwork Starlight · Japanese healing · 21 lines · 160 s.srt
+ + +
+
-1. **The Chopped Outro**: Setting `max_duration = 240.0s` while feeding a full lyric - sheet resulted in the track hitting the final timestamp while the acoustic model was - still executing the chorus. All 12 initial test runs ended at exactly `240.000s`—an - unusable hard cliff. -2. **The Panic Accelerando**: If you prompt the model to fit too many lines into a tight - window, the auto-regressive acoustic token predictor compresses the vocal pacing, - turning an emotional Celtic ballad into an unnatural rap cadence. +
+
+ Waltz of the Dandelion album cover +
+
+
Waltz of the Dandelion · Japanese healing · 18 lines · 179 s.srt
+ + +
+
-The fundamental mistake was treating `max_duration` as the *desired song length*. -In generative auto-regressive audio, **max duration is not your target length; it is your -acoustic safety wall.** +
+
+ Whale in the Milky Way album cover +
+
+
Whale in the Milky Way · Japanese healing · 20 lines · 164 s.srt
+ + +
+
-## The Three Architecture Fixes +
+
+ Puddle Jumping Parade album cover +
+
+
Puddle Jumping Parade · Japanese cute · 21 lines · 105 s.srt
+ + +
+
-### 1. The 295.0s Headroom Formula +
+
+ Strawberry Cloud Bakery album cover +
+
+
Strawberry Cloud Bakery · Japanese cute · 19 lines · 100 s.srt
+ + +
+
-To produce songs that naturally conclude with a 4-minute runtime, you must structure -the lyric density for **210s to 260s**, but provision the generator with **295.0s** -of ceiling: +
+
+ Captain Marshmallow album cover +
+
+
Captain Marshmallow · Japanese cute · 22 lines · 95 s.srt
+ + +
+
-$$\text{Headroom Buffer} = \text{Max Duration (295s)} - \text{Expected Lyric Duration (220s)} \approx 75\text{s}$$ +
+
+ Lanterns on the Lotus Pond album cover +
+
+
Lanterns on the Lotus Pond · Guofeng fusion · 19 lines · 131 s.srt
+ + +
+
-This ~75-second buffer provides the acoustic model with the token budget required to: -- Complete the final verse without panic compression; -- Introduce a dedicated, melodic instrumental outro; -- Execute natural acoustic decay, room reverb dissipation, and final silence. +
+
+ Wings of the Kunpeng album cover +
+
+
Wings of the Kunpeng · Guofeng fusion · 19 lines · 213 s.srt
+ + +
+
-### 2. Automated Hard-Cut Quarantine Gate +
+
+ The Puppeteer's Thread album cover +
+
+
The Puppeteer's Thread · Guofeng fusion · 19 lines · 221 s.srt
+ + +
+
-Any take whose generated length measures within 0.1s of the hard ceiling (`>= 294.9s`) -is treated as an unresolved truncation failure and automatically quarantined. Only takes -that resolve naturally before the ceiling (`duration < 294.0s`) are accepted into the -production archive. Across our 60-song run, all 60 finalized masters ended between -**129.9s and 285.6s** with zero abrupt cutoffs. +
+
+ Paper Airplane in September album cover +
+
+
Paper Airplane in September · Acoustic folk · 18 lines · 128 s.srt
+ + +
+
-### 3. VRAM Stabilization on a 12GB RTX 3060 +
+
+ Porridge and Evening Smoke album cover +
+
+
Porridge and Evening Smoke · Acoustic folk · 17 lines · 144 s.srt
+ + +
+
-Running YuE2-3B in BF16 on consumer hardware requires tight memory lifecycle management: +
+
+ Green Train to the Sea album cover +
+
+
Green Train to the Sea · Acoustic folk · 17 lines · 108 s.srt
+ + +
+
-```bash -# Launch ComfyUI standalone with minimal VRAM overhead -cd /d/app/ComfyUI -./python_embeded/python.exe -s ComfyUI/main.py \ - --windows-standalone-build \ - --port 8188 \ - --disable-pinned-memory \ - --disable-nvml-pressure \ - --cache-classic +
+
+ Sanctuary of the Dawn album cover +
+
+
Sanctuary of the Dawn · Choral · 19 lines · 182 s.srt
+ + +
+
+ +
+
+ Look Upon Your Path album cover +
+
+
Look Upon Your Path · Choral · 17 lines · 122 s.srt
+ + +
+
+ +
+
+ Bells of the Winter Star album cover +
+
+
Bells of the Winter Star · Choral · 18 lines · 177 s.srt
+ + +
+
+ +```` + +### 🇬🇧 英文版原文 (English Markdown Primary) +````markdown +--- +title: "Generating Subtitles for Songs That Ship Without Timestamps" +description: "AI music tools give you audio and nothing else — no line timings, no karaoke. Here is the pipeline I use to produce LRC and SRT subtitles for 18 songs, why plain transcription fails, and the two bugs that made the lyrics box scroll to the bottom." +pubDate: 2026-10-20 +category: tutorials +blueprint: ai-agent-automation +blueprintLayer: L3-local-gpu-inference +stream: creative-lab +tags: ["subtitles", "lrc", "srt", "whisper", "forced-alignment", "ai-music", "python"] +ogImage: "/og/generating-subtitles-for-songs-without-timestamps.jpg" +banner: "/banners/generating-subtitles-for-songs-without-timestamps.jpg" +draft: false +--- +Every AI music tool I have tried returns the same shape of output: an audio +file. Not one of them tells you *when* each line is sung. That is fine if you +only want to listen, and useless the moment you want lyrics on screen — a +karaoke line, a video caption, a scrolling block under a player. + +Timing is the whole problem. The audio is the easy part. + +This is the pipeline I ended up with, running on 18 English songs generated +locally, plus the two places it broke in the browser. Everything below is +reproducible: the only thing it needs from you is the **lyrics you already +wrote**. + +## What you actually need + +Two subtitle formats, one shared data structure: + +| Format | Used by | Shape | +|---|---|---| +| **LRC** | music players, karaoke blocks, `