post(batch-07): apply consultant micro-fixes to A/B/C (EN+ZH)
Deploy / build (push) Successful in 20s
Deploy / build (push) Successful in 20s
A: drop the arrow artefact in the log block, name the upstream SSO gate as the party that writes 499, add I/O headroom to the latency lesson, make the 53-line snippet self-contained, correct summary line counts (53/18 not 52/22). B: remove second person from the closing paragraph, <details class=tech-deep-dive> + summary with deliverable and 28 lines, ZH jumpLinks -> explicit English anchors. C: three->four failure modes, remove 4 second-person spots, details class + 61/63 lines, ZH anchors, and an explicit <a id> on the EN H2 whose backticked heading slugified to 'the-naive--map-0-...' (double hyphen).
This commit is contained in:
1 parent
a503f1cdc8
commit
cf2b77d0dd
6 files changed
+45
-43
No files matched your search
@@ -99,8 +99,10 @@ ambiguous; one of them actively misleads.
|
||||
| Application log (uvicorn) | **No entry at all** for the path during the incident window | Proves the request never reached the app — the proxy aborted first |
|
||||
| In-container probe | Same call direct to the app: **17.7 s / 98.5 s / 35.8 s** on three consecutive attempts | Confirms the slowness is real and inside the audit, not a network artefact |
|
||||
|
||||
`499` is the status nginx writes when the **client** closes the connection before the response is
|
||||
sent. It is not an error the server acknowledges, it is not a 5xx, and FastAPI's access log never
|
||||
`499` is the status portal nginx writes when its **caller** — here, the upstream SSO gate whose own
|
||||
timer expired well before the portal's 120-second limit — closes the connection before the response
|
||||
is sent, while simultaneously serving its own HTML timeout page to the browser. It is not an error
|
||||
the server acknowledges, it is not a 5xx, and FastAPI's access log never
|
||||
sees it — so the panel could stay broken for weeks while every uptime check, every error-rate
|
||||
dashboard and every "is the service healthy?" question answered green. The status that describes
|
||||
the failure is written by the hop that gave up, about a request the application never received.
|
||||
@@ -125,7 +127,7 @@ automation counts refreshed in 76.1s
|
||||
automation counts refreshed in 82.3s
|
||||
automation counts refreshed in 93.5s
|
||||
automation counts refreshed in 124.2s
|
||||
automation counts refreshed in 93.5s → 165.1s
|
||||
automation counts refreshed in 165.1s
|
||||
automation counts refreshed in 633.6s
|
||||
automation counts refreshed in 1282.5s
|
||||
automation counts refreshed in 2507.2s
|
||||
@@ -138,9 +140,9 @@ failure that arrives at 120 s into a failure that arrives at 300 s, five minutes
|
||||
longer window of consumed capacity.
|
||||
|
||||
The lesson is the one I keep relearning at the edge: **an endpoint's latency is not a property of
|
||||
its code, it is a property of the CPU it is allowed to borrow.** A one-core cgroup limit turns an
|
||||
O(n) file audit into an unbounded operation whose runtime is dictated by whatever else is running
|
||||
beside it.
|
||||
its code, it is a property of the CPU and I/O headroom it is allowed to borrow.** A one-core cgroup
|
||||
limit alongside heavy host workloads turns an O(n) file audit into an unbounded operation whose
|
||||
runtime is dictated by whatever else is running beside it.
|
||||
|
||||
---
|
||||
|
||||
@@ -155,7 +157,7 @@ and a flag, so a burst of polls cannot start a burst of audits. The cache is war
|
||||
inside the app's lifespan hook, so the first human to open the panel does not pay for it.
|
||||
|
||||
<details class="tech-deep-dive">
|
||||
<summary>🔍 View the stale-while-revalidate implementation — cache, single-flight refresh thread, startup warm (52 lines)</summary>
|
||||
<summary>🔍 View the stale-while-revalidate implementation — cache, single-flight refresh thread, startup warm (53 lines)</summary>
|
||||
|
||||
```python
|
||||
_AUTOMATION_TTL_SEC = float(os.environ.get("AUTOMATION_COUNTS_TTL_SEC", "300") or 300)
|
||||
@@ -177,7 +179,7 @@ def automation_get(refresh: bool = False):
|
||||
"flags": flags,
|
||||
"counts": (cached["data"] or {}).get("counts")
|
||||
or {"notes": 0, "missing_cover": 0, "tts_stale": 0},
|
||||
"refreshing": _automation_refreshing() or not fresh,
|
||||
"refreshing": _automation_refreshing_flag[0] or not fresh,
|
||||
"cached_at": (datetime.fromtimestamp(cached["at"]).isoformat(timespec="seconds")
|
||||
if cached["at"] else None),
|
||||
}
|
||||
@@ -225,7 +227,7 @@ inferred "malformed JSON" from "HTML error page" and discarded both the status c
|
||||
that would have named the failing hop.
|
||||
|
||||
<details class="tech-deep-dive">
|
||||
<summary>🔍 View the text-first response reader and the retry contract (22 lines)</summary>
|
||||
<summary>🔍 View the text-first response reader and the retry contract (18 lines)</summary>
|
||||
|
||||
```javascript
|
||||
function autoReadJson(res) {
|
||||
|
||||
@@ -120,8 +120,8 @@ If an encode used >= 97% of the allowable source bitrate (`obps >= vbps * SIZE_C
|
||||
the file was capped. Retrying with a lower CRF would mathematically yield zero benefit. The pipeline
|
||||
now detects this condition and cancels the retry immediately.
|
||||
|
||||
<details>
|
||||
<summary>View pipeline quality-gate logic (Python excerpt)</summary>
|
||||
<details class="tech-deep-dive">
|
||||
<summary>🔍 View the two-tier quality gate and cap-bound bypass logic — evaluate_quality_gate() (28 lines)</summary>
|
||||
|
||||
```python
|
||||
SIZE_CEIL = 1.0 # Output video bitrate must not exceed the source's
|
||||
@@ -165,4 +165,4 @@ After deploying this change to the production nodes:
|
||||
* The 106-hour wasted compute overhead on boundary files dropped to **zero**.
|
||||
* Pipeline throughput increased by **31%** across the archival queue because the supervisor stopped running provably inert second passes.
|
||||
|
||||
When building automated transcoding pipelines, never evaluate quality metrics in isolation from your rate-control constraints. When CRF meets `-maxrate`, the ceiling always wins.
|
||||
In an automated transcoding pipeline, quality metrics can never be evaluated in isolation from the rate-control ceiling. When CRF meets `-maxrate`, the ceiling always wins.
|
||||
@@ -27,7 +27,7 @@ summary:
|
||||
dynamicCta: true
|
||||
---
|
||||
|
||||
## The naive `-map 0` that breaks in production
|
||||
## The naive `-map 0` that breaks in production <a id="the-naive-map-0-that-breaks-in-production"></a>
|
||||
|
||||
Every developer automating FFmpeg reaches for the same shortcut when preserving secondary
|
||||
tracks during a transcode:
|
||||
@@ -39,7 +39,7 @@ ffmpeg -i source.mkv -map 0 -c:v libx265 -c:a copy output.mp4
|
||||
The logic seems bulletproof: map all streams (`-map 0`), transcode the video stream to save bits,
|
||||
and copy every accompanying audio track, subtitle, and chapter without losing data.
|
||||
|
||||
In production across a large, diverse video archive, this command fails catastrophically in three
|
||||
In production across a large, diverse video archive, this command fails catastrophically in four
|
||||
distinct ways:
|
||||
|
||||
1. **It hard-crashes on Matroska output**: If an MP4 source contains Apple QuickTime text chapters (`bin_data`) or iPhone timecode tracks (`tmcd`), FFmpeg aborts instantly:
|
||||
@@ -50,8 +50,8 @@ distinct ways:
|
||||
3. **It silently strips font attachments**: When remuxing into MP4, attached `.ttf` or `.otf` styling fonts cause fatal errors: `Could not find tag for codec ttf`.
|
||||
4. **It silently wipes audio track names on iOS**: Apple devices read track labels from `udta/name`, whereas VLC, ExoPlayer, and ffprobe read from `hdlr/handler_name`. FFmpeg's demuxer only reads `handler_name` back, so a secondary remux silently drops the Apple tag.
|
||||
|
||||
To build an automated, crash-proof media pipeline, you cannot treat containers as transparent
|
||||
envelopes. You need an explicit, policy-driven stream planner.
|
||||
Building an automated, crash-proof media pipeline means never treating containers as transparent
|
||||
envelopes. It requires an explicit, policy-driven stream planner.
|
||||
|
||||
---
|
||||
|
||||
@@ -90,8 +90,8 @@ The design follows three rules:
|
||||
3. **Dual metadata synchronization**: Whenever a track has a human-readable title, write both keys:
|
||||
`-metadata:s:a:0 title="Commentary"` and `-metadata:s:a:0 handler_name="Commentary"`.
|
||||
|
||||
<details>
|
||||
<summary>View tracks.py pure planning implementation (Python excerpt)</summary>
|
||||
<details class="tech-deep-dive">
|
||||
<summary>🔍 View the deterministic stream planner — tracks.py plan() and build_ffmpeg_args() (61 lines)</summary>
|
||||
|
||||
```python
|
||||
TEXT_SUB = {"subrip", "srt", "ass", "ssa", "webvtt", "mov_text", "text"}
|
||||
@@ -163,8 +163,8 @@ def build_ffmpeg_args(plan):
|
||||
|
||||
## Verifying the output at packet level
|
||||
|
||||
A plan is only as good as its verification gate. After remuxing, how do you prove that an audio
|
||||
stream copied with `-c:a copy` was genuinely untouched?
|
||||
A plan is only as good as its verification gate. After remuxing, the pipeline still has to prove
|
||||
that an audio stream copied with `-c:a copy` was genuinely untouched.
|
||||
|
||||
Standard duration checks are insufficient because timestamps can drift across containers. My validation gate
|
||||
uses **packet payload MD5 hashing**:
|
||||
|
||||
@@ -81,7 +81,7 @@ portal 的 `/api/` 位置在现行配置里是 `proxy_read_timeout 120s`,而
|
||||
| 应用日志(uvicorn) | 故障窗口内该路径**一条记录都没有** | 证明请求根本没到应用,是代理先放弃的 |
|
||||
| 容器内直连打点 | 同一调用直连应用:连续三次 **17.7 秒 / 98.5 秒 / 35.8 秒** | 确认慢是真的、且慢在审计内部,不是网络抖动 |
|
||||
|
||||
`499` 是 nginx 在**客户端**先断开连接时写下的状态码。它不是服务端承认的错误,不是 5xx,FastAPI 的访问日志也永远看不到它——所以这张卡片可以坏上几个星期,而所有存活探测、错误率看板、「服务健康吗」的问答统统是绿的。真正描述这次故障的状态码,由放弃等待的那一层写下,指向的是一个应用从未收到的请求。
|
||||
`499` 是 portal nginx 在其**上游调用方**主动掐断连接时写下的状态码——这里是排在它前面、计时器比 portal 的 120 秒更早到期的 SSO 门禁,它转身把自己的 HTML 超时页吐给了浏览器,而 portal nginx 只看到调用方提前离场。它不是服务端承认的错误,不是 5xx,FastAPI 的访问日志也永远看不到它——所以这张卡片可以坏上几个星期,而所有存活探测、错误率看板、「服务健康吗」的问答统统是绿的。真正描述这次故障的状态码,由放弃等待的那一层写下,指向的是一个应用从未收到的请求。
|
||||
|
||||
而最响的那条证据恰好在误导:`SyntaxError`。它对文档的描述是准确的,对原因的判断是错的。
|
||||
|
||||
@@ -106,7 +106,7 @@ automation counts refreshed in 2507.2s
|
||||
|
||||
同一份代码、同一个 341 篇的库,一个数量级的跨度,最坏值比空闲机上量到的数字**慢 184 倍**。这一条就足以否定那个「朴素修法」:把反代超时从 120 秒抬到 300 秒,解决不了一个 2507 秒的计算,只是把 120 秒失败改成 300 秒失败,晚五分钟、多占五分钟容量。
|
||||
|
||||
这就是我在边缘层反复吃到的教训:**端点延迟不是代码的属性,而是它被允许借用多少 CPU 的属性。** 单核 cgroup 限流会把一次 O(n) 文件审计变成耗时由邻居决定的无界操作。
|
||||
这就是我在边缘层反复吃到的教训:**端点延迟不是代码的属性,而是它被允许借用多少 CPU 与 I/O 余量的属性。** 单核 cgroup 限流叠加宿主机的重载任务,会把一次 O(n) 文件审计变成耗时由邻居决定的无界操作。
|
||||
|
||||
---
|
||||
|
||||
@@ -117,7 +117,7 @@ automation counts refreshed in 2507.2s
|
||||
**一、读路径绝不计算。** 计数放进 TTL 缓存(300 秒,`AUTOMATION_COUNTS_TTL_SEC` 可调)。请求要么拿到新鲜缓存,要么立刻拿到上一次的值并带 `refreshing: true`,同时由一个后台 daemon 线程重算。单飞用一把锁加一个标记守住,成串的轮询不会变成成串的审计。缓存还会在应用 lifespan 里预热一次,第一个打开面板的人不用替所有人付这笔账。
|
||||
|
||||
<details class="tech-deep-dive">
|
||||
<summary>🔍 查看 stale-while-revalidate 实现 —— 缓存、单飞刷新线程与启动预热(52 行)</summary>
|
||||
<summary>🔍 查看 stale-while-revalidate 实现 —— 缓存、单飞刷新线程与启动预热(53 行)</summary>
|
||||
|
||||
```python
|
||||
_AUTOMATION_TTL_SEC = float(os.environ.get("AUTOMATION_COUNTS_TTL_SEC", "300") or 300)
|
||||
@@ -139,7 +139,7 @@ def automation_get(refresh: bool = False):
|
||||
"flags": flags,
|
||||
"counts": (cached["data"] or {}).get("counts")
|
||||
or {"notes": 0, "missing_cover": 0, "tts_stale": 0},
|
||||
"refreshing": _automation_refreshing() or not fresh,
|
||||
"refreshing": _automation_refreshing_flag[0] or not fresh,
|
||||
"cached_at": (datetime.fromtimestamp(cached["at"]).isoformat(timespec="seconds")
|
||||
if cached["at"] else None),
|
||||
}
|
||||
@@ -182,7 +182,7 @@ def _start_automation_refresh() -> None:
|
||||
**三、客户端绝不盲解析。** 真正躺在浏览器侧的缺陷是 `res.json()`:它把「收到了 HTML 错误页」推断成「JSON 格式错误」,顺手丢掉了状态码和本该指出故障层级的那段正文。
|
||||
|
||||
<details class="tech-deep-dive">
|
||||
<summary>🔍 查看「先读文本」的响应读取器与重试契约(22 行)</summary>
|
||||
<summary>🔍 查看「先读文本」的响应读取器与重试契约(18 行)</summary>
|
||||
|
||||
```javascript
|
||||
function autoReadJson(res) {
|
||||
|
||||
@@ -21,13 +21,13 @@ summary:
|
||||
detail: "彻底旁路封顶文件的无效重试,将临界视频的重试开销从 106 小时直接清零。"
|
||||
stack: ["FFmpeg", "libx265", "SSIM", "Python", "Linux 生产节点"]
|
||||
jumpLinks:
|
||||
business: "#消耗了-106-小时却毫无进展的转码批次"
|
||||
architecture: "#架构重构二档门禁与封顶自适应旁路"
|
||||
result: "#交付结果"
|
||||
business: "#the-batch-that-burned-106-hours-doing-nothing"
|
||||
architecture: "#the-architectural-solution-two-tier-gates-and-cap-awareness"
|
||||
result: "#the-result"
|
||||
dynamicCta: true
|
||||
---
|
||||
|
||||
## 消耗了 106 小时却毫无进展的转码批次
|
||||
## 消耗了 106 小时却毫无进展的转码批次 <a id="the-batch-that-burned-106-hours-doing-nothing"></a>
|
||||
|
||||
在我所维护的私有高吞吐媒体转码基础设施中,自动化入库队列设定了一条铁律:**转码输出文件体积绝对不可超过原文件**(`SIZE_CEIL = 1.0`)。
|
||||
|
||||
@@ -85,7 +85,7 @@ dynamicCta: true
|
||||
|
||||
---
|
||||
|
||||
## 架构重构:二档门禁与封顶自适应旁路
|
||||
## 架构重构:二档门禁与封顶自适应旁路 <a id="the-architectural-solution-two-tier-gates-and-cap-awareness"></a>
|
||||
|
||||
为了根除算力浪费,我对调度器的裁决引擎进行了三重重构:
|
||||
|
||||
@@ -99,8 +99,8 @@ dynamicCta: true
|
||||
### 2. 封顶阻断判定(`CAP_BOUND_SHARE`)
|
||||
如果编码输出已经消耗了源视频码率的 97%(`obps >= vbps * SIZE_CEIL * 0.97`),说明该视频已经撞死在码率墙上。此时调低 CRF 重试在数学上注定无效。调度器检测到此状态后,直接取消重试并记录原因。
|
||||
|
||||
<details>
|
||||
<summary>点击查看质量门禁自适应裁决核心代码(Python 关键片段)</summary>
|
||||
<details class="tech-deep-dive">
|
||||
<summary>🔍 查看二档质量门禁与封顶阻断裁决实现 —— evaluate_quality_gate()(28 行)</summary>
|
||||
|
||||
```python
|
||||
SIZE_CEIL = 1.0 # 输出码率严禁超过源视频
|
||||
@@ -137,7 +137,7 @@ def evaluate_quality_gate(info, src, mean_ssim, bad_share, vbps, obps):
|
||||
|
||||
---
|
||||
|
||||
## 交付结果
|
||||
## 交付结果 <a id="the-result"></a>
|
||||
|
||||
将新策略部署至生产节点后:
|
||||
* 此前在 `crf 19` 与 `crf 14` 之间死循环报错 `FAIL quality 0.99192` 的测试文件,在首轮直接以 `PASS_SECOND_TIER` 顺利入库(体积为原片的 87.5%,零重试,单次编码用时 120 秒)。
|
||||
|
||||
@@ -21,13 +21,13 @@ summary:
|
||||
detail: "音轨与字幕名称同时双写至 udta/name(Apple 规范)与 hdlr(ffprobe/VLC 规范),解决二次混流导致的标签丢失。"
|
||||
stack: ["FFmpeg", "ffprobe", "Python", "Matroska/MP4 规范"]
|
||||
jumpLinks:
|
||||
business: "#生产环境中频繁暴雷的朴素--map-0"
|
||||
architecture: "#跨容器兼容性矩阵真相"
|
||||
result: "#交付结果"
|
||||
business: "#the-naive-map-0-that-breaks-in-production"
|
||||
architecture: "#the-hidden-matrix-of-container-behavior"
|
||||
result: "#the-result"
|
||||
dynamicCta: true
|
||||
---
|
||||
|
||||
## 生产环境中频繁暴雷的朴素 `-map 0`
|
||||
## 生产环境中频繁暴雷的朴素 `-map 0` <a id="the-naive-map-0-that-breaks-in-production"></a>
|
||||
|
||||
任何使用自动化脚本调用 FFmpeg 的开发者,在需要保留原片多音轨与字幕时,都会自然写下这条看似稳妥的命令:
|
||||
|
||||
@@ -37,7 +37,7 @@ ffmpeg -i source.mkv -map 0 -c:v libx265 -c:a copy output.mp4
|
||||
|
||||
从直觉上看,这个逻辑无懈可击:映射所有流(`-map 0`),转码视频层以节约体积,同时将原片伴随的所有多语言音轨、字幕以及章节完整复制(`copy`)。
|
||||
|
||||
然而在跨越数万个多源视频文件的私有转码流水线中,这条命令引发了三类严重的生产故障:
|
||||
然而在跨越数万个多源视频文件的私有转码流水线中,这条命令引发了四类严重的生产故障:
|
||||
|
||||
1. **目标为 Matroska(MKV)时直接崩溃**:如果源 MP4 文件中包含 Apple QuickTime 章节文本轨(`data/bin_data/text`)或手机录制带入的时间码轨(`tmcd`),FFmpeg 的 Matroska 混流器会立即报错中断退出:
|
||||
`Only audio, video, and subtitles are supported for Matroska.`
|
||||
@@ -47,11 +47,11 @@ ffmpeg -i source.mkv -map 0 -c:v libx265 -c:a copy output.mp4
|
||||
3. **字体附件直接炸裂**:尝试将带有嵌入 `.ttf` / `.otf` 字体的源文件 remux 到 MP4 时,混流器会无情报错:`Could not find tag for codec ttf`。
|
||||
4. **iOS 设备上的音轨名称神秘消失**:Apple 设备从 MP4 的 `udta/name` 字段读取音轨显示名,而 VLC、ExoPlayer 与 ffprobe 则读取 `hdlr/handler_name`。FFmpeg 的 demuxer 在回读 MP4 时仅提取 `handler_name`,导致文件在二次混流后,Apple 端的音轨名称直接被静默抹除。
|
||||
|
||||
在企业级媒体工程中,容器绝对不能被简单视作无差别的“透明包装箱”。你必须建立基于策略的显式轨道规划体系。
|
||||
在企业级媒体工程中,容器绝对不能被简单视作无差别的“透明包装箱”,工程上必须建立基于策略的显式轨道规划体系。
|
||||
|
||||
---
|
||||
|
||||
## 跨容器兼容性矩阵真相
|
||||
## 跨容器兼容性矩阵真相 <a id="the-hidden-matrix-of-container-behavior"></a>
|
||||
|
||||
通过搭建受控的自动化测试用例,并在包级别计算解复用数据包的 MD5 签名,我整理出了 MP4 与 Matroska(`.mkv`)在多轨处理上的边界差异:
|
||||
|
||||
@@ -82,8 +82,8 @@ ffmpeg -i source.mkv -map 0 -c:v libx265 -c:a copy output.mp4
|
||||
3. **跨平台双标签元数据同步**:当音轨或字幕存在命名时,同时注入两组元数据:
|
||||
`-metadata:s:a:0 title="解说轨"` 配合 `-metadata:s:a:0 handler_name="解说轨"`。
|
||||
|
||||
<details>
|
||||
<summary>点击查看 tracks.py 核心规划实现(Python 关键片段)</summary>
|
||||
<details class="tech-deep-dive">
|
||||
<summary>🔍 查看确定性轨道规划器实现 —— tracks.py plan() 与 build_ffmpeg_args()(63 行)</summary>
|
||||
|
||||
```python
|
||||
TEXT_SUB = {"subrip", "srt", "ass", "ssa", "webvtt", "mov_text", "text"}
|
||||
@@ -177,7 +177,7 @@ def stream_signature(path, stream_selector):
|
||||
|
||||
---
|
||||
|
||||
## 交付结果
|
||||
## 交付结果 <a id="the-result"></a>
|
||||
|
||||
将显式轨道规划器投入生产环境后:
|
||||
* **混流层致命错误降为 0**:在处理跨越不同来源的 14,000+ 视频文件过程中,再未发生过因未知轨道或不支持编码导致的容器崩溃。
|
||||
|
||||
Reference in new issue
Block a user