post(devops): every player stuttered on the same video — the file was blameless (EN+ZH) + og/banner

The converted HEVC stuttered in VLC, Chrome and on an iPad (offline) while the source played fine. Container timing was perfect (5818/5818 frame intervals at exactly 0.040000 s, no dup/backward PTS, ctts not flat) and a full decode printed nothing. Cross-decoder SSIM found the real defect: NVDEC's legacy av1_cuvid wrapper returns frames from the wrong timestamps - 106 of 5819 frames, SSIM down to 0.330, deterministically, roughly every 30 frames - and the HEVC output was a faithful encode of those wrong pictures (min 0.991 against the NVDEC decode, min 0.330 against a software decode). Fix is one line (-hwaccel cuda -hwaccel_output_format cuda -c:v av1, verified 1.000000 over 1500 frames; vp9_cuvid unaffected). Adds the content gate (structure AND pictures, defect-shaped thresholds) to the pipeline.
2026-10-08 04:52:04 +08:00
parent c2f8452372
commit e6476e1d15
6 changed files with 347 additions and 0 deletions
@@ -1149,6 +1149,24 @@ BANNERS['putting-a-front-door-on-an-app-you-cant-modify'] = {
  ],
 };

+BANNERS['every-player-stuttered-the-file-was-fine'] = {
+  titlebar: "root@media - the file was blameless",
+  lines: [
+    { t: 'prompt', text: "$" }, { t: 'cmd', text: "tsconv --file 5YkGHtaduGU" },
+    { t: 'dim', text: "container: 5818/5818 frames @ 0.040000 s" }, { t: 'dim', text: "decode: rc=0, zero errors printed" },
+    { t: 'err', text: "pictures: 106 frames wrong (SSIM 0.33)" }, { t: 'prompt', text: "$" },
+    { t: 'cmd', text: "-hwaccel cuda -c:v av1" }, { t: 'ok', text: "0 wrong frames  SSIM 1.000000" },
+    { t: 'hl', text: "gate: structure AND content, then replace" }, { t: 'dim', text: "65/65 affected files re-encoded" },
+  ],
+  flow: [
+    { n: '1', label: "decode" },
+    { n: '2', label: "encode" },
+    { n: '3', label: "verify" },
+    { n: '4', label: "content" },
+    { n: '5', label: "replace" },
+  ],
+};
+
 // ---------- read frontmatter ----------
 const postPath = join(ROOT, 'src', 'content', 'posts', `${slug}.md`);
 let category = 'devops';
@@ -349,6 +349,12 @@ TERMINALS['putting-a-front-door-on-an-app-you-cant-modify'] = `
    <div class="line"><span class="prompt">$</span><span class="cmd">docker compose up -d   (gateway takes the public port)</span><span class="fix">→ front page 200 ✓</span></div>
    <div class="line"><span class="prompt">$</span><span class="cmd">curl -s /index.html | grep -o lang=</span><span class="fix">→ zh-CN ✓</span></div>`;

+TERMINALS['every-player-stuttered-the-file-was-fine'] = `
+    <div class="line"><span class="prompt">$</span><span class="cmd">ffmpeg -v error -i out.mp4 -f null -   ->  0 errors</span></div>
+    <div class="line"><span class="prompt">&nbsp;</span><span class="err">106/5819 frames show the WRONG picture</span></div>
+    <div class="line"><span class="prompt">$</span><span class="cmd">-c:v av1_cuvid   ->   -hwaccel cuda -c:v av1</span></div>
+    <div class="line"><span class="prompt">&nbsp;</span><span class="cmd">0 wrong frames  SSIM 1.000000</span><span class="fix">-> fixed</span></div>`;
+
 // ---------- read frontmatter ----------
 const postPath = join(ROOT, 'src', 'content', 'posts', `${slug}.md`);
 if (!existsSync(postPath)) {
@@ -0,0 +1,189 @@
+---
+title: "Every Player Stuttered on the Same Video. The File Was Blameless."
+description: "Same HEVC file, same stutter in VLC, Chrome and an iPad — offline. The container timing was perfect. The pictures were not: NVDEC's legacy AV1 wrapper was returning frames from the wrong timestamps."
+pubDate: 2026-10-08
+category: devops
+tags: [ffmpeg, nvdec, hevc, transcoding, video]
+ogImage: /og/every-player-stuttered-the-file-was-fine.png
+banner: /banners/every-player-stuttered-the-file-was-fine.png
+draft: false
+---
+
+## The question I could not answer
+
+A converted video stuttered. Not badly — a hitch every few seconds, then it
+carried on. The kind of thing you blame on Wi-Fi.
+
+Except it happened on an iPad Pro M1 playing the file **offline**, in VLC,
+and in Chrome on a desktop. Different decoders, different platforms, same
+periodic hitch. The original download played perfectly everywhere. So the
+network was out, the device was out, and the file was in.
+
+I spent a long time measuring the file, and every measurement said the file
+was fine. It was not fine. It was a faithful re-encode of a decoder bug.
+
+## Step 1: measure the container, not the picture
+
+The converted file is HEVC Main 8-bit in MP4 (`hvc1`), 3840×2160, 25 fps,
+yuv420p, bt709, average 15.49 Mbps, 232.8 seconds, 5819 frames, `moov`
+before `mdat`.
+
+Everything about its timeline is textbook:
+
+```text
+frame intervals       5818 of 5818 = exactly 0.040000 s
+duplicate PTS         0
+negative PTS          0
+gaps > 1.5x median    0
+edit list (ctts)      5796 entries  (not a flat list)
+has_b_frames          2
+A/V start offset      0.000 s
+B-frame pattern       I B B B P B B B ...  (744 groups, all 3 B-frames)
+```
+
+A flat `ctts` — every presentation offset set to zero — is the classic way a
+re-encode ends up with PTS sitting in decode order, and it makes players
+drop roughly a quarter of the frames. That is what I expected to find. It
+was not there.
+
+Full decode, both verbosity levels:
+
+```bash
+ffmpeg -v warning -i converted.mp4 -f null -
+ffmpeg -v error   -i converted.mp4 -f null -
+```
+
+Both exit 0, both print nothing, 174 seconds of wall time for 232 seconds of
+video. No corrupt frames, no missing references, no non-monotonic DTS.
+
+The GOP looked different from the source (24 keyframes, a constant 10.0 s,
+because `hevc_nvenc` defaults to `-g 250`; the source has 59 keyframes at an
+average 3.88 s) and the bitrate was 17% higher than the source — but the
+source's own peak was higher than the output's, so neither explains a hitch
+every few seconds.
+
+**Everything measurable in the container was correct.** Which is the point
+where you have to stop measuring the container.
+
+## Step 2: compare pictures, from two different decoders
+
+If the timing is right and the file still looks wrong, the question becomes
+whether the *pictures* are right. I compared the source file decoded two
+ways, frame by frame, with SSIM:
+
+```bash
+# software decode of the source
+ffmpeg -i source.mkv -vf "scale=640:360" -f rawvideo -pix_fmt yuv420p sw.yuv
+
+# the same frames through NVDEC's AV1 path
+ffmpeg -c:v av1_cuvid -i source.mkv -vf "scale=640:360" -f rawvideo -pix_fmt yuv420p hw.yuv
+```
+
+106 frames out of 5819 disagree — 1.8% — with SSIM as low as **0.330**. Not
+uniform noise: the interval histogram peaks at about every 30 frames.
+
+Two control tests made it worse and better at the same time:
+
+- **Deterministic.** Decoding the same source twice through `av1_cuvid`
+gives SSIM 1.000000 between the two runs. This is not a race; every
+conversion will weld the same bad frames into the output.
+- **Not all AV1.** A 1080p AV1 sample decoded identically through both paths
+(SSIM 1.000000). A second 4K AV1 stream did not. It is stream-dependent.
+
+Then the decisive comparison — which side does the converted file match?
+
+| Comparison | Mean SSIM | Minimum | Frames below 0.98 |
+|---|---|---|---|
+| Converted HEVC vs software decode of the source | 0.99091 | **0.330** | **106** |
+| Converted HEVC vs NVDEC decode of the source | **0.99742** | 0.991 | **0** |
+
+The converted file agrees with the *wrong* decode, frame for frame. It is
+not a bad encode. It is an accurate encode of the wrong pictures.
+
+A single extracted frame shows it plainly: frame 546 through software decode
+is a wide shot of a choir; through `av1_cuvid` it is a close-up from a
+different moment in the video — and the converted file has the close-up.
+
+## Step 3: the fix is one line
+
+The bug is not the hardware. It is the legacy decoder wrapper. `av1_cuvid`
+is an old-style wrapped decoder; the modern hardware path goes through
+`hwaccel`:
+
+```bash
+# before
+-c:v av1_cuvid
+
+# after
+-hwaccel cuda -hwaccel_output_format cuda -c:v av1
+```
+
+Same stream, same source, 1500 frames compared against the software
+reference:
+
+```text
+-hwaccel cuda -c:v av1        SSIM 1.000000   (bit-identical, every frame)
+vp9_cuvid vs libvpx-vp9       SSIM 0.99991    (0 frames below 0.98 - unaffected)
+```
+
+Encoding speed did not change: a 30-second 4K slice in 24.1 s, 1.24×
+realtime either way.
+
+To be precise about the scope: this is a bug in the `*_cuvid` wrapper for
+**AV1** on this driver/library combination, reproducible on demand. It says
+nothing about your GPU's AV1 decoder — the hardware path is clean.
+
+## Step 4: make it impossible to ship again
+
+The uncomfortable part is that my pipeline *had* a verification step. It
+checked the duration, the resolution, the codec, the frame rate and the
+frame rhythm — all container properties. Every one of them passed on a file
+with 106 wrong frames.
+
+So the pipeline is now five steps, and the fourth one is the picture check:
+
+```text
+decode -> encode -> verify structure -> verify content -> replace the file
+```
+
+The content gate compares the output's frames against a software decode of
+the source (both sides decoded sequentially, cropped by timestamp), and uses
+the defect's *shape* rather than a single-frame threshold:
+
+- fail if any frame falls below 0.85
+- fail if more than 1% of frames fall below 0.95
+- fail if the mean falls below 0.98
+- ignore near-flat frames (SSIM is meaningless on frames that are almost
+entirely black — two nearly-black frames can score 0.25 while being visually
+identical)
+
+Those numbers are calibrated against measured data, not intuition. Good
+encodes in this library sit at 0 frames of 1500 below 0.95. The defective
+files sit at 2.9% of frames below 0.85 with a depressed mean. A perfectly
+good 4K60 encode of a dance video once measured min 0.949 — a single frame
+below 0.95, at a hard cut — which a single-frame threshold would have
+rejected forever.
+
+Scoped to the library: 75 files had AV1 originals in the snapshots, 65 of
+them (87%) failed the content gate. All 65 were re-encoded from the pristine
+snapshot originals — never from the already-corrupted file — and every one
+landed with min SSIM 0.997–0.999.
+
+## What I would do differently
+
+1. **"It decodes without errors" is not a quality check.** Decoding a frame
+successfully says nothing about whether it is the *right* frame. The picture
+comparison is the check that found it. 2. **Compare against a second
+decoder, not against your expectations.** A software decode is free and it
+is the reference. If your hardware path disagrees with it, you have found
+something worth understanding. 3. **Verify the output against the source,
+not the source's metadata.** 4. **Do not assume the modern and legacy
+hardware paths are the same code.** `av1_cuvid` and `-hwaccel cuda -c:v av1`
+reach the same silicon through different plumbing. A bug in one is not a
+verdict on the hardware. 5. **A gate that fails safe is worth more than a
+gate that never fails.** My first gate rejected that 4K60 encode. The file
+was fine and the gate was wrong — but the failure cost one wasted encode,
+not a bad file in the library.
+
+The stutter is gone. Not because the encoder got better, but because the
+decoder stopped lying.
@@ -0,0 +1,134 @@
+---
+title: "每个播放器都在同一个视频上卡顿，而文件本身是无辜的"
+description: "同一个 HEVC 文件，在 VLC、Chrome 和 iPad 上（离线）都出现同样的周期卡顿。容器层时间轴完美无缺，画面却不是——NVDEC 的旧版 AV1 包装器返回了错误时间点的帧。"
+pubDate: 2026-10-08
+category: devops
+tags: [ffmpeg, nvdec, hevc, transcoding, video]
+ogImage: /og/every-player-stuttered-the-file-was-fine.png
+banner: /banners/every-player-stuttered-the-file-was-fine.png
+draft: false
+---
+
+## 一个我回答不了的问题
+
+有个转好的视频会卡顿。不严重——每隔几秒顿一下，然后继续放。这种毛病你通常会怪到 Wi-Fi 头上。
+
+问题是它出现在一台 **离线**播放的 iPad Pro M1 上、出现在 VLC 里、也出现在桌面版 Chrome 里。不同的解码器、不同的平台、同样的周期性卡顿。而原始下载的视频在所有地方都播放正常。所以网络被排除、设备被排除，剩下的嫌疑只有那个文件。
+
+我花了很长时间测量这个文件，所有测量都说它是好的。它并不是好的。它是**对一个解码器 bug 的忠实重编码**。
+
+## 第一步：先量容器，而不是量画面
+
+这个转好的文件是 MP4 容器里的 HEVC Main 8-bit（`hvc1`）、3840×2160、25 fps、yuv420p、bt709、平均 15.49 Mbps、232.8 秒、5819 帧，`moov` 在 `mdat` 之前。
+
+它的时间轴是教科书级别的干净：
+
+```text
+帧间隔              5818 个全部 = 恰好 0.040000 秒
+重复 PTS            0
+负值 PTS            0
+超过中位数 1.5 倍的缺口   0
+edit list (ctts)    5796 条（不是扁平表）
+has_b_frames        2
+音视频起始偏移        0.000 秒
+B 帧图案            I B B B P B B B ……（744 组，每组都是 3 个 B）
+```
+
+扁平的 `ctts`——把每个呈现偏移都设成零——是重编码后 PTS 停留在解码顺序上的经典症状，会让播放器丢掉大约四分之一的帧。这正是我预期会找到的东西。它不在那里。
+
+完整解码，两个详细级别都跑：
+
+```bash
+ffmpeg -v warning -i converted.mp4 -f null -
+ffmpeg -v error   -i converted.mp4 -f null -
+```
+
+两条都以 0 退出、什么都不打印，232 秒的视频用了 174 秒跑完。没有损坏帧、没有缺失参考帧、没有非单调 DTS。
+
+GOP 确实和源不同（24 个关键帧、恒定 10.0 秒，因为 `hevc_nvenc` 默认 `-g 250`；源是 59 个关键帧、平均 3.88 秒），码率也比源高 17%——但源自身的峰值比输出的峰值还高，所以两者都解释不了"每隔几秒顿一下"。
+
+**容器里所有可测的东西都是正确的。** 而这恰恰是"你就该停止测量容器"的那个点。
+
+## 第二步：用两个不同的解码器去比画面
+
+如果时间轴是对的、文件看起来却仍是错的，问题就变成：**画面**本身对不对。我把源文件用两种方式解码，再逐帧做 SSIM 对比：
+
+```bash
+# 软件解码源文件
+ffmpeg -i source.mkv -vf "scale=640:360" -f rawvideo -pix_fmt yuv420p sw.yuv
+
+# 同一批帧走 NVDEC 的 AV1 路径
+ffmpeg -c:v av1_cuvid -i source.mkv -vf "scale=640:360" -f rawvideo -pix_fmt yuv420p hw.yuv
+```
+
+5819 帧里有 **106 帧**不一致——占 1.8%——SSIM 最低到 **0.330**。不是均匀的噪声：间隔直方图的峰值大约落在每 30 帧一次。
+
+两个对照实验，一个让情况更糟、一个让情况更清楚：
+
+- **确定性。** 同一个源用 `av1_cuvid` 解两次，两次之间 SSIM 是 1.000000。这不是竞争条件；每一次转换都会把同一批坏帧焊进输出里。
+- **不是所有 AV1 都中招。** 一个 1080p 的 AV1 样本两条路径解出来完全一致（SSIM 1.000000），而另一个 4K AV1 流不一致。这跟具体的流有关。
+
+然后是决定性的对比——转好的文件到底跟哪一边一致？
+
+| 对比 | 平均 SSIM | 最低 | 低于 0.98 的帧数 |
+|---|---|---|---|
+| 转好的 HEVC vs 源的软件解码 | 0.99091 | **0.330** | **106** |
+| 转好的 HEVC vs 源的 NVDEC 解码 | **0.99742** | 0.991 | **0** |
+
+转好的文件和**错误的那一份**解码逐帧吻合。它不是一次糟糕的编码，它是**对错误画面的一次准确编码**。
+
+抽出一帧就能看得很清楚：第 546 帧在软件解码里是合唱团的远景；在 `av1_cuvid` 里是另一个时刻的特写镜头——而转好的文件里是那张特写。
+
+## 第三步：修复只要一行
+
+Bug 不在硬件，在旧的解码器包装器。`av1_cuvid` 属于老式的 wrapper 解码路径；现代的硬件路径要走 `hwaccel`：
+
+```bash
+# 修复前
+-c:v av1_cuvid
+
+# 修复后
+-hwaccel cuda -hwaccel_output_format cuda -c:v av1
+```
+
+同一个流、同一个源，1500 帧对软件参考逐帧比较：
+
+```text
+-hwaccel cuda -c:v av1        SSIM 1.000000   （逐帧逐位一致）
+vp9_cuvid vs libvpx-vp9       SSIM 0.99991    （0 帧低于 0.98——不受影响）
+```
+
+编码速度没有变化：一段 30 秒的 4K 素材 24.1 秒编完，两种路径都是 1.24× 实时。
+
+范围要说准确：这是这套驱动/库组合下 **AV1** 的 `*_cuvid` 包装器的 bug，可稳定复现。它与你显卡的 AV1 解码器无关——现代硬件路径是干净的。
+
+## 第四步：让它再也不可能悄悄发出去
+
+最让人不安的地方在于：我的流水线**本来就有**验证步骤。它检查时长、分辨率、编码器、帧率和帧节奏——全是容器属性。这些检查在一个带着 106 帧错误画面的文件上**全部通过**。
+
+所以现在流水线是五步，第四步才是画面检查：
+
+```text
+解码 -> 编码 -> 结构验证 -> 内容验证 -> 替换文件
+```
+
+内容闸门把输出帧与源的软件解码逐帧对比（两侧都顺序解码、按时间戳裁剪取窗口），并且判的是缺陷的**形状**，而不是某一帧的硬线：
+
+- 任何一帧低于 0.85 → 失败
+- 低于 0.95 的帧超过 1% → 失败
+- 均值低于 0.98 → 失败
+- 剔除近平坦帧（对几乎全黑的帧，SSIM 没有意义——两帧几乎全黑也能算出 0.25，而它们在视觉上完全相同）
+
+这些阈值是拿实测数据标定出来的，不是凭直觉。这个素材库里好的编码在 1500 帧里有 0 帧低于 0.95；有缺陷的文件有 2.9% 的帧低于 0.85 且均值下滑。而一段完全正常的 4K60 舞蹈视频曾经测出最低 0.949——一个硬切处只有 1 帧低于 0.95——用单帧硬线判的话，它会被永久拒绝。
+
+范围：快照里有 AV1 原档的共 75 个文件，其中 **65 个（87%）**没通过内容闸门。这 65 个全部用**快照里的干净原档**重转（绝不用已经被弄坏的文件），每一个都以 min SSIM 0.997–0.999 落地。
+
+## 我会怎么做得不一样
+
+1. **"解码没有报错"不是质量检查。** 一帧能解码成功，跟它是不是**正确的那一帧**毫无关系。找出问题的是画面比对。
+2. **跟第二个解码器比，而不是跟你的预期比。** 软件解码是免费的，而且它就是参考基准。如果你的硬件路径跟它不一致，你就找到了值得搞清楚的东西。
+3. **验证输出要对着源，而不是对着源的元数据。**
+4. **别以为现代硬件路径和旧包装器是同一条代码路径。** `av1_cuvid` 和 `-hwaccel cuda -c:v av1` 走的是不同的管道、到的是同一块硅片。其中一个有 bug，不等于对硬件下判决。
+5. **一个"失败得安全"的闸门，比一个从不失败的闸门值钱得多。** 我的第一版闸门拒绝了那段 4K60 编码，文件是好的、闸门是错的——但这个错误的代价只是白烧一次编码，而不是让一个坏文件进了库。
+
+卡顿消失了。不是因为编码器变好了，而是因为解码器不再说谎了。