Hermes Agent
0cdb54f910
Give it a path; it reads the source, picks the recipe, encodes, and verifies the result before accepting it. The problem it exists to remove: a file can drop ~23% of its frames in a browser while r_frame_rate == avg_frame_rate reads EXACTLY equal, so every fps-based check calls it healthy and it never enters a repair queue. The cause is deterministic and header-readable -- the MP4 composition-offset table is unusable, either a single constant ctts entry (PTS == DTS + const) or no ctts box at all (PTS == DTS) -- so the frame timestamps are in decode order and the decoder's reordered output frames carry the wrong times. - three recipes chosen from the source FILE, never the filename: lossless -c copy when the timeline allows it, AAC-only fix when just the audio is wrong, full HEVC (or x264 for reach) when the timeline cannot be copied - quality policy: never upscale; HEVC -cq 27 measured at SSIM 0.9937 ~= source size; audio copied when already AAC, else AAC 256k - the two flags that must never be dropped: -fps_mode cfr and -pix_fmt yuv420p - verified before acceptance: duration, resolution, codec, hvc1 tag, 8-bit, uniform cadence, no flat composition offsets, full decode pass, and a read-back after the write - built for cron: lock file (two concurrent NVENC jobs starve one encoder and DROP FRAMES), idempotent, leaves the input untouched on failure, aborts below 5 GB free - scripts/selftest.py builds both defect shapes and asserts the detector, including a Matroska case that must NOT be flagged Measured on the file that prompted this: 23.0% of frames dropped before, 0.0% after.
125 lines
5.9 KiB
Markdown
125 lines
5.9 KiB
Markdown
# Verifying "it plays smoothly" — the gate, the arbiter, and what is automatable
|
||
|
||
## The question this answers
|
||
|
||
"Is there a dropped frame?" and "do I have to play it in a browser to know?" — no. The
|
||
browser is the *arbiter* of the user-visible symptom; it is not the *gate*. Automate the
|
||
causal check, spot-check with the browser.
|
||
|
||
The symptom comes from a deterministic property of the container, readable from the header:
|
||
whether the frame timestamps are in **presentation** order or **decode** order.
|
||
|
||
## The defect classes
|
||
|
||
Three distinct causes produce the same complaint ("一卡一卡 / choppy"), and a file can have
|
||
any combination. Only the first is visible to an fps test.
|
||
|
||
| # | class | signature | fps test sees it? |
|
||
|---|---|---|---|
|
||
| 1 | non-uniform cadence | `r_frame_rate` != `avg_frame_rate` (gap > 0.01) | **yes** |
|
||
| 2 | encoder dropped frames | doubled intervals (0.040 and 0.080 where 0.040 belongs) | yes (avg drops) |
|
||
| 3 | **unusable composition offsets** | `r == avg` **exactly** | **NO** |
|
||
|
||
Class 3 is the one that hides. Two shapes, both the same defect:
|
||
|
||
- MP4 with a **single constant `ctts` entry** → `PTS == DTS + const`
|
||
- MP4 with **no `ctts` box at all** → `PTS == DTS`
|
||
|
||
Either way the timestamps are in decode order. Decoders still reorder by POC, so the
|
||
timestamps attached to the *output* frames are wrong, and the compositor drops ~23% of them.
|
||
|
||
> **A missing `ctts` is only a defect on an MP4 whose video stream reorders.** Matroska stores
|
||
> presentation timestamps per block and legitimately has no `ctts` box — so a detector that
|
||
> treats "no ctts" as flat will flag every `.mkv`. Distinguish "no MP4 sample table" from
|
||
> "MP4 with no ctts box", and use the packet-PTS test for Matroska.
|
||
|
||
## The gate (automatable, runs on every file)
|
||
|
||
Both checks are header-only (~50 ms) and **work on HEVC**, unlike the browser.
|
||
|
||
**MP4 — count the `ctts` entries.** Walk the `moov` from the first 2 MB (the `moov` is at the
|
||
front after `+faststart`). `<= 1` entry on a stream with `has_b_frames > 0` is the defect.
|
||
Healthy files in this library carry **1,500–43,000** entries.
|
||
|
||
**Any container — count PTS inversions.** A stream with B-frames *must* present frames out of
|
||
decode order, so its container PTS go backwards somewhere:
|
||
|
||
```sh
|
||
ffprobe -v error -select_streams v:0 -read_intervals "%+3" \
|
||
-show_entries packet=pts -of csv=p=0 FILE
|
||
```
|
||
|
||
Count steps where `pts[i+1] < pts[i]`. **0 inversions = flat/broken**; 20–45 = healthy.
|
||
This is the only option for Matroska, which has no `ctts`.
|
||
|
||
Measured separation, no overlap:
|
||
|
||
| file | ctts entries | PTS inversions | Chrome dropped |
|
||
|---|---|---|---|
|
||
| defective | **1** | **0** | **22.8–27.3%** |
|
||
| defective (no ctts box) | **absent** | **0** | **22.8–23.5%** |
|
||
| same file, x264 CFR re-encode | 985 | 35 | **0.0%** |
|
||
| known-good 1080p | 1549 | 23 | 0.0% |
|
||
| known-good 360p | 4262 | 35 | 0.2% |
|
||
| our HEVC 2160p output | many | 44 | (not measurable in Chrome) |
|
||
|
||
A file that is *already* a playable MP4 must also be checked before being skipped, or a
|
||
"nothing to do" verdict is issued for a file that stutters.
|
||
|
||
## The arbiter (real Chrome) and its blind spot
|
||
|
||
```sh
|
||
python measure_playback.py <url> [seconds] # add a second url to A/B in one run
|
||
```
|
||
|
||
Plays muted in real headless Chrome over CDP and reports `getVideoPlaybackQuality()` —
|
||
decoded frames, `droppedVideoFrames`, and the media-time/wall-clock ratio.
|
||
|
||
- **The ratio is not the verdict.** A defective file advances at ratio **1.000** while dropping
|
||
20–23% of frames. Only the drop count separates healthy from broken.
|
||
- **Threshold, not equality.** A 1–2 frame drop at startup is noise (healthy files measure
|
||
0.0–0.2%); a real defect measures ~20%+. Treat `< 1%` as smooth.
|
||
- **Chrome cannot decode HEVC.** `canPlayType('video/mp4; codecs="hvc1…"')` returns `''`, so an
|
||
HEVC file reports **0 decoded / 0 dropped** and looks perfect while nothing played. A 0-frame
|
||
result is **not a pass** — it means "not measurable here". Judge HEVC by the structural gate
|
||
and confirm on the Apple device.
|
||
|
||
So the browser is: too slow (~30 s/file), unable to judge the main output codec, and
|
||
statistical. It is the right tool for an H.264 A/B spot-check, not for a library gate.
|
||
|
||
## Why this is causal, not a heuristic — the 2x2
|
||
|
||
Correlation would not be enough (a defect could be a symptom of something else). The 2x2
|
||
crosses the container timing against the bitstream:
|
||
|
||
| | flat container | correct offsets |
|
||
|---|---|---|
|
||
| **healthy bitstream** | **23.5% dropped** (stutter) | 0.0% |
|
||
| **defective bitstream** | 22.8% dropped | **0.0%** (fixed) |
|
||
|
||
Writing the same flat container around a *healthy* bitstream reproduces the stutter, and
|
||
writing proper offsets around the *broken* bitstream fixes it. The bitstream is irrelevant —
|
||
**the container timing is the cause**. That is what licenses a header-only gate.
|
||
|
||
## What cannot fix it
|
||
|
||
- `-c copy` cannot retime. ffmpeg will not rewrite `stts`/`ctts` under stream copy; output
|
||
timing stays byte-identical and the drop count is unchanged.
|
||
- Repacking the raw H.264 (`-fflags +genpts -r 25 -c copy`) stays flat — measured.
|
||
- `-bsf:v h264_metadata=tick_rate=…` retimes only the SPS; timestamps unchanged.
|
||
- Re-downloading reproduces it: the `.mkv` yt-dlp writes is itself flat.
|
||
|
||
Only a real video re-encode regenerates the timeline. Cost on a copy-tier file: 1.28x at
|
||
crf 23, 1.74x at crf 20, 2.17x at crf 18.
|
||
|
||
## Provenance: inherited, not introduced by the remux
|
||
|
||
The defect arrives with the download. Some `.mkv` files yt-dlp writes are already flat (PTS
|
||
monotonic in file order, `dts = pts - 40`), and `-c copy` carries that through unchanged. An
|
||
mp4→mkv→mp4 round trip does not repair it.
|
||
|
||
Measured on one channel folder: **8 of 8** `360p-avc1` source `.mkv` files were flat, and the
|
||
same key was flat in both its `.mkv` and its converted `.mp4`. It is source-specific, not
|
||
library-wide: ~1.3% of AVC1 files, and **0 of 60** in the re-encoded HEVC tier (a re-encode
|
||
regenerates timing, so the HEVC path is structurally immune).
|