universal-video-transcode: any file in, one iPad+Android-playable MP4 out
Give it a path; it reads the source, picks the recipe, encodes, and verifies the result before accepting it. The problem it exists to remove: a file can drop ~23% of its frames in a browser while r_frame_rate == avg_frame_rate reads EXACTLY equal, so every fps-based check calls it healthy and it never enters a repair queue. The cause is deterministic and header-readable -- the MP4 composition-offset table is unusable, either a single constant ctts entry (PTS == DTS + const) or no ctts box at all (PTS == DTS) -- so the frame timestamps are in decode order and the decoder's reordered output frames carry the wrong times. - three recipes chosen from the source FILE, never the filename: lossless -c copy when the timeline allows it, AAC-only fix when just the audio is wrong, full HEVC (or x264 for reach) when the timeline cannot be copied - quality policy: never upscale; HEVC -cq 27 measured at SSIM 0.9937 ~= source size; audio copied when already AAC, else AAC 256k - the two flags that must never be dropped: -fps_mode cfr and -pix_fmt yuv420p - verified before acceptance: duration, resolution, codec, hvc1 tag, 8-bit, uniform cadence, no flat composition offsets, full decode pass, and a read-back after the write - built for cron: lock file (two concurrent NVENC jobs starve one encoder and DROP FRAMES), idempotent, leaves the input untouched on failure, aborts below 5 GB free - scripts/selftest.py builds both defect shapes and asserts the detector, including a Matroska case that must NOT be flagged Measured on the file that prompted this: 23.0% of frames dropped before, 0.0% after.
This commit is contained in:
Hermes Agent
@@ -0,0 +1,95 @@
|
||||
# Compatibility matrix — what actually plays where
|
||||
|
||||
Sources: Android's [Supported media formats](https://developer.android.com/media/platform/supported-formats)
|
||||
(read 2026-10-04) and Apple's HEVC hardware baseline (A9, announced 2015-09-09 with the
|
||||
iPhone 6S — its first chip with a hardware HEVC decoder, **Main 8 and Main 10**).
|
||||
|
||||
## The intersection
|
||||
|
||||
The target is the set that plays on **both** platforms with **no third-party player**:
|
||||
|
||||
```text
|
||||
Container : MP4 (ISO BMFF)
|
||||
Video : H.264 High 8-bit yuv420p ── universal ──
|
||||
or HEVC Main 8-bit yuv420p, tagged hvc1 ── smaller ──
|
||||
Audio : AAC-LC
|
||||
Extras : moov before mdat (+faststart), uniform frame cadence
|
||||
```
|
||||
|
||||
**Apple is the constraint, not Android.** Android decodes Matroska and Opus natively; iOS has
|
||||
no Matroska demuxer and cannot play Opus at all. So `.mkv` + Opus → MP4 + AAC is the change
|
||||
that forces the whole conversion.
|
||||
|
||||
## Android (from the official table)
|
||||
|
||||
| Format | Decoder | Containers | Notes |
|
||||
|---|---|---|---|
|
||||
| H.264 AVC **Baseline** | **YES** (all versions) | 3GP, MP4, MPEG-TS | the only unconditional guarantee |
|
||||
| H.264 AVC **Main** | **YES** | — | "The decoder is required, the encoder is recommended"; encoder Main is Android 6.0+ |
|
||||
| **H.265 HEVC** | **Android 5.0+** | MP4, Matroska | **"Main Profile Level 3 for mobile devices and Main Profile Level 4.1 for Android TV"** |
|
||||
| VP8 | Android 2.3.3+ | WebM, Matroska | |
|
||||
| VP9 | Android 4.4+ | WebM, Matroska, MP4 | |
|
||||
| AV1 | Android 10+ (mandatory 14+) | MP4, Matroska | |
|
||||
| AAC-LC | YES | 3GP, MP4, m4a, ADTS | mono/stereo/5.0/5.1, 8–48 kHz |
|
||||
| Opus | Android 5.0+ | Ogg, MP4, Matroska | irrelevant for iOS targets |
|
||||
|
||||
### The HEVC level caveat — read this before promising "HEVC plays on Android"
|
||||
|
||||
Android's guarantee is **Main profile Level 3**, which is roughly SD. Our 1080p output declares
|
||||
**Level 4.0** and 4K declares **5.0** (measured). Modern phones decode far beyond that, but a
|
||||
low-end or old device is only *required* to reach Level 3 — so "Android supports HEVC" does not
|
||||
imply "Android supports *this* HEVC".
|
||||
|
||||
- Audience includes cheap/old Android hardware → prefer **H.264** (`--codec x264`).
|
||||
- Modern devices only → HEVC is fine and buys a large size reduction.
|
||||
|
||||
## Apple / iOS
|
||||
|
||||
| Format | Support |
|
||||
|---|---|
|
||||
| Container | **MP4 / MOV only.** No Matroska demuxer in Photos, Files, TV or Safari. |
|
||||
| H.264 | All devices. |
|
||||
| HEVC | **A9 and newer** (iPhone 6S / 2015 iPad and later). Main 8 and Main 10. |
|
||||
| `hvc1` vs `hev1` | **`hvc1` required.** Apple refuses `hev1`; ffmpeg defaults to `hev1`. |
|
||||
| VP9 / VP09 | **Never.** |
|
||||
| AV1 | Only A17 Pro / M3 and newer. |
|
||||
| Opus | **Never.** |
|
||||
| AAC | Yes. |
|
||||
|
||||
`-tag:v hvc1` is the single most commonly missed flag: a file that is otherwise perfect will be
|
||||
refused by an iPad purely because it carries the `hev1` sample entry.
|
||||
|
||||
## Pixel format and bit depth
|
||||
|
||||
| pix_fmt | Verdict |
|
||||
|---|---|
|
||||
| **`yuv420p`** | **use this** — 8-bit 4:2:0, universal on both platforms |
|
||||
| `yuv420p10le` | Apple accepts it (A9+, Main 10); Android's guarantee is Main profile, so treat as a compatibility risk. Not needed for SDR sources. |
|
||||
| `yuv422p` / `yuv444p` | **avoid** — outside the guaranteed profile for both |
|
||||
|
||||
**HDR is a separate question.** All VP9/VP09/AV1 sources sampled from this library were
|
||||
**Profile 0 = 8-bit, yuv420p, bt709**; HDR was 1 file in 4431. For SDR sources there is nothing
|
||||
to preserve, and tone-mapping is out of scope for the tool.
|
||||
|
||||
## Container and streaming details
|
||||
|
||||
- `moov` **before** `mdat` (`-movflags +faststart`) so playback starts without downloading the
|
||||
whole file. Verify: the first 32 bytes are `…ftyp isom`, then `moov`, then `mdat`.
|
||||
- **Range requests must work** — the player seeks with them. Expect `206` +
|
||||
`Content-Range: bytes …` + `Accept-Ranges: bytes`.
|
||||
- `Content-Type` should be `video/mp4`. A converted file served as `video/matroska` can be
|
||||
refused by AVPlayer even though the bytes are a valid MP4.
|
||||
- **The filename extension is not authoritative.** A `.mkv` name on an MP4 body streams fine
|
||||
when `Content-Type` follows the container. Do not "fix" the extension; do fix the MIME.
|
||||
|
||||
## Audio decisions
|
||||
|
||||
| source audio | action | result |
|
||||
|---|---|---|
|
||||
| AAC | **copy** (`-c:a copy`) | bit-identical to the original |
|
||||
| Opus / Vorbis / anything else | **AAC 256k** | required — Opus is invalid in MP4 and iOS cannot play it |
|
||||
|
||||
Measured cascade error (source Opus decode vs AAC decode, sample-aligned): 128k = 22.7 dB,
|
||||
192k = 29.0 dB, **256k = 33.5 dB**, 320k = 36.1 dB. Above 256k the residual is the source's own
|
||||
Opus loss, so 320k buys almost nothing. Cost of 256k over a 626-hour library: +0.5 MB/min
|
||||
≈ +19 GB (~3%), which is why audio stops being a concern at 256k.
|
||||
@@ -0,0 +1,124 @@
|
||||
# Verifying "it plays smoothly" — the gate, the arbiter, and what is automatable
|
||||
|
||||
## The question this answers
|
||||
|
||||
"Is there a dropped frame?" and "do I have to play it in a browser to know?" — no. The
|
||||
browser is the *arbiter* of the user-visible symptom; it is not the *gate*. Automate the
|
||||
causal check, spot-check with the browser.
|
||||
|
||||
The symptom comes from a deterministic property of the container, readable from the header:
|
||||
whether the frame timestamps are in **presentation** order or **decode** order.
|
||||
|
||||
## The defect classes
|
||||
|
||||
Three distinct causes produce the same complaint ("一卡一卡 / choppy"), and a file can have
|
||||
any combination. Only the first is visible to an fps test.
|
||||
|
||||
| # | class | signature | fps test sees it? |
|
||||
|---|---|---|---|
|
||||
| 1 | non-uniform cadence | `r_frame_rate` != `avg_frame_rate` (gap > 0.01) | **yes** |
|
||||
| 2 | encoder dropped frames | doubled intervals (0.040 and 0.080 where 0.040 belongs) | yes (avg drops) |
|
||||
| 3 | **unusable composition offsets** | `r == avg` **exactly** | **NO** |
|
||||
|
||||
Class 3 is the one that hides. Two shapes, both the same defect:
|
||||
|
||||
- MP4 with a **single constant `ctts` entry** → `PTS == DTS + const`
|
||||
- MP4 with **no `ctts` box at all** → `PTS == DTS`
|
||||
|
||||
Either way the timestamps are in decode order. Decoders still reorder by POC, so the
|
||||
timestamps attached to the *output* frames are wrong, and the compositor drops ~23% of them.
|
||||
|
||||
> **A missing `ctts` is only a defect on an MP4 whose video stream reorders.** Matroska stores
|
||||
> presentation timestamps per block and legitimately has no `ctts` box — so a detector that
|
||||
> treats "no ctts" as flat will flag every `.mkv`. Distinguish "no MP4 sample table" from
|
||||
> "MP4 with no ctts box", and use the packet-PTS test for Matroska.
|
||||
|
||||
## The gate (automatable, runs on every file)
|
||||
|
||||
Both checks are header-only (~50 ms) and **work on HEVC**, unlike the browser.
|
||||
|
||||
**MP4 — count the `ctts` entries.** Walk the `moov` from the first 2 MB (the `moov` is at the
|
||||
front after `+faststart`). `<= 1` entry on a stream with `has_b_frames > 0` is the defect.
|
||||
Healthy files in this library carry **1,500–43,000** entries.
|
||||
|
||||
**Any container — count PTS inversions.** A stream with B-frames *must* present frames out of
|
||||
decode order, so its container PTS go backwards somewhere:
|
||||
|
||||
```sh
|
||||
ffprobe -v error -select_streams v:0 -read_intervals "%+3" \
|
||||
-show_entries packet=pts -of csv=p=0 FILE
|
||||
```
|
||||
|
||||
Count steps where `pts[i+1] < pts[i]`. **0 inversions = flat/broken**; 20–45 = healthy.
|
||||
This is the only option for Matroska, which has no `ctts`.
|
||||
|
||||
Measured separation, no overlap:
|
||||
|
||||
| file | ctts entries | PTS inversions | Chrome dropped |
|
||||
|---|---|---|---|
|
||||
| defective | **1** | **0** | **22.8–27.3%** |
|
||||
| defective (no ctts box) | **absent** | **0** | **22.8–23.5%** |
|
||||
| same file, x264 CFR re-encode | 985 | 35 | **0.0%** |
|
||||
| known-good 1080p | 1549 | 23 | 0.0% |
|
||||
| known-good 360p | 4262 | 35 | 0.2% |
|
||||
| our HEVC 2160p output | many | 44 | (not measurable in Chrome) |
|
||||
|
||||
A file that is *already* a playable MP4 must also be checked before being skipped, or a
|
||||
"nothing to do" verdict is issued for a file that stutters.
|
||||
|
||||
## The arbiter (real Chrome) and its blind spot
|
||||
|
||||
```sh
|
||||
python measure_playback.py <url> [seconds] # add a second url to A/B in one run
|
||||
```
|
||||
|
||||
Plays muted in real headless Chrome over CDP and reports `getVideoPlaybackQuality()` —
|
||||
decoded frames, `droppedVideoFrames`, and the media-time/wall-clock ratio.
|
||||
|
||||
- **The ratio is not the verdict.** A defective file advances at ratio **1.000** while dropping
|
||||
20–23% of frames. Only the drop count separates healthy from broken.
|
||||
- **Threshold, not equality.** A 1–2 frame drop at startup is noise (healthy files measure
|
||||
0.0–0.2%); a real defect measures ~20%+. Treat `< 1%` as smooth.
|
||||
- **Chrome cannot decode HEVC.** `canPlayType('video/mp4; codecs="hvc1…"')` returns `''`, so an
|
||||
HEVC file reports **0 decoded / 0 dropped** and looks perfect while nothing played. A 0-frame
|
||||
result is **not a pass** — it means "not measurable here". Judge HEVC by the structural gate
|
||||
and confirm on the Apple device.
|
||||
|
||||
So the browser is: too slow (~30 s/file), unable to judge the main output codec, and
|
||||
statistical. It is the right tool for an H.264 A/B spot-check, not for a library gate.
|
||||
|
||||
## Why this is causal, not a heuristic — the 2x2
|
||||
|
||||
Correlation would not be enough (a defect could be a symptom of something else). The 2x2
|
||||
crosses the container timing against the bitstream:
|
||||
|
||||
| | flat container | correct offsets |
|
||||
|---|---|---|
|
||||
| **healthy bitstream** | **23.5% dropped** (stutter) | 0.0% |
|
||||
| **defective bitstream** | 22.8% dropped | **0.0%** (fixed) |
|
||||
|
||||
Writing the same flat container around a *healthy* bitstream reproduces the stutter, and
|
||||
writing proper offsets around the *broken* bitstream fixes it. The bitstream is irrelevant —
|
||||
**the container timing is the cause**. That is what licenses a header-only gate.
|
||||
|
||||
## What cannot fix it
|
||||
|
||||
- `-c copy` cannot retime. ffmpeg will not rewrite `stts`/`ctts` under stream copy; output
|
||||
timing stays byte-identical and the drop count is unchanged.
|
||||
- Repacking the raw H.264 (`-fflags +genpts -r 25 -c copy`) stays flat — measured.
|
||||
- `-bsf:v h264_metadata=tick_rate=…` retimes only the SPS; timestamps unchanged.
|
||||
- Re-downloading reproduces it: the `.mkv` yt-dlp writes is itself flat.
|
||||
|
||||
Only a real video re-encode regenerates the timeline. Cost on a copy-tier file: 1.28x at
|
||||
crf 23, 1.74x at crf 20, 2.17x at crf 18.
|
||||
|
||||
## Provenance: inherited, not introduced by the remux
|
||||
|
||||
The defect arrives with the download. Some `.mkv` files yt-dlp writes are already flat (PTS
|
||||
monotonic in file order, `dts = pts - 40`), and `-c copy` carries that through unchanged. An
|
||||
mp4→mkv→mp4 round trip does not repair it.
|
||||
|
||||
Measured on one channel folder: **8 of 8** `360p-avc1` source `.mkv` files were flat, and the
|
||||
same key was flat in both its `.mkv` and its converted `.mp4`. It is source-specific, not
|
||||
library-wide: ~1.3% of AVC1 files, and **0 of 60** in the re-encoded HEVC tier (a re-encode
|
||||
regenerates timing, so the HEVC path is structurally immune).
|
||||
Reference in New Issue
Block a user