library_run.py: policy-driven runner (never bigger, quality pinned, tiered) + measured YouTube H.264 numbers
The previous runner encoded everything at a fixed -cq 27 with -maxrate = 1.5x the FILE bitrate, which is how a library ended up at 111% of its source size. This replaces the encode step with the policy the skill documents, and adds the gates that make it a guarantee: * classify by bpp: <0.025 copy the video (bit-identical), 0.025-0.040 hevc_nvenc, >0.040 libx265 * -maxrate = 1.0x the source's VIDEO bitrate (never the file total) + a post-encode assertion * SSIM gate: per-frame comparison of a 60 s window against the source, retry once at +2 quality if it fails, then fail the item (the original is never touched) * audio: copy when AAC, else AAC at the SOURCE's bitrate (a fixed 256k cancelled the video saving) * x265 tier pinned to the E-cores; --tier to run tiers separately; --keep-local for a pilot that touches nothing * defaults cq31 / crf26 from the measurement below, not cq27 Measured on three real sources (60 s window, video-only): cq27 GROWS two of three files (115%, 119%); cq31 lands at 43-74%; x265 crf26 at 34-67% with min SSIM 0.988-0.993 - and x265 beats NVENC by 10-22% at matched SSIM. Pinning min SSIM >= 0.996 leaves only cq27, which is bigger than the source, so ~0.99 is the usable quality frontier and the owner gets to make that call explicitly (references/hevc-conversion-policy.md section 11, SKILL.md scope section).
This commit is contained in:
@@ -1,7 +1,8 @@
|
||||
# HEVC conversion policy — never bigger, same look, no judder
|
||||
# HEVC conversion policy — playable everywhere first, shrink where there is headroom, no judder
|
||||
|
||||
Owner's standing requirements for a HEVC conversion. Read this before starting a library job;
|
||||
the default settings in `transcode.py` do **not** satisfy them.
|
||||
the default settings in `transcode.py` do **not** satisfy them. Note the title: the size rule is
|
||||
**conditional** (section 1), not a blanket "never grow".
|
||||
|
||||
## 1. Size: shrink where there is headroom, growth is acceptable where there is not
|
||||
|
||||
@@ -84,6 +85,10 @@ One library: 131 files < 0.025 vs 243 files > 0.040 out of 572. A single global
|
||||
cannot be right for both — that is precisely how a job ends up growing some files and shrinking
|
||||
others. Calibrate per class on 1-2 representatives (60-120 s, sweep 2-3 quality levels).
|
||||
|
||||
bpp and the size regime are **different axes** and can disagree: bpp picks the *recipe* (how much
|
||||
effort is worth spending), while the pre-flight sample in section 2 picks the *size expectation*
|
||||
(hard assertion or not). When they disagree the pre-flight wins — it is measured on the actual file.
|
||||
|
||||
## 4. Same sound quality
|
||||
|
||||
- Source audio already **AAC** -> `-c:a copy`, bit-identical. Always preferable.
|
||||
@@ -135,7 +140,8 @@ Per item, in this order:
|
||||
3. **Full decode pass** (`ffmpeg -v error -i OUT -f null -`) — the only check that proves the
|
||||
payload is not truncated or corrupt.
|
||||
4. **SSIM vs the source window >= target** (section 2).
|
||||
5. **Size assertion**: output video bitrate <= source (section 1).
|
||||
5. **Size assertion**, per the regime in section 1: hard `out <= src` where the source has
|
||||
headroom; runaway guard `out <= 1.25x src` where the pre-flight says it is already compressed.
|
||||
6. **Upload to a temp name, read the share copy back end-to-end, compare SHA-256 + byte count**,
|
||||
then replace atomically. A duration probe is not enough: with `+faststart` the `moov` is written
|
||||
first, so a truncated upload still reports the full duration.
|
||||
@@ -216,6 +222,40 @@ Only **20% slower**, not the ~3x that per-core throughput suggests: the 10 E-cor
|
||||
while half of the P-core threads are SMT siblings, and x265 does not scale linearly anyway. So
|
||||
E-cores-only is the right default when the priority is efficiency over speed.
|
||||
|
||||
## 11. Measured: what H.264 -> HEVC actually saves (3 real sources, 2026-10-07)
|
||||
|
||||
A 60 s window from the middle of three real library files, video-only, SSIM vs the source at
|
||||
480x270. **Read this before promising a saving**: it ranges from 16% to 66% on the same settings,
|
||||
and the setting that a fixed `-cq 27` implies does not exist.
|
||||
|
||||
| source | src video | cq27 | cq30 | cq31 | x265 crf24 | **x265 crf26** | x265 crf28 |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 1080p30 music, bpp 0.0437 | 2720 kbps | 72.4% | 51.0% | 43.5% | 43.0% | **33.7%** | 26.4% |
|
||||
| 720p30, bpp 0.0437 | 1208 kbps | **115.2%** | 82.6% | 72.9% | 71.4% | **56.7%** | 45.0% |
|
||||
| 1080x1920@30 (vertical), bpp 0.0722 | 4490 kbps | **119.3%** | 83.8% | 74.3% | 84.3% | **67.0%** | 53.2% |
|
||||
|
||||
(SSIM min/mean at those settings: cq27 0.993-0.996; cq30 0.990-0.995; cq31 0.989-0.994;
|
||||
crf24 0.990-0.993; crf26 0.988-0.993; crf28 0.985-0.991.)
|
||||
|
||||
Three findings that change the recipe:
|
||||
|
||||
1. **`-cq 27` GROWS two of the three files (115%, 119%).** That is the same defect that made a
|
||||
whole library land at 111%: a fixed cq is a quality target, and on an already-efficient H.264
|
||||
source it targets a quality *above* the source. Start at **cq 31 / crf 26**, not 27.
|
||||
2. **"Same quality" and "smaller" cannot both be pushed to the limit.** Pinning min SSIM >= 0.996
|
||||
(the earlier target) leaves exactly one option on these files - cq27 - which is *bigger* than
|
||||
the source. The usable frontier is min SSIM ~0.99: **x265 crf26 at 34-67% of the source**, i.e.
|
||||
33-66% smaller. Put that choice to the owner explicitly instead of quietly dropping quality.
|
||||
3. **x265 beats NVENC by 10-22% at MATCHED SSIM** (720p: cq31 72.9% vs crf26 56.7% at min SSIM
|
||||
0.9894/0.9879; 1080p: 43.5% vs 33.7% at 0.9938/0.9927) - consistent with the earlier 9-16%
|
||||
estimate. So: NVENC for the mid tier (speed), x265 for the fat tier (size). Note the cost is real:
|
||||
on these files x265 ran ~1.9-3x realtime vs 5x+ for hevc_nvenc.
|
||||
|
||||
Defaults that follow from this, and the gates that enforce them, are in `scripts/library_run.py`:
|
||||
`CQ_NVENC=31`, `CRF_X265=26`, `SSIM_TARGET=0.985` (per-frame min), `SIZE_CEIL=1.0` (the output's
|
||||
video bitrate may not exceed the source's), a one-step quality retry when the SSIM gate fails, and
|
||||
`-maxrate` = 1.0x the source's **video** bitrate.
|
||||
|
||||
## 9. CPU and GPU tiers in parallel
|
||||
|
||||
Different tiers can encode **at the same time** — NVENC is a fixed-function ASIC, x265 is CPU.
|
||||
@@ -241,6 +281,8 @@ serial 96 h (4.0 days) -> parallel **66 h (2.7 days)**. Requirements to run it s
|
||||
|
||||
`SRC` = the local staged source, `SRC_V` = `src_video_bps` from section 1, `A` = `src_audio_kbps`
|
||||
(section 4). Affinitize the process to the E-cores (`0x3FF0000`) before the tier-C command.
|
||||
`-maxrate SRC_V` below is the **headroom** setting; use `1.5x SRC_V` when the pre-flight says the
|
||||
source is already compressed and growth is acceptable (section 1).
|
||||
|
||||
```sh
|
||||
# ---- tier A: lean (bpp < 0.025) — no video re-encode at all, size cannot change
|
||||
|
||||
Reference in New Issue
Block a user