docs: scope the size policy to H.264 sources; publish the HEVC conversion policy
The installed skill had drifted ahead of the repo: SKILL.md carried the whole owner-requirements section and references/hevc-conversion-policy.md was missing here entirely. Synced both. Also states the boundary the policy needs: this skill's size hunt (SSIM-pinned quality, maxrate 1.0x source video bitrate, bpp tiering, matched audio) is for H.264 / MPEG-4 sources. YouTube's VP9/AV1 sources are already near-optimal for their quality, so a re-encode can only grow them or lose quality - the accepted answer there is 'same size, do not chase smaller', and the job reduces to: never decode with *_cuvid, keep -fps_mode cfr, gate the output on pictures vs the source, and source any repair from the pre-conversion snapshot backup rather than the converted file.
This commit is contained in:
@@ -23,6 +23,25 @@ python scripts/transcode.py "D:/video" --recursive --dry-run # plan only
|
||||
python scripts/transcode.py "D:/video" --out D:/converted
|
||||
```
|
||||
|
||||
## Scope — read this before applying the size policy
|
||||
|
||||
This policy is for **H.264 (and older MPEG-4/DivX) sources**, where the source is fat enough that
|
||||
HEVC at matched quality lands 25-50% smaller (the course-library case: 1080p H.264 at
|
||||
550-2400 kbps, mp3 audio, files 300 MB-1.3 GB).
|
||||
|
||||
**Do NOT run the size hunt on YouTube-sourced libraries (VP9/AV1).** YouTube already serves those
|
||||
at a near-optimal bitrate for their quality, so re-encoding an already-lean VP9/AV1 file can only
|
||||
grow it or lose quality - "the source is already as small as it gets, and that is accepted". The
|
||||
job there is narrower and different:
|
||||
|
||||
| | H.264 library (this policy) | YouTube VP9/AV1 library |
|
||||
|---|---|---|
|
||||
| goal | same look, **smaller** file | same look, same size - no shrink sought |
|
||||
| decode | any correct decoder | **never `*_cuvid`** (wrong frames, see below) |
|
||||
| quality gate | SSIM >= 0.996 + size ceiling | **per-frame content check vs the software decode** |
|
||||
| source for a repair | the file itself | **the pre-conversion snapshot backup**, never the already-converted file |
|
||||
| frame cadence | `-fps_mode cfr` | `-fps_mode cfr` |
|
||||
|
||||
## What "playable everywhere" actually means
|
||||
|
||||
The intersection that both platforms decode **without a third-party player**:
|
||||
@@ -142,6 +161,99 @@ python /path/to/scripts/transcode.py "Z:/docker/tubesync/download" --recursive \
|
||||
For a TubeSync library specifically, prefer its own DB-driven pipeline (`tubesync-selfhosted`)
|
||||
— it also updates the database rows. Use this tool for anything outside that library.
|
||||
|
||||
## Whole-library runs on a network share (in-place, safe)
|
||||
|
||||
`transcode.py` writes `dst + ".part.mp4"` **next to the source**. Over SMB/WebDAV/Dokan that
|
||||
is the failure mode to avoid, and when the source is already `.mp4` then `dst == src`, so the
|
||||
only copy of the file is the one being written. For a library run, drive ffmpeg directly
|
||||
from a small runner instead:
|
||||
|
||||
1. copy the source **off the share** to local disk (chunked; treat a short read as failure)
|
||||
2. encode locally, with the same mandatory flags (`-fps_mode cfr`, `-pix_fmt yuv420p`)
|
||||
3. run the structural gate **and** a full decode pass on the local output
|
||||
4. upload to `<name>.new.mp4` (never the final name)
|
||||
5. read the share copy back end-to-end and compare **SHA-256 + byte count** with the local file
|
||||
6. only then `os.replace()` it over the original, and re-probe the result on the share
|
||||
|
||||
Step 5 costs one extra network read (~1 h per 330 GiB) and is the only thing that actually
|
||||
proves the remote copy is complete — `getsize()` cannot, and neither can a duration probe:
|
||||
with `+faststart` the `moov` is written first, so a truncated file still reports full duration.
|
||||
Keep a JSONL ledger and a per-file skip test ("already HEVC hvc1") so a reboot resumes safely.
|
||||
A single instance only; the lock matters even more here because two runs would race on `dst`.
|
||||
|
||||
### `plan_for()` gaps to know before a library run
|
||||
|
||||
- **`mpeg4` (DivX/Xvid) returns `None` = "unsupported video codec"** — it is skipped, not
|
||||
converted, so `.avi`/`mp4v` files silently keep a codec no iPad can play. Give them an
|
||||
explicit re-encode.
|
||||
- **Repairing a defective H.264 timeline returns `x264`, not HEVC** (hardcoded in the H.264
|
||||
branch). If the requirement is "everything in HEVC", force it — the recipe is a policy, not
|
||||
the tool's default.
|
||||
- A truncated file (`moov atom not found`) is a **re-download**, not a transcode job; report it
|
||||
instead of retrying.
|
||||
|
||||
### Measured throughput (RTX 3060, hevc_nvenc p6, `-tune hq`, cq 27)
|
||||
|
||||
Use these to quote an ETA before starting — the estimate is the user's decision input.
|
||||
|
||||
| source | speed |
|
||||
|---|---|
|
||||
| 1080p25 H.264 | **6.4x realtime** (~160 fps) |
|
||||
| 480x270 | ~34x realtime |
|
||||
| per-file overhead | ~45 s (local copy + decode pass + upload + read-back hash) |
|
||||
|
||||
A 520 h / 324 GiB library is therefore ~73 h of wall clock, not the ~40 h a bitrate-based
|
||||
guess suggests. Encode order is a real choice: newest-first finishes the currently-watched
|
||||
content in hours instead of days.
|
||||
|
||||
## "Same quality" is not the same cq number (measured)
|
||||
|
||||
`hevc_nvenc -cq 27` and `libx265 -crf 27` are **not the same quality**. Same number,
|
||||
different scale: on one 1080p25 source (source video 1.20 Mbps), 8-minute window, video only:
|
||||
|
||||
| setting | Mbps | % of source video | SSIM |
|
||||
|---|---|---|---|
|
||||
| source H.264 (x264 2-pass) | 1.20 | 100% | ref |
|
||||
| hevc_nvenc cq27 | 1.31 | **109%** | 0.9971 |
|
||||
| hevc_nvenc cq30 | 0.91 | 76% | 0.9963 |
|
||||
| libx265 crf24 | 0.83 | 69.5% | 0.9963 |
|
||||
| hevc_nvenc cq32 | 0.69 | 58% | 0.9955 |
|
||||
| libx265 crf27 | 0.58 | 48% | 0.9952 |
|
||||
| hevc_nvenc cq34 | 0.54 | 45% | 0.9947 |
|
||||
|
||||
**Compare only at matched SSIM, never at matched cq/crf numbers.** Matched pairs here:
|
||||
NVENC cq30 vs x265 crf24 (both 0.9963) -> x265 9% smaller; NVENC cq32 vs x265 crf27
|
||||
(0.9955/0.9952) -> x265 16% smaller. So software is **9-16% more bit-efficient at equal
|
||||
quality**, not the 2-2.5x that a raw cq27-vs-crf27 comparison implies - most of that gap is a
|
||||
quality difference. Price: ~3x realtime (x265 fast, 1080p) vs 6.4x (NVENC).
|
||||
|
||||
**Why a library can grow instead of shrink.** On sources that are already efficiently
|
||||
encoded H.264, NVENC cq27 is a *higher* quality target than the source itself, so the video
|
||||
comes back at ~109% and the file grows. Quality must be pinned to a number, e.g. "SSIM >= 0.996
|
||||
against the source", which is the only self-consistent definition when the source is already
|
||||
lossy and no master exists to measure against. SSIM >= 0.996 lands the video near 70% of the
|
||||
source bitrate - that is where the actual saving lives.
|
||||
|
||||
**Classify the source before choosing a setting.** bpp = bitrate / (w * h * fps) measures how
|
||||
much fat is left, and it is readable in milliseconds:
|
||||
|
||||
| bpp | meaning |
|
||||
|---|---|
|
||||
| < 0.015 | already soft - re-encoding gains almost nothing, prefer `-c copy` |
|
||||
| 0.015-0.025 | lean (typical streaming) |
|
||||
| 0.025-0.040 | normal |
|
||||
| > 0.040 | real headroom - worth a real re-encode |
|
||||
|
||||
One library measured 131 files < 0.025 against 243 files > 0.040 out of 572 - a single global
|
||||
cq cannot be right for both. Calibrate per class (1-2 representatives, 60-120 s sample, sweep 2-3
|
||||
cq values, keep the cheapest that hits the SSIM target); per-file calibration costs 1-2 min each.
|
||||
A `-maxrate` of 1.0x the source bitrate is a cheap hard guarantee that output never grows.
|
||||
|
||||
**Audio must match the source, not a fixed 256k.** A weekly series shipping mp3 at 128 kbps was
|
||||
re-encoded to `-b:a 256k`: that added ~128 kbps for no audible gain, ~+10% file size on a
|
||||
1.3 Mbps file - enough to cancel the video saving on its own. AAC is more efficient than mp3, so
|
||||
`-b:a` = the source's own audio bitrate (clamped, e.g. 96-256k) is already a quality upgrade.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Never write the encode output directly to a network share.** A streaming `.part` write over
|
||||
@@ -163,6 +275,25 @@ For a TubeSync library specifically, prefer its own DB-driven pipeline (`tubesyn
|
||||
bash cannot hand a Chinese filename to native `ffmpeg.exe` even when `ls` shows it.
|
||||
- **`-hwaccel cuda` alone may still software-decode VP9**; `-c:v vp9_cuvid` forces NVDEC.
|
||||
|
||||
## The owner's standing requirements — read this first
|
||||
|
||||
When the ask is "convert this library to HEVC, same look, same sound, playable on iPad+Android,
|
||||
never bigger, no judder", the defaults here are **not** enough: a fixed `-cq 27` targets a quality
|
||||
*above* an already-efficient H.264 source, so the library grows instead of shrinking.
|
||||
|
||||
| requirement | the answer |
|
||||
|---|---|
|
||||
| never bigger than the source | `-maxrate` = 1.0x the source **video** bitrate + a post-encode size assertion |
|
||||
| same picture quality | pin it to a number: **SSIM >= 0.996** vs the source (video lands near 70% of the source bitrate) |
|
||||
| same sound quality | copy AAC; for mp3/opus, AAC at **the source's own bitrate**, never a fixed 256k |
|
||||
| iPad + Android | MP4 + `hvc1` + `yuv420p` + AAC |
|
||||
| no judder | `-fps_mode cfr` on every re-encode, the three cadence defects gated, and **no `*_cuvid` decoders** |
|
||||
| verification | mandatory, and it must assert on the output, not on the input |
|
||||
| CPU encoding | allowed, but **tell the owner first** and quote the wall clock (~3x realtime vs 6.4x) |
|
||||
|
||||
Full policy, including the measured numbers, the source-classification table, the judder detector
|
||||
table and every CPU/threading knob: **`references/hevc-conversion-policy.md`**.
|
||||
|
||||
## Support files
|
||||
|
||||
- `scripts/transcode.py` — the tool. `--dry-run`, `--codec`, `--cq/--crf`, `--audio`,
|
||||
@@ -171,3 +302,6 @@ For a TubeSync library specifically, prefer its own DB-driven pipeline (`tubesyn
|
||||
the HEVC level caveat, and what breaks on each device.
|
||||
- `references/verification-and-automation.md` — the defect classes, the structural gate, the
|
||||
browser arbiter and its blind spot, and the 2x2 causality proof.
|
||||
- `references/hevc-conversion-policy.md` — the owner's standing requirements for a HEVC library job
|
||||
(size ceiling, SSIM-pinned quality, bpp classification, audio matched to the source, the three
|
||||
judder classes, the mandatory verification chain, and the CPU/throttling knobs, efficiency first).
|
||||
|
||||
@@ -0,0 +1,271 @@
|
||||
# HEVC conversion policy — never bigger, same look, no judder
|
||||
|
||||
Owner's standing requirements for a HEVC conversion. Read this before starting a library job;
|
||||
the default settings in `transcode.py` do **not** satisfy them.
|
||||
|
||||
## 1. Size: shrink where there is headroom, growth is acceptable where there is not
|
||||
|
||||
**Playability on iPad + Android is the primary goal. Size is secondary and conditional** — it is
|
||||
NOT a universal "must never grow" rule. Two regimes, and they need opposite handling:
|
||||
|
||||
| regime | detector | expectation | `-maxrate` | output assertion |
|
||||
|---|---|---|---|---|
|
||||
| **headroom** — an ordinary H.264 encode that is not yet at the transparency floor | bpp > ~0.030, or a sampled pre-flight encode needs **fewer** bits than the source | output **should be smaller**. If it grows, that is a **recipe bug**, not an acceptable outcome | 1.0x the source **video** bitrate | `out <= src` (hard) |
|
||||
| **already compressed** — YouTube / streaming downloads, near-optimal 2-pass sources | bpp < ~0.030, or a sampled pre-flight encode needs **>=** the source bits | growth is **acceptable**. The file already sits at the transparency floor; only playability matters | 1.5x the source video bitrate | `out <= 1.25x src` (runaway-bloat guard only) |
|
||||
|
||||
The authority is the **sampled pre-flight** (section 2): encode a 60-120 s window at the target
|
||||
quality and compare bits. bpp is just the cheap prior that tells you which way you expect it to go.
|
||||
|
||||
**Do not clamp to 1.0x on an already-compressed source.** Capping at the source bitrate forces the
|
||||
encoder *below* the transparency point, which is a visible quality loss **on top of** the
|
||||
second-generation loss — strictly worse than accepting a slightly larger file. A hard cap is only
|
||||
correct in the headroom regime, where shrinking is the actual goal.
|
||||
|
||||
Why an already-compressed source can legitimately grow: HEVC's ~40% efficiency advantage is
|
||||
measured against H.264 at *equal encoding effort*. A near-optimal 2-pass AVC stream is already
|
||||
closer to the floor than a single-pass hardware HEVC encode is, so at matched perceived quality
|
||||
there may simply be no bits left to win.
|
||||
|
||||
### An H.264 source with headroom is expected to come out smaller
|
||||
|
||||
When the source is ordinary H.264 with real headroom and the output still grows, the recipe is
|
||||
wrong, not the source. The two measured causes, both in this history:
|
||||
|
||||
- a quality target **above the source** (`-cq 27` on NVENC = SSIM 0.9971 while the source is
|
||||
already at ~0.9963-equivalent), landing the video at 109% of the source bitrate;
|
||||
- audio re-encoded to a **fixed 256k** when the source ships mp3 at 128k: ~+10% for no audible gain.
|
||||
|
||||
### HEVC is not what makes a file playable
|
||||
|
||||
H.264 + AAC in MP4 **already plays on both iPad and Android** — no conversion needed. HEVC is only
|
||||
*required* for sources that are not already in that shape: Matroska, Opus/Vorbis audio, DivX/Xvid,
|
||||
MPEG-TS, or a broken frame cadence. For an already-playable, already-lean H.264 file the only
|
||||
benefit is size; where there is no headroom a re-encode buys nothing and costs a generation, so
|
||||
`-c copy` (or leaving it alone) is the better answer. Say this to the owner before spending days of
|
||||
GPU/CPU on files that are already fine.
|
||||
|
||||
The two derived values every rule below needs, from one ffprobe of the source:
|
||||
|
||||
```python
|
||||
src_video_bps = format_bit_rate - audio_stream_bit_rate # NOT the file's total bitrate
|
||||
src_audio_kbps = max(96, min(256, audio_stream_bit_rate // 1000))
|
||||
```
|
||||
|
||||
Comparing against the file's **total** bitrate instead of the video's is the specific mistake to
|
||||
avoid: it turned a real 109% growth into a false 86% shrink in one analysis.
|
||||
|
||||
## 2. Same picture quality — pin it to a number
|
||||
|
||||
"Same quality" is meaningless unless it is measurable, and with an already-lossy source and no
|
||||
master, the only self-consistent definition is **SSIM against the source**.
|
||||
|
||||
- Target **SSIM >= 0.996** on a sampled window. Measured on 1080p25 H.264: this lands the video
|
||||
near **70% of the source bitrate** — that is where the saving is. `cq 27` on NVENC gives 0.9971
|
||||
and *grows*; `cq 30` gives 0.9963 at 76%; `cq 32` gives 0.9955 at 58%.
|
||||
- **Never compare cq/crf numbers across encoders.** `-cq 27` (NVENC) = SSIM 0.9971 while
|
||||
`-crf 27` (x265) = 0.9952. Match on SSIM, then compare bitrate.
|
||||
- Software x265 is **9-16% more bit-efficient at matched SSIM**, not 2x — most of the raw
|
||||
difference between cq27 and crf27 is a quality difference, not an efficiency difference.
|
||||
- Verify the claim per file (or per class): encode a 60-120 s window, measure SSIM vs the same
|
||||
source window, and fail the item if it is under target.
|
||||
|
||||
## 3. Classify the source before choosing a setting
|
||||
|
||||
`bpp = video_bitrate / (w * h * fps)`, readable in milliseconds, tells you how much fat is left:
|
||||
|
||||
| bpp | verdict |
|
||||
|---|---|
|
||||
| < 0.015 | already soft — re-encoding gains nothing, use `-c copy` |
|
||||
| 0.015-0.025 | lean |
|
||||
| 0.025-0.040 | normal |
|
||||
| > 0.040 | real headroom — worth a real re-encode |
|
||||
|
||||
One library: 131 files < 0.025 vs 243 files > 0.040 out of 572. A single global quality setting
|
||||
cannot be right for both — that is precisely how a job ends up growing some files and shrinking
|
||||
others. Calibrate per class on 1-2 representatives (60-120 s, sweep 2-3 quality levels).
|
||||
|
||||
## 4. Same sound quality
|
||||
|
||||
- Source audio already **AAC** -> `-c:a copy`, bit-identical. Always preferable.
|
||||
- Source audio **mp3/opus/vorbis** -> must be re-encoded; those are not valid in MP4.
|
||||
- **`-b:a` must track the source, never a fixed 256k.** A weekly series shipping mp3 at 128 kbps
|
||||
re-encoded to 256k AAC added ~128 kbps for zero audible gain: ~+10% file size, enough to cancel
|
||||
the video saving by itself. AAC is more efficient than mp3, so **matching** the source bitrate is
|
||||
already an upgrade. Clamp to a sane floor (96k).
|
||||
|
||||
## 5. Playable on iPad AND Android
|
||||
|
||||
MP4 + `-tag:v hvc1` (Apple rejects `hev1`) + `-pix_fmt yuv420p` + AAC. No Matroska, never Opus.
|
||||
HEVC 1080p is Main profile Level 4.0 — every modern iPhone/iPad (A9+) and Android 5+ handles it,
|
||||
but a cheap/old Android device is only guaranteed Main Level 3 (~SD).
|
||||
|
||||
## 6. No judder — three defect classes, same symptom
|
||||
|
||||
`-fps_mode cfr` is mandatory on every re-encode: with the default `auto`, ffmpeg **drops** frames
|
||||
it cannot place (0.5-1.3% measured), leaving doubled frame intervals that play as judder. CFR
|
||||
duplicates a frame instead, which costs almost nothing. Never pass `-r` unless you want a
|
||||
different playing speed.
|
||||
|
||||
| class | signature | detector | fix |
|
||||
|---|---|---|---|
|
||||
| source-inherited VFR | `r_frame_rate != avg_frame_rate` | ffprobe header, gap > 0.01 | re-encode (remux cannot repair) |
|
||||
| encoder-dropped | doubled intervals (0.040 + 0.080) | histogram `packet=duration_time` shows 2+ values | `-fps_mode cfr` |
|
||||
| **flat composition offsets** | `r == avg` **exactly**, uniform `stts`, clean decode, still drops ~23% | `ctts` entry count <= 1 on a B-frame stream, or 0 PTS inversions over 60 s | real re-encode; **no lossless repair exists** |
|
||||
|
||||
The third class is the trap: every cadence test passes and the file still stutters on real
|
||||
hardware. Gate it explicitly (see `scripts/transcode.py`: `_ctts_entries()`, `pts_inversions()`).
|
||||
|
||||
**Never decode with a `*_cuvid` hardware decoder.** Measured on this hardware (RTX 3060,
|
||||
ffmpeg 9.0.1): `av1_cuvid` emitted **the wrong picture on 1.8% of frames** (a frame from a
|
||||
different moment, not a corrupt one). The container stayed perfect — exact intervals, correct
|
||||
`ctts`, clean decode — so every structural check passed while the output visibly jumped. Use
|
||||
software decode (4K AV1 ran ~150 fps on the i9-13900K) or the modern
|
||||
`-hwaccel cuda -hwaccel_output_format cuda` path, and if you ever must use cuvid, compare its
|
||||
frames against the software decode with per-frame SSIM (>=0.99 everywhere).
|
||||
|
||||
**Judge the picture, not the container, when the complaint is "it jumps".**
|
||||
|
||||
## 7. Verification — yes, mandatory, and it must assert on the output
|
||||
|
||||
Per item, in this order:
|
||||
|
||||
1. **Local staging.** Fetch the source off the share first; never encode over SMB/WebDAV/Dokan.
|
||||
2. **Structural gate** on the encode: codec/tag/duration/resolution/pixel format, `r` vs `avg`
|
||||
within 0.01, `ctts`/PTS inversion check, one distinct frame duration.
|
||||
3. **Full decode pass** (`ffmpeg -v error -i OUT -f null -`) — the only check that proves the
|
||||
payload is not truncated or corrupt.
|
||||
4. **SSIM vs the source window >= target** (section 2).
|
||||
5. **Size assertion**: output video bitrate <= source (section 1).
|
||||
6. **Upload to a temp name, read the share copy back end-to-end, compare SHA-256 + byte count**,
|
||||
then replace atomically. A duration probe is not enough: with `+faststart` the `moov` is written
|
||||
first, so a truncated upload still reports the full duration.
|
||||
7. **Re-probe on the share** after the replace.
|
||||
|
||||
Cheap confidence check on a healthy file, one distinct value expected:
|
||||
|
||||
```sh
|
||||
ffprobe -v error -select_streams v:0 -read_intervals "%+60" \
|
||||
-show_entries packet=duration_time -of csv=p=0 FILE | sort -u | wc -l
|
||||
```
|
||||
|
||||
**Do not hand-sample videos before converting** — nothing here is spec-dependent, so the guarantee
|
||||
belongs in the output assertion. Chrome cannot arbitrate HEVC either: `canPlayType('...hvc1...')`
|
||||
returns `''`, so an HEVC file reports 0 decoded / 0 dropped and looks perfect while nothing played.
|
||||
HEVC smoothness is confirmable only on the Apple device, via the frame-duration histogram, or on
|
||||
an Android player. For an owner-facing A/B, serve the original and the variant side by side in a
|
||||
real browser and let them compare (the H.264-only variant is the one Chrome can measure).
|
||||
|
||||
## 8. CPU (software) encoding — every knob, and what each one costs
|
||||
|
||||
**Remind the owner before starting a CPU job and quote the wall-clock time.** Software x265 runs
|
||||
3.5-4.4x realtime at 1080p depending on cores and preset (vs 6.4x for `hevc_nvenc`), so a 520 h
|
||||
library is days of CPU. Say so before starting, not after — see the measured table in the topology
|
||||
section and the tier timings in section 9.
|
||||
|
||||
Efficiency first, and the knobs do different jobs:
|
||||
|
||||
| goal | knob | effect |
|
||||
|---|---|---|
|
||||
| better bits-per-quality | `-preset medium|slow|slower` | **the only lever that buys efficiency**; each step ~5-8% smaller at equal quality, ~1.5-2x slower |
|
||||
| fewer cores | `-x265-params pools=N` (N threads) — for `libx265`, ffmpeg's `-threads N` maps to frame threads | fewer cores, roughly linear slowdown, **no efficiency gain** |
|
||||
| fewer frame threads | `-x265-params frame-threads=2` | less parallelism, marginal efficiency *gain*, slower |
|
||||
| CPU affinity | `start /affinity <hexmask> ffmpeg ...`, or PowerShell `(Get-Process ffmpeg).ProcessorAffinity` | pins to chosen cores; keeps the machine responsive |
|
||||
| low priority | PowerShell `(Get-Process ffmpeg).PriorityClass='BelowNormal'` | yields to everything else at no efficiency cost |
|
||||
| **no high frequency** | `powercfg /setacvalueindex SCHEME_CURRENT SUB_PROCESSOR PROCTHROTTLEMAX 60` then `powercfg /setactive SCHEME_CURRENT` | caps clocks: best perf/watt, quietest, slowest — match `pools` to the capped cores |
|
||||
|
||||
**Preset buys efficiency; thread count does not.** If the priority is efficiency rather than speed,
|
||||
prefer a slower preset at a lower core count over a faster preset with all cores. Never run two
|
||||
encoders of the same kind at once: they compete for the same unit and drop frames — the exact
|
||||
defect the gates exist for. One CPU encoder running *alongside* one GPU encoder is fine (different
|
||||
units), see section 9.
|
||||
|
||||
### This host's measured topology and the P-core-avoidance recipe
|
||||
|
||||
Windows reports the i9-13900K VM as "13 cores / 26 logical" — a misreport of a heterogeneous
|
||||
topology. A per-logical-CPU microbenchmark settles it in 30 seconds:
|
||||
|
||||
| logical CPU | relative single-thread throughput |
|
||||
|---|---|
|
||||
| **0-15** (8 P-cores x HT) | 0.43-0.52 s for a fixed workload |
|
||||
| **16-25** (10 E-cores) | 0.79-0.91 s — **56% of a P-core** |
|
||||
|
||||
Pin to the E-cores only (best perf/watt, keeps the P-cores free for the desktop):
|
||||
|
||||
```sh
|
||||
# affinity mask = bits 16-25 set = 0x3FF0000 (= 67,043,328)
|
||||
start /affinity 3FF0000 ffmpeg ... # cmd
|
||||
```
|
||||
or set the mask on the driving Python process (children inherit it):
|
||||
|
||||
```python
|
||||
ctypes.windll.kernel32.SetProcessAffinityMask(
|
||||
ctypes.windll.kernel32.GetCurrentProcess(), ctypes.c_size_t(0x3FF0000))
|
||||
```
|
||||
|
||||
Match the thread count to the mask (`-x265-params pools=10`). **`pools` does not choose cores —
|
||||
the affinity mask does**; `pools` only caps threads.
|
||||
|
||||
Measured cost of E-core-only encoding (x265 preset medium, 1080p25, 120 s sample):
|
||||
|
||||
| cores | speed | output |
|
||||
|---|---|---|
|
||||
| all 26 threads | 4.37x realtime | 0.76 Mbps |
|
||||
| **E-cores only (10 threads)** | **3.51x realtime** | 0.76 Mbps |
|
||||
|
||||
Only **20% slower**, not the ~3x that per-core throughput suggests: the 10 E-cores are real cores
|
||||
while half of the P-core threads are SMT siblings, and x265 does not scale linearly anyway. So
|
||||
E-cores-only is the right default when the priority is efficiency over speed.
|
||||
|
||||
## 9. CPU and GPU tiers in parallel
|
||||
|
||||
Different tiers can encode **at the same time** — NVENC is a fixed-function ASIC, x265 is CPU.
|
||||
Pair an E-core-pinned CPU job with a GPU job and the P-cores stay free for the user. Wall clock
|
||||
becomes `max(cpu_tier, gpu_tier)` instead of the sum:
|
||||
|
||||
| tier | files | GiB | content h | recipe | time |
|
||||
|---|---|---|---|---|---|
|
||||
| fat (bpp > 0.040) | 252 | 153 | 219 | x265 crf24, E-cores | 65 h |
|
||||
| mid (0.025-0.040) | 198 | 117 | 183 | hevc_nvenc cq30 | 31 h |
|
||||
| lean (< 0.025) | 131 | 54 | 118 | `-c copy` + audio match | <1 h |
|
||||
|
||||
serial 96 h (4.0 days) -> parallel **66 h (2.7 days)**. Requirements to run it safely:
|
||||
|
||||
- **a lock per tier**, not one global lock, and a separate local staging dir per instance;
|
||||
- the share is the shared resource: two instances double the WebDAV/Dokan traffic (staging read +
|
||||
upload + read-back hash). Test a small batch in parallel before committing the library;
|
||||
- run the fast `-c copy` tier on its own first, so it is not contending while the two big jobs settle;
|
||||
- `-c copy` only where the timing gate passes — a file in the flat-ctts class must be re-encoded
|
||||
whatever its bpp says.
|
||||
|
||||
## 10. The three recipes, ready to run
|
||||
|
||||
`SRC` = the local staged source, `SRC_V` = `src_video_bps` from section 1, `A` = `src_audio_kbps`
|
||||
(section 4). Affinitize the process to the E-cores (`0x3FF0000`) before the tier-C command.
|
||||
|
||||
```sh
|
||||
# ---- tier A: lean (bpp < 0.025) — no video re-encode at all, size cannot change
|
||||
ffmpeg -y -i SRC -map 0:v:0 -map 0:a:0? -map_metadata 0 \
|
||||
-c:v copy -c:a copy -movflags +faststart OUT.mp4 # -c:a aac -b:a A when the source is mp3
|
||||
|
||||
# ---- tier B: mid (0.025-0.040) — GPU, quality pinned to SSIM >= 0.996
|
||||
ffmpeg -y -i SRC -map 0:v:0 -map 0:a:0? -map_metadata 0 \
|
||||
-c:v hevc_nvenc -preset p6 -tune hq -rc vbr -cq 30 -b:v 0 \
|
||||
-maxrate SRC_V -bufsize <2.5x SRC_V> \
|
||||
-profile:v main -pix_fmt yuv420p -tag:v hvc1 -fps_mode cfr \
|
||||
-c:a copy -movflags +faststart OUT.mp4 # -c:a aac -b:a A when the source is mp3
|
||||
|
||||
# ---- tier C: fat (bpp > 0.040) — CPU, E-cores only, efficiency first
|
||||
ffmpeg -y -i SRC -map 0:v:0 -map 0:a:0? -map_metadata 0 \
|
||||
-c:v libx265 -preset medium -crf 24 -x265-params pools=10 \
|
||||
-maxrate SRC_V -bufsize <2.5x SRC_V> \
|
||||
-pix_fmt yuv420p -tag:v hvc1 -fps_mode cfr \
|
||||
-c:a copy -movflags +faststart OUT.mp4
|
||||
```
|
||||
|
||||
Why these exact numbers: `cq 30` (NVENC) and `crf 24` (x265) both land near **SSIM 0.9963** — the
|
||||
matched-quality point where the video comes back at ~70-76% of the source bitrate. Neither is
|
||||
`cq 27`/`crf 27`: those are different qualities from each other, and `cq 27` is *above* an
|
||||
already-efficient source, which is what made a library grow.
|
||||
|
||||
Every tier still runs the whole verification chain from section 7 — tier A included: a `-c copy`
|
||||
can still originate a truncation, and a flat-ctts source cannot be copied at all.
|
||||
Reference in New Issue
Block a user