universal-video-transcode/SKILL.md

name, description, version, author, license, platforms, metadata

name	description	version	author	license	platforms	metadata
universal-video-transcode	Use when transcoding video for iPad+Android playback.	1.0.0	Hermes Agent	MIT	linux windows	hermes tags related_skills ffmpeg hevc h264 mp4 ipad ios android transcode nvenc cfr frame-timing tubesync-selfhosted local-audio-to-text

Universal video transcode — anything in, one MP4 that plays on iPad AND Android

One tool: scripts/transcode.py. Hand it a path; a file converts, a folder converts everything in it. It reads the source, picks the recipe itself, encodes, and verifies the result before accepting it.

python scripts/transcode.py "D:/video/clip.mkv"
python scripts/transcode.py "D:/video" --recursive --dry-run   # plan only
python scripts/transcode.py "D:/video" --out D:/converted

Scope — read this before applying the size policy

This policy is for H.264 (and older MPEG-4/DivX) sources, where the source is fat enough that HEVC at matched quality lands 25-50% smaller (the course-library case: 1080p H.264 at 550-2400 kbps, mp3 audio, files 300 MB-1.3 GB).

Do NOT run the size hunt on YouTube-sourced libraries whose codec is VP9/AV1. YouTube already serves those at a near-optimal bitrate for their quality, so re-encoding an already-lean VP9/AV1 file can only grow it or lose quality - "the source is already as small as it gets, and that is accepted". YouTube's H.264 (AVC1) ladder is a different story, and worth measuring before dismissing: the compatibility encodes are fat (measured median bpp 0.053 on one library's 1080p AVC1 files, 2.4-4.7 Mbps), and they shrank to 34-67% of the source video bitrate at matched quality (min SSIM ~0.99) - see section 11 of the policy reference. Classify by bpp and treat them like any other H.264 source; only the VP9/AV1 half is "no shrink sought".

	H.264 / AVC1 source (this policy)	VP9 / AV1 source (YouTube)
goal	same look, smaller file	same look, same size - no shrink sought
quality gate	SSIM pinned + size ceiling (min ~0.99 is the usable frontier)	per-frame content check vs the software decode
source for a repair	the file itself	the pre-conversion snapshot backup, never the already-converted file
frame cadence	-fps_mode cfr	-fps_mode cfr
decode	any correct decoder	never *_cuvid (wrong frames, see below)

What "playable everywhere" actually means

The intersection that both platforms decode without a third-party player:

	iPad / iOS	Android
Container	MP4/MOV only — no Matroska	MP4 and MKV both fine
Video	H.264 (all), HEVC hvc1 (A9+)	H.264 Baseline mandatory; HEVC 5.0+
Audio	AAC — never Opus	AAC and Opus both fine

Apple is the constraint, not Android. iOS has no Matroska demuxer and cannot play Opus at all, so MP4 + AAC is what forces the conversion. Android would have accepted the .mkv.

Two facts that decide the video codec:

• HEVC must be tagged hvc1. ffmpeg defaults to hev1, which Apple refuses. -tag:v hvc1 is not optional.
• Android only guarantees HEVC Main profile Level 3 (~SD). 1080p output is Level 4.0 and 4K is 5.0 — fine on modern phones, but a low-end/old device may only reach Level 3. When the audience includes cheap Android hardware, prefer H.264 (--codec x264), which is mandatory from Android 3.0 and universal on iOS.

Full matrix, per-platform citations, and the HDR/10-bit caveats: references/compatibility-matrix.md.

The three recipes it picks automatically

Decided from the source file, never from its filename or extension:

recipe	when	video	audio
copy	H.264/HEVC with a usable timeline	-c copy — bit-identical	copy if AAC
fix-audio	same, audio is not AAC	-c copy	AAC 256k
encode	VP9/AV1, or H.264/HEVC whose timeline is unusable	HEVC (or x264)	copy if AAC, else AAC 256k

copy is used whenever a copy is possible, because a copy is lossless by definition — that is the strongest possible answer to "keep the quality the same".

Quality policy ("as close to the original as possible")

• Never upscale. The source resolution is preserved; there is no scale in the pipeline.
• HEVC targets -cq 27, measured on this hardware at SSIM 0.9937 (visually transparent) while landing at ~source size. -cq 28-30 buys 14-32% smaller at SSIM still >=0.992.
• H.264 (when chosen for reach) targets -crf 20.
• Audio: copied when already AAC (bit-identical); otherwise AAC 256k, where the residual error is the source's own lossy error — 320k buys nothing measurable.
• Bitrate cap: -maxrate = 1.5x the source bitrate, -bufsize = 2.5x that cap. Never set bufsize equal to maxrate (a 1-second VBV buffer starves the encoder and collapses quality on complex scenes); never cap at exactly the source bitrate (that pushes quality below the original — a second-generation loss).

The two flags that must never be dropped

-fps_mode cfr      # without it the encoder silently drops 0.5-1.3% of frames,
                   # leaving doubled frame intervals that play as judder
-pix_fmt yuv420p   # 8-bit 4:2:0; 10-bit / 4:2:2 / 4:4:4 break device compatibility

-fps_mode cfr makes ffmpeg duplicate a frame it cannot place instead of dropping it. Duplicated frames cost almost nothing; a gap is visible. The tool adds both automatically and fails the item if the output comes back non-uniform.

Verifying "no dropped frames" — and whether a browser is needed

Short answer: the browser is the arbiter, but it is NOT the gate. Automate the causal check; use the browser only to spot-check.

The user-visible symptom (choppy playback) comes from a deterministic property of the container: whether the frame timestamps are in presentation order or decode order. That is readable from the header, so it needs no playback at all.

The defect this matters for: a B-frame stream whose MP4 composition-offset table is unusable — either a single constant ctts entry (PTS == DTS + const) or no ctts box at all (PTS == DTS). The decoder still reorders by POC, so the timestamps attached to the output frames are wrong and the compositor drops ~23% of them — while r_frame_rate == avg_frame_rate stays EXACTLY equal, so every fps-based test calls the file healthy.

gate	cost	covers HEVC?	role
r_frame_rate vs avg_frame_rate	~50 ms	yes	catches dropped/irregular cadence only
ctts / PTS-inversion structural check	~50 ms	yes	the automatable gate — causal
full decode (ffmpeg -f null -)	minutes	yes	decode integrity
real Chrome droppedVideoFrames	~30 s/file	NO	arbiter for H.264 spot-checks

Three traps that make the browser check unsuitable as an automated gate:

1. Chrome has no HEVC decoder. canPlayType('video/mp4; codecs="hvc1…"') returns '', so an HEVC file reports 0 decoded / 0 dropped and looks perfect while nothing played. A 0-frame result is not a pass. HEVC can only be confirmed on the Apple device itself.
2. It needs a real Chrome and a real page; ~30 s per file is far too slow for a library.
3. The measurement is statistical (a 1-frame drop is noise), so it needs a threshold, not equality.

The tool therefore runs the structural gate on every file automatically, plus a full decode pass, and refuses the output otherwise. Measurements, the exact thresholds, and the 2x2 experiment that proved causality: references/verification-and-automation.md.

Running it unattended (cron)

Safe to schedule, because it is idempotent and self-locking:

• A lock file (default ~/.transcode.lock) refuses to start a second instance. This is not optional hygiene: two concurrent NVENC jobs share one encoder unit, starve each other, and drop frames — the exact defect the tool exists to prevent.
• Idempotent: a file that is already a playable MP4 is skipped, so a re-run costs a probe and nothing else. --dry-run prints the plan without encoding.
• A failed item leaves the input untouched, so the next run simply retries it.
• Disk guard: aborts below 5 GB free rather than filling the work drive (an ENOSPC mid-encode is reported as a misleading codec error and costs every remaining item).

# hourly; exits non-zero if anything failed
python /path/to/scripts/transcode.py "Z:/docker/tubesync/download" --recursive \
  --out "Z:/docker/tubesync/converted" >> /var/log/transcode.log 2>&1

For a TubeSync library specifically, prefer its own DB-driven pipeline (tubesync-selfhosted) — it also updates the database rows. Use this tool for anything outside that library.

Whole-library runs on a network share (in-place, safe)

transcode.py writes dst + ".part.mp4" next to the source. Over SMB/WebDAV/Dokan that is the failure mode to avoid, and when the source is already .mp4 then dst == src, so the only copy of the file is the one being written. For a library run, drive ffmpeg directly from a small runner instead:

1. copy the source off the share to local disk (chunked; treat a short read as failure)
2. encode locally, with the same mandatory flags (-fps_mode cfr, -pix_fmt yuv420p)
3. run the structural gate and a full decode pass on the local output
4. upload to <name>.new.mp4 (never the final name)
5. read the share copy back end-to-end and compare SHA-256 + byte count with the local file
6. only then os.replace() it over the original, and re-probe the result on the share

Step 5 costs one extra network read (~1 h per 330 GiB) and is the only thing that actually proves the remote copy is complete — getsize() cannot, and neither can a duration probe: with +faststart the moov is written first, so a truncated file still reports full duration. Keep a JSONL ledger and a per-file skip test ("already HEVC hvc1") so a reboot resumes safely. A single instance only; the lock matters even more here because two runs would race on dst.

plan_for() gaps to know before a library run

• mpeg4 (DivX/Xvid) returns None = "unsupported video codec" — it is skipped, not converted, so .avi/mp4v files silently keep a codec no iPad can play. Give them an explicit re-encode.
• Repairing a defective H.264 timeline returns x264, not HEVC (hardcoded in the H.264 branch). If the requirement is "everything in HEVC", force it — the recipe is a policy, not the tool's default.
• A truncated file (moov atom not found) is a re-download, not a transcode job; report it instead of retrying.

Measured throughput (RTX 3060, hevc_nvenc p6, -tune hq, cq 27)

Use these to quote an ETA before starting — the estimate is the user's decision input.

source	speed
1080p25 H.264	6.4x realtime (~160 fps)
480x270	~34x realtime
per-file overhead	~45 s (local copy + decode pass + upload + read-back hash)

A 520 h / 324 GiB library is therefore ~73 h of wall clock, not the ~40 h a bitrate-based guess suggests. Encode order is a real choice: newest-first finishes the currently-watched content in hours instead of days.

"Same quality" is not the same cq number (measured)

hevc_nvenc -cq 27 and libx265 -crf 27 are not the same quality. Same number, different scale: on one 1080p25 source (source video 1.20 Mbps), 8-minute window, video only:

setting	Mbps	% of source video	SSIM
source H.264 (x264 2-pass)	1.20	100%	ref
hevc_nvenc cq27	1.31	109%	0.9971
hevc_nvenc cq30	0.91	76%	0.9963
libx265 crf24	0.83	69.5%	0.9963
hevc_nvenc cq32	0.69	58%	0.9955
libx265 crf27	0.58	48%	0.9952
hevc_nvenc cq34	0.54	45%	0.9947

Compare only at matched SSIM, never at matched cq/crf numbers. Matched pairs here: NVENC cq30 vs x265 crf24 (both 0.9963) -> x265 9% smaller; NVENC cq32 vs x265 crf27 (0.9955/0.9952) -> x265 16% smaller. So software is 9-16% more bit-efficient at equal quality, not the 2-2.5x that a raw cq27-vs-crf27 comparison implies - most of that gap is a quality difference. Price: ~3x realtime (x265 fast, 1080p) vs 6.4x (NVENC).

Why a library can grow instead of shrink. On sources that are already efficiently encoded H.264, NVENC cq27 is a higher quality target than the source itself, so the video comes back at ~109% and the file grows. Quality must be pinned to a number, e.g. "SSIM >= 0.996 against the source", which is the only self-consistent definition when the source is already lossy and no master exists to measure against. SSIM >= 0.996 lands the video near 70% of the source bitrate - that is where the actual saving lives.

Classify the source before choosing a setting. bpp = bitrate / (w * h * fps) measures how much fat is left, and it is readable in milliseconds:

bpp	meaning
< 0.015	already soft - re-encoding gains almost nothing, prefer -c copy
0.015-0.025	lean (typical streaming)
0.025-0.040	normal
> 0.040	real headroom - worth a real re-encode

One library measured 131 files < 0.025 against 243 files > 0.040 out of 572 - a single global cq cannot be right for both. Calibrate per class (1-2 representatives, 60-120 s sample, sweep 2-3 cq values, keep the cheapest that hits the SSIM target); per-file calibration costs 1-2 min each. A -maxrate of 1.0x the source bitrate is a cheap hard guarantee that output never grows.

Audio must match the source, not a fixed 256k. A weekly series shipping mp3 at 128 kbps was re-encoded to -b:a 256k: that added ~128 kbps for no audible gain, ~+10% file size on a 1.3 Mbps file - enough to cancel the video saving on its own. AAC is more efficient than mp3, so -b:a = the source's own audio bitrate (clamped, e.g. 96-256k) is already a quality upgrade.

• Never build a verification harness on a stream-copy cut. -ss X -c copy on an H.264 file with B-frames loses packets (measured: a 60 s cut of a 25 fps source yielded 1270 frames instead of 1500, with the content shifted after each dropped packet), so an encode of that cut can never line up with anything and the comparison reports a fake SSIM 0.25 minimum. Compare whole files (or byte-copies of them): a byte copy is faithful, a stream copy is not. Corollary - before believing any picture verdict, make the harness prove itself on a file against ITSELF (must be lo=1.0).
• Time-box a single-file mystery. When one file resists diagnosis after a couple of experiments, park it (leave it untouched, log it as 'gate could not verify'), report it, and move on - a library run must not stall on one item. Revisit parked files in a batch later.

Pitfalls

• A RaiDrive/Dokan write can return with NO error while the file is a fraction of what it should be. Measured: a 403 MB upload left 12.5 MB on the share, and RaiDrive's own "the operation didn't complete within specific time limit" toast arrived after the write had already returned success. Never trust a write to a Dokan-backed share: hash the local file, hash the share copy back, compare size AND SHA-256, and only then rename. On a mismatch, retry the upload (it usually works on the second attempt) rather than failing the item. Uploads should use small chunks (4 MB) with a periodic flush+fsync; a partial *.new.mp4/*.part.mp4 must be deleted, never resumed from os.path.getsize() on the share (RaiDrive reports stale sizes, so a resume offset taken from it corrupts the file).

• Those same toasts are an overload signal, not a permission problem. Three jobs reading/writing one share provoked both the truncated write above and [Errno 9] Bad file descriptor on a 1 GB staged read; writing into the library folder itself was perfectly fine (verified by creating and deleting a file there). Cap the concurrency at two jobs (one CPU tier + one GPU tier), give each tier its own lock file, and make the lock take over a stale one (pid gone).

• Compare pictures by TIMESTAMP, never by frame index after an input seek. An input -ss snaps to a keyframe, and it snaps to a different keyframe for the source and for the encode - measured: two 1500-frame decodes whose start times differed by ~10 s, so every frame met the wrong one and a perfectly good encode scored min SSIM 0.25. Cut the window with the trim filter on absolute PTS (-copyts -ss <before window> -i in -vf "trim=start=X:end=Y,setpts=PTS-STARTPTS,..."). Validate the method by comparing a file against ITSELF: it must read lo=1.0/mean=1.0, otherwise the harness is broken and every verdict it produces is noise.

• If the frame counts in the sampled window differ by more than ~1%, the picture gate is NOT APPLICABLE - do not fail the file. A source with holes in its timeline (measured: 1459 frames where 1500 belong, a 1.7 s hole) is supposed to gain duplicated frames from -fps_mode cfr; after the hole every frame pairs with the wrong one and SSIM collapses. Report the condition and let the structural/decode/size gates carry the verdict.

• File paths with a space before the extension are NOT the problem they look like - 1130 维特根斯坦十讲 .mp4 opened and streamed fine. Suspect the share, not the name.

• Never kill encoders by process name. Get-Process ffmpeg | Stop-Process -Force also killed an unrelated pilot encode that was running legitimately (rc=-1 mid-encode). Kill the specific background PID, or the specific ffmpeg PID you started.

• Never write the encode output directly to a network share. A streaming .part write over SMB/SFTP/Dokan leaves a corrupt stub when the connection blips. Encode to local disk, verify, then copy the complete file across. The tool writes dst + ".part.mp4" and renames only after verification — keep it on a local path if the destination is remote.

• os.path.getsize() lies on RaiDrive/Dokan — it returns cached metadata from the previous file handle. Verify a network copy by sequential chunked read, never by size.

• A .mkv extension on a converted file is cosmetic. TubeSync's rename pass rewrites finished .mp4 files back to .mkv while downloaded_container still reads mp4, and Content-Type follows the container. Do not "fix" it and do not report it as a playback bug.

• Do not re-download to fix a timing defect. The mkv that yt-dlp writes is itself flat, so a re-download reproduces the defect exactly.

• A -c copy remux cannot repair timing. ffmpeg will not rewrite stts/ctts under stream copy, and retiming only the SPS (-bsf:v h264_metadata=tick_rate=…) changes nothing either. Repairing a copy-tier file means re-encoding its video — quote the size cost (1.28x at crf 23, 1.74x at crf 20) before applying it to a whole library.

• Unicode paths: on Windows, drive ffmpeg from Python subprocess with an argv list. MSYS bash cannot hand a Chinese filename to native ffmpeg.exe even when ls shows it.

• -hwaccel cuda alone may still software-decode VP9; -c:v vp9_cuvid forces NVDEC.

The owner's standing requirements — read this first

When the ask is "convert this library to HEVC, same look, same sound, playable on iPad+Android, never bigger, no judder", the defaults here are not enough: a fixed -cq 27 targets a quality above an already-efficient H.264 source, so the library grows instead of shrinking.

requirement	the answer
size	shrink where the source has headroom; growth is acceptable for already-compressed sources (YouTube-style) — playability is the goal there
same picture quality	pin it to a number: SSIM >= 0.996 vs the source (video lands near 70% of the source bitrate)
same sound quality	copy AAC; for mp3/opus, AAC at the source's own bitrate, never a fixed 256k
iPad + Android	MP4 + hvc1 + yuv420p + AAC
no judder	-fps_mode cfr on every re-encode, the three cadence defects gated, and no *_cuvid decoders
verification	mandatory, and it must assert on the output, not on the input
CPU encoding	allowed, but tell the owner first and quote the wall clock (~3x realtime vs 6.4x)

Full policy, including the measured numbers, the source-classification table, the judder detector table and every CPU/threading knob: references/hevc-conversion-policy.md.

Support files

• scripts/library_run.py — the whole-library driver that enforces the policy above: bpp tiering (copy / nvenc / x265), -maxrate at 1.0x the source video bitrate + a size assertion, a per-frame SSIM gate with a one-step quality retry, audio matched to the source, local staging, SHA-256 read-back before an atomic replace, a JSONL ledger, a run lock, E-core pinning for the CPU tier, --tier to run tiers separately, --dry-run for the classification plan, --keep-local to pilot without touching anything, and --origin-root to redo files that an earlier bad setting bloated, sourcing them from a snapshot of the originals (never deleting it).
• scripts/transcode.py — the tool. --dry-run, --codec, --cq/--crf, --audio, --channels, --cpu, --keep, --overwrite, --no-decode-check, --lock.
• references/compatibility-matrix.md — per-platform codec/container facts with citations, the HEVC level caveat, and what breaks on each device.
• references/verification-and-automation.md — the defect classes, the structural gate, the browser arbiter and its blind spot, and the 2x2 causality proof.
• references/hevc-conversion-policy.md — the owner's standing requirements for a HEVC library job (size ceiling, SSIM-pinned quality, bpp classification, audio matched to the source, the three judder classes, the mandatory verification chain, and the CPU/throttling knobs, efficiency first).