STATE.md carries what the next session needs: the two commands that keep the tiers running, where
the logs/stage/locks/ledger live, the eight bugs this round produced (each with the measurement that
proved it), and the open items. The skill's body now keeps only the load-bearing rules; every
measured case moved to references/pitfalls.md, so the always-loaded part stays readable.
Two bugs the library run exposed on real files:
* An input seek is not a faithful decode. On an H.264 source with open GOPs the frames after the
landing point decode differently from a sequential decode - the SAME file compared with itself,
once seeked and once decoded in order, scored min 0.25 / mean 0.82, and the gate condemned good
encodes because of it. Both sides are now decoded sequentially with the window cut by the trim
filter, which is also the only decode guaranteed to match what the encoder itself saw.
* Staleness of the run lock. Killing a background session kills the wrapper but not the python
underneath, so a live run's lock survives and the next start is correctly refused (it looked like
a pid-reuse bug; it was not). The lock now carries a heartbeat, staleness is decided by age as
well as pid, and a takeover that cannot delete a file the OS still holds overwrites it in place.
Two lessons from chasing one stubborn file. A 60 s stream copy (-ss X -c copy) of an H.264 source
with B-frames lost 230 of its 1500 frames and shifted the content, so every encode of that cut
looked broken (SSIM minimum 0.25) - harnesses must sample with byte copies, never stream copies,
and must prove themselves against a file compared with itself. And when a single file resists
diagnosis, park it and report it instead of stalling a library run on one item.
Every one of these cost real time today and each has a detector now: RaiDrive returning success
on a truncated write (12.5 MB of 403 MB, hash read-back caught it), the share's toasts being an
overload signal rather than a permissions problem (3 concurrent jobs), a picture gate comparing
by frame index after an input seek (false FAIL at min SSIM 0.25 on a good encode; fixed by
trimming on PTS), frame-count differences meaning 'gate not applicable' instead of 'broken file',
a filename with a space before its extension being innocent, and killing encoders by process name
(which took out an unrelated pilot encode).
The old runner (fixed -cq 27 + fixed 256k audio + maxrate at 1.5x the FILE bitrate) left 54 files
BIGGER than their originals - the worst at 124.8%. Re-encoding those from their current HEVC would
be second-generation loss, so the source must be the pre-conversion original; a dated snapshot of
the library holds it (582 files / 324 GiB measured).
A file already HEVC is now re-done only when it is bigger than its original (>1.01x), which makes
the run idempotent with no extra state. The tier and every gate use the ORIGINAL, dst stays the
live path, and os.remove(src) is suppressed whenever the source came from the origin tree so the
archive of originals can never be deleted by a replace.
The previous runner encoded everything at a fixed -cq 27 with -maxrate = 1.5x the FILE bitrate,
which is how a library ended up at 111% of its source size. This replaces the encode step with the
policy the skill documents, and adds the gates that make it a guarantee:
* classify by bpp: <0.025 copy the video (bit-identical), 0.025-0.040 hevc_nvenc, >0.040 libx265
* -maxrate = 1.0x the source's VIDEO bitrate (never the file total) + a post-encode assertion
* SSIM gate: per-frame comparison of a 60 s window against the source, retry once at +2 quality
if it fails, then fail the item (the original is never touched)
* audio: copy when AAC, else AAC at the SOURCE's bitrate (a fixed 256k cancelled the video saving)
* x265 tier pinned to the E-cores; --tier to run tiers separately; --keep-local for a pilot that
touches nothing
* defaults cq31 / crf26 from the measurement below, not cq27
Measured on three real sources (60 s window, video-only): cq27 GROWS two of three files (115%,
119%); cq31 lands at 43-74%; x265 crf26 at 34-67% with min SSIM 0.988-0.993 - and x265 beats NVENC
by 10-22% at matched SSIM. Pinning min SSIM >= 0.996 leaves only cq27, which is bigger than the
source, so ~0.99 is the usable quality frontier and the owner gets to make that call explicitly
(references/hevc-conversion-policy.md section 11, SKILL.md scope section).
The installed skill had drifted ahead of the repo: SKILL.md carried the whole owner-requirements
section and references/hevc-conversion-policy.md was missing here entirely. Synced both.
Also states the boundary the policy needs: this skill's size hunt (SSIM-pinned quality, maxrate
1.0x source video bitrate, bpp tiering, matched audio) is for H.264 / MPEG-4 sources. YouTube's
VP9/AV1 sources are already near-optimal for their quality, so a re-encode can only grow them or
lose quality - the accepted answer there is 'same size, do not chase smaller', and the job reduces
to: never decode with *_cuvid, keep -fps_mode cfr, gate the output on pictures vs the source, and
source any repair from the pre-conversion snapshot backup rather than the converted file.
Give it a path; it reads the source, picks the recipe, encodes, and verifies the
result before accepting it.
The problem it exists to remove: a file can drop ~23% of its frames in a browser
while r_frame_rate == avg_frame_rate reads EXACTLY equal, so every fps-based check
calls it healthy and it never enters a repair queue. The cause is deterministic and
header-readable -- the MP4 composition-offset table is unusable, either a single
constant ctts entry (PTS == DTS + const) or no ctts box at all (PTS == DTS) -- so the
frame timestamps are in decode order and the decoder's reordered output frames carry
the wrong times.
- three recipes chosen from the source FILE, never the filename: lossless -c copy when
the timeline allows it, AAC-only fix when just the audio is wrong, full HEVC (or
x264 for reach) when the timeline cannot be copied
- quality policy: never upscale; HEVC -cq 27 measured at SSIM 0.9937 ~= source size;
audio copied when already AAC, else AAC 256k
- the two flags that must never be dropped: -fps_mode cfr and -pix_fmt yuv420p
- verified before acceptance: duration, resolution, codec, hvc1 tag, 8-bit, uniform
cadence, no flat composition offsets, full decode pass, and a read-back after the write
- built for cron: lock file (two concurrent NVENC jobs starve one encoder and DROP
FRAMES), idempotent, leaves the input untouched on failure, aborts below 5 GB free
- scripts/selftest.py builds both defect shapes and asserts the detector, including a
Matroska case that must NOT be flagged
Measured on the file that prompted this: 23.0% of frames dropped before, 0.0% after.
hoelee
Hermes Agent