Every item left its SSIM windows (291 MB each), per-offset pairs, VMAF pairs and stats logs behind;
anything killed mid-item left them for ever. Measured 2026-10-12: 6 windows + 123 gate logs, work
drive at 3.9 GB, both lanes idle on the guard. process() now purges its own tag/pid artefacts in the
finally, and sweep_stage() clears leftovers older than STAGE_MAX_AGE_H while skipping any tag that has
a fresh <tag>_out.mp4.err (i.e. is encoding right now). Regression: scripts/test_stage_cleanup.py.
Measured 2026-10-11: an item parked at 2 rounds was released by the owner (park entry deleted) so it
could be retried at a lower crf - and the very next failure of ANOTHER item restored the entry, from
the stale in-memory snapshot of the engine that had loaded the file before the edit. The released
item was then SKIPped as parked. save_policy now re-reads the file and writes only the counters this
process bumped (POLICY_DIRTY), so a deleted park entry or an added accept entry survives.
Regression: scripts/test_policy_merge.py (4 checks: released item stays released, external accept
entry kept, counter reaches disk, and the old behaviour would have restored it).
The queues run the engines under pythonw.exe (no console). A console-subsystem child spawned by a
console-less parent without CREATE_NO_WINDOW makes Windows allocate a NEW console and SHOW it - one
visible window per ffprobe/ffmpeg call (measured 2026-10-11: Windows Terminal windows titled with the
ffmpeg binary while two encodes ran).
Fixed: transcode.run(), library_run.run()/run_watched() and the three gate helpers that call ffmpeg
directly (SSIM filtergraph, per-offset pairs, VMAF).
Proof (cache/scratch/nowin_test4.py, run in a DETACHED pythonw: GetConsoleWindow()==0, like the
engine): raw Popen without the flag -> 1 new visible console window; with the flag -> none) -> 0;
transcode.run() -> 0 windows, library_run.run() -> 0, library_run.run_watched() -> 0.
The SSIM gate compared frame i with frame i of the other file. Measured 2026-10-11 on two
installed outputs: -fps_mode cfr inserts a duplicate wherever a frame is dropped, so the whole
window sits at a constant -1 frame; the old sweep stepped by 10 (it could never see +-1) and its
max(0, off) seek made every NEGATIVE offset identical to 0. Cost: 14 items refused in 9.6 h
(12.4 h of lane time, then parked) while healthy encodes read 0.9808/0.9862 instead of
0.9976/0.9974.
* alignment: fine step-1 +-6 -> coarse step-10 +-100 -> per-frame +-2 best match, each stage named
in the verdict, PID-tagged temp files (two lanes share the stage dir), pair stats logs pruned
* the alignment note is INFORMATION now: it used to arrive in the irregular-timeline slot, which
skips the whole picture verdict - the canary and two x265 items were accepted that way with the
mean bar never enforced
* a window decode that fails is retried once and its ffmpeg stderr is kept (a transient decode
killed a finished 16.7-minute encode with a bare 'no frames compared'); gate decodes get their
own 60-minute stall window because trim writes nothing before the window starts; the ssim filter
and the pair comparisons are time-bounded so a wedged ffmpeg cannot hang a lane
* bars (owner decision): mean 0.995, redo/owner-released 0.990
* VMAF, observe-only and reported as vmaf480 because the window is the 480x270 one the gate already
decoded (480p reads ~+2.4/+3.2 points above native on the item measured) - the evidence base for
a threshold is two items, so it accumulates instead of gating
An owner decision can say a LEAN source is still worth re-encoding (measured: x265 crf27 lands at
33-53% of the source video bitrate at SSIM mean >=0.996). --tier cannot express that: it SKIPS any
file whose classification disagrees. --force-tier applies the requested tier instead, with two
guards that stay: copy is refused for a VP9/AV1 source (iOS cannot play the result) and a re-encode
is refused for a file that is already HEVC hvc1 (that is the --overwrite decision).
Owner decision 2026-10-10. Eight items were livelocked in the nvenc tier, burning ~3.3
GPU-hours per pass (initial encode + quality+2 retry) and never converting: they missed the
mean bar by 0.2-1.7 thousandths (0.9626-0.9778) while keeping only 0.20-0.93% of frames
below 0.85 - and per this repo's own comment the FRAME-SHARE test is the one that actually
catches a defect, the mean bar splits a continuum. With SIZE_CEIL=1.0 the mean bar is
unreachable for them by construction.
- library_run.py: POLICY (C:\AI\transcode-run\gate_policy.json) with
* accept - named items are judged by the frame-share test; a mean under the bar is logged,
not enforced. An item that trips the frame-share test is refused even if listed
(0912 洞见, 1.33%, is deliberately absent).
* park - failed quality ROUNDS per item; at PARK_AFTER=2 the item is skipped with a
visible SKIP status instead of being re-encoded. Counters persist in the same file.
- scripts/build_gate_policy.py: builds that file from the run log, taking each key verbatim
from the '--- i/n <relative path>' line (no hand-copying). Result: 7 accepted, 1 refused.
- run/relaunch_nvenc.sh: start the GPU tier ALONE (relaunch_tiers.sh hardcodes both tiers).
- STATE.md: the policy, its rationale and the round counter.
- run/relaunch_x265.sh: start ONLY the x265 tier. run/relaunch_tiers.sh hardcodes
`for t in x265 nvenc`, and the nvenc tier must not start while another job owns the
GPU (the TubeSync 58-file repair batch holds it since 2026-10-09 20:36). x265 is
pinned to the E-cores, so it shares only the RaiDrive mount -- 2 concurrent share
jobs is the documented safe limit.
- The launcher redirects its own banners into the log (under -WindowStyle Hidden,
stdout goes nowhere) and documents the -ArgumentList form: the script path alone.
- STATE.md: LastBootUpTime 2026-10-09 16:51:58 -- both tiers stopped by a REBOOT, not a
bug (x265 404/589, nvenc 308/589); the stale lock pids need no cleanup because
transcode.Lock is staleness-aware. Memory after reboot: 13 GB free / 50.3 of 55.9 GB
committed, but the commit limit fell from 79.6 GB because D: is 92% full.
崇祯传: the 493 MB <live>.new.mp4 the MemoryError left behind was complete. Passed, on a
local copy, structure + FULL DECODE + the picture gate (min 0.9980 / mean 0.9988, 1500
frames, 0 below 0.85) and a hash read-back against the share copy, only then renamed over
the live file (install read-back sha256 5a903ab8a3d1, 493,366,137 B, hevc/hvc1, 3616.8 s).
Saves a 3616 s re-encode: 822,288,651 B h264 -> 493,366,137 B HEVC.
The other 7 were truncated prefixes -> deleted; their live originals are untouched and the
tiers will redo them. 0 *.new.mp4 remain on the share.
Host hit 0 GB physical free / 76.8 of 79.6 GB commit while both tiers ran (pagefile
capped at 32 GB on a 97%-full D:), and the Dokan write of the batch's largest output
raised MemoryError. except OSError does not catch that, so the item was logged
'!! FAIL MemoryError:' with an empty message and the complete 493 MB <live>.new.mp4
stayed on the share; 8 such orphans had accumulated.
- _sha_remote / copy_remote_to_local / copy_local_to_remote: except (OSError, MemoryError)
so a transient host hiccup retries instead of failing the item
- same widening for both partial-file cleanups
- per-item handler: delete <dest>.new.mp4 unconditionally (os.remove + ignore
FileNotFoundError, never gated on os.path.exists - stale metadata on this share)
Verified by injecting MemoryError: the upload path returns False with the partial
deleted, the source read retries then raises a clean IOError, the hash retries twice.
One pass measured 11 attempts rejected purely on the window mean (0.9655-0.9795)
while accepted encodes had worse frame minima (0.33-0.90): the bar was splitting a
continuum, not separating defects. 0718 马斯克原理 failed with min 0.887 and 0.00% of
frames below 0.85. Those files already play (HEVC hvc1 + AAC) and a re-do exists only
to reclaim an old setting's bloat; reaching a 0.98 mean costs more than the source's
own bitrate, which the size ceiling forbids - so the strict bar stranded them with
their bloat while burning an encode every pass.
SSIM_MEAN_MIN_REDO = 0.975 now applies only when the run is re-doing a file that is
genuinely playable (HEVC hvc1 AND aac) from the snapshot original. The frame-share
test (>1% of frames below 0.85) is unchanged and is what catches a real defect.
Verified on ONE file before the tiers were restarted: 0321 随他们去 (rejected last
pass at mean 0.9789) -> ok, mean 0.9786, min 0.5092 with 5 of 1500 frames below 0.85
(fades), 633 MB -> 418 MB installed, live byte count == the encoded output.
The x265/nvenc tiers launched as Hermes background processes were killed
mid-encode at 10:37 when the user's session was closed, so persist_on_release
is not enough here: run/relaunch_tiers.sh starts them orphaned from Hermes
(Start-Process) with a retry loop, which is why they are no longer tracked and
must be stopped by PID only.
STATE.md now carries the progress table (242/589 done, 347 pending: 169 x265 +
174 nvenc + 4 copy), the reason a full `--tier copy` pass must NOT be run (its
only skip test is "already HEVC", which a copy output never satisfies - it
would redo all 193), and the four new weekly episodes handled one file at a
time.
STATE.md carries what the next session needs: the two commands that keep the tiers running, where
the logs/stage/locks/ledger live, the eight bugs this round produced (each with the measurement that
proved it), and the open items. The skill's body now keeps only the load-bearing rules; every
measured case moved to references/pitfalls.md, so the always-loaded part stays readable.
The GPU tier sat frozen for 1.5 hours: the user updated the NVIDIA driver while a hevc_nvenc encode
was running, which wedges in-flight GPU work with no error at all - the process stays alive, the
output stops growing, the encoder reads 0%, and nothing anywhere reports a problem. A multi-day
unattended run cannot park like that, so every long command that writes a file (the encode, and the
gate's decodes) is now watched: no growth for 300 s and the wedged process is killed and reported as
a failure for the normal retry path to pick up. Self-checked with a process that writes once and
then sleeps: killed at ~35 s for a 20 s stall window.
The three tiers run side by side by design (copy is I/O, x265 is CPU, nvenc is the GPU ASIC) and
they share the stage directory - but the gate's raw frame files had FIXED names, so two concurrent
gates overwrote each other's decodes. Measured: a comparison reported 100% of frames below 0.85 -
which reads exactly like a systematic defect and was nothing but two runs writing ssim_ref.yuv at
once. The raws are now tagged with the pid and the file tag. A single-file conversion failed its
gate on this and the 'defect' was entirely my own harness.
A freshly downloaded replacement has to go through the SAME pipeline as the library - same bpp
tier rule, same gates, same hash-verified install - rather than a hand-rolled ffmpeg one-off. So:
--path converts one file anywhere on disk (its tier is still chosen by bpp; the dest defaults next
to it), --dest installs the result at an exact path (e.g. back into the library), --keep-source
never deletes the source after a verified install. Also fixes a cross-drive crash: os.path.relpath
raises ValueError when the file is on D: and the library on Z:, so logs/ledger fall back to the
absolute path (measured on the first run of this feature).
A 30 MB file timed out a 120 s ffprobe on the share. The main loop's per-item try/except keeps the
run alive (good) but turns that hiccup into a FAIL, leaving the file unconverted for the rest of the
pass - so a probe is now retried three times before it is allowed to fail an item.
One file was reported converted and was not: 'ok 1085 MB -> 478 MB' printed, every metadata probe
agreed, and the live file was still the H.264 original afterwards, with no error anywhere. RaiDrive
/ Dokan serves cached attributes, so a rename can appear to succeed while nothing landed. The
install step now reads the live file back and compares its SHA-256 with the encode it just made -
the one check a cache cannot satisfy. Costs a read per file; catches a whole class of silent
no-ops that today's 'success' reports hid.
The gate failed a perfectly good encode (min 0.2552, mean 0.9841) because it demanded a per-frame
floor of 0.985 for EVERY frame. Two measured facts say that is the wrong test:
* SSIM collapses on near-flat frames. Two almost-black frames of a fade score ~0.25 while their
neighbours score 0.998, so a minimum over all frames condemns good encodes. The gate now measures
each source frame's luma std and drops the flat ones from the verdict (reported, never hidden;
if more than half the window is flat the exclusion is skipped so the gate still means something).
* A defect has a SHAPE. Measured distributions on this library: good encodes 0 of 1500 frames below
0.95; the file that triggered this 5 of 1500 below 0.85 (0.33%, isolated) at a 0.984 mean; the
genuine decoder-defect files 44 of 1499 (2.9%) with the mean falling. So the verdict is now
'mean < 0.98 OR more than 1% of frames below 0.85' -> fail; a handful of scattered low frames at a
healthy mean is logged as fades, not a defect - and the log line carries the counts, so the
decision is auditable.
A read of the RIGHT LENGTH can still be wrong on this share - RaiDrive reports stale sizes, and a
full-file listing read here stopped halfway with no error at all (it looked exactly like a
truncated file). A corrupt staged source would be silently encoded into a corrupt output, so the
copy now compares a hash of what it wrote against a fresh read of the share copy (files over 64 MB,
one extra pass - cheap next to the encode it protects). Today's evidence: a 921 MB snapshot original
threw 'Error splitting the input into NAL units' in one pass and decoded perfectly in the next.
Two bugs the library run exposed on real files:
* An input seek is not a faithful decode. On an H.264 source with open GOPs the frames after the
landing point decode differently from a sequential decode - the SAME file compared with itself,
once seeked and once decoded in order, scored min 0.25 / mean 0.82, and the gate condemned good
encodes because of it. Both sides are now decoded sequentially with the window cut by the trim
filter, which is also the only decode guaranteed to match what the encoder itself saw.
* Staleness of the run lock. Killing a background session kills the wrapper but not the python
underneath, so a live run's lock survives and the next start is correctly refused (it looked like
a pid-reuse bug; it was not). The lock now carries a heartbeat, staleness is decided by age as
well as pid, and a takeover that cannot delete a file the OS still holds overwrites it in place.
Two lessons from chasing one stubborn file. A 60 s stream copy (-ss X -c copy) of an H.264 source
with B-frames lost 230 of its 1500 frames and shifted the content, so every encode of that cut
looked broken (SSIM minimum 0.25) - harnesses must sample with byte copies, never stream copies,
and must prove themselves against a file compared with itself. And when a single file resists
diagnosis, park it and report it instead of stalling a library run on one item.
Every one of these cost real time today and each has a detector now: RaiDrive returning success
on a truncated write (12.5 MB of 403 MB, hash read-back caught it), the share's toasts being an
overload signal rather than a permissions problem (3 concurrent jobs), a picture gate comparing
by frame index after an input seek (false FAIL at min SSIM 0.25 on a good encode; fixed by
trimming on PTS), frame-count differences meaning 'gate not applicable' instead of 'broken file',
a filename with a space before its extension being innocent, and killing encoders by process name
(which took out an unrelated pilot encode).
Two failures from today's library run, both caught, both now handled:
1. The picture gate compared frames by INDEX after an input seek. A seek snaps to a keyframe, and
it snapped to a DIFFERENT keyframe for the source and for the encode - measured: two 1500-frame
decodes whose start times differed by ~10 s, so every frame met the wrong one and a perfectly
good encode scored min SSIM 0.25 (and a file with a hole in its source's timeline scored 0.45).
The window is now cut with the trim filter on absolute PTS (-copyts), so both sides cover
exactly the same interval no matter where the seek lands; a self-comparison then measures
lo=1.0/mean=1.0 (validated), and a frame-count difference >1% is reported as 'gate NOT
APPLICABLE' rather than failing a file whose source is simply missing frames.
2. RaiDrive/Dokan returned from a 403 MB write with NO error while the remote file held 12.5 MB,
and surfaced 'the operation didn't complete within specific time limit' afterwards. The
SHA-256 read-back caught it (nothing was replaced), and it now retries the upload instead of
failing outright. Uploads use 4 MB chunks with a periodic flush; source copies retry 5x with a
longer backoff and smaller chunks.
The old runner (fixed -cq 27 + fixed 256k audio + maxrate at 1.5x the FILE bitrate) left 54 files
BIGGER than their originals - the worst at 124.8%. Re-encoding those from their current HEVC would
be second-generation loss, so the source must be the pre-conversion original; a dated snapshot of
the library holds it (582 files / 324 GiB measured).
A file already HEVC is now re-done only when it is bigger than its original (>1.01x), which makes
the run idempotent with no extra state. The tier and every gate use the ORIGINAL, dst stays the
live path, and os.remove(src) is suppressed whenever the source came from the origin tree so the
archive of originals can never be deleted by a replace.
The previous runner encoded everything at a fixed -cq 27 with -maxrate = 1.5x the FILE bitrate,
which is how a library ended up at 111% of its source size. This replaces the encode step with the
policy the skill documents, and adds the gates that make it a guarantee:
* classify by bpp: <0.025 copy the video (bit-identical), 0.025-0.040 hevc_nvenc, >0.040 libx265
* -maxrate = 1.0x the source's VIDEO bitrate (never the file total) + a post-encode assertion
* SSIM gate: per-frame comparison of a 60 s window against the source, retry once at +2 quality
if it fails, then fail the item (the original is never touched)
* audio: copy when AAC, else AAC at the SOURCE's bitrate (a fixed 256k cancelled the video saving)
* x265 tier pinned to the E-cores; --tier to run tiers separately; --keep-local for a pilot that
touches nothing
* defaults cq31 / crf26 from the measurement below, not cq27
Measured on three real sources (60 s window, video-only): cq27 GROWS two of three files (115%,
119%); cq31 lands at 43-74%; x265 crf26 at 34-67% with min SSIM 0.988-0.993 - and x265 beats NVENC
by 10-22% at matched SSIM. Pinning min SSIM >= 0.996 leaves only cq27, which is bigger than the
source, so ~0.99 is the usable quality frontier and the owner gets to make that call explicitly
(references/hevc-conversion-policy.md section 11, SKILL.md scope section).
The installed skill had drifted ahead of the repo: SKILL.md carried the whole owner-requirements
section and references/hevc-conversion-policy.md was missing here entirely. Synced both.
Also states the boundary the policy needs: this skill's size hunt (SSIM-pinned quality, maxrate
1.0x source video bitrate, bpp tiering, matched audio) is for H.264 / MPEG-4 sources. YouTube's
VP9/AV1 sources are already near-optimal for their quality, so a re-encode can only grow them or
lose quality - the accepted answer there is 'same size, do not chase smaller', and the job reduces
to: never decode with *_cuvid, keep -fps_mode cfr, gate the output on pictures vs the source, and
source any repair from the pre-conversion snapshot backup rather than the converted file.
A public repository with no license is "all rights reserved" by default: nobody may
legally use, modify or redistribute it. MIT grants exactly the permissions a public
utility repo implies, while the warranty disclaimer keeps liability with the user.
Copyright (c) 2026 Lee Teong Hoe (Hoelee Enterprise). Linked from the README.
docs/WHY.md records the headline case and the guards, all traceable to failures this
pipeline actually hit rather than hypotheticals:
- the file that dropped 23.0% of frames while r_frame_rate == avg_frame_rate read
exactly equal, and which a full library scan did not flag
- 27 invisible stutterers (1.2% of the library) with the same signature
- a detector that missed one of the two defect shapes (no ctts box) entirely
- a "lossless" remux that faithfully carried the stutter through -c copy
- a re-download that reproduced the defect, because the source mkv is already flat
- a video that played only its first 10 seconds after a short network copy, where
getsize() returned stale metadata and size-based verification passed
- a 115-file batch lost to a full disk
- two concurrent encoders starving each other and dropping frames
- an iPad refusing an otherwise perfect file over hev1 vs hvc1
- Opus copied into an MP4
Also carries the reproduction (scripts/selftest.py), why a browser cannot be the
automated gate (Chrome decodes no HEVC, so an HEVC file reports 0/0 and looks perfect),
the 2x2 that proves the cause is container timing and not the bitstream, and cron /
Task Scheduler recipes with the reason each is safe to schedule.
Give it a path; it reads the source, picks the recipe, encodes, and verifies the
result before accepting it.
The problem it exists to remove: a file can drop ~23% of its frames in a browser
while r_frame_rate == avg_frame_rate reads EXACTLY equal, so every fps-based check
calls it healthy and it never enters a repair queue. The cause is deterministic and
header-readable -- the MP4 composition-offset table is unusable, either a single
constant ctts entry (PTS == DTS + const) or no ctts box at all (PTS == DTS) -- so the
frame timestamps are in decode order and the decoder's reordered output frames carry
the wrong times.
- three recipes chosen from the source FILE, never the filename: lossless -c copy when
the timeline allows it, AAC-only fix when just the audio is wrong, full HEVC (or
x264 for reach) when the timeline cannot be copied
- quality policy: never upscale; HEVC -cq 27 measured at SSIM 0.9937 ~= source size;
audio copied when already AAC, else AAC 256k
- the two flags that must never be dropped: -fps_mode cfr and -pix_fmt yuv420p
- verified before acceptance: duration, resolution, codec, hvc1 tag, 8-bit, uniform
cadence, no flat composition offsets, full decode pass, and a read-back after the write
- built for cron: lock file (two concurrent NVENC jobs starve one encoder and DROP
FRAMES), idempotent, leaves the input untouched on failure, aborts below 5 GB free
- scripts/selftest.py builds both defect shapes and asserts the detector, including a
Matroska case that must NOT be flagged
Measured on the file that prompted this: 23.0% of frames dropped before, 0.0% after.
hoelee
Hermes Agent