gate: a re-do of an already-playable file gets a 0.975 mean bar (owner decision)
One pass measured 11 attempts rejected purely on the window mean (0.9655-0.9795) while accepted encodes had worse frame minima (0.33-0.90): the bar was splitting a continuum, not separating defects. 0718 马斯克原理 failed with min 0.887 and 0.00% of frames below 0.85. Those files already play (HEVC hvc1 + AAC) and a re-do exists only to reclaim an old setting's bloat; reaching a 0.98 mean costs more than the source's own bitrate, which the size ceiling forbids - so the strict bar stranded them with their bloat while burning an encode every pass. SSIM_MEAN_MIN_REDO = 0.975 now applies only when the run is re-doing a file that is genuinely playable (HEVC hvc1 AND aac) from the snapshot original. The frame-share test (>1% of frames below 0.85) is unchanged and is what catches a real defect. Verified on ONE file before the tiers were restarted: 0321 随他们去 (rejected last pass at mean 0.9789) -> ok, mean 0.9786, min 0.5092 with 5 of 1500 frames below 0.85 (fades), 633 MB -> 418 MB installed, live byte count == the encoded output.
This commit is contained in:
@@ -278,6 +278,7 @@ re-encoded to `-b:a 256k`: that added ~128 kbps for no audible gain, ~+10% file
|
||||
- **Kill by PID only**, one lock per tier, one NVENC job at a time, and pause GPU tiers before a driver update (an update wedges GPU work silently — the output-growth watchdog is the net).
|
||||
- **A multi-day run must not be tied to the agent session.** Two tiers launched as Hermes background jobs with `persist_on_release` were both killed mid-encode the moment the user closed their session (measured 2026-10-08 10:37: `*_out.mp4` and `.err` frozen in the stage dir, no `PASS-DONE` in either log, ~40 min of work lost). Launch them **orphaned** instead — `powershell -NoProfile -Command "Start-Process -FilePath <bash.exe> -ArgumentList <script.sh> -WindowStyle Hidden"`, the script itself looping `library_run.py` with a retry — so the parent exits immediately and the agent's process cleanup cannot reach the tree. Verify the chain: the job's parent must be a bash whose own parent is already gone. Cost: no exit notification (watch the log) and **stopping is by PID only**.
|
||||
- **The `copy` tier is not idempotent — never re-run a whole `--tier copy` pass.** Its only skip test is "already HEVC hvc1", and a copy output is H.264 by definition, so a second pass redoes every file it ever produced (193 of them here, ~40 GB of share I/O). For a handful of newly-arrived files use the single-file path per file: `library_run.py --tier copy --path <file> --keep-source`.
|
||||
- **A re-do of an already-playable file earns a 0.975 mean bar; a first-time conversion does not.** Measured over one pass of this library: 11 attempts were rejected purely on the window mean (0.9655–0.9795) while accepted encodes had *worse* frame minima (0.33–0.90) — the bar was splitting a continuum, not separating defects (`0718 马斯克原理` failed with min 0.887 and **0.00 %** of frames below 0.85). A re-do exists only to reclaim an old setting's bloat from a file that already plays (HEVC hvc1 + AAC), and reaching a 0.98 mean costs more bits than the source's own bitrate — which the size ceiling forbids — so a strict bar strands the file with its bloat for ever while burning an encode every pass. `SSIM_MEAN_MIN_REDO = 0.975` therefore applies **only** when the live file is genuinely playable (HEVC hvc1 **and** AAC) and this run is replacing it from the snapshot original. The frame-share test (>1 % of frames below 0.85) is unchanged in both cases and remains the test that catches a real defect — under the relaxed bar `失控` (0.9655) is still rejected, and `0808 陌生人效应` (0.9739) keeps the full 0.98 bar because its live file is H.264 — not a re-do at all.
|
||||
- **Full measured detail, every failure mode and its detector: `references/pitfalls.md`.**
|
||||
|
||||
## The owner's standing requirements — read this first
|
||||
|
||||
@@ -54,6 +54,20 @@ bash /d/tmp/xcode/convert_new4.sh # 逐个 --tier copy --path <file> --keep-
|
||||
|
||||
- `1004 反内卷` 10:48:53 完成 ✓(535 MB → 535 MB,视频 100% 逐位不变,音频转 AAC,149 s,rc=0)
|
||||
|
||||
## 图片闸门:re-do 用 0.975(业主决定 2026-10-08)
|
||||
|
||||
第一轮实测:一趟里 11 次编码**只差在 mean**(0.9655–0.9795)就被拒,而被接受的那些帧最低分反而更差
|
||||
(0.33–0.90)—— 阈值在切一条连续的分布,不是在分开缺陷(`0718 马斯克原理`:min 0.887、**0.00%** 帧
|
||||
<0.85,照样被拒)。被拒的文件本来就已能播(HEVC hvc1 + AAC),re-do 的唯一目的是清掉旧设定的膨胀;
|
||||
而要够到 mean 0.98 得花掉超过源片自身的码率,会被 size ceiling 拦住 —— 结果是文件连同膨胀被永久搁置,
|
||||
而且每一趟还白烧一次编码。
|
||||
|
||||
新规则(`library_run.SSIM_MEAN_MIN_REDO = 0.975`):**只有**"重做一个已经在播的文件"才用 0.975,条件为
|
||||
live = HEVC hvc1 **且**音轨是 AAC(真能播)**且**本次是从快照原档重做;首次转换仍用 0.98。
|
||||
帧占比判据(>1% 帧 <0.85)两档都不变 —— 它才是抓真缺陷的那条:`失控`(0.9655)仍会被拒,
|
||||
`0808 陌生人效应`(0.9739)因为 live 是 H.264、根本不是 re-do,也仍走 0.98。
|
||||
先在一个文件上验证(`0321 随他们去`,上一轮 mean 0.9789 被拒)→ 见 `/d/tmp/xcode/run_pilot.log`。
|
||||
|
||||
## 目录与日志
|
||||
|
||||
| 用途 | 路径 |
|
||||
|
||||
+31
-6
@@ -88,6 +88,15 @@ SIZE_CEIL = 1.0 # output video bitrate must not exceed the source's
|
||||
QUALITY_STEPS = 2 # on a quality-gate failure, retry with the quality raised this many
|
||||
# steps (cq -2 / crf -2) before giving up; the retry only fires on a
|
||||
# file the cheaper setting could not carry
|
||||
SSIM_MEAN_MIN = 0.98 # the window-mean bar for a normal encode
|
||||
SSIM_MEAN_MIN_REDO = 0.975 # ... and for a RE-DO of a file that already plays everywhere
|
||||
# (HEVC hvc1 + AAC). Owner decision 2026-10-08: measured on this library,
|
||||
# 11 of a pass's attempts died on a mean of 0.9655-0.9795 while files with
|
||||
# WORSE frame minima (0.33-0.90) passed - the bar was splitting a
|
||||
# continuum, not separating defects (0718 马斯克原理 failed with 0.00% of
|
||||
# frames below 0.85 and min 0.887). A re-do's only job is to reclaim the
|
||||
# old settings' bloat, so it must not be stranded with the bigger file for
|
||||
# ever; 0.975 still rejects the two genuine stragglers (0.9655, 0.9739).
|
||||
|
||||
|
||||
def log(*a):
|
||||
@@ -599,6 +608,7 @@ def process(src, args, dest=None, origin=None):
|
||||
if info["dur"] < 1.0:
|
||||
return "FAIL stub (%.1fs) - needs a re-download, not a transcode" % info["dur"], None
|
||||
live = dest or dst_path_for(src)
|
||||
redo_playable = False # True once we know this run is re-doing a file that ALREADY plays
|
||||
|
||||
if info["vcodec"] == "hevc" and info["tag"] == "hvc1" and info["pix"] in ("yuv420p",) \
|
||||
and not T.unusable_timeline(info, src)[0]:
|
||||
@@ -614,6 +624,14 @@ def process(src, args, dest=None, origin=None):
|
||||
info["size"] / 1e6, o_size / 1e6), None
|
||||
log(" re-doing from the original: live %.0f MB > original %.0f MB" % (
|
||||
info["size"] / 1e6, o_size / 1e6))
|
||||
# A re-do replaces a file that already plays on iPad + Android; its only purpose is to
|
||||
# reclaim the old settings' bloat, so it earns the relaxed mean bar (owner decision,
|
||||
# SSIM_MEAN_MIN_REDO) - but only when the live file really is playable end to end
|
||||
# (HEVC hvc1 video, which the branch above guarantees, plus AAC audio).
|
||||
redo_playable = info["acodec"] in ("aac", "mp4a")
|
||||
if not redo_playable:
|
||||
log(" re-do of a file whose audio is %s (not playable) - the normal bar applies"
|
||||
% (info["acodec"] or "none"))
|
||||
src, info = origin, probe(origin)
|
||||
if not info:
|
||||
return "FAIL original unreadable", None
|
||||
@@ -687,17 +705,24 @@ def process(src, args, dest=None, origin=None):
|
||||
# Measured on this library: good encodes 0 of 1500 frames below 0.95; the file that
|
||||
# triggered this rule 5 of 1500 below 0.85 (0.33%) at 0.98 mean; the genuine
|
||||
# decoder-defect files 44 of 1499 (2.9%) with the mean falling too.
|
||||
# The MEAN bar is 0.975 instead of 0.98 when this is a re-do of a file that already
|
||||
# plays (SSIM_MEAN_MIN_REDO) - the frame-share test below is unchanged and is what
|
||||
# actually catches a defect.
|
||||
bad_share = (sstat.get("below", 0) / float(nfr)) if nfr else 0.0
|
||||
systematic = (mean is not None and mean < 0.98) or bad_share > MAX_BAD_SHARE
|
||||
mean_min = SSIM_MEAN_MIN_REDO if redo_playable else SSIM_MEAN_MIN
|
||||
systematic = (mean is not None and mean < mean_min) or bad_share > MAX_BAD_SHARE
|
||||
if systematic:
|
||||
if q_off < QUALITY_STEPS and (attempt == 1):
|
||||
q_off = QUALITY_STEPS
|
||||
log(" quality gate: mean %.4f, %d of %d frames below %.2f (%.2f%%) - "
|
||||
"retrying at quality+%d" % (mean, sstat.get("below", 0), nfr,
|
||||
BAD_FRAME_SSIM, 100.0 * bad_share, q_off))
|
||||
log(" quality gate: mean %.4f < bar %.3f, %d of %d frames below %.2f "
|
||||
"(%.2f%%) - retrying at quality+%d" % (
|
||||
mean, mean_min, sstat.get("below", 0), nfr,
|
||||
BAD_FRAME_SSIM, 100.0 * bad_share, q_off))
|
||||
continue
|
||||
return ("FAIL quality: %s - systematic: mean %.4f, %.2f%% of frames below %.2f"
|
||||
% (ssim_note, mean, 100.0 * bad_share, BAD_FRAME_SSIM)), None
|
||||
return ("FAIL quality: %s - systematic: mean %.4f < bar %.3f, %.2f%% of frames "
|
||||
"below %.2f%s"
|
||||
% (ssim_note, mean, mean_min, 100.0 * bad_share, BAD_FRAME_SSIM,
|
||||
" (re-do of an already playable file)" if redo_playable else "")), None
|
||||
if lo is not None and lo < SSIM_TARGET:
|
||||
log(" %d scattered frame(s) below %.2f (%.2f%% of the window) with a healthy "
|
||||
"mean - treated as fades/black, not a defect"
|
||||
|
||||
Reference in New Issue
Block a user