gate policy: frame-share is the verdict for owner-approved items; park after 2 failed rounds

Owner decision 2026-10-10. Eight items were livelocked in the nvenc tier, burning ~3.3 GPU-hours per pass (initial encode + quality+2 retry) and never converting: they missed the mean bar by 0.2-1.7 thousandths (0.9626-0.9778) while keeping only 0.20-0.93% of frames below 0.85 - and per this repo's own comment the FRAME-SHARE test is the one that actually catches a defect, the mean bar splits a continuum. With SIZE_CEIL=1.0 the mean bar is unreachable for them by construction. - library_run.py: POLICY (C:\AI\transcode-run\gate_policy.json) with * accept - named items are judged by the frame-share test; a mean under the bar is logged, not enforced. An item that trips the frame-share test is refused even if listed (0912 洞见, 1.33%, is deliberately absent). * park - failed quality ROUNDS per item; at PARK_AFTER=2 the item is skipped with a visible SKIP status instead of being re-encoded. Counters persist in the same file. - scripts/build_gate_policy.py: builds that file from the run log, taking each key verbatim from the '--- i/n <relative path>' line (no hand-copying). Result: 7 accepted, 1 refused. - run/relaunch_nvenc.sh: start the GPU tier ALONE (relaunch_tiers.sh hardcodes both tiers). - STATE.md: the policy, its rationale and the round counter.
2026-10-10 17:22:45 +08:00
parent bd8382e334
commit 99d03ff072
4 changed files with 217 additions and 24 deletions
@@ -62,6 +62,26 @@ bash /d/tmp/xcode/convert_new4.sh     # 逐个 --tier copy --path <file> --keep-

 - `1004 反内卷` 10:48:53 完成 ✓（535 MB → 535 MB，视频 100% 逐位不变，音频转 AAC，149 s，rc=0）

+## 闸门策略文件（业主决定 2026-10-10）：`C:\AI\transcode-run\gate_policy.json`
+
+8 个项在 nvenc 档里反复失败、**每轮白烧 ~3.3 GPU 小时**（初始一次 + quality+2 一次），而且永远过不了：它们的
+mean 差 0.2–1.7 个千分点（0.9626–0.9778），但**帧占比只有 0.20–0.93%** —— 按本仓库自己的注释，抓真缺陷的是
+**帧占比 >1%** 那条，mean 那条只是在切一条连续分布。
+
+```json
+{
+ "accept": [ "7 个项（帧占比 ≤1%）的库内相对路径" ],
+ "park":   { "8 个项": 1 }
+}
+```
+
+- `accept`：名单内的项以**帧占比测试**为准，mean 低于门槛只记录不拦截。**帧占比 >1% 的项即使列在名单里也照样拒**
+  （《2020年\0912 洞见》1.33% 故意不在名单里）。
+- `park`：每项**失败轮数**；达到 `PARK_AFTER = 2` 就 **SKIP**（不再重编码），要重试就把它的条目删掉。
+  没有这条，同样的 8 个项每趟都重演一次、永远转不出来。
+- 生成器：`scripts/build_gate_policy.py`（键**逐字取自日志**的 `--- i/n <相对路径>` 行，不手抄）。
+- 停/起的单档启动器：`run/relaunch_nvenc.sh`（`relaunch_tiers.sh` 会连带拉起 x265，别用它单独起 GPU 档）。
+
 ## 图片闸门：re-do 用 0.975（业主决定 2026-10-08）

 第一轮实测：一趟里 11 次编码**只差在 mean**（0.9655–0.9795）就被拒，而被接受的那些帧最低分反而更差
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+# Start ONLY the nvenc (GPU) tier, DETACHED from Hermes.
+#
+# run/relaunch_tiers.sh hardcodes `for t in x265 nvenc`, so it cannot be used to start the GPU tier
+# alone; this is its nvenc-only twin. One lock per tier (C:\AI\transcode-run\run-nvenc.lock, and
+# transcode.Lock is staleness-aware so a lock left by a reboot needs no cleanup).
+#
+# 2026-10-10: the run is governed by C:\AI\transcode-run\gate_policy.json (owner decision):
+#   accept = items where the FRAME-SHARE test is the verdict and the mean bar only a warning
+#            (7 items that failed by 0.2-1.7 thousandths of mean while keeping 0.20-0.93% of frames
+#            below 0.85; 0912 洞见 at 1.33% is deliberately NOT on the list and keeps being refused)
+#   park   = failed quality ROUNDS per item; at PARK_AFTER=2 the item is skipped instead of
+#            re-encoded, which stops the ~3.3 GPU-hours/pass those items used to burn. Clear an
+#            item's entry there to retry it.
+#
+# Detached because a Hermes-scoped job dies with the session (measured 2026-10-08). No exit
+# notification: watch /d/tmp/xcode/run_nvenc.log. Stop by PID only, never by image name.
+
+set -u
+TOOLS=/c/Users/hoelee/AppData/Local/hermes/tools
+PY="$(ls -d $TOOLS/python-*/python.exe 2>/dev/null | head -1)"
+LIB='Z:\Class\#snapshot\GMT+08-2026.09.25-09.50.20\樊登读书\樊登读书会（每周更新）'
+NLOG=/d/tmp/xcode/run_nvenc.log
+
+exec >> "$NLOG" 2>&1
+cd /d/dev/universal-video-transcode || { echo "FATAL cannot cd to the repo"; exit 1; }
+echo "=== nvenc-only relauncher start $(date '+%F %T')  py=$PY"
+echo "=== origin root: $LIB"
+echo "=== gate policy: $(cat /c/AI/transcode-run/gate_policy.json 2>/dev/null | tr -d '\n' | cut -c1-160)"
+if [ ! -x "$PY" ]; then echo "FATAL python not found under $TOOLS"; exit 1; fi
+
+for i in 1 2 3 4 5 6 7 8; do
+  echo "=== relaunch attempt $i $(date '+%F %T')"
+  "$PY" -u scripts/library_run.py --tier nvenc --origin-root "$LIB" --reverse
+  rc=$?
+  echo "nvenc PASS-DONE rc=$rc attempt=$i $(date '+%F %T')"
+  [ $rc -eq 0 ] && break
+  sleep 60
+done
@@ -0,0 +1,63 @@
+#!/usr/bin/env python3
+"""Build C:\\AI\\transcode-run\\gate_policy.json from today's nvenc run log.
+
+Owner decision 2026-10-10: items that failed the quality gate today are split by the FRAME-SHARE
+test (>1% of frames below 0.85 = the documented real defect test, and the only one of the eight that
+trips it is 0912 洞见 at 1.33%). Items passing the frame-share test go on the accept list, so the
+mean bar stops stranding them; every item that failed a round today gets park=1, so one more failure
+parks it (PARK_AFTER=2) instead of re-encoding it forever.
+
+Keys are taken verbatim from the log's "--- i/n <disp key>" lines, never retyped.
+"""
+import json, os, re, subprocess, sys
+
+LOG = r"D:\tmp\xcode\run_nvenc.log"
+POLICY = r"C:\AI\transcode-run\gate_policy.json"
+MARK = "nvenc relaunch attempt 1"      # only today's run
+
+items = []          # (disp key, bad share %, mean, bar)
+last = None
+seen = False
+for line in open(LOG, encoding="utf-8", errors="replace"):
+    if MARK in line:
+        seen = True
+    if not seen:
+        continue
+    m = re.match(r"^\d\d:\d\d:\d\d --- \d+/\d+ (.+?)\s*$", line)
+    if m:
+        last = m.group(1)
+        continue
+    if "!! FAIL quality" in line and last:
+        sh = re.search(r"([\d.]+)% of frames below", line)
+        mn = re.search(r"mean ([\d.]+) < bar ([\d.]+)", line)
+        items.append((last, float(sh.group(1)) if sh else None,
+                      float(mn.group(1)) if mn else None, float(mn.group(2)) if mn else None))
+        last = None
+
+# de-duplicate (an item may fail more than once in a log), keep the worst share
+worst = {}
+for k, sh, mn, bar in items:
+    if k not in worst or (sh or 0) > (worst[k] or 0):
+        worst[k] = sh
+
+accept, park, refused = [], {}, []
+for k, sh in worst.items():
+    park[k] = 1                     # one failed round today; the next failure parks it
+    if sh is not None and sh > 1.0:
+        refused.append((k, sh))
+    else:
+        accept.append(k)
+
+os.makedirs(os.path.dirname(POLICY), exist_ok=True)
+with open(POLICY, "w", encoding="utf-8") as fh:
+    json.dump({"accept": sorted(accept), "park": park}, fh, ensure_ascii=False, indent=1)
+
+print("failed items today:", len(worst))
+print("ACCEPT (frame-share <= 1%%): %d" % len(accept))
+for k in sorted(accept):
+    print("   ", k, "  (worst share %.2f%%)" % worst[k])
+print("REFUSED (frame-share > 1%%, keeps failing): %d" % len(refused))
+for k, sh in refused:
+    print("   ", k, "  (%.2f%%)" % sh)
+print("park seeded at 1 round for all of them -> parked at PARK_AFTER=2")
+print("wrote", POLICY)
@@ -115,6 +115,45 @@ def disp(p):
        return p


+# ---- owner gate policy (2026-10-10) -----------------------------------------------------------
+# Two settings, both owner decisions, kept OUT of the code so they can be changed without touching
+# the pipeline:
+#
+#  * accept  - items for which the FRAME-SHARE test is the verdict and the mean bar is only a
+#              warning. The bar was splitting a continuum rather than separating defects: on
+#              2026-10-10 eight items failed by 0.2-1.7 thousandths of mean while keeping
+#              0.20-0.93 % of frames below the floor, and only one of them (0912 洞见, 1.33 %)
+#              actually tripped the frame-share test that is documented as the real defect test.
+#              An item that fails the frame-share test is refused even when listed here.
+#  * park    - {item: failed quality ROUNDS}. A round is an initial attempt plus its quality+2
+#              retry. At PARK_AFTER rounds the item is skipped instead of re-encoded: without this
+#              the same eight items burned ~3.3 GPU-hours every pass, forever, and never converted.
+POLICY = os.path.join(WORK, "gate_policy.json")
+PARK_AFTER = 2
+
+
+def load_policy():
+    """{"accept": set(disp keys), "park": {disp key: failed rounds}} - missing file means neither."""
+    try:
+        with open(POLICY, encoding="utf-8") as fh:
+            d = json.load(fh)
+        return {"accept": set(d.get("accept") or []), "park": dict(d.get("park") or {})}
+    except (OSError, ValueError):
+        return {"accept": set(), "park": {}}
+
+
+def save_policy(p):
+    try:
+        with open(POLICY, "w", encoding="utf-8") as fh:
+            json.dump({"accept": sorted(p["accept"]), "park": p["park"]}, fh,
+                      ensure_ascii=False, indent=1)
+    except OSError as e:
+        log("   ! could not save %s: %s" % (POLICY, e))
+
+
+POLICY_CACHE = load_policy()
+
+
 def probe(p, tries=3):
    """T.probe with retries: this share times a probe out (measured: 120 s for a 30 MB read), and
    one hiccup must not fail an item that is otherwise fine. The per-item try/except in the main
@@ -710,7 +749,18 @@ def process(src, args, dest=None, origin=None):
                # actually catches a defect.
                bad_share = (sstat.get("below", 0) / float(nfr)) if nfr else 0.0
                mean_min = SSIM_MEAN_MIN_REDO if redo_playable else SSIM_MEAN_MIN
-                systematic = (mean is not None and mean < mean_min) or bad_share > MAX_BAD_SHARE
+                # Owner decision 2026-10-10: for an item on the policy's accept list the FRAME-SHARE
+                # test is the verdict (that is the test documented as the one that actually catches
+                # a defect) and a mean under the bar is recorded, not enforced. An item that fails
+                # the frame-share test is refused even when listed, and every other item is judged
+                # exactly as before.
+                approved = disp(src) in POLICY_CACHE["accept"]
+                systematic = bad_share > MAX_BAD_SHARE or (
+                    mean is not None and mean < mean_min and not approved)
+                if approved and mean is not None and mean < mean_min and bad_share <= MAX_BAD_SHARE:
+                    log("   accepted on the frame-share test (%.2f%% below %.2f, bar %.2f%%) for an "
+                        "owner-approved item; mean %.4f under the %.3f bar recorded, not enforced"
+                        % (100.0 * bad_share, BAD_FRAME_SSIM, 100.0 * MAX_BAD_SHARE, mean, mean_min))
                if systematic:
                    if q_off < QUALITY_STEPS and (attempt == 1):
                        q_off = QUALITY_STEPS
@@ -896,6 +946,7 @@ def main():
        log("=== plan: %s" % plan)
        return

+    POLICY_CACHE.update(load_policy())      # pick up an edited gate_policy.json between runs
    with T.Lock(lockf):
        stats = {"ok": 0, "skip": 0, "fail": 0}
        ledger = open(LEDGER, "a", encoding="utf-8")
@@ -907,30 +958,39 @@ def main():
            log("--- %d/%d %s" % (i, len(files), disp(f)))
            t0 = time.time()
            origin = None
-            if args.origin_root:
+            parked = POLICY_CACHE["park"].get(disp(f), 0)
+            if parked >= PARK_AFTER:
+                # Parked: it has failed this many quality rounds already and every further attempt
+                # costs an encode (plus its quality+2 retry) without ever converting. Recorded as a
+                # SKIP so it is visible, not silent; clear its entry in gate_policy.json to retry.
+                status = ("SKIP parked: %d quality round(s) already failed - not re-encoding "
+                          "(clear the entry in %s to retry)" % (parked, os.path.basename(POLICY)))
+                extra = None
+            else:
+                if args.origin_root:
+                    try:
+                        cand = os.path.join(args.origin_root, os.path.relpath(f, ROOT))
+                    except ValueError:
+                        cand = None
+                    origin = cand if cand and os.path.exists(cand) else None
                try:
-                    cand = os.path.join(args.origin_root, os.path.relpath(f, ROOT))
-                except ValueError:
-                    cand = None
-                origin = cand if cand and os.path.exists(cand) else None
-            try:
-                status, extra = process(f, args, dest=args.dest or dst_path_for(f), origin=origin)
-            except Exception as e:
-                status, extra = "FAIL %s: %s" % (type(e).__name__, e), None
-                # Any failure raised out of process() may leave the staged upload target on the
-                # share. Measured 2026-10-09: the host ran out of memory/commit while installing
-                # the largest output of the batch, `MemoryError` came out of a Dokan write, and the
-                # complete 493 MB `.new.mp4` stayed next to the live file (8 such orphans had piled
-                # up). Delete it unconditionally - a metadata check on this share can serve a stale
-                # answer, so "not found" is the only outcome that is ignored.
-                try:
-                    staged = (args.dest or dst_path_for(f)) + ".new.mp4"
-                    os.remove(staged)
-                    log("   ! removed the orphaned staging file %s" % staged)
-                except FileNotFoundError:
-                    pass
-                except (OSError, MemoryError) as ex:
-                    log("   ! could not remove the staging file: %s" % ex)
+                    status, extra = process(f, args, dest=args.dest or dst_path_for(f), origin=origin)
+                except Exception as e:
+                    status, extra = "FAIL %s: %s" % (type(e).__name__, e), None
+                    # Any failure raised out of process() may leave the staged upload target on the
+                    # share. Measured 2026-10-09: the host ran out of memory/commit while installing
+                    # the largest output of the batch, `MemoryError` came out of a Dokan write, and
+                    # the complete 493 MB `.new.mp4` stayed next to the live file (8 such orphans had
+                    # piled up). Delete it unconditionally - a metadata check on this share can serve
+                    # a stale answer, so "not found" is the only outcome that is ignored.
+                    try:
+                        staged = (args.dest or dst_path_for(f)) + ".new.mp4"
+                        os.remove(staged)
+                        log("   ! removed the orphaned staging file %s" % staged)
+                    except FileNotFoundError:
+                        pass
+                    except (OSError, MemoryError) as ex:
+                        log("   ! could not remove the staging file: %s" % ex)
            el = time.time() - t0
            if status == "ok":
                stats["ok"] += 1
@@ -943,6 +1003,17 @@ def main():
            else:
                stats["fail"] += 1
                log("   !! %s" % status)
+                if status.startswith("FAIL quality"):
+                    # Count the round so a repeat offender stops consuming the encoder. The counter
+                    # is persisted: without it every pass re-encoded the same eight items forever.
+                    k = disp(f)
+                    POLICY_CACHE["park"][k] = POLICY_CACHE["park"].get(k, 0) + 1
+                    save_policy(POLICY_CACHE)
+                    log("   -> quality round %d/%d failed for this item%s"
+                        % (POLICY_CACHE["park"][k], PARK_AFTER,
+                           (" - PARKED, skipped from now on (clear its entry in %s)"
+                            % os.path.basename(POLICY))
+                           if POLICY_CACHE["park"][k] >= PARK_AFTER else ""))
            ledger.write(json.dumps({"t": time.strftime("%Y-%m-%dT%H:%M:%S"),
                                     "path": disp(f), "status": status,
                                     "secs": round(el, 1), "extra": extra},