diff --git a/public/banners/calibrating-intel-quick-sync-hevc-quality.png b/public/banners/calibrating-intel-quick-sync-hevc-quality.png new file mode 100644 index 0000000..289a39c Binary files /dev/null and b/public/banners/calibrating-intel-quick-sync-hevc-quality.png differ diff --git a/public/banners/docker-reported-exit-0-the-oom-killer-disagreed.png b/public/banners/docker-reported-exit-0-the-oom-killer-disagreed.png new file mode 100644 index 0000000..21013b5 Binary files /dev/null and b/public/banners/docker-reported-exit-0-the-oom-killer-disagreed.png differ diff --git a/public/og/calibrating-intel-quick-sync-hevc-quality.png b/public/og/calibrating-intel-quick-sync-hevc-quality.png new file mode 100644 index 0000000..11eda10 Binary files /dev/null and b/public/og/calibrating-intel-quick-sync-hevc-quality.png differ diff --git a/public/og/docker-reported-exit-0-the-oom-killer-disagreed.png b/public/og/docker-reported-exit-0-the-oom-killer-disagreed.png new file mode 100644 index 0000000..49ed9d1 Binary files /dev/null and b/public/og/docker-reported-exit-0-the-oom-killer-disagreed.png differ diff --git a/src/content/posts/calibrating-intel-quick-sync-hevc-quality.md b/src/content/posts/calibrating-intel-quick-sync-hevc-quality.md new file mode 100644 index 0000000..322a746 --- /dev/null +++ b/src/content/posts/calibrating-intel-quick-sync-hevc-quality.md @@ -0,0 +1,220 @@ +--- +title: "Calibrating Intel Quick Sync HEVC: A Cliff at 28, and a Preset That Does Nothing" +description: "I measured QSV HEVC on real 1080p files: ICQ 18–22 inflate thin sources, the quality cliff sits between 28 and 30, and fast equals medium byte for byte." +pubDate: 2026-10-14 +updatedDate: 2026-10-16 +category: tutorials +tags: ["ffmpeg", "hevc", "qsv", "transcoding", "video"] +ogImage: /og/calibrating-intel-quick-sync-hevc-quality.png +banner: /banners/calibrating-intel-quick-sync-hevc-quality.png +draft: false +--- + +## The question I could not answer: which number do I ship? + +My transcoding queue decides, per file, which encoder does the work: `copy` +when the source is already efficient, NVENC on the GPU, x265 on the CPU — +and now Intel Quick Sync. Each tier has two knobs that decide quality, and +every tier except QSV had numbers I had measured on my own library. + +QSV had none. And the code refused to guess: a shipping default of +`qsv_quality = 0` / `qsv_preset = ""` means *nobody has calibrated this*, +and the engine treats that as a hard exclusion — QSV can exist in the code +and never be selected on any machine, including the one that has the +hardware. + +So the task was not "turn on QSV". It was "produce measured numbers for two +knobs, so the encoder becomes selectable honestly". + +## Problem one: the hardware is not in the machine where the app runs + +The queue runs on Windows 11 in a VM. That VM has an RTX 3060 passed through +and no Intel graphics of any kind — which is exactly why QSV had never been +measured: the obvious move is to run one test locally and read the numbers +off. + +The iGPU is on the host instead (unRaid, i9-13900K), idle and unclaimed: + +```text +00:02.0 VGA compatible controller: Intel Raptor Lake-S GT1 [UHD Graphics 770] + driver: i915 refcount: 0 +/dev/dri/card0 /dev/dri/renderD128 (present, world-accessible) +no running container declares a dri device +virsh dumpxml Win11: hostdevs = 01:00.0, 01:00.1 (the 3060 + its HDMI audio) +``` + +Two conclusions, both useful. The VM's `Error creating a MFX session` is +**expected** and not a bug to chase — the VM can never see that iGPU. And +the only way to measure QSV is on the Linux side, in a container, on the +host. + +## Measure with the engine's own gate, not with ffmpeg + +This is the part that makes the numbers worth anything. I did not write +ffmpeg commands and compute SSIM myself — the calibration runs **the engine +that will do the work**, with its own verification and its own accept/reject +gate: + +```bash +python3 library_run.py --work-dir /work --tier qsv --force-tier qsv \ + --qsv-quality Q --qsv-preset P --path --keep-local --overwrite +``` + +`--keep-local` means encode, verify, run the gate, and then *do not* land +the file — the verdict is what I am collecting, not a new library. Sources +came in read-only over an existing SMB mount, and I hashed each source +before and after: unchanged. + +The image is `linuxserver/ffmpeg` (ffmpeg 9.0.2, iHD 26.3.5, oneVPL 2.17) +plus `python3` — plus **numpy**, which is not optional. Without it the gate +prints `flat-frame scan skipped: No module named 'numpy'` and quietly +changes how near-flat frames are judged, which changes verdicts. + +`-global_quality` maps to `RateControlMethod: ICQ` on `hevc_qsv`, which is +the knob being calibrated. + +## The ladder: on a thin source, low ICQ makes the file *bigger* + +First run, on a real 1080p25 source at 951 kbps (0.0184 bits per pixel — a +"thin" file, the kind that is already squeezed): + +| ICQ | engine verdict | mean SSIM | video = source | +|---|---|---|---| +| 18 | **FAIL size** — 176.8% | — | — | +| 20 | **FAIL size** — 141.8% | — | — | +| 22 | **FAIL size** — 107.6% | — | — | +| 24 | ok | 0.9967 | 81.1% | +| 26 | ok | 0.9959 | 61.0% | +| 28 | ok | 0.9951 | 46.4% | +| 30 | **FAIL quality** | 0.9942 | — | +| 32 | **FAIL quality** | 0.9942 | — | + +Three things fall out of that table. + +The counterintuitive one: **low ICQ does not mean safe.** ICQ 18–22 produce +files *larger* than the source, so the engine's size ceiling (100% of the +source's video bitrate) rejects them. QSV is not alone in this — NVENC at a +low `cq` does the same thing on thin inputs — but the boundary sits higher +here: the usable range starts at 24, not 18. + +Second, **there is a cliff between 28 and 30.** 30 and 32 fail on quality, +and they fail on the *first attempt*. That phrase matters, because the +engine retries a failed quality attempt at `global_quality - 2` — lower +ICQ, higher quality — and logs the retry. A line reading "ok at quality+2" +is a **first-attempt failure**, not a pass. I read that line backwards in my +own notes the first time; the rule is to read the first attempt, not the +recovery. + +Third, the floor is where the choice gets made: with a **0.995 mean-SSIM +floor**, q28 passes with 0.0001 of margin and q26 with 0.0010. + +## Full files confirm the ladder — and one shortcut survives + +Excerpts (180-second lossless slices) are how you rank ten candidates +without ten full encodes. They are only legitimate if they agree with +full-file runs, so I ran the real files end to end: + +| source | ICQ | preset | video = source | file | SSIM min / mean | encode | +|---|---|---|---|---|---|---| +| 254 MB, 2023 s, 872 kbps | 26 | medium | **61.0%** | 168,435,266 B (66%) | 0.9938 / **0.9960** | 360 s ≈ 5.6× realtime | +| the same file | 28 | medium | 46.3% | 54% | 0.9927 / 0.9952 | 330 s | +| 315 MB, 3280 s, 546 kbps | 26 | medium | 50.4% | 60% | 0.9965 / **0.9977** | 600 s ≈ 5.5× realtime | + +The 180-second excerpt of the first file measured mean 0.9959; the full file +measured 0.9960. That is close enough to trust the shortcut **for quality +ranking** — and I now have the evidence to say so rather than hope so. + +## The preset that does nothing + +The excerpt runs said preset only changes speed: veryfast 30 s / 61.6%, fast +30 s / 61.0%, medium 60 s / 61.0%, slow and veryslow 60 s. Clean story: pick +veryfast, save 30 seconds per file. + +Then the full files disagreed with it. The same source at ICQ 26, one run +with `fast` and one with `medium`: + +```text +fast: 168,435,266 B enc 360 s mean SSIM 0.9960 +medium: 168,435,266 B enc 360 s mean SSIM 0.9960 +``` + +Byte-for-byte identical, same encode time, same quality. The 30-vs-60-second +difference in the excerpt layer was per-run fixed overhead, not encoding +speed — and it is the kind of number that would have gone into a config file +and a blog post if I had stopped one step early. + +So: **excerpts rank quality, never speed.** Speed has to be measured on full +files, where the fixed overhead disappears. + +## What I shipped + +`qsv_quality = 26`, `qsv_preset = medium` — 61.0% of the source's video +bitrate at mean SSIM 0.9960 (min 0.9938) at roughly 5.6× realtime, and 50.4% +at mean 0.9977 on the thinnest file in the library. q28's margin over the +floor is 0.0002, and the cost of a quality failure is two full encodes plus +a park, so the extra 15 percentage points of savings are not worth the tail +risk. + +Two supporting measurements worth recording, because they answer "is it QSV +or is it the file?": + +- **Cross-encoder control.** One 720p source with non-uniform cadence scored + QSV q24 mean 0.9154, QSV q28 mean 0.9146, and x265 crf26 mean 0.9164 — and + the engine's alignment search reported + `no alignment explains the mismatch`. The rejection belongs to the source, + not to Quick Sync. +- **Concurrency.** The engine refuses a second encoder on a shared work + root: + `another transcode is already running (pid 7, /work/run-qsv.lock, 3 s old)`. + With separate work roots, two jobs run — and take 201 s where one takes + 107 s. Twice the wall clock, zero throughput gain. One QSV job at a time. + +## Four traps, each of which cost a cycle + +1. **Never mount a shared library at `/lib`.** I mounted the SMB share into + the container at `/lib` — which is where the container's own dynamic + loader lives. All eight jobs died in under a second with + `exec /bin/bash: no such file or directory`, which reads like a broken + image, not a bad mount. `/library` works. +2. **Synthetic sources lie about ICQ.** `hevc_qsv`'s rate control is + rejected by the driver for `rgb24` input + (`Selected ratecontrol mode is unsupported`) — so a + `testsrc`/`mandelbrot` self-test concludes the opposite of the truth. Use + a real file, or add `format=nv12`. +3. **Already-HEVC sources cannot be calibrated at all.** The gate answers + `SKIP already HEVC hvc1` and `--overwrite` does not lift it. The fat + 1080p H.264 originals in this library had all been converted long ago; + the remaining fat files are HEVC — so that tier has nothing left to + measure on. That is a missing input, not a skipped task. +4. **Read the first attempt, not the retry** (see the cliff above). + +## What I would do differently + +1. **Measure speed on the artifact you will actually run**, not on a slice + of it. My one wrong number in this whole exercise came from trusting an + excerpt's wall clock. +2. **Make the machine prove it has the hardware before you plan the test.** + Six minutes of `lspci` and `virsh dumpxml` retired the "why does QSV fail + in the VM" question that had been open for weeks. +3. **Keep the gate in the loop.** Because the engine's own verification + judged every run, the tables above are decisions, not opinions — the same + gate that will reject a bad file in production rejected my ICQ 30. +4. **Write the honest scope line into the notes.** These numbers are Linux + iHD 26.3.5 + oneVPL 2.17 on a UHD 770; the production target is the + Windows MSDK path on the same silicon, and I cannot claim they agree. + What this establishes is that the command runs on real hardware, where + the quality↔size curve sits, and how the lock behaves — leaving a + re-measurement on Windows+Intel as one table, instead of another + afternoon. + +## The result + +QSV went from "present in the code and never selectable" to a calibrated +tier with published numbers, and the safety rule changed with it: the test +that used to pin *shipping defaults must mean uncalibrated* now pins +*shipping defaults must be measured values*, while keeping every assertion +that an empty value is a hard exclusion. The doctor output on a machine with +no Intel GPU moved from an informational "waiting for hardware" line to a +warning — which is its own small lesson about what a calibrated encoder +failing self-check should look like, and why hardware the machine does not +have belongs in a summary row rather than a warning. diff --git a/src/content/posts/docker-reported-exit-0-the-oom-killer-disagreed.md b/src/content/posts/docker-reported-exit-0-the-oom-killer-disagreed.md new file mode 100644 index 0000000..d629a58 --- /dev/null +++ b/src/content/posts/docker-reported-exit-0-the-oom-killer-disagreed.md @@ -0,0 +1,196 @@ +--- +title: "Docker Reported Exit 0. The Kernel Had Already OOM-Killed It." +description: "A container on my NAS kept stopping unexpectedly while Docker reported exit 0 and OOMKilled=false. The honest evidence was in dmesg: six cgroup OOM kills." +pubDate: 2026-10-16 +category: devops +tags: ["docker", "oom", "traefik", "synology", "crawlers"] +ogImage: /og/docker-reported-exit-0-the-oom-killer-disagreed.png +banner: /banners/docker-reported-exit-0-the-oom-killer-disagreed.png +draft: false +--- + +## The notification I expected to be a config mistake + +My NAS sent the notification every Docker user learns to skim: **container +stopped unexpectedly**. This one is the edge proxy in front of my +self-hosted wiki, and it had been up for two weeks. Nobody had deployed +anything that morning and nothing in the compose file had been touched, so +the two available stories were "it crashed" and "something restarted it". + +Both were wrong, and the fact that I could not tell from Docker's own report +is the part worth writing down. + +## Docker's report says nothing happened + +```text +State: + Status: running + Running: true + ExitCode: 0 <- a clean exit + OOMKilled: false <- not killed by the kernel + FinishedAt: 2026-10-15T19:39:33.63057879Z + StartedAt: 2026-10-15T19:39:38.769326672Z <- restarted 5 s later +RestartCount: 6 +RestartPolicy: always +``` + +Read that as written and the conclusion is: the process exited cleanly, six +times, and the restart policy gave it a new life each time. A long-running +proxy does not exit with status 0 on its own — and a clean exit that repeats +six times is a contradiction, not an answer. `ExitCode` and `OOMKilled` are +exactly the two fields I would have trusted here. On this host, both were +wrong. + +## The kernel keeps its own record + +On Linux, a cgroup memory kill is logged by the kernel even when the +container runtime's bookkeeping does not carry it forward: + +```text +[1737797.044856] traefik invoked oom-killer: gfp_mask=0x2400040, order=0, oom_score_adj=0 +[1737797.123736] Memory cgroup out of memory: Kill process 615 (traefik) score 970 or sacrifice child +[1737797.132800] Killed process 615 (traefik) total-vm:2525800kB, anon-rss:478480kB, file-rss:0kB +``` + +The uptime stamp `1737797` corresponds to 19:39:33 UTC — to the second, the +container's `FinishedAt`. That is the whole diagnosis, and it took one +`dmesg` to get it. The container did not exit; it was killed inside its own +memory limit and restarted 5 seconds later. + +dmesg holds **six** such events, and `RestartCount` was **6**. Every other +container in that stack has a restart count of zero and a start time from +weeks ago, so the six kills pin to this one container: + +| dmesg uptime | local time | anon-rss at kill | +|---|---|---| +| 1406539 | Oct 12, ~07:37 | 491 MB | +| 1407759 | Oct 12, ~07:57 | 494 MB | +| 1432574 | Oct 12, ~14:49 | 486 MB | +| 1734793 | Oct 16, ~02:45 | 487 MB | +| 1735495 | Oct 16, ~02:58 | 485 MB | +| 1737797 | Oct 16, 03:39 ← the notification | 478 MB | + +Every kill happens at 478–494 MB against a limit of 512 MiB (`536,870,912` +bytes). That is a ceiling being touched, not a runaway. + +Two more readings made the mechanism clear. The cgroup's recorded peak — the +counter that survives the restart — was already back at exactly +`536,870,912` **18 minutes** after the restart, i.e. the container touched +its ceiling again in the same cycle *without* being killed. And the usage +curve oscillated (176 → 308 → 361 MB) instead of climbing monotonically, so +this is allocation churn under load hitting a small cap, not a provable slow +leak. The prime suspect is a geoblock plugin doing an external HTTPS lookup +for every client IP it has not seen before. + +A seventh kill landed at 05:33 that morning, before the fix went in. + +On this DSM build the truth was only in `dmesg` plus the cgroup's +`memory.max_usage_in_bytes`. If you are chasing the same shape on a normal +host, `docker inspect -f '{{.State.OOMKilled}}'` is the field to read first +— it is supposed to be authoritative. Here it read `false` five seconds +after the kernel had logged the kill, which is the reason this post exists. + +## What was actually breaking: a crawler flood + +Host memory was never the problem — 64 GB installed, ~39 GB available. One +container's 512 MiB ceiling was. + +The driver was traffic, and the numbers came out of the container's own +access log. In the last 50,000 lines: + +```text +distinct client IPs 25,976 +403 responses 33,636 (geoblock / deny-list doing its job) +429 responses 16,082 (inflight and rate limiting doing their job) +container CPU 118–230% +host load average 26–32 +``` + +Each restart refilled the heap in 12 to 40 minutes. Nothing in the +configuration was wrong, and neither were the 403s and 429s — they are the +filter rejecting crawlers exactly as designed. The problem is who is +knocking: this container is the public edge for a multi-terabyte Wikipedia +corpus served from a home NAS, behind a country allowlist and a bot filter. +That is an extremely attractive target for a residential-proxy crawler +cloud, and the responses telling it to go away are still work. + +## The fix, and the sentence that has to come with it + +```yaml + deploy: + resources: + limits: +- memory: 512M ++ memory: 2g + restart: always ++memswap_limit: 2g +``` + +Verification, because "I raised the limit and it feels better" is not a +verification: + +- **Only that container was recreated.** The wiki, the proxy core and the + robots container still reported two weeks of uptime — the gap users saw + was seconds, not a stack restart. +- `HostConfig.Memory = 2147483648`, `MemorySwap = 2147483648`, and the + cgroup's `memory.limit_in_bytes` agreed with both; `RestartCount` went + back to 0. +- Peak usage after 18 minutes: **210 MB**, against a ceiling it had been + pinning at 512 MB. By midday it sat at 94 MiB. +- The mount list still matched the file, including the plugin directory + mount — losing that one had once disabled every router in this stack. +- Both plugins were loaded and active (country allowlist, bot filter), and + the routes returned 200 once the filter's score window expired. + +**And what it is not:** this is a stopgap. The flood is the driver, so an +order-of-magnitude bigger flood reaches 2 GB too. The real fix is shrinking +the public entry surface, not enlarging the cage around it — and a limit +bump that is not labelled as a stopgap becomes a permanent explanation. + +I chose to disable swap for this container rather than let it swap: +`memswap_limit` equal to the memory limit does exactly that. The host +already had 16 GB of its 40 GB swap in use with committed memory at 92 GB, +and an edge proxy that starts swapping slows the entire NAS — the disk array +is the bottleneck here, so swapping buys a slower failure rather than a +longer life. + +## The trap that nearly made me report a self-inflicted outage + +During verification, my first bare `curl` probes against the site returned +**403**, and it looked exactly like a regression I had just caused by +restarting the proxy. It was not: the bot filter had scored **my own client +IP** (`blocked user-agent`, score 80) and placed it in a 15-minute score +window, because I had been hammering the host. + +The tell was in the same test run — `/robots.txt` returned 200 and the +internal `/cdn` path returned 400 at the same moment, i.e. the router was +healthy and only my own requests were being filtered. On a host where you +are also a client, never diagnose with your own bare curl: probe a path the +filter ignores, or read a 200 out of the access log. + +## What I would do differently + +1. **Treat "stopped unexpectedly" plus `ExitCode 0` as an instruction to + read `dmesg`**, not as evidence of a clean exit. A restart count above + zero is a memory investigation until proven otherwise. +2. **Alert on the cgroup peak, not only on restarts.** The post-restart + reading that was already pinned at the ceiling told me a seventh kill was + coming about eighteen minutes before the notification arrived. +3. **Get the "who is hitting me" number early.** 25,976 distinct IPs in one + log sample is the entire story, and I found it after spending time on the + memory curve. +4. **Question the container, not the server.** "The server is out of memory" + and "one container has a 512 MiB ceiling on a 64 GB machine" need + completely different responses, and the second one is invisible unless + you look at the limit. + +## The result + +No OOM kill since the change, with a peak at 210 MB after 18 minutes against +a ceiling that used to be touched in every single cycle — and the seven +kills that had been silently restarting the public edge for four days +stopped appearing in dmesg. The crawler flood did not stop, which is why I +recorded the failure mode in my ops notes next to the stack: "container +stopped unexpectedly" plus `exit 0` / `OOMKilled=false` is a misleading +signature, and on this platform the honest evidence lives in the kernel log +and the cgroup's high-water mark. diff --git a/src/content/posts/zh/calibrating-intel-quick-sync-hevc-quality.md b/src/content/posts/zh/calibrating-intel-quick-sync-hevc-quality.md new file mode 100644 index 0000000..115e390 --- /dev/null +++ b/src/content/posts/zh/calibrating-intel-quick-sync-hevc-quality.md @@ -0,0 +1,127 @@ +--- +title: "校准 Intel Quick Sync HEVC：28 与 30 之间的悬崖，和一个毫无作用的 preset" +description: "我在真实 1080p 文件上量了 QSV HEVC：ICQ 18–22 会把瘦文件撑大，质量悬崖落在 28 与 30 之间，而 fast 与 medium 逐字节同大小。" +pubDate: 2026-10-14 +updatedDate: 2026-10-16 +category: tutorials +tags: ["ffmpeg", "hevc", "qsv", "transcoding", "video"] +ogImage: /og/calibrating-intel-quick-sync-hevc-quality.png +banner: /banners/calibrating-intel-quick-sync-hevc-quality.png +draft: false +--- + +## 一个我回答不了的问题：到底该出货哪个数字？ + +我的转码队列按文件逐个决定用哪个编码器：源文件本来就高效就用 `copy`、GPU 走 NVENC、CPU 走 x265——现在又多了一个 Intel Quick Sync。每一档都有两个决定质量的旋钮，除了 QSV，其他每一档我都有在自己片库上量过的数字。 + +QSV 一个都没有。而代码拒绝猜：出货默认值是 `qsv_quality = 0` / `qsv_preset = ""`，含义是「没人标定过这一档」，引擎把它当成硬排除——于是 QSV 可以存在于代码里，却在任何机器上都不会被选中，包括真的有这块显卡的那台。 + +所以任务不是「把 QSV 打开」，而是「为两个旋钮产出量过的数字，让这个编码器能被诚实地选中」。 + +## 问题一：硬件不在跑这个程序的机器里 + +队列跑在 Windows 11 虚拟机里。那台 VM 直通了一张 RTX 3060，没有任何 Intel 核显——这正是一直没量过 QSV 的原因：最顺手的主意就是在本地跑一次、把数字读出来。 + +核显在宿主上（unRaid，i9-13900K），而且空闲、无人占用： + +```text +00:02.0 VGA compatible controller: Intel Raptor Lake-S GT1 [UHD Graphics 770] + driver: i915 refcount: 0 +/dev/dri/card0 /dev/dri/renderD128 （存在，全局可读） +没有任何运行中的容器声明 dri 设备 +virsh dumpxml Win11: hostdev = 01:00.0, 01:00.1（3060 + 它的 HDMI 音频） +``` + +两个结论都很有用。VM 里那句 `Error creating a MFX session` 是**预期行为**，不是一个要去查的 bug——那台 VM 永远看不到这块核显。而量 QSV 的唯一办法，是在宿主上、在容器里、用 Linux 侧去量。 + +## 用量尺的那个人自己的闸门来量，而不是用 ffmpeg + +这是让上面这些数字真正有价值的部分。我没有自己写 ffmpeg 命令、自己算 SSIM——整场标定跑的是**将来真正干活的那个引擎本体**，带着它自己的校验和自己的判收闸门： + +```bash +python3 library_run.py --work-dir /work --tier qsv --force-tier qsv \ + --qsv-quality Q --qsv-preset P --path <真原件> --keep-local --overwrite +``` + +`--keep-local` 的意思是：编码、校验、过闸门，但**不落盘**——我收集的是判决，不是一批新文件。素材通过已有的 SMB 挂载只读进入，而且每个源档在编码前后都做了哈希核对：未变。 + +镜像是 `linuxserver/ffmpeg`（ffmpeg 9.0.2、iHD 26.3.5、oneVPL 2.17）加 `python3`，再加 **numpy**——后者不是可选项。没有它闸门会打 `flat-frame scan skipped: No module named 'numpy'`，悄悄改变近平坦帧的判定方式，也就改变了判决。 + +`-global_quality` 在 `hevc_qsv` 上对应 `RateControlMethod: ICQ`，这就是被标定的旋钮。 + +## 质量阶梯：在瘦源上，低 ICQ 会把文件**变大** + +第一轮，源是一支真实的 1080p25、951 kbps（bpp 0.0184——已经很瘦、榨过一遍的那种）： + +| ICQ | 引擎判决 | mean SSIM | 视频 = 源 | +|---|---|---|---| +| 18 | **FAIL size** — 176.8% | — | — | +| 20 | **FAIL size** — 141.8% | — | — | +| 22 | **FAIL size** — 107.6% | — | — | +| 24 | ok | 0.9967 | 81.1% | +| 26 | ok | 0.9959 | 61.0% | +| 28 | ok | 0.9951 | 46.4% | +| 30 | **FAIL quality** | 0.9942 | — | +| 32 | **FAIL quality** | 0.9942 | — | + +这张表掉出三个结论。 + +第一个反直觉的：**低 ICQ 不等于安全**。ICQ 18–22 产出的文件比源还**大**，于是被引擎的体积闸门（源视频码率的 100%）拒掉。这不是 QSV 独有的毛病——NVENC 对瘦输入用低 `cq` 也是同一个形状——但这里的边界更靠上：可用区间从 24 起，不是 18。 + +第二个，**28 与 30 之间有一道悬崖**。30 和 32 都质量失败，而且失败在**首次尝试**。这句话很重要，因为引擎在质量失败时会用 `global_quality - 2` 重试一次——ICQ 更低、质量更高——并把重试写进日志。一行写着「ok at quality+2」的日志其实是**首次尝试失败**，不是通过。我第一遍在自己的笔记里把这条读反了；规矩是读首次尝试，不读救济结果。 + +第三个，地板决定取舍：在 **0.995 的 mean SSIM 下限**下，q28 的余量是 0.0001，q26 的余量是 0.0010。 + +## 完整文件确认了这张阶梯——而且有一个捷径站得住 + +摘录（180 秒无损切片）是「不用跑十次完整编码就能排出候选顺序」的办法。它只有在与完整文件结果一致时才成立，所以我把真原件端到端跑了一遍： + +| 源 | ICQ | preset | 视频 = 源 | 文件 | SSIM min / mean | 编码 | +|---|---|---|---|---|---|---| +| 254 MB，2023 秒，872 kbps | 26 | medium | **61.0%** | 168,435,266 B（66%） | 0.9938 / **0.9960** | 360 秒 ≈ 5.6× 实时 | +| 同一支 | 28 | medium | 46.3% | 54% | 0.9927 / 0.9952 | 330 秒 | +| 315 MB，3280 秒，546 kbps | 26 | medium | 50.4% | 60% | 0.9965 / **0.9977** | 600 秒 ≈ 5.5× 实时 | + +第一支的 180 秒摘录量到 mean 0.9959，完整文件量到 0.9960。这个接近程度足以让人相信「摘录用来排质量档」这条路——而且现在这是有证据的，不是靠希望。 + +## 那个毫无作用的 preset + +摘录那一层给出的故事很干净：preset 只影响速度——veryfast 30 秒 / 61.6%、fast 30 秒 / 61.0%、medium 60 秒 / 61.0%、slow 与 veryslow 60 秒。结论似乎就是选 veryfast，每支文件省 30 秒。 + +然后完整文件否掉了它。同一支源、同一个 ICQ 26，一次用 `fast`、一次用 `medium`： + +```text +fast: 168,435,266 B 编码 360 秒 mean SSIM 0.9960 +medium: 168,435,266 B 编码 360 秒 mean SSIM 0.9960 +``` + +逐字节同大小、同编码时间、同质量。摘录层的 30 秒对 60 秒，是每一轮固定开销造成的差别，不是编码速度——这正是那种「如果只走到一步就停」会写进配置文件、再写进一篇博客的数字。 + +所以：**摘录只能排质量，永远不能排速度。** 速度必须在完整文件上量，固定开销在那里才会消失。 + +## 最后落了什么 + +`qsv_quality = 26`、`qsv_preset = medium`——源视频码率的 61.0%，mean SSIM 0.9960（min 0.9938），约 5.6× 实时；在片库里最瘦的一支上是 50.4%、mean 0.9977。q28 相对下限的余量只有 0.0002，而一次质量失败的代价是两次整片编码再加一次搁置，所以那多出来的 15 个百分点不值得换这条尾部风险。 + +另外两条值得记下来的量测，它们回答的是「到底是 QSV 的问题还是文件的问题」： + +- **跨编码器对照。** 一支 720p、节奏不均匀的源，QSV q24 量到 mean 0.9154、QSV q28 量到 0.9146、x265 crf26 量到 0.9164——引擎的对齐搜索同时报 `no alignment explains the mismatch`。这条拒绝属于源档，不属于 Quick Sync。 +- **并发。** 引擎拒绝在共享 work root 上起第二个编码器：`another transcode is already running (pid 7, /work/run-qsv.lock, 3 s old)`。换成各自独立的 work root，两个作业能跑——但墙钟 201 秒对单人 107 秒。两倍时间，零吞吐收益。QSV 一次只跑一个作业。 + +## 四个坑，每个都花掉一轮 + +1. **永远不要把共享片库挂到容器的 `/lib`。** 我把 SMB 共享挂进了容器的 `/lib`——而那是容器自己的动态加载器目录。八个作业全在 1 秒内死掉，报 `exec /bin/bash: no such file or directory`，读起来像镜像坏了，而不是挂载点错了。挂 `/library` 就正常。 +2. **合成源会给出与事实相反的 ICQ 结论。** `hevc_qsv` 的码率控制对 `rgb24` 输入会被驱动拒绝（`Selected ratecontrol mode is unsupported`），所以用 `testsrc`/`mandelbrot` 自检会得出相反的结论。要么用真文件，要么加 `format=nv12`。 +3. **已经是 HEVC 的源根本没法标定。** 闸门会回 `SKIP already HEVC hvc1`，`--overwrite` 也不解除。这个片库里肥档 1080p H.264 原件早就转完了，剩下还在肥档的都是 HEVC——所以那一档已经没有可量的素材了。这是缺输入，不是漏做任务。 +4. **读首次尝试，不读重试**（见上面那道悬崖）。 + +## 如果重来一次 + +1. **速度要在你真正会跑的那个产物上量**，不要在它的一片切片上量。整件事里我唯一报错的数字，就是信了摘录的墙钟。 +2. **先让机器证明它有没有这块硬件，再排测试计划。** 六分钟的 `lspci` 加 `virsh dumpxml`，就把「为什么 QSV 在 VM 里失败」这个悬了几周的问题结掉了。 +3. **把闸门留在回路里。** 因为每一次运行都由引擎自己的校验判收，上面这些表格是决策，不是观点——将来在生产里会拒掉坏文件的，正是这个把我的 ICQ 30 拒掉的闸门。 +4. **把诚实的适用范围写进文档。** 这些数字是 UHD 770 上 Linux iHD 26.3.5 + oneVPL 2.17 的数；生产目标是同一块硅片上的 Windows MSDK 路径，我不能声称两者一致。这里站得住的是：命令在真硬件上跑得通、质量↔体积曲线的位置、以及锁的行为——留给将来的，只是「在 Windows+Intel 上按同一张表复测一次」，而不是又一个下午。 + +## 结果 + +QSV 从「存在于代码里、永远选不到」变成了一个带公开数字的已标定档，安全规则也跟着改了：原来那条钉住「**出货默认值必须代表未标定**」的测试，现在钉的是「**出货默认值必须是量过的值**」，同时保留「空值 = 硬排除」的全部断言。在没装 Intel 显卡的机器上，doctor 那行也从信息级的「等硬件」变成了警告——这本身是另一个小教训：一个已标定的编码器自检失败该长什么样，以及为什么机器上不存在的硬件应该进一行汇总，而不是进警告。 diff --git a/src/content/posts/zh/docker-reported-exit-0-the-oom-killer-disagreed.md b/src/content/posts/zh/docker-reported-exit-0-the-oom-killer-disagreed.md new file mode 100644 index 0000000..8724b12 --- /dev/null +++ b/src/content/posts/zh/docker-reported-exit-0-the-oom-killer-disagreed.md @@ -0,0 +1,120 @@ +--- +title: "Docker 说它干净退出，内核说它被 OOM 杀掉了" +description: "NAS 上的容器不停「意外停止」，而 Docker 报的是 exit 0 / OOMKilled=false。真正的证据在 dmesg 里：六次 cgroup 内存击杀。" +pubDate: 2026-10-16 +category: devops +tags: ["docker", "oom", "traefik", "synology", "crawlers"] +ogImage: /og/docker-reported-exit-0-the-oom-killer-disagreed.png +banner: /banners/docker-reported-exit-0-the-oom-killer-disagreed.png +draft: false +--- + +## 一条我以为又是配置写错的告警 + +我的 NAS 发来那条所有 Docker 用户都学会一眼扫过的通知：**容器意外停止**。出事的是我自托管 wiki 前面的那个边缘代理，它已经连着跑了两周。那天早上没人部署过东西，compose 文件也没被动过，所以当时只有两个故事可选：「它崩了」，或者「有东西把它重启了」。 + +两个都错。而真正值得写下来的是：**光看 Docker 自己的报告，我根本分不出来。** + +## Docker 的报告说「什么都没发生」 + +```text +State: + Status: running + Running: true + ExitCode: 0 <- 干净退出 + OOMKilled: false <- 不是被内核杀的 + FinishedAt: 2026-10-15T19:39:33.63057879Z + StartedAt: 2026-10-15T19:39:38.769326672Z <- 5 秒后重启 +RestartCount: 6 +RestartPolicy: always +``` + +照字面读，结论就是：进程干净退出了六次，每次都由重启策略续了命。一个长期运行的代理不会自己以状态 0 退出——而「干净退出」重复六次本身是矛盾，不是答案。`ExitCode` 和 `OOMKilled` 恰好是我在这里最信任的两个字段。在这台机器上，两个都是错的。 + +## 内核自己留了另一份记录 + +在 Linux 上，cgroup 的内存击杀由内核记录，哪怕容器运行时的账本没有把它带过来： + +```text +[1737797.044856] traefik invoked oom-killer: gfp_mask=0x2400040, order=0, oom_score_adj=0 +[1737797.123736] Memory cgroup out of memory: Kill process 615 (traefik) score 970 or sacrifice child +[1737797.132800] Killed process 615 (traefik) total-vm:2525800kB, anon-rss:478480kB, file-rss:0kB +``` + +开机秒数 `1737797` 对应 19:39:33 UTC——和容器的 `FinishedAt` 一秒不差。整个诊断到此结束，代价只是一次 `dmesg`。容器没有退出，它是在自己的内存上限内被杀掉，5 秒后又被拉起来。 + +dmesg 里这样的记录一共 **6 条**，而 `RestartCount` 正是 **6**。同一个 stack 里其他容器的重启计数都是 0、启动时间都是几周前，所以这六次击杀可以钉在这一个容器上： + +| dmesg 开机秒 | 本地时间 | 被杀时 anon-rss | +|---|---|---| +| 1406539 | 10-12 约 07:37 | 491 MB | +| 1407759 | 10-12 约 07:57 | 494 MB | +| 1432574 | 10-12 约 14:49 | 486 MB | +| 1734793 | 10-16 约 02:45 | 487 MB | +| 1735495 | 10-16 约 02:58 | 485 MB | +| 1737797 | 10-16 03:39 ← 那条通知 | 478 MB | + +每一次被杀都发生在 478–494 MB，而上限是 512 MiB（`536,870,912` 字节）。这是天花板被摸到，不是失控。 + +还有两个读数把机制说清了。cgroup 记录的高水位（重启后仍然保留的那个计数器）在重启 **18 分钟**后就已经回到了正好 `536,870,912`——也就是说同一个周期里它又摸到了天花板，只是这次没被杀。而用量曲线是来回震荡的（176 → 308 → 361 MB），不是单调爬升，所以这是负载下的分配抖动撞上小上限，不是一个能证明的缓慢泄漏。最大嫌疑是那个 geoblock 插件：每遇到一个没见过的客户端 IP，它就要做一次外部 HTTPS 查询。 + +同一天早上 05:33 又来第 7 次击杀——在改动落下去之前。 + +在这台 DSM 上，真相只存在于 `dmesg` 加上 cgroup 的 `memory.max_usage_in_bytes`。如果你在普通主机上追同一个形状，先看 `docker inspect -f '{{.State.OOMKilled}}'`——它本该是权威。而这里它读出来是 `false`，就在内核记下那次击杀的 5 秒后。这就是这篇文存在的理由。 + +## 真正在崩的是什么：一场爬虫洪水 + +宿主内存从来不是问题——64 GB 里约 39 GB 可用。出问题的是某个容器的 512 MiB 上限。 + +驱动因素是流量，数字是从容器自己的访问日志里读出来的。最后 50,000 行里： + +```text +不同客户端 IP 25,976 +403 响应 33,636 （geoblock / 拒绝名单在正常工作） +429 响应 16,082 （inflight 与限流在正常工作） +容器 CPU 118–230% +宿主 load average 26–32 +``` + +每次重启后，堆在 12 到 40 分钟内又被填满。配置里没有任何东西是错的，那些 403 和 429 也不是——它们正是过滤器按设计拒绝爬虫的结果。问题在于谁在敲门：这个容器是一个多 TB 级 Wikipedia 语料的公网入口，托管在家庭 NAS 上，前面挂着一份国家白名单和一个机器人过滤器。对住宅代理爬虫云来说，这是个极有吸引力的目标；而那些「请你离开」的响应，本身也是工作量。 + +## 修法，以及必须跟着的那句话 + +```yaml + deploy: + resources: + limits: +- memory: 512M ++ memory: 2g + restart: always ++memswap_limit: 2g +``` + +验证（因为「我把上限调大之后感觉好多了」不叫验证）： + +- **只有那一个容器被重建。** wiki、代理内核、robots 容器都还报着两周的 uptime——用户看到的是几秒的空窗，而不是整个 stack 重启。 +- `HostConfig.Memory = 2147483648`、`MemorySwap = 2147483648`，cgroup 的 `memory.limit_in_bytes` 与两者一致；`RestartCount` 归零。 +- 18 分钟后峰值用量 **210 MB**——而之前它一直贴着 512 MB 的天花板。到中午是 94 MiB。 +- 挂载列表仍与文件一致，包括插件目录那条挂载——丢掉它曾经让这个 stack 里所有路由全部失效。 +- 两个插件都加载且在生效（国家白名单、机器人过滤器），等过滤器的计分窗口过期后，各条路由都返回 200。 + +**以及它不是什么：** 这是权宜之计。驱动因素是洪水，所以大一个数量级的洪水照样能摸到 2 GB。真正的修法是缩小公网入口面，而不是把笼子做大——一条没被标明为权宜之计的上限调整，最终会变成一个永久解释。 + +我选择让这个容器不换页：`memswap_limit` 与内存上限相等就是关掉它的 swap。这台宿主当时已在 40 GB swap 里用掉 16 GB，commit 总量 92 GB，而一个开始换页的边缘代理会把整台 NAS 拖慢——磁盘阵列才是瓶颈，所以换页买到的是更慢的失败，不是更长的命。 + +## 那个差点让我报告「自伤事故」的坑 + +验证期间，我第一次用裸 `curl` 打过去，返回 **403**，看起来完全像我刚重启代理造成的事故。并不是：机器人过滤器给我**自己的客户端 IP** 打了分（`blocked user-agent`，分数 80），然后把它放进一个 15 分钟的计分窗口——因为我一直在猛敲这台机器。 + +线索就在同一轮测试里：`/robots.txt` 返回 200，内部的 `/cdn` 路径返回 400，而两者是同一时刻——说明路由是健康的，只有我自己的请求被过滤。在一台「你同时是客户端」的宿主上，永远不要用裸 curl 做判断：去探一个过滤器会忽略的路径，或者从访问日志里读出一个 200。 + +## 如果重来一次 + +1. **把「意外停止」加 `ExitCode 0` 当成「去读 dmesg」的指令**，而不是干净退出的证据。重启计数大于零的容器，在证明清白之前都当作内存事故来查。 +2. **对 cgroup 高水位告警，而不是只对重启告警。** 重启后那个已经贴着天花板的读数，在通知到达前约十八分钟就告诉我第 7 次击杀要来了。 +3. **早点拿到「谁在打我」这个数字。** 一份日志样本里 25,976 个不同 IP 就是全部故事，而我是在看完整条内存曲线之后才找到它。 +4. **怀疑容器，别怀疑服务器。** 「服务器内存不够」和「一台 64 GB 的机器上某个容器只有 512 MiB 上限」需要完全不同的应对，而后者你不去看那个上限就根本看不见。 + +## 结果 + +改动之后再没有出现 OOM 击杀，18 分钟后的峰值是 210 MB，而过去每一个周期都会摸到上限——四天里一直悄悄重启公网入口的那七次击杀，也从 dmesg 里消失了。爬虫洪水并没有停，所以我把这个失败形态记进了那套 stack 的运维笔记：**「容器意外停止」+ `exit 0` / `OOMKilled=false` 是一个会误导人的签名**，而在这类平台上，诚实的证据住在内核日志和 cgroup 的高水位里。