post: publish the QSV calibration + the OOM-killer pair (EN+ZH) + og/banner
Deploy / build (push) Successful in 15s
Deploy / build (push) Successful in 15s
- calibrating-intel-quick-sync-hevc-quality (tutorials): pubDate 2026-10-14, updatedDate 2026-10-16 (its real work date; sitemap lastmod stays honest). ffmpeg 9.0.2 pins it to 2026-09-18 at the earliest, so it cannot be backdated — see project-state B2l. - docker-reported-exit-0-the-oom-killer-disagreed (devops): pubDate 2026-10-16, date-pinned by its own dmesg table, ships on the real date. - og measured rows=1/overflow=0, banners delta=2/overflow=0 on both; build 159 pages, no warnings.
This commit is contained in:
hoelee committed
1 parent
609fc75609
commit
7b001dc630
8 files changed
+663
No files matched your search
@@ -0,0 +1,196 @@
|
||||
---
|
||||
title: "Docker Reported Exit 0. The Kernel Had Already OOM-Killed It."
|
||||
description: "A container on my NAS kept stopping unexpectedly while Docker reported exit 0 and OOMKilled=false. The honest evidence was in dmesg: six cgroup OOM kills."
|
||||
pubDate: 2026-10-16
|
||||
category: devops
|
||||
tags: ["docker", "oom", "traefik", "synology", "crawlers"]
|
||||
ogImage: /og/docker-reported-exit-0-the-oom-killer-disagreed.png
|
||||
banner: /banners/docker-reported-exit-0-the-oom-killer-disagreed.png
|
||||
draft: false
|
||||
---
|
||||
|
||||
## The notification I expected to be a config mistake
|
||||
|
||||
My NAS sent the notification every Docker user learns to skim: **container
|
||||
stopped unexpectedly**. This one is the edge proxy in front of my
|
||||
self-hosted wiki, and it had been up for two weeks. Nobody had deployed
|
||||
anything that morning and nothing in the compose file had been touched, so
|
||||
the two available stories were "it crashed" and "something restarted it".
|
||||
|
||||
Both were wrong, and the fact that I could not tell from Docker's own report
|
||||
is the part worth writing down.
|
||||
|
||||
## Docker's report says nothing happened
|
||||
|
||||
```text
|
||||
State:
|
||||
Status: running
|
||||
Running: true
|
||||
ExitCode: 0 <- a clean exit
|
||||
OOMKilled: false <- not killed by the kernel
|
||||
FinishedAt: 2026-10-15T19:39:33.63057879Z
|
||||
StartedAt: 2026-10-15T19:39:38.769326672Z <- restarted 5 s later
|
||||
RestartCount: 6
|
||||
RestartPolicy: always
|
||||
```
|
||||
|
||||
Read that as written and the conclusion is: the process exited cleanly, six
|
||||
times, and the restart policy gave it a new life each time. A long-running
|
||||
proxy does not exit with status 0 on its own — and a clean exit that repeats
|
||||
six times is a contradiction, not an answer. `ExitCode` and `OOMKilled` are
|
||||
exactly the two fields I would have trusted here. On this host, both were
|
||||
wrong.
|
||||
|
||||
## The kernel keeps its own record
|
||||
|
||||
On Linux, a cgroup memory kill is logged by the kernel even when the
|
||||
container runtime's bookkeeping does not carry it forward:
|
||||
|
||||
```text
|
||||
[1737797.044856] traefik invoked oom-killer: gfp_mask=0x2400040, order=0, oom_score_adj=0
|
||||
[1737797.123736] Memory cgroup out of memory: Kill process 615 (traefik) score 970 or sacrifice child
|
||||
[1737797.132800] Killed process 615 (traefik) total-vm:2525800kB, anon-rss:478480kB, file-rss:0kB
|
||||
```
|
||||
|
||||
The uptime stamp `1737797` corresponds to 19:39:33 UTC — to the second, the
|
||||
container's `FinishedAt`. That is the whole diagnosis, and it took one
|
||||
`dmesg` to get it. The container did not exit; it was killed inside its own
|
||||
memory limit and restarted 5 seconds later.
|
||||
|
||||
dmesg holds **six** such events, and `RestartCount` was **6**. Every other
|
||||
container in that stack has a restart count of zero and a start time from
|
||||
weeks ago, so the six kills pin to this one container:
|
||||
|
||||
| dmesg uptime | local time | anon-rss at kill |
|
||||
|---|---|---|
|
||||
| 1406539 | Oct 12, ~07:37 | 491 MB |
|
||||
| 1407759 | Oct 12, ~07:57 | 494 MB |
|
||||
| 1432574 | Oct 12, ~14:49 | 486 MB |
|
||||
| 1734793 | Oct 16, ~02:45 | 487 MB |
|
||||
| 1735495 | Oct 16, ~02:58 | 485 MB |
|
||||
| 1737797 | Oct 16, 03:39 ← the notification | 478 MB |
|
||||
|
||||
Every kill happens at 478–494 MB against a limit of 512 MiB (`536,870,912`
|
||||
bytes). That is a ceiling being touched, not a runaway.
|
||||
|
||||
Two more readings made the mechanism clear. The cgroup's recorded peak — the
|
||||
counter that survives the restart — was already back at exactly
|
||||
`536,870,912` **18 minutes** after the restart, i.e. the container touched
|
||||
its ceiling again in the same cycle *without* being killed. And the usage
|
||||
curve oscillated (176 → 308 → 361 MB) instead of climbing monotonically, so
|
||||
this is allocation churn under load hitting a small cap, not a provable slow
|
||||
leak. The prime suspect is a geoblock plugin doing an external HTTPS lookup
|
||||
for every client IP it has not seen before.
|
||||
|
||||
A seventh kill landed at 05:33 that morning, before the fix went in.
|
||||
|
||||
On this DSM build the truth was only in `dmesg` plus the cgroup's
|
||||
`memory.max_usage_in_bytes`. If you are chasing the same shape on a normal
|
||||
host, `docker inspect -f '{{.State.OOMKilled}}'` is the field to read first
|
||||
— it is supposed to be authoritative. Here it read `false` five seconds
|
||||
after the kernel had logged the kill, which is the reason this post exists.
|
||||
|
||||
## What was actually breaking: a crawler flood
|
||||
|
||||
Host memory was never the problem — 64 GB installed, ~39 GB available. One
|
||||
container's 512 MiB ceiling was.
|
||||
|
||||
The driver was traffic, and the numbers came out of the container's own
|
||||
access log. In the last 50,000 lines:
|
||||
|
||||
```text
|
||||
distinct client IPs 25,976
|
||||
403 responses 33,636 (geoblock / deny-list doing its job)
|
||||
429 responses 16,082 (inflight and rate limiting doing their job)
|
||||
container CPU 118–230%
|
||||
host load average 26–32
|
||||
```
|
||||
|
||||
Each restart refilled the heap in 12 to 40 minutes. Nothing in the
|
||||
configuration was wrong, and neither were the 403s and 429s — they are the
|
||||
filter rejecting crawlers exactly as designed. The problem is who is
|
||||
knocking: this container is the public edge for a multi-terabyte Wikipedia
|
||||
corpus served from a home NAS, behind a country allowlist and a bot filter.
|
||||
That is an extremely attractive target for a residential-proxy crawler
|
||||
cloud, and the responses telling it to go away are still work.
|
||||
|
||||
## The fix, and the sentence that has to come with it
|
||||
|
||||
```yaml
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
- memory: 512M
|
||||
+ memory: 2g
|
||||
restart: always
|
||||
+memswap_limit: 2g
|
||||
```
|
||||
|
||||
Verification, because "I raised the limit and it feels better" is not a
|
||||
verification:
|
||||
|
||||
- **Only that container was recreated.** The wiki, the proxy core and the
|
||||
robots container still reported two weeks of uptime — the gap users saw
|
||||
was seconds, not a stack restart.
|
||||
- `HostConfig.Memory = 2147483648`, `MemorySwap = 2147483648`, and the
|
||||
cgroup's `memory.limit_in_bytes` agreed with both; `RestartCount` went
|
||||
back to 0.
|
||||
- Peak usage after 18 minutes: **210 MB**, against a ceiling it had been
|
||||
pinning at 512 MB. By midday it sat at 94 MiB.
|
||||
- The mount list still matched the file, including the plugin directory
|
||||
mount — losing that one had once disabled every router in this stack.
|
||||
- Both plugins were loaded and active (country allowlist, bot filter), and
|
||||
the routes returned 200 once the filter's score window expired.
|
||||
|
||||
**And what it is not:** this is a stopgap. The flood is the driver, so an
|
||||
order-of-magnitude bigger flood reaches 2 GB too. The real fix is shrinking
|
||||
the public entry surface, not enlarging the cage around it — and a limit
|
||||
bump that is not labelled as a stopgap becomes a permanent explanation.
|
||||
|
||||
I chose to disable swap for this container rather than let it swap:
|
||||
`memswap_limit` equal to the memory limit does exactly that. The host
|
||||
already had 16 GB of its 40 GB swap in use with committed memory at 92 GB,
|
||||
and an edge proxy that starts swapping slows the entire NAS — the disk array
|
||||
is the bottleneck here, so swapping buys a slower failure rather than a
|
||||
longer life.
|
||||
|
||||
## The trap that nearly made me report a self-inflicted outage
|
||||
|
||||
During verification, my first bare `curl` probes against the site returned
|
||||
**403**, and it looked exactly like a regression I had just caused by
|
||||
restarting the proxy. It was not: the bot filter had scored **my own client
|
||||
IP** (`blocked user-agent`, score 80) and placed it in a 15-minute score
|
||||
window, because I had been hammering the host.
|
||||
|
||||
The tell was in the same test run — `/robots.txt` returned 200 and the
|
||||
internal `/cdn` path returned 400 at the same moment, i.e. the router was
|
||||
healthy and only my own requests were being filtered. On a host where you
|
||||
are also a client, never diagnose with your own bare curl: probe a path the
|
||||
filter ignores, or read a 200 out of the access log.
|
||||
|
||||
## What I would do differently
|
||||
|
||||
1. **Treat "stopped unexpectedly" plus `ExitCode 0` as an instruction to
|
||||
read `dmesg`**, not as evidence of a clean exit. A restart count above
|
||||
zero is a memory investigation until proven otherwise.
|
||||
2. **Alert on the cgroup peak, not only on restarts.** The post-restart
|
||||
reading that was already pinned at the ceiling told me a seventh kill was
|
||||
coming about eighteen minutes before the notification arrived.
|
||||
3. **Get the "who is hitting me" number early.** 25,976 distinct IPs in one
|
||||
log sample is the entire story, and I found it after spending time on the
|
||||
memory curve.
|
||||
4. **Question the container, not the server.** "The server is out of memory"
|
||||
and "one container has a 512 MiB ceiling on a 64 GB machine" need
|
||||
completely different responses, and the second one is invisible unless
|
||||
you look at the limit.
|
||||
|
||||
## The result
|
||||
|
||||
No OOM kill since the change, with a peak at 210 MB after 18 minutes against
|
||||
a ceiling that used to be touched in every single cycle — and the seven
|
||||
kills that had been silently restarting the public edge for four days
|
||||
stopped appearing in dmesg. The crawler flood did not stop, which is why I
|
||||
recorded the failure mode in my ops notes next to the stack: "container
|
||||
stopped unexpectedly" plus `exit 0` / `OOMKilled=false` is a misleading
|
||||
signature, and on this platform the honest evidence lives in the kernel log
|
||||
and the cgroup's high-water mark.
|
||||
Reference in new issue
Block a user