hoelee-blog/src/content/posts/docker-reported-exit-0-the-oom-killer-disagreed.md

title, description, pubDate, category, tags, ogImage, banner, draft, summary, dynamicCta

title	description	pubDate	category	tags	ogImage	banner	draft	summary	dynamicCta
Docker Reported Exit 0. The Kernel Had Already OOM-Killed It.	A container on my NAS kept stopping unexpectedly while Docker reported exit 0 and OOMKilled=false. The honest evidence was in dmesg: six cgroup OOM kills.	2026-10-16	devops	docker oom traefik synology crawlers	/og/docker-reported-exit-0-the-oom-killer-disagreed.jpg	/banners/docker-reported-exit-0-the-oom-killer-disagreed.jpg	false	audience painPoint impact stack jumpLinks SREs, DevOps Leads & Systems Engineers A public edge proxy (Traefik) crashed 7 times over 4 days while Docker daemon reported ExitCode=0 and OOMKilled=false, rendering standard monitoring completely blind. metric detail Exposed Root Cause Bypassed Docker's false status via kernel dmesg and cgroup high-water marks, pinning the crash to 25,976 crawler IPs hitting a 512MB limit. metric detail Zero-Downtime Hotfix Hot-scaled proxy container to 2GB with memory swap locked, dropping peak memory to 210MB in 18 minutes with 0 subsequent crashes. metric detail Anti-Lockout SOP Established edge crawler stress-testing and verified probe procedures to prevent diagnostic self-lockout from security filters. Docker Linux Kernel (dmesg/cgroups) Traefik Synology DSM Security Filtering business architecture result #the-result #the-kernel-keeps-its-own-record #what-i-would-do-differently	true

The notification I expected to be a config mistake

My NAS sent the notification every Docker user learns to skim: container stopped unexpectedly. This one is the edge proxy in front of my self-hosted wiki, and it had been up for two weeks. Nobody had deployed anything that morning and nothing in the compose file had been touched, so the two available stories were "it crashed" and "something restarted it".

Both were wrong, and the fact that I could not tell from Docker's own report is the part worth writing down.

Docker's report says nothing happened

State:
  Status:      running
  Running:     true
  ExitCode:    0            <- a clean exit
  OOMKilled:   false        <- not killed by the kernel
  FinishedAt:  2026-10-15T19:39:33.63057879Z
  StartedAt:   2026-10-15T19:39:38.769326672Z   <- restarted 5 s later
RestartCount: 6
RestartPolicy: always

Read that as written and the conclusion is: the process exited cleanly, six times, and the restart policy gave it a new life each time. A long-running proxy does not exit with status 0 on its own — and a clean exit that repeats six times is a contradiction, not an answer. ExitCode and OOMKilled are exactly the two fields I would have trusted here. On this host, both were wrong.

The kernel keeps its own record

On Linux, a cgroup memory kill is logged by the kernel even when the container runtime's bookkeeping does not carry it forward:

[1737797.044856] traefik invoked oom-killer: gfp_mask=0x2400040, order=0, oom_score_adj=0
[1737797.123736] Memory cgroup out of memory: Kill process 615 (traefik) score 970 or sacrifice child
[1737797.132800] Killed process 615 (traefik) total-vm:2525800kB, anon-rss:478480kB, file-rss:0kB

The uptime stamp 1737797 corresponds to 19:39:33 UTC — to the second, the container's FinishedAt. That is the whole diagnosis, and it took one dmesg to get it. The container did not exit; it was killed inside its own memory limit and restarted 5 seconds later.

dmesg holds six such events, and RestartCount was 6. Every other container in that stack has a restart count of zero and a start time from weeks ago, so the six kills pin to this one container:

dmesg uptime	local time	anon-rss at kill
1406539	Oct 12, ~07:37	491 MB
1407759	Oct 12, ~07:57	494 MB
1432574	Oct 12, ~14:49	486 MB
1734793	Oct 16, ~02:45	487 MB
1735495	Oct 16, ~02:58	485 MB
1737797	Oct 16, 03:39 ← the notification	478 MB

Every kill happens at 478–494 MB against a limit of 512 MiB (536,870,912 bytes). That is a ceiling being touched, not a runaway.

Two more readings made the mechanism clear. The cgroup's recorded peak — the counter that survives the restart — was already back at exactly 536,870,912 18 minutes after the restart, i.e. the container touched its ceiling again in the same cycle without being killed. And the usage curve oscillated (176 → 308 → 361 MB) instead of climbing monotonically, so this is allocation churn under load hitting a small cap, not a provable slow leak. The prime suspect is a geoblock plugin doing an external HTTPS lookup for every client IP it has not seen before.

A seventh kill landed at 05:33 that morning, before the fix went in.

On this DSM build the truth was only in dmesg plus the cgroup's memory.max_usage_in_bytes. If you are chasing the same shape on a normal host, docker inspect -f '{{.State.OOMKilled}}' is the field to read first — it is supposed to be authoritative. Here it read false five seconds after the kernel had logged the kill, which is the reason this post exists.

What was actually breaking: a crawler flood

Host memory was never the problem — 64 GB installed, ~39 GB available. One container's 512 MiB ceiling was.

The driver was traffic, and the numbers came out of the container's own access log. In the last 50,000 lines:

distinct client IPs      25,976
403 responses            33,636   (geoblock / deny-list doing its job)
429 responses            16,082   (inflight and rate limiting doing their job)
container CPU            118–230%
host load average        26–32

Each restart refilled the heap in 12 to 40 minutes. Nothing in the configuration was wrong, and neither were the 403s and 429s — they are the filter rejecting crawlers exactly as designed. The problem is who is knocking: this container is the public edge for a multi-terabyte Wikipedia corpus served from a home NAS, behind a country allowlist and a bot filter. That is an extremely attractive target for a residential-proxy crawler cloud, and the responses telling it to go away are still work.

The fix, and the sentence that has to come with it

 deploy:
   resources:
     limits:
-      memory: 512M
+      memory: 2g
 restart: always
+memswap_limit: 2g

Verification, because "I raised the limit and it feels better" is not a verification:

• Only that container was recreated. The wiki, the proxy core and the robots container still reported two weeks of uptime — the gap users saw was seconds, not a stack restart.
• HostConfig.Memory = 2147483648, MemorySwap = 2147483648, and the cgroup's memory.limit_in_bytes agreed with both; RestartCount went back to 0.
• Peak usage after 18 minutes: 210 MB, against a ceiling it had been pinning at 512 MB. By midday it sat at 94 MiB.
• The mount list still matched the file, including the plugin directory mount — losing that one had once disabled every router in this stack.
• Both plugins were loaded and active (country allowlist, bot filter), and the routes returned 200 once the filter's score window expired.

And what it is not: this is a stopgap. The flood is the driver, so an order-of-magnitude bigger flood reaches 2 GB too. The real fix is shrinking the public entry surface, not enlarging the cage around it — and a limit bump that is not labelled as a stopgap becomes a permanent explanation.

I chose to disable swap for this container rather than let it swap: memswap_limit equal to the memory limit does exactly that. The host already had 16 GB of its 40 GB swap in use with committed memory at 92 GB, and an edge proxy that starts swapping slows the entire NAS — the disk array is the bottleneck here, so swapping buys a slower failure rather than a longer life.

The trap that nearly made me report a self-inflicted outage

During verification, my first bare curl probes against the site returned 403, and it looked exactly like a regression I had just caused by restarting the proxy. It was not: the bot filter had scored my own client IP (blocked user-agent, score 80) and placed it in a 15-minute score window, because I had been hammering the host.

The tell was in the same test run — /robots.txt returned 200 and the internal /cdn path returned 400 at the same moment, i.e. the router was healthy and only my own requests were being filtered. On a host where you are also a client, never diagnose with your own bare curl: probe a path the filter ignores, or read a 200 out of the access log.

What I would do differently

1. Treat "stopped unexpectedly" plus ExitCode 0 as an instruction to read dmesg, not as evidence of a clean exit. A restart count above zero is a memory investigation until proven otherwise.
2. Alert on the cgroup peak, not only on restarts. The post-restart reading that was already pinned at the ceiling told me a seventh kill was coming about eighteen minutes before the notification arrived.
3. Get the "who is hitting me" number early. 25,976 distinct IPs in one log sample is the entire story, and I found it after spending time on the memory curve.
4. Question the container, not the server. "The server is out of memory" and "one container has a 512 MiB ceiling on a 64 GB machine" need completely different responses, and the second one is invisible unless you look at the limit.

The result

No OOM kill since the change, with a peak at 210 MB after 18 minutes against a ceiling that used to be touched in every single cycle — and the seven kills that had been silently restarting the public edge for four days stopped appearing in dmesg. The crawler flood did not stop, which is why I recorded the failure mode in my ops notes next to the stack: "container stopped unexpectedly" plus exit 0 / OOMKilled=false is a misleading signature, and on this platform the honest evidence lives in the kernel log and the cgroup's high-water mark.