post: when a proxy timeout returns HTML instead of JSON (EN+ZH)

499 post-mortem: an O(n) library audit inside a polled GET on a one-core
cgroup, three reverse-proxy hops, and the status no monitor watches.

- measured: 13.6 s idle / 17.7-98.5-35.8 s probes / 76-2507 s under load
- evidence triad: 7x 499 @ portal, zero app-log entries, in-container repro
- fix: 300 s TTL cache + single-flight refresh + refreshing/cached_at flags,
  text-first client reader, ?refresh=1
- result: cached path 200 in 0.24-0.36 s (median 0.26 s)
- art: ideogram4 base frame + 4 derived targets, art-meta evidence line
- LinkedIn companion: docs/linkedin/batch-07-linkedin-posts.md

docs/linkedin/batch-07-linkedin-posts.md

@@ -0,0 +1,53 @@
+# LinkedIn Technical Posts — Batch 07
+
+> **Target Audience**: CTOs, Engineering Directors, and Principal/Staff Hiring Managers in the US, Europe, and Singapore.
+> **Positioning**: Senior Full-Stack / DevOps / Platform Architect.
+> **Tone & Persona**: Authoritative, production-tested, quantitative, first-person singular (`I`), zero-fluff, no-seller/no-homelab voice.
+> **Status**: Ready for publication / scheduling.
+
+---
+
+## Post 01: The Proxy That Answered a JSON Request With HTML
+
+* **Target Article**: `when-a-proxy-timeout-returns-html-instead-of-json`
+* **Canonical URL**: `https://blog.hoelee.com/posts/when-a-proxy-timeout-returns-html-instead-of-json/`
+* **Category**: Full-Stack / Edge Engineering
+* **Character Count**: ~2,150 chars
+
+```text
+A production admin card failed with one line: "SyntaxError: Unexpected token '<', "<!DOCTYPE ".
+
+Read literally, that message accuses the frontend. It sent the investigation into the wrong process for hours.
+
+The payload was HTML. An application server does not do that on a JSON route. A reverse proxy does, when it stops waiting and serves its own error page.
+
+Root cause: GET /api/automation computed a library-wide audit inside the request thread. 289 notes, each split into chapters, each chapter hashed, each audio target stat()'d. Then the number was rendered on a polled card.
+
+At 289 notes that request took 13.6 s. Three consecutive probes: 17.7 s / 98.5 s / 35.8 s. The request had to cross three proxy hops, each with its own timeout, none of them visible to the application.
+
+Three artefacts found the fault without guessing:
+
+- Portal access log: 7 requests, every one 499 with 0 body bytes, zero 4xx, zero 5xx.
+- Application log: no entry at all for that path. The request never arrived.
+- In-container probe: reproduced the 17.7 / 98.5 / 35.8 s timings.
+
+499 is what nginx writes when the CLIENT hangs up first. It is not a 5xx, so no error-rate dashboard saw it, and FastAPI never logged it. The card could stay broken for weeks while every health check answered green.
+
+The fix was not a bigger timeout. Once the audit moved off the request path, the app began logging its own duration:
+
+76.1 s. 82.3 s. 93.5 s. 124.2 s. 165.1 s. 633.6 s. 1282.5 s. 2507.2 s.
+
+Same code, same library, a 184x spread — because the container is pinned to one core at a quarter of default CPU weight. Raising the proxy timeout from 120 s to 300 s does not fix a 2,507-second computation. It only moves the wall.
+
+What I shipped: a 300 s TTL cache with single-flight background refresh and startup warm-up; a refreshing flag plus cached_at so stale numbers render as "computing" instead of a fake zero; and a text-first response reader that keeps the status and the first 80 bytes, so a proxy abort now prints "HTTP 503 · <!DOCTYPE html>" and names the failing layer.
+
+Result: the cached read path answers 200 in 0.24–0.36 s across five consecutive calls. Median 0.26 s against the 13.6 s audit it replaced — 52x at the median, 38–57x across the spread.
+
+The most useful line in the whole post-mortem was the one that lied.
+
+An HTML error page handed to a JSON caller is not a syntax error. It is a proxy telling you, in the only language it has, that it stopped waiting before your code woke up.
+
+Full measurement chain, both code diffs, and the five rules I now apply to every panel endpoint: https://blog.hoelee.com/posts/when-a-proxy-timeout-returns-html-instead-of-json/
+
+What does your access-log 499 ratio look like on the endpoints nobody polls?
+```