hoelee-blog/docs/linkedin/batch-07-linkedin-posts.md

LinkedIn Technical Posts — Batch 07

Target Audience: CTOs, Engineering Directors, and Principal/Staff Hiring Managers in the US, Europe, and Singapore. Positioning: Senior Full-Stack / DevOps / Platform Architect. Tone & Persona: Authoritative, production-tested, quantitative, first-person singular (I), zero-fluff, no-seller/no-homelab voice. Status: Ready for publication / scheduling.

---

Post 01: The Proxy That Answered a JSON Request With HTML

• Target Article: when-a-proxy-timeout-returns-html-instead-of-json
• Canonical URL: https://blog.hoelee.com/posts/when-a-proxy-timeout-returns-html-instead-of-json/
• Category: Full-Stack / Edge Engineering
• Character Count: ~2,150 chars

A production admin card failed with one line: "SyntaxError: Unexpected token '<', "<!DOCTYPE ".

Read literally, that message accuses the frontend. It sent the investigation into the wrong process for hours.

The payload was HTML. An application server does not do that on a JSON route. A reverse proxy does, when it stops waiting and serves its own error page.

Root cause: GET /api/automation computed a library-wide audit inside the request thread. 289 notes, each split into chapters, each chapter hashed, each audio target stat()'d. Then the number was rendered on a polled card.

At 289 notes that request took 13.6 s. Three consecutive probes: 17.7 s / 98.5 s / 35.8 s. The request had to cross three proxy hops, each with its own timeout, none of them visible to the application.

Three artefacts found the fault without guessing:

- Portal access log: 7 requests, every one 499 with 0 body bytes, zero 4xx, zero 5xx.
- Application log: no entry at all for that path. The request never arrived.
- In-container probe: reproduced the 17.7 / 98.5 / 35.8 s timings.

499 is what nginx writes when the CLIENT hangs up first. It is not a 5xx, so no error-rate dashboard saw it, and FastAPI never logged it. The card could stay broken for weeks while every health check answered green.

The fix was not a bigger timeout. Once the audit moved off the request path, the app began logging its own duration:

76.1 s. 82.3 s. 93.5 s. 124.2 s. 165.1 s. 633.6 s. 1282.5 s. 2507.2 s.

Same code, same library, a 184x spread — because the container is pinned to one core at a quarter of default CPU weight. Raising the proxy timeout from 120 s to 300 s does not fix a 2,507-second computation. It only moves the wall.

What I shipped: a 300 s TTL cache with single-flight background refresh and startup warm-up; a refreshing flag plus cached_at so stale numbers render as "computing" instead of a fake zero; and a text-first response reader that keeps the status and the first 80 bytes, so a proxy abort now prints "HTTP 503 · <!DOCTYPE html>" and names the failing layer.

Result: the cached read path answers 200 in 0.24–0.36 s across five consecutive calls. Median 0.26 s against the 13.6 s audit it replaced — 52x at the median, 38–57x across the spread.

The most useful line in the whole post-mortem was the one that lied.

An HTML error page handed to a JSON caller is not a syntax error. It is a proxy telling you, in the only language it has, that it stopped waiting before your code woke up.

Full measurement chain, both code diffs, and the five rules I now apply to every panel endpoint: https://blog.hoelee.com/posts/when-a-proxy-timeout-returns-html-instead-of-json/

What does your access-log 499 ratio look like on the endpoints nobody polls?