hoelee-blog/src/content/posts/when-a-proxy-timeout-returns-html-instead-of-json.md

ogImage, banner, title, description, pubDate, category, blueprint, blueprintLayer, tags, draft, summary, dynamicCta

ogImage	banner	title	description	pubDate	category	blueprint	blueprintLayer	tags	draft	summary	dynamicCta
/og/when-a-proxy-timeout-returns-html-instead-of-json.jpg	/banners/when-a-proxy-timeout-returns-html-instead-of-json.jpg	When a Proxy Timeout Returns HTML Instead of JSON: A 499 Post-Mortem	Why a self-hosted admin card failed with 'Unexpected token <': an O(n) library audit running inside a GET request, three reverse-proxy hops, and the 499 status no monitor watches — measured from 13.6 s on an idle box to 2,507 s on a busy one.	2026-10-28	engineering	self-hosted-infrastructure	L2-edge-proxy-runtime	nginx fastapi reverse-proxy timeouts caching observability stale-while-revalidate	false	audience painPoint impact stack jumpLinks Full-Stack Engineers, SREs & Platform Architects Who Own Their Own Edge An admin dashboard card died with a JavaScript syntax error, which pointed every investigation at the frontend — while the real failure was an O(n) audit burning one pinned CPU core behind three proxy hops that silently abort at their own timeouts. metric detail Endpoint Latency −98% Moved a 289-to-341-note library audit off the request thread into a single-flight background refresher; the cached read path now answers in 0.24–0.36 s over five consecutive calls instead of 13.6 s idle and 2,507 s under load. metric detail Failure Now Names Its Layer Replaced a blind res.json() with a text-first reader that preserves the HTTP status and the first 80 bytes of an HTML error page, so a proxy abort reports 'HTTP 503 · <!DOCTYPE …' instead of an unactionable SyntaxError. metric detail Zero Fabricated Zeros Added a refreshing flag plus cached_at timestamp with a 3-second client retry, so a stale audit renders as 'computing' rather than as a false zero on a compliance-adjacent counter. FastAPI Python threads nginx authentik Docker cgroups business architecture result #the-card-that-blamed-javascript #failure-mode-an-audit-inside-a-get #the-result	true

The card that blamed JavaScript

The bug report arrived as one line of text from a live panel:

读取自动处理状态失败：SyntaxError: Unexpected token '<', "<!DOCTYPE "

Read literally, that message accuses the frontend. A SyntaxError on a < character is what JSON.parse throws when the payload is not JSON, and the reflexive conclusion — a stale bundle, a typo in a serialiser, a wrong Content-Type — puts the investigation in the wrong process for hours.

The payload was HTML. Something in front of the application had produced a document starting with <!DOCTYPE. An application server does not do that on a JSON route; a reverse proxy does. The error message was therefore a map of the fault, drawn by the layer that was not at fault: the browser faithfully reported that it had been handed the wrong kind of document.

What follows is the post-mortem, the measurements that identified the actual cause, and the three contract changes that made the class of failure impossible to misdiagnose again.

---

Failure mode: an audit inside a GET

The failing route was a status endpoint behind a publishing console:

GET /api/automation

It returns three counters and three feature flags. Two of the counters are cheap; the third is tts_stale, and computing it calls a library-wide audit — enumerate every markdown note, split each into chapters, hash the content of each chapter, then stat the corresponding audio target for each one. All of it ran inside the request thread, on every single poll of the card.

# The shape that broke: a GET that measures the whole library
@app.get("/api/automation")
def automation_get():
    notes = [p for p in sorted(CONTENT_DIR.glob("*.md")) if not p.stem.endswith(".en")]
    missing_cover = [...]
    stale = tts_mod.stale_targets(content_dir=CONTENT_DIR, audio_root=AUDIO_DIR)  # O(n) audit
    return {"counts": {"notes": len(notes), "tts_stale": len(stale)}, ...}

At 289 notes this is a 13.6-second request. That number alone should have disqualified it from living on a polled card, but the second half of the problem is what turned it into an outage: the request has to cross three reverse-proxy hops before it reaches that function — an SSO gate, a portal nginx, and the container network — and each hop owns a timeout that the application neither sets nor can see.

The portal's /api/ location is proxy_read_timeout 120s in the live configuration. The SSO gate in front of it gave up earlier. Once a hop's clock expires it does what an nginx does: it stops waiting, closes the upstream connection, and serves its own HTML error page to the browser. The frontend then ran await res.json() on that document and produced the SyntaxError quoted above.

Nothing in that chain returns a 500. The application never learned the request existed.

---

The evidence triad that proved nothing had failed

Three artefacts, taken together, located the fault without guesswork. Individually, each is ambiguous; one of them actively misleads.

Evidence	Observation	What it eliminates
Portal access log	7 requests for the path, every one 499 with 0 body bytes, zero 4xx, zero 5xx	Rules out application errors and authentication rejects
Application log (uvicorn)	No entry at all for the path during the incident window	Proves the request never reached the app — the proxy aborted first
In-container probe	Same call direct to the app: 17.7 s / 98.5 s / 35.8 s on three consecutive attempts	Confirms the slowness is real and inside the audit, not a network artefact

499 is the status nginx writes when the client closes the connection before the response is sent. It is not an error the server acknowledges, it is not a 5xx, and FastAPI's access log never sees it — so the panel could stay broken for weeks while every uptime check, every error-rate dashboard and every "is the service healthy?" question answered green. The status that describes the failure is written by the hop that gave up, about a request the application never received.

The misleading artefact was the loudest one: the SyntaxError. It is a true statement about the document and a false statement about the cause.

---

The hidden variable: 13.6 s idle, 2,507 s on a busy box

Before the fix, the audit's cost looked like a constant. It is not. The console container is pinned to a single core (cpuset "4") at a quarter of default CPU weight (cpu_shares 256), on a host that also runs an HEVC transcode gate and a content pipeline. The audit competes for that one core with everything else on it.

After the fix moved the audit off the request path, the application began logging its own duration, which turned the hypothesis into a measurement:

automation counts refreshed in 76.1s
automation counts refreshed in 82.3s
automation counts refreshed in 93.5s
automation counts refreshed in 124.2s
automation counts refreshed in 93.5s → 165.1s
automation counts refreshed in 633.6s
automation counts refreshed in 1282.5s
automation counts refreshed in 2507.2s

Same code, same 341-note library, one order of magnitude of spread — and a worst case 184× slower than the figure measured on an idle box. That single fact dissolves the naive fix. Raising the proxy timeout from 120 s to 300 s does not solve a 2,507-second computation; it only converts a failure that arrives at 120 s into a failure that arrives at 300 s, five minutes later, with a longer window of consumed capacity.

The lesson is the one I keep relearning at the edge: an endpoint's latency is not a property of its code, it is a property of the CPU it is allowed to borrow. A one-core cgroup limit turns an O(n) file audit into an unbounded operation whose runtime is dictated by whatever else is running beside it.

---

The fix: separate the read path from the compute path

Three contract changes, each small enough to test independently.

1. The read path never computes. Counts live behind a TTL cache (300 s, AUTOMATION_COUNTS_TTL_SEC to tune). A request either gets a fresh cached value or gets the previous value immediately with refreshing: true, while a single daemon thread recomputes. Single-flight is enforced with a lock and a flag, so a burst of polls cannot start a burst of audits. The cache is warmed once at startup, inside the app's lifespan hook, so the first human to open the panel does not pay for it.

🔍 View the stale-while-revalidate implementation — cache, single-flight refresh thread, startup warm (52 lines)

_AUTOMATION_TTL_SEC = float(os.environ.get("AUTOMATION_COUNTS_TTL_SEC", "300") or 300)
_automation_cache: Dict[str, Any] = {"data": None, "at": 0.0}
_automation_refreshing_flag = [False]
_automation_lock = threading.RLock()


@app.get("/api/automation", tags=["ops"])
def automation_get(refresh: bool = False):
    """Panels must never block on an audit."""
    flags = autoflow.settings(db=db)
    now = time.time()
    cached = dict(_automation_cache)
    fresh = cached["data"] is not None and (now - cached["at"]) < _AUTOMATION_TTL_SEC
    if refresh or not fresh:
        _start_automation_refresh()
    return {
        "flags": flags,
        "counts": (cached["data"] or {}).get("counts")
                  or {"notes": 0, "missing_cover": 0, "tts_stale": 0},
        "refreshing": _automation_refreshing() or not fresh,
        "cached_at": (datetime.fromtimestamp(cached["at"]).isoformat(timespec="seconds")
                      if cached["at"] else None),
    }


def _automation_counts() -> Dict[str, Any]:
    """The slow half: cover gap + stale TTS over the whole library."""
    notes = [p for p in sorted(CONTENT_DIR.glob("*.md")) if not p.stem.endswith(".en")]
    missing_cover = [...]
    stale = tts_mod.stale_targets(content_dir=CONTENT_DIR, audio_root=AUDIO_DIR)
    return {"counts": {"notes": len(notes), "missing_cover": len(missing_cover),
                       "tts_stale": len(stale)},
            "missing_cover": missing_cover[:20]}


def _start_automation_refresh() -> None:
    with _automation_lock:
        if _automation_refreshing_flag[0]:
            return
        _automation_refreshing_flag[0] = True

    def _run() -> None:
        started = time.time()
        try:
            data = _automation_counts()
            _automation_cache.update({"data": data, "at": time.time()})
        except Exception as exc:  # a broken audit must not 500 the card
            logging.warning("automation counts failed: %s", exc)
        finally:
            _automation_refreshing_flag[0] = False
            logging.info("automation counts refreshed in %.1fs", time.time() - started)

    threading.Thread(target=_run, name="automation-counts", daemon=True).start()

2. A stale value must announce itself. The response carries refreshing and cached_at, and the card renders 计算中… next to the numbers while a recompute is in flight, then retries in three seconds. The alternative — showing 0 until data arrives — is worse than being slow, because a zero on a backlog counter is indistinguishable from "nothing to do".

3. The client never parses blindly. res.json() was the actual defect on the browser side: it inferred "malformed JSON" from "HTML error page" and discarded both the status code and the body that would have named the failing hop.

🔍 View the text-first response reader and the retry contract (22 lines)

function autoReadJson(res) {
  // A proxy timeout / 502 / 503 returns an HTML error page. Blind res.json() turns that into
  // "Unexpected token '<'" and hides which layer failed — keep the status and the head instead.
  return res.text().then(function (text) {
    try {
      return { data: JSON.parse(text) };
    } catch (e) {
      var head = String(text || '').trim().slice(0, 80).replace(/\s+/g, ' ');
      return { error: 'HTTP ' + res.status + ' · ' + (head || 'empty response') };
    }
  });
}

// refreshing: show the last known numbers, label them, come back in three seconds.
document.getElementById('auto-counts').textContent =
  '笔记 ' + counts.notes + ' · 缺封面 ' + counts.missing_cover
  + ' · 待朗读 ' + counts.tts_stale + (data.refreshing ? ' · 计算中…' : '');
if (data.refreshing) { autoRetryTimer = setTimeout(window.autoReload, 3000); }

A fourth change is smaller but load-bearing: ?refresh=1 forces a recompute. Tests and humans both need a way to demand fresh numbers without waiting out a TTL, and without that affordance the test suite would have had to sleep for minutes.

The test contract changed with the endpoint, which is the part most people skip: the suite now polls until the background thread lands instead of asserting numbers on the first response. 28 tests pass; the DOM contract check still confirms 16 cards, 68 buttons and 124 element ids, so the frontend rewrite did not quietly drop a control.

---

Reflection: five rules for panel endpoints

1. A dashboard endpoint is not a query interface. It is the most frequently called and least cached surface in an admin UI. Anything O(n) over user data belongs in a background job with a publish step, not in the GET that a browser polls on focus.
2. Your timeout budget is the smallest hop, not the sum. Four proxies each "allowing" 120 s do not add up; the request dies at whichever clock expires first, and the application cannot see any of them. The application-side budget has to be an order of magnitude under the smallest one.
3. 499 deserves an alert. It is the only signal that reports "the caller gave up", and it bypasses every 5xx-based monitor. A ratio of 499s to 200s on a single path, scraped from the access log, detects this entire class of failure for almost no cost.
4. Never trust res.json() on a proxied route. Read text, keep the status, keep the first line. The error message is the only artefact the next engineer will have.
5. Stale-while-revalidate beats a bigger number. Raising a timeout moves the wall; decoupling compute from read removes it — and it removes it for the worst case you have not measured yet (2,507 s here), not just the one you did (13.6 s).

---

The result

• Cached read path: 200 in 0.24–0.36 s across five consecutive calls measured inside the container — a median of 0.26 s against the 13.6 s audit it replaced, 52× faster at the median and 38–57× across the spread.
• Worst observed case eliminated as a user-facing risk. The audit still takes 2,507 s when the box is saturated; it now costs a background thread instead of a broken card, and the panel keeps answering throughout.
• Correctness preserved, not masked. Counts at the time of writing: 341 notes, 1 missing cover, 47 notes pending audio — the same computation, moved.
• The failure can no longer masquerade. If any hop aborts again, the card prints HTTP 503 · <!DOCTYPE html>… and the investigation starts at the right layer.

The single most useful line in this whole exercise was the one that lied: Unexpected token '<'. An HTML error page handed to a JSON caller is not a syntax error — it is a proxy telling you, in the only language it has, that it stopped waiting before your code ever woke up.