---
ogImage: "/og/when-a-proxy-timeout-returns-html-instead-of-json.jpg"
banner: "/banners/when-a-proxy-timeout-returns-html-instead-of-json.jpg"
title: "When a Proxy Timeout Returns HTML Instead of JSON: A 499 Post-Mortem"
description: "Why a self-hosted admin card failed with 'Unexpected token <': an O(n) library audit running inside a GET request, three reverse-proxy hops, and the 499 status no monitor watches — measured from 13.6 s on an idle box to 2,507 s on a busy one."
pubDate: 2026-10-28
category: engineering
blueprint: self-hosted-infrastructure
blueprintLayer: L2-edge-proxy-runtime
tags: ["nginx", "fastapi", "reverse-proxy", "timeouts", "caching", "observability", "stale-while-revalidate"]
draft: false
summary:
  audience: "Full-Stack Engineers, SREs & Platform Architects Who Own Their Own Edge"
  painPoint: "An admin dashboard card died with a JavaScript syntax error, which pointed every investigation at the frontend — while the real failure was an O(n) audit burning one pinned CPU core behind three proxy hops that silently abort at their own timeouts."
  impact:
    - metric: "Endpoint Latency −98%"
      detail: "Moved a 289-to-341-note library audit off the request thread into a single-flight background refresher; the cached read path now answers in 0.24–0.36 s over five consecutive calls instead of 13.6 s idle and 2,507 s under load."
    - metric: "Failure Now Names Its Layer"
      detail: "Replaced a blind res.json() with a text-first reader that preserves the HTTP status and the first 80 bytes of an HTML error page, so a proxy abort reports 'HTTP 503 · <!DOCTYPE …' instead of an unactionable SyntaxError."
    - metric: "Zero Fabricated Zeros"
      detail: "Added a refreshing flag plus cached_at timestamp with a 3-second client retry, so a stale audit renders as 'computing' rather than as a false zero on a compliance-adjacent counter."
  stack: ["FastAPI", "Python threads", "nginx", "authentik", "Docker", "cgroups"]
  jumpLinks:
    business: "#the-card-that-blamed-javascript"
    architecture: "#failure-mode-an-audit-inside-a-get"
    result: "#the-result"
dynamicCta: true
---

## The card that blamed JavaScript

The bug report arrived as one line of text from a live panel:

```
读取自动处理状态失败：SyntaxError: Unexpected token '<', "<!DOCTYPE "
```

Read literally, that message accuses the frontend. A `SyntaxError` on a `<` character is what
`JSON.parse` throws when the payload is not JSON, and the reflexive conclusion — a stale bundle, a
typo in a serialiser, a wrong `Content-Type` — puts the investigation in the wrong process for
hours.

The payload was HTML. Something in front of the application had produced a document starting with
`<!DOCTYPE`. An application server does not do that on a JSON route; a reverse proxy does. The
error message was therefore a map of the fault, drawn by the layer that was *not* at fault: the
browser faithfully reported that it had been handed the wrong kind of document.

What follows is the post-mortem, the measurements that identified the actual cause, and the three
contract changes that made the class of failure impossible to misdiagnose again.

---

## Failure mode: an audit inside a GET

The failing route was a status endpoint behind a publishing console:

```
GET /api/automation
```

It returns three counters and three feature flags. Two of the counters are cheap; the third is
`tts_stale`, and computing it calls a library-wide audit — enumerate every markdown note, split
each into chapters, hash the content of each chapter, then `stat` the corresponding audio target
for each one. All of it ran **inside the request thread**, on every single poll of the card.

```python
# The shape that broke: a GET that measures the whole library
@app.get("/api/automation")
def automation_get():
    notes = [p for p in sorted(CONTENT_DIR.glob("*.md")) if not p.stem.endswith(".en")]
    missing_cover = [...]
    stale = tts_mod.stale_targets(content_dir=CONTENT_DIR, audio_root=AUDIO_DIR)  # O(n) audit
    return {"counts": {"notes": len(notes), "tts_stale": len(stale)}, ...}
```

At 289 notes this is a 13.6-second request. That number alone should have disqualified it from
living on a polled card, but the second half of the problem is what turned it into an outage: the
request has to cross three reverse-proxy hops before it reaches that function — an SSO gate, a
portal nginx, and the container network — and each hop owns a timeout that the application neither
sets nor can see.

The portal's `/api/` location is `proxy_read_timeout 120s` in the live configuration. The SSO gate
in front of it gave up earlier. Once a hop's clock expires it does what an nginx does: it stops
waiting, closes the upstream connection, and serves **its own HTML error page** to the browser. The
frontend then ran `await res.json()` on that document and produced the `SyntaxError` quoted above.

Nothing in that chain returns a 500. The application never learned the request existed.

---

## The evidence triad that proved nothing had failed

Three artefacts, taken together, located the fault without guesswork. Individually, each is
ambiguous; one of them actively misleads.

| Evidence | Observation | What it eliminates |
|---|---|---|
| Portal access log | 7 requests for the path, **every one `499` with 0 body bytes**, zero 4xx, zero 5xx | Rules out application errors and authentication rejects |
| Application log (uvicorn) | **No entry at all** for the path during the incident window | Proves the request never reached the app — the proxy aborted first |
| In-container probe | Same call direct to the app: **17.7 s / 98.5 s / 35.8 s** on three consecutive attempts | Confirms the slowness is real and inside the audit, not a network artefact |

`499` is the status nginx writes when the **client** closes the connection before the response is
sent. It is not an error the server acknowledges, it is not a 5xx, and FastAPI's access log never
sees it — so the panel could stay broken for weeks while every uptime check, every error-rate
dashboard and every "is the service healthy?" question answered green. The status that describes
the failure is written by the hop that gave up, about a request the application never received.

The misleading artefact was the loudest one: the `SyntaxError`. It is a true statement about the
document and a false statement about the cause.

---

## The hidden variable: 13.6 s idle, 2,507 s on a busy box

Before the fix, the audit's cost looked like a constant. It is not. The console container is
pinned to a single core (`cpuset "4"`) at a quarter of default CPU weight (`cpu_shares 256`), on a
host that also runs an HEVC transcode gate and a content pipeline. The audit competes for that one
core with everything else on it.

After the fix moved the audit off the request path, the application began logging its own duration,
which turned the hypothesis into a measurement:

```
automation counts refreshed in 76.1s
automation counts refreshed in 82.3s
automation counts refreshed in 93.5s
automation counts refreshed in 124.2s
automation counts refreshed in 93.5s → 165.1s
automation counts refreshed in 633.6s
automation counts refreshed in 1282.5s
automation counts refreshed in 2507.2s
```

Same code, same 341-note library, one order of magnitude of spread — and a worst case **184×
slower than the figure measured on an idle box**. That single fact dissolves the naive fix. Raising
the proxy timeout from 120 s to 300 s does not solve a 2,507-second computation; it only converts a
failure that arrives at 120 s into a failure that arrives at 300 s, five minutes later, with a
longer window of consumed capacity.

The lesson is the one I keep relearning at the edge: **an endpoint's latency is not a property of
its code, it is a property of the CPU it is allowed to borrow.** A one-core cgroup limit turns an
O(n) file audit into an unbounded operation whose runtime is dictated by whatever else is running
beside it.

---

## The fix: separate the read path from the compute path

Three contract changes, each small enough to test independently.

**1. The read path never computes.** Counts live behind a TTL cache (300 s, `AUTOMATION_COUNTS_TTL_SEC`
to tune). A request either gets a fresh cached value or gets the previous value immediately with
`refreshing: true`, while a single daemon thread recomputes. Single-flight is enforced with a lock
and a flag, so a burst of polls cannot start a burst of audits. The cache is warmed once at startup,
inside the app's lifespan hook, so the first human to open the panel does not pay for it.

<details class="tech-deep-dive">
<summary>🔍 View the stale-while-revalidate implementation — cache, single-flight refresh thread, startup warm (52 lines)</summary>

```python
_AUTOMATION_TTL_SEC = float(os.environ.get("AUTOMATION_COUNTS_TTL_SEC", "300") or 300)
_automation_cache: Dict[str, Any] = {"data": None, "at": 0.0}
_automation_refreshing_flag = [False]
_automation_lock = threading.RLock()


@app.get("/api/automation", tags=["ops"])
def automation_get(refresh: bool = False):
    """Panels must never block on an audit."""
    flags = autoflow.settings(db=db)
    now = time.time()
    cached = dict(_automation_cache)
    fresh = cached["data"] is not None and (now - cached["at"]) < _AUTOMATION_TTL_SEC
    if refresh or not fresh:
        _start_automation_refresh()
    return {
        "flags": flags,
        "counts": (cached["data"] or {}).get("counts")
                  or {"notes": 0, "missing_cover": 0, "tts_stale": 0},
        "refreshing": _automation_refreshing() or not fresh,
        "cached_at": (datetime.fromtimestamp(cached["at"]).isoformat(timespec="seconds")
                      if cached["at"] else None),
    }


def _automation_counts() -> Dict[str, Any]:
    """The slow half: cover gap + stale TTS over the whole library."""
    notes = [p for p in sorted(CONTENT_DIR.glob("*.md")) if not p.stem.endswith(".en")]
    missing_cover = [...]
    stale = tts_mod.stale_targets(content_dir=CONTENT_DIR, audio_root=AUDIO_DIR)
    return {"counts": {"notes": len(notes), "missing_cover": len(missing_cover),
                       "tts_stale": len(stale)},
            "missing_cover": missing_cover[:20]}


def _start_automation_refresh() -> None:
    with _automation_lock:
        if _automation_refreshing_flag[0]:
            return
        _automation_refreshing_flag[0] = True

    def _run() -> None:
        started = time.time()
        try:
            data = _automation_counts()
            _automation_cache.update({"data": data, "at": time.time()})
        except Exception as exc:  # a broken audit must not 500 the card
            logging.warning("automation counts failed: %s", exc)
        finally:
            _automation_refreshing_flag[0] = False
            logging.info("automation counts refreshed in %.1fs", time.time() - started)

    threading.Thread(target=_run, name="automation-counts", daemon=True).start()
```

</details>

**2. A stale value must announce itself.** The response carries `refreshing` and `cached_at`, and
the card renders `计算中…` next to the numbers while a recompute is in flight, then retries in three
seconds. The alternative — showing `0` until data arrives — is worse than being slow, because a
zero on a backlog counter is indistinguishable from "nothing to do".

**3. The client never parses blindly.** `res.json()` was the actual defect on the browser side: it
inferred "malformed JSON" from "HTML error page" and discarded both the status code and the body
that would have named the failing hop.

<details class="tech-deep-dive">
<summary>🔍 View the text-first response reader and the retry contract (22 lines)</summary>

```javascript
function autoReadJson(res) {
  // A proxy timeout / 502 / 503 returns an HTML error page. Blind res.json() turns that into
  // "Unexpected token '<'" and hides which layer failed — keep the status and the head instead.
  return res.text().then(function (text) {
    try {
      return { data: JSON.parse(text) };
    } catch (e) {
      var head = String(text || '').trim().slice(0, 80).replace(/\s+/g, ' ');
      return { error: 'HTTP ' + res.status + ' · ' + (head || 'empty response') };
    }
  });
}

// refreshing: show the last known numbers, label them, come back in three seconds.
document.getElementById('auto-counts').textContent =
  '笔记 ' + counts.notes + ' · 缺封面 ' + counts.missing_cover
  + ' · 待朗读 ' + counts.tts_stale + (data.refreshing ? ' · 计算中…' : '');
if (data.refreshing) { autoRetryTimer = setTimeout(window.autoReload, 3000); }
```

</details>

A fourth change is smaller but load-bearing: `?refresh=1` forces a recompute. Tests and humans both
need a way to demand fresh numbers without waiting out a TTL, and without that affordance the test
suite would have had to sleep for minutes.

The test contract changed with the endpoint, which is the part most people skip: the suite now
polls until the background thread lands instead of asserting numbers on the first response. 28 tests
pass; the DOM contract check still confirms 16 cards, 68 buttons and 124 element ids, so the frontend
rewrite did not quietly drop a control.

---

## Reflection: five rules for panel endpoints

1. **A dashboard endpoint is not a query interface.** It is the most frequently called and least
   cached surface in an admin UI. Anything O(n) over user data belongs in a background job with a
   publish step, not in the GET that a browser polls on focus.
2. **Your timeout budget is the smallest hop, not the sum.** Four proxies each "allowing" 120 s do
   not add up; the request dies at whichever clock expires first, and the application cannot see any
   of them. The application-side budget has to be an order of magnitude under the smallest one.
3. **`499` deserves an alert.** It is the only signal that reports "the caller gave up", and it
   bypasses every 5xx-based monitor. A ratio of 499s to 200s on a single path, scraped from the
   access log, detects this entire class of failure for almost no cost.
4. **Never trust `res.json()` on a proxied route.** Read text, keep the status, keep the first line.
   The error message is the only artefact the next engineer will have.
5. **Stale-while-revalidate beats a bigger number.** Raising a timeout moves the wall; decoupling
   compute from read removes it — and it removes it for the worst case you have not measured yet
   (2,507 s here), not just the one you did (13.6 s).

---

## The result

- **Cached read path: 200 in 0.24–0.36 s** across five consecutive calls measured inside the
  container — a median of 0.26 s against the 13.6 s audit it replaced, **52× faster at the median
  and 38–57× across the spread**.
- **Worst observed case eliminated as a user-facing risk.** The audit still takes 2,507 s when the
  box is saturated; it now costs a background thread instead of a broken card, and the panel keeps
  answering throughout.
- **Correctness preserved, not masked.** Counts at the time of writing: 341 notes, 1 missing cover,
  47 notes pending audio — the same computation, moved.
- **The failure can no longer masquerade.** If any hop aborts again, the card prints
  `HTTP 503 · <!DOCTYPE html>…` and the investigation starts at the right layer.

The single most useful line in this whole exercise was the one that lied: `Unexpected token '<'`.
An HTML error page handed to a JSON caller is not a syntax error — it is a proxy telling you, in the
only language it has, that it stopped waiting before your code ever woke up.