post: model gateway OAuth boundary post-mortem (EN+ZH) + LinkedIn/dev.to copy
Deploy / build (push) Successful in 39s
Deploy / build (push) Successful in 39s
Argon reviewed twice: blog body, then the distribution copy. Adopted: no absolute timestamps, seats framed as workflow isolation, 403->502 error contract fix, two-tier cost math with output pricing, task-dispatch naming.
This commit is contained in:
hoelee committed
1 parent
f30b5b17ae
commit
b75899c875
14 files changed
+1822
-1326
No files matched your search
+1339
-1326
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,96 @@
|
||||
*Originally published on my blog — [read it there with the full context, code and updates](https://blog.hoelee.com/posts/when-a-self-hosted-model-router-tripped-trust-and-safety/).*
|
||||
|
||||
A single self-hosted OpenAI-compatible gateway served every model tier: long-context reading, refactors, cheap background jobs. One key, one model picker, one dashboard where spend was visible in one place. Six independent workflow seats — repo indexing, CI triage, interactive refactors — sat behind it, each authenticating with its own subscription OAuth credential.
|
||||
|
||||
Then all six came back disabled inside one short detection window. None of them had been doing anything unusual. What tripped was a scheduled credential check, and what it found was a block already applied upstream.
|
||||
|
||||
The interesting part was not the block. It was the ten minutes I spent rotating credentials that were never wrong.
|
||||
|
||||
## The error that sent me the wrong way
|
||||
|
||||
For the first ten minutes, every consumer of the gateway's error reported a credential failure — orchestrator, shell probe, CLI — because that is what the body said: `401 invalid_api_key`. The upstream had actually returned a policy decision.
|
||||
|
||||
Three probes separated the layers, and they are worth keeping for any "the model stopped answering" ticket: confirm the gateway answers its own health endpoint; confirm a model from an unrelated provider succeeds on the same key; then loop the subscription models and compare the error codes. One model failing means a broken model. Every model on one provider failing identically, while other providers succeed on the same key, means the provider relationship is closed — and no key rotation moves that.
|
||||
|
||||
The real defect is the error contract, not the outage:
|
||||
|
||||
```python
|
||||
# Before: policy decisions disguised as auth failures
|
||||
if upstream.status == 403:
|
||||
return jsonify(error="invalid_api_key"), 401
|
||||
|
||||
# After: distinct classes, distinct client behaviour
|
||||
if "Terms of Service" in upstream.text or "disabled in this account" in upstream.text:
|
||||
return jsonify(error="upstream_policy_block",
|
||||
upstream_status=403,
|
||||
alert="credential-policy"), 502
|
||||
return jsonify(error="invalid_api_key"), 401
|
||||
```
|
||||
|
||||
A 401 that means "your key is wrong" and a 403 that means "the vendor closed this seat" must never collapse into the same response. Flattening them costs the responder the first ten minutes of every incident.
|
||||
|
||||
## Why a shared token is not a shared key
|
||||
|
||||
An API key is a bearer credential: whoever holds it can use it. An OAuth token is bound to a client identity — client id, consent record, scopes. When a different process presents that token, the request keeps the claims and loses the behaviour: headers, TLS fingerprint, request cadence.
|
||||
|
||||
Two properties of the initial gateway topology made the pattern obvious to any detection system:
|
||||
|
||||
- **One egress.** Every seat left through one container on a single public address.
|
||||
- **One fingerprint.** The gateway normalised headers and stripped client telemetry, so six workflow seats looked like one client at six times the volume.
|
||||
|
||||
Simultaneous failures across independent workflows are not six human reviews. They are one deterministic cluster rule firing. The vendor's terms had said it in advance: scripting the official CLI is acceptable, pooling seats or automating rotation across them is not.
|
||||
|
||||
## Task dispatch instead of routing
|
||||
|
||||
The lesson is not "use fewer seats". It is that a subscription is a seat, not an API. The moment its token leaves the vendor's client, it becomes a shared credential.
|
||||
|
||||
So the gateway keeps serving models paid for by API key, and subscription work moved to task dispatch: the orchestrator runs isolated, single-task invocations of the official CLI bound to a dedicated workflow seat. No pooled credentials, no hijacked persistent session, no rotation. The token never leaves the vendor's binary.
|
||||
|
||||
```
|
||||
orchestrator -> task dispatcher -> official CLI (owns the token) -> vendor
|
||||
|
|
||||
+- cwd, permission scope, timeout, tree-kill, no retries
|
||||
```
|
||||
|
||||
Four dispatcher properties mattered more than any prompt:
|
||||
|
||||
1. **Explicit working directory** — each task starts inside one repository, never a user home directory or a filesystem root, because an agent with edit rights inherits the whole shell.
|
||||
2. **Closed stdin** — interactive prompts in a headless run become errors instead of silent hangs. One forgotten `-y` otherwise blocks the parent forever.
|
||||
3. **Process-group kill on timeout** — killing the parent leaves grandchildren running; kill the group and return a timeout code, not a half-finished diff.
|
||||
4. **No automatic retries** — an inner agent loop times an outer retry policy is hundreds of upstream calls in minutes, the exact cadence that got the previous design flagged.
|
||||
|
||||
```python
|
||||
proc = subprocess.Popen(
|
||||
argv, cwd=repo, stdin=subprocess.DEVNULL, # no interactive hangs
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
start_new_session=True, # own process group
|
||||
)
|
||||
try:
|
||||
out, err = proc.communicate(timeout=task_timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
os.killpg(proc.pid, signal.SIGKILL) # kill the whole tree
|
||||
raise
|
||||
```
|
||||
|
||||
Context stopped travelling in prompts as well: the delegate reads repository instruction files and queries a self-hosted memory service over MCP, so a brief stays six lines instead of a transcript dump. Delegate writes are tagged `status: draft` and final archiving stays with the orchestrator.
|
||||
|
||||
## The economics survived, because the workload splits
|
||||
|
||||
Nine turns in ten are routine orchestration — tool loops, retrieval, cron, log triage, short edits. Those run on a curated open-weight plan: $10/month carrying a **$100 monthly allowance** on the fast model, with per-model pools beside it. At $0.15 per million input tokens and $0.60 per million output, a 4k-in/500-out turn costs about `$0.0009` — under a tenth of a cent.
|
||||
|
||||
The remaining tenth is heavy engineering: multi-file refactors, legacy archaeology, anything with a real build-and-test contract. That dispatches to an authorised CLI seat, where the vendor's client owns the credential and the allowance belongs to the subscription rather than to a pooled bucket.
|
||||
|
||||
Pay for frequency with API-key models, pay for depth with authorised seats, and never let the two share a token.
|
||||
|
||||
### 💬 Let's Compare Notes
|
||||
|
||||
If a credential is bound to a client, where should a platform team draw the execution boundary when metered APIs and client-bound CLI tools have to coexist: standardise on pure APIs and pay the premium, or run a dispatcher that governs official binaries? I have moved to the second, and I would like to hear from anyone who has run the first at team scale and made it survive an audit.
|
||||
|
||||
Two related posts from the same infrastructure work:
|
||||
|
||||
- [Slashing agent context with a code graph](https://blog.hoelee.com/posts/slashing-agent-context-with-a-code-graph/)
|
||||
- [It reported success. Nothing had changed.](https://blog.hoelee.com/posts/it-reported-success-nothing-had-changed/)
|
||||
|
||||
---
|
||||
|
||||
*I write about self-hosted infrastructure, DevOps and the failures behind them at [blog.hoelee.com](https://blog.hoelee.com/).*
|
||||
+1
@@ -0,0 +1 @@
|
||||
🔗 Full post-mortem, the isolation probes and the dispatcher core: https://blog.hoelee.com/posts/when-a-self-hosted-model-router-tripped-trust-and-safety/
|
||||
@@ -0,0 +1,13 @@
|
||||
I spent ten minutes rotating credentials that were never wrong — because my AI gateway translated an upstream policy block into HTTP 401.
|
||||
|
||||
In the initial architecture, a self-hosted gateway unified every model tier behind a single interface: long-context reading, refactors, cheap background jobs. One key, one picker, one spend dashboard. Good engineering for a real problem, missing exactly one thing: a boundary.
|
||||
|
||||
An OAuth token is bound to the client that obtained it. Isolated workflow seats egressing through a single container with normalised headers strip client telemetry and read as one anomalous cluster. Simultaneous trips across independent workflows are not manual reviews — they are one deterministic cluster rule firing.
|
||||
|
||||
The root failure was not the upstream block, it was the error contract: turning an upstream HTTP 403 policy rejection into an OpenAI-style 401 invalid_api_key blinded every downstream consumer. That mapping is now fixed — policy blocks return 502 with an explicit upstream_policy_block signal and raise an alert, while credential failures stay 401.
|
||||
|
||||
What replaced the pooled design: task dispatch. The orchestrator dispatches isolated, single-task invocations to the vendor's official CLI bound to a dedicated workflow seat — no credential pooling, no session hijacking, no rotation. Explicit working directory, closed stdin, process-group kill on timeout, zero automatic retries. Context moves through repository instruction files and an MCP memory service, not transcript dumps.
|
||||
|
||||
The economics held because the workload splits: nine turns in ten are routine orchestration on a $10 plan carrying a $100 model allowance (about $0.0009 for a 4k/500 turn), and the last tenth is heavy engineering on an authorised seat.
|
||||
|
||||
For platform teams building internal AI planes: where do you draw the execution boundary when mixing metered APIs with client-bound CLI tools — strict pure-API standardisation at ten times the cost, or a hybrid dispatcher governing official binaries? Where has that boundary broken under load?
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 53 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 32 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 4.2 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 63 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 56 KiB |
@@ -364,5 +364,10 @@
|
||||
"evidence": "SSIM >= 0.995 · 10,000+ assets · 0 corrupt stubs",
|
||||
"ogTitle_en": "5 Quality Gates for FFmpeg",
|
||||
"ogTitle_zh": "视频转码流水线五道质量门禁"
|
||||
},
|
||||
"when-a-self-hosted-model-router-tripped-trust-and-safety": {
|
||||
"evidence": "6 seats - 9 s - 403 to 502 mapping",
|
||||
"ogTitle_en": "The Gateway That Tripped Trust & Safety",
|
||||
"ogTitle_zh": "网关撞上风控"
|
||||
}
|
||||
}
|
||||
@@ -2403,6 +2403,30 @@
|
||||
1360,
|
||||
768
|
||||
]
|
||||
},
|
||||
"when-a-self-hosted-model-router-tripped-trust-and-safety": {
|
||||
"sw": 1930,
|
||||
"sh": 1090,
|
||||
"left": 604,
|
||||
"top": 102,
|
||||
"fill": [
|
||||
14,
|
||||
18,
|
||||
30
|
||||
],
|
||||
"zoom": 1.419,
|
||||
"silhouette": 0.47,
|
||||
"cut": 0.0,
|
||||
"band": [
|
||||
764,
|
||||
123,
|
||||
1294,
|
||||
655
|
||||
],
|
||||
"source_size": [
|
||||
1360,
|
||||
768
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,172 @@
|
||||
---
|
||||
ogImage: "/og/when-a-self-hosted-model-router-tripped-trust-and-safety.jpg"
|
||||
banner: "/banners/when-a-self-hosted-model-router-tripped-trust-and-safety.jpg"
|
||||
title: "When a Self-Hosted Model Gateway Tripped Trust & Safety: Six Seats, Nine Seconds"
|
||||
description: "A post-mortem of consolidating subscription seats behind one OpenAI-compatible gateway: the OAuth client boundary that was crossed, the cluster block that followed nine seconds later, and the task-dispatch architecture that replaced it."
|
||||
pubDate: 2026-11-06
|
||||
category: ai
|
||||
blueprint: ai-agent-automation
|
||||
blueprintLayer: L2-orchestration
|
||||
tags: ["ai-agents", "oauth", "terms-of-service", "post-mortem", "self-hosting", "model-routing"]
|
||||
draft: false
|
||||
summary:
|
||||
audience: "Platform Engineers, AI Tooling Builders & Engineering Leads"
|
||||
painPoint: "One OpenAI-compatible gateway served every model behind a single key, until six subscription seats were disabled inside a nine-second window and every chat request started failing with a misleading 401."
|
||||
impact:
|
||||
- metric: "6 Seats, 9 Seconds"
|
||||
detail: "A scheduled credential health check hit all six connections at once; every one came back disabled. Cluster-level detection, not six individual reviews."
|
||||
- metric: "1 Wrong Error Code"
|
||||
detail: "The gateway translated an upstream policy block into an OpenAI-style 401 invalid_api_key, so the first ten minutes of diagnosis went into credentials that were never wrong."
|
||||
- metric: "10x Cost Leverage Kept"
|
||||
detail: "The replacement two-tier design keeps a $10 subscription carrying a $100 model allowance for routine orchestration, while heavy work dispatches to official CLI seats."
|
||||
stack: ["OpenAI-compatible gateway", "OAuth 2.0 token reuse", "Official vendor CLI", "MCP", "Structured post-mortem"]
|
||||
jumpLinks:
|
||||
business: "#the-nine-second-block"
|
||||
architecture: "#why-a-shared-token-is-not-a-shared-key"
|
||||
result: "#task-dispatch-instead-of-routing"
|
||||
dynamicCta: true
|
||||
---
|
||||
|
||||
## The Nine Second Block <a id="the-nine-second-block"></a>
|
||||
|
||||
For eight months a self-hosted gateway on my internal orchestration node served every model behind one key and one model picker: a long-context model for reading large repos, a frontier model for refactors, an open-weight model for cheap background jobs. One endpoint, one dashboard, one place to see spend. It worked well enough that I stopped noticing what it was actually doing — presenting subscription OAuth tokens from software the vendor never authorised.
|
||||
|
||||
On a recent Friday morning, in the middle of a routine maintenance window, that arrangement ended in nine seconds. The gateway's scheduled credential health check fired across all six connections at once, each returning:
|
||||
|
||||
```
|
||||
403 This service has been disabled in this account
|
||||
for violation of Terms of Service.
|
||||
Please submit an appeal to continue using this product.
|
||||
```
|
||||
|
||||
The first connection failed immediately, and the last followed inside the same nine-second window. None of the six seats had been doing anything unusual; the sweep itself was the trigger, and it found a block that had already been applied upstream. The gateway's own database flipped each connection to `test_status = banned`, `error_code = 403` in a single pass, and the first failed chat in the orchestrator log landed two minutes later.
|
||||
|
||||
The failure mode that mattered was not the block. It was the ten minutes of confident misdiagnosis that followed, because the gateway surfaced an upstream policy decision as an HTTP 401 with `invalid_api_key`. Every client that consumed that error — the orchestrator, shell probes, the CLI — reported a credential problem. Those ten minutes went into rotating keys that were never the issue.
|
||||
|
||||
## Why a Shared Token Is Not a Shared Key <a id="why-a-shared-token-is-not-a-shared-key"></a>
|
||||
|
||||
An API key is a bearer credential issued to whoever holds it. An OAuth token is not. It is bound to a client identity: a client id, a consent record, and scopes granted to a specific application. When a third-party process presents that token, the request keeps the original client's claims but loses the original client's behaviour — headers, TLS fingerprint, request cadence.
|
||||
|
||||
That gap is precisely what Trust & Safety systems are built to detect, and the vendor's terms said so in advance. The acceptable-access clause prohibited reaching the service through third-party software, and the FAQ drew the line finer still: scripting the official CLI directly was acceptable, while pooling seats or automating rotation across them was not supported.
|
||||
|
||||
Two properties of my setup made the pattern unmistakable:
|
||||
|
||||
- **One egress.** All six seats left through the same container on the same public address, so every request shared one network identity.
|
||||
- **One fingerprint.** The gateway normalised headers and stripped client telemetry, so six seats looked like one client generating six times the traffic.
|
||||
|
||||
Nine seconds for six connections is not six human reviews. It is one rule matching one cluster. The seats had been configured for separate workflows — interactive work, CI jobs, staging probes — and I had collapsed their credentials into a single routing layer for convenience. That consolidation is the mistake, and it is independent of how many seats exist.
|
||||
|
||||
## The Forensic Trail <a id="the-forensic-trail"></a>
|
||||
|
||||
The useful part of a post-mortem is the evidence an engineer can re-derive, not the conclusion. Three artifacts settled it within an hour.
|
||||
|
||||
| Artifact | What it proved |
|
||||
|---|---|
|
||||
| Gateway health endpoint | The gateway itself was healthy: HTTP 200, models listed, unrelated providers answering normally |
|
||||
| A non-subscription model on the same key | Returned 200 with tool calls, so the credential and the routing layer were both fine |
|
||||
| Every subscription model | Returned the identical 401, seconds apart, with the upstream 403 embedded in the body |
|
||||
|
||||
The third row is the one that matters. When one model fails, the model is broken. When every model on one provider fails identically while every other provider on the same key succeeds, the failure lives in the provider relationship, and no amount of key rotation will move it.
|
||||
|
||||
The gateway's connection table confirmed the state, and its call log gave the shape of the event:
|
||||
|
||||
```
|
||||
t+0s connectivity sweep 6 connections -> 403 policy block
|
||||
t+9s last connection 6/6 disabled
|
||||
t+28s orchestrator first failed chat request
|
||||
t+96s connection table rows flipped to banned / 403
|
||||
```
|
||||
|
||||
<details class="tech-deep-dive">
|
||||
<summary>🔍 The full isolation probe: four commands that separate credential, gateway and provider (27 lines)</summary>
|
||||
|
||||
```bash
|
||||
# 1. Is the gateway itself alive?
|
||||
curl -s -o /dev/null -w '%{http_code}\n' "$GW/api/health"
|
||||
200
|
||||
|
||||
# 2. Does a model from an unrelated provider answer on the same key?
|
||||
curl -s "$GW/v1/chat/completions" \
|
||||
-H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
|
||||
-d '{"model":"open-weight/unrelated","messages":[{"role":"user","content":"reply OK"}]}' \
|
||||
| jq -r '.choices[0].finish_reason'
|
||||
tool_calls
|
||||
|
||||
# 3. Do the subscription models fail identically?
|
||||
for m in subscription/fast subscription/deep subscription/alt; do
|
||||
curl -s "$GW/v1/chat/completions" \
|
||||
-H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
|
||||
-d "{\"model\":\"$m\",\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}]}" \
|
||||
| jq -r '.error.code // .choices[0].finish_reason'
|
||||
done
|
||||
invalid_api_key
|
||||
invalid_api_key
|
||||
invalid_api_key
|
||||
|
||||
# 4. What does the gateway's own connection table say?
|
||||
sqlite3 gateway.db \
|
||||
"select provider, test_status, error_code, count(*)
|
||||
from connections group by 1,2,3;"
|
||||
subscription|banned|403|6
|
||||
```
|
||||
|
||||
Row 2 succeeding while row 3 fails three times identically is the whole diagnosis: the key works, the gateway works, the provider relationship is closed.
|
||||
|
||||
</details>
|
||||
|
||||
## Task Dispatch Instead of Routing <a id="task-dispatch-instead-of-routing"></a>
|
||||
|
||||
The lesson is not "use fewer seats". The lesson is that a subscription is a **seat**, not an **API**. A seat grants a human, through the vendor's own client, the right to work. The moment its token leaves that client, the seat becomes a shared credential, and enforcement treats it as one.
|
||||
|
||||
So the architecture changed shape. The gateway keeps serving models paid for by API key — that part was never the problem. Subscription work moved to **task dispatch**: the orchestrator runs isolated, single-task invocations of the vendor's official CLI under a seat it is authorised to use, and the token never leaves the vendor's binary. No pooled credentials, no persistent hijacked session, no rotation across seats.
|
||||
|
||||
```
|
||||
orchestrator -> task dispatcher -> official CLI (owns the token) -> vendor
|
||||
|
|
||||
+- cwd, permission scope, timeout, tree-kill, no retries
|
||||
```
|
||||
|
||||
Four properties of the dispatcher did more for safety than any prompt:
|
||||
|
||||
1. **Explicit working directory.** Each task starts inside one repository, never in a user home directory or a filesystem root, because an agent with edit rights inherits the whole shell of the user running it.
|
||||
2. **Closed stdin.** Interactive prompts inside a headless run — a package manager asking to initialise, an SSH client asking to trust a host key — become errors instead of silent hangs. Without this, one forgotten `-y` blocks the parent process forever.
|
||||
3. **Process-group kill on timeout.** Killing the parent leaves grandchildren running; the dispatcher kills the whole group and returns a timeout code instead of a half-finished diff.
|
||||
4. **No automatic retries.** An inner agent loop multiplied by an outer retry policy produces hundreds of upstream calls in minutes — exactly the cadence the previous design was flagged for.
|
||||
|
||||
A minimal version of the execution core is small enough to audit in one screen:
|
||||
|
||||
```python
|
||||
proc = subprocess.Popen(
|
||||
argv, cwd=repo, stdin=subprocess.DEVNULL, # no interactive hangs
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
start_new_session=True, # own process group
|
||||
)
|
||||
try:
|
||||
out, err = proc.communicate(timeout=task_timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
os.killpg(proc.pid, signal.SIGKILL) # kill the whole tree
|
||||
out, err = proc.communicate()
|
||||
return {"status": "timeout", "detail": err[-2000:]}
|
||||
```
|
||||
|
||||
Context stopped travelling in prompts as well. The delegate reads repository instruction files and queries a self-hosted memory service over MCP, so a brief stays six lines instead of a transcript dump. Delegate writes are tagged `status: draft`, and final archiving stays with the orchestrator — a subagent that records its own failed attempts as fact poisons every later job.
|
||||
|
||||
One more fix belongs here, because it addresses the original ten wasted minutes: retained API-key traffic still passes through the gateway, and its error mapping now distinguishes classes instead of flattening them. An upstream policy block surfaces as `502` with `upstream_policy_block` metadata and fires an alert; authentication failures stay `401`. A real credential problem and a vendor decision no longer look identical from the client side.
|
||||
|
||||
## The Economics Still Work <a id="the-economics-still-work"></a>
|
||||
|
||||
Dropping OAuth reuse could have cost more than it saved, so the replacement had to clear a cost bar. It does, because the workload splits cleanly in two.
|
||||
|
||||
**Routine orchestration — roughly nine turns in ten.** High-frequency tool loops, retrieval, cron jobs, log triage and short code edits run on a curated open-weight plan: $10/month carrying a **$100 monthly allowance** on the fast model, with separate per-model pools beside it. Off-peak pricing is $0.15 per million input tokens and $0.60 per million output, so a 4k-in/500-out turn costs `4000 × 0.15/1e6 + 500 × 0.60/1e6 ≈ $0.0009` — just under a tenth of a cent. Two guardrails keep that allowance honest: per-model pools, so throttling one model does not stop the pipeline, and hard escalation rules — two failed attempts on the same error, three or more files touched, more than five exploratory steps, or any irreversible production operation moves the task off the cheap model before it burns the allowance.
|
||||
|
||||
**Heavy engineering — the remaining tenth.** Multi-file refactors, legacy archaeology and anything with a real build-and-test contract dispatch to an official CLI seat, where the vendor's own client owns the credential and the allowance is the subscription's, not a pooled one.
|
||||
|
||||
That split is the actual answer to keeping this cheap without crossing the boundary: pay for frequency with API-key models, pay for depth with authorised seats, and never let the two share a token.
|
||||
|
||||
## Reflection <a id="reflection"></a>
|
||||
|
||||
What I keep from this is that the gateway was genuinely good engineering solving a real problem — one interface for many providers, one place to see spend, failover that worked. What it lacked was a boundary. A credential issued to one client cannot be re-hosted by another and expected to keep its identity, and no amount of clean code changes what the enforcement layer sees: one address, six connections, one fingerprint.
|
||||
|
||||
The pooled pattern is now permanently retired. The seats that remain are used the way the vendor intends — through its client, one task at a time, at a cadence a human developer could plausibly produce — and the compliance work to move subscription access onto properly licensed tiers is in progress.
|
||||
|
||||
Task dispatch is slower to build and uglier to debug than a router that answers for everything. It is also the only version of this that survives an audit.
|
||||
@@ -0,0 +1,172 @@
|
||||
---
|
||||
ogImage: "/og/zh-when-a-self-hosted-model-router-tripped-trust-and-safety.jpg"
|
||||
banner: "/banners/when-a-self-hosted-model-router-tripped-trust-and-safety.jpg"
|
||||
title: "自建模型网关触发风控复盘：九秒之内，六个席位全部熔断"
|
||||
description: "一次把订阅席位收敛到单一 OpenAI 兼容网关的事故复盘：被越过的 OAuth 客户端边界、九秒后到来的集群封锁，以及取代它的任务分派架构。"
|
||||
pubDate: 2026-11-06
|
||||
category: ai
|
||||
blueprint: ai-agent-automation
|
||||
blueprintLayer: L2-orchestration
|
||||
tags: ["ai-agents", "oauth", "terms-of-service", "post-mortem", "self-hosting", "model-routing"]
|
||||
draft: false
|
||||
summary:
|
||||
audience: "平台工程师、AI 工具链开发者与研发负责人"
|
||||
painPoint: "一个 OpenAI 兼容网关把全部模型收在单一密钥之后，直到六个订阅席位在九秒内同时被停用，此后每一次对话请求都以一条误导性的 401 失败。"
|
||||
impact:
|
||||
- metric: "六席位 · 九秒"
|
||||
detail: "一次计划内的凭据健康检查同时打到六条连接，六条全部返回停用。这是集群级检测，不是六次独立审核。"
|
||||
- metric: "一个错误码"
|
||||
detail: "网关把上游的策略决定翻译成 OpenAI 风格的 401 invalid_api_key，前十分钟的诊断因此全部指向了从未出错的凭据。"
|
||||
- metric: "十倍成本杠杆保留"
|
||||
detail: "替代的双层设计保住了 10 美元订阅换 100 美元模型额度用于常规编排，重度工作则分派给官方 CLI 席位。"
|
||||
stack: ["OpenAI 兼容网关", "OAuth 2.0 令牌复用", "厂商官方 CLI", "MCP", "结构化复盘"]
|
||||
jumpLinks:
|
||||
business: "#the-nine-second-block"
|
||||
architecture: "#why-a-shared-token-is-not-a-shared-key"
|
||||
result: "#task-dispatch-instead-of-routing"
|
||||
dynamicCta: true
|
||||
---
|
||||
|
||||
## 九秒熔断 <a id="the-nine-second-block"></a>
|
||||
|
||||
八个月里，内部编排节点上的一个自建网关把我用到的全部模型收进一把密钥、一个模型选择器：长上下文模型读大仓库，前沿模型做重构，开源权重模型跑廉价后台任务。一个端点、一块面板、一处花费视图。它运转得足够顺，以至于我不再去注意它实际在做的事——用厂商从未授权的软件出示订阅的 OAuth 令牌。
|
||||
|
||||
某个周五凌晨的维护窗口里，这套安排以九秒收场。网关计划内的凭据健康检查同时打到全部六条连接，每一条都返回：
|
||||
|
||||
```
|
||||
403 This service has been disabled in this account
|
||||
for violation of Terms of Service.
|
||||
Please submit an appeal to continue using this product.
|
||||
```
|
||||
|
||||
第一条响应立刻失败，最后一条落在同一个九秒窗口内。六个席位当时都没有异常行为；触发者是这次探针本身，它撞上的是上游早已生效的封锁标记。网关自己的数据库在一次写入里把每条连接改成 `test_status = banned`、`error_code = 403`，编排日志里第一条失败的对话在两分钟后出现。
|
||||
|
||||
真正要命的不是封锁，而是随后十分钟十分自信的误判：网关把上游的策略决定以 HTTP 401 `invalid_api_key` 的形式抛出。消费这条错误的每一个客户端——编排器、shell 探测、CLI——都报告凭据有问题。那十分钟全花在轮换从未出错的密钥上。
|
||||
|
||||
## 共享令牌为何不等于共享密钥 <a id="why-a-shared-token-is-not-a-shared-key"></a>
|
||||
|
||||
API 密钥是签发给持有一方的持有者凭据。OAuth 令牌不是。它绑定的是一个客户端身份：客户端 ID、一份同意记录、一组授予特定应用的权限范围。当第三方进程出示这份令牌，请求仍然带着原客户端的声明，却丢掉了原客户端的行为——请求头、TLS 指纹、请求节奏。
|
||||
|
||||
这个缺口正是风控系统被造出来要检测的东西，而厂商条款事先就写清了。可接受访问的条款禁止通过第三方软件触达服务，FAQ 把界线划得更细：直接脚本化官方 CLI 可以接受，而汇集席位、或自动化地在席位之间轮换，不受支持。
|
||||
|
||||
我自己的两个设计决定，让这个模式一目了然：
|
||||
|
||||
- **单一出口。** 六个席位全部经由同一台容器、同一个公网地址发出，于是每条请求共享一个网络身份。
|
||||
- **单一指纹。** 网关规范化了请求头、剥离了客户端遥测，于是六个席位看起来像一个客户端制造了六倍流量。
|
||||
|
||||
六条连接只用九秒，不是六个人在审核六个案例，而是一条规则命中了一个集群。这些席位原本服务于不同工作流——交互式工作、CI 任务、预发布探针——我为了省事把它们的凭据收敛进了同一个路由层。那次收敛才是错误本身，与席位数无关。
|
||||
|
||||
## 取证链路 <a id="the-forensic-trail"></a>
|
||||
|
||||
复盘里有价值的是别人能重新推演的证据，不是结论。三个产物在一小时内定了案。
|
||||
|
||||
| 产物 | 它证明了什么 |
|
||||
|---|---|
|
||||
| 网关健康端点 | 网关本身健康：HTTP 200，模型列表正常，非订阅类供应商照常应答 |
|
||||
| 同一把密钥上的非订阅模型 | 返回 200 且带工具调用，说明凭据与路由层都没问题 |
|
||||
| 全部订阅类模型 | 返回完全一致的 401，前后相差数秒，响应体里嵌着上游 403 |
|
||||
|
||||
第三行是关键。单个模型失败，是模型坏了。一个供应商下所有模型以同样方式失败、而同一把密钥上其他供应商全部成功，故障就在供应商关系里，轮换密钥一步也挪不动它。
|
||||
|
||||
网关的连接表确认了状态，调用日志给出了事件的形状：
|
||||
|
||||
```
|
||||
t+0s connectivity sweep 6 connections -> 403 policy block
|
||||
t+9s last connection 6/6 disabled
|
||||
t+28s orchestrator first failed chat request
|
||||
t+96s connection table rows flipped to banned / 403
|
||||
```
|
||||
|
||||
<details class="tech-deep-dive">
|
||||
<summary>🔍 完整隔离探测：四条命令分开凭据、网关与供应商（27 行）</summary>
|
||||
|
||||
```bash
|
||||
# 1. 网关自己活着吗？
|
||||
curl -s -o /dev/null -w '%{http_code}\n' "$GW/api/health"
|
||||
200
|
||||
|
||||
# 2. 同一把密钥上，无关供应商的模型能答吗？
|
||||
curl -s "$GW/v1/chat/completions" \
|
||||
-H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
|
||||
-d '{"model":"open-weight/unrelated","messages":[{"role":"user","content":"reply OK"}]}' \
|
||||
| jq -r '.choices[0].finish_reason'
|
||||
tool_calls
|
||||
|
||||
# 3. 订阅类模型是否以同样方式失败？
|
||||
for m in subscription/fast subscription/deep subscription/alt; do
|
||||
curl -s "$GW/v1/chat/completions" \
|
||||
-H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
|
||||
-d "{\"model\":\"$m\",\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}]}" \
|
||||
| jq -r '.error.code // .choices[0].finish_reason'
|
||||
done
|
||||
invalid_api_key
|
||||
invalid_api_key
|
||||
invalid_api_key
|
||||
|
||||
# 4. 网关自己的连接表怎么说？
|
||||
sqlite3 gateway.db \
|
||||
"select provider, test_status, error_code, count(*)
|
||||
from connections group by 1,2,3;"
|
||||
subscription|banned|403|6
|
||||
```
|
||||
|
||||
第 2 行成功、第 3 行连续三次以同样方式失败，诊断就齐了：密钥没坏，网关没坏，是供应商关系被关闭了。
|
||||
|
||||
</details>
|
||||
|
||||
## 改为任务分派，而不是路由 <a id="task-dispatch-instead-of-routing"></a>
|
||||
|
||||
教训不是"少用几个席位"。教训是：订阅是一把**席位**，不是一个**API**。席位授予的是一个人通过厂商自己的客户端工作的权利。令牌一旦离开那个客户端，席位就变成共享凭据，执法也照共享凭据处理。
|
||||
|
||||
于是架构改了形状。网关继续服务那些用 API 密钥付费的模型——那部分从来不是问题。订阅类工作改成**任务分派**：编排器在获得授权的席位下，运行厂商官方 CLI 的隔离式单任务调用，令牌始终不离开厂商的二进制文件。没有汇集凭据，没有劫持长连接会话，也没有跨席位轮换。
|
||||
|
||||
```
|
||||
orchestrator -> task dispatcher -> official CLI (owns the token) -> vendor
|
||||
|
|
||||
+- cwd, permission scope, timeout, tree-kill, no retries
|
||||
```
|
||||
|
||||
这个分派器的四条属性，比任何提示词都更管用：
|
||||
|
||||
1. **显式工作目录。** 每个任务只在某一个仓库内启动，绝不在用户主目录或文件系统根目录启动；一个拥有编辑权限的 agent 继承的是运行它的用户的整个 shell。
|
||||
2. **关闭标准输入。** headless 运行里的交互式提问——包管理器询问是否初始化、SSH 客户端询问是否信任主机密钥——会变成错误而不是静默挂起。少了这一步，一个忘写的 `-y` 就能把父进程永远卡住。
|
||||
3. **超时杀进程组。** 只杀父进程会留下孙进程；分派器杀掉整个进程组并返回超时码，而不是交回一份做了一半的 diff。
|
||||
4. **不自动重试。** 内层 agent 循环乘以外层重试策略，几分钟内就是几百次上游调用——正是上一套设计被标记的节奏。
|
||||
|
||||
执行内核的最小实现，小到可以一屏审完：
|
||||
|
||||
```python
|
||||
proc = subprocess.Popen(
|
||||
argv, cwd=repo, stdin=subprocess.DEVNULL, # no interactive hangs
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
start_new_session=True, # own process group
|
||||
)
|
||||
try:
|
||||
out, err = proc.communicate(timeout=task_timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
os.killpg(proc.pid, signal.SIGKILL) # kill the whole tree
|
||||
out, err = proc.communicate()
|
||||
return {"status": "timeout", "detail": err[-2000:]}
|
||||
```
|
||||
|
||||
上下文也不再靠提示词搬运。委托进程从仓库级指令文件读项目规则，通过 MCP 查询自建记忆服务，所以一份委托简报只有六行，而不是一份对话倾泻。委托方写入的记忆带 `status: draft` 标记，最终归档权留在编排者手里——一个把自己失败的尝试当事实记下来的子 agent，会污染之后每一次任务。
|
||||
|
||||
还有一处修复属于这里，因为它正好回答最初浪费的那十分钟：保留下来的 API 密钥流量仍然经过网关，而它的错误映射已经改为区分类别，而不是抹平。上游的策略封锁以 `502` 加 `upstream_policy_block` 元数据呈现并触发告警；认证失败仍然是 `401`。真正的凭据问题与厂商的决定，在客户端一侧不再长得一模一样。
|
||||
|
||||
## 经济性依然成立 <a id="the-economics-still-work"></a>
|
||||
|
||||
丢掉 OAuth 复用本可能让成本高过收益，所以替代方案必须先过成本线。它过了，因为负载干净地分成两半。
|
||||
|
||||
**常规编排——大约每十轮里的九轮。** 高频工具循环、检索、cron、日志分诊与短代码修改，跑在一份精选开源权重计划上：每月 10 美元，快速模型带**每月 100 美元额度**，旁边还有按模型划分的独立池。低峰价是每百万输入 token 0.15 美元、每百万输出 0.60 美元，于是一次 4k 输入 / 500 输出的回合成本为 `4000 × 0.15/1e6 + 500 × 0.60/1e6 ≈ $0.0009`，略低于十分之一美分。两个护栏让这份额度不至于成为幻想：按模型分池，一个模型被限流不至于让流水线停摆；以及硬性升级规则——同一错误连续失败两次、触及三个以上文件、超过五步探索性诊断、或任何不可逆的生产操作，都要在烧掉额度之前转到更强模型。
|
||||
|
||||
**重度工程——剩下的那一成。** 多文件重构、遗产代码考古、以及任何带真实构建与测试契约的任务，分派给官方 CLI 席位：凭据由厂商自己的客户端持有，额度属于那份订阅，而不是一个汇集池。
|
||||
|
||||
这个划分才是"怎样既便宜又不越过边界"的答案：用 API 密钥模型为频率付费，用授权席位为深度付费，绝不让两者共享同一份令牌。
|
||||
|
||||
## 反思 <a id="reflection"></a>
|
||||
|
||||
我要留下的是这一点：那个网关本身是解决真问题的好工程——多供应商一个接口、一处看花费、可用的故障转移。它缺的是一条边界。签发给某个客户端的凭据，不能被另一个客户端重新托管并指望它保留身份；代码写得再干净，也改变不了执法层看到的事实：一个地址、六条连接、一个指纹。
|
||||
|
||||
汇集模式现已永久退役。保留下来的席位按厂商意图使用——经由它的客户端、一次一个任务、频率落在人类开发者能做到的范围；而把订阅访问迁到正规授权层级的合规工作正在进行。
|
||||
|
||||
任务分派比一个什么都答的路由器更难建、更难调。它也是这套东西里唯一能扛过一次审计的版本。
|
||||
Reference in new issue
Block a user