post: model gateway OAuth boundary post-mortem (EN+ZH) + LinkedIn/dev.to copy
Deploy / build (push) Successful in 39s
Deploy / build (push) Successful in 39s
Argon reviewed twice: blog body, then the distribution copy. Adopted: no absolute timestamps, seats framed as workflow isolation, 403->502 error contract fix, two-tier cost math with output pricing, task-dispatch naming.
This commit is contained in:
1 parent
f30b5b17ae
commit
b75899c875
14 files changed
+1822
-1326
No files matched your search
@@ -0,0 +1,96 @@
|
||||
*Originally published on my blog — [read it there with the full context, code and updates](https://blog.hoelee.com/posts/when-a-self-hosted-model-router-tripped-trust-and-safety/).*
|
||||
|
||||
A single self-hosted OpenAI-compatible gateway served every model tier: long-context reading, refactors, cheap background jobs. One key, one model picker, one dashboard where spend was visible in one place. Six independent workflow seats — repo indexing, CI triage, interactive refactors — sat behind it, each authenticating with its own subscription OAuth credential.
|
||||
|
||||
Then all six came back disabled inside one short detection window. None of them had been doing anything unusual. What tripped was a scheduled credential check, and what it found was a block already applied upstream.
|
||||
|
||||
The interesting part was not the block. It was the ten minutes I spent rotating credentials that were never wrong.
|
||||
|
||||
## The error that sent me the wrong way
|
||||
|
||||
For the first ten minutes, every consumer of the gateway's error reported a credential failure — orchestrator, shell probe, CLI — because that is what the body said: `401 invalid_api_key`. The upstream had actually returned a policy decision.
|
||||
|
||||
Three probes separated the layers, and they are worth keeping for any "the model stopped answering" ticket: confirm the gateway answers its own health endpoint; confirm a model from an unrelated provider succeeds on the same key; then loop the subscription models and compare the error codes. One model failing means a broken model. Every model on one provider failing identically, while other providers succeed on the same key, means the provider relationship is closed — and no key rotation moves that.
|
||||
|
||||
The real defect is the error contract, not the outage:
|
||||
|
||||
```python
|
||||
# Before: policy decisions disguised as auth failures
|
||||
if upstream.status == 403:
|
||||
return jsonify(error="invalid_api_key"), 401
|
||||
|
||||
# After: distinct classes, distinct client behaviour
|
||||
if "Terms of Service" in upstream.text or "disabled in this account" in upstream.text:
|
||||
return jsonify(error="upstream_policy_block",
|
||||
upstream_status=403,
|
||||
alert="credential-policy"), 502
|
||||
return jsonify(error="invalid_api_key"), 401
|
||||
```
|
||||
|
||||
A 401 that means "your key is wrong" and a 403 that means "the vendor closed this seat" must never collapse into the same response. Flattening them costs the responder the first ten minutes of every incident.
|
||||
|
||||
## Why a shared token is not a shared key
|
||||
|
||||
An API key is a bearer credential: whoever holds it can use it. An OAuth token is bound to a client identity — client id, consent record, scopes. When a different process presents that token, the request keeps the claims and loses the behaviour: headers, TLS fingerprint, request cadence.
|
||||
|
||||
Two properties of the initial gateway topology made the pattern obvious to any detection system:
|
||||
|
||||
- **One egress.** Every seat left through one container on a single public address.
|
||||
- **One fingerprint.** The gateway normalised headers and stripped client telemetry, so six workflow seats looked like one client at six times the volume.
|
||||
|
||||
Simultaneous failures across independent workflows are not six human reviews. They are one deterministic cluster rule firing. The vendor's terms had said it in advance: scripting the official CLI is acceptable, pooling seats or automating rotation across them is not.
|
||||
|
||||
## Task dispatch instead of routing
|
||||
|
||||
The lesson is not "use fewer seats". It is that a subscription is a seat, not an API. The moment its token leaves the vendor's client, it becomes a shared credential.
|
||||
|
||||
So the gateway keeps serving models paid for by API key, and subscription work moved to task dispatch: the orchestrator runs isolated, single-task invocations of the official CLI bound to a dedicated workflow seat. No pooled credentials, no hijacked persistent session, no rotation. The token never leaves the vendor's binary.
|
||||
|
||||
```
|
||||
orchestrator -> task dispatcher -> official CLI (owns the token) -> vendor
|
||||
|
|
||||
+- cwd, permission scope, timeout, tree-kill, no retries
|
||||
```
|
||||
|
||||
Four dispatcher properties mattered more than any prompt:
|
||||
|
||||
1. **Explicit working directory** — each task starts inside one repository, never a user home directory or a filesystem root, because an agent with edit rights inherits the whole shell.
|
||||
2. **Closed stdin** — interactive prompts in a headless run become errors instead of silent hangs. One forgotten `-y` otherwise blocks the parent forever.
|
||||
3. **Process-group kill on timeout** — killing the parent leaves grandchildren running; kill the group and return a timeout code, not a half-finished diff.
|
||||
4. **No automatic retries** — an inner agent loop times an outer retry policy is hundreds of upstream calls in minutes, the exact cadence that got the previous design flagged.
|
||||
|
||||
```python
|
||||
proc = subprocess.Popen(
|
||||
argv, cwd=repo, stdin=subprocess.DEVNULL, # no interactive hangs
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
start_new_session=True, # own process group
|
||||
)
|
||||
try:
|
||||
out, err = proc.communicate(timeout=task_timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
os.killpg(proc.pid, signal.SIGKILL) # kill the whole tree
|
||||
raise
|
||||
```
|
||||
|
||||
Context stopped travelling in prompts as well: the delegate reads repository instruction files and queries a self-hosted memory service over MCP, so a brief stays six lines instead of a transcript dump. Delegate writes are tagged `status: draft` and final archiving stays with the orchestrator.
|
||||
|
||||
## The economics survived, because the workload splits
|
||||
|
||||
Nine turns in ten are routine orchestration — tool loops, retrieval, cron, log triage, short edits. Those run on a curated open-weight plan: $10/month carrying a **$100 monthly allowance** on the fast model, with per-model pools beside it. At $0.15 per million input tokens and $0.60 per million output, a 4k-in/500-out turn costs about `$0.0009` — under a tenth of a cent.
|
||||
|
||||
The remaining tenth is heavy engineering: multi-file refactors, legacy archaeology, anything with a real build-and-test contract. That dispatches to an authorised CLI seat, where the vendor's client owns the credential and the allowance belongs to the subscription rather than to a pooled bucket.
|
||||
|
||||
Pay for frequency with API-key models, pay for depth with authorised seats, and never let the two share a token.
|
||||
|
||||
### 💬 Let's Compare Notes
|
||||
|
||||
If a credential is bound to a client, where should a platform team draw the execution boundary when metered APIs and client-bound CLI tools have to coexist: standardise on pure APIs and pay the premium, or run a dispatcher that governs official binaries? I have moved to the second, and I would like to hear from anyone who has run the first at team scale and made it survive an audit.
|
||||
|
||||
Two related posts from the same infrastructure work:
|
||||
|
||||
- [Slashing agent context with a code graph](https://blog.hoelee.com/posts/slashing-agent-context-with-a-code-graph/)
|
||||
- [It reported success. Nothing had changed.](https://blog.hoelee.com/posts/it-reported-success-nothing-had-changed/)
|
||||
|
||||
---
|
||||
|
||||
*I write about self-hosted infrastructure, DevOps and the failures behind them at [blog.hoelee.com](https://blog.hoelee.com/).*
|
||||
Reference in new issue
Block a user