hoelee
b75899c875
Deploy / build (push) Successful in 39s
Argon reviewed twice: blog body, then the distribution copy. Adopted: no absolute timestamps, seats framed as workflow isolation, 403->502 error contract fix, two-tier cost math with output pricing, task-dispatch naming.
97 lines
7.1 KiB
Markdown
97 lines
7.1 KiB
Markdown
*Originally published on my blog — [read it there with the full context, code and updates](https://blog.hoelee.com/posts/when-a-self-hosted-model-router-tripped-trust-and-safety/).*
|
|
|
|
A single self-hosted OpenAI-compatible gateway served every model tier: long-context reading, refactors, cheap background jobs. One key, one model picker, one dashboard where spend was visible in one place. Six independent workflow seats — repo indexing, CI triage, interactive refactors — sat behind it, each authenticating with its own subscription OAuth credential.
|
|
|
|
Then all six came back disabled inside one short detection window. None of them had been doing anything unusual. What tripped was a scheduled credential check, and what it found was a block already applied upstream.
|
|
|
|
The interesting part was not the block. It was the ten minutes I spent rotating credentials that were never wrong.
|
|
|
|
## The error that sent me the wrong way
|
|
|
|
For the first ten minutes, every consumer of the gateway's error reported a credential failure — orchestrator, shell probe, CLI — because that is what the body said: `401 invalid_api_key`. The upstream had actually returned a policy decision.
|
|
|
|
Three probes separated the layers, and they are worth keeping for any "the model stopped answering" ticket: confirm the gateway answers its own health endpoint; confirm a model from an unrelated provider succeeds on the same key; then loop the subscription models and compare the error codes. One model failing means a broken model. Every model on one provider failing identically, while other providers succeed on the same key, means the provider relationship is closed — and no key rotation moves that.
|
|
|
|
The real defect is the error contract, not the outage:
|
|
|
|
```python
|
|
# Before: policy decisions disguised as auth failures
|
|
if upstream.status == 403:
|
|
return jsonify(error="invalid_api_key"), 401
|
|
|
|
# After: distinct classes, distinct client behaviour
|
|
if "Terms of Service" in upstream.text or "disabled in this account" in upstream.text:
|
|
return jsonify(error="upstream_policy_block",
|
|
upstream_status=403,
|
|
alert="credential-policy"), 502
|
|
return jsonify(error="invalid_api_key"), 401
|
|
```
|
|
|
|
A 401 that means "your key is wrong" and a 403 that means "the vendor closed this seat" must never collapse into the same response. Flattening them costs the responder the first ten minutes of every incident.
|
|
|
|
## Why a shared token is not a shared key
|
|
|
|
An API key is a bearer credential: whoever holds it can use it. An OAuth token is bound to a client identity — client id, consent record, scopes. When a different process presents that token, the request keeps the claims and loses the behaviour: headers, TLS fingerprint, request cadence.
|
|
|
|
Two properties of the initial gateway topology made the pattern obvious to any detection system:
|
|
|
|
- **One egress.** Every seat left through one container on a single public address.
|
|
- **One fingerprint.** The gateway normalised headers and stripped client telemetry, so six workflow seats looked like one client at six times the volume.
|
|
|
|
Simultaneous failures across independent workflows are not six human reviews. They are one deterministic cluster rule firing. The vendor's terms had said it in advance: scripting the official CLI is acceptable, pooling seats or automating rotation across them is not.
|
|
|
|
## Task dispatch instead of routing
|
|
|
|
The lesson is not "use fewer seats". It is that a subscription is a seat, not an API. The moment its token leaves the vendor's client, it becomes a shared credential.
|
|
|
|
So the gateway keeps serving models paid for by API key, and subscription work moved to task dispatch: the orchestrator runs isolated, single-task invocations of the official CLI bound to a dedicated workflow seat. No pooled credentials, no hijacked persistent session, no rotation. The token never leaves the vendor's binary.
|
|
|
|
```
|
|
orchestrator -> task dispatcher -> official CLI (owns the token) -> vendor
|
|
|
|
|
+- cwd, permission scope, timeout, tree-kill, no retries
|
|
```
|
|
|
|
Four dispatcher properties mattered more than any prompt:
|
|
|
|
1. **Explicit working directory** — each task starts inside one repository, never a user home directory or a filesystem root, because an agent with edit rights inherits the whole shell.
|
|
2. **Closed stdin** — interactive prompts in a headless run become errors instead of silent hangs. One forgotten `-y` otherwise blocks the parent forever.
|
|
3. **Process-group kill on timeout** — killing the parent leaves grandchildren running; kill the group and return a timeout code, not a half-finished diff.
|
|
4. **No automatic retries** — an inner agent loop times an outer retry policy is hundreds of upstream calls in minutes, the exact cadence that got the previous design flagged.
|
|
|
|
```python
|
|
proc = subprocess.Popen(
|
|
argv, cwd=repo, stdin=subprocess.DEVNULL, # no interactive hangs
|
|
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
|
start_new_session=True, # own process group
|
|
)
|
|
try:
|
|
out, err = proc.communicate(timeout=task_timeout)
|
|
except subprocess.TimeoutExpired:
|
|
os.killpg(proc.pid, signal.SIGKILL) # kill the whole tree
|
|
raise
|
|
```
|
|
|
|
Context stopped travelling in prompts as well: the delegate reads repository instruction files and queries a self-hosted memory service over MCP, so a brief stays six lines instead of a transcript dump. Delegate writes are tagged `status: draft` and final archiving stays with the orchestrator.
|
|
|
|
## The economics survived, because the workload splits
|
|
|
|
Nine turns in ten are routine orchestration — tool loops, retrieval, cron, log triage, short edits. Those run on a curated open-weight plan: $10/month carrying a **$100 monthly allowance** on the fast model, with per-model pools beside it. At $0.15 per million input tokens and $0.60 per million output, a 4k-in/500-out turn costs about `$0.0009` — under a tenth of a cent.
|
|
|
|
The remaining tenth is heavy engineering: multi-file refactors, legacy archaeology, anything with a real build-and-test contract. That dispatches to an authorised CLI seat, where the vendor's client owns the credential and the allowance belongs to the subscription rather than to a pooled bucket.
|
|
|
|
Pay for frequency with API-key models, pay for depth with authorised seats, and never let the two share a token.
|
|
|
|
### 💬 Let's Compare Notes
|
|
|
|
If a credential is bound to a client, where should a platform team draw the execution boundary when metered APIs and client-bound CLI tools have to coexist: standardise on pure APIs and pay the premium, or run a dispatcher that governs official binaries? I have moved to the second, and I would like to hear from anyone who has run the first at team scale and made it survive an audit.
|
|
|
|
Two related posts from the same infrastructure work:
|
|
|
|
- [Slashing agent context with a code graph](https://blog.hoelee.com/posts/slashing-agent-context-with-a-code-graph/)
|
|
- [It reported success. Nothing had changed.](https://blog.hoelee.com/posts/it-reported-success-nothing-had-changed/)
|
|
|
|
---
|
|
|
|
*I write about self-hosted infrastructure, DevOps and the failures behind them at [blog.hoelee.com](https://blog.hoelee.com/).*
|