8.3 KiB
8.3 KiB
LinkedIn Technical Posts — Batch 08 (Argon 审核定稿版)
Target Audience: CTOs, Engineering Directors, and Principal/Staff Hiring Managers in the US, Europe, and Singapore. Positioning: Senior Full-Stack / DevOps / Platform Architect. Tone & Persona: Authoritative, production-tested, quantitative, first-person singular (
I), zero-fluff, no-homelab/no-shopkeeper voice. Status: Approved by Consultant Argon (2026-11-01).
Post 01: The OIDC Redirect Loop Where Every Service Logged 100% Success
- Target Article:
authentik-oidc-redirect-loop-node-set-header-overwrite - Canonical URL:
https://blog.hoelee.com/posts/authentik-oidc-redirect-loop-node-set-header-overwrite/ - Category: Identity Architecture / Node.js
- Character Count: ~2,100 chars
- Argon Verdict: Approved
A custom admin portal failed with the most frustrating error a browser can show: ERR_TOO_MANY_REDIRECTS.
Users entered their credentials at Authentik, completed multi-factor challenges, consented to permissions, and hit the callback. Then the browser bounced right back to Authentik, which immediately issued a new code and bounced them back again.
Inside the container, every single request logged HTTP 302 Found. Authentik logged successful token exchanges. The OpenID Connect tokens were cryptographically sound. Zero errors, zero warnings.
The bug lived in a helper function written years ago:
function setCookie(res, cookieStr) {
res.setHeader('Set-Cookie', cookieStr);
}
In the Node.js http module (OutgoingMessage), calling res.setHeader() with a string argument is an atomic replacement. It does not append.
During the OIDC callback, two cookie operations happen:
1. Establish the authenticated session: app_sess=<token>; HttpOnly; Secure; SameSite=Lax
2. Expire the transient verification token: app_oidc=; Max-Age=0
The second call completely erased the first.
A curl -v probe revealed what the application logs never saw: the browser received an HTTP 302 to /admin accompanied by exactly one Set-Cookie header—the expired transient state. The session cookie never touched the client. The browser hit /admin, carried no credentials, failed the route guard, and was dispatched back to Authentik.
Because each component executed its isolated logic flawlessly, the system was caught in an infinite loop while reporting 100% uptime.
The fix was converting setCookie into a defensive array accumulator:
const existing = res.getHeader('Set-Cookie');
const cookies = Array.isArray(existing) ? existing.concat(cookieStr) : existing ? [existing, cookieStr] : [cookieStr];
res.setHeader('Set-Cookie', cookies);
Verified end-to-end with Playwright headless browser automation: cookie preservation validated, full 21 dashboard rows rendered.
The hardest bugs are never the ones that crash. They are the ones where every layer quietly does exactly what you told it to do.
Discussion: What is the most insidious "100% green metrics, 100% broken flow" silent failure you’ve diagnosed in production authentication pipelines?
Full post-mortem and Playwright verification script:
https://blog.hoelee.com/posts/authentik-oidc-redirect-loop-node-set-header-overwrite/
Post 02: Why Six Chrome Containers Locked a Production Edge Storage Node
- Target Article:
six-chrome-containers-locked-my-nas-btrfs-d-state-igpu-offload - Canonical URL:
https://blog.hoelee.com/posts/six-chrome-containers-locked-my-nas-btrfs-d-state-igpu-offload/ - Category: DevOps / SRE / Storage Architecture
- Character Count: ~2,250 chars
- Argon Verdict: Approved (scrubbed hardware model to edge storage node)
At 09:42, every reverse-proxied service on a production edge storage node stopped responding. HTTPS handshakes established, SSL certificates validated, but backend requests hung indefinitely.
SSH terminal load average: 27.42 on a 4-core AMD Ryzen V1500B.
Memory: 7,410 MB used out of 7,842 MB. Swap: 3,980 MB used out of 4,096 MB.
The culprit: six containerized desktop Chrome instances running headless browser automation.
The architectural mismatch:
The Ryzen V1500B has no integrated GPU. Modern browser interfaces, canvas rendering, and video previews rely heavily on WebGL. Without a hardware GPU passed through via /dev/dri, Chromium falls back to CPU emulation using SwiftShader and llvmpipe.
What happened next was a textbook cascading kernel freeze:
1. CPU Saturation: Six browser instances software-rasterized graphics simultaneously, pegging all 4 cores at 100%.
2. Aggressive Swapping: Graphic buffer allocations exhausted physical RAM, forcing anonymous pages out to mechanical disks.
3. Storage Deadlock: When Docker attempted to prune an ephemeral container, the daemon invoked `btrfs subvolume delete`. Under saturated disk I/O, the filesystem transaction stalled in kernel D-state (uninterruptible sleep).
Because the Btrfs root transaction lock was held, every system process that touched the filesystem—PostgreSQL, proxy sockets, and internal managers—froze in the queue.
The fix: separating storage infrastructure from GPU-bound compute workloads.
I migrated the browser automation fleet to an unRaid host powered by an Intel Core i9-13900K, passing through the integrated Intel UHD Graphics 770 (/dev/dri/renderD128). I also switched container policies from restart: unless-stopped to an on-demand model (0% CPU and 0 MB RAM when idle).
Results:
- Recovered 7.5 GB of committed memory (RAM + swap) on the node.
- Host load average dropped from 27.42 to 6.80 within minutes.
- Sub-second response restored across all edge proxies.
- Browser automation execution latency decreased by over 60% on native silicon.
Discussion: How do you architect resource boundaries for headless workloads to prevent containerized browser rendering from starving storage daemons?
Full breakdown and Compose configurations:
https://blog.hoelee.com/posts/six-chrome-containers-locked-my-nas-btrfs-d-state-igpu-offload/
Post 03: The 768 MB Limit That Caused a Microservice Retry Storm
- Target Article:
cascading-docker-failures-nocodb-heap-limit-starved-n8n - Canonical URL:
https://blog.hoelee.com/posts/cascading-docker-failures-nocodb-heap-limit-starved-n8n/ - Category: Systems Integration / High Availability
- Character Count: ~2,150 chars
- Argon Verdict: Approved
The alert triggered at 04:00: dozens of business-critical n8n workflows failed with `connect ECONNREFUSED :10380`.
Telegram event webhooks, customer inquiry synchronizations, and database pipelines all stalled simultaneously. Several long-running batch jobs hung in permanent `running` state.
The target service—NocoDB on port 10380—was trapped in a crash loop.
Container logs held the fatal line:
`FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory`
When provisioned, NocoDB was given a strict container cgroup constraint: `mem_limit: 768m`.
Under a tight 768 MB cgroup boundary, Node.js V8's default heap heuristics leave insufficient headroom for spiky aggregation workloads. When scheduled cron tasks triggered multi-table joins across tens of thousands of rows, memory climbed past 700 MB. V8's garbage collector attempted aggressive mark-sweep cycles, failed to reclaim contiguous memory, and aborted.
Then came the retry storm:
When NocoDB died, n8n database nodes triggered automatic retries. Hundreds of retries hammered port 10380 just as Docker brought the container back up, instantly re-saturating the heap and triggering another OOM kill within seconds.
The fix required tuning both layers:
1. Aligned Node.js runtime heap: NODE_OPTIONS="--max-old-space-size=1536"
2. Increased Docker container envelope: 2048 MB limit with 512 MB reserved for OS buffers and native bindings.
3. Purged stalled orphaned executions in n8n's internal database to release worker locks.
Outcome: 100% workflow recovery, 0 connection drops, and stable 450 MB memory usage during peak analytical queries.
Never set a Docker memory limit on a Node.js container without explicitly tuning --max-old-space-size to leave room for native memory.
Discussion: Where do you draw the line between Docker cgroup limits and runtime internal heap tuning in production Node.js microservices?
Full post-mortem:
https://blog.hoelee.com/posts/cascading-docker-failures-nocodb-heap-limit-starved-n8n/