post(batch-08): publish 5 incident & infrastructure posts after Argon review (EN+ZH)
Deploy / build (push) Successful in 1m35s
No files matched your search
@@ -1259,5 +1259,70 @@
|
||||
"notes": "",
|
||||
"en_chars": 6896,
|
||||
"zh_chars": 4766
|
||||
},
|
||||
"authentik-oidc-redirect-loop-node-set-header-overwrite": {
|
||||
"slug": "authentik-oidc-redirect-loop-node-set-header-overwrite",
|
||||
"title": "The Authentik OIDC Redirect Loop: Why Node.js res.setHeader Overwrites Rather Than Appends",
|
||||
"zh_title": "Authentik OIDC 重定向死循环:Node.js res.setHeader 整头覆盖机制与会话丢失排障",
|
||||
"category": "engineering",
|
||||
"pub_date": "2025-02-14",
|
||||
"status": "approved",
|
||||
"batch_id": "Batch-08",
|
||||
"approved_at": "2026-11-01 14:55",
|
||||
"notes": "Batch-08 顾问审阅「通过」:修正 Bento 卡片行数与摘要对齐、中文标题去个人化、details 标注 12 行。Playwright 真实测试通过。",
|
||||
"en_chars": 6367,
|
||||
"zh_chars": 3970
|
||||
},
|
||||
"six-chrome-containers-locked-my-nas-btrfs-d-state-igpu-offload": {
|
||||
"slug": "six-chrome-containers-locked-my-nas-btrfs-d-state-igpu-offload",
|
||||
"title": "Why Six Desktop Chrome Containers Locked a Production NAS: CPU WebGL Emulation, Btrfs D-State, and Offloading to iGPU",
|
||||
"zh_title": "六个 Chrome 容器打崩存储 NAS:CPU 软解 WebGL、Btrfs D 状态死锁与核显架构迁移",
|
||||
"category": "devops",
|
||||
"pub_date": "2025-03-20",
|
||||
"status": "approved",
|
||||
"batch_id": "Batch-08",
|
||||
"approved_at": "2026-11-01 14:55",
|
||||
"notes": "Batch-08 顾问审阅「通过」:修正 7.5GB 提交内存空间、折叠 docker-compose 配置(17行)、正文对齐 unRaid 宿主、英文标题去个人化。",
|
||||
"en_chars": 6351,
|
||||
"zh_chars": 3892
|
||||
},
|
||||
"cascading-docker-failures-nocodb-heap-limit-starved-n8n": {
|
||||
"slug": "cascading-docker-failures-nocodb-heap-limit-starved-n8n",
|
||||
"title": "Cascading Docker Failures: When a Node.js Heap Limit in NocoDB Starved an n8n Automation Stack",
|
||||
"zh_title": "Docker 级联故障复盘:NocoDB 内存溢出如何拖垮整套 n8n 自动化栈",
|
||||
"category": "devops",
|
||||
"pub_date": "2025-04-12",
|
||||
"status": "approved",
|
||||
"batch_id": "Batch-08",
|
||||
"approved_at": "2026-11-01 14:55",
|
||||
"notes": "Batch-08 顾问审阅「通过」:校准 V8 堆内存与 cgroup 启发式机制描述、修正 details 为 17 行、对齐 Compose 语法。",
|
||||
"en_chars": 5885,
|
||||
"zh_chars": 3834
|
||||
},
|
||||
"apple-icloud-smtp-port-465-timeout-starttls-587-trap": {
|
||||
"slug": "apple-icloud-smtp-port-465-timeout-starttls-587-trap",
|
||||
"title": "Apple iCloud SMTP Times Out on Port 465: The 587 STARTTLS-Only Trap",
|
||||
"zh_title": "Apple iCloud 邮箱发信 465 超时:仅支持 587 STARTTLS 的协议陷阱",
|
||||
"category": "notes",
|
||||
"pub_date": "2026-06-18",
|
||||
"status": "approved",
|
||||
"batch_id": "Batch-08",
|
||||
"approved_at": "2026-11-01 14:55",
|
||||
"notes": "Batch-08 顾问审阅「通过」:明确 Windows Winsock 10060 运行环境、校准 RFC 6409/3207 与 RFC 8314 演进史、中文标签去个人化、details 标注 11 行。",
|
||||
"en_chars": 5070,
|
||||
"zh_chars": 3550
|
||||
},
|
||||
"home-assistant-container-upgrade-ladder-four-hops": {
|
||||
"slug": "home-assistant-container-upgrade-ladder-four-hops",
|
||||
"title": "Home Assistant Container: Upgrading Thirteen Releases Across a Four-Hop Ladder",
|
||||
"zh_title": "Home Assistant 容器版跨版本升级:四阶安全平跳十三期迭代",
|
||||
"category": "devops",
|
||||
"pub_date": "2026-07-22",
|
||||
"status": "approved",
|
||||
"batch_id": "Batch-08",
|
||||
"approved_at": "2026-11-01 14:55",
|
||||
"notes": "Batch-08 顾问审阅「通过」:将智能家居升维为机房物联网动环监控、平滑 2026.8 反代表述、修正 details 为 14 行。",
|
||||
"en_chars": 5593,
|
||||
"zh_chars": 3422
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
eval 'ps -ef | grep gemini_cdp
|
||||
eval 'ps -ef | grep gemini_cdp
|
||||
Binary file (standard input) matches
|
||||
|
After Width: | Height: | Size: 89 KiB |
|
After Width: | Height: | Size: 65 KiB |
|
After Width: | Height: | Size: 75 KiB |
|
After Width: | Height: | Size: 33 KiB |
|
After Width: | Height: | Size: 54 KiB |
|
After Width: | Height: | Size: 45 KiB |
|
After Width: | Height: | Size: 70 KiB |
|
After Width: | Height: | Size: 50 KiB |
|
After Width: | Height: | Size: 89 KiB |
|
After Width: | Height: | Size: 73 KiB |
|
After Width: | Height: | Size: 67 KiB |
|
After Width: | Height: | Size: 74 KiB |
|
After Width: | Height: | Size: 74 KiB |
|
After Width: | Height: | Size: 73 KiB |
|
After Width: | Height: | Size: 79 KiB |
|
After Width: | Height: | Size: 8.6 KiB |
|
After Width: | Height: | Size: 22 KiB |
|
After Width: | Height: | Size: 4.6 KiB |
|
After Width: | Height: | Size: 11 KiB |
|
After Width: | Height: | Size: 6.5 KiB |
|
After Width: | Height: | Size: 18 KiB |
|
After Width: | Height: | Size: 7.5 KiB |
|
After Width: | Height: | Size: 18 KiB |
|
After Width: | Height: | Size: 10 KiB |
|
After Width: | Height: | Size: 28 KiB |
|
After Width: | Height: | Size: 69 KiB |
|
After Width: | Height: | Size: 81 KiB |
|
After Width: | Height: | Size: 79 KiB |
|
After Width: | Height: | Size: 76 KiB |
|
After Width: | Height: | Size: 85 KiB |
@@ -1,338 +1,353 @@
|
||||
{
|
||||
"_about": "Inputs for scripts/og-gen/render_from_art.mjs ('one art, four targets'). One entry per post: `art` = the zero-text base frame (default <art-dir>/<slug>.png), `evidence` = the short factual line shown in the card's evidence bar (REQUIRED, written by hand — never derived), optional `ogTitle_en` / `ogTitle_zh` = shortened display titles for the card only (consultant question 1, option C: the article keeps its real title), optional `tags` = override the three badge tags. Cards render from the post's own frontmatter (title/description/category) unless an ogTitle override is present.",
|
||||
"_status": "73 of 73 posts have a hand-written evidence line; base art is generated from C:/AI/ideogram4/captions/<slug>.json",
|
||||
"running-ai-video-locally-on-a-12gb-gpu": {
|
||||
"evidence": "1280x736 -> 1080p - 32 min / 5 s - $0",
|
||||
"ogTitle_en": "AI Video on a 12 GB GPU",
|
||||
"ogTitle_zh": "12GB 显卡跑 1080p 视频"
|
||||
},
|
||||
"adding-a-domain-to-someone-elses-mail-server": {
|
||||
"evidence": "4 DNS records · 554 antispam reject"
|
||||
},
|
||||
"adding-english-mode-to-a-chinese-only-web-app": {
|
||||
"evidence": "no browser API · sub_filter EN mode"
|
||||
},
|
||||
"ai-furniture-compositing-with-flux-kontext": {
|
||||
"evidence": "kontext multi-image · tainted canvas fix"
|
||||
},
|
||||
"authentik-css-greater-than-bug": {
|
||||
"evidence": "1 escaped '>' · CSS silently dropped"
|
||||
},
|
||||
"authentik-forward-auth-gate-wasnt-live": {
|
||||
"evidence": "host 302 · public 200 · 2 ingress layers"
|
||||
},
|
||||
"authentik-major-upgrade-gotchas": {
|
||||
"evidence": "2025.8 → 2026.8 · auth_flow mixup"
|
||||
},
|
||||
"automating-cyberpanel-without-the-ui": {
|
||||
"evidence": "v2 JSON API gone · session + CSRF"
|
||||
},
|
||||
"best-ai-video-generators-2026": {
|
||||
"evidence": "free vs paid · local LTX-2.3 test"
|
||||
},
|
||||
"calibrating-intel-quick-sync-hevc-quality": {
|
||||
"evidence": "ICQ 28-30 cliff · fast == medium"
|
||||
},
|
||||
"chainlink-vrf-v2-lottery-contract": {
|
||||
"evidence": "VRF v2.5 · mock-funding bug"
|
||||
},
|
||||
"docker-reported-exit-0-the-oom-killer-disagreed": {
|
||||
"evidence": "exit 0 · dmesg: 6x cgroup OOM kills"
|
||||
},
|
||||
"does-hevc-actually-shrink-your-files": {
|
||||
"evidence": "NVENC cq27 bigger · x265 -10-22%"
|
||||
},
|
||||
"embedding-fonts-so-a-pdf-looks-the-same-everywhere": {
|
||||
"evidence": "4 families · 8 faces · 2 renderers"
|
||||
},
|
||||
"every-player-stuttered-the-file-was-fine": {
|
||||
"evidence": "5818/5818 intervals · 0 duplicate PTS"
|
||||
},
|
||||
"five-step-ai-roadmap-what-i-actually-run": {
|
||||
"evidence": "whisper + mem0 in production · step 4 skipped"
|
||||
},
|
||||
"fully-on-chain-svg-nfts": {
|
||||
"evidence": "base64 SVG in tokenURI() · mood flip on-chain"
|
||||
},
|
||||
"hardening-a-tor-onion-service": {
|
||||
"evidence": "bind 0.0.0.0:80 denied · onion v3"
|
||||
},
|
||||
"hello-world": {
|
||||
"evidence": "~140-container homelab · self-hosted"
|
||||
},
|
||||
"how-i-built-the-digikedai-telegram-bot": {
|
||||
"evidence": "Telegram + WhatsApp · LiteLLM gateway"
|
||||
},
|
||||
"how-i-host-this-blog": {
|
||||
"evidence": "git push · act_runner · nginx"
|
||||
},
|
||||
"how-i-made-my-own-songs-with-suno-ai": {
|
||||
"evidence": "7 songs · lyrics + prompts"
|
||||
},
|
||||
"how-i-vetted-20-vps-providers-with-parallel-subagents": {
|
||||
"evidence": "20 providers · 3 hours · subagents"
|
||||
},
|
||||
"how-to-verify-a-hosting-provider-before-you-buy": {
|
||||
"evidence": "rdap.org domain age · AUP read"
|
||||
},
|
||||
"i-let-an-agent-manage-my-shopee-vouchers": {
|
||||
"evidence": "mode=propose gate · RM140 voucher unapproved"
|
||||
},
|
||||
"it-reported-success-nothing-had-changed": {
|
||||
"evidence": "12.5 MB of 403 MB · rename never landed"
|
||||
},
|
||||
"jpa-version-field-lost-update": {
|
||||
"evidence": "@Version · stale PUT → 409"
|
||||
},
|
||||
"loop-engineering-without-a-coding-agent": {
|
||||
"evidence": "11 cron jobs · 8 zero-AI loops"
|
||||
},
|
||||
"migrating-codeigniter-iis-to-openlitespeed": {
|
||||
"evidence": "17,000-line CI4 · env() fatal"
|
||||
},
|
||||
"my-backup-was-using-someone-elses-api-quota": {
|
||||
"evidence": "1 TB restic seed · shared rclone quota"
|
||||
},
|
||||
"n8n-v1-to-v2-upgrade-gotchas": {
|
||||
"evidence": "1.x → 2.40.1 · 7 silent breaks"
|
||||
},
|
||||
"nocodb-sso-is-a-licensed-feature": {
|
||||
"evidence": "OIDC env accepted · first login crashes"
|
||||
},
|
||||
"one-hostname-public-tracker-sso-dashboard": {
|
||||
"evidence": "1 host · tracker public, dashboard SSO"
|
||||
},
|
||||
"one-prometheus-for-unraid-synology-and-a-vps": {
|
||||
"evidence": "3 hosts · 1 prometheus · 0 false panels"
|
||||
},
|
||||
"passbolt-hang-three-failure-modes": {
|
||||
"evidence": "null fingerprint · 3 failure modes"
|
||||
},
|
||||
"patching-workbench-26-for-mariadb": {
|
||||
"evidence": "26.7.0 TypeError · 3 patches"
|
||||
},
|
||||
"putting-a-front-door-on-an-app-you-cant-modify": {
|
||||
"evidence": "5879-byte SPA · 3 injected lines"
|
||||
},
|
||||
"read-only-nocodb-dashboard-for-a-remote-database": {
|
||||
"evidence": "SNAT source IP · 0 rows copied"
|
||||
},
|
||||
"reading-a-containers-own-api-docs": {
|
||||
"evidence": "4 docker commands · image is the spec"
|
||||
},
|
||||
"replacing-rdpguard-with-ipban": {
|
||||
"evidence": "RDPGuard 7.8.7 → IPBan 4.1.0 · 12 IPs"
|
||||
},
|
||||
"running-production-infrastructure-solo": {
|
||||
"evidence": "162 containers · 3 hosts · 78 compose stacks"
|
||||
},
|
||||
"running-tts-as-a-service-with-token-sidecars": {
|
||||
"evidence": "tokens expire 10 min / 1 h · sidecars"
|
||||
},
|
||||
"scraping-bot-walled-marketplace-warm-browser-session": {
|
||||
"evidence": "warm CDP session · split waits"
|
||||
},
|
||||
"self-healing-digital-goods-entitlements": {
|
||||
"evidence": "5 workflows · NocoDB + n8n + AList"
|
||||
},
|
||||
"self-hosted-speech-to-text-api": {
|
||||
"evidence": "whisper.cpp large-v3 · ~5x typing"
|
||||
},
|
||||
"self-hosting-mem0-memory-stack": {
|
||||
"evidence": "mem0 + LiteLLM + pgvector · :20015"
|
||||
},
|
||||
"shipping-an-ai-photo-editor-as-a-wordpress-plugin": {
|
||||
"evidence": "22 versions · fal.ai async queue"
|
||||
},
|
||||
"smartctl-exit-code-32-skips-the-disks-that-matter": {
|
||||
"evidence": "rc=32 is not failure · skips bad disks"
|
||||
},
|
||||
"space-reclamation-wasnt-stuck-it-was-waiting": {
|
||||
"evidence": "121 GiB returned · scheduled 09:29"
|
||||
},
|
||||
"stop-writing-agent-prompts-for-deterministic-work": {
|
||||
"evidence": "29 straight failures · 1 string compare"
|
||||
},
|
||||
"syncing-a-self-improving-ai-agent-across-machines": {
|
||||
"evidence": "git-backed profile · manual sync"
|
||||
},
|
||||
"synology-api-401-with-the-correct-password": {
|
||||
"evidence": "401 twice · 2 causes · 3 commands"
|
||||
},
|
||||
"synology-spreadsheet-api-is-a-container": {
|
||||
"evidence": "1515 APIs · 4 traps"
|
||||
},
|
||||
"that-dying-ssd-was-just-a-bad-sata-cable": {
|
||||
"evidence": "WRITE FPDMA timeout · 1 SATA cable"
|
||||
},
|
||||
"the-cause-was-trim-not-the-ssds": {
|
||||
"evidence": "corruption_errs 27 / 31 · queued TRIM"
|
||||
},
|
||||
"the-nocodb-attachment-that-wouldnt-update": {
|
||||
"evidence": "300 rows · 1 lookahead regex miss"
|
||||
},
|
||||
"the-upstream-was-deleted-then-came-back-rewritten": {
|
||||
"evidence": "repo 404 · 233 commits rewrite"
|
||||
},
|
||||
"two-ways-a-javascript-file-stops-working-and-looks-fine": {
|
||||
"evidence": "shadowed name · let below call"
|
||||
},
|
||||
"unraid-stop-array-hangs-on-swapfile": {
|
||||
"evidence": "8 GB swapfile pins /mnt/cache"
|
||||
},
|
||||
"upgrading-codeigniter-46-to-47": {
|
||||
"evidence": "4.6 → 4.7 · 2 unlisted fatals"
|
||||
},
|
||||
"using-chinese-llm-apis-from-malaysia": {
|
||||
"evidence": "GLM-5.3 · DeepSeek V4 · RMB top-up"
|
||||
},
|
||||
"verifying-a-pdf-report-page-by-page": {
|
||||
"evidence": "19 pages · every deviation < 2 pt"
|
||||
},
|
||||
"vetting-an-open-source-dependency-before-you-bet-on-it": {
|
||||
"evidence": "6 checks · grep beats the README"
|
||||
},
|
||||
"when-smart-says-healthy-but-your-raid-is-corrupting-data": {
|
||||
"evidence": "SMART healthy · checksum 0x8941f998"
|
||||
},
|
||||
"why-500000-lines-of-commit-churn-only-left-230000-in-production": {
|
||||
"evidence": "518,000 added · 235,000 kept · 5 repos"
|
||||
},
|
||||
"why-chrome-forgets-its-tabs-in-a-container": {
|
||||
"evidence": "RestoreOnStartup=1 · 70-line keeper"
|
||||
},
|
||||
"why-i-still-bought-a-local-gpu": {
|
||||
"evidence": "stronger frontier · local for retries + batch"
|
||||
},
|
||||
"why-my-on-chain-nft-art-changed-on-windows": {
|
||||
"evidence": "autocrlf CRLF · base64 differs"
|
||||
},
|
||||
"why-telegram-bot-notifications-die": {
|
||||
"evidence": "400 on sendPhoto · getMe fine · IPv6-only DNS"
|
||||
},
|
||||
"why-your-cdp-clicks-silently-fail": {
|
||||
"evidence": "scroll-behavior:smooth · rect expired"
|
||||
},
|
||||
"why-your-grafana-dashboard-shows-no-data": {
|
||||
"evidence": "41 panels No data · variable filtered itself"
|
||||
},
|
||||
"why-your-headless-browser-cant-scrape-everything": {
|
||||
"evidence": "Goofish blocks · Carousell serves"
|
||||
},
|
||||
"your-disk-full-alert-is-lying": {
|
||||
"evidence": "3% free = 500 GB · alert fires wrong way"
|
||||
},
|
||||
"yue2-on-a-12gb-gpu-songs-that-lost-their-endings": {
|
||||
"evidence": "12 takes - all 240.000 s - cap now 300 s"
|
||||
},
|
||||
"generating-subtitles-for-songs-without-timestamps": {
|
||||
"evidence": "18 songs - 342 lines - 0.96 anchor coverage"
|
||||
},
|
||||
"why-i-built-a-video-knowledge-base": {
|
||||
"evidence": "263 pages · 20ms search · 0 cloud STT fees"
|
||||
},
|
||||
"how-i-built-an-automated-music-video-pipeline": {
|
||||
"evidence": "LRC + Remotion 1080p · CDP upload · 8-lang subs published",
|
||||
"ogTitle_en": "Automated MV Pipeline with Remotion & CDP",
|
||||
"ogTitle_zh": "Remotion 与 Docker CDP 自动生成发布 MV"
|
||||
},
|
||||
"uploading-large-videos-to-youtube-via-cdp": {
|
||||
"evidence": "130MB video via Portainer archive PUT · 0 RAM spikes",
|
||||
"ogTitle_en": "Upload 100MB+ Videos to YouTube via CDP",
|
||||
"ogTitle_zh": "CDP 上传百兆视频:Docker 归档法"
|
||||
},
|
||||
"debugging-remotion-video-jitter": {
|
||||
"evidence": "bassScale 30fps tremor killed · 2-layer opacity crossfade",
|
||||
"ogTitle_en": "Debugging Remotion Video Jitter",
|
||||
"ogTitle_zh": "排查 Remotion 视频抽搐与透明度叠化"
|
||||
},
|
||||
"self-hosted-tts-cosyvoice2-n8n-gpu": {
|
||||
"evidence": "3.2GB VRAM · 300ms first-packet · 0 cloud API fees",
|
||||
"ogTitle_en": "Self-Hosted Neural TTS with CosyVoice 2 & N8N",
|
||||
"ogTitle_zh": "自建 CosyVoice 2 高保真语音栈与 N8N 联动"
|
||||
},
|
||||
"why-is-my-tts-model-repeating-percent": {
|
||||
"evidence": "URL percent-encoding trap killed · 2-layer defensive fix",
|
||||
"ogTitle_en": "Why Your TTS Model Keeps Saying \"Percent\"",
|
||||
"ogTitle_zh": "为什么本地语音合成一直在念 Percent?"
|
||||
},
|
||||
"php2shell-patching-every-php-runtime-i-manage": {
|
||||
"evidence": "8 public sites in 2 min · 7 on 8.5.12 · 8 dead rules"
|
||||
},
|
||||
"giving-ai-your-mouse-and-keyboard-safely": {
|
||||
"evidence": "PowerShell & Registry stripped · 0.5x downscale · 0 flash",
|
||||
"ogTitle_en": "Safe Guardrails for Windows Desktop MCP",
|
||||
"ogTitle_zh": "给 AI 鼠标键盘前的四道安全防线"
|
||||
},
|
||||
"plugging-windows-desktop-control-into-four-ai-agents": {
|
||||
"evidence": "1 Python 3.14 server · 4 agent configs · 19 native tools",
|
||||
"ogTitle_en": "Plugging Windows Desktop Control into 4 Agents",
|
||||
"ogTitle_zh": "把 Windows 桌面控制接进 4 款 AI Agent"
|
||||
},
|
||||
"why-your-windows-mcp-calls-silently-fail": {
|
||||
"evidence": "CONTROL_PREEMPTED killed · 10s idle gate bypassed",
|
||||
"ogTitle_en": "Why Windows MCP Calls Silently Fail",
|
||||
"ogTitle_zh": "攻破 Windows 桌面 MCP 10 秒空闲门禁"
|
||||
},
|
||||
"when-to-give-your-agent-physical-clicks": {
|
||||
"evidence": "API > CLI > CUA > Physical clicks · 4-tier matrix",
|
||||
"ogTitle_en": "When to Give Your Agent Physical Clicks",
|
||||
"ogTitle_zh": "什么时候该让 AI 操控物理鼠标"
|
||||
},
|
||||
"the-single-writer-rule-for-coding-agents": {
|
||||
"evidence": "1 write process · 0 merge collisions · isolated review",
|
||||
"ogTitle_en": "The Single-Writer Rule for AI Agents",
|
||||
"ogTitle_zh": "代码只让一个 AI 改:单写多审架构"
|
||||
},
|
||||
"slashing-agent-context-with-a-code-graph": {
|
||||
"evidence": "AST graph indexing · 80% token churn cut · 0 hallucination",
|
||||
"ogTitle_en": "Slashing Context With Code Graph",
|
||||
"ogTitle_zh": "用 Code Graph 斩断 Token 膨胀与遗忘"
|
||||
},
|
||||
"why-multi-agent-coding-fails": {
|
||||
"evidence": "Telephone game distortion · implicit architectural drift",
|
||||
"ogTitle_en": "Why Multi-Agent Coding Fails",
|
||||
"ogTitle_zh": "多 AI 写代码烂尾与隐式决策困境"
|
||||
},
|
||||
"generating-full-length-songs-with-yue2-comfyui": {
|
||||
"evidence": "60 songs - 9-part structure - 295s headroom cap"
|
||||
},
|
||||
"when-lowering-crf-does-nothing-maxrate-deadlock": {
|
||||
"evidence": "CRF 19→14 · SSIM +0.00003 · 106.2h wasted",
|
||||
"ogTitle_en": "When Lowering CRF Does Nothing",
|
||||
"ogTitle_zh": "降低 CRF 为何毫无效果:maxrate 死锁"
|
||||
},
|
||||
"why-ffmpeg-map-0-breaks-your-mp4-and-mkv-remux": {
|
||||
"evidence": "-map 0 copy · tx3g CC · udta/name dual-write",
|
||||
"ogTitle_en": "Why ffmpeg -map 0 Breaks Remux",
|
||||
"ogTitle_zh": "ffmpeg -map 0 搞崩跨容器混流"
|
||||
},
|
||||
"traefik-yaegi-plugin-package-name-trap": {
|
||||
"evidence": "Yaegi lookup · hyphen to underscore · panic",
|
||||
"ogTitle_en": "Traefik Yaegi Plugin Package Trap",
|
||||
"ogTitle_zh": "Traefik Yaegi 插件启动崩溃陷阱"
|
||||
},
|
||||
"architecting-sea-payment-and-offline-licensing-pipelines": {
|
||||
"evidence": "toyyibPay FPX · FOR UPDATE · Ed25519 offline",
|
||||
"ogTitle_en": "SEA Payment & Offline Licensing",
|
||||
"ogTitle_zh": "东南亚网银支付与离线机器码授权"
|
||||
},
|
||||
"batch-generating-80-technical-covers-with-comfyui": {
|
||||
"evidence": "89 covers in 48m · 35% safe zone · Laplacian filter",
|
||||
"ogTitle_en": "Batch Generating 80+ Technical Covers",
|
||||
"ogTitle_zh": "ComfyUI 批量闭环 80+ 张技术封面"
|
||||
},
|
||||
"when-a-proxy-timeout-returns-html-instead-of-json": {
|
||||
"evidence": "13.6 s -> 0.26 s · 2507 s worst · 499 at the gate",
|
||||
"ogTitle_en": "The Proxy That Replied in HTML",
|
||||
"ogTitle_zh": "反代用 HTML 回答了 JSON",
|
||||
"tags": [
|
||||
"nginx",
|
||||
"timeouts",
|
||||
"caching"
|
||||
]
|
||||
},
|
||||
"read-amplification-504-gateway-timeout-swr-cache": {
|
||||
"evidence": "4 reads/file · 960 per page · 0.3 µs cache hit"
|
||||
}
|
||||
"_about": "Inputs for scripts/og-gen/render_from_art.mjs ('one art, four targets'). One entry per post: `art` = the zero-text base frame (default <art-dir>/<slug>.png), `evidence` = the short factual line shown in the card's evidence bar (REQUIRED, written by hand — never derived), optional `ogTitle_en` / `ogTitle_zh` = shortened display titles for the card only (consultant question 1, option C: the article keeps its real title), optional `tags` = override the three badge tags. Cards render from the post's own frontmatter (title/description/category) unless an ogTitle override is present.",
|
||||
"_status": "73 of 73 posts have a hand-written evidence line; base art is generated from C:/AI/ideogram4/captions/<slug>.json",
|
||||
"running-ai-video-locally-on-a-12gb-gpu": {
|
||||
"evidence": "1280x736 -> 1080p - 32 min / 5 s - $0",
|
||||
"ogTitle_en": "AI Video on a 12 GB GPU",
|
||||
"ogTitle_zh": "12GB 显卡跑 1080p 视频"
|
||||
},
|
||||
"adding-a-domain-to-someone-elses-mail-server": {
|
||||
"evidence": "4 DNS records · 554 antispam reject"
|
||||
},
|
||||
"adding-english-mode-to-a-chinese-only-web-app": {
|
||||
"evidence": "no browser API · sub_filter EN mode"
|
||||
},
|
||||
"ai-furniture-compositing-with-flux-kontext": {
|
||||
"evidence": "kontext multi-image · tainted canvas fix"
|
||||
},
|
||||
"authentik-css-greater-than-bug": {
|
||||
"evidence": "1 escaped '>' · CSS silently dropped"
|
||||
},
|
||||
"authentik-forward-auth-gate-wasnt-live": {
|
||||
"evidence": "host 302 · public 200 · 2 ingress layers"
|
||||
},
|
||||
"authentik-major-upgrade-gotchas": {
|
||||
"evidence": "2025.8 → 2026.8 · auth_flow mixup"
|
||||
},
|
||||
"automating-cyberpanel-without-the-ui": {
|
||||
"evidence": "v2 JSON API gone · session + CSRF"
|
||||
},
|
||||
"best-ai-video-generators-2026": {
|
||||
"evidence": "free vs paid · local LTX-2.3 test"
|
||||
},
|
||||
"calibrating-intel-quick-sync-hevc-quality": {
|
||||
"evidence": "ICQ 28-30 cliff · fast == medium"
|
||||
},
|
||||
"chainlink-vrf-v2-lottery-contract": {
|
||||
"evidence": "VRF v2.5 · mock-funding bug"
|
||||
},
|
||||
"docker-reported-exit-0-the-oom-killer-disagreed": {
|
||||
"evidence": "exit 0 · dmesg: 6x cgroup OOM kills"
|
||||
},
|
||||
"does-hevc-actually-shrink-your-files": {
|
||||
"evidence": "NVENC cq27 bigger · x265 -10-22%"
|
||||
},
|
||||
"embedding-fonts-so-a-pdf-looks-the-same-everywhere": {
|
||||
"evidence": "4 families · 8 faces · 2 renderers"
|
||||
},
|
||||
"every-player-stuttered-the-file-was-fine": {
|
||||
"evidence": "5818/5818 intervals · 0 duplicate PTS"
|
||||
},
|
||||
"five-step-ai-roadmap-what-i-actually-run": {
|
||||
"evidence": "whisper + mem0 in production · step 4 skipped"
|
||||
},
|
||||
"fully-on-chain-svg-nfts": {
|
||||
"evidence": "base64 SVG in tokenURI() · mood flip on-chain"
|
||||
},
|
||||
"hardening-a-tor-onion-service": {
|
||||
"evidence": "bind 0.0.0.0:80 denied · onion v3"
|
||||
},
|
||||
"hello-world": {
|
||||
"evidence": "~140-container homelab · self-hosted"
|
||||
},
|
||||
"how-i-built-the-digikedai-telegram-bot": {
|
||||
"evidence": "Telegram + WhatsApp · LiteLLM gateway"
|
||||
},
|
||||
"how-i-host-this-blog": {
|
||||
"evidence": "git push · act_runner · nginx"
|
||||
},
|
||||
"how-i-made-my-own-songs-with-suno-ai": {
|
||||
"evidence": "7 songs · lyrics + prompts"
|
||||
},
|
||||
"how-i-vetted-20-vps-providers-with-parallel-subagents": {
|
||||
"evidence": "20 providers · 3 hours · subagents"
|
||||
},
|
||||
"how-to-verify-a-hosting-provider-before-you-buy": {
|
||||
"evidence": "rdap.org domain age · AUP read"
|
||||
},
|
||||
"i-let-an-agent-manage-my-shopee-vouchers": {
|
||||
"evidence": "mode=propose gate · RM140 voucher unapproved"
|
||||
},
|
||||
"it-reported-success-nothing-had-changed": {
|
||||
"evidence": "12.5 MB of 403 MB · rename never landed"
|
||||
},
|
||||
"jpa-version-field-lost-update": {
|
||||
"evidence": "@Version · stale PUT → 409"
|
||||
},
|
||||
"loop-engineering-without-a-coding-agent": {
|
||||
"evidence": "11 cron jobs · 8 zero-AI loops"
|
||||
},
|
||||
"migrating-codeigniter-iis-to-openlitespeed": {
|
||||
"evidence": "17,000-line CI4 · env() fatal"
|
||||
},
|
||||
"my-backup-was-using-someone-elses-api-quota": {
|
||||
"evidence": "1 TB restic seed · shared rclone quota"
|
||||
},
|
||||
"n8n-v1-to-v2-upgrade-gotchas": {
|
||||
"evidence": "1.x → 2.40.1 · 7 silent breaks"
|
||||
},
|
||||
"nocodb-sso-is-a-licensed-feature": {
|
||||
"evidence": "OIDC env accepted · first login crashes"
|
||||
},
|
||||
"one-hostname-public-tracker-sso-dashboard": {
|
||||
"evidence": "1 host · tracker public, dashboard SSO"
|
||||
},
|
||||
"one-prometheus-for-unraid-synology-and-a-vps": {
|
||||
"evidence": "3 hosts · 1 prometheus · 0 false panels"
|
||||
},
|
||||
"passbolt-hang-three-failure-modes": {
|
||||
"evidence": "null fingerprint · 3 failure modes"
|
||||
},
|
||||
"patching-workbench-26-for-mariadb": {
|
||||
"evidence": "26.7.0 TypeError · 3 patches"
|
||||
},
|
||||
"putting-a-front-door-on-an-app-you-cant-modify": {
|
||||
"evidence": "5879-byte SPA · 3 injected lines"
|
||||
},
|
||||
"read-only-nocodb-dashboard-for-a-remote-database": {
|
||||
"evidence": "SNAT source IP · 0 rows copied"
|
||||
},
|
||||
"reading-a-containers-own-api-docs": {
|
||||
"evidence": "4 docker commands · image is the spec"
|
||||
},
|
||||
"replacing-rdpguard-with-ipban": {
|
||||
"evidence": "RDPGuard 7.8.7 → IPBan 4.1.0 · 12 IPs"
|
||||
},
|
||||
"running-production-infrastructure-solo": {
|
||||
"evidence": "162 containers · 3 hosts · 78 compose stacks"
|
||||
},
|
||||
"running-tts-as-a-service-with-token-sidecars": {
|
||||
"evidence": "tokens expire 10 min / 1 h · sidecars"
|
||||
},
|
||||
"scraping-bot-walled-marketplace-warm-browser-session": {
|
||||
"evidence": "warm CDP session · split waits"
|
||||
},
|
||||
"self-healing-digital-goods-entitlements": {
|
||||
"evidence": "5 workflows · NocoDB + n8n + AList"
|
||||
},
|
||||
"self-hosted-speech-to-text-api": {
|
||||
"evidence": "whisper.cpp large-v3 · ~5x typing"
|
||||
},
|
||||
"self-hosting-mem0-memory-stack": {
|
||||
"evidence": "mem0 + LiteLLM + pgvector · :20015"
|
||||
},
|
||||
"shipping-an-ai-photo-editor-as-a-wordpress-plugin": {
|
||||
"evidence": "22 versions · fal.ai async queue"
|
||||
},
|
||||
"smartctl-exit-code-32-skips-the-disks-that-matter": {
|
||||
"evidence": "rc=32 is not failure · skips bad disks"
|
||||
},
|
||||
"space-reclamation-wasnt-stuck-it-was-waiting": {
|
||||
"evidence": "121 GiB returned · scheduled 09:29"
|
||||
},
|
||||
"stop-writing-agent-prompts-for-deterministic-work": {
|
||||
"evidence": "29 straight failures · 1 string compare"
|
||||
},
|
||||
"syncing-a-self-improving-ai-agent-across-machines": {
|
||||
"evidence": "git-backed profile · manual sync"
|
||||
},
|
||||
"synology-api-401-with-the-correct-password": {
|
||||
"evidence": "401 twice · 2 causes · 3 commands"
|
||||
},
|
||||
"synology-spreadsheet-api-is-a-container": {
|
||||
"evidence": "1515 APIs · 4 traps"
|
||||
},
|
||||
"that-dying-ssd-was-just-a-bad-sata-cable": {
|
||||
"evidence": "WRITE FPDMA timeout · 1 SATA cable"
|
||||
},
|
||||
"the-cause-was-trim-not-the-ssds": {
|
||||
"evidence": "corruption_errs 27 / 31 · queued TRIM"
|
||||
},
|
||||
"the-nocodb-attachment-that-wouldnt-update": {
|
||||
"evidence": "300 rows · 1 lookahead regex miss"
|
||||
},
|
||||
"the-upstream-was-deleted-then-came-back-rewritten": {
|
||||
"evidence": "repo 404 · 233 commits rewrite"
|
||||
},
|
||||
"two-ways-a-javascript-file-stops-working-and-looks-fine": {
|
||||
"evidence": "shadowed name · let below call"
|
||||
},
|
||||
"unraid-stop-array-hangs-on-swapfile": {
|
||||
"evidence": "8 GB swapfile pins /mnt/cache"
|
||||
},
|
||||
"upgrading-codeigniter-46-to-47": {
|
||||
"evidence": "4.6 → 4.7 · 2 unlisted fatals"
|
||||
},
|
||||
"using-chinese-llm-apis-from-malaysia": {
|
||||
"evidence": "GLM-5.3 · DeepSeek V4 · RMB top-up"
|
||||
},
|
||||
"verifying-a-pdf-report-page-by-page": {
|
||||
"evidence": "19 pages · every deviation < 2 pt"
|
||||
},
|
||||
"vetting-an-open-source-dependency-before-you-bet-on-it": {
|
||||
"evidence": "6 checks · grep beats the README"
|
||||
},
|
||||
"when-smart-says-healthy-but-your-raid-is-corrupting-data": {
|
||||
"evidence": "SMART healthy · checksum 0x8941f998"
|
||||
},
|
||||
"why-500000-lines-of-commit-churn-only-left-230000-in-production": {
|
||||
"evidence": "518,000 added · 235,000 kept · 5 repos"
|
||||
},
|
||||
"why-chrome-forgets-its-tabs-in-a-container": {
|
||||
"evidence": "RestoreOnStartup=1 · 70-line keeper"
|
||||
},
|
||||
"why-i-still-bought-a-local-gpu": {
|
||||
"evidence": "stronger frontier · local for retries + batch"
|
||||
},
|
||||
"why-my-on-chain-nft-art-changed-on-windows": {
|
||||
"evidence": "autocrlf CRLF · base64 differs"
|
||||
},
|
||||
"why-telegram-bot-notifications-die": {
|
||||
"evidence": "400 on sendPhoto · getMe fine · IPv6-only DNS"
|
||||
},
|
||||
"why-your-cdp-clicks-silently-fail": {
|
||||
"evidence": "scroll-behavior:smooth · rect expired"
|
||||
},
|
||||
"why-your-grafana-dashboard-shows-no-data": {
|
||||
"evidence": "41 panels No data · variable filtered itself"
|
||||
},
|
||||
"why-your-headless-browser-cant-scrape-everything": {
|
||||
"evidence": "Goofish blocks · Carousell serves"
|
||||
},
|
||||
"your-disk-full-alert-is-lying": {
|
||||
"evidence": "3% free = 500 GB · alert fires wrong way"
|
||||
},
|
||||
"yue2-on-a-12gb-gpu-songs-that-lost-their-endings": {
|
||||
"evidence": "12 takes - all 240.000 s - cap now 300 s"
|
||||
},
|
||||
"generating-subtitles-for-songs-without-timestamps": {
|
||||
"evidence": "18 songs - 342 lines - 0.96 anchor coverage"
|
||||
},
|
||||
"why-i-built-a-video-knowledge-base": {
|
||||
"evidence": "263 pages · 20ms search · 0 cloud STT fees"
|
||||
},
|
||||
"how-i-built-an-automated-music-video-pipeline": {
|
||||
"evidence": "LRC + Remotion 1080p · CDP upload · 8-lang subs published",
|
||||
"ogTitle_en": "Automated MV Pipeline with Remotion & CDP",
|
||||
"ogTitle_zh": "Remotion 与 Docker CDP 自动生成发布 MV"
|
||||
},
|
||||
"uploading-large-videos-to-youtube-via-cdp": {
|
||||
"evidence": "130MB video via Portainer archive PUT · 0 RAM spikes",
|
||||
"ogTitle_en": "Upload 100MB+ Videos to YouTube via CDP",
|
||||
"ogTitle_zh": "CDP 上传百兆视频:Docker 归档法"
|
||||
},
|
||||
"debugging-remotion-video-jitter": {
|
||||
"evidence": "bassScale 30fps tremor killed · 2-layer opacity crossfade",
|
||||
"ogTitle_en": "Debugging Remotion Video Jitter",
|
||||
"ogTitle_zh": "排查 Remotion 视频抽搐与透明度叠化"
|
||||
},
|
||||
"self-hosted-tts-cosyvoice2-n8n-gpu": {
|
||||
"evidence": "3.2GB VRAM · 300ms first-packet · 0 cloud API fees",
|
||||
"ogTitle_en": "Self-Hosted Neural TTS with CosyVoice 2 & N8N",
|
||||
"ogTitle_zh": "自建 CosyVoice 2 高保真语音栈与 N8N 联动"
|
||||
},
|
||||
"why-is-my-tts-model-repeating-percent": {
|
||||
"evidence": "URL percent-encoding trap killed · 2-layer defensive fix",
|
||||
"ogTitle_en": "Why Your TTS Model Keeps Saying \"Percent\"",
|
||||
"ogTitle_zh": "为什么本地语音合成一直在念 Percent?"
|
||||
},
|
||||
"php2shell-patching-every-php-runtime-i-manage": {
|
||||
"evidence": "8 public sites in 2 min · 7 on 8.5.12 · 8 dead rules"
|
||||
},
|
||||
"giving-ai-your-mouse-and-keyboard-safely": {
|
||||
"evidence": "PowerShell & Registry stripped · 0.5x downscale · 0 flash",
|
||||
"ogTitle_en": "Safe Guardrails for Windows Desktop MCP",
|
||||
"ogTitle_zh": "给 AI 鼠标键盘前的四道安全防线"
|
||||
},
|
||||
"plugging-windows-desktop-control-into-four-ai-agents": {
|
||||
"evidence": "1 Python 3.14 server · 4 agent configs · 19 native tools",
|
||||
"ogTitle_en": "Plugging Windows Desktop Control into 4 Agents",
|
||||
"ogTitle_zh": "把 Windows 桌面控制接进 4 款 AI Agent"
|
||||
},
|
||||
"why-your-windows-mcp-calls-silently-fail": {
|
||||
"evidence": "CONTROL_PREEMPTED killed · 10s idle gate bypassed",
|
||||
"ogTitle_en": "Why Windows MCP Calls Silently Fail",
|
||||
"ogTitle_zh": "攻破 Windows 桌面 MCP 10 秒空闲门禁"
|
||||
},
|
||||
"when-to-give-your-agent-physical-clicks": {
|
||||
"evidence": "API > CLI > CUA > Physical clicks · 4-tier matrix",
|
||||
"ogTitle_en": "When to Give Your Agent Physical Clicks",
|
||||
"ogTitle_zh": "什么时候该让 AI 操控物理鼠标"
|
||||
},
|
||||
"the-single-writer-rule-for-coding-agents": {
|
||||
"evidence": "1 write process · 0 merge collisions · isolated review",
|
||||
"ogTitle_en": "The Single-Writer Rule for AI Agents",
|
||||
"ogTitle_zh": "代码只让一个 AI 改:单写多审架构"
|
||||
},
|
||||
"slashing-agent-context-with-a-code-graph": {
|
||||
"evidence": "AST graph indexing · 80% token churn cut · 0 hallucination",
|
||||
"ogTitle_en": "Slashing Context With Code Graph",
|
||||
"ogTitle_zh": "用 Code Graph 斩断 Token 膨胀与遗忘"
|
||||
},
|
||||
"why-multi-agent-coding-fails": {
|
||||
"evidence": "Telephone game distortion · implicit architectural drift",
|
||||
"ogTitle_en": "Why Multi-Agent Coding Fails",
|
||||
"ogTitle_zh": "多 AI 写代码烂尾与隐式决策困境"
|
||||
},
|
||||
"generating-full-length-songs-with-yue2-comfyui": {
|
||||
"evidence": "60 songs - 9-part structure - 295s headroom cap"
|
||||
},
|
||||
"when-lowering-crf-does-nothing-maxrate-deadlock": {
|
||||
"evidence": "CRF 19→14 · SSIM +0.00003 · 106.2h wasted",
|
||||
"ogTitle_en": "When Lowering CRF Does Nothing",
|
||||
"ogTitle_zh": "降低 CRF 为何毫无效果:maxrate 死锁"
|
||||
},
|
||||
"why-ffmpeg-map-0-breaks-your-mp4-and-mkv-remux": {
|
||||
"evidence": "-map 0 copy · tx3g CC · udta/name dual-write",
|
||||
"ogTitle_en": "Why ffmpeg -map 0 Breaks Remux",
|
||||
"ogTitle_zh": "ffmpeg -map 0 搞崩跨容器混流"
|
||||
},
|
||||
"traefik-yaegi-plugin-package-name-trap": {
|
||||
"evidence": "Yaegi lookup · hyphen to underscore · panic",
|
||||
"ogTitle_en": "Traefik Yaegi Plugin Package Trap",
|
||||
"ogTitle_zh": "Traefik Yaegi 插件启动崩溃陷阱"
|
||||
},
|
||||
"architecting-sea-payment-and-offline-licensing-pipelines": {
|
||||
"evidence": "toyyibPay FPX · FOR UPDATE · Ed25519 offline",
|
||||
"ogTitle_en": "SEA Payment & Offline Licensing",
|
||||
"ogTitle_zh": "东南亚网银支付与离线机器码授权"
|
||||
},
|
||||
"batch-generating-80-technical-covers-with-comfyui": {
|
||||
"evidence": "89 covers in 48m · 35% safe zone · Laplacian filter",
|
||||
"ogTitle_en": "Batch Generating 80+ Technical Covers",
|
||||
"ogTitle_zh": "ComfyUI 批量闭环 80+ 张技术封面"
|
||||
},
|
||||
"when-a-proxy-timeout-returns-html-instead-of-json": {
|
||||
"evidence": "13.6 s -> 0.26 s · 2507 s worst · 499 at the gate",
|
||||
"ogTitle_en": "The Proxy That Replied in HTML",
|
||||
"ogTitle_zh": "反代用 HTML 回答了 JSON",
|
||||
"tags": [
|
||||
"nginx",
|
||||
"timeouts",
|
||||
"caching"
|
||||
]
|
||||
},
|
||||
"read-amplification-504-gateway-timeout-swr-cache": {
|
||||
"evidence": "4 reads/file · 960 per page · 0.3 µs cache hit"
|
||||
},
|
||||
"authentik-oidc-redirect-loop-node-set-header-overwrite": {
|
||||
"evidence": "Set-Cookie overwrite · 0 server errors · 21 links verified"
|
||||
},
|
||||
"six-chrome-containers-locked-my-nas-btrfs-d-state-igpu-offload": {
|
||||
"evidence": "7.5 GB RAM freed · Load 27.4 → 6.8 · iGPU render node"
|
||||
},
|
||||
"cascading-docker-failures-nocodb-heap-limit-starved-n8n": {
|
||||
"evidence": "768 MB OOM · max-old-space-size 1536 · 100% workflow health"
|
||||
},
|
||||
"apple-icloud-smtp-port-465-timeout-starttls-587-trap": {
|
||||
"evidence": "Port 465 timeout · 587 STARTTLS only · exit code 0"
|
||||
},
|
||||
"home-assistant-container-upgrade-ladder-four-hops": {
|
||||
"evidence": "13 releases · 4-hop ladder · 259 → 265 entities intact"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
ogImage: "/og/apple-icloud-smtp-port-465-timeout-starttls-587-trap.jpg"
|
||||
banner: "/banners/apple-icloud-smtp-port-465-timeout-starttls-587-trap.jpg"
|
||||
title: "Apple iCloud SMTP Times Out on Port 465: The 587 STARTTLS-Only Trap"
|
||||
description: "Why configuring an automated mail client for iCloud SMTP on port 465 results in silent connection timeouts (WSAETIMEDOUT), and why Apple's outbound mail architecture requires explicit STARTTLS on port 587."
|
||||
pubDate: 2026-06-18
|
||||
updatedDate: 2026-11-01
|
||||
category: notes
|
||||
blueprint: self-hosted-infrastructure
|
||||
blueprintLayer: L2-edge-proxy-runtime
|
||||
tags: ["email", "smtp", "icloud", "networking", "troubleshooting", "himalaya"]
|
||||
draft: false
|
||||
summary:
|
||||
audience: "System Administrators, CLI Tool Users & Mail Automation Engineers"
|
||||
painPoint: "Automated mail monitors and CLI tools failed to send email via Apple iCloud accounts, hanging indefinitely on connection attempts with os error 10060 while IMAP reading worked perfectly."
|
||||
impact:
|
||||
- metric: "Zero Timeout Latency"
|
||||
detail: "Switched transport from implicit SSL on port 465 to explicit STARTTLS on port 587, eliminating connection timeouts."
|
||||
- metric: "Silent Cron Execution"
|
||||
detail: "Restored exit code 0 across all account health checks, stopping false-positive failure alerts from waking on-call operators."
|
||||
- metric: "Protocol Compliance"
|
||||
detail: "Aligned client configuration directly with RFC 6409 / RFC 3207 submission standards enforced by Apple mail infrastructure."
|
||||
stack: ["Himalaya CLI", "SMTP / STARTTLS", "iCloud Mail", "Python", "Cron"]
|
||||
jumpLinks:
|
||||
business: "#the-alert-that-broke-the-silence"
|
||||
architecture: "#implicit-tls-versus-starttls"
|
||||
result: "#the-result"
|
||||
dynamicCta: true
|
||||
---
|
||||
|
||||
## The alert that broke the silence
|
||||
|
||||
An automated account health monitor running as a background maintenance task on a Windows administration host dispatched an alert during scheduled telemetry:
|
||||
|
||||
```text
|
||||
Cronjob Response: himalaya account health (job_id: 2f09440986e4)
|
||||
-------------
|
||||
📧 himalaya account health check — FAILED
|
||||
|
||||
❌ icloud (smtp)
|
||||
smtp: FAIL (connect smtp.mail.me.com:465: A connection attempt failed
|
||||
because the connected party did not properly respond after a period of time,
|
||||
or established connection failed because connected host has failed to respond. (os error 10060 / WSAETIMEDOUT))
|
||||
```
|
||||
|
||||
Reading mail over IMAP (`imap.mail.me.com:993`) continued to function with sub-second response times. Authentication tokens and app-specific passwords were confirmed valid. Yet every outbound delivery attempt over SMTP stalled for thirty seconds before aborting with a TCP timeout error (`WSAETIMEDOUT` / OS error 10060).
|
||||
|
||||
## Implicit TLS versus STARTTLS
|
||||
|
||||
The root cause lay in a subtle assumption regarding outbound mail security standards. In the CLI client configuration (`~/.config/himalaya/config.toml`), the iCloud profile had been configured following the conventional pattern used by Gmail and custom mail servers:
|
||||
|
||||
```toml
|
||||
# Initial configuration
|
||||
[accounts.icloud]
|
||||
email = "[email protected]"
|
||||
backend = "imap"
|
||||
imap.server = "imap.mail.me.com"
|
||||
imap.port = 993
|
||||
imap.encryption = "tls"
|
||||
|
||||
message.send.backend = "smtp"
|
||||
smtp.server = "smtps://smtp.mail.me.com:465"
|
||||
```
|
||||
|
||||
Port 465 was originally allocated for SMTPS (implicit TLS, where the TLS handshake happens immediately upon TCP connection establishment). While many commercial mail providers support both port 465 and port 587, Apple's mail infrastructure (`smtp.mail.me.com`) strictly enforces legacy RFC 6409 / RFC 3207 submission standards rather than RFC 8314 implicit TLS on port 465:
|
||||
|
||||
1. **Port 465 is completely closed or dropped** on Apple edge firewalls. Packets sent to port 465 are silently discarded (leading to `WSAETIMEDOUT` / `ETIMEDOUT` rather than an immediate `ECONNREFUSED`).
|
||||
2. Outbound message submission is **only accepted on port 587** via cleartext TCP followed by an explicit `STARTTLS` handshake.
|
||||
|
||||
Attempting an implicit SSL connection (`smtps://`) to port 465 meant the client waited for a TLS Client Hello response that Apple's gateway was never configured to send.
|
||||
|
||||
## The fix: switching to port 587 with starttls
|
||||
|
||||
The configuration was adjusted to enforce explicit STARTTLS over port 587:
|
||||
|
||||
<details class="tech-deep-dive">
|
||||
<summary>🔍 config.toml — iCloud SMTP configuration (11 lines)</summary>
|
||||
|
||||
```toml
|
||||
[accounts.icloud]
|
||||
email = "[email protected]"
|
||||
backend = "imap"
|
||||
imap.server = "imap.mail.me.com"
|
||||
imap.port = 993
|
||||
imap.encryption = "tls"
|
||||
|
||||
message.send.backend = "smtp"
|
||||
smtp.server = "smtp://smtp.mail.me.com:587"
|
||||
smtp.starttls = true
|
||||
```
|
||||
</details>
|
||||
|
||||
Notice the change from the `smtps://` URI scheme to `smtp://`, combined with `smtp.starttls = true`.
|
||||
|
||||
## The result
|
||||
|
||||
Testing the connection immediately after the configuration change verified the resolution:
|
||||
|
||||
```text
|
||||
$ himalaya account check --account icloud
|
||||
Account: icloud
|
||||
imap: OK
|
||||
smtp: OK
|
||||
```
|
||||
|
||||
The health check cron script executed with an exit code of `0`. All six configured mail profiles returned green status, and monitoring returned to its normal silent operation.
|
||||
@@ -0,0 +1,123 @@
|
||||
---
|
||||
ogImage: "/og/authentik-oidc-redirect-loop-node-set-header-overwrite.jpg"
|
||||
banner: "/banners/authentik-oidc-redirect-loop-node-set-header-overwrite.jpg"
|
||||
title: "The Authentik OIDC Redirect Loop: Why Node.js res.setHeader Overwrites Rather Than Appends"
|
||||
description: "A production admin portal entered an infinite login loop after successful Authentik OIDC authorization with zero server errors logged. The root cause was Node.js res.setHeader replacing earlier Set-Cookie headers instead of appending."
|
||||
pubDate: 2025-02-14
|
||||
updatedDate: 2026-11-01
|
||||
category: engineering
|
||||
blueprint: self-hosted-infrastructure
|
||||
blueprintLayer: L3-application-runtime
|
||||
tags: ["authentik", "oidc", "nodejs", "cookies", "authentication", "docker"]
|
||||
draft: false
|
||||
summary:
|
||||
audience: "Full-Stack Developers, Identity Engineers & Node.js System Maintainers"
|
||||
painPoint: "Users logging into a custom administration portal via Authentik OIDC were bounced back into an infinite redirect loop between the gateway and callback, while container logs showed 100% HTTP 302 redirects and successful token exchanges with zero warnings."
|
||||
impact:
|
||||
- metric: "Zero Phantom Redirects"
|
||||
detail: "Eliminated the OIDC callback bounce loop by replacing bare res.setHeader calls with an array-appending cookie accumulator."
|
||||
- metric: "Verified Session Retention"
|
||||
detail: "Proved end-to-end authentication via Playwright headless browser automation, confirming cookie persistence and 21 dashboard rows rendered."
|
||||
- metric: "Zero Overhead Defense"
|
||||
detail: "Fixed the state machine with a lightweight array accumulator without introducing session stores or stateful server dependencies."
|
||||
stack: ["Node.js", "Authentik", "OIDC", "Playwright", "Docker"]
|
||||
jumpLinks:
|
||||
business: "#the-infinite-login-loop-with-clean-logs"
|
||||
architecture: "#root-cause-how-res-set-header-erased-sessions"
|
||||
result: "#the-result"
|
||||
dynamicCta: true
|
||||
---
|
||||
|
||||
## The infinite login loop with clean logs
|
||||
|
||||
The incident presented as a silent access failure. Users attempting to access the protected administration dashboard at `/admin` were redirected to the Authentik identity provider, prompted for multi-factor credentials, granted consent, and redirected back to the application callback.
|
||||
|
||||
Instead of landing on the dashboard, the browser immediately rebounded to Authentik. Authentik recognised the existing active user session, issued a fresh authorization code, and redirected back once again. This ping-pong loop repeated until the browser aborted with `ERR_TOO_MANY_REDIRECTS`.
|
||||
|
||||
Inspecting the service logs inside the container showed no errors whatsoever:
|
||||
|
||||
```text
|
||||
127.0.0.1 - - [14/Feb/2025:08:12:04 +0000] "GET /admin HTTP/1.1" 302 -
|
||||
127.0.0.1 - - [14/Feb/2025:08:12:05 +0000] "GET /callback?code=... HTTP/1.1" 302 -
|
||||
127.0.0.1 - - [14/Feb/2025:08:12:05 +0000] "GET /admin HTTP/1.1" 302 -
|
||||
127.0.0.1 - - [14/Feb/2025:08:12:06 +0000] "GET /callback?code=... HTTP/1.1" 302 -
|
||||
```
|
||||
|
||||
Every single hop reported HTTP 302 Found. Authentik logged successful token exchanges. The OpenID Connect tokens were mathematically valid and unexpired. Yet the application refused to recognize that authentication had succeeded.
|
||||
|
||||
## Root cause: how res.setHeader erased sessions
|
||||
|
||||
The application runtime used a lightweight native Node.js HTTP server without bulky frameworks. During the OIDC callback handling cycle, two distinct cookie operations needed to happen in sequence:
|
||||
|
||||
1. Setting the permanent authenticated session cookie (`app_sess=<token>; HttpOnly; Secure; SameSite=Lax`).
|
||||
2. Expiring and clearing the transient OIDC state cookie (`app_oidc=; Max-Age=0`).
|
||||
|
||||
The naive response helper had been written like this:
|
||||
|
||||
```javascript
|
||||
function setCookie(res, cookieStr) {
|
||||
res.setHeader('Set-Cookie', cookieStr);
|
||||
}
|
||||
```
|
||||
|
||||
In the Node.js `http` module (`OutgoingMessage`), calling `res.setHeader(name, value)` with a string argument is an **atomic replacement**, not an append operation.
|
||||
|
||||
When the callback controller executed:
|
||||
|
||||
```javascript
|
||||
// Step 1: Establish session
|
||||
setCookie(res, `app_sess=${sessionToken}; Path=/; HttpOnly; Secure; SameSite=Lax`);
|
||||
|
||||
// Step 2: Clear transient state
|
||||
setCookie(res, `app_oidc=; Path=/; Max-Age=0; HttpOnly; Secure; SameSite=Lax`);
|
||||
```
|
||||
|
||||
The second call completely blew away the `Set-Cookie` header established by the first call.
|
||||
|
||||
When inspecting the actual response headers using `curl -v`:
|
||||
|
||||
```text
|
||||
< HTTP/1.1 302 Found
|
||||
< Location: /admin
|
||||
< Set-Cookie: app_oidc=; Path=/; Max-Age=0; HttpOnly; Secure; SameSite=Lax
|
||||
```
|
||||
|
||||
The `app_sess` cookie never reached the browser. The browser followed the redirect to `/admin`, sent no session token, failed the route guard, and was promptly dispatched right back to Authentik. Because every step succeeded according to its own isolated logic, not a single line of warning was emitted to stderr.
|
||||
|
||||
## The fix: array-based header aggregation
|
||||
|
||||
The fix required ensuring that any invocation of `setCookie` preserves previously registered cookies in the response header array.
|
||||
|
||||
<details class="tech-deep-dive">
|
||||
<summary>🔍 server.mjs — setCookie implementation (12 lines)</summary>
|
||||
|
||||
```javascript
|
||||
function setCookie(res, cookieStr) {
|
||||
const existing = res.getHeader('Set-Cookie');
|
||||
let cookies = [];
|
||||
if (Array.isArray(existing)) {
|
||||
cookies = existing.concat(cookieStr);
|
||||
} else if (typeof existing === 'string') {
|
||||
cookies = [existing, cookieStr];
|
||||
} else {
|
||||
cookies = [cookieStr];
|
||||
}
|
||||
res.setHeader('Set-Cookie', cookies);
|
||||
}
|
||||
```
|
||||
</details>
|
||||
|
||||
By coercing existing header values into an array and passing the array back to `res.setHeader()`, Node's serialization layer produces multiple distinct `Set-Cookie` lines in the raw HTTP response.
|
||||
|
||||
## The result
|
||||
|
||||
After deploying the patch, the OIDC handshake was verified using an automated Playwright headless browser script.
|
||||
|
||||
The test script completed the full authentication lifecycle:
|
||||
- Origin request to `/admin` intercepted and redirected to Authentik.
|
||||
- Single sign-on form filled and submitted.
|
||||
- OIDC authorization consent granted.
|
||||
- Callback returned with both `app_sess` and expired `app_oidc` cookies present.
|
||||
- Destination page loaded with HTTP 200, successfully rendering all 21 management rows in the admin dashboard.
|
||||
|
||||
The entire failure mode was traced and resolved at the HTTP protocol layer without altering identity provider settings or degrading token security.
|
||||
@@ -0,0 +1,111 @@
|
||||
---
|
||||
ogImage: "/og/cascading-docker-failures-nocodb-heap-limit-starved-n8n.jpg"
|
||||
banner: "/banners/cascading-docker-failures-nocodb-heap-limit-starved-n8n.jpg"
|
||||
title: "Cascading Docker Failures: When a Node.js Heap Limit in NocoDB Starved an n8n Automation Stack"
|
||||
description: "A 768 MB container limit caused NocoDB to crash with JavaScript heap out of memory, triggering hundreds of ECONNREFUSED failures across dependent n8n workflows. Resolving it required max-old-space-size tuning and pruning zombie executions."
|
||||
pubDate: 2025-04-12
|
||||
updatedDate: 2026-11-01
|
||||
category: devops
|
||||
blueprint: self-hosted-infrastructure
|
||||
blueprintLayer: L3-application-runtime
|
||||
tags: ["n8n", "nocodb", "docker", "nodejs", "memory", "automation", "troubleshooting"]
|
||||
draft: false
|
||||
summary:
|
||||
audience: "DevOps Engineers, Integration Architects & Automation Workflow Maintainers"
|
||||
painPoint: "Dozens of business-critical n8n automations failed simultaneously with ECONNREFUSED 10380, filling error logs and leaving long-running workflows permanently stuck in execution limbo."
|
||||
impact:
|
||||
- metric: "Zero Upstream Connection Drops"
|
||||
detail: "Raised the NocoDB container envelope to 2048 MB and aligned Node.js max-old-space-size to 1536 MB, eliminating the memory crashes."
|
||||
- metric: "100% Workflow Recovery"
|
||||
detail: "Purged orphaned running records from the orchestration database and restored end-to-end webhook delivery without dropped payloads."
|
||||
- metric: "SATA Array I/O Protection"
|
||||
detail: "Prevented disk contention caused by cascading retry storms against unresponsive storage backends."
|
||||
stack: ["n8n", "NocoDB", "Docker Compose", "Node.js", "PostgreSQL", "Portainer"]
|
||||
jumpLinks:
|
||||
business: "#the-morning-the-automation-board-turned-red"
|
||||
architecture: "#root-cause-the-default-v8-heap-ceiling"
|
||||
result: "#the-result"
|
||||
dynamicCta: true
|
||||
---
|
||||
|
||||
## The morning the automation board turned red
|
||||
|
||||
The failure appeared during a routine system inspection on a self-hosted automation node. The n8n administration dashboard reported dozens of workflow failures overnight:
|
||||
|
||||
```text
|
||||
NodeApiError: connect ECONNREFUSED 192.168.1.1:10380
|
||||
at Object.request (/usr/local/lib/node_modules/n8n/node_modules/axios/lib/adapters/http.js:64:14)
|
||||
at processTicksAndRejections (node:internal/process/task_queues:95:5)
|
||||
```
|
||||
|
||||
Every integration responsible for syncing Telegram registrations, form submissions, and customer inquiries into the central database was failing. In addition, several scheduled batch jobs were marked as permanently `running`, accumulating memory without making progress.
|
||||
|
||||
Checking the health of the target service—NocoDB on port `10380`—confirmed that the container was not serving traffic. It was caught in a restart crash loop.
|
||||
|
||||
## Root cause: the default V8 heap ceiling
|
||||
|
||||
Inspecting the container logs via Portainer revealed an explicit Node.js runtime crash:
|
||||
|
||||
```text
|
||||
<--- Last few GCs --->
|
||||
[1:0x55d7b8a8b000] 184201 ms: Mark-sweep (reduce) 712.4 (724.8) -> 710.2 (725.1) MB, 412.5 / 0.0 ms
|
||||
[1:0x55d7b8a8b000] 184712 ms: Mark-sweep (reduce) 715.1 (725.1) -> 714.8 (726.0) MB, 511.0 / 0.0 ms
|
||||
|
||||
<--- JS stacktrace --->
|
||||
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
|
||||
1: 0xb83a30 node::Abort() [node]
|
||||
2: 0xa94415 node::FatalError(char const*, char const*) [node]
|
||||
3: 0xd45062 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
|
||||
```
|
||||
|
||||
When NocoDB had been provisioned in Docker Compose, the container had been given a strict cgroup limit:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
nocodb:
|
||||
image: nocodb/nocodb:latest
|
||||
mem_limit: 768m
|
||||
```
|
||||
|
||||
Under a tight 768 MB cgroup boundary, V8's default heap heuristics leave insufficient headroom for spiky aggregation workloads. Under complex aggregation queries and table joins across tens of thousands of rows, memory usage expanded beyond 700 MB. The V8 garbage collector attempted aggressive mark-sweep cycles, failed to reclaim enough contiguous pages, and triggered a fatal abort.
|
||||
|
||||
When NocoDB died, every dependent n8n workflow experienced connection refused errors. Because n8n had retry policies configured on its database nodes, hundreds of retries hammered the port simultaneously as Docker restarted the container, creating an avalanche of queued requests that re-triggered the OOM crash within seconds of boot.
|
||||
|
||||
## The fix: memory alignment and orphan cleanup
|
||||
|
||||
Resolving the cascade required addressing both the container headroom and the runtime heap sizing:
|
||||
|
||||
<details class="tech-deep-dive">
|
||||
<summary>🔍 docker-compose.yml — NocoDB memory configuration (17 lines)</summary>
|
||||
|
||||
```yaml
|
||||
services:
|
||||
nocodb:
|
||||
image: nocodb/nocodb:latest
|
||||
container_name: nocodb-server
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- NC_DB=pg://postgres:5432?u=...
|
||||
- NODE_OPTIONS=--max-old-space-size=1536
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 2048M
|
||||
reservations:
|
||||
memory: 512M
|
||||
ports:
|
||||
- "10380:8080"
|
||||
```
|
||||
</details>
|
||||
|
||||
By setting `NODE_OPTIONS: "--max-old-space-size=1536"` alongside a 2048 MB Docker memory ceiling, V8 was given 1.5 GB of user-space heap while leaving 512 MB of container memory for native bindings, libuv buffers, and OS overhead.
|
||||
|
||||
Next, the database state in n8n was cleaned up. Long-running executions that had stalled during the database outage were identified in the orchestration storage and marked as failed so worker threads could release their execution locks.
|
||||
|
||||
## The result
|
||||
|
||||
Following the stack restart and configuration alignment:
|
||||
- NocoDB initialized cleanly, reporting `{"message":"OK"}` on its `/api/v1/health` endpoint.
|
||||
- Webhook events and data pipelines processed without a single connection drop.
|
||||
- Host memory on the container host stabilized without further OOM kills.
|
||||
- The automation failure counter dropped back to zero and stayed clean across subsequent batch cron cycles.
|
||||
@@ -0,0 +1,83 @@
|
||||
---
|
||||
ogImage: "/og/home-assistant-container-upgrade-ladder-four-hops.jpg"
|
||||
banner: "/banners/home-assistant-container-upgrade-ladder-four-hops.jpg"
|
||||
title: "Home Assistant Container: Upgrading Thirteen Releases Across a Four-Hop Ladder"
|
||||
description: "Upgrading a containerized Home Assistant installation from 2025.9 to 2026.10 in a single leap risks irreversible SQLite database corruption and broken reverse-proxy gates. Here is how a verified 4-hop ladder preserved 259 entities."
|
||||
pubDate: 2026-07-22
|
||||
updatedDate: 2026-11-01
|
||||
category: devops
|
||||
blueprint: self-hosted-infrastructure
|
||||
blueprintLayer: L3-application-runtime
|
||||
tags: ["iot", "docker", "upgrade", "sqlite", "traefik", "reverse-proxy"]
|
||||
draft: false
|
||||
summary:
|
||||
audience: "IoT Telemetry Engineers, Edge Infrastructure Architects & Container Maintainers"
|
||||
painPoint: "Upgrading Home Assistant Container across thirteen monthly releases in one blind jump risks database migration deadlock, Python 3.14 syntax deprecations, and silent reverse-proxy lockout."
|
||||
impact:
|
||||
- metric: "259 → 265 Entities Intact"
|
||||
detail: "Diffed entity registries before and after each migration step, confirming zero dropped devices or broken state machines."
|
||||
- metric: "Zero Recorder Schema Corruption"
|
||||
detail: "Safely migrated SQLite recorder schema 50 to 53 across isolated stops without table locks or WAL truncation."
|
||||
- metric: "Reverse Proxy Gate Preserved"
|
||||
detail: "Preserved reverse-proxy trusted_proxies and use_x_forwarded_for validation checks alongside .storage/http state transitions, avoiding HTTP 400 bad gateway lockouts."
|
||||
stack: ["Home Assistant Core", "Docker Compose", "SQLite", "Traefik", "Python 3.14"]
|
||||
jumpLinks:
|
||||
business: "#the-thirteen-version-maintenance-debt"
|
||||
architecture: "#isolating-hazards-across-four-hops"
|
||||
result: "#the-result"
|
||||
dynamicCta: true
|
||||
---
|
||||
|
||||
## The thirteen-version maintenance debt
|
||||
|
||||
When operating Home Assistant as a dedicated IoT telemetry and environmental control service inside a standalone Docker container (`homeassistant/home-assistant`)—aggregating rack thermal sensors, UPS power metrics, and automated alerts—system updates do not come from a one-click dashboard button. Every update requires pulling a new container image, recreating the service, and running database migrations on startup.
|
||||
|
||||
Over thirteen months, a production installation had remained pinned to version **2025.9.4**. Upstream releases had moved to **2026.10.4**. Jumping straight across thirteen major versions is the single fastest way to brick an automation stack:
|
||||
|
||||
1. **Irreversible Database Migrations**: The recorder component upgraded its SQLite schema from version 50 to 53. If an intermediate migration fails mid-stream, the SQLite file cannot be rolled back without cold backups.
|
||||
2. **Python Runtime Shifts**: The base container migrated from Python 3.13 to 3.14, introducing stricter asynchronous task constraints and deprecating legacy helper functions.
|
||||
3. **The 2026.8 HTTP Gate Rewrite**: In 2026.8, Home Assistant restructured its reverse-proxy security subsystem, migrating network parameters into internal storage files (`.storage/http`). A failed migration turns external access through reverse proxies into immediate `HTTP 400: Bad Request` errors.
|
||||
|
||||
To eliminate risk, the upgrade was partitioned into a verified 4-hop ladder.
|
||||
|
||||
## Isolating hazards across four hops
|
||||
|
||||
Official guidelines recommend stepping through no more than five to six minor releases per increment. The selected upgrade path isolated each architectural breaking change:
|
||||
|
||||
| Hop | Target Image | Primary Hazard Isolated | Verification Gate |
|
||||
|---|---|---|---|
|
||||
| **1** | `2025.12.5` | Recorder SQLite schema migration (v50 → v53) | Database schema verified, WAL clean |
|
||||
| **2** | `2026.3.4` | Python 3.13 → 3.14 runtime & zstd container layers | Log inspection: 0 setup failures |
|
||||
| **3** | `2026.8.3` | `http:` reverse-proxy trusted_proxies validation & `.storage/http` transition | Reverse proxy headers pass; HTTP 200 |
|
||||
| **4** | `2026.10.4` | Current stable target & final auth cleanup | All 259 baseline entities verified |
|
||||
|
||||
Before pulling images, a cold snapshot of the `/config` volume was created with the container stopped, ensuring database consistency.
|
||||
|
||||
<details class="tech-deep-dive">
|
||||
<summary>🔍 docker-compose.yml — Home Assistant container definition (14 lines)</summary>
|
||||
|
||||
```yaml
|
||||
services:
|
||||
homeassistant:
|
||||
container_name: home-assistant
|
||||
image: homeassistant/home-assistant:2026.10.4
|
||||
volumes:
|
||||
- /volume1/docker/home-assistant/config:/config
|
||||
- /etc/localtime:/etc/localtime:ro
|
||||
environment:
|
||||
- TZ=Asia/Kuala_Lumpur
|
||||
network_mode: host
|
||||
restart: unless-stopped
|
||||
stop_grace_period: 60s
|
||||
```
|
||||
</details>
|
||||
|
||||
Notice `stop_grace_period: 60s`: under large SQLite databases, Home Assistant requires adequate time to commit SQLite WAL pages and flush event buffers. The default 10-second Docker timeout frequently causes SIGKILL aborts (exit code 137).
|
||||
|
||||
## The result
|
||||
|
||||
The upgrade proceeded through all four stages without rollback:
|
||||
|
||||
- **Entity Preservation**: The baseline entity count expanded from **259 to 265** (the six additional entries were new native power-distribution metrics), with zero lost automations or unavailable sensors.
|
||||
- **Recorder Health**: The database transitioned across schema versions in less than one second, with total row counts growing cleanly from 249,875 to 249,897.
|
||||
- **Gateway Integrity**: The reverse proxy gate continued to enforce mutual authentication, with `X-Forwarded-For` validation passing without configuration regression.
|
||||
@@ -0,0 +1,99 @@
|
||||
---
|
||||
ogImage: "/og/six-chrome-containers-locked-my-nas-btrfs-d-state-igpu-offload.jpg"
|
||||
banner: "/banners/six-chrome-containers-locked-my-nas-btrfs-d-state-igpu-offload.jpg"
|
||||
title: "Why Six Desktop Chrome Containers Locked a Production NAS: CPU WebGL Emulation, Btrfs D-State, and Offloading to iGPU"
|
||||
description: "Running multiple containerized Chrome instances on an embedded 4-core NAS drove load average past 27, freezing Btrfs transactions in uninterruptible sleep. Migrating them to an Intel iGPU host freed 7.5 GB of committed memory and dropped host load to 6.8."
|
||||
pubDate: 2025-03-20
|
||||
updatedDate: 2026-11-01
|
||||
category: devops
|
||||
blueprint: self-hosted-infrastructure
|
||||
blueprintLayer: L1-host-storage-compute
|
||||
tags: ["docker", "synology", "btrfs", "gpu", "chrome", "unraid", "cdp"]
|
||||
draft: false
|
||||
summary:
|
||||
audience: "Private Cloud Architects, SREs & Infrastructure Engineers Managing High-Density Workloads"
|
||||
painPoint: "Six desktop Chrome containers running automation tasks on a 4-core embedded NAS without an integrated GPU consumed 100% CPU on WebGL emulation, driving the host into memory swap and deadlocking Btrfs storage transactions."
|
||||
impact:
|
||||
- metric: "7.5 GB Committed Memory Freed"
|
||||
detail: "Purged redundant browser instances and swap pressure from the storage appliance, reclaiming 7.5 GB of committed memory (physical RAM + swap) and returning host metrics to healthy operational baselines."
|
||||
- metric: "Load Average 27.4 → 6.8"
|
||||
detail: "Eliminated CPU soft-rendering contention, unfreezing container daemon removal locks and eliminating upstream 502 gateway timeouts."
|
||||
- metric: "Zero-Idle Architecture"
|
||||
detail: "Re-architected browser automation on an Intel UHD 770 host with on-demand execution, idling at 0% CPU and 0 MB RAM when inactive."
|
||||
stack: ["Synology DSM", "Btrfs", "Docker", "Intel QuickSync / DRI", "unRaid", "CDP"]
|
||||
jumpLinks:
|
||||
business: "#the-storage-appliance-that-stopped-responding"
|
||||
architecture: "#anatomy-of-a-btrfs-d-state-freeze"
|
||||
result: "#the-result"
|
||||
dynamicCta: true
|
||||
---
|
||||
|
||||
## The storage appliance that stopped responding
|
||||
|
||||
The incident surfaced when reverse-proxied management dashboards and cloud drive services on a Synology DS1821+ NAS simultaneously timed out. Direct HTTPS handshakes succeeded and certificates matched, but backend requests hung indefinitely.
|
||||
|
||||
Opening an SSH terminal to the host revealed extreme resource exhaustion:
|
||||
|
||||
```text
|
||||
# uptime
|
||||
09:42:15 up 42 days, 18:22, 1 user, load average: 27.42, 25.18, 22.04
|
||||
|
||||
# free -m
|
||||
total used free shared buff/cache available
|
||||
Mem: 7842 7410 112 340 320 180
|
||||
Swap: 4096 3980 116
|
||||
```
|
||||
|
||||
On a 4-core AMD Ryzen V1500B embedded platform, a load average above 27 meant that more than twenty processes were trapped in execution queues or uninterruptible kernel sleep. Memory was 94% utilized, and swap was almost completely exhausted.
|
||||
|
||||
The primary consumers were six desktop Chrome containers (`lscr.io/linuxserver/chrome`) hosting automation sessions for generative AI, document processing, and browser pipelines.
|
||||
|
||||
## Anatomy of a Btrfs D-state freeze
|
||||
|
||||
The DS1821+ platform lacks an integrated GPU. Modern web applications—particularly canvas-heavy interfaces, document renderers, and video previews—rely heavily on WebGL. Without a hardware acceleration device passed through (`/dev/dri`), Chromium transparently falls back to CPU emulation using SwiftShader or llvmpipe.
|
||||
|
||||
This architectural mismatch created a three-stage cascade:
|
||||
|
||||
1. **CPU Saturation**: Six headless/streaming browser instances ran software rasterization simultaneously, pegging all 4 physical cores at 100% utilization.
|
||||
2. **Aggressive Swapping**: As browser renderers allocated temporary graphic buffers, the kernel was forced to swap anonymous pages to mechanical spinning rust (`/volume1`), introducing massive I/O wait.
|
||||
3. **Storage Transaction Lockup**: When the container manager attempted to delete an ephemeral diagnostic container, the Docker daemon invoked a Btrfs subvolume deletion (`btrfs subvolume delete`). Under saturated disk queues, the filesystem transaction stalled in kernel D-state (uninterruptible sleep).
|
||||
|
||||
Because the Btrfs root transaction lock was held, other system services—including local PostgreSQL instances, socket listeners, and reverse proxy workers—locked up as soon as they performed synchronous filesystem writes. The entire machine froze.
|
||||
|
||||
## The migration: offloading to an Intel iGPU host
|
||||
|
||||
The long-term fix was to separate storage appliances from GPU-bound browser automation workloads.
|
||||
|
||||
The browser automation fleet was relocated to a dedicated unRaid compute server equipped with an Intel Core i9-13900K and an integrated Intel UHD Graphics 770 processor.
|
||||
|
||||
<details class="tech-deep-dive">
|
||||
<summary>🔍 docker-compose.yml — iGPU hardware-accelerated Chrome worker (17 lines)</summary>
|
||||
|
||||
```yaml
|
||||
services:
|
||||
chrome-worker:
|
||||
image: lscr.io/linuxserver/chrome:latest
|
||||
container_name: chrome-worker
|
||||
devices:
|
||||
- /dev/dri/renderD128:/dev/dri/renderD128
|
||||
- /dev/dri/card0:/dev/dri/card0
|
||||
environment:
|
||||
- CHROME_CLI=--remote-debugging-port=9222 --user-data-dir=/config/profile --enable-features=VaapiVideoDecodeLinuxGL --ignore-gpu-blocklist --use-gl=angle --use-angle=gl
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 3072M
|
||||
restart: "no"
|
||||
```
|
||||
</details>
|
||||
|
||||
Passing `/dev/dri/renderD128` enabled native hardware decoding and WebGL rendering directly on the silicon. Furthermore, changing container policy from `restart: unless-stopped` to an on-demand orchestration model ensured that idle worker browsers stay shut down at 0% CPU and 0 MB RAM until explicitly awakened by automation jobs.
|
||||
|
||||
## The result
|
||||
|
||||
The impact on the storage appliance was immediate:
|
||||
|
||||
- Terminating the six soft-rendering containers on the NAS recovered **7.5 GB of committed memory (physical RAM + swap)** and stopped swap thrashing.
|
||||
- Host load average plunged from **27.42 down to 6.80** within minutes.
|
||||
- Trapped Btrfs transactions committed cleanly, restoring cloud drive and portal responses to sub-second latencies.
|
||||
- On the Intel compute node, automated browser workflows executed with native GPU hardware acceleration, reducing graphic-heavy script latency by more than 60%.
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
ogImage: "/og/zh-apple-icloud-smtp-port-465-timeout-starttls-587-trap.jpg"
|
||||
banner: "/banners/apple-icloud-smtp-port-465-timeout-starttls-587-trap.jpg"
|
||||
title: "Apple iCloud 邮箱发信 465 超时:仅支持 587 STARTTLS 的协议陷阱"
|
||||
description: "为什么在自动化邮件客户端中将 Apple iCloud SMTP 设为 465 端口会导致连接超时(WSAETIMEDOUT / os error 10060)?剖析 Apple 邮件网关对 587 端口显式 STARTTLS 的硬性要求。"
|
||||
pubDate: 2026-06-18
|
||||
updatedDate: 2026-11-01
|
||||
category: notes
|
||||
blueprint: self-hosted-infrastructure
|
||||
blueprintLayer: L2-edge-proxy-runtime
|
||||
tags: ["邮件协议", "smtp", "icloud", "网络排障", "协议规范", "himalaya"]
|
||||
draft: false
|
||||
summary:
|
||||
audience: "系统管理员、CLI 生产力工具用户与邮件自动化开发者"
|
||||
painPoint: "配置 CLI 邮件工具自动化发送 iCloud 邮件时,连接无休止挂起并报错 os error 10060 超时,而 IMAP 读信完全正常且应用专用密码有效。"
|
||||
impact:
|
||||
- metric: "根除连接超时"
|
||||
detail: "将传输方式由 465 隐式 TLS 直连切换为 587 端口显式 STARTTLS 握手,连接耗时由 30 秒超时缩减至毫秒级成功。"
|
||||
- metric: "监控自动静默"
|
||||
detail: "全量恢复健康巡检脚本 0 退出码,消除因假阳性报错引发的告警轰炸。"
|
||||
- metric: "RFC 规范对齐"
|
||||
detail: "严格对齐 Apple 邮件网关所强制遵循的 RFC 6409 / RFC 3207 邮件提交传输协议。"
|
||||
stack: ["Himalaya CLI", "SMTP / STARTTLS", "iCloud Mail", "Python", "Cron"]
|
||||
jumpLinks:
|
||||
business: "#the-alert-that-broke-the-silence"
|
||||
architecture: "#implicit-tls-versus-starttls"
|
||||
result: "#the-result"
|
||||
dynamicCta: true
|
||||
---
|
||||
|
||||
## 刺破宁静的自动化告警 <a id="the-alert-that-broke-the-silence"></a>
|
||||
|
||||
在运行于 Windows 管理宿主机的后台邮件健康巡检任务中,监控频道突然弹出了告警通知:
|
||||
|
||||
```text
|
||||
Cronjob Response: himalaya account health (job_id: 2f09440986e4)
|
||||
-------------
|
||||
📧 himalaya 账号健康检查 — 存在异常
|
||||
|
||||
❌ icloud (smtp)
|
||||
smtp: FAIL (connect smtp.mail.me.com:465: A connection attempt failed
|
||||
because the connected party did not properly respond after a period of time,
|
||||
or established connection failed because connected host has failed to respond. (os error 10060 / WSAETIMEDOUT))
|
||||
```
|
||||
|
||||
经过测试,使用跨平台 CLI 客户端通过 IMAP 协议(`imap.mail.me.com:993`)拉取收件箱邮件毫秒级响应,App-Specific Password(应用专用密码)也已反复确认有效。但只要尝试通过 SMTP 发送测试信件,底层套接字就会死等 30 秒,最终抛出 Windows Winsock 体系特有的超时异常(`os error 10060 / WSAETIMEDOUT`,对应 Linux 系统的 `ETIMEDOUT 110`)。
|
||||
|
||||
## 隐式 TLS 与 STARTTLS 的协议分水岭 <a id="implicit-tls-versus-starttls"></a>
|
||||
|
||||
排查的突破点在于发信端口规范的细微差别。在客户端配置(`~/.config/himalaya/config.toml`)中,此前参考了 Gmail 与通用自建企业邮的习惯进行填写:
|
||||
|
||||
```toml
|
||||
# 初始有问题的配置
|
||||
[accounts.icloud]
|
||||
email = "[email protected]"
|
||||
backend = "imap"
|
||||
imap.server = "imap.mail.me.com"
|
||||
imap.port = 993
|
||||
imap.encryption = "tls"
|
||||
|
||||
message.send.backend = "smtp"
|
||||
smtp.server = "smtps://smtp.mail.me.com:465"
|
||||
```
|
||||
|
||||
465 端口历史上被用于 SMTPS(隐式 TLS 直连,即 TCP 建立后直接开始 TLS 握手)。虽然许多主流邮件服务商为了向下兼容同时开放 465 与 587 端口,但 Apple 的发信基础设施(`smtp.mail.me.com`)严格执行传统 RFC 6409 / RFC 3207 邮件提交规范,而未开放 RFC 8314 所推荐的 465 隐式 TLS 端口:
|
||||
|
||||
1. **465 端口在 Apple 边缘防火墙上被直接静默丢包**。发往 465 端口的 SYN 报文收不到任何回应,因而客户端表现为漫长的 `WSAETIMEDOUT`,而不是快速返回拒绝(`ECONNREFUSED`)。
|
||||
2. Apple 的邮件网关**仅在 587 端口**提供发信提交服务,并且要求客户端以明文建立 TCP 连接后,立即发起 `STARTTLS` 扩展指令升级为加密通道。
|
||||
|
||||
使用 `smtps://` 直连 465 端口,无异于在一个根本没有服务监听的端口上等待 TLS Client Hello 应答。
|
||||
|
||||
## 修复:切换为 587 端口并启用 STARTTLS <a id="the-fix"></a>
|
||||
|
||||
在配置文件中调整协议 Scheme 与端口定义:
|
||||
|
||||
<details class="tech-deep-dive">
|
||||
<summary>🔍 config.toml — 规范后的 iCloud 账户配置(11 行)</summary>
|
||||
|
||||
```toml
|
||||
[accounts.icloud]
|
||||
email = "[email protected]"
|
||||
backend = "imap"
|
||||
imap.server = "imap.mail.me.com"
|
||||
imap.port = 993
|
||||
imap.encryption = "tls"
|
||||
|
||||
message.send.backend = "smtp"
|
||||
smtp.server = "smtp://smtp.mail.me.com:587"
|
||||
smtp.starttls = true
|
||||
```
|
||||
</details>
|
||||
|
||||
重点是将连接前缀从 `smtps://` 改回标准的 `smtp://`,同时声明 `smtp.starttls = true`。
|
||||
|
||||
## 结果 <a id="the-result"></a>
|
||||
|
||||
配置保存后,使用 CLI 单独测试 iCloud 账户连通性:
|
||||
|
||||
```text
|
||||
$ himalaya account check --account icloud
|
||||
Account: icloud
|
||||
imap: OK
|
||||
smtp: OK
|
||||
```
|
||||
|
||||
随后重新触发全量巡检脚本,全站配置的 6 个邮箱账号全部以绿色 OK 顺利通过,脚本正常以退出码 `0` 结束,巡检监控重新回归无异常推送的静默健康状态。
|
||||
@@ -0,0 +1,120 @@
|
||||
---
|
||||
ogImage: "/og/zh-authentik-oidc-redirect-loop-node-set-header-overwrite.jpg"
|
||||
banner: "/banners/authentik-oidc-redirect-loop-node-set-header-overwrite.jpg"
|
||||
title: "Authentik OIDC 重定向死循环:Node.js res.setHeader 整头覆盖机制与会话丢失排障"
|
||||
description: "自托管管理面板在 Authentik OIDC 授权成功后陷入无限重定向死循环,服务端日志零报错。排查发现根因是原生 Node.js 的 res.setHeader 默认替换而非追加 Set-Cookie 响应头。"
|
||||
pubDate: 2025-02-14
|
||||
updatedDate: 2026-11-01
|
||||
category: engineering
|
||||
blueprint: self-hosted-infrastructure
|
||||
blueprintLayer: L3-application-runtime
|
||||
tags: ["authentik", "oidc", "nodejs", "cookies", "身份认证", "docker"]
|
||||
draft: false
|
||||
summary:
|
||||
audience: "全栈开发工程师、身份认证架构师与 Node.js 服务维护者"
|
||||
painPoint: "用户在 Authentik 单点登录授权成功后,在回调页与受保护面板之间陷入死循环,容器日志全是 302 重定向与成功的 Token 交换,没有任何错误提示。"
|
||||
impact:
|
||||
- metric: "根除死循环重定向"
|
||||
detail: "将裸调 res.setHeader 改为数组累加机制,彻底解决 OIDC 回调时会话 Cookie 被静默清空的问题。"
|
||||
- metric: "自动化端到端验证"
|
||||
detail: "使用 Playwright 真实无头浏览器验证完整登录链路,确认 Cookie 正确保留且后台 21 行数据正常渲染。"
|
||||
- metric: "零性能开销"
|
||||
detail: "仅用十余行原生防御性代码修复状态机,无需引入额外会话存储或第三方框架开销。"
|
||||
stack: ["Node.js", "Authentik", "OIDC", "Playwright", "Docker"]
|
||||
jumpLinks:
|
||||
business: "#the-infinite-login-loop-with-clean-logs"
|
||||
architecture: "#root-cause-how-res-set-header-erased-sessions"
|
||||
result: "#the-result"
|
||||
dynamicCta: true
|
||||
---
|
||||
|
||||
## 全是成功日志的重定向死循环 <a id="the-infinite-login-loop-with-clean-logs"></a>
|
||||
|
||||
这是一起表现极为诡异的静默故障。管理员访问受保护的 `/admin` 后台,系统按预期重定向到 Authentik 统一认证网关。在完成二次验证并点击授权后,页面跳回应用的 OIDC 回调地址。
|
||||
|
||||
然而浏览器并没有进入后台,而是瞬间再次被踢回 Authentik。Authentik 检测到用户已经持有有效会话,立刻签发新的授权码并再次跳回回调页。这个过程周而复始,直到浏览器触发死循环熔断并报错 `ERR_TOO_MANY_REDIRECTS`。
|
||||
|
||||
查看应用容器的服务端日志,里面一片祥和,甚至没有任何警告信息:
|
||||
|
||||
```text
|
||||
127.0.0.1 - - [14/Feb/2025:08:12:04 +0000] "GET /admin HTTP/1.1" 302 -
|
||||
127.0.0.1 - - [14/Feb/2025:08:12:05 +0000] "GET /callback?code=... HTTP/1.1" 302 -
|
||||
127.0.0.1 - - [14/Feb/2025:08:12:05 +0000] "GET /admin HTTP/1.1" 302 -
|
||||
127.0.0.1 - - [14/Feb/2025:08:12:06 +0000] "GET /callback?code=... HTTP/1.1" 302 -
|
||||
```
|
||||
|
||||
每一个跳转都是合法的 HTTP 302。Authentik 那边记录的 Token 交换全部成功。OIDC 的 JWT 签名、过期时间均完全合规。但下游应用就是顽固地认为“当前请求未登录”。
|
||||
|
||||
## 根因:res.setHeader 是如何抹掉会话的 <a id="root-cause-how-res-set-header-erased-sessions"></a>
|
||||
|
||||
该服务采用原生 Node.js 构建,没有包裹沉重的全栈框架。在处理 OIDC 回调握手时,服务端需要先后完成两项 Cookie 操作:
|
||||
|
||||
1. 写入刚生成的长期业务登录凭证(`app_sess=<token>; HttpOnly; Secure; SameSite=Lax`)。
|
||||
2. 将防重放的临时 OIDC 校验 Cookie 清空作废(`app_oidc=; Max-Age=0`)。
|
||||
|
||||
初始的辅助写入函数实现非常直接:
|
||||
|
||||
```javascript
|
||||
function setCookie(res, cookieStr) {
|
||||
res.setHeader('Set-Cookie', cookieStr);
|
||||
}
|
||||
```
|
||||
|
||||
在 Node.js 原生 `http` 模块(`OutgoingMessage`)的设计中,如果给 `setHeader(name, value)` 传入字符串参数,它的行为是**整头覆盖**,而不是追加。
|
||||
|
||||
在回调控制器里,代码先后执行了两次调用:
|
||||
|
||||
```javascript
|
||||
// 第一步:写入登录会话
|
||||
setCookie(res, `app_sess=${sessionToken}; Path=/; HttpOnly; Secure; SameSite=Lax`);
|
||||
|
||||
// 第二步:清理临时状态
|
||||
setCookie(res, `app_oidc=; Path=/; Max-Age=0; HttpOnly; Secure; SameSite=Lax`);
|
||||
```
|
||||
|
||||
第二次调用执行的瞬间,第一步刚写入的 `app_sess` 被整个抹掉。
|
||||
|
||||
使用 `curl -v` 抓取回调的原始响应头,真相一目了然:
|
||||
|
||||
```text
|
||||
< HTTP/1.1 302 Found
|
||||
< Location: /admin
|
||||
< Set-Cookie: app_oidc=; Path=/; Max-Age=0; HttpOnly; Secure; SameSite=Lax
|
||||
```
|
||||
|
||||
浏览器从未收到过 `app_sess`。在跟随 302 跳转到 `/admin` 时,由于请求中缺乏凭据,路由门禁再次拦截,并重新将请求丢回 Authentik。由于链路上每个组件都在忠实执行自身逻辑,整条故障链路没有产生任何一行 stderr 报错。
|
||||
|
||||
## 修复:基于数组的多响应头合并 <a id="the-fix"></a>
|
||||
|
||||
修复的核心在于:任何向 `Set-Cookie` 追加内容的操作,必须先取出已存在的字段并转为数组保存。
|
||||
|
||||
<details class="tech-deep-dive">
|
||||
<summary>🔍 server.mjs — setCookie 修复实现(12 行)</summary>
|
||||
|
||||
```javascript
|
||||
function setCookie(res, cookieStr) {
|
||||
const existing = res.getHeader('Set-Cookie');
|
||||
let cookies = [];
|
||||
if (Array.isArray(existing)) {
|
||||
cookies = existing.concat(cookieStr);
|
||||
} else if (typeof existing === 'string') {
|
||||
cookies = [existing, cookieStr];
|
||||
} else {
|
||||
cookies = [cookieStr];
|
||||
}
|
||||
res.setHeader('Set-Cookie', cookies);
|
||||
}
|
||||
```
|
||||
</details>
|
||||
|
||||
当传入值为字符串数组时,Node 的 HTTP 序列化器会自动将其拆解为多条独立的 `Set-Cookie:` 标头吐给客户端。
|
||||
|
||||
## 结果 <a id="the-result"></a>
|
||||
|
||||
代码更新后,编写了专用的 Playwright 自动化探测脚本进行端到端全链路验收:
|
||||
- 请求 `/admin` 正确触发 302 并转跳 Authentik。
|
||||
- 自动完成 SSO 表单提交与授权确认。
|
||||
- 回调响应头中完整包含 `app_sess` 与过期的 `app_oidc`。
|
||||
- 最终落地页以 HTTP 200 打开,管理后台的 21 行数据表格全部渲染成功。
|
||||
|
||||
无需修改上游身份提供商的任何参数,单靠规范底层 HTTP 头输出,就彻底斩断了重定向死循环。
|
||||
@@ -0,0 +1,111 @@
|
||||
---
|
||||
ogImage: "/og/zh-cascading-docker-failures-nocodb-heap-limit-starved-n8n.jpg"
|
||||
banner: "/banners/cascading-docker-failures-nocodb-heap-limit-starved-n8n.jpg"
|
||||
title: "Docker 级联故障复盘:NocoDB 内存溢出如何拖垮整套 n8n 自动化栈"
|
||||
description: "容器配置的 768 MB 限制导致 NocoDB 频繁触发 JavaScript 堆内存溢出崩溃,进而引发依赖该数据源的大量 n8n 工作流报错 ECONNREFUSED。通过对齐 max-old-space-size 与清理孤儿任务彻底根治。"
|
||||
pubDate: 2025-04-12
|
||||
updatedDate: 2026-11-01
|
||||
category: devops
|
||||
blueprint: self-hosted-infrastructure
|
||||
blueprintLayer: L3-application-runtime
|
||||
tags: ["n8n", "nocodb", "docker", "nodejs", "内存排障", "自动化", "排错复盘"]
|
||||
draft: false
|
||||
summary:
|
||||
audience: "DevOps 运维工程师、系统集成架构师与自动化工作流维护者"
|
||||
painPoint: "多条关键业务的 n8n 自动化流同时报错 ECONNREFUSED 10380,错误日志塞满面板,部分长时间运行的任务永久挂在 running 状态无法退出。"
|
||||
impact:
|
||||
- metric: "根除上游连接中断"
|
||||
detail: "将 NocoDB 容器内存配额提升至 2048 MB,并注入 max-old-space-size=1536,彻底消除内存崩溃。"
|
||||
- metric: "100% 工作流恢复健康"
|
||||
detail: "清理编排数据库中的悬挂孤儿记录,全量恢复 Webhook 接收与数据入库,零数据丢失。"
|
||||
- metric: "保护 SATA 阵列 I/O"
|
||||
detail: "阻断下游重试风暴引发的磁盘恶性竞争,避免存储控制器被雪崩式重试打死。"
|
||||
stack: ["n8n", "NocoDB", "Docker Compose", "Node.js", "PostgreSQL", "Portainer"]
|
||||
jumpLinks:
|
||||
business: "#the-morning-the-automation-board-turned-red"
|
||||
architecture: "#root-cause-the-default-v8-heap-ceiling"
|
||||
result: "#the-result"
|
||||
dynamicCta: true
|
||||
---
|
||||
|
||||
## 自动化面板一片飘红的早晨 <a id="the-morning-the-automation-board-turned-red"></a>
|
||||
|
||||
在对自托管生产节点进行早间巡检时,n8n 自动化中枢的失败计数器突然出现大面积暴涨。控制台日志里铺天盖地全是网络拒绝错误:
|
||||
|
||||
```text
|
||||
NodeApiError: connect ECONNREFUSED 192.168.1.1:10380
|
||||
at Object.request (/usr/local/lib/node_modules/n8n/node_modules/axios/lib/adapters/http.js:64:14)
|
||||
at processTicksAndRejections (node:internal/process/task_queues:95:5)
|
||||
```
|
||||
|
||||
所有负责将外部消息通知、客户线索表单同步至本地中心库的工作流全面停摆。更严重的是,几个定时批处理任务永远停留在 `running` 状态,占用着后台线程却没有任何数据产出。
|
||||
|
||||
使用探针检查下游目标服务——运行在 `10380` 端口的 NocoDB 容器,发现它根本没有提供服务,正深陷于反复重启的死循环中。
|
||||
|
||||
## 根因:V8 默认堆上限与容器限额的错配 <a id="root-cause-the-default-v8-heap-ceiling"></a>
|
||||
|
||||
通过 Portainer 调出该容器的崩溃日志,终端赫然打印着致命的 Node.js 内存越界报错:
|
||||
|
||||
```text
|
||||
<--- Last few GCs --->
|
||||
[1:0x55d7b8a8b000] 184201 ms: Mark-sweep (reduce) 712.4 (724.8) -> 710.2 (725.1) MB, 412.5 / 0.0 ms
|
||||
[1:0x55d7b8a8b000] 184712 ms: Mark-sweep (reduce) 715.1 (725.1) -> 714.8 (726.0) MB, 511.0 / 0.0 ms
|
||||
|
||||
<--- JS stacktrace --->
|
||||
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
|
||||
1: 0xb83a30 node::Abort() [node]
|
||||
2: 0xa94415 node::FatalError(char const*, char const*) [node]
|
||||
3: 0xd45062 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
|
||||
```
|
||||
|
||||
在原先的编排配置中,为了防止单个轻量服务抢占过多物理宿主机资源,给容器加上了硬性配额:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
nocodb:
|
||||
image: nocodb/nocodb:latest
|
||||
mem_limit: 768m
|
||||
```
|
||||
|
||||
在 768 MB 的严苛容器边界下,V8 默认的堆内存启发式阈值无法应对突发的多表聚合峰值。当外部多表联查与复杂视图聚合触发、数据量瞬间拉升时,进程内存占用逼近 700 MB。V8 垃圾回收器高频发起全量 Mark-sweep,在无法回收足够连续物理内存的情况下直接触发致命退出。
|
||||
|
||||
NocoDB 的崩溃直接引发了下游级联效应:数十个 n8n 节点在遇到拒绝连接时触发了配置的重试策略。数百次重试请求在容器重启拉起的瞬间疯狂涌入,短时间内再次耗尽内存,造成了持续数十次的雪崩式死循环。
|
||||
|
||||
## 修复:内存分层对齐与孤儿记录清理 <a id="the-fix"></a>
|
||||
|
||||
彻底修复这一级联隐患需要同时解决运行时内存边界与业务积压:
|
||||
|
||||
<details class="tech-deep-dive">
|
||||
<summary>🔍 docker-compose.yml — NocoDB 内存精细化调优配置(17 行)</summary>
|
||||
|
||||
```yaml
|
||||
services:
|
||||
nocodb:
|
||||
image: nocodb/nocodb:latest
|
||||
container_name: nocodb-server
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- NC_DB=pg://postgres:5432?u=...
|
||||
- NODE_OPTIONS=--max-old-space-size=1536
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 2048M
|
||||
reservations:
|
||||
memory: 512M
|
||||
ports:
|
||||
- "10380:8080"
|
||||
```
|
||||
</details>
|
||||
|
||||
将容器的硬限制放宽至 2048 MB,并向运行时注入环境变量 `NODE_OPTIONS: "--max-old-space-size=1536"`。这为 V8 用户态堆空间分配了完整的 1.5 GB,同时给底层操作系统与原生网络 I/O 预留出充足的 512 MB 缓冲区。
|
||||
|
||||
随后,连接至 n8n 内部元数据库,将故障期间因连接中断而处于挂起锁死状态的历史孤儿执行记录批量标记为已终结,释放执行上下文。
|
||||
|
||||
## 结果 <a id="the-result"></a>
|
||||
|
||||
配置应用并重新启动栈服务后:
|
||||
- NocoDB 容器恢复平稳运行,健康探测接口 `/api/v1/health` 持续稳定返回 `{"message":"OK"}`。
|
||||
- Webhook 链路与自动化数据管线全部顺畅通过,未再出现单次丢包或连接拒绝。
|
||||
- 容器内存平稳维持在 400~600 MB 健康区间,再未触碰崩溃阈值。
|
||||
- n8n 控制台失败计数器彻底归零,经受住了后续全量批处理的检验。
|
||||
@@ -0,0 +1,83 @@
|
||||
---
|
||||
ogImage: "/og/zh-home-assistant-container-upgrade-ladder-four-hops.jpg"
|
||||
banner: "/banners/home-assistant-container-upgrade-ladder-four-hops.jpg"
|
||||
title: "Home Assistant 容器版跨版本升级:四阶安全平跳十三期迭代"
|
||||
description: "把容器化部署的 Home Assistant 从 2025.9 一跃升级到 2026.10 往往会导致 SQLite 数据库损坏与反代配置失效。剖析如何通过四阶梯级跳方案安全保留全部 259 个实体与配置。"
|
||||
pubDate: 2026-07-22
|
||||
updatedDate: 2026-11-01
|
||||
category: devops
|
||||
blueprint: self-hosted-infrastructure
|
||||
blueprintLayer: L3-application-runtime
|
||||
tags: ["物联网监控", "docker", "系统升级", "sqlite", "traefik", "反向代理"]
|
||||
draft: false
|
||||
summary:
|
||||
audience: "物联网(IoT)系统工程师、私有云架构师与容器化基础设施维护者"
|
||||
painPoint: "将落后 13 个发布周期的 Home Assistant 容器一次性盲跳升级到最新版,面临 SQLite 数据库死锁损坏、Python 3.14 语法变动以及反代门禁配置静默失效的巨大风险。"
|
||||
impact:
|
||||
- metric: "259 → 265 实体完整无损"
|
||||
detail: "在每个阶段前后对比实体注册表,确认零设备掉线、零自动化失效。"
|
||||
- metric: "零数据库模式损坏"
|
||||
detail: "平滑完成 SQLite 记录器从 Schema 50 至 53 的单向迁移,WAL 日志干净收敛。"
|
||||
- metric: "反代访问无缝保留"
|
||||
detail: "隔离 reverse-proxy trusted_proxies 与 use_x_forwarded_for 验证校验并平稳过渡 .storage/http 状态,未发生 HTTP 400 阻断。"
|
||||
stack: ["Home Assistant Core", "Docker Compose", "SQLite", "Traefik", "Python 3.14"]
|
||||
jumpLinks:
|
||||
business: "#the-thirteen-version-maintenance-debt"
|
||||
architecture: "#isolating-hazards-across-four-hops"
|
||||
result: "#the-result"
|
||||
dynamicCta: true
|
||||
---
|
||||
|
||||
## 跨越十三期的技术债务 <a id="the-thirteen-version-maintenance-debt"></a>
|
||||
|
||||
在使用独立 Docker 容器(`homeassistant/home-assistant`)运行 Home Assistant 作为机房环境动力监控、UPS 电力遥测与自动化告警中枢时,系统更新无法依靠面板里的一键按钮完成。每一次升级,都需要运维人员手动更新镜像标签、重建容器并监控首次启动时的数据库迁移流程。
|
||||
|
||||
在实际生产环境中,一套长期稳定运行的系统停留在 **2025.9.4** 版本长达十三个月,而官方最新版已经推进至 **2026.10.4**。面对整整 13 个大版本的跨度,如果直接拉取最新镜像盲跳启动,极易引发严重故障:
|
||||
|
||||
1. **单向不可逆的数据库迁移**:系统自带的 Recorder 历史记录组件在这期间完成了 SQLite Schema 50 至 53 的重大架构变更。一旦跨版本迁移途中因超时断电中断,数据库将陷入不可读死锁。
|
||||
2. **底层 Python 运行时更迭**:基础镜像由 Python 3.13 升级到 3.14,引入了更严苛的异步语法限制,旧组件中的废弃 API 会导致配置加载直接报红。
|
||||
3. **2026.8 反向代理配置大变动**:官方在 2026.8 重构了网络安全子系统,将反代信任参数纳入 `.storage/http` 内部存储中。若迁移不当,反向代理接入将全量报 `HTTP 400: Bad Request`。
|
||||
|
||||
为了彻底消除隐患,必须将漫长的跨度拆解为四级平稳阶梯。
|
||||
|
||||
## 隔离风险的四阶升级阶梯 <a id="isolating-hazards-across-four-hops"></a>
|
||||
|
||||
官方文档通常建议单次跨越不超过 5 至 6 个小版本。经过研判,确立了如下四个独立验收的升级阶段:
|
||||
|
||||
| 阶梯 | 目标版本镜像 | 重点隔离的架构风险 | 验收门禁 |
|
||||
|---|---|---|---|
|
||||
| **1** | `2025.12.5` | 隔离 SQLite recorder 模式升级(v50 → v53) | 数据库模式就绪,WAL 归一无损 |
|
||||
| **2** | `2026.3.4` | 隔离 Python 3.13 → 3.14 运行时与 zstd 镜像层 | 查看容器日志:0 个 Setup failed |
|
||||
| **3** | `2026.8.3` | 隔离 `http:` trusted_proxies 校验与 `.storage/http` 配置迁移 | 反向代理标头验证通过,返回 200 |
|
||||
| **4** | `2026.10.4` | 落地当前最新稳定版,完成旧版认证组件清理 | 基线 259 个实体与传感器全部在线 |
|
||||
|
||||
在启动升级前,先停止正在运行的容器,对挂载的 `/config` 目录进行冷备份,确保持久化数据库和 WAL 文件绝对一致。
|
||||
|
||||
<details class="tech-deep-dive">
|
||||
<summary>🔍 docker-compose.yml — Home Assistant 容器生产配置(14 行)</summary>
|
||||
|
||||
```yaml
|
||||
services:
|
||||
homeassistant:
|
||||
container_name: home-assistant
|
||||
image: homeassistant/home-assistant:2026.10.4
|
||||
volumes:
|
||||
- /volume1/docker/home-assistant/config:/config
|
||||
- /etc/localtime:/etc/localtime:ro
|
||||
environment:
|
||||
- TZ=Asia/Kuala_Lumpur
|
||||
network_mode: host
|
||||
restart: unless-stopped
|
||||
stop_grace_period: 60s
|
||||
```
|
||||
</details>
|
||||
|
||||
特别注意 `stop_grace_period: 60s` 的设定:当 SQLite 数据表较大时,Home Assistant 在收到退出信号后需要数十秒将内存缓冲区与 WAL 日志写回磁盘。Docker 默认的 10 秒超时往往会触发内核 SIGKILL 强制杀进程(Exit Code 137),造成数据库锁损坏。
|
||||
|
||||
## 结果 <a id="the-result"></a>
|
||||
|
||||
四阶升级全部顺利执行完毕,无一例回滚:
|
||||
|
||||
- **实体完整性**:实体总数由基线的 **259 个无缝扩展至 265 个**(新增 6 个为官方新版内置的电能监控统计),零配置报错,零传感器掉线。
|
||||
- **记录器健康**:数据库无缝完成迁移,耗时不到 1 秒,历史数据行数从 249,875 行平稳增长至 249,897 行。
|
||||
- **反代网关稳固**:上游反代网关继续维持单点登录保护,`X-Forwarded-For` 传递被系统精准信任,外部访问完全恢复顺畅。
|
||||
@@ -0,0 +1,99 @@
|
||||
---
|
||||
ogImage: "/og/zh-six-chrome-containers-locked-my-nas-btrfs-d-state-igpu-offload.jpg"
|
||||
banner: "/banners/six-chrome-containers-locked-my-nas-btrfs-d-state-igpu-offload.jpg"
|
||||
title: "六个 Chrome 容器打崩存储 NAS:CPU 软解 WebGL、Btrfs D 状态死锁与核显架构迁移"
|
||||
description: "在无核显的 4 核嵌入式 NAS 上跑 6 个 Chrome 桌面容器,软解 WebGL 将系统负载推过 27,并引发 Btrfs 事务陷入 D 状态死锁。将浏览器集群迁移至 Intel 核显节点后,释放了 7.5 GB 提交内存,负载回落至 6.8。"
|
||||
pubDate: 2025-03-20
|
||||
updatedDate: 2026-11-01
|
||||
category: devops
|
||||
blueprint: self-hosted-infrastructure
|
||||
blueprintLayer: L1-host-storage-compute
|
||||
tags: ["docker", "synology", "btrfs", "gpu", "chrome", "unraid", "cdp"]
|
||||
draft: false
|
||||
summary:
|
||||
audience: "私有云架构师、SRE 与运行高密度容器集群的基础设施工程师"
|
||||
painPoint: "在缺乏核显加速的 4 核嵌入式 NAS 上同时运行 6 个 Chrome 自动化容器,CPU 软解渲染 WebGL 打满整机算力,引发频繁 Swap 换页并最终导致底层 Btrfs 文件系统陷入内核 D 状态死锁。"
|
||||
impact:
|
||||
- metric: "释放 7.5 GB 提交内存空间"
|
||||
detail: "彻底剥离常驻存储设备的浏览器实例,释放 7.5 GB 提交内存(含物理内存与 Swap 交换区),根除颠簸。"
|
||||
- metric: "系统负载 27.4 → 6.8"
|
||||
detail: "解除 CPU 软解算力竞争与 Docker 守护进程移除锁,彻底消除反向代理网关 502 与上游超时。"
|
||||
- metric: "按需零驻留架构"
|
||||
detail: "在带 Intel UHD 770 核显的计算宿主上重构工作流,闲置时保持停止,达成 0% CPU 与 0 MB 资源占用。"
|
||||
stack: ["Synology DSM", "Btrfs", "Docker", "Intel QuickSync / DRI", "unRaid", "CDP"]
|
||||
jumpLinks:
|
||||
business: "#the-storage-appliance-that-stopped-responding"
|
||||
architecture: "#anatomy-of-a-btrfs-d-state-freeze"
|
||||
result: "#the-result"
|
||||
dynamicCta: true
|
||||
---
|
||||
|
||||
## 失去响应的集中存储设备 <a id="the-storage-appliance-that-stopped-responding"></a>
|
||||
|
||||
故障最初表现为网络边缘服务的集体超时:部署在一台 DS1821+ NAS 上的管理后台与私有网盘页面打不开。外部 HTTPS 握手瞬间完成、证书完全匹配,但一旦反代向后端发起请求,连接便无休止地转圈挂起。
|
||||
|
||||
通过 SSH 登录宿主机终端,看到的资源消耗状况令人震惊:
|
||||
|
||||
```text
|
||||
# uptime
|
||||
09:42:15 up 42 days, 18:22, 1 user, load average: 27.42, 25.18, 22.04
|
||||
|
||||
# free -m
|
||||
total used free shared buff/cache available
|
||||
Mem: 7842 7410 112 340 320 180
|
||||
Swap: 4096 3980 116
|
||||
```
|
||||
|
||||
对于一台搭载 AMD Ryzen V1500B(4 核 4 线程)嵌入式处理器的机器来说,超过 27 的 Load Average 意味着有二十多个进程在排队抢夺算力,或者卡在不可中断的内核休眠中。物理内存占用高达 94%,4GB 的 Swap 分区几乎彻底告罄。
|
||||
|
||||
资源消耗大户是 6 个常驻的桌面版 Chrome 容器(`lscr.io/linuxserver/chrome`),用于跑生图预览、自动化审计与网页渲染。
|
||||
|
||||
## Btrfs 内核 D 状态死锁剖析 <a id="anatomy-of-a-btrfs-d-state-freeze"></a>
|
||||
|
||||
这台 NAS 芯片没有集成核显。现代 Web 应用——尤其是富媒体界面、交互看板与视频画布——高度依赖 WebGL 渲染管线。在没有透传图形加速设备(`/dev/dri`)的环境下,Chromium 会静默降级为 CPU 软件光栅化(SwiftShader 或 llvmpipe)。
|
||||
|
||||
这种硬件特质与软件需求的错配,引发了三级连环雪崩:
|
||||
|
||||
1. **算力被软解耗尽**:6 个浏览器容器同时用 CPU 计算像素,把 4 颗物理核心全部顶死在 100%。
|
||||
2. **内存溢出与磁盘颠簸**:随着渲染进程申请图形缓冲区,系统内存见底,内核被迫高频将内存页换出到机械阵列(`/volume1`)的 Swap 分区,引入天量 I/O Wait。
|
||||
3. **存储事务挂起**:当容器守护进程尝试删除一个废弃的临时探针容器时,触发了 `btrfs subvolume delete`。在磁盘队列彻底堵死的极端压力下,Btrfs 的全局删除事务陷入了内核不可中断睡眠(D-state)。
|
||||
|
||||
由于 Btrfs 全局锁未能释放,宿主机上依赖文件写入的其它核心进程(包括本地元数据库、日志记录器与反代工作线程)在发起同步系统调用时全部被锁死在等待队列中,整机因而瘫痪。
|
||||
|
||||
## 架构迁移:将负载剥离至 Intel 核显计算节点 <a id="the-migration"></a>
|
||||
|
||||
彻底的解法是将“存储节点”与“图形化自动化算力”在物理层面上做隔离。
|
||||
|
||||
浏览器集群被全量迁移至配备 Intel Core i9-13900K 且具备 Intel UHD Graphics 770 的专用 unRaid 计算节点上。
|
||||
|
||||
<details class="tech-deep-dive">
|
||||
<summary>🔍 docker-compose.yml — 核显硬件加速 Chrome 容器配置(17 行)</summary>
|
||||
|
||||
```yaml
|
||||
services:
|
||||
chrome-worker:
|
||||
image: lscr.io/linuxserver/chrome:latest
|
||||
container_name: chrome-worker
|
||||
devices:
|
||||
- /dev/dri/renderD128:/dev/dri/renderD128
|
||||
- /dev/dri/card0:/dev/dri/card0
|
||||
environment:
|
||||
- CHROME_CLI=--remote-debugging-port=9222 --user-data-dir=/config/profile --enable-features=VaapiVideoDecodeLinuxGL --ignore-gpu-blocklist --use-gl=angle --use-angle=gl
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 3072M
|
||||
restart: "no"
|
||||
```
|
||||
</details>
|
||||
|
||||
透传 `/dev/dri/renderD128` 让渲染和硬件解码直接跑在处理器的核显执行单元上,彻底解放了 CPU。此外,容器策略从 `restart: unless-stopped` 改为**按需启停模型**:在未执行任务时保持 Stopped 状态,实现 0% CPU 与 0 MB 内存的真正零驻留。
|
||||
|
||||
## 结果 <a id="the-result"></a>
|
||||
|
||||
架构调整后,存储宿主机立刻恢复了健康弹性:
|
||||
|
||||
- 停用 NAS 上的软解容器瞬间回收了 **7.5 GB 提交内存空间(含物理内存与 Swap)**,彻底消除了 Swap 换页抖动。
|
||||
- 宿主机 Load Average 在数分钟内由 **27.42 暴跌至 6.80**。
|
||||
- 堆积的 Btrfs 事务顺利提交完成,内部驱动服务与网关代理的响应耗时回归毫秒级。
|
||||
- 在 Intel 硬件加速节点上,依赖图形界面的自动化脚本执行耗时平均下降了 60% 以上。
|
||||