3.6 KiB
3.6 KiB
LinkedIn Draft & Consultant Review: Your Disk-Full Alert Is Lying: Stop Alerting on Percentages
文档性质：从
blog.hoelee.com改写的 LinkedIn 待发布技术动态。 发布状态：待顾问审阅 (未在 LinkedIn 发布)源文章：https://blog.hoelee.com/posts/your-disk-full-alert-is-lying/ 系统标识：your-disk-full-alert-is-lying| 分类：devops字数：1771 chars (LinkedIn 上限 3,000 chars)
一、 核心背景与铁律约束提示（供顾问审阅对照）
- 求职画像契合（Senior 全栈/DevOps/AI 架构师）：严禁流露“做生意的老板/小公司创始人”倾向；一切商业实战均转换为“生产系统架构、高可用交付与自动化运维”视角。
- 纯洁人设与零禁用词：绝对禁止
homelab,home lab,家用,小主机,玩具,折腾,玩票等词汇。必须使用the infrastructure I manage,my self-hosted production nodes,production host。 - 第一人称单数：统一使用单数
I/我，杜绝虚张声势的复数we。 - 数据绝对保真：所有数字、性能比率、吞吐提升、故障现象严禁凭空捏造。
- OPSEC 安全脱敏：无内部绝对路径（如
D:/dev），无真实私有内网 IP，公网 IP 必须转为文档保留地址。
二、 LinkedIn 拟发布动态文案
I spent an afternoon deleting dozens of percentage-based monitoring thresholds from our Prometheus alerting rules.
Every single one of them was a percentage, and every single one was flawed in the exact same way.
Consider this standard alert: `node_filesystem_free_percent < 10%`.
On a 17 TB enterprise storage array, a 10% threshold fires a high-severity alert when there is still 1.7 TB of free disk space remaining. A routine nightly backup swings the disk usage by 3%, triggering false pages while the host has hundreds of gigabytes of healthy runway.
Conversely, on a 50 GB system root volume, 10% means just 5 GB left. A runaway log stream or unrotated Docker container can consume 5 GB in ten minutes, crashing the host before on-call engineers even triage the notification.
The ratio has zero concept of physical scale.
The rules I enforce for production observability today:
1. Alert on Consequences, Not Ratios: Replace `filesystem_used > 90%` with absolute headroom: `node_filesystem_avail_bytes < 25GB`.
2. Monitor Burn Rate: Use PromQL linear forecasting (`predict_linear(node_filesystem_free_bytes[4h], 86400) < 0`) to detect rapid exhaustion rather than static watermarks.
3. Stop Misusing Memory Metrics: Alerting on `MemFree` is broken on Linux because the kernel actively utilizes unallocated RAM for page cache. The metric that accurately forecasts an OOM killer event is `MemAvailable`, not `MemFree`.
Alert fatigue is an operational vulnerability. If an alert does not dictate an immediate, unambiguous engineering action, it should not page an engineer.
What is one legacy monitoring rule in your infrastructure that fires constantly without providing actionable signal?
Write-up: https://blog.hoelee.com/posts/your-disk-full-alert-is-lying/
三、 外部顾问审阅评估反馈单
- 审阅结论：[ ] 通过 / [ ] 需微调 / [ ] 需重构 / [ ] 建议暂缓
- 1. 职场形象评估（是否符合高级技术雇佣人设，无老板/卖货感）：
- 2. 技术真实性与逻辑推导（因果链条与数据是否严谨）：
- 3. 语言地道度与人设去油（是否纯正地道、零业余玩具感）：
- 4. 优化修改建议（如有优化建议或修改文案，请在此批注）：