hoelee-blog/docs/linkedin/06-your-disk-full-alert-is-lying.md

LinkedIn Draft & Consultant Review: Your Disk-Full Alert Is Lying: Stop Alerting on Percentages

文档性质：从 blog.hoelee.com 改写并在 NocoDB 排期的 LinkedIn 生产级动态。 发布状态：已通过外部顾问审阅 / NocoDB 已排期待发布 排期波次：Wave 2 (纵深期) - 槽位 6 (拟发布日期: 2026-11-17 Tue) 源文章：https://blog.hoelee.com/posts/your-disk-full-alert-is-lying/ 系统标识：your-disk-full-alert-is-lying | 分类：devops 字数：1779 chars (严格控制在 3,000 字符内)

---

一、 核心背景与铁律约束提示（供复核对照）

1. 求职画像契合（Senior 全栈/DevOps/AI 架构师）：彻底脱敏“做生意的老板/小公司创始人”倾向；一切商业实战均转换为“生产系统架构、高可用交付与自动化运维”视角。
2. 纯洁人设与零禁用词：绝对禁止 homelab, home lab, 家用, 小主机, 玩具, 折腾, 玩票 等词汇。必须使用 the infrastructure I manage, my self-hosted production nodes, production host。
3. 第一人称单数：统一使用单数 I / 我，杜绝虚张声势的复数 we。
4. 数据绝对保真：所有数字、性能比率、吞吐提升、故障现象严禁凭空捏造。
5. OPSEC 安全脱敏：无内部绝对路径（如 D:/dev），无真实私有内网 IP，公网 IP 必须转为文档保留地址。

---

二、 LinkedIn 拟发布动态文案（已按顾问意见修订）

I spent an afternoon deleting dozens of percentage-based monitoring thresholds from my Prometheus alerting rules.

Every single one of them was a ratio, and every single one was flawed in the exact same way.

Consider a standard threshold: alerting when free disk space drops below 10%.
On a 17 TB enterprise storage array, a 10% threshold fires a high-severity alert when there is still 1.7 TB of free space remaining. A routine nightly backup swings disk usage by 3%, triggering false pages while the host has hundreds of gigabytes of healthy runway.

Conversely, on a 50 GB system root volume, 10% means just 5 GB left. A runaway log stream or unrotated Docker container can consume 5 GB in ten minutes, crashing the host before an engineer even triages the notification.

A percentage ratio has zero concept of physical scale.

The rules I enforce for production observability today:
1. Alert on Absolute Headroom, Not Ratios: Replace `filesystem_used > 90%` with explicit byte boundaries (`node_filesystem_avail_bytes < 25 * 1024^3`).
2. Monitor Burn Rate: Use PromQL linear forecasting (`predict_linear(node_filesystem_avail_bytes[4h], 86400) < 0`) to detect rapid exhaustion rather than static watermarks.
3. Stop Misusing Memory Metrics: Alerting on `MemFree` is broken on Linux because the kernel actively utilizes unallocated RAM for page cache. The metric that accurately forecasts an OOM killer event is `MemAvailable`, not `MemFree`.

Alert fatigue is an operational vulnerability. If an alert does not dictate an immediate, unambiguous engineering action, it should not fire.

What is one legacy monitoring rule in your infrastructure that fires constantly without providing actionable signal?

Write-up: https://blog.hoelee.com/posts/your-disk-full-alert-is-lying/

---

三、 外部顾问审阅评估反馈单与归档记录

• 审阅结论：需微调（已更新修订稿）
• 1. 诊断与修改批注：按顾问修订稿纠正第一人称复数 our -> my，统一 PromQL 指标为 avail_bytes，规范 PromQL 字节表达式。
• 2. NocoDB 归档记录：
  • Candidate Key: cand:your-disk-full-alert-is-lying (Status: APPROVED, Priority: HIGH)
  • Draft Key: cand:your-disk-full-alert-is-lying:v1 (Status: APPROVED)
  • 拟定发布节奏：每周 2 篇（周二/周四），平稳推进 5 周。