58 lines
3.8 KiB
Markdown
58 lines
3.8 KiB
Markdown
# LinkedIn Draft & Consultant Review: Your Disk-Full Alert Is Lying: Stop Alerting on Percentages
|
||
|
||
> **文档性质**：从 `blog.hoelee.com` 改写并在 NocoDB 排期的 LinkedIn 生产级动态。
|
||
> **发布状态**：`已通过外部顾问审阅 / NocoDB 已排期待发布`
|
||
> **排期波次**：Wave 2 (纵深期) - 槽位 6 (拟发布日期: 2026-11-17 Tue)
|
||
> **源文章**：https://blog.hoelee.com/posts/your-disk-full-alert-is-lying/
|
||
> **系统标识**：`your-disk-full-alert-is-lying` | **分类**：`devops`
|
||
> **字数**：1779 chars (严格控制在 3,000 字符内)
|
||
|
||
---
|
||
|
||
## 一、 核心背景与铁律约束提示（供复核对照）
|
||
|
||
1. **求职画像契合（Senior 全栈/DevOps/AI 架构师）**：彻底脱敏“做生意的老板/小公司创始人”倾向；一切商业实战均转换为“生产系统架构、高可用交付与自动化运维”视角。
|
||
2. **纯洁人设与零禁用词**：绝对禁止 `homelab`, `home lab`, `家用`, `小主机`, `玩具`, `折腾`, `玩票` 等词汇。必须使用 `the infrastructure I manage`, `my self-hosted production nodes`, `production host`。
|
||
3. **第一人称单数**：统一使用单数 `I` / `我`，杜绝虚张声势的复数 `we`。
|
||
4. **数据绝对保真**：所有数字、性能比率、吞吐提升、故障现象严禁凭空捏造。
|
||
5. **OPSEC 安全脱敏**：无内部绝对路径（如 `D:/dev`），无真实私有内网 IP，公网 IP 必须转为文档保留地址。
|
||
|
||
---
|
||
|
||
## 二、 LinkedIn 拟发布动态文案（已按顾问意见修订）
|
||
|
||
```text
|
||
I spent an afternoon deleting dozens of percentage-based monitoring thresholds from my Prometheus alerting rules.
|
||
|
||
Every single one of them was a ratio, and every single one was flawed in the exact same way.
|
||
|
||
Consider a standard threshold: alerting when free disk space drops below 10%.
|
||
On a 17 TB enterprise storage array, a 10% threshold fires a high-severity alert when there is still 1.7 TB of free space remaining. A routine nightly backup swings disk usage by 3%, triggering false pages while the host has hundreds of gigabytes of healthy runway.
|
||
|
||
Conversely, on a 50 GB system root volume, 10% means just 5 GB left. A runaway log stream or unrotated Docker container can consume 5 GB in ten minutes, crashing the host before an engineer even triages the notification.
|
||
|
||
A percentage ratio has zero concept of physical scale.
|
||
|
||
The rules I enforce for production observability today:
|
||
1. Alert on Absolute Headroom, Not Ratios: Replace `filesystem_used > 90%` with explicit byte boundaries (`node_filesystem_avail_bytes < 25 * 1024^3`).
|
||
2. Monitor Burn Rate: Use PromQL linear forecasting (`predict_linear(node_filesystem_avail_bytes[4h], 86400) < 0`) to detect rapid exhaustion rather than static watermarks.
|
||
3. Stop Misusing Memory Metrics: Alerting on `MemFree` is broken on Linux because the kernel actively utilizes unallocated RAM for page cache. The metric that accurately forecasts an OOM killer event is `MemAvailable`, not `MemFree`.
|
||
|
||
Alert fatigue is an operational vulnerability. If an alert does not dictate an immediate, unambiguous engineering action, it should not fire.
|
||
|
||
What is one legacy monitoring rule in your infrastructure that fires constantly without providing actionable signal?
|
||
|
||
Write-up: https://blog.hoelee.com/posts/your-disk-full-alert-is-lying/
|
||
```
|
||
|
||
---
|
||
|
||
## 三、 外部顾问审阅评估反馈单与归档记录
|
||
|
||
- [x] **审阅结论**：需微调（已更新修订稿）
|
||
- **1. 诊断与修改批注**：按顾问修订稿纠正第一人称复数 our -> my，统一 PromQL 指标为 avail_bytes，规范 PromQL 字节表达式。
|
||
- **2. NocoDB 归档记录**：
|
||
- Candidate Key: `cand:your-disk-full-alert-is-lying` (Status: `APPROVED`, Priority: `HIGH`)
|
||
- Draft Key: `cand:your-disk-full-alert-is-lying:v1` (Status: `APPROVED`)
|
||
- 拟定发布节奏：每周 2 篇（周二/周四），平稳推进 5 周。
|