Add TRIM corruption sequel post + publish SATA cable post (en/zh, og+banner)
Deploy / build (push) Successful in 19s
Deploy / build (push) Successful in 19s
- New post: 'The Corruption Came Back on Different Drives — the Cause Was TRIM, Not the SSDs' — same 0x8941f998 zeros fingerprint on a second drive stack (IronWolf 110 SATA pair), root cause queued TRIM (FreeBSD gag 264139), fix diskAutotrim=off + nodiscard, scrub evidence, enterprise SATA buying guidance. en + zh, custom OG + banner. - Publish formerly-draft SATA cable post (draft:false) with OG + banner. - Correct drive identity in both: ZA960NM10001 is Seagate IronWolf 110, not 'Samsung PM9A3' (PM9A3 is NVMe; FPDMA errors are SATA-only). - Old RAID post (en+zh): cross-link to sequel, fix 'a Samsung' -> IronWolf 110.
This commit is contained in:
@@ -0,0 +1,231 @@
|
||||
---
|
||||
title: "The Corruption Came Back on Different Drives — the Cause Was TRIM, Not the SSDs"
|
||||
description: "Two weeks after a btrfs RAID1 pool corrupted on healthy-looking NVMe drives, the same all-zero checksum reappeared on a different drive stack. The culprit was queued TRIM (NCQ) on Seagate IronWolf 110 SSDs — a firmware bug FreeBSD blacklisted but Linux never did. How I caught it and the fix."
|
||||
pubDate: 2026-09-18
|
||||
category: devops
|
||||
tags: [unraid, btrfs, trim, ssd, raid, data-loss, smart, troubleshooting, seagate]
|
||||
ogImage: /og/the-cause-was-trim-not-the-ssds.png
|
||||
banner: /banners/the-cause-was-trim-not-the-ssds.png
|
||||
---
|
||||
|
||||
Two weeks ago I published how a btrfs RAID1 pool on two healthy-looking Samsung
|
||||
PM9A3 NVMe drives silently corrupted a Windows VM — reads returning
|
||||
**all-zero** blocks that no single bad drive could explain. That investigation
|
||||
ended with an honest shrug: the drives tested clean, the strongest suspect was
|
||||
the shared M.2 riser.
|
||||
|
||||
Then it happened again. On *different* drives, on a *different* bus. The exact
|
||||
same checksum fingerprint. This time I found the root cause — and it wasn't
|
||||
the SSDs.
|
||||
|
||||
## Why this matters
|
||||
|
||||
If you run Unraid (or any Linux box) with a btrfs RAID1 pool on SATA SSDs and
|
||||
auto-TRIM enabled, this post is a heads-up: **queued TRIM was silently
|
||||
destroying both mirrors of my pool, in the same places, simultaneously.** RAID
|
||||
redundancy gives you zero protection against it. And SMART — which flagged a
|
||||
drive as *failing* through all of this — had nothing to do with the actual
|
||||
corruption.
|
||||
|
||||
---
|
||||
|
||||
## The setup
|
||||
|
||||
- **unRaid 7.x** on an Intel i9-13900K.
|
||||
- **Pool `ssd`**: two **Seagate IronWolf 110 960 GB** SATA SSDs
|
||||
(`ZA960NM10001`, serials `HKR02TFK` + `HKR02L9P`) in **btrfs RAID1**,
|
||||
mounted at `/mnt/ssd`, holding Docker's data root (`/var/lib/docker`) and
|
||||
the Windows 11 VM's virtual disk (`domains/Win11Enterprise/vdisk1.img`).
|
||||
- The VM's vdisk is **not** `NOCOW`, so btrfs validates every read against a
|
||||
checksum — the same setting that caught the previous incident.
|
||||
|
||||
Trivia: one of these drives (`sdd`) is the very drive from *["I Was About to
|
||||
RMA This SSD — the Fault Was a SATA Cable"](/posts/that-dying-ssd-was-just-a-bad-sata-cable/)*
|
||||
— after the cable swap it formatted clean and joined this pool.
|
||||
|
||||
## Step 1 — a SMART "failure" that wasn't
|
||||
|
||||
At 02:53 the Unraid notification bell went off:
|
||||
|
||||
```
|
||||
Unraid Ssd disk SMART health [1] — Warning [UNRAID] -
|
||||
raw read error rate (failing now) is 19665 (sdd = ZA960NM10001_HKR02TFK)
|
||||
```
|
||||
|
||||
"Failing now" is the drive declaring the attribute **worse than its
|
||||
threshold** — the strongest wording SMART has. But the raw value is a red
|
||||
herring on these Seagates: on IronWolf 110s, attribute 1's raw field is a
|
||||
vendor-encoded composite (errors in the high half, an operation counter in the
|
||||
low half). `19665 = 0x00004CD1` → high half = **0 errors**. The *normalized*
|
||||
value is what tripped: `093` with a **worst of 088**, below the `090`
|
||||
threshold, so the drive self-reports `FAILING_NOW` / `IN_THE_PAST` on its
|
||||
SMART return status. The twin (`sdb`) showed the same shape with a raw of
|
||||
only `341`.
|
||||
|
||||
Every real failure counter was zero: reallocated 0, grown bad block 0,
|
||||
program/erase fails 0, uncorrectable ECC 0, UDMA CRC 0, SSD life left 99 %,
|
||||
SMART overall **PASSED**, and a just-completed extended self-test was clean.
|
||||
This is a documented IronWolf 110 behaviour — the same "Failing" flag at 100 %
|
||||
lifespan hit Synology users years ago, and it's why the community treats that
|
||||
model's attribute-1 trip as firmware noise unless the real counters move.
|
||||
|
||||
So: false alarm on the *drive*, but the alarm made me go look — and that's
|
||||
when I found the real damage.
|
||||
|
||||
## Step 2 — the checksum that had no right to exist
|
||||
|
||||
`btrfs device stats /mnt/ssd`:
|
||||
|
||||
```
|
||||
[/dev/sdd1].corruption_errs 27 [/dev/sdb1].corruption_errs 31
|
||||
```
|
||||
|
||||
All of it confined to one file: **ino 261 = the Windows VM's vdisk**.
|
||||
`dmesg` showed the same signature I'd spent a week staring at in the NVMe
|
||||
incident:
|
||||
|
||||
```
|
||||
btrfs: checksum verify failed on logical 1505629372416 mirror 1 wanted 0x8941f998
|
||||
```
|
||||
|
||||
**`0x8941f998` is the CRC32C of 4 KiB of zeros.** The pool was reading back
|
||||
empty blocks — and crucially, on **both mirrors at the same logical offsets**.
|
||||
I verified it against my notes from the PM9A3 pool: identical value, identical
|
||||
"deterministic zeros" behaviour, identical RAID1-defeating correlation.
|
||||
|
||||
That match is a fingerprint. Whatever this is, it isn't the drives and it
|
||||
isn't the M.2 riser — the IronWolf pair lives on **SATA ports**, a
|
||||
completely different controller path. It's the *layer above* the drives
|
||||
deciding to erase the same data on both members at once. There is exactly one
|
||||
mechanism in Linux that does that: **discard / TRIM**.
|
||||
|
||||
## Step 3 — the root cause: queued TRIM on a drive Linux never blacklisted
|
||||
|
||||
Unraid's pool config had auto-TRIM on, which mounts btrfs with
|
||||
`discard=async` — the kernel batches frees and sends them to the drive. On
|
||||
SATA SSDs that means **queued TRIM** (the `DATA SET MANAGEMENT` command
|
||||
deferred as an NCQ command, `SEND FPDMA QUEUED`).
|
||||
|
||||
The IronWolf 110 is *known to be unstable with queued TRIM*. FreeBSD's
|
||||
kernel has a documented report — [bug 264139](https://bugs.freebsd.org/264139),
|
||||
*"ata: NCQ_DSM_TRIM trim method for Seagate IronWolf 110 SATA SSD hangs
|
||||
drives"* — and the fix, merged in 2024, is blunt:
|
||||
[commit `a6cef617660a`](https://lists.freebsd.org/archives/freebsd-fs/2024-March/003273.html):
|
||||
"The Seagate IronWolf 110 SATA SSD drive has been reported to be unstable with
|
||||
NCQ trim enabled."
|
||||
|
||||
Meanwhile Linux's own quirk list (`drivers/ata/libata-core.c`) — where the
|
||||
kernel deliberately downgrades broken drives to non-queued or disabled TRIM —
|
||||
contains Micron M500/M550/1100, Crucial M500/M550/MX100, Samsung 840/850/860/870
|
||||
and a few others, but **no Seagate entry at all**. My own `dmesg` proved the
|
||||
mechanism exists and is selective:
|
||||
|
||||
```
|
||||
ata6.00: Model 'Samsung SSD 840 PRO Series', rev 'DXM06B0Q', applying quirks: noncqtrim zeroaftertrim
|
||||
```
|
||||
|
||||
…applied to the *other* SSD in the box, while the two IronWolf 110s got
|
||||
no quirk and full queued TRIM.
|
||||
|
||||
Why does that corrupt a RAID1? Because btrfs RAID1 stores the *same* free-space
|
||||
map on both members — it discards the same logical extents on both drives in
|
||||
the same pass. A TRIM path that invalidates more (or the wrong) LBAs on this
|
||||
firmware zeroes the **same live blocks on both mirrors at once**. Your mirror
|
||||
becomes two copies of the same hole: the checksum fails on both, self-heal has
|
||||
nothing to copy from, and the data is gone.
|
||||
|
||||
> The honest caveat: FreeBSD documents *instability/hangs* from the drive
|
||||
> side, and "queued TRIM erased live data" is the mechanism I infer from that
|
||||
> plus the fingerprint (zeros on both mirrors, SMART-clean, CRC error counter
|
||||
> untouched). It fits every observation. The proof is the test — see the
|
||||
> result below.
|
||||
|
||||
## Step 4 — the fix (persistent, one variable at a time)
|
||||
|
||||
One change, applied twice — persistent config and live mount:
|
||||
|
||||
```bash
|
||||
# /boot/config/pools/ssd.cfg (backup: /root/ssd.cfg.bak.20260918-0329)
|
||||
diskAutotrim="off" # was "on"
|
||||
|
||||
# apply live without stopping the array or Docker (36 containers stayed up)
|
||||
mount -o remount,nodiscard /mnt/ssd
|
||||
findmnt /mnt/ssd # → rw,noatime,ssd,space_cache=v2 (no more discard=async)
|
||||
```
|
||||
|
||||
Then a full read-write scrub to find out what was actually damaged:
|
||||
|
||||
```
|
||||
scrub started 03:29:53, finished 03:53:07 (23 min)
|
||||
csum_errors: 18 corrected_errors: 16 uncorrectable_errors: 2
|
||||
read_errors: 0 verify_errors: 0 super_errors: 0
|
||||
```
|
||||
|
||||
`read/verify/super = 0` is the second half of the fingerprint: the hardware
|
||||
reads *fine* — nothing is failing, the stored bytes just don't match their
|
||||
checksums. Of the 18 bad blocks, **16 were bad on one mirror and were healed
|
||||
from the clean copy. Two uncorrectable entries were one single 4 KiB block
|
||||
that was bad on both mirrors** — permanent loss, ~45.2 GiB into the VM's
|
||||
vdisk (`logical 1505629372416`, file offset `48560156672`). I zero-filled
|
||||
that one block so the VM's reads at least don't error; the original bytes are
|
||||
unrecoverable either way.
|
||||
|
||||
And a wrinkle worth knowing: **`btrfs device stats` counters did not move
|
||||
during the scrub** even though it found 18 errors — scrub-detected errors and
|
||||
read-detected errors are tracked separately. You cannot watch only `device
|
||||
stats`; you have to watch dmesg + scrub output too. (My watchdog checks all
|
||||
three; the details are at the end.)
|
||||
|
||||
## What I'd do differently
|
||||
|
||||
1. **Never enable auto-TRIM on a btrfs pool without checking the drive's
|
||||
queued-TRIM record.** FreeBSD keeps a "don't TRIM this drive" list for a
|
||||
reason. Linux has the same mechanism in `libata-core.c` — an afternoon of
|
||||
grep + a `dmesg` check after the first boot beats a restored-from-backup VM.
|
||||
2. **Keep TRIM on the menu, just not queued.** If you want fstrim on SATA
|
||||
SSDs, force non-queued TRIM (`libata.force=…:noncqtrim`) and verify with a
|
||||
scrub after the first trim pass. `discard=sync` does **not** help — it
|
||||
still uses the queued path on drives that advertise it.
|
||||
3. **Watch all three corruption signals**, not one: `btrfs device stats`,
|
||||
`dmesg | grep 'csum failed'`, and periodic scrubs. Each can move
|
||||
independently (see above).
|
||||
4. **Corroborate before joining pairs.** Same-batch, same-firmware drives are
|
||||
a correlated-failure risk; a pair sharing one firmware bug is the
|
||||
correlated-failure worst case. The mirror didn't save me either time.
|
||||
5. **When buying enterprise SATA SSDs**: the drives on Linux's quirk list
|
||||
(Micron M500/M550/1100, Crucial M500/M550/MX100, Samsung 840/850/860/870)
|
||||
and anything with a documented NCQ-TRIM bug (IronWolf 110) are out.
|
||||
Samsung PM893/PM897, Micron 5400 PRO/MAX, Solidigm D3-S4610/S4620,
|
||||
WD Ultrastar DC SA620/SA630, and Kingston DC600M all carry PLP plus no
|
||||
Linux-quirk entries, and vendors that publish firmware release notes
|
||||
(Micron, Solidigm, WD, Kingston) can actually ship you a fix. NVMe
|
||||
sidesteps the SATA queued-TRIM class entirely — my PM9A3s aren't in the
|
||||
NVMe quirk table either.
|
||||
|
||||
## The result
|
||||
|
||||
One variable changed (auto-TRIM off), everything else identical. Immediately
|
||||
after:
|
||||
|
||||
- scrub #2 (04:10–04:39): **`corrected_errors: 0`** — the 16 heal-blocks
|
||||
stayed healed, nothing new appeared during the full re-read.
|
||||
- `corruption_errs` flat at sdd1=27 / sdb1=31 for hours of normal Docker +
|
||||
VM-less operation (counters had been climbing daily before).
|
||||
- No new `csum failed` lines in dmesg.
|
||||
|
||||
The confirmation window runs a few days: Win11 VM back on (it produces the
|
||||
write pattern that reproduced this), daily checks against a baseline file,
|
||||
and a re-scrub on 2026-09-22. If corruption reappears with TRIM off, the next
|
||||
suspect isn't the drives — it's the RAM (no ECC on this box), and memtest is
|
||||
the test. If it stays quiet, the IronWolf 110 queued-TRIM firmware bug is
|
||||
confirmed as the cause of both this incident and — I'd bet — a good share of
|
||||
the "healthy drives corrupting my RAID" reports the forums keep collecting.
|
||||
|
||||
A "failing" SMART warning, a drive with zero errors, and corruption on
|
||||
both mirrors of a fresh pool — every sign pointed somewhere else, and the
|
||||
actual culprit was a storage feature most of us enable without a second
|
||||
thought.
|
||||
|
||||
---
|
||||
|
||||
*Part of my storage-failure series: [the NVMe incident](/posts/when-smart-says-healthy-but-your-raid-is-corrupting-data/) and the [SATA cable wake-up call](/posts/that-dying-ssd-was-just-a-bad-sata-cable/).*
|
||||
Reference in New Issue
Block a user