hoelee-blog/src/content/posts/space-reclamation-wasnt-stuck-it-was-waiting.md

title, description, pubDate, updatedDate, category, tags, ogImage, banner, draft

title	description	pubDate	updatedDate	category	tags	ogImage	banner	draft
Space Reclamation Wasn't Stuck. It Was Waiting for Tomorrow at 09:29.	The Storage Manager card said 'running' all day and free space hadn't moved. 121 GiB had come back in two steps — and the afternoon ate 47 GiB of it.	2026-10-14	2026-10-16	devops	synology btrfs snapshots storage nas	/og/space-reclamation-wasnt-stuck-it-was-waiting.jpg	/banners/space-reclamation-wasnt-stuck-it-was-waiting.jpg	false

The question I asked was the wrong question

I emptied a few folders, kicked off Space Reclamation on the NAS, and waited. Hours later the Storage Manager card still read running, and free space looked the way it had looked in the morning. My question was "why didn't it reclaim anything".

It had reclaimed — twice — and one of those reclaims was larger than anything I had deleted. The measurement answered a different question than the one I asked, which is the interesting part.

Sample free space on a timer before believing anything

I sampled volume1 free space every five minutes for the day instead of looking at a progress card:

time	free	event
08:59 → 09:24	1134.1 GiB	flat — nothing reclaiming
09:29	+69.0 GiB	Class snapshot …10.14-09.29.02 rotated
09:34	+46.8 GiB	Video snapshot …10.14-09.30.35 rotated
15:28:45	—	DSM log: Start to empty all recycle bins.
15:34	+5.0 GiB	recycle-bin empty finished — only 5 GiB was in there
16:35 → 17:39	−39 GiB	new data written (a burst)
15:00 → 20:44	−47.2 GiB	net: the volume lost space that afternoon

Two reclaims, ~121 GiB in total, and then 47 GiB of new writes swallowed the afternoon. "Nothing came back" was a reading error on my side: I compared the morning to the evening, and the evening had a day of writing in it.

The recycle bin — the thing I had bothered to empty by hand — was 5 GiB. The snapshots, which I had not thought about at all, were 115.8 GiB. That ratio is the whole story.

Why the card said "running" and meant nothing

$ synostgreclaim --get-all-status --verbose
{"state":"running","num_subvolume_remain":1,"now_runnable":true}
   schedule_table: 168 × 1
   run_time_option: run_all_time

state: running with a schedule table of 168 ones and run_all_time means the task is configured to be "running" 24×7, by design. It is a schedule, not a progress indicator — so the card will read the same thing on a day when nothing is scheduled to happen at all. Meanwhile the machine had no synostgreclaim process, no btrfs balance, and md2/dm-3 I/O of roughly zero. Nothing was being reclaimed at that moment, and nothing was stuck either.

Then the logs, which are mostly red herrings on DSM:

• the reclaim task itself logs nothing;
• /var/log/space_operation.log is LUKS and volume-list spam;
• /var/log/synorecycle.log is synounlinkd path noise;
• Log Center for the day contained only my own recycle-bin line and a few UPS on-battery/back-online blips.

A UI state field that is always true and three log files that never contain the answer: the only instrument left is free space over time.

The mechanism nobody tells you about

Space Reclamation frees deleted btrfs subvolumes. It cannot free data that a snapshot is still pinning. Those two shares keep retention 1 — one generation each — so anything I delete becomes free space only at the next day's rotation, around 09:29 / 09:30, in a step of tens of gigabytes, and then it is silent for 24 hours.

That is exactly what the two jumps were. It also means the honest answer to "I deleted 50 GB and nothing happened" on this box is: "wait for tomorrow morning, then check again" — and if the deleted data lived inside a snapshot, the answer is "it frees when that snapshot expires, and not before".

What I would do differently

1. Sample the number, not the status. A five-minute timer on free space answers "is anything happening" in a way no task card can, and it made the regrowth visible at the same time.
2. Read the schedule field, not the state field. run_all_time explains a permanent "running" without any bug existing.
3. Know each share's retention. It is the clock that decides when a delete becomes free space; without it, every deletion looks broken for a day.
4. Separate the two ledgers. Recycle bin (5 GiB) and snapshots (115.8 GiB) behave nothing alike and are two different mental models.
5. Re-check the denominator. The volume was losing ~1.5 GiB/h to normal writes; a reclaim that returns 121 GiB and a day that consumes 47 GiB look identical if you only compare two snapshots of the same number.

What the measurement actually says about the machine

I went in expecting to fix a stuck job and came out with a storage trend instead:

volume1   42 TB   98% used   1.2 TB free   ~-1.5 GiB/h lately  (-344 GiB over 8 days)
volume2   16 TB  100% used    70 GB free   ~-19 GiB / 2 days
health    btrfs corruption_errs 0 on both volumes
          all md arrays [UUUU] / [UU], no balance running
          read/write_io_errs identical to the September baseline

So the useful conclusion is not "the reclaim works". It is that reclaim is a rounding error next to the write rate, deletion is not the lever I thought it was while retention is 1, and the trend — not the task card — is what I should be alerting on. That is the same lesson as Your Disk-Full Alert Is Lying, arriving from the other direction: alert on the consequence (bytes free over time), because the mechanism's own status is not evidence.