post: four posts — rclone/Drive quota, DSM space reclamation, mail domain/PTR, embedded fonts (EN+ZH) + og/banner
Deploy / build (push) Successful in 32s
Deploy / build (push) Successful in 32s
- my-backup-was-using-someone-elses-api-quota (notes, 2026-10-15, pinned by its own evidence) - space-reclamation-wasnt-stuck-it-was-waiting (devops, 2026-10-14, pinned by its own samples) - adding-a-domain-to-someone-elses-mail-server (notes, 2025-05-06, backdated: no date/version pins) - embedding-fonts-so-a-pdf-looks-the-same-everywhere (engineering, 2025-01-22, backdated: no date/version pins) All four updatedDate 2026-10-16 so every lastmod stays honest; the two backdated posts carry no prose links (a backdated post may only link backwards). og measured rows=1/overflow=0, banners 8 rows/delta=2/overflow=0.
This commit is contained in:
hoelee committed
1 parent
636e6f9624
commit
08fa7a9412
18 files changed
+1036
No files matched your search
@@ -0,0 +1,124 @@
|
||||
---
|
||||
title: "Space Reclamation Wasn't Stuck. It Was Waiting for Tomorrow at 09:29."
|
||||
description: "The Storage Manager card said 'running' all day and free space hadn't moved. 121 GiB had come back in two steps — and the afternoon ate 47 GiB of it."
|
||||
pubDate: 2026-10-14
|
||||
updatedDate: 2026-10-16
|
||||
category: devops
|
||||
tags: ["synology", "btrfs", "snapshots", "storage", "nas"]
|
||||
ogImage: /og/space-reclamation-wasnt-stuck-it-was-waiting.png
|
||||
banner: /banners/space-reclamation-wasnt-stuck-it-was-waiting.png
|
||||
draft: false
|
||||
---
|
||||
|
||||
## The question I asked was the wrong question
|
||||
|
||||
I emptied a few folders, kicked off Space Reclamation on the NAS, and
|
||||
waited. Hours later the Storage Manager card still read **running**, and
|
||||
free space looked the way it had looked in the morning. My question was "why
|
||||
didn't it reclaim anything".
|
||||
|
||||
It had reclaimed — twice — and one of those reclaims was larger than
|
||||
anything I had deleted. The measurement answered a different question than
|
||||
the one I asked, which is the interesting part.
|
||||
|
||||
## Sample free space on a timer before believing anything
|
||||
|
||||
I sampled `volume1` free space every five minutes for the day instead of
|
||||
looking at a progress card:
|
||||
|
||||
| time | free | event |
|
||||
|---|---|---|
|
||||
| 08:59 → 09:24 | 1134.1 GiB | flat — nothing reclaiming |
|
||||
| **09:29** | **+69.0 GiB** | Class snapshot `…10.14-09.29.02` rotated |
|
||||
| **09:34** | **+46.8 GiB** | Video snapshot `…10.14-09.30.35` rotated |
|
||||
| **15:28:45** | — | DSM log: `Start to empty all recycle bins.` |
|
||||
| **15:34** | **+5.0 GiB** | recycle-bin empty finished — only 5 GiB was in there |
|
||||
| 16:35 → 17:39 | **−39 GiB** | new data written (a burst) |
|
||||
| 15:00 → 20:44 | **−47.2 GiB** | net: the volume *lost* space that afternoon |
|
||||
|
||||
Two reclaims, ~121 GiB in total, and then 47 GiB of new writes swallowed the
|
||||
afternoon. "Nothing came back" was a reading error on my side: I compared
|
||||
the morning to the evening, and the evening had a day of writing in it.
|
||||
|
||||
The recycle bin — the thing I had bothered to empty by hand — was **5 GiB**.
|
||||
The snapshots, which I had not thought about at all, were **115.8 GiB**.
|
||||
That ratio is the whole story.
|
||||
|
||||
## Why the card said "running" and meant nothing
|
||||
|
||||
```text
|
||||
$ synostgreclaim --get-all-status --verbose
|
||||
{"state":"running","num_subvolume_remain":1,"now_runnable":true}
|
||||
schedule_table: 168 × 1
|
||||
run_time_option: run_all_time
|
||||
```
|
||||
|
||||
`state: running` with a schedule table of 168 ones and `run_all_time` means
|
||||
the task is configured to be "running" **24×7, by design**. It is a
|
||||
schedule, not a progress indicator — so the card will read the same thing on
|
||||
a day when nothing is scheduled to happen at all. Meanwhile the machine had
|
||||
no `synostgreclaim` process, no btrfs balance, and md2/dm-3 I/O of roughly
|
||||
zero. Nothing was being reclaimed at that moment, and nothing was stuck
|
||||
either.
|
||||
|
||||
Then the logs, which are mostly red herrings on DSM:
|
||||
|
||||
- the reclaim task itself logs nothing;
|
||||
- `/var/log/space_operation.log` is LUKS and volume-list spam;
|
||||
- `/var/log/synorecycle.log` is `synounlinkd` path noise;
|
||||
- Log Center for the day contained only my own recycle-bin line and a few
|
||||
UPS on-battery/back-online blips.
|
||||
|
||||
A UI state field that is always true and three log files that never contain
|
||||
the answer: the only instrument left is free space over time.
|
||||
|
||||
## The mechanism nobody tells you about
|
||||
|
||||
Space Reclamation frees **deleted btrfs subvolumes**. It cannot free data
|
||||
that a snapshot is still pinning. Those two shares keep **retention 1** —
|
||||
one generation each — so anything I delete becomes free space only at the
|
||||
**next day's rotation, around 09:29 / 09:30**, in a step of tens of
|
||||
gigabytes, and then it is silent for 24 hours.
|
||||
|
||||
That is exactly what the two jumps were. It also means the honest answer to
|
||||
"I deleted 50 GB and nothing happened" on this box is: "wait for tomorrow
|
||||
morning, then check again" — and if the deleted data lived inside a
|
||||
snapshot, the answer is "it frees when that snapshot expires, and not
|
||||
before".
|
||||
|
||||
## What I would do differently
|
||||
|
||||
1. **Sample the number, not the status.** A five-minute timer on free space
|
||||
answers "is anything happening" in a way no task card can, and it made
|
||||
the regrowth visible at the same time.
|
||||
2. **Read the schedule field, not the state field.** `run_all_time` explains
|
||||
a permanent "running" without any bug existing.
|
||||
3. **Know each share's retention.** It is the clock that decides when a
|
||||
delete becomes free space; without it, every deletion looks broken for a
|
||||
day.
|
||||
4. **Separate the two ledgers.** Recycle bin (5 GiB) and snapshots (115.8
|
||||
GiB) behave nothing alike and are two different mental models.
|
||||
5. **Re-check the denominator.** The volume was losing ~1.5 GiB/h to normal
|
||||
writes; a reclaim that returns 121 GiB and a day that consumes 47 GiB
|
||||
look identical if you only compare two snapshots of the same number.
|
||||
|
||||
## What the measurement actually says about the machine
|
||||
|
||||
I went in expecting to fix a stuck job and came out with a storage trend
|
||||
instead:
|
||||
|
||||
```text
|
||||
volume1 42 TB 98% used 1.2 TB free ~-1.5 GiB/h lately (-344 GiB over 8 days)
|
||||
volume2 16 TB 100% used 70 GB free ~-19 GiB / 2 days
|
||||
health btrfs corruption_errs 0 on both volumes
|
||||
all md arrays [UUUU] / [UU], no balance running
|
||||
read/write_io_errs identical to the September baseline
|
||||
```
|
||||
|
||||
So the useful conclusion is not "the reclaim works". It is that reclaim is a
|
||||
rounding error next to the write rate, deletion is not the lever I thought
|
||||
it was while retention is 1, and the trend — not the task card — is what I
|
||||
should be alerting on. That is the same lesson as
|
||||
[Your Disk-Full Alert Is Lying](/posts/your-disk-full-alert-is-lying/),
|
||||
arriving from the other direction: alert on the consequence (bytes free over
|
||||
time), because the mechanism's own status is not evidence.
|
||||
Reference in new issue
Block a user