post: four posts — rclone/Drive quota, DSM space reclamation, mail domain/PTR, embedded fonts (EN+ZH) + og/banner
Deploy / build (push) Successful in 32s
Deploy / build (push) Successful in 32s
- my-backup-was-using-someone-elses-api-quota (notes, 2026-10-15, pinned by its own evidence) - space-reclamation-wasnt-stuck-it-was-waiting (devops, 2026-10-14, pinned by its own samples) - adding-a-domain-to-someone-elses-mail-server (notes, 2025-05-06, backdated: no date/version pins) - embedding-fonts-so-a-pdf-looks-the-same-everywhere (engineering, 2025-01-22, backdated: no date/version pins) All four updatedDate 2026-10-16 so every lastmod stays honest; the two backdated posts carry no prose links (a backdated post may only link backwards). og measured rows=1/overflow=0, banners 8 rows/delta=2/overflow=0.
This commit is contained in:
hoelee committed
1 parent
636e6f9624
commit
08fa7a9412
18 files changed
+1036
No files matched your search
@@ -0,0 +1,169 @@
|
||||
---
|
||||
title: "My Backup Was Crawling Because I Was Using Someone Else's API Quota"
|
||||
description: "A 1 TB restic seed stalled at zero bytes. The log named a Google project I did not own: rclone's shared OAuth client, quota pooled with every rclone user alive."
|
||||
pubDate: 2026-10-15
|
||||
updatedDate: 2026-10-16
|
||||
category: notes
|
||||
tags: ["rclone", "backup", "google-drive", "oauth", "restic"]
|
||||
ogImage: /og/my-backup-was-using-someone-elses-api-quota.png
|
||||
banner: /banners/my-backup-was-using-someone-elses-api-quota.png
|
||||
draft: false
|
||||
---
|
||||
|
||||
## The subscription ended, and I had to move a terabyte
|
||||
|
||||
The call came in the least convenient way: the OneDrive account holding my
|
||||
nightly backups had no subscription any more, and I had a 1.5-year Google
|
||||
subscription sitting unused. Move the backup, keep the history, don't lose a
|
||||
night.
|
||||
|
||||
The first thing I hit was not a technical problem: I could not remember
|
||||
*how* the backup was wired. rclone? Duplicati? Something Synology shipped?
|
||||
On a three-host estate with 78 compose stacks — the one in
|
||||
[Running Production Infrastructure Solo](/posts/running-production-infrastructure-solo/)
|
||||
— "which tool writes my backups" is a question I should have been able to
|
||||
answer from memory and could not. So step one was reading the machine.
|
||||
|
||||
## What was actually there
|
||||
|
||||
Two rclone configs, which is the kind of detail that bites: the GUI config
|
||||
that Backrest (the restic front end) uses, plus a **separate stale CLI
|
||||
copy** that a cron job or a human shell had been using at some point. Change
|
||||
one and you have changed half the system.
|
||||
|
||||
The migration itself is boring when done in the right order, and I did it
|
||||
read-first:
|
||||
|
||||
```text
|
||||
rclone.conf append [gdrive-*] + [gdrive-*-crypt]
|
||||
rclone/rclone.conf (CLI copy) same two remotes
|
||||
Portainer stack 177 env → the new crypt remote; config :ro → :rw
|
||||
restic init new repo on Google Drive
|
||||
Backrest SetConfig dsm-repo URI → the new location
|
||||
```
|
||||
|
||||
Before touching the plan URIs I checked one property in the source, because
|
||||
everything depended on it: a restic repository's identity is
|
||||
`guid = sha256(id)` — a stable value derived from the repo id, **not** from
|
||||
its URL. So repointing a plan at a new backend cannot silently fork the
|
||||
history. All six plans came out byte-identical to the originals, and the
|
||||
repo `id`/`guid` were untouched.
|
||||
|
||||
## Then the seed stopped moving, and the log named a stranger
|
||||
|
||||
The new repo seeded to 2.88 GB of roughly 1075 GB — 0.27% — and then slowed
|
||||
to a crawl. The live log had the answer:
|
||||
|
||||
```text
|
||||
'Read requests per minute' ... reason: RATE_LIMIT_EXCEEDED
|
||||
consumer: projects/202264815644 <- not my project
|
||||
quota_limit_value: 840000
|
||||
```
|
||||
|
||||
`202264815644` is **rclone's own shared OAuth project**. My terabyte seed
|
||||
was drawing on a per-minute read quota pooled with every rclone user on
|
||||
earth, which is why it degraded from ~750 KB/s to nothing while my NAS, my
|
||||
disk and my link were all idle. By the time I measured, the seed was fully
|
||||
stalled — **0 bytes in 90 seconds**, 159 errors, still `rateLimitExceeded` —
|
||||
and the six nightly plans had all failed with the same wall:
|
||||
|
||||
```text
|
||||
Fatal: unable to save snapshot: unexpected HTTP response (500)
|
||||
googleapi: Error 403: Quota exceeded for quota metric 'Queries'
|
||||
and limit 'Requests per minute' ...
|
||||
```
|
||||
|
||||
"Slow" and "blocked" look similar in a progress bar and are completely
|
||||
different problems. Ninety seconds of sampled zeros is what made the call.
|
||||
|
||||
## You need your own OAuth client — and the console is unavoidable
|
||||
|
||||
There is exactly one real fix: stop using the shared client and bring your
|
||||
own. That means a Google Cloud project, an OAuth consent screen, and a
|
||||
credential. Two settings cost real time:
|
||||
|
||||
- **Application type = Desktop app.** rclone's redirect is
|
||||
`http://127.0.0.1:53682`, and that client type permits it without extra
|
||||
configuration.
|
||||
- **Publish the app.** Leave it in "Testing" and Google expires your refresh
|
||||
token **every 7 days** — which turns "working" into a silent weekly
|
||||
outage.
|
||||
|
||||
I checked whether I could shortcut the console with the CLI that was already
|
||||
authenticated here. I could not, and the reason is worth knowing: `gcloud`
|
||||
was signed in only as a **service account** in an unrelated project, and
|
||||
service accounts cannot create consent screens or OAuth client IDs. The
|
||||
credential has to belong to a project a human owns. It was a dead end, and I
|
||||
verified that instead of asserting it.
|
||||
|
||||
## Three traps in the authorization flow
|
||||
|
||||
1. **Publishing an unverified app can hard-block a restricted scope.**
|
||||
`.../auth/drive` is restricted, and an app that is published *without*
|
||||
completing Google's verification review is sometimes blocked outright
|
||||
with no bypass. In my case the bypass existed — the "Google hasn't
|
||||
verified this app" screen, then **Advanced → Go to rclone-backup
|
||||
(unsafe)** — but that is a coin flip on scope and review state, not a
|
||||
guarantee, and "unsafe" is only a label meaning the app never went
|
||||
through a paid audit.
|
||||
2. **The callback accepts exactly one request.** A reload, or a second tab,
|
||||
hits `127.0.0.1:53682` with no code and kills the attempt. My first try
|
||||
died exactly that way (`no code returned`), and it looks like a
|
||||
credentials problem. One tab. Don't reload it.
|
||||
3. **Finish the consent screen on the machine that runs rclone.** The
|
||||
redirect goes to `localhost`, so a phone or another PC cannot complete it
|
||||
— the browser has to be the one on the same host as the process waiting
|
||||
on that port.
|
||||
|
||||
Then: read the token, write `client_id` + `client_secret` + `token` into
|
||||
**both** configs (the GUI one and the stale CLI copy), and restore the two
|
||||
flags that matter for a long unattended upload: `use_trash = false` and
|
||||
`stop_on_upload_limit = true`.
|
||||
|
||||
## The old side: verify by reading, not by asking
|
||||
|
||||
Purging the old remote reported an error — `itemNotFound` — and the space
|
||||
came back anyway:
|
||||
|
||||
```text
|
||||
Used: 1.012 TiB -> 3.800 GiB
|
||||
Free: ~0 -> 1.007 TiB
|
||||
DSM folder: present -> gone Trashed: 0 B
|
||||
```
|
||||
|
||||
The error was rclone complaining *after* the work finished. Also worth
|
||||
recording: the tool's documentation says personal accounts cannot
|
||||
`permanentDelete`, which would have meant emptying the recycle bin by hand —
|
||||
in practice the trash was already empty and roughly 1 TB was reclaimed with
|
||||
no browser step. Documentation describes the API; the API is not always what
|
||||
you get.
|
||||
|
||||
One more measured thing, because it changed my next decision: pausing the
|
||||
file-sync container that shared that machine halved the load average — 16.70
|
||||
→ 8.89 — which is the difference between "the network is slow" and "the box
|
||||
is busy".
|
||||
|
||||
## What I would do differently
|
||||
|
||||
1. **Own the OAuth client from day one.** A shared client id is a queue you
|
||||
are standing in, with no status page and no one to call.
|
||||
2. **Answer "what writes my backups" from a document, not from memory.** Two
|
||||
rclone configs is a normal amount of drift for a system nobody has drawn
|
||||
in a year.
|
||||
3. **Read the destination, not the exit code.** The purge "failed" and freed
|
||||
a terabyte; a successful-looking rename can land nothing. Same lesson as
|
||||
[It Reported Success. Nothing Had Changed.](/posts/it-reported-success-nothing-had-changed/)
|
||||
— this estate keeps teaching it.
|
||||
4. **Never start a multi-day upload while the last few percent of your
|
||||
attention is elsewhere.** Checking for `RATE_LIMIT_EXCEEDED` at 0.27%
|
||||
costs minutes; discovering it after two days costs the window.
|
||||
|
||||
## The result, honestly stated
|
||||
|
||||
The plumbing works: the plans live on Google Drive, the repo identity is
|
||||
unchanged, the seed is now spending **my** quota instead of a shared one,
|
||||
and the old side gave back about a terabyte. What it is not, yet, is a
|
||||
backup: at the time I wrote this, the first seed had uploaded 2.88 GB of
|
||||
~1075 GB and the new repo had **zero snapshots** — nothing is restorable
|
||||
until the first snapshot lands. That is the number I watch, because "the
|
||||
transfer is running" and "you can restore" are different claims.
|
||||
Reference in new issue
Block a user