hoelee-blog/src/content/posts/my-backup-was-using-someone-elses-api-quota.md

title, description, pubDate, updatedDate, category, tags, ogImage, banner, draft

title	description	pubDate	updatedDate	category	tags	ogImage	banner	draft
My Backup Was Crawling Because I Was Using Someone Else's API Quota	A 1 TB restic seed stalled at zero bytes. The log named a Google project I did not own: rclone's shared OAuth client, quota pooled with every rclone user alive.	2026-10-15	2026-10-16	notes	rclone backup google-drive oauth restic	/og/my-backup-was-using-someone-elses-api-quota.jpg	/banners/my-backup-was-using-someone-elses-api-quota.jpg	false

The subscription ended, and I had to move a terabyte

The call came in the least convenient way: the OneDrive account holding my nightly backups had no subscription any more, and I had a 1.5-year Google subscription sitting unused. Move the backup, keep the history, don't lose a night.

The first thing I hit was not a technical problem: I could not remember how the backup was wired. rclone? Duplicati? Something Synology shipped? On a three-host estate with 78 compose stacks — the one in Running Production Infrastructure Solo — "which tool writes my backups" is a question I should have been able to answer from memory and could not. So step one was reading the machine.

What was actually there

Two rclone configs, which is the kind of detail that bites: the GUI config that Backrest (the restic front end) uses, plus a separate stale CLI copy that a cron job or a human shell had been using at some point. Change one and you have changed half the system.

The migration itself is boring when done in the right order, and I did it read-first:

rclone.conf                     append [gdrive-*] + [gdrive-*-crypt]
rclone/rclone.conf (CLI copy)   same two remotes
Portainer stack 177             env → the new crypt remote; config :ro → :rw
restic init                     new repo on Google Drive
Backrest SetConfig              dsm-repo URI → the new location

Before touching the plan URIs I checked one property in the source, because everything depended on it: a restic repository's identity is guid = sha256(id) — a stable value derived from the repo id, not from its URL. So repointing a plan at a new backend cannot silently fork the history. All six plans came out byte-identical to the originals, and the repo id/guid were untouched.

Then the seed stopped moving, and the log named a stranger

The new repo seeded to 2.88 GB of roughly 1075 GB — 0.27% — and then slowed to a crawl. The live log had the answer:

'Read requests per minute' ... reason: RATE_LIMIT_EXCEEDED
consumer: projects/202264815644        <- not my project
quota_limit_value: 840000

202264815644 is rclone's own shared OAuth project. My terabyte seed was drawing on a per-minute read quota pooled with every rclone user on earth, which is why it degraded from ~750 KB/s to nothing while my NAS, my disk and my link were all idle. By the time I measured, the seed was fully stalled — 0 bytes in 90 seconds, 159 errors, still rateLimitExceeded — and the six nightly plans had all failed with the same wall:

Fatal: unable to save snapshot: unexpected HTTP response (500)
googleapi: Error 403: Quota exceeded for quota metric 'Queries'
  and limit 'Requests per minute' ...

"Slow" and "blocked" look similar in a progress bar and are completely different problems. Ninety seconds of sampled zeros is what made the call.

You need your own OAuth client — and the console is unavoidable

There is exactly one real fix: stop using the shared client and bring your own. That means a Google Cloud project, an OAuth consent screen, and a credential. Two settings cost real time:

• Application type = Desktop app. rclone's redirect is http://127.0.0.1:53682, and that client type permits it without extra configuration.
• Publish the app. Leave it in "Testing" and Google expires your refresh token every 7 days — which turns "working" into a silent weekly outage.

I checked whether I could shortcut the console with the CLI that was already authenticated here. I could not, and the reason is worth knowing: gcloud was signed in only as a service account in an unrelated project, and service accounts cannot create consent screens or OAuth client IDs. The credential has to belong to a project a human owns. It was a dead end, and I verified that instead of asserting it.

Three traps in the authorization flow

1. Publishing an unverified app can hard-block a restricted scope. .../auth/drive is restricted, and an app that is published without completing Google's verification review is sometimes blocked outright with no bypass. In my case the bypass existed — the "Google hasn't verified this app" screen, then Advanced → Go to rclone-backup (unsafe) — but that is a coin flip on scope and review state, not a guarantee, and "unsafe" is only a label meaning the app never went through a paid audit.
2. The callback accepts exactly one request. A reload, or a second tab, hits 127.0.0.1:53682 with no code and kills the attempt. My first try died exactly that way (no code returned), and it looks like a credentials problem. One tab. Don't reload it.
3. Finish the consent screen on the machine that runs rclone. The redirect goes to localhost, so a phone or another PC cannot complete it — the browser has to be the one on the same host as the process waiting on that port.

Then: read the token, write client_id + client_secret + token into both configs (the GUI one and the stale CLI copy), and restore the two flags that matter for a long unattended upload: use_trash = false and stop_on_upload_limit = true.

The old side: verify by reading, not by asking

Purging the old remote reported an error — itemNotFound — and the space came back anyway:

Used:   1.012 TiB  ->  3.800 GiB
Free:   ~0         ->  1.007 TiB
DSM folder: present -> gone      Trashed: 0 B

The error was rclone complaining after the work finished. Also worth recording: the tool's documentation says personal accounts cannot permanentDelete, which would have meant emptying the recycle bin by hand — in practice the trash was already empty and roughly 1 TB was reclaimed with no browser step. Documentation describes the API; the API is not always what you get.

One more measured thing, because it changed my next decision: pausing the file-sync container that shared that machine halved the load average — 16.70 → 8.89 — which is the difference between "the network is slow" and "the box is busy".

What I would do differently

1. Own the OAuth client from day one. A shared client id is a queue you are standing in, with no status page and no one to call.
2. Answer "what writes my backups" from a document, not from memory. Two rclone configs is a normal amount of drift for a system nobody has drawn in a year.
3. Read the destination, not the exit code. The purge "failed" and freed a terabyte; a successful-looking rename can land nothing. Same lesson as It Reported Success. Nothing Had Changed. — this estate keeps teaching it.
4. Never start a multi-day upload while the last few percent of your attention is elsewhere. Checking for RATE_LIMIT_EXCEEDED at 0.27% costs minutes; discovering it after two days costs the window.

The result, honestly stated

The plumbing works: the plans live on Google Drive, the repo identity is unchanged, the seed is now spending my quota instead of a shared one, and the old side gave back about a terabyte. What it is not, yet, is a backup: at the time I wrote this, the first seed had uploaded 2.88 GB of ~1075 GB and the new repo had zero snapshots — nothing is restorable until the first snapshot lands. That is the number I watch, because "the transfer is running" and "you can restore" are different claims.