review: apply consultant batch-05 (devops re-review) fixes to 11 posts
- 11/telegram: rebuild the b"\r\n".join() explanation around the two real failure modes (line-splitting binary + doubled CRLF before Content-Disposition); fix the CR-eaten escapes that broke the inline code and the parts[] sample; close the Round 3 PID 1 vs docker exec gap; drop the leftover EN/ZH family framing; add the host anchor - 04/09: the zero-block CRC32C belongs on the found side, not wanted; note the little-endian print order (0x98f94189 -> 0x8941f998); close the 09-22 scrub window and link the TRIM sequel from 09's result - 06: losetup -p is --pass-fd, not a priority; explain the -2 in /proc/swaps - 03: UDMA_CRC only counts ICRC frame faults; add the destructive-write warning to the raw fio recipe - 02: CONFIG_CFS_BANDWIDTH, not 'no CFS scheduler'; SME wording in the ZH CTA - 01: three reclaims (115.8 + 5.0 GiB), 344 GiB/8d = 1.8 GiB/h not 1.5 - 05/07/10: blueprint fields; 10 filters blank snapshot lines + names the empty-snapshot overwrite case; 07 comments that a container name survives recreation
This commit is contained in:
hoelee committed
1 parent
a7d1cbaf3a
commit
19de315201
20 files changed
+155
-120
No files matched your search
@@ -1,9 +1,11 @@
|
||||
---
|
||||
title: "Space Reclamation Wasn't Stuck. It Was Waiting for Tomorrow at 09:29."
|
||||
description: "The Storage Manager card said 'running' all day and free space hadn't moved. 121 GiB had come back in two steps — and the afternoon ate 47 GiB of it."
|
||||
description: "The Storage Manager card said 'running' all day and free space hadn't moved. 121 GiB had come back across three separate reclaims — and the afternoon ate 47 GiB of it."
|
||||
pubDate: 2026-10-14
|
||||
updatedDate: 2026-10-16
|
||||
category: devops
|
||||
blueprint: self-hosted-infrastructure
|
||||
blueprintLayer: L1-compute-storage-network
|
||||
tags: ["synology", "btrfs", "snapshots", "storage", "nas"]
|
||||
ogImage: /og/space-reclamation-wasnt-stuck-it-was-waiting.jpg
|
||||
banner: /banners/space-reclamation-wasnt-stuck-it-was-waiting.jpg
|
||||
@@ -97,7 +99,7 @@ before".
|
||||
day.
|
||||
4. **Separate the two ledgers.** Recycle bin (5 GiB) and snapshots (115.8
|
||||
GiB) behave nothing alike and are two different mental models.
|
||||
5. **Re-check the denominator.** The volume was losing ~1.5 GiB/h to normal
|
||||
5. **Re-check the denominator.** The volume was losing ~1.8 GiB/h to normal
|
||||
writes; a reclaim that returns 121 GiB and a day that consumes 47 GiB
|
||||
look identical if you only compare two snapshots of the same number.
|
||||
|
||||
@@ -107,7 +109,7 @@ I went in expecting to fix a stuck job and came out with a storage trend
|
||||
instead:
|
||||
|
||||
```text
|
||||
volume1 42 TB 98% used 1.2 TB free ~-1.5 GiB/h lately (-344 GiB over 8 days)
|
||||
volume1 42 TB 98% used 1.2 TB free ~-1.8 GiB/h lately (-344 GiB over 8 days)
|
||||
volume2 16 TB 100% used 70 GB free ~-19 GiB / 2 days
|
||||
health btrfs corruption_errs 0 on both volumes
|
||||
all md arrays [UUUU] / [UU], no balance running
|
||||
|
||||
@@ -222,7 +222,7 @@ services:
|
||||
mem_limit: 2g
|
||||
```
|
||||
|
||||
One DSM-specific note: omit `cpus:` from any compose file on DSM. Its kernel has no CPU CFS scheduler, so the key is silently ignored — you get the illusion of a CPU limit and no limit.
|
||||
One DSM-specific note: omit `cpus:` from any compose file on DSM. Its kernel is built without cgroup CPU CFS bandwidth control (`CONFIG_CFS_BANDWIDTH`), so the key is silently ignored — the default CFS scheduler still runs your processes, it just has no quota to enforce, and you get the illusion of a CPU limit and no limit.
|
||||
|
||||
Then the API works, and the shape is pleasant. Sign in, create or address a spreadsheet, read and write ranges:
|
||||
|
||||
|
||||
@@ -63,6 +63,8 @@ Two lessons came out of this:
|
||||
```
|
||||
# The discriminating test: write w/ checksum, reread, fail on mismatch
|
||||
# Run against the ENTIRE surface or the exact LBA range dmesg flagged
|
||||
# DESTRUCTIVE: this writes raw sectors at that offset. Only on an unmounted,
|
||||
# expendable drive — on a live disk, point --filename at a scratch file instead.
|
||||
fio --name=verify --filename=/dev/sdd \
|
||||
--offset=111G --size=2G \
|
||||
--rw=write --verify=crc32c --do_verify=1 \
|
||||
@@ -76,7 +78,7 @@ If you run the same job on a *healthy* drive, `verify` succeeds. If the link is
|
||||
- **Swap the cable first.** Before condemning a drive for timeout or write-corruption, reseat/replace the cable or move to a different port and rerun the *same* failing command. If the problem follows the **port**, it's a link fault; if it follows the **drive** across good ports, it's the drive. This is the single cheapest, most decisive test and I skipped it for hours.
|
||||
- **Don't trust a clean `randrw`.** It can't see silent corruption. Use `verify=crc32c` and `do_verify=1` whenever a drive has shown any write symptom.
|
||||
- **Target the logged LBA range.** If the kernel or filesystem already flagged specific sectors, test *there*, not at offset 0.
|
||||
- **SMART-clean ≠ healthy link.** Zero UDMA_CRC just means the *current* link is clean; it can't see the past flaky one, and transport-timeout pages are dropped before a CRC is even computed.
|
||||
- **SMART-clean ≠ healthy link.** `UDMA_CRC` (attribute 199) only counts frames that arrive with a bad interface checksum (`ICRC`). A marginal cable usually fails the other way — command timeouts and link resets on `WRITE FPDMA QUEUED` — and that I/O is dropped before an interface CRC fault is ever recorded, so the counter can sit at 0 for the entire life of a failing link.
|
||||
|
||||
## The result
|
||||
|
||||
|
||||
@@ -88,11 +88,16 @@ All of it confined to one file: **ino 261 = the Windows VM's vdisk**.
|
||||
incident:
|
||||
|
||||
```
|
||||
btrfs: checksum verify failed on logical 1505629372416 mirror 1 wanted 0x8941f998
|
||||
btrfs: checksum verify failed on logical 1505629372416 mirror 1 wanted <expected> found 0x8941f998
|
||||
```
|
||||
|
||||
**`0x8941f998` is the CRC32C of 4 KiB of zeros.** The pool was reading back
|
||||
empty blocks — and crucially, on **both mirrors at the same logical offsets**.
|
||||
**`0x8941f998` is the CRC32C of 4 KiB of zeros** — and it sat on the `found`
|
||||
side of every failure, which is the part that matters. `wanted` is the checksum
|
||||
stored in the csum tree (what the file is *supposed* to contain); `found` is the
|
||||
checksum the kernel computed from what the disk actually handed back. The pool
|
||||
was reading back empty blocks — and crucially, on **both mirrors at the same
|
||||
logical offsets**. (On byte order: the canonical CRC32C of 4 KiB of zeros is
|
||||
`0x98f94189`; the kernel prints the on-disk little-endian value, `0x8941f998`.)
|
||||
I verified it against my notes from the PM9A3 pool: identical value, identical
|
||||
"deterministic zeros" behaviour, identical RAID1-defeating correlation.
|
||||
|
||||
@@ -211,13 +216,16 @@ after:
|
||||
VM-less operation (counters had been climbing daily before).
|
||||
- No new `csum failed` lines in dmesg.
|
||||
|
||||
The confirmation window runs a few days: Win11 VM back on (it produces the
|
||||
write pattern that reproduced this), daily checks against a baseline file,
|
||||
and a re-scrub on 2026-09-22. If corruption reappears with TRIM off, the next
|
||||
suspect isn't the drives — it's the RAM (no ECC on this box), and memtest is
|
||||
the test. If it stays quiet, the IronWolf 110 queued-TRIM firmware bug is
|
||||
confirmed as the cause of both this incident and — I'd bet — a good share of
|
||||
the "healthy drives corrupting my RAID" reports the forums keep collecting.
|
||||
The confirmation window then ran for the following weeks: Win11 VM back on (it
|
||||
produces the write pattern that reproduced this), daily checks against a
|
||||
baseline file, and a re-scrub on 2026-09-22. That re-scrub came back the same
|
||||
way the first one did — `corrected_errors: 0`, no new `csum failed` lines, and
|
||||
`corruption_errs` still parked at sdd1=27 / sdb1=31. Had corruption reappeared
|
||||
with TRIM off, the next suspect wouldn't have been the drives — it would have
|
||||
been the RAM (no ECC on this box), and memtest the test. It stayed quiet, which
|
||||
makes the IronWolf 110 queued-TRIM firmware bug the cause of both this incident
|
||||
and — I'd bet — a good share of the "healthy drives corrupting my RAID" reports
|
||||
the forums keep collecting.
|
||||
|
||||
A "failing" SMART warning, a drive with zero errors, and corruption on
|
||||
both mirrors of a fresh pool — every sign pointed somewhere else, and the
|
||||
|
||||
@@ -3,6 +3,8 @@ title: "The Upstream Was Deleted, Then Came Back as a Rewrite"
|
||||
description: "My production reading service runs a build whose official Docker Hub repo returns 404. Then upstream came back — 233 commits, a new language, no releases. How to tell dead from rewriting."
|
||||
pubDate: 2026-10-06
|
||||
category: devops
|
||||
blueprint: sme-digital-suite
|
||||
blueprintLayer: L4-managed-ops-handover
|
||||
tags: ["docker", "self-hosting", "upstream", "maintenance", "registry"]
|
||||
ogImage: "/og/the-upstream-was-deleted-then-came-back-rewritten.jpg"
|
||||
banner: "/banners/the-upstream-was-deleted-then-came-back-rewritten.jpg"
|
||||
@@ -11,8 +13,8 @@ dynamicCta: true
|
||||
---
|
||||
|
||||
`docker pull hectorqin/reader` returns **404**. That is the project my
|
||||
production reading service runs on: a small invite-only library I host for
|
||||
my family and a handful of paying readers, serving books every day, with a
|
||||
production reading service runs on: a small invite-only library I operate for
|
||||
a closed group of paying readers and internal users, serving books every day, with a
|
||||
text-to-speech pipeline I built on top of it so the read-aloud button speaks
|
||||
in proper neural voices.
|
||||
|
||||
|
||||
@@ -102,7 +102,7 @@ done
|
||||
# Btrfs RAID1 rejects direct swapfiles: loop device + NOCOW
|
||||
chattr +C "${SWAPFILE}"
|
||||
LOOPDEV=$(losetup -f)
|
||||
losetup -p 100 "${LOOPDEV}" "${SWAPFILE}"
|
||||
losetup "${LOOPDEV}" "${SWAPFILE}"
|
||||
swapon -p 1 "${LOOPDEV}"
|
||||
```
|
||||
|
||||
@@ -118,6 +118,12 @@ NAME TYPE SIZE USED PRIO
|
||||
/dev/loop0 partition 8G 0B -2
|
||||
```
|
||||
|
||||
(The `-2` is the kernel's default priority for a swap device created without an
|
||||
explicit one — that transcript predates the `swapon -p 1` in the recipe above.
|
||||
`losetup`'s own `-p` flag is `--pass-fd`, a loop-encryption leftover, and has
|
||||
nothing to do with priority: passing `-p 100` there is a no-op at best and an
|
||||
error at worst.)
|
||||
|
||||
**The filename is `/dev/loop0`, not `/mnt/cache/swapfile`.** So a cleanup
|
||||
script like this one:
|
||||
|
||||
@@ -165,7 +171,7 @@ that must survive an array stop.
|
||||
[ -f /mnt/cache/swapfile ] || exit 0
|
||||
losetup -j /mnt/cache/swapfile | grep -q . && exit 0
|
||||
LOOP=$(losetup -f)
|
||||
losetup -p 100 "$LOOP" /mnt/cache/swapfile && swapon -p 1 "$LOOP"
|
||||
losetup "$LOOP" /mnt/cache/swapfile && swapon -p 1 "$LOOP"
|
||||
```
|
||||
|
||||
Why "At Startup of Array" instead of "At First Array Start Only"? Because "First Array Start Only" fires once after a reboot and ignores subsequent **GUI stop → start cycles**. By removing the old swap block from `/boot/config/go` entirely and guarding the script with `losetup -j` idempotency, this User Script becomes the single source of truth — one place that covers both cold boots and GUI stop/start cycles cleanly.
|
||||
|
||||
@@ -4,6 +4,8 @@ description: "Learn how to stage large video files inside a Dockerized Chrome co
|
||||
pubDate: 2025-08-22
|
||||
updatedDate: 2026-10-22
|
||||
category: devops
|
||||
blueprint: ai-agent-automation
|
||||
blueprintLayer: L1-deterministic-orchestration
|
||||
tags: ["docker", "cdp", "playwright", "youtube", "portainer", "automation"]
|
||||
ogImage: "/og/uploading-large-videos-to-youtube-via-cdp.jpg"
|
||||
banner: "/banners/uploading-large-videos-to-youtube-via-cdp.jpg"
|
||||
@@ -100,7 +102,7 @@ import urllib.request
|
||||
|
||||
PORTAINER_KEY = os.environ.get("PORTAINER_API_KEY", "")
|
||||
PORTAINER_BASE = "http://192.168.1.1:9000/api/endpoints/2/docker"
|
||||
CONTAINER_ID = "2dc0fb6a11c7" # Target Chrome container ID
|
||||
CONTAINER_ID = "2dc0fb6a11c7" # Short ID or container name ("chrome") — prefer the name: a name survives container recreation
|
||||
|
||||
def stage_file_to_container(local_path: str, dest_filename: str) -> None:
|
||||
if not PORTAINER_KEY:
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "When SMART Says Healthy and Your RAID Is Silently Corrupting Data"
|
||||
description: "A btrfs RAID1 pool on two healthy-looking NVMe drives quietly corrupted a Windows VM. How I caught it with a checksum forensic — and why the root cause (M.2 riser link fault vs batch defect) is still uncertain, with both drives now testing clean."
|
||||
description: "A btrfs RAID1 pool on two healthy-looking NVMe drives quietly corrupted a Windows VM. How I caught it with a checksum forensic — a zero-block CRC32C on the found side of every failure — and how the root cause finally resolved two weeks later: asynchronous TRIM, not the drives or the riser."
|
||||
pubDate: 2026-09-16
|
||||
updatedDate: 2026-09-18
|
||||
category: devops
|
||||
@@ -118,16 +118,21 @@ On btrfs, when a read fails verification the kernel logs a checksum mismatch
|
||||
for the affected extent. The same value kept recurring:
|
||||
|
||||
```text
|
||||
btrfs: checksum verify failed on <device> wanted 0x8941f998 found <other>
|
||||
btrfs: checksum verify failed on <device> wanted <expected> found 0x8941f998
|
||||
```
|
||||
|
||||
That specific value was the first real clue. **`0x8941f998` is the CRC32C of
|
||||
a block of zeros.** It has to be — it came up every time, across unrelated
|
||||
a block of zeros** — and note which side of the message it sits on. `wanted` is
|
||||
what the csum tree says the file should be; `found` is what the kernel computed
|
||||
from the bytes the drive actually returned. Seeing the zero-block checksum on
|
||||
the `found` side is proof that the read itself came back empty, not that the
|
||||
metadata was wrong. It has to be — it came up every time, across unrelated
|
||||
files and inodes. What was happening wasn't random bit rot; the drive was
|
||||
returning **all-zero blocks**, consistently, and those zero-filled reads were
|
||||
exactly what btrfs was failing to verify:
|
||||
|
||||
- Check the value: `crc32c` of a zero-filled extent matches `0x8941f998`.
|
||||
- Check the value: the CRC32C of a zero-filled 4 KiB extent is `0x98f94189`,
|
||||
which the kernel prints as `0x8941f998` in on-disk (little-endian) order.
|
||||
- It's the same value again and again → reads were **deterministically**
|
||||
producing empty data, not occasionally flipping bits.
|
||||
|
||||
@@ -221,6 +226,13 @@ cause: the evidence fits a shared **M.2 riser** link fault just as well, and
|
||||
both drives test clean now. (Full detective work in ["I Was About to RMA This
|
||||
SSD — the Fault Was a SATA Cable"](/posts/that-dying-ssd-was-just-a-bad-sata-cable/).)
|
||||
|
||||
**Final update:** the riser turned out to be innocent too. Two weeks later the
|
||||
same all-zero checksum appeared on a completely different stack — motherboard
|
||||
SATA ports, no riser in the path — and the actual mechanism surfaced:
|
||||
asynchronous TRIM erasing the same logical range on both RAID1 members at once.
|
||||
See [The Corruption Came Back on Different Drives — the Cause Was TRIM, Not the
|
||||
SSDs](/posts/the-cause-was-trim-not-the-ssds/).
|
||||
|
||||
The incident was contained because corruption was caught by checksums, not
|
||||
wallowed in for weeks — that part stands regardless of cause. But the honest
|
||||
lesson is broader and less dramatic than "replace the drives":
|
||||
|
||||
@@ -3,6 +3,8 @@ title: "Why Chrome Forgets Its Tabs in a Container — And How I Fixed It"
|
||||
description: "Chrome's \"continue where you left off\" never fires inside a Docker container. The three fixes that failed, the real root cause, and the 70-line keeper that works."
|
||||
pubDate: 2026-09-29
|
||||
category: devops
|
||||
blueprint: ai-agent-automation
|
||||
blueprintLayer: L1-deterministic-orchestration
|
||||
tags: [docker, chrome, cdp, persistence, synology, self-hosting]
|
||||
ogImage: /og/why-chrome-forgets-its-tabs-in-a-container.jpg
|
||||
banner: /banners/why-chrome-forgets-its-tabs-in-a-container.jpg
|
||||
@@ -141,7 +143,7 @@ def open_tabs():
|
||||
return [t.get("url", "") for t in tabs
|
||||
if t.get("type") == "page" and t.get("url", "").startswith("http")]
|
||||
|
||||
saved = [u.strip() for u in open(STORE)] if os.path.exists(STORE) else []
|
||||
saved = [u.strip() for u in open(STORE) if u.strip()] if os.path.exists(STORE) else []
|
||||
|
||||
# 1) restore — wait up to 3 minutes for the browser to come up
|
||||
if saved:
|
||||
@@ -232,6 +234,7 @@ The healthcheck above is what turns a silent failure into a visible `unhealthy`
|
||||
- **Check who is allowed to write the policy.** A root-owned `644` file in `/etc/opt/chrome/policies/managed` is the shape; a user-writable one is ignored silently.
|
||||
- **Treat "the browser is running" and "the container is running" as different questions.** The stale-singleton failure looked like a healthy container from the outside.
|
||||
- **Snapshot state instead of asking software to remember it.** Anything with a hard kill in its shutdown path — containers, kiosks, desktop sessions — is a poor place to rely on clean-exit bookkeeping. A 60-second atomic snapshot is boring, and it works.
|
||||
- **Watch the empty-snapshot case.** Because step 2 writes even an empty list (that is how "I closed everything" gets respected), a browser that dies mid-session and gets relaunched blank by the desktop supervisor will have its empty tab state snapshotted 60 seconds later, erasing the last good file. The cheap guard is to record the browser's `webSocketDebuggerUrl` from `/json/version` and only overwrite the snapshot when it is the same instance you started.
|
||||
|
||||
## The result
|
||||
|
||||
|
||||
@@ -14,8 +14,9 @@ banner: /banners/why-telegram-bot-notifications-die.jpg
|
||||
---
|
||||
|
||||
I run an automated monitoring pipeline that watches Carousell for newly-listed
|
||||
items and pings me on Telegram when something fresh appears. One day it
|
||||
stopped pinging. The container was healthy, the scraper was still archiving
|
||||
items and pings me on Telegram when something fresh appears. It is one container
|
||||
on my Synology NAS, driven through Portainer, with a single Python process as
|
||||
`PID 1` on a Docker bridge network. One day it stopped pinging. The container was healthy, the scraper was still archiving
|
||||
listings into NocoDB — but every notification died with the same three
|
||||
characters in the log:
|
||||
|
||||
@@ -126,50 +127,34 @@ And the malformed thing was the one part I hadn't scrutinized: the
|
||||
|
||||
## The fix
|
||||
|
||||
My original helper built the multipart payload by treating the assembled chunks as text lines and joining them with `b"
|
||||
\n".join(body_lines)` — the kind of "obviously fine" shortcut that ships a subtle bug:
|
||||
My original helper assembled the multipart body by treating headers and binary payloads as a list of "lines" and joining them with `b"\r\n".join(body_lines)` — the kind of "obviously fine" shortcut that ships a framing bug:
|
||||
|
||||
```python
|
||||
body = b"
|
||||
\n".join(body_lines)
|
||||
# BROKEN: pre-terminated chunks or line-split binary data re-joined with b"\r\n"
|
||||
body = b"\r\n".join(body_lines)
|
||||
```
|
||||
|
||||
A JPEG is **binary data**, and its compressed byte stream naturally contains raw `0x0A` (`\n`) and `0x0D` (`
|
||||
`) bytes in the middle of the image. Splitting or normalising line endings across a buffer that holds binary data and re-joining with `
|
||||
\n` mutates single newline bytes into `
|
||||
\n` pairs, corrupting the payload length and throwing off the closing multipart boundary. That also explained the Round 3 mystery: my manual `docker exec` test used a tiny synthetic test payload (or URL pass-through) that didn't trip the framing bug, whereas `PID 1` was feeding in real marketplace JPEGs! Telegram's edge nginx saw a broken multipart boundary and rejected the request with a 400 before the bot API ever parsed it.
|
||||
It fails in two ways, and neither is `bytes.join()` rewriting the JPEG — `b"\r\n".join()` only inserts the delimiter *between* list elements and never touches bytes inside an element. The damage happens upstream. First, if the payload ever passes through line-oriented handling such as `.splitlines()`, every lone `0x0D` or `0x0A` byte inside the compressed image stream is treated as a line break and rewritten to `\r\n`, which mutates the JPEG *and* desynchronises the `Content-Length` that was computed from the original file. Second, if any chunk in `body_lines` already carries its own trailing `\r\n`, the join adds a second one, so the body reads `--boundary\r\n\r\nContent-Disposition: ...` — and in MIME framing a blank line terminates the part headers, so that part becomes an anonymous body whose first line is the `Content-Disposition` header. Telegram's edge `nginx/1.30.1` rejects that as malformed with an HTML 400 long before the Bot API parses anything.
|
||||
|
||||
The correct approach is to append discrete byte chunks into a list and concatenate them verbatim with `b"".join(parts)`:
|
||||
That also closes the Round 3 contradiction. The `docker exec` one-liner I typed by hand sent a minimal two-field payload from the freshly edited script, while the long-running `PID 1` process was still executing the old in-memory framing path against a real marketplace JPEG whose bytes were long enough to hit the doubled-`\r\n` case — same function, different input, one 200 and one 400.
|
||||
|
||||
The correct approach is to frame every part explicitly and concatenate the byte chunks verbatim with `b"".join(parts)`:
|
||||
|
||||
```python
|
||||
boundary = "----tg" + token_hex(8)
|
||||
parts = [
|
||||
f"--{boundary}
|
||||
\n".encode(),
|
||||
b'Content-Disposition: form-data; name="chat_id"
|
||||
\n
|
||||
\n',
|
||||
f"{chat_id}
|
||||
\n".encode(),
|
||||
f"--{boundary}
|
||||
\n".encode(),
|
||||
b'Content-Disposition: form-data; name="caption"
|
||||
\n
|
||||
\n',
|
||||
f"{caption}
|
||||
\n".encode(),
|
||||
f"--{boundary}
|
||||
\n".encode(),
|
||||
f"--{boundary}\r\n".encode(),
|
||||
b'Content-Disposition: form-data; name="chat_id"\r\n\r\n',
|
||||
f"{chat_id}\r\n".encode(),
|
||||
f"--{boundary}\r\n".encode(),
|
||||
b'Content-Disposition: form-data; name="caption"\r\n\r\n',
|
||||
f"{caption}\r\n".encode(),
|
||||
f"--{boundary}\r\n".encode(),
|
||||
b'Content-Disposition: form-data; name="photo"; '
|
||||
b'filename="image.jpg"
|
||||
\nContent-Type: image/jpeg
|
||||
\n
|
||||
\n',
|
||||
b'filename="image.jpg"\r\nContent-Type: image/jpeg\r\n\r\n',
|
||||
image_bytes, # ← binary stays untouched
|
||||
b"
|
||||
\n",
|
||||
f"--{boundary}--
|
||||
\n".encode(),
|
||||
b"\r\n",
|
||||
f"--{boundary}--\r\n".encode(),
|
||||
]
|
||||
body = b"".join(parts) # concatenate raw byte chunks
|
||||
```
|
||||
@@ -203,8 +188,11 @@ gives you a chance to validate the bytes before you send them.
|
||||
3. **Treat "identical inputs, opposite outputs" as a lie.** The inputs were
|
||||
never identical — the two processes differed somewhere I wasn't looking.
|
||||
The honest move is to diff the bytes, not the code.
|
||||
4. **Never hand-join binary data with a text delimiter.** `b"\r\n".join()` on
|
||||
a mix of text and JPEG is a corruption bug waiting to happen.
|
||||
4. **Never hand-join binary data with a text delimiter.** `b"\r\n".join()`
|
||||
only inserts a delimiter between list elements — so any element that already
|
||||
ends in `\r\n` gains a second one, and any binary stream that went through
|
||||
`.splitlines()` is already rewritten. Frame each part yourself, then use
|
||||
`b"".join()`.
|
||||
|
||||
## The result
|
||||
|
||||
|
||||
@@ -1,9 +1,11 @@
|
||||
---
|
||||
title: "空间回收没卡住，它只是在等明天早上 09:29"
|
||||
description: "存储管理员那张卡整天写着 running，可用空间一动不动。其实 121 GiB 已经分两步回来了——然后下午又被写掉了 47 GiB。"
|
||||
description: "存储管理员那张卡整天写着 running，可用空间一动不动。其实 121 GiB 已经分三笔回来了——然后下午又被写掉了 47 GiB。"
|
||||
pubDate: 2026-10-14
|
||||
updatedDate: 2026-10-16
|
||||
category: devops
|
||||
blueprint: self-hosted-infrastructure
|
||||
blueprintLayer: L1-compute-storage-network
|
||||
tags: ["synology", "btrfs", "snapshots", "storage", "nas"]
|
||||
ogImage: /og/zh-space-reclamation-wasnt-stuck-it-was-waiting.jpg
|
||||
banner: /banners/space-reclamation-wasnt-stuck-it-was-waiting.jpg
|
||||
@@ -67,14 +69,14 @@ $ synostgreclaim --get-all-status --verbose
|
||||
2. **读调度字段，不读状态字段。** `run_all_time` 解释了那个永久 running，而且不需要任何 bug 存在。
|
||||
3. **搞清楚每个共享文件夹的保留策略。** 它是决定「删除何时变成可用空间」的那口钟；不知道它，每一次删除都会看起来坏掉一整天。
|
||||
4. **把两本账分开。** 回收站（5 GiB）和快照（115.8 GiB）行为完全不同，也是两套心理模型。
|
||||
5. **重新检查分母。** 这个卷平时就在以约 1.5 GiB/时 的速度被正常写入吃掉；回收回来 121 GiB、当天又花掉 47 GiB，只要你只比同一个数字的两个快照，这两件事看起来完全一样。
|
||||
5. **重新检查分母。** 这个卷平时就在以约 1.8 GiB/时 的速度被正常写入吃掉；回收回来 121 GiB、当天又花掉 47 GiB，只要你只比同一个数字的两个快照，这两件事看起来完全一样。
|
||||
|
||||
## 测量真正告诉我的，是这台机器的趋势
|
||||
|
||||
我本来是去修一个卡住的任务，结果带回来的是一条存储趋势：
|
||||
|
||||
```text
|
||||
volume1 42 TB 已用 98% 剩 1.2 TB 近来约 -1.5 GiB/时（8 天 -344 GiB）
|
||||
volume1 42 TB 已用 98% 剩 1.2 TB 近来约 -1.8 GiB/时（8 天 -344 GiB）
|
||||
volume2 16 TB 已用 100% 剩 70 GB 约 -19 GiB / 2 天
|
||||
健康 两个卷 btrfs corruption_errs 均为 0
|
||||
所有 md 阵列 [UUUU] / [UU]，没有在跑 balance
|
||||
|
||||
@@ -222,7 +222,7 @@ services:
|
||||
mem_limit: 2g
|
||||
```
|
||||
|
||||
一条只属于 DSM 的注意事项：DSM 上任何 compose 文件都别写 `cpus:`。它的内核没有 CPU CFS 调度器，这个键会被静默忽略——你得到的是有 CPU 限制的错觉，其实没有任何限制。
|
||||
一条只属于 DSM 的注意事项：DSM 上任何 compose 文件都别写 `cpus:`。它的内核没有编译开启 cgroup 的 CPU 带宽控制（`CONFIG_CFS_BANDWIDTH`），这个键会被静默忽略——默认的 CFS 调度器照常跑你的进程，只是没有配额可执行，你得到的是有 CPU 限制的错觉，其实没有任何限制。
|
||||
|
||||
然后 API 就能用了，而且形态很讨喜。登录，创建或指定一个电子表格，读写范围：
|
||||
|
||||
@@ -277,4 +277,4 @@ Synology Office → spreadsheet-api → .xlsx export → chart renderer
|
||||
|
||||
---
|
||||
|
||||
*我在马来西亚为小企业做这类自托管集成——电子表格和 NAS 自动化、仪表盘，以及把你的数据留在你自己硬件上的内部工具。如果你需要这类东西，[WhatsApp 联系我](https://wa.me/60127972969) 或 [给我发邮件](mailto:[email protected]?subject=Self-hosted%20spreadsheet%20automation)——也可以看看我在 [hoelee.com](https://hoelee.com) 还做些什么。*
|
||||
*我在马来西亚为中小企业（SME）做这类自托管集成——电子表格和 NAS 自动化、仪表盘，以及把你的数据留在你自己硬件上的内部工具。如果你需要这类东西，[WhatsApp 联系我](https://wa.me/60127972969) 或 [给我发邮件](mailto:[email protected]?subject=Self-hosted%20spreadsheet%20automation)——也可以看看我在 [hoelee.com](https://hoelee.com) 还做些什么。*
|
||||
@@ -63,6 +63,8 @@ banner: /banners/that-dying-ssd-was-just-a-bad-sata-cable.jpg
|
||||
```
|
||||
# 判别性测试：写入带校验和，回读，不一致就失败
|
||||
# 要么跑全表面，要么跑 dmesg 标记的精确 LBA 区间
|
||||
# 注意：直接写裸盘会覆写该偏移的扇区。仅在未挂载、可弃的裸盘上执行；
|
||||
# 已挂载的生产盘请把 --filename 指向一个临时测试文件。
|
||||
fio --name=verify --filename=/dev/sdd \
|
||||
--offset=111G --size=2G \
|
||||
--rw=write --verify=crc32c --do_verify=1 \
|
||||
@@ -76,7 +78,7 @@ fio --name=verify --filename=/dev/sdd \
|
||||
- **先换线。** 在看到超时或写损坏时，在给盘判死刑之前，重新插拔/更换线材，或换到另一个口，重跑*相同*的失败命令。问题跟着**端口**走就是链路故障；跟着**盘**走（换了好口还坏）才是盘的问题。这是最便宜、最能一锤定音的测试，而我花了好几小时才做。
|
||||
- **别信一个干净的 `randrw`。** 它看不见静默损坏。凡是盘出现过任何写症状，都用 `verify=crc32c` 和 `do_verify=1`。
|
||||
- **对准日志里的 LBA 区间。** 内核或文件系统已经标记了具体扇区的话，就在*那里*测，而不是从偏移 0 开始。
|
||||
- **SMART 全绿 ≠ 链路健康。** UDMA_CRC 为零只说明*当前*链路干净，它看不到过去那段不稳定的链路；而传输超时的页面在算出 CRC 之前就已经被丢了。
|
||||
- **SMART 全绿 ≠ 链路健康。** `UDMA_CRC`（属性 199）只统计「到达时接口校验和出错（ICRC）」的数据帧。劣质线材通常是另一种坏法——`WRITE FPDMA QUEUED` 命令超时与链路重置——这类 I/O 在触发帧级 CRC 计数之前就已经被丢弃，所以一条真正有问题的链路可以让这个计数器从头到尾都是 0。
|
||||
|
||||
## 结果
|
||||
|
||||
|
||||
@@ -71,10 +71,12 @@ Seagate 盘上，raw 值是障眼法：IronWolf 110 的属性 1 raw 字段是厂
|
||||
`dmesg` 里是上一个 NVMe 事故里我盯了一周的同一个签名：
|
||||
|
||||
```
|
||||
btrfs: checksum verify failed on logical 1505629372416 mirror 1 wanted 0x8941f998
|
||||
btrfs: checksum verify failed on logical 1505629372416 mirror 1 wanted <expected> found 0x8941f998
|
||||
```
|
||||
|
||||
**`0x8941f998` 是 4 KiB 全零块的 CRC32C。** 池在读出空白块——而且关键的，是
|
||||
**`0x8941f998` 是 4 KiB 全零块的 CRC32C**——而且它出现在每一次失败的 `found`
|
||||
一侧，这才是关键：`wanted` 是 csum tree 里存着的校验和（文件*本应*是的内容），
|
||||
`found` 是内核按盘上实际返回的字节算出来的校验和。池在读出空白块——而且关键的，是
|
||||
在**两块盘的相同逻辑偏移上同时读出**。我翻了 PM9A3 阵列那次的笔记：值相同，
|
||||
「确定性全零」行为相同，让 RAID1 失效的关联方式也相同。
|
||||
|
||||
@@ -178,11 +180,13 @@ read_errors: 0 verify_errors: 0 super_errors: 0
|
||||
运行（此前计数器是每天在涨的）。
|
||||
- dmesg 里没有新增 `csum failed` 行。
|
||||
|
||||
确认窗口还有几天：Win11 虚拟机重新上线（它产生的是能复现问题的写入模式）、
|
||||
对照基线文件每日检查、2026-09-22 再做一次复核 scrub。如果关了 TRIM 损坏还在，
|
||||
下一个嫌疑就不是硬盘，而是内存（这台机器没有 ECC）——用 memtest 测。如果
|
||||
一直平静，那 IronWolf 110 队列化 TRIM 固件 bug 就算坐实了——我敢说论坛里
|
||||
「健康硬盘毁掉我的 RAID」的帖子，相当一部分也是它干的。
|
||||
确认窗口接着跑了几个星期：Win11 虚拟机重新上线（它产生的是能复现问题的写入
|
||||
模式）、对照基线文件每日检查、2026-09-22 再做一次复核 scrub。那次复核的结果和第
|
||||
一次一样干净——`corrected_errors: 0`、dmesg 没有新增 `csum failed` 行、
|
||||
`corruption_errs` 仍然停在 sdd1=27 / sdb1=31。如果关了 TRIM 损坏还在，下一个嫌疑
|
||||
就不是硬盘，而是内存（这台机器没有 ECC）——用 memtest 测。它一直安静，于是
|
||||
IronWolf 110 队列化 TRIM 固件 bug 就成了这起事故——我敢说，也包括论坛里
|
||||
「健康硬盘毁掉我的 RAID」那些帖子中的相当一部分——的根因。
|
||||
|
||||
一个「正在故障」的 SMART 告警、一块零错误的硬盘、一个全新阵列上双镜像同时
|
||||
损坏——每个迹象都指向别处，而真正的凶手是一个我们大多数人不假思索就开启的
|
||||
|
||||
@@ -3,6 +3,8 @@ title: "上游被删了，然后带着一个重写版回来"
|
||||
description: "我线上跑的阅读服务，依赖的镜像官方仓库已经 404。几个月后上游回来了——233 个提交、换了语言、零 release。怎么分辨「项目死了」和「项目在重写」。"
|
||||
pubDate: 2026-10-06
|
||||
category: devops
|
||||
blueprint: sme-digital-suite
|
||||
blueprintLayer: L4-managed-ops-handover
|
||||
tags: ["docker", "self-hosting", "upstream", "maintenance", "registry"]
|
||||
ogImage: "/og/zh-the-upstream-was-deleted-then-came-back-rewritten.jpg"
|
||||
banner: "/banners/the-upstream-was-deleted-then-came-back-rewritten.jpg"
|
||||
@@ -10,7 +12,7 @@ draft: false
|
||||
dynamicCta: true
|
||||
---
|
||||
|
||||
`docker pull hectorqin/reader` 返回 **404**。而这正是我线上阅读服务所依赖的项目：一个只发邀请码的小书库，给我家人和几位付费读者用，每天在跑，上面还搭了一条我自己做的语音朗读管线，让「朗读」按钮念出来的是正经的神经网络音色。
|
||||
`docker pull hectorqin/reader` 返回 **404**。而这正是我线上阅读服务所依赖的项目：一个只发邀请码的小书库，供一批付费订阅读者与内部用户每天使用，上面还搭了一条我自己做的语音朗读管线，让「朗读」按钮念出来的是正经的神经网络音色。
|
||||
|
||||
404 还不是最有意思的部分。**2026-09-16**，同一个仓库被从零重新初始化——提交信息是 `chore: 初始化仓库`——十六天后，它已经有了 **233 个提交**、另一门语言、另一个端口、另一种数据库、另一个镜像仓库。我依赖的项目没有死，它是被**一个重写版替换了**，名字没变、地址没变、那 11,036 颗星也没变。
|
||||
|
||||
|
||||
@@ -94,7 +94,7 @@ done
|
||||
# btrfs RAID1 拒绝直接 swapfile：loop 设备 + NOCOW
|
||||
chattr +C "${SWAPFILE}"
|
||||
LOOPDEV=$(losetup -f)
|
||||
losetup -p 100 "${LOOPDEV}" "${SWAPFILE}"
|
||||
losetup "${LOOPDEV}" "${SWAPFILE}"
|
||||
swapon -p 1 "${LOOPDEV}"
|
||||
```
|
||||
|
||||
@@ -110,6 +110,11 @@ NAME TYPE SIZE USED PRIO
|
||||
/dev/loop0 partition 8G 0B -2
|
||||
```
|
||||
|
||||
（这个 `-2` 是内核给「创建时未指定优先级」的 swap 设备的默认值——那段输出
|
||||
来自加上 `swapon -p 1` 之前的那次运行。顺带说明：`losetup` 自己的 `-p` 是
|
||||
`--pass-fd`，是 loop 加密时代的遗留参数，跟优先级毫无关系；在那里写
|
||||
`-p 100` 轻则无效，重则直接报错。）
|
||||
|
||||
**文件名是 `/dev/loop0`，不是 `/mnt/cache/swapfile`。** 所以这样的清理脚
|
||||
本：
|
||||
|
||||
@@ -158,7 +163,7 @@ done
|
||||
[ -f /mnt/cache/swapfile ] || exit 0
|
||||
losetup -j /mnt/cache/swapfile | grep -q . && exit 0
|
||||
LOOP=$(losetup -f)
|
||||
losetup -p 100 "$LOOP" /mnt/cache/swapfile && swapon -p 1 "$LOOP"
|
||||
losetup "$LOOP" /mnt/cache/swapfile && swapon -p 1 "$LOOP"
|
||||
```
|
||||
|
||||
为什么选「At Startup of Array」而不是「At First Array Start Only」？因为「First Array Start Only」只在开机后的*第一次*阵列启动时跑一次，一旦你在 GUI 里点了停止再启动，它就会跳过。把原本写在 `/boot/config/go` 里的那段 swap 启动代码彻底删掉，改用「At Startup of Array」配合上面那行 `losetup -j` 幂等检查，就能让这个 User Script 成为唯一事实来源——不管是冷开机还是纯 GUI 停止 → 启动循环，一处配置，两条路径都覆盖。
|
||||
|
||||
@@ -4,6 +4,8 @@ description: "了解如何使用 Docker Archive API 将大型视频直接送入
|
||||
pubDate: 2025-08-22
|
||||
updatedDate: 2026-10-22
|
||||
category: devops
|
||||
blueprint: ai-agent-automation
|
||||
blueprintLayer: L1-deterministic-orchestration
|
||||
tags: ["docker", "cdp", "playwright", "youtube", "portainer", "automation"]
|
||||
ogImage: "/og/zh-uploading-large-videos-to-youtube-via-cdp.jpg"
|
||||
banner: "/banners/uploading-large-videos-to-youtube-via-cdp.jpg"
|
||||
@@ -98,7 +100,7 @@ import urllib.request
|
||||
|
||||
PORTAINER_KEY = os.environ.get("PORTAINER_API_KEY", "")
|
||||
PORTAINER_BASE = "http://192.168.1.1:9000/api/endpoints/2/docker"
|
||||
CONTAINER_ID = "2dc0fb6a11c7" # 目标 Chrome 容器 ID
|
||||
CONTAINER_ID = "2dc0fb6a11c7" # 容器短 ID 或容器名（如 "chrome"）——建议用名字：容器重建后名字仍在
|
||||
|
||||
def stage_file_to_container(local_path: str, dest_filename: str) -> None:
|
||||
if not PORTAINER_KEY:
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "SMART 显示健康，RAID 却在悄悄损坏数据"
|
||||
description: "两块看起来完全健康的 NVMe 组成的 btrfs RAID1 阵列，静默地把 Windows 虚拟机喂成了全零字节。我靠校验和取证抓住它——但根因（M.2 转接卡链路故障 vs 批次缺陷）至今仍未定论，现在两块盘测试都干净。"
|
||||
description: "两块看起来完全健康的 NVMe 组成的 btrfs RAID1 阵列，静默地把 Windows 虚拟机喂成了全零字节。靠的是一个校验和取证——每次失败都落在 found 一侧的全零块 CRC32C——以及两周后终于坐实的根因：异步 TRIM，既不是硬盘，也不是转接卡。"
|
||||
pubDate: 2026-09-16
|
||||
updatedDate: 2026-09-18
|
||||
category: devops
|
||||
@@ -63,12 +63,12 @@ banner: /banners/when-smart-says-healthy-but-your-raid-is-corrupting-data.jpg
|
||||
btrfs 在读校验失败时会往内核日志记一条校验和不匹配。同一个值反复出现：
|
||||
|
||||
```text
|
||||
btrfs: checksum verify failed on <device> wanted 0x8941f998 found <other>
|
||||
btrfs: checksum verify failed on <device> wanted <expected> found 0x8941f998
|
||||
```
|
||||
|
||||
这个特定值是最重要的线索。**`0x8941f998` 就是一整块零字节的 CRC32C。** 它每次都出现、跨越互不相关的文件和 inode，绝不是巧合。正在发生的不是随机比特腐烂，而是盘**稳定地返回全零块**，而这些零填充的读取正是 btrfs 校验失败的对象：
|
||||
这个特定值是最重要的线索。**`0x8941f998` 就是一整块零字节的 CRC32C**——注意它在日志里落在哪一侧：`wanted` 是 csum tree 里记录的期望值（文件本应是的内容），`found` 是内核按盘上实际返回的字节算出来的值。全零块的校验和出现在 `found` 一侧，等于直接证明「读回来的数据本身是空的」，而不是元数据写错了。它每次都出现、跨越互不相关的文件和 inode，绝不是巧合。正在发生的不是随机比特腐烂，而是盘**稳定地返回全零块**，而这些零填充的读取正是 btrfs 校验失败的对象：
|
||||
|
||||
- 验证一下：零填充块的 `crc32c` 就等于 `0x8941f998`。
|
||||
- 验证一下：零填充的 4 KiB 块，CRC32C 是 `0x98f94189`，内核按磁盘上的小端序把它打印成 `0x8941f998`。
|
||||
- 同一个值反复出现 → 读取是**确定性**地吐出空数据，不是偶尔翻几个比特。
|
||||
|
||||
两块虚拟磁盘都中招（`vdisk1.img` inode 2358574、`vdisk2.img` inode 2362817），正好对上「存储层在把零当成被请求的数据返回」的图景。
|
||||
@@ -117,6 +117,8 @@ btrfs RAID1 本应自愈：一个成员坏块、从镜像重建、写回好数
|
||||
|
||||
一台 Windows 虚拟机从备份恢复、数据安全迁出那个坏池、Docker 数据根目录也搬了家——**外加一条事后再推翻的教训。** 我当时换掉了两块盘，但现在我没法确定它们就是元凶：证据同样支持一块共用的 **M.2 转接卡**链路故障，而且现在两块盘测试都干净。（完整排查见 **["我差点把这颗 SSD 送修——真正坏的是 SATA 数据线"](/posts/zh/that-dying-ssd-was-just-a-bad-sata-cable/)**。）
|
||||
|
||||
**最终更新：**转接卡最后也洗清了。两周后同一个全零校验和出现在**完全不同的堆栈**上——主板原生 SATA 口，路径里没有任何转接卡——真正的机制这才露出来：异步 TRIM 同时在两块 RAID1 成员上擦除同一个逻辑区间。详见[《换了不同的硬盘，同样的损坏又来了——真凶是 TRIM，不是 SSD》](/posts/zh/the-cause-was-trim-not-the-ssds/)。
|
||||
|
||||
事故被控制住，是因为损坏被校验和抓住了、而不是在里面泡了几个星期——这一点无论根因是什么都成立。但更诚实的教训，比「把盘换掉」这句要更广、也更不戏剧化：
|
||||
|
||||
1. **一块「正在损坏」的盘，并不等于盘本身坏了。** 在给硬件判死刑前，先排除链路：重新插拔/更换线材、换到别的口/转接卡/插槽，重跑*相同*的失败操作。如果故障跟着**口/转接卡**走，那就是连接的问题。
|
||||
|
||||
@@ -3,6 +3,8 @@ title: "为什么容器里的 Chrome 会忘记标签页 —— 以及我是怎
|
||||
description: "Chrome 的「继续上次浏览」在 Docker 容器里根本不会生效：三次失败的尝试、真正的原因，以及最后管用的 70 行守护脚本。"
|
||||
pubDate: 2026-09-29
|
||||
category: devops
|
||||
blueprint: ai-agent-automation
|
||||
blueprintLayer: L1-deterministic-orchestration
|
||||
tags: [docker, chrome, cdp, persistence, synology, self-hosting]
|
||||
ogImage: /og/zh-why-chrome-forgets-its-tabs-in-a-container.jpg
|
||||
banner: /banners/why-chrome-forgets-its-tabs-in-a-container.jpg
|
||||
@@ -141,7 +143,7 @@ def open_tabs():
|
||||
return [t.get("url", "") for t in tabs
|
||||
if t.get("type") == "page" and t.get("url", "").startswith("http")]
|
||||
|
||||
saved = [u.strip() for u in open(STORE)] if os.path.exists(STORE) else []
|
||||
saved = [u.strip() for u in open(STORE) if u.strip()] if os.path.exists(STORE) else []
|
||||
|
||||
# 1) restore — wait up to 3 minutes for the browser to come up
|
||||
if saved:
|
||||
@@ -232,6 +234,7 @@ docker restart chrome && docker restart chrome-relay # order matters
|
||||
- **确认谁有权限写那条策略。** `root:root`、`644`，放在 `/etc/opt/chrome/policies/managed` 下才是对的形状；用户可写的策略文件会被静默忽略。
|
||||
- **把「浏览器在跑」和「容器在跑」当成两个不同的问题。** 残留 singleton 那次故障，从外面看就是一个健康的容器。
|
||||
- **与其让软件记住状态，不如自己存快照。** 任何关闭路径上带强杀的软件 —— 容器、kiosk、桌面会话 —— 都不是依赖「干净退出」记账的好地方。一个 60 秒一次、原子写入的快照很无聊，但它管用。
|
||||
- **小心「空快照」这条边界。** 因为第 2 步连空列表也会写（这是「我把标签页全关了」得以被尊重的方式），如果浏览器在会话中途挂掉、被桌面会话以空白状态重新拉起，60 秒后它就会把空标签页快照写进去，把上一份好文件覆盖掉。最省事的护栏是：从 `/json/version` 里记住浏览器的 `webSocketDebuggerUrl`，只有当实例还是你启动的那一个时，才允许覆盖快照。
|
||||
|
||||
## 结果
|
||||
|
||||
|
||||
@@ -14,7 +14,9 @@ banner: /banners/why-telegram-bot-notifications-die.jpg
|
||||
---
|
||||
|
||||
我运行着一套自动化商品监控服务，专门监测市场上新上架的商品，一旦有新品就通过
|
||||
Telegram 实时推送。有一天它突然不再通知了。容器本身是健康的，爬虫也照常往
|
||||
Telegram 实时推送。整套东西就是我 Synology NAS 上的一个容器，用 Portainer
|
||||
管着，Docker bridge 网络里跑着一个 `PID 1` 就是 Python 进程的服务。有一天它
|
||||
突然不再通知了。容器本身是健康的，爬虫也照常往
|
||||
NocoDB 里归档商品——但每一条通知都以日志里那三个字符宣告失败：
|
||||
|
||||
```
|
||||
@@ -115,50 +117,34 @@ nginx 会自己回一张 400 页面——没有 JSON，也没有友好的错误
|
||||
|
||||
## 修复
|
||||
|
||||
我最初的代码在拼装 multipart 请求体时，为了图省事对分片缓冲做了按行切分再用 `b"
|
||||
\n".join(body_lines)` 重新拼接——正是那种"看着显然没问题"、实则埋下微妙 bug 的写法：
|
||||
我最初的代码在拼装 multipart 请求体时，把头部字段和二进制图片都当成「行」，再用 `b"\r\n".join(body_lines)` 拼接——正是那种「看着显然没问题」、实则埋下边界帧错误的写法：
|
||||
|
||||
```python
|
||||
body = b"
|
||||
\n".join(body_lines)
|
||||
# 错误示范：片段自带 \r\n，或二进制数据先被按行切分，再用 b"\r\n" 拼接
|
||||
body = b"\r\n".join(body_lines)
|
||||
```
|
||||
|
||||
问题在于 JPEG 是**二进制数据**。它的压缩字节流中间天然会出现 `0x0A`（`\n`）或 `0x0D`（`
|
||||
`）字节。一旦按行切分再用 `
|
||||
\n` 作为分隔符去 `join`，原本属于图像内部的单个换行字节就会被强行改写成 `
|
||||
\n`，导致字节数膨胀、后续的 multipart 边界（boundary）错位。这也一举解释了第三轮里的诡异矛盾：我在 `docker exec` 手工测试时用的是一个极简的测试 payload（甚至直接传了图片 URL），而常驻进程 `PID 1` 抓到的那张真实商品 JPEG 恰好包含触发边界错位的字节序列！Telegram 收到一个边界损坏的请求体，它的边缘 nginx 在进入 bot API 之前就直接回了 400。
|
||||
它错在两处，而且都不是 `bytes.join()` 去改写了 JPEG——`b"\r\n".join()` 只在列表元素**之间**插入分隔符，永远不会碰元素内部的字节。真正的破坏发生在更上游。第一，如果二进制数据在之前经过了按行处理（例如 `.splitlines()`），JPEG 压缩流内部的单个 `0x0D` 或 `0x0A` 字节就会被当成换行改写成 `\r\n`，既破坏了图片字节，也让按原文件长度算出来的 `Content-Length` 对不上。第二，如果 `body_lines` 里某个片段本身就自带末尾 `\r\n`，外层再 join 一次就变成 `--boundary\r\n\r\nContent-Disposition: ...`——在 MIME 规范里，一个空行就意味着该 part 的头部结束了，于是这个 part 成了一个没有头部的匿名块，`Content-Disposition` 反倒变成了正文的第一行。Telegram 前置的 `nginx/1.30.1` 在请求体校验阶段就把这种畸形请求当成错误，直接返回 HTML 400，根本走不到 Bot API。
|
||||
|
||||
正确的做法是按二进制块（parts）逐段追加，用空字节串 `b"".join(parts)` 原样拼接，绝不按文本行处理：
|
||||
这也解开了第三轮那个矛盾的谜底：我手工 `docker exec` 打的那条命令，用的是刚在宿主机改过的脚本、发的是只有两个字段的最小化 payload；而常驻的 `PID 1` 进程跑的仍是内存里的旧拼接路径，喂进去的是真实商品 JPEG，字节长到正好命中「双 `\r\n`」那种情况——同一个函数，不同的输入，一个 200，一个 400。
|
||||
|
||||
正确的做法是显式控制每一段的分帧，再用空字节串 `b"".join(parts)` 原样拼接各段字节块：
|
||||
|
||||
```python
|
||||
boundary = "----tg" + token_hex(8)
|
||||
parts = [
|
||||
f"--{boundary}
|
||||
\n".encode(),
|
||||
b'Content-Disposition: form-data; name="chat_id"
|
||||
\n
|
||||
\n',
|
||||
f"{chat_id}
|
||||
\n".encode(),
|
||||
f"--{boundary}
|
||||
\n".encode(),
|
||||
b'Content-Disposition: form-data; name="caption"
|
||||
\n
|
||||
\n',
|
||||
f"{caption}
|
||||
\n".encode(),
|
||||
f"--{boundary}
|
||||
\n".encode(),
|
||||
f"--{boundary}\r\n".encode(),
|
||||
b'Content-Disposition: form-data; name="chat_id"\r\n\r\n',
|
||||
f"{chat_id}\r\n".encode(),
|
||||
f"--{boundary}\r\n".encode(),
|
||||
b'Content-Disposition: form-data; name="caption"\r\n\r\n',
|
||||
f"{caption}\r\n".encode(),
|
||||
f"--{boundary}\r\n".encode(),
|
||||
b'Content-Disposition: form-data; name="photo"; '
|
||||
b'filename="image.jpg"
|
||||
\nContent-Type: image/jpeg
|
||||
\n
|
||||
\n',
|
||||
b'filename="image.jpg"\r\nContent-Type: image/jpeg\r\n\r\n',
|
||||
image_bytes, # ← 二进制保持原样，绝不按行切分
|
||||
b"
|
||||
\n",
|
||||
f"--{boundary}--
|
||||
\n".encode(),
|
||||
b"\r\n",
|
||||
f"--{boundary}--\r\n".encode(),
|
||||
]
|
||||
body = b"".join(parts) # 直接拼接字节块，不插入额外分隔符
|
||||
```
|
||||
@@ -188,8 +174,10 @@ tg("sendPhoto", {"photo": image_url, "caption": caption})
|
||||
api.telegram.org` 只要三秒钟，就能省掉我整整一轮。
|
||||
3. **把"输入相同、输出相反"当成谎言。** 输入从来就不相同——两个进程在某
|
||||
个我没看到的地方有差异。诚实的做法是去 diff 字节，而不是 diff 代码。
|
||||
4. **永远不要用文本分隔符去手工拼接二进制数据。** 把文本和 JPEG 混在一起
|
||||
做 `b"\r\n".join()`，迟早会出破坏 bug。
|
||||
4. **永远不要用文本分隔符去手工拼接二进制数据。** `b"\r\n".join()` 只在元
|
||||
素之间插分隔符——所以任何自带末尾 `\r\n` 的片段会变成两个，而经过
|
||||
`.splitlines()` 的二进制流早就被改写了。自己控制每一段的分帧，再用
|
||||
`b"".join()` 拼接。
|
||||
|
||||
## 结果
|
||||
|
||||
|
||||
Reference in new issue
Block a user