Add post: Why Telegram Bot Notifications Die (IPv6/DNS/multipart debugging)
Deploy / build (push) Successful in 36s
Deploy / build (push) Successful in 36s
This commit is contained in:
+100
-32
@@ -4,9 +4,11 @@ Living list of what's done and what's next for blog.hoelee.com. Work through the
|
||||
|
||||
**Status key:** ✅ done · 🔵 in progress · ⬜ not started
|
||||
|
||||
> **How to resume the project:** start at the top-most ⬜ item in the **Execution Plan** (§2) below and work downward. Each step is self-contained, has a "done when" criterion, and references the governing doc. Don't jump ahead — earlier steps unlock later ones.
|
||||
|
||||
---
|
||||
|
||||
## Done (foundation)
|
||||
## 1. Done (foundation)
|
||||
|
||||
- ✅ Astro 5 static + Markdown, Gitea Actions CI/CD → nginx → Cloudflare (deployed)
|
||||
- ✅ Design system: hoelee.com brand palette, Inter + JetBrains Mono, light/dark, sticky nav, code copy button
|
||||
@@ -14,53 +16,119 @@ Living list of what's done and what's next for blog.hoelee.com. Work through the
|
||||
- ✅ Author card + `Person`/`ProfilePage` JSON-LD (E-E-A-T), article meta, reading time, related posts
|
||||
- ✅ Full SEO: canonical, Open Graph (+dims), twitter:card, favicon (all sizes), RSS + sitemap
|
||||
- ✅ Category pages (`/categories/`, `/categories/[category]/`)
|
||||
- ✅ Locale scheme: English flat in `posts/`, Chinese in `posts/zh/` (lang derived from folder, no `lang:` frontmatter)
|
||||
- ✅ Language switcher in nav — links to the **same post** in the other language (auto-matches by `zh/` prefix; falls back to section landing when no translation)
|
||||
- ✅ Locale-aware nav labels (EN: Posts/Categories/About/RSS; ZH: 文章/分类/关于/RSS)
|
||||
- ✅ Dark mode default (light is opt-in via toggle)
|
||||
- ✅ Case study post "How I Host This Blog" — docker-compose sample + real install gotchas (from commit history)
|
||||
- ✅ Locale scheme: English flat in `posts/`, Chinese in `posts/zh/` (lang derived from folder)
|
||||
- ✅ Language switcher in nav — links to the **same post** in the other language (auto-matches by `zh/` prefix)
|
||||
- ✅ Locale-aware nav labels (EN/ZH)
|
||||
- ✅ Dark mode default (light opt-in via toggle)
|
||||
- ✅ Case study "How I Host This Blog"
|
||||
- ✅ Chinese translation of the case study (`posts/zh/how-i-host-this-blog.md`)
|
||||
- ✅ hello-world intro post — "What I write about" skills map + corrected business framing
|
||||
- ✅ Business framing corrected everywhere: website design & development = primary; email hosting = secondary (not the focus)
|
||||
- ✅ Knowledge guides in `docs/` (content, design, seo, ops, post-guideline) + README index + `hoelee-blog` skill
|
||||
- ✅ hello-world intro post
|
||||
- ✅ Case study "How I Built the DigiKedai Telegram AI Bot" + Chinese twin (`posts/zh/how-i-built-the-digikedai-telegram-bot/`)
|
||||
- ✅ Business framing corrected (website design & development = primary; email hosting = secondary)
|
||||
- ✅ Knowledge guides in `docs/` + README index + `hoelee-blog` skill
|
||||
- ✅ Both repos public (Gitea + GitHub) with title/description/homepage/topics + `v1.0.0` release
|
||||
|
||||
---
|
||||
|
||||
## Tier 1 — Content (80% of value; do this first)
|
||||
## 2. Execution Plan (work top → bottom, one step at a time)
|
||||
|
||||
- ⬜ **Write the 2 flagship case studies** — highest ROI, these are the portfolio:
|
||||
- ⬜ "How I Built the DigiKedai Telegram AI Bot" (DSM Docker + Cloudflare tunnel webhook + LiteLLM)
|
||||
- ⬜ "Self-Hosting a Mem0 Memory Stack" (API + LiteLLM + pgvector)
|
||||
- ⬜ **2–3 gotcha posts** from real debugging history (short, Google-friendly):
|
||||
- ⬜ "The Traefik forward-auth gotcha that cost me a day"
|
||||
- ⬜ "Site-to-site OpenVPN behind CGNAT"
|
||||
- ⬜ "Fixing the WordPress /cv 301→404 chain" (from own audit)
|
||||
- ⬜ **"Hard job → post" habit** — every solved problem becomes a `notes` entry the same week (template in `docs/content-guide.md` §4)
|
||||
> This plan comes from a full research pass (Sept 2026) comparing blog.hoelee.com against reference developer blogs (Simon Willison, Josh Comeau, Dan Abramov/overreacted, Julia Evans) + industry surveys. Priority is fixed: **identity & content → discovery → polish.** Don't reorder unless the user says so.
|
||||
|
||||
### Phase A — Identity (highest ROI, ~2–3 hrs total)
|
||||
|
||||
**Step A1 — Add a real author photo (headshot).**
|
||||
The #1 gap vs. every reference blog: the avatar is a letter "M" placeholder and there's no photo anywhere on the site. Every credible personal dev blog has a human face.
|
||||
- [ ] User provides one headshot (square, ≥800×800 for avatar; also source for og).
|
||||
- [ ] Replace `avatar` letter with the photo in: nav/brand (optional), author card (About + every post), `ProfilePage` JSON-LD `image`.
|
||||
- [ ] Add the photo to `og-default.png` template so the default share card has a face.
|
||||
- **Governing doc:** `design-guide.md` §2 (author box), §5 (E-E-A-T name+photo consistency).
|
||||
- **Done when:** a real face renders in the author card on About + every post; `curl` shows no 404 for the asset.
|
||||
|
||||
**Step A2 — Align the homepage title & hero framing.**
|
||||
The `<title>` says "engineering, DevOps & self-hosting" but the hero says "full-stack developer and DevOps engineer" — two slightly different framings.
|
||||
- [ ] Pick one line (recommend: "full-stack developer & DevOps engineer") and use it in both `<title>`/meta description and hero paragraph.
|
||||
- **Done when:** homepage title, meta description, and hero all say the same thing about who Hoelee is.
|
||||
|
||||
**Step A3 — Add a "Start here" / featured posts route.**
|
||||
New visitors land on reverse-chronological "Latest posts" with no guidance to the best content (Julia Evans' Favorites, Josh Comeau's featured posts both solve this).
|
||||
- [ ] Add a "Start here" (or "Featured") section on the homepage surfacing 2–3 flagship case studies.
|
||||
- [ ] Optionally add a `/favorites` or `/start-here` page (defer the dedicated page until ≥6 strong posts; the homepage strip is the immediate win).
|
||||
- **Done when:** homepage shows a featured/start-here strip above or beside "Latest posts".
|
||||
|
||||
### Phase B — Content (80% of value; the long game)
|
||||
|
||||
**Step B1 — Write the 2nd flagship case study: "Self-Hosting a Mem0 Memory Stack".**
|
||||
The Mem0 flagship is already the single highest-value unwritten post in the backlog.
|
||||
- [ ] Write `src/content/posts/self-hosting-mem0.md` (category `case-studies`).
|
||||
- [ ] Write Chinese twin `src/content/posts/zh/self-hosting-mem0.md` (same filename → auto language-switch).
|
||||
- [ ] Follow the "hard job → post" template (§4 content-guide) + open with "why it matters" + end with hire CTA (§8 post-guideline).
|
||||
- **Governing doc:** `content-guide.md` §4/§8, `post-guideline.md` §8.
|
||||
- **Done when:** both EN + ZH pages live, language-switch works, hire CTA present.
|
||||
|
||||
**Step B2 — Write 2–3 short "gotcha" posts (Google-friendly, compound over time).**
|
||||
- [ ] "The Traefik forward-auth gotcha that cost me a day"
|
||||
- [ ] "Site-to-site OpenVPN behind CGNAT"
|
||||
- [ ] "Fixing the WordPress /cv 301→404 chain" (from own audit)
|
||||
- **Governing doc:** `content-guide.md` §3 (post type #3), `post-guideline.md`.
|
||||
- **Done when:** ≥2 gotcha posts live (these are `notes`/`devops`, no Chinese translation required per §8).
|
||||
|
||||
**Step B3 — Adopt the "hard job → post" habit.**
|
||||
Every solved problem becomes a `notes` entry the same week.
|
||||
- [ ] Revisit cadence target: 2 posts/month → 1/week (`content-guide.md` §5).
|
||||
- **Done when:** 3 consecutive months hit the 2-posts/month floor.
|
||||
|
||||
### Phase C — Discovery & structure (Tier 2)
|
||||
|
||||
**Step C1 — Per-post custom OG images (at least for case studies).**
|
||||
Currently every post shares the generic 14KB `og-default.png` — flagship posts share the same bland card as category pages.
|
||||
- [ ] Build a branded 1200×630 OG template (name + face + title).
|
||||
- [ ] Generate a custom `ogImage` for each case study (frontmatter `ogImage:` field already supported).
|
||||
- **Governing doc:** `design-guide.md` §2/§3, `content-guide.md` §6 (ogImage field).
|
||||
- **Done when:** each case study's `og:image` is unique and 1200×630.
|
||||
|
||||
**Step C2 — Tag pages** (`/tags/[tag]/` archive pages for fine-grained discovery + internal linking).
|
||||
- [ ] Add tag archive routes (tags currently render as labels only).
|
||||
- **Done when:** clicking a tag on any post opens a working `/tags/<tag>/` page.
|
||||
|
||||
**Step C3 — Categories page shows all 7 categories** (not just those with posts), with "0 posts / coming soon" for empty ones — signals intended coverage.
|
||||
- **Done when:** `/categories/` lists all 7 categories including empty ones with a placeholder.
|
||||
|
||||
**Step C4 — Dedicated `/zh/posts/` and `/zh/categories/` archive pages.**
|
||||
- [ ] Split zh nav from `/zh/` landing into real archives once Chinese content grows (≥4 zh posts).
|
||||
- **Done when:** zh nav links to real `/zh/posts/` + `/zh/categories/` archives.
|
||||
|
||||
### Phase D — Polish / later (Tier 3)
|
||||
|
||||
**Step D1 — Search** (AstroPaper-style fuzzy search). Low priority until >20 posts.
|
||||
**Step D2 — Google Search Console submission** — submit `sitemap-index.xml` for faster indexing.
|
||||
**Step D3 — Newsletter / email capture** — only after real traffic exists (agree: do NOT add yet).
|
||||
|
||||
---
|
||||
|
||||
## Tier 2 — Structural gaps
|
||||
## 3. Research Findings Snapshot (Sept 2026)
|
||||
|
||||
- ⬜ **Tag pages** — tags currently render as labels only; add `/tags/[tag]/` archive pages for fine-grained discovery + internal linking
|
||||
- ⬜ **Categories page shows all 7 categories** (not just those with posts) — signal intended coverage; show "0 posts / coming soon" for empty ones
|
||||
- ⬜ **Search** — AstroPaper-style fuzzy search (low priority until >20 posts)
|
||||
- ⬜ **Dedicated `/zh/posts/` and `/zh/categories/` archive pages** — currently zh nav links point to `/zh/` landing; split into real archives when Chinese content grows
|
||||
What the reference blogs do that blog.hoelee.com should mirror, ranked:
|
||||
|
||||
| Finding | Reference example | Status on blog.hoelee.com |
|
||||
|---|---|---|
|
||||
| Real name + photo + one-line identity | All four | ⚠️ name ✅, photo ❌ (letter "M") — **Step A1** |
|
||||
| Focused thesis (one sentence on what it's about) | Julia Evans, Simon Willison | ⚠️ has it, but title/hero drift — **Step A2** |
|
||||
| Honesty about what you *don't* know | Simon, Dan Abramov | ✅ strong (DigiKedai "bugs that ate an afternoon") |
|
||||
| Specific detail: code, diagrams, numbers, bug stories | All four | ✅ strong |
|
||||
| Consistent cadence (slow is fine, dead is not) | Julia (~monthly), Simon (daily) | ⚠️ only 3 posts, all Sept 4–6 — **Phase B** |
|
||||
| "Start here" / Favorites route | Julia Evans, Josh Comeau | ❌ — **Step A3** |
|
||||
| RSS + sitemap + clean SEO | All four | ✅ |
|
||||
| Per-post OG images | Josh Comeau | ❌ — **Step C1** |
|
||||
| Search (once >15–20 posts) | Josh Comeau | ❌ deferred — **Step D1** |
|
||||
|
||||
---
|
||||
|
||||
## Tier 3 — Polish / later
|
||||
|
||||
- ⬜ **Google Search Console submission** — submit `sitemap-index.xml` for faster indexing
|
||||
- ⬜ **Newsletter / email capture** — only after real traffic exists
|
||||
|
||||
---
|
||||
|
||||
## Conventions (non-negotiable)
|
||||
## 4. Conventions (non-negotiable)
|
||||
|
||||
- Push git.hoelee.com first, then GitHub
|
||||
- English-first; Chinese selective (2–3 flagship case studies); no Malay
|
||||
- No overclaiming, especially Web3
|
||||
- Name identity: "Lee Teong Hoe" / "Mr Hoelee" + same photo + same `sameAs` handles everywhere
|
||||
- Business framing: website design & development is primary; email hosting is secondary
|
||||
- See `docs/post-guideline.md` for post-writing rules (title case, locale structure, frontmatter, etc.)
|
||||
- English post titles use Title Case
|
||||
- See `docs/post-guideline.md` for post-writing rules; `docs/content-guide.md` for strategy; `docs/design-guide.md` for UI
|
||||
|
||||
@@ -0,0 +1,210 @@
|
||||
---
|
||||
title: "Why Telegram Bot Notifications Die: IPv6, DNS, and a 400 That Took a Day"
|
||||
description: "A debugging story: a Dockerized telegram monitor whose sendPhoto calls returned 400 from nginx/1.30.1 while getMe worked fine — an IPv6-only DNS answer, a wrong extra_hosts IP, and a multipart join that corrupted JPEG bytes."
|
||||
pubDate: 2026-09-09
|
||||
category: devops
|
||||
tags: [docker, telegram, python, dns, ipv6, debugging, portainer]
|
||||
---
|
||||
|
||||
I run a small monitoring bot that watches Carousell for newly-listed
|
||||
listings and pings me on Telegram when something fresh appears. One day it
|
||||
stopped pinging. The container was healthy, the scraper was still archiving
|
||||
listings into NocoDB — but every notification died with the same three
|
||||
characters in the log:
|
||||
|
||||
```
|
||||
telegram sendPhoto failed: 400 ... nginx/1.30.1
|
||||
```
|
||||
|
||||
That string, `nginx/1.30.1`, became the villain of a day-long debugging
|
||||
session. This is the story of how a deceptively small bug turned out to be
|
||||
three separate problems stacked on top of each other — and how I'd find each
|
||||
one faster next time.
|
||||
|
||||
## The problem
|
||||
|
||||
The bot had two jobs: **collect** listings, and **notify** me. Collection
|
||||
worked. Notification didn't. The log said `400 Bad Request` and pointed at a
|
||||
server identifying itself as `nginx/1.30.1`.
|
||||
|
||||
My first assumption was textbook: **something in the network is intercepting
|
||||
the request.** `nginx/1.30.1` is not a header I associated with Telegram's
|
||||
API. A 400 with an HTML body (Telegram returns JSON even on errors) smelled
|
||||
like a local reverse proxy or a VPN gateway rejecting traffic before it ever
|
||||
left the machine.
|
||||
|
||||
Assumptions like that are cheap. They're also wrong a lot. Let me walk
|
||||
through what I actually found.
|
||||
|
||||
## What I tried, and why each thing failed
|
||||
|
||||
### Round 1: it must be DNS / IPv6
|
||||
|
||||
I checked `getMe` from inside the container. It worked. Then I checked
|
||||
`sendMessage`. It failed. Same token, same container, one method up, one down.
|
||||
|
||||
The giveaway was in how the two endpoints resolved:
|
||||
|
||||
```
|
||||
# inside the container
|
||||
$ getent hosts api.telegram.org
|
||||
2001:67c:4e8:f004::9 api.telegram.org
|
||||
```
|
||||
|
||||
The container resolved `api.telegram.org` to **IPv6 only** — a single `AAAA`
|
||||
record and no `A` record. The container ran on a Docker bridge network with
|
||||
**no IPv6 connectivity**. So every request that needed `api.telegram.org` was
|
||||
trying to reach an IPv6 address it had no route to.
|
||||
|
||||
The fix looked obvious: pin the IPv4 address with `extra_hosts`. I did, and
|
||||
the DNS answer became exactly what I expected:
|
||||
|
||||
```
|
||||
149.154.167.220 api.telegram.org
|
||||
```
|
||||
|
||||
And the notifications **still failed.** Same 400. My proud fix did nothing.
|
||||
|
||||
### Round 2: I picked the wrong IP
|
||||
|
||||
Here's a mistake worth remembering: I hard-coded `149.154.167.220` from
|
||||
memory. When I finally queried a public resolver, the real `A` record for
|
||||
`api.telegram.org` was different:
|
||||
|
||||
```
|
||||
149.154.166.110 api.telegram.org # what DNS actually says
|
||||
149.154.167.220 api.telegram.org # what I hard-coded
|
||||
```
|
||||
|
||||
`167.220` is *inside* Telegram's IP range (`149.154.160.0/20`), so it's not
|
||||
"wrong" in the sense of being somebody else's server — but it's not the
|
||||
active endpoint, and bot-API traffic to it behaved unpredictably. I switched
|
||||
to `166.110`. It still failed. So the IP wasn't the whole story either.
|
||||
|
||||
### Round 3: the same command, two different outcomes
|
||||
|
||||
This was the moment the whole thing got weird. Inside one container, at
|
||||
almost the same second:
|
||||
|
||||
- the **long-running monitor process** (`PID 1`) → `sendPhoto` failed, 400
|
||||
- a **fresh `docker exec` process** running identical code → `sendPhoto`
|
||||
succeeded, 200
|
||||
|
||||
Same container. Same code. Same payload. Same second.
|
||||
|
||||
I ruled out token, chat_id, emoji in the caption, Unicode quotes, image
|
||||
format, environment variables, and the image URL itself — I download-tested
|
||||
the photo and it was a perfectly valid JPEG. Nothing in the code differed
|
||||
between the two paths.
|
||||
|
||||
When you hit a contradiction like that — "identical inputs, opposite
|
||||
outputs" — the answer is usually that the inputs *aren't* identical. But
|
||||
finding the difference took one more frame change.
|
||||
|
||||
### Round 4: `nginx/1.30.1` is Telegram
|
||||
|
||||
I re-read the raw 400 response instead of assuming. The response was an HTML
|
||||
"400 Bad Request" page, and the `Server` header was `nginx/1.30.1`.
|
||||
|
||||
**That nginx is Telegram's own edge.** Telegram's API sits behind nginx
|
||||
(version 1.30.1), and when a request is malformed *before* it reaches their
|
||||
application layer, nginx itself answers with a plain 400 page — no JSON, no
|
||||
friendly error code.
|
||||
|
||||
That reframed everything. The request wasn't being intercepted. It was
|
||||
**reaching Telegram, and Telegram's nginx was rejecting it as malformed.**
|
||||
|
||||
And the malformed thing was the one part I hadn't scrutinized: the
|
||||
`sendPhoto` multipart body.
|
||||
|
||||
## The fix
|
||||
|
||||
My original code built the multipart body like this — the kind of
|
||||
"obviously fine" line that ships a subtle bug:
|
||||
|
||||
```python
|
||||
body = b"\r\n".join(body_lines)
|
||||
```
|
||||
|
||||
That `join` looks convenient, but a JPEG is **binary data**. Its bytes include
|
||||
`\r\n` byte sequences in the middle of the image data. Joining with `\r\n`
|
||||
as the delimiter corrupts the exact bytes that are supposed to be the image
|
||||
payload. Telegram receives a body whose multipart boundaries are broken, so
|
||||
its nginx rejects the request with a 400 before the bot API ever parses it.
|
||||
|
||||
The correct approach is to assemble the body field by field, keeping binary
|
||||
data untouched:
|
||||
|
||||
```python
|
||||
boundary = "----tg" + token_hex(8)
|
||||
|
||||
parts = []
|
||||
parts.append(f"--{boundary}\r\n".encode())
|
||||
parts.append(b'Content-Disposition: form-data; name="chat_id"\r\n\r\n')
|
||||
parts.append(f"{chat_id}\r\n".encode())
|
||||
|
||||
parts.append(f"--{boundary}\r\n".encode())
|
||||
parts.append(b'Content-Disposition: form-data; name="caption"\r\n\r\n')
|
||||
parts.append(f"{caption}\r\n".encode())
|
||||
|
||||
parts.append(f"--{boundary}\r\n".encode())
|
||||
parts.append(
|
||||
b'Content-Disposition: form-data; name="photo"; '
|
||||
b'filename="image.jpg"\r\nContent-Type: image/jpeg\r\n\r\n'
|
||||
)
|
||||
parts.append(image_bytes) # ← binary stays binary
|
||||
parts.append(b"\r\n")
|
||||
|
||||
parts.append(f"--{boundary}--\r\n".encode())
|
||||
|
||||
body = b"".join(parts) # join bytes, not lines
|
||||
```
|
||||
|
||||
The difference is `b"".join(parts)` instead of `b"\r\n".join(...)`: each part
|
||||
is already a complete chunk with its own framing, and we concatenate them
|
||||
verbatim rather than inserting a delimiter *between every element*.
|
||||
|
||||
There's a second, higher-level fix worth naming: **stop asking Telegram to
|
||||
download the image for you.** My earlier code passed the image as a *URL* and
|
||||
let Telegram fetch it:
|
||||
|
||||
```python
|
||||
tg("sendPhoto", {"photo": image_url, "caption": caption})
|
||||
```
|
||||
|
||||
That means Telegram's servers have to reach out to the source CDN — and if
|
||||
that CDN is flaky or geo-blocks Telegram's crawlers, you get intermittent
|
||||
failures that are impossible to reproduce locally. Downloading the image
|
||||
yourself, then uploading the bytes, removes an entire class of flakiness and
|
||||
gives you a chance to validate the bytes before you send them.
|
||||
|
||||
## What I'd do differently
|
||||
|
||||
1. **Read the raw response, not the assumption.** `nginx/1.30.1` sat in logs
|
||||
for hours while I chased phantom proxies. Look up who owns the header
|
||||
before inventing a man-in-the-middle.
|
||||
2. **Verify a hard-coded IP against a live resolver.** Memory is not DNS.
|
||||
`dig +short api.telegram.org` takes three seconds and would have saved me
|
||||
a whole round.
|
||||
3. **Treat "identical inputs, opposite outputs" as a lie.** The inputs were
|
||||
never identical — the two processes differed somewhere I wasn't looking.
|
||||
The honest move is to diff the bytes, not the code.
|
||||
4. **Never hand-join binary data with a text delimiter.** `b"\r\n".join()` on
|
||||
a mix of text and JPEG is a corruption bug waiting to happen.
|
||||
|
||||
## The result
|
||||
|
||||
After the multipart fix, the monitor pushed a real notification through on
|
||||
the next tick — image and all. I also folded in a retry queue so a failed
|
||||
notification stays marked un-sent and is retried 30 seconds later instead of
|
||||
silently vanishing. The whole episode cost the better part of a day, but it
|
||||
turned into exactly the kind of bug I'll never trip over twice.
|
||||
|
||||
---
|
||||
|
||||
*The bot is a personal project, but the lesson generalizes to any service
|
||||
that relies on outbound notifications. If you've got a monitor, a scraper, or
|
||||
an alert pipeline that needs to reach people reliably, let's talk — I build
|
||||
and fix these for a living.*
|
||||
|
||||
**WhatsApp +60 12-797 2969 · me@hoelee.com · hoelee.com**
|
||||
@@ -0,0 +1,192 @@
|
||||
---
|
||||
title: "Telegram 机器人通知为什么发不出去：IPv6、DNS 和一个折腾了一整天的 400"
|
||||
description: "一次排障记录：一个跑在 Docker 里的 Telegram 监控机器人，sendPhoto 一直返回 nginx/1.30.1 的 400，但 getMe 却正常——根因是只返回 IPv6 的 DNS 应答、一个写错的 extra_hosts IP，以及一个用 join 拼接破坏了 JPEG 二进制数据的 multipart body。"
|
||||
pubDate: 2026-09-09
|
||||
category: devops
|
||||
tags: [docker, telegram, python, dns, ipv6, debugging, portainer]
|
||||
---
|
||||
|
||||
我跑着一个小监控机器人，专门盯着 Carousell 上新上架的商品，一旦有新品就通过
|
||||
Telegram 通知我。有一天它突然不再通知了。容器本身是健康的，爬虫也照常往
|
||||
NocoDB 里归档商品——但每一条通知都以日志里那三个字符宣告失败：
|
||||
|
||||
```
|
||||
telegram sendPhoto failed: 400 ... nginx/1.30.1
|
||||
```
|
||||
|
||||
`nginx/1.30.1` 这个字符串，成了我接下来一整天排障的"反派"。这是一个看似
|
||||
微不足道的 bug，实际却是由三个问题层层堆叠而成的故事——以及下一次我会
|
||||
如何更快地逐个定位它们。
|
||||
|
||||
## 问题本身
|
||||
|
||||
这个机器人有两件事要做：**采集**商品，以及**通知**我。采集正常，通知挂了。
|
||||
日志显示 `400 Bad Request`，并指向一个自称 `nginx/1.30.1` 的服务器。
|
||||
|
||||
我的第一反应很教科书：**网络里有什么东西在拦截请求。** 因为
|
||||
`nginx/1.30.1` 这个 header 跟我印象里的 Telegram API 对不上——而且一个
|
||||
带 HTML 响应体的 400（Telegram 就算出错也返回 JSON）看起来就像本地反向
|
||||
代理或 VPN 网关在请求还没离开机器之前就把它拒掉了。
|
||||
|
||||
这种假设很省力，但也常常是错的。下面说说我到底发现了什么。
|
||||
|
||||
## 我尝试了什么，以及为什么每个都失败了
|
||||
|
||||
### 第一轮：一定是 DNS / IPv6
|
||||
|
||||
我先在容器里跑了 `getMe`——它能通。然后又跑了 `sendMessage`——它挂了。
|
||||
同一个 token、同一个容器，一个方法通、一个方法挂。
|
||||
|
||||
线索藏在两个端点的解析方式里：
|
||||
|
||||
```
|
||||
# 容器内部
|
||||
$ getent hosts api.telegram.org
|
||||
2001:67c:4e8:f004::9 api.telegram.org
|
||||
```
|
||||
|
||||
容器把 `api.telegram.org` 解析成了**只有 IPv6**——一条 `AAAA` 记录，
|
||||
没有 `A` 记录。而容器跑在一个 **没有 IPv6 连通性的** Docker bridge
|
||||
网络上。于是每个需要访问 `api.telegram.org` 的请求，都在试图连接一个
|
||||
它根本走不到的 IPv6 地址。
|
||||
|
||||
修复看起来很简单：用 `extra_hosts` 把 IPv4 地址钉死。我照做了，DNS
|
||||
应答变得完全符合预期：
|
||||
|
||||
```
|
||||
149.154.167.220 api.telegram.org
|
||||
```
|
||||
|
||||
结果通知**还是失败**，同一个 400。我那得意的修复啥也没改变。
|
||||
|
||||
### 第二轮：我选错了 IP
|
||||
|
||||
这里有一个值得记住的教训：我是**凭记忆**硬编码了 `149.154.167.220`。
|
||||
等我终于去查公共解析器时，`api.telegram.org` 真正的 `A` 记录并不是它：
|
||||
|
||||
```
|
||||
149.154.166.110 api.telegram.org # DNS 实际返回的
|
||||
149.154.167.220 api.telegram.org # 我硬编码的
|
||||
```
|
||||
|
||||
`167.220` 确实**在** Telegram 的 IP 段内（`149.154.160.0/20`），所以它
|
||||
并不是"别人的服务器"这种错——但它不是当前活跃的端点，bot-API 的流量打到
|
||||
它上面的行为是不可预测的。我把它换成 `166.110`，仍然失败。所以 IP 也
|
||||
不是全部原因。
|
||||
|
||||
### 第三轮：同样的命令，两个相反的结果
|
||||
|
||||
这是整个事情开始变得诡异的一刻。在同一个容器里，几乎同一秒：
|
||||
|
||||
- **常驻的监控进程**（`PID 1`）→ `sendPhoto` 失败，400
|
||||
- 一个**新起的 `docker exec` 进程**跑着完全相同的代码 → `sendPhoto`
|
||||
成功，200
|
||||
|
||||
同一个容器、同样的代码、同样的 payload、同一秒。
|
||||
|
||||
我逐一排除了 token、chat_id、caption 里的 emoji、Unicode 引号、图片格式、
|
||||
环境变量、以及图片 URL 本身——我把那张图下载下来测过，是一张完全合法的
|
||||
JPEG。两条路径的代码没有任何差别。
|
||||
|
||||
当你撞上这种"输入相同、输出相反"的矛盾时，答案通常是：**输入其实并不
|
||||
相同**。但找到那个差异，还需要我再换一次思路。
|
||||
|
||||
### 第四轮：`nginx/1.30.1` 就是 Telegram
|
||||
|
||||
我重新读了原始的 400 响应，而不是去猜。响应是一张 HTML 的
|
||||
"400 Bad Request" 页面，`Server` 头是 `nginx/1.30.1`。
|
||||
|
||||
**那个 nginx 就是 Telegram 自家的边缘服务器。** Telegram 的 API 背后就是
|
||||
nginx（版本 1.30.1），当一个请求在到达它们应用层**之前**就已经畸形时，
|
||||
nginx 会自己回一张 400 页面——没有 JSON，也没有友好的错误码。
|
||||
|
||||
这一下彻底改变了整个视角。请求并没有被拦截，而是**确实到达了 Telegram，
|
||||
然后被 Telegram 的 nginx 以"畸形"为由拒绝了**。
|
||||
|
||||
而那个畸形的东西，正是我一直没仔细看的部分：`sendPhoto` 的 multipart
|
||||
请求体。
|
||||
|
||||
## 修复
|
||||
|
||||
我最初的代码是这样拼 multipart 请求体的——正是那种"看着显然没问题"、
|
||||
实则埋下微妙 bug 的一行：
|
||||
|
||||
```python
|
||||
body = b"\r\n".join(body_lines)
|
||||
```
|
||||
|
||||
这个 `join` 看起来很省事，但 JPEG 是**二进制数据**。它的字节里本身就包含
|
||||
`\r\n` 这样的字节序列，出现在图片数据中间。用 `\r\n` 作为分隔符去 join，
|
||||
就把本该是图片 payload 的字节给破坏了。Telegram 收到的是一个 multipart
|
||||
边界被破坏的请求体，于是它的 nginx 在 bot API 还没来得及解析之前就回了个
|
||||
400。
|
||||
|
||||
正确的做法是逐字段拼装 body，让二进制数据保持原样：
|
||||
|
||||
```python
|
||||
boundary = "----tg" + token_hex(8)
|
||||
|
||||
parts = []
|
||||
parts.append(f"--{boundary}\r\n".encode())
|
||||
parts.append(b'Content-Disposition: form-data; name="chat_id"\r\n\r\n')
|
||||
parts.append(f"{chat_id}\r\n".encode())
|
||||
|
||||
parts.append(f"--{boundary}\r\n".encode())
|
||||
parts.append(b'Content-Disposition: form-data; name="caption"\r\n\r\n')
|
||||
parts.append(f"{caption}\r\n".encode())
|
||||
|
||||
parts.append(f"--{boundary}\r\n".encode())
|
||||
parts.append(
|
||||
b'Content-Disposition: form-data; name="photo"; '
|
||||
b'filename="image.jpg"\r\nContent-Type: image/jpeg\r\n\r\n'
|
||||
)
|
||||
parts.append(image_bytes) # ← 二进制保持二进制
|
||||
parts.append(b"\r\n")
|
||||
|
||||
parts.append(f"--{boundary}--\r\n".encode())
|
||||
|
||||
body = b"".join(parts) # join 的是字节块，不是行
|
||||
```
|
||||
|
||||
区别在于用 `b"".join(parts)` 而不是 `b"\r\n".join(...)`：每一块本身就是
|
||||
一个带完整边框的片段，我们只是把它们的字节**原样拼接**，而不是在
|
||||
**每个元素之间**都插入一个分隔符。
|
||||
|
||||
还有一个更高层的修复值得点名：**别再让 Telegram 替你去下载图片。** 我
|
||||
早期的代码是把图片当作 *URL* 传过去，让 Telegram 自己去抓：
|
||||
|
||||
```python
|
||||
tg("sendPhoto", {"photo": image_url, "caption": caption})
|
||||
```
|
||||
|
||||
这意味着 Telegram 的服务器得主动去访问源 CDN——如果那个 CDN 不稳定、
|
||||
或者对 Telegram 的爬虫做了地域封锁，你就会得到本地根本无法复现的间歇性
|
||||
失败。自己先把图片下载下来、再上传字节，就消除了一整类的不稳定因素，
|
||||
还给了你在发送前校验字节的机会。
|
||||
|
||||
## 如果重来一次我会怎么做
|
||||
|
||||
1. **读原始响应，而不是读假设。** `nginx/1.30.1` 在日志里躺了好几个小时，
|
||||
我却在追一个根本不存在的中间人。在凭空捏造拦截者之前，先查清楚那个
|
||||
header 到底属于谁。
|
||||
2. **对硬编码的 IP 用实时解析器验证。** 记忆不是 DNS。`dig +short
|
||||
api.telegram.org` 只要三秒钟，就能省掉我整整一轮。
|
||||
3. **把"输入相同、输出相反"当成谎言。** 输入从来就不相同——两个进程在某
|
||||
个我没看到的地方有差异。诚实的做法是去 diff 字节，而不是 diff 代码。
|
||||
4. **永远不要用文本分隔符去手工拼接二进制数据。** 把文本和 JPEG 混在一起
|
||||
做 `b"\r\n".join()`，迟早会出破坏 bug。
|
||||
|
||||
## 结果
|
||||
|
||||
multipart 修复之后，监控机器人在下一个 tick 就真正把通知发出去了——
|
||||
图片和文字都在。我还一并加了一个重试队列：一条失败的通知会保持"未发送"
|
||||
状态，30 秒后自动重试，而不是悄无声息地消失。整件事耗掉了大半天的功夫，
|
||||
但它变成了那种我这辈子不会再踩第二次的坑。
|
||||
|
||||
---
|
||||
|
||||
*这个机器人是我个人的项目，但教训可以推广到任何依赖外发通知的服务。如果你
|
||||
有一个监控、抓取，或者需要可靠地触达用户的告警管线，我们可以聊聊——这就
|
||||
是我日常工作里搭建和修复的东西。*
|
||||
|
||||
**WhatsApp +60 12-797 2969 · me@hoelee.com · hoelee.com**
|
||||
Reference in New Issue
Block a user