Add two posts: headless-browser scraping limits + NocoDB backfill gotchas

- why-your-headless-browser-cant-scrape-everything (case study, en+zh)
- the-nocodb-attachment-that-wouldnt-update (note, en+zh)
- og + banner images for both, generator entries added
- gitignore .og/ scratch dir (also removes previously-committed scratch files)

src/content/posts/the-nocodb-attachment-that-wouldnt-update.md

@@ -0,0 +1,93 @@
+---
+title: "The NocoDB Attachment That Wouldn't Update (and a Regex That Missed)"
+description: "Two small gotchas from backfilling 300 rows: NocoDB ignores a path change when the attachment keeps its id, and a lookahead regex silently skipped URLs without a file extension."
+pubDate: 2026-09-10
+category: notes
+tags: [nocodb, backfill, python, regex, data-pipeline]
+ogImage: /og/the-nocodb-attachment-that-wouldnt-update.png
+banner: /banners/the-nocodb-attachment-that-wouldnt-update.png
+---
+
+Backfilling 300 rows in NocoDB surfaced two small gotchas that each ate a chunk of
+time. Both are the kind of thing that only bites you when the data you're fixing
+is *almost* — but not quite — in the shape the code assumes.
+
+## Gotcha 1: updating an Attachment's `path` doesn't matter if you keep the `id`
+
+I wanted to swap every thumbnail URL in an Attachment column for its full-size
+version. The obvious move is to edit the JSON the column stores:
+
+```python
+img = json.loads(row["image"])          # [{"id": "...", "path": "...", "signedPath": "..."}]
+img[0]["path"] = new_fullsize_url
+img[0]["signedPath"] = new_fullsize_url
+```
+
+It looked like it worked — the PATCH returned success. But reading the rows back,
+every `path` was still the old thumbnail.
+
+The culprit: the attachment carries an NocoDB-generated `id`, and as long as it's
+present, NocoDB resolves the attachment *by that id* and happily ignores the new
+`path`. The field I thought I was updating was effectively read-only while the `id`
+was still there.
+
+The fix is to drop the server-assigned fields and let NocoDB re-register the
+attachment from scratch:
+
+```python
+img[0] = {
+    "path": new_fullsize_url,
+    "mimetype": "image/jpeg",
+    "title": img[0].get("title", "image.jpg"),
+}
+# no "id", no "signedPath" — NocoDB assigns fresh ones
+```
+
+With `id` gone, NocoDB registers a new attachment from the `path` and generates a
+fresh signed URL. The value finally sticks.
+
+**Rule:** if a NocoDB Attachment edit isn't sticking, check for the `id` field.
+Strip it and re-register instead of patching in place.
+
+## Gotcha 2: a lookahead regex that silently skipped extension-less URLs
+
+To build the full-size URL I needed to strip the `_progressive_thumbnail` suffix.
+My first pass used a lookahead:
+
+```python
+clean = re.sub(r"_progressive_thumbnail(?=\.\w+$)", "", url)
+```
+
+That works for:
+
+```
+...70776f54_progressive_thumbnail.jpg   →  ...70776f54.jpg   ✅
+```
+
+But Carousell serves *two* URL shapes. Some thumbnails end with the bare suffix
+and **no extension**:
+
+```
+...f94af259_progressive_thumbnail      →  (lookahead doesn't match)  ❌
+```
+
+Because the lookahead requires `.\w+` at the end, the second shape slipped through
+unmatched — 25 of my 300 rows. The regex "worked," it just silently didn't do
+anything on that slice of the data.
+
+The robust version is a plain replacement, which handles both:
+
+```python
+clean = url.replace("_progressive_thumbnail", "")
+```
+
+**Rule:** when normalizing a string that has multiple real-world shapes, prefer
+`replace()` over a regex unless you've enumerated every shape. A regex that
+silently skips is worse than one that fails loudly.
+
+---
+
+Nothing here is exotic. But both are the *boring* kind of bug that only reveals
+itself in real data — the attachment the API swears it updated, the regex that
+matches 91% of the rows and says nothing about the other 9%. If backfilling data
+into NocoDB ever feels like it's half-working, these two are where I'd look first.

src/content/posts/why-your-headless-browser-cant-scrape-everything.md

@@ -0,0 +1,160 @@
+---
+title: "Why Your Headless Browser Can't Scrape Everything"
+description: "Two second-hand marketplaces, two very different walls: Goofish blocks a headless browser with 'illegal access' while Carousell keeps serving data. The lesson is to read how a page delivers data before you reach for Playwright."
+pubDate: 2026-09-10
+category: case-studies
+tags: [scraping, playwright, browserless, docker, goofish, carousell, anti-bot]
+ogImage: /og/why-your-headless-browser-cant-scrape-everything.png
+banner: /banners/why-your-headless-browser-cant-scrape-everything.png
+---
+
+I hit the same wall a few weeks apart, on two different second-hand marketplaces,
+and the contrast taught me more about scraping than a year of tutorials did.
+
+On **Carousell**, a headless browser pulled freshly-listed products into a database
+with prices, conditions, and photo URLs — no problem, running for months. On
+**Goofish (闲鱼)**, the exact same setup returned the same three words every time:
+
+```
+为了保障您的体验，请使用正常浏览器访问闲鱼~
+(translation: "to protect your experience, please use a normal browser")
+```
+
+Two marketplaces, one browser, opposite outcomes. This is the story of *why*, and
+the one decision that would have saved me most of a day if I'd made it first.
+
+## The goal, badly framed
+
+I wanted to search Goofish for products and pull the results out — titles, prices,
+links. The obvious first answer is "use a headless browser." It's the answer
+everyone gives: render the page with Playwright or Puppeteer, wait for the
+JavaScript, read the DOM.
+
+So I built exactly that. I deployed a self-hosted `browserless` container — a
+ready-made headless-Chrome-as-an-API — and pointed it at Goofish search. Then I
+spent hours trying every lever the tool exposes:
+
+- `--disable-blink-features=AutomationControlled` to hide the automation flag
+- patching `navigator.webdriver` to `undefined` via `evaluateOnNewDocument`
+- a realistic Chrome user-agent
+- a `stealth` launch option
+
+Each time, the product list stayed stuck at "loading…" — the page frame rendered,
+the header rendered, the footer rendered, but the actual listings never populated.
+The HTML came back with zero prices and zero product links.
+
+Meanwhile I had a memory nagging me: my Carousell monitor grabs the same kind of
+data with *no* stealth, *no* anti-detection tricks, and it works flawlessly. Why?
+
+## The frame change: read how the data gets to the page
+
+The answer wasn't in the browser flags. It was in **how each site delivers its data**.
+
+**Carousell embeds its search results in the page's initial payload.** The server
+sends the listings as JSON baked into the HTML (a `__PRELOADED_STATE__`-style blob).
+The JavaScript enhances them afterward, but a plain HTTP request already contains
+everything — product ID, title, price, photos. A scraper can parse that JSON out of
+the raw HTML without ever rendering anything.
+
+**Goofish delivers nothing server-side.** Its search results are fetched *after*
+the page loads, from a signed API endpoint. That endpoint requires:
+
+- a signature the app computes from request parameters and a secret
+- anti-bot checks that flag a headless browser's fingerprint (WebDriver detection,
+  CDP artifacts, canvas/audio fingerprints)
+
+Strip the signature away, and the API returns nothing. Strip the browser off, and
+the page returns "illegal access."
+
+That single distinction — **server-rendered JSON vs. a signed async API** — is the
+whole game. It tells you in one minute whether "render the page in a browser" will
+take you an afternoon or a month.
+
+## Why the headless browser hit a wall on Goofish
+
+The honest reality: **an open-source, self-hosted browser is not a stealth tool.**
+
+`browserless` (the container I deployed) is a *generic* renderer. Its free tier
+renders pages, executes JavaScript, takes screenshots, converts PDFs. The
+anti-detection features — real `stealth` mode, captcha solving, unblocked proxies —
+are explicitly the **paid Cloud/Enterprise** tier. The message in their docs is
+unambiguous.
+
+So I was asking a general-purpose renderer to beat a platform whose entire
+business depends on defeating automated clients. That's not a configuration bug;
+it's a mismatch between the tool and the adversary.
+
+To actually scrape Goofish I'd have to choose one of a few much heavier paths:
+
+1. **Reverse the app's signing algorithm** and call the search API directly. This
+   is how the serious scrapers do it — but the signature changes, so it's an
+   ongoing maintenance cost, not a one-time build.
+2. **Buy a third-party data API** (Alibaba Cloud's marketplace and similar sell
+   `item_search` endpoints). Clean data, but you pay per call and you're at the
+   mercy of the vendor.
+3. **Drive a real, logged-in browser at low frequency** and hope the rate stays
+   under the radar. Works for occasional manual checks, useless for reliable
+   monitoring at scale.
+
+None of these are "spin up a container and go." That's the point.
+
+## The reusable checklist
+
+Before I touch Playwright again on a new target, I now answer three questions in
+order:
+
+1. **Where does the data actually live?** Open the page with JavaScript disabled
+   (or read the raw HTML). If the data is in the HTML, I don't need a browser at
+   all — I need an HTTP client and a parser. The browser is waste.
+2. **If it's not in the HTML, what does the async request look like?** Open
+   DevTools → Network, find the XHR/fetch that returns the data, and read its
+   headers and query params. Is there a signature? Is it stable, or does it change
+   per request?
+3. **Is there a signed API?** If yes, stop. Decide up front whether the target is
+   worth either (a) reverse-engineering the sign, or (b) paying a data vendor. If
+   neither, the honest answer is "this costs more than it's worth" — and *knowing*
+   that in the first hour is itself a win.
+
+Most scraping guides skip all three and jump straight to "install Puppeteer."
+That's the expensive path, the one that burns a day discovering the hard way what
+a 60-second investigation would have told you.
+
+## What I'd do differently
+
+- **Disable JavaScript before reaching for a browser.** It's the single fastest
+  signal for "server-rendered vs. client-fetched." Ten seconds, no setup.
+- **Don't buy stealth from a generic renderer.** Stealth is either a product you
+  pay for (browserless Cloud, ScrapingBee, etc.) or something you build and
+  maintain. It is not a checkbox in an open-source container.
+- **Distinguish "rendering" from "extraction."** A headless browser returns HTML;
+  it does not hand you structured product fields. Even when rendering works,
+  someone still has to write the selectors or the `evaluate` script. That's a
+  separate task — plan for it, don't assume it's free.
+
+## The result
+
+The contrast sharpened into a rule I now apply to every project: **decide whether
+the target is server-rendered or API-gated before choosing a tool.** On Carousell
+(server-rendered), a plain HTTP parser has archived thousands of listings with
+prices and photos for months. On Goofish (signed API), I stopped after an honest
+look at the effort and the maintenance burden — which is itself the right
+outcome for a project where a manual search once in a while does everything I
+needed.
+
+A headless browser is the right tool *sometimes* — for screenshots, PDFs, taking
+over a human session, or targets that render server-side. It is not a universal
+scraping key, and treating it as one is the expensive mistake.
+
+---
+
+## Want me to look at a site before you commit to scraping it?
+
+Most scraping projects fail in the first hour because nobody checked whether the
+target even *lets* data be scraped cheaply. I do that check before writing any
+code — and I build the monitors, notification pipelines, and self-hosted
+infrastructure that actually run. If you're eyeing a data source and want to know
+if it's an afternoon or a month, let's talk:
+
+- 📱 **WhatsApp:** [+60 12-797 2969](https://wa.me/60127972969)
+- 📧 **Email:** [[email protected]](mailto:[email protected])
+- 🌐 **Website:** [hoelee.com](https://hoelee.com)

src/content/posts/zh/the-nocodb-attachment-that-wouldnt-update.md

@@ -0,0 +1,88 @@
+---
+title: "NocoDB 附件字段怎么改都不生效，和一个漏网的正则"
+description: "回填 300 行数据时踩到的两个小坑：NocoDB 在附件保留 id 的情况下会无视 path 的改动，以及一个 lookahead 正则悄悄跳过了没有扩展名的 URL。"
+pubDate: 2026-09-10
+category: notes
+tags: [nocodb, backfill, python, regex, data-pipeline]
+ogImage: /og/the-nocodb-attachment-that-wouldnt-update.png
+banner: /banners/the-nocodb-attachment-that-wouldnt-update.png
+---
+
+在 NocoDB 里回填 300 行数据的经历，暴露了两个小坑，每一个都耗掉了我一段时间。
+它们都属于那种「只有当你修的数据恰好接近、但又没完全符合代码假设的形状时」
+才会咬你的 bug。
+
+## 坑 1：Attachment 改了 `path` 没用，只要你还留着 `id`
+
+我想把一个 Attachment 列里的缩略图 URL 全部换成高清版。直觉上就是去改这个
+列存的 JSON：
+
+```python
+img = json.loads(row["image"])          # [{"id": "...", "path": "...", "signedPath": "..."}]
+img[0]["path"] = new_fullsize_url
+img[0]["signedPath"] = new_fullsize_url
+```
+
+它看起来成了——PATCH 返回成功。但把行读回来，每个 `path` 还是旧的缩略图。
+
+元凶是：这个附件带着一个 NocoDB 生成的 `id`，只要它还在，NocoDB 就会**按这个
+id 去解析附件**，然后心安理得地无视新的 `path`。我以为自己在更新的那个字段，
+其实是只读的，只要 `id` 还在。
+
+修复的办法是丢掉服务端分配的字段，让 NocoDB 从头重新注册这个附件：
+
+```python
+img[0] = {
+    "path": new_fullsize_url,
+    "mimetype": "image/jpeg",
+    "title": img[0].get("title", "image.jpg"),
+}
+# 不要 "id"，也不要 "signedPath" —— NocoDB 会重新分配
+```
+
+没有 `id` 之后，NocoDB 会根据 `path` 注册一个新附件，并生成一个新的签名 URL。
+这下值才真正写进去了。
+
+**规则：** 如果 NocoDB 的 Attachment 改来改去都不生效，检查一下 `id` 字段。
+把它剥掉、重新注册，而不是原地打补丁。
+
+## 坑 2：一个会悄悄跳过「无扩展名 URL」的 lookahead 正则
+
+为了拼出高清 URL，我需要去掉 `_progressive_thumbnail` 这个后缀。第一版我用了
+一个 lookahead：
+
+```python
+clean = re.sub(r"_progressive_thumbnail(?=\.\w+$)", "", url)
+```
+
+这对这种情况有效：
+
+```
+...70776f54_progressive_thumbnail.jpg   →  ...70776f54.jpg   ✅
+```
+
+但 Carousell 有**两种** URL 形态。有些缩略图以光秃秃的后缀结尾，**没有扩展名**：
+
+```
+...f94af259_progressive_thumbnail      →  (lookahead 匹配不上)  ❌
+```
+
+因为 lookahead 要求结尾是 `.\w+`，第二种形态就漏掉了——300 行里有 25 行。
+这个正则「没报错」，它只是在那部分数据上什么都没做。
+
+更稳的写法是直接 `replace`，两种形态都能覆盖：
+
+```python
+clean = url.replace("_progressive_thumbnail", "")
+```
+
+**规则：** 当你要规范化一个有多种真实形态的字符串时，除非你已经把所有形态都
+枚举清楚了，否则优先用 `replace()` 而不是正则。一个悄悄跳过的正则，比一个
+直接报错的更糟。
+
+---
+
+这里头没什么高深的东西。但这两类都是那种**只有在真实数据里才会现形**的无聊
+bug——一个 API 信誓旦旦说更新成功了的附件，一个匹配了 91% 行、却对剩下 9%
+一声不吭的正则。如果你给 NocoDB 回填数据时总觉得「只生效了一半」，先来查这两
+个地方。

src/content/posts/zh/why-your-headless-browser-cant-scrape-everything.md

@@ -0,0 +1,142 @@
+---
+title: "为什么你的无头浏览器不是万能爬虫"
+description: "两个二手交易平台，两面完全不同的墙：闲鱼用「非法访问」把无头浏览器挡在门外，Carousell 却照常吐数据。教训是——在伸手拿 Playwright 之前，先搞清页面到底是怎么把数据交给你的。"
+pubDate: 2026-09-10
+category: case-studies
+tags: [scraping, playwright, browserless, docker, goofish, carousell, anti-bot]
+ogImage: /og/why-your-headless-browser-cant-scrape-everything.png
+banner: /banners/why-your-headless-browser-cant-scrape-everything.png
+---
+
+我在短短几周里，分别在两个不同的二手交易平台上撞上了同一堵墙，而这组对照
+教给我的，比学一年教程还多。
+
+在 **Carousell** 上，一个无头浏览器把新上架的商品连价格、成色、图片链接一起
+抓进数据库——毫无问题，稳定跑了几个月。在**闲鱼**上，一模一样的设置，每次都
+只返回同样三个词：
+
+```
+为了保障您的体验，请使用正常浏览器访问闲鱼~
+```
+
+两个平台，同一个浏览器，结果完全相反。这篇文章讲的就是**为什么**，以及那个
+本该在一开始就做、能帮我省下大半天时间的决定。
+
+## 一个从一开始就没问对的问题
+
+我想在闲鱼上搜索商品，把结果抽出来——标题、价格、链接。第一个跳出来的答案
+自然是「用无头浏览器」。每个人都会这么答：用 Playwright 或 Puppeteer 把页面
+渲染出来，等 JavaScript 执行完，再读 DOM。
+
+于是我就真这么搭了。我部署了一个自托管的 `browserless` 容器——一个现成的
+「无头 Chrome 即 API」，把它指向闲鱼的搜索页。然后我花了几个小时，把那个工具
+暴露出来的每个开关都试了个遍：
+
+- `--disable-blink-features=AutomationControlled` 隐藏自动化标志
+- 用 `evaluateOnNewDocument` 把 `navigator.webdriver` 抹成 `undefined`
+- 一个逼真的 Chrome user-agent
+- 一个 `stealth` 启动选项
+
+每一次，商品列表都卡在「加载中…」——页面框架渲染出来了，页头渲染出来了，
+页脚渲染出来了，但真正的商品列表永远加载不出来。返回的 HTML 里价格是 0 条，
+商品链接是 0 条。
+
+与此同时，脑子里有个声音一直在提醒我：我的 Carousell 监控器抓的也是同样的
+数据，而且**没有任何 stealth、没有任何反检测手段**，却一直稳稳地工作。为什么？
+
+## 换个思路：先看数据是怎么到页面上的
+
+答案根本不在浏览器的那些启动参数里，而在**每个网站是怎么把数据交给页面的**。
+
+**Carousell 是把搜索结果嵌在页面的初始响应里的。** 服务器把商品列表作为
+JSON 烤进了 HTML（类似 `__PRELOADED_STATE__` 这种结构）。JavaScript 之后再把
+它们增强一下，但一次普通的 HTTP 请求就已经包含了所有的东西——商品 ID、标题、
+价格、图片。爬虫甚至不需要渲染任何东西，直接从这个原始 HTML 里把 JSON 解析
+出来就行。
+
+**闲鱼什么都不在服务端给。** 它的搜索结果是在页面加载**之后**，才从一个带
+签名的 API 端点上拉回来的。这个端点要求：
+
+- 一个由 App 根据请求参数和一个密钥算出来的签名
+- 一系列会标记无头浏览器指纹的反爬检查（WebDriver 检测、CDP 痕迹、canvas/
+  audio 指纹）
+
+把签名去掉，API 就什么都不返回。把浏览器特征去掉，页面就回你一句「非法访问」。
+
+**服务端渲染的 JSON vs. 带签名的异步 API**——这一个区别，就是全部的游戏。
+它让你在一分钟内就判断出「用浏览器渲染」这条路是花一个下午，还是花一个月。
+
+## 无头浏览器为什么在闲鱼上撞了墙
+
+说句实在话：**开源的、自托管的浏览器本身不是隐身工具。**
+
+`browserless`（我部署的那个容器）是个**通用**渲染器。它的免费版能渲染页面、
+执行 JavaScript、截图、转 PDF。而反检测的能力——真正的 `stealth` 模式、验证码
+识别、解锁代理——明确是**付费的 Cloud/Enterprise 版**才有的。它们文档里写得
+毫不含糊。
+
+所以说，我是在拿一个通用渲染器，去硬刚一个整个商业模式都建立在「击败自动化
+客户端」之上的平台。这不是配置 bug，是工具和对手之间的错配。
+
+真要抓闲鱼，我有几条重得多的路可选：
+
+1. **逆向 App 的签名算法**，直接调搜索 API。严肃的爬虫都是这么干的——但签名
+   会变，所以这是持续的维护成本，不是一次性做完就完事。
+2. **买第三方数据接口**（阿里云市场之类卖 `item_search` 端点）。数据干净，但
+   按次计费，而且受制于供应商。
+3. **驱动一个真实登录的浏览器、低频访问**，指望频率始终低于雷达线。偶尔手动
+   查查还行，想稳定的大规模监控，没用。
+
+没有一条是「起个容器就能跑」的。这才是重点。
+
+## 可以复用的检查清单
+
+在给一个新目标动用 Playwright 之前，我现在会按顺序问自己三个问题：
+
+1. **数据到底存在哪里？** 关掉 JavaScript 打开页面（或者直接读原始 HTML）。
+   如果数据就在 HTML 里，我根本不需要浏览器——我需要的是一个 HTTP 客户端和一个
+   解析器。浏览器纯属浪费。
+2. **如果不在 HTML 里，那个异步请求长什么样？** 打开 DevTools → Network，找到
+   那个返回数据的 XHR/fetch，看它的 header 和查询参数。有没有签名？它是稳定的，
+   还是每个请求都变？
+3. **是不是带签名的 API？** 如果是，停下来。先想清楚这个目标值不值得 (a) 逆向
+   签名，或者 (b) 付费买数据供应商。如果都不值，诚实的答案就是「这事成本高于
+   收益」——能在一个小时内就**知道**这一点，本身就是种胜利。
+
+大多数爬虫教程会跳过这三步，直接跳到「装 Puppeteer」。那是昂贵的那条路——
+花一整天用最痛的方式，才搞懂一个 60 秒的调查早就该告诉你的事。
+
+## 如果重来一次我会怎么做
+
+- **在伸手拿浏览器之前，先禁用 JavaScript。** 这是判断「服务端渲染 vs. 客户端
+  拉取」最快的信号，十秒钟，零配置。
+- **别指望从通用渲染器里白嫖 stealth。** stealth 要么是你花钱买的产品
+  （browserless Cloud、ScrapingBee 之类），要么是你自己搭、自己维护的东西。它
+  不是一个开源容器里的勾选框。
+- **把「渲染」和「提取」分开看。** 无头浏览器返回的是 HTML，它不会直接给你
+  结构化的商品字段。就算渲染成功了，还是得有人写选择器或 `evaluate` 脚本。那是
+  另一件事——要为它做计划，别以为它是免费的。
+
+## 结果
+
+这组对照沉淀成了一条我现在每个项目都会用的规则：**在选工具之前，先判断目标是
+服务端渲染还是 API 门槛。** 在 Carousell（服务端渲染）上，一个普通的 HTTP 解析器
+几个月来归档了几千条带价格和图片的商品。在闲鱼（签名 API）上，我诚实地评估了
+工作量和维护成本之后选择了停下——对一个偶尔手动搜一下就够用的需求来说，这本身
+就是正确的结局。
+
+无头浏览器**有时候**是对的工具——截图、转 PDF、接管一个人工会话，或者目标本身
+就是服务端渲染。但它不是一把通用的爬虫钥匙，把它当成钥匙，才是那个昂贵的错误。
+
+---
+
+## 想知道一个网站值不值得花力气去抓？
+
+大多数爬虫项目都在第一个小时内失败，因为没人先去查一眼——这个目标到底允不
+允许你用低成本抓到数据。我写代码之前会先做这个判断——我也真的在搭那些能跑起来
+的监控器、通知管线，和背后的自托管基础设施。如果你盯上了一个数据源，想知道它是
+一个下午的活还是一个月的活，欢迎聊聊：
+
+- 📱 **WhatsApp:** [+60 12-797 2969](https://wa.me/60127972969)
+- 📧 **Email:** [[email protected]](mailto:[email protected])
+- 🌐 **Website:** [hoelee.com](https://hoelee.com)