diff --git a/src/content/posts/scraping-bot-walled-marketplace-warm-browser-session.md b/src/content/posts/scraping-bot-walled-marketplace-warm-browser-session.md
new file mode 100644
index 0000000..5b7d987
--- /dev/null
+++ b/src/content/posts/scraping-bot-walled-marketplace-warm-browser-session.md
@@ -0,0 +1,165 @@
+---
+title: "Scraping a Bot-Walled Marketplace With a Warm Browser Session"
+description: "How I scraped Shopee's bot-walled listings with a warm headless-Chrome session over CDP — fail-fast health checks, split-wait rendering, and why I refused to parallelize it."
+pubDate: 2026-09-13
+category: engineering
+tags: ["scraping", "cdp", "headless-chrome", "anti-bot", "shopee", "python"]
+ogImage: "/og/scraping-bot-walled-marketplace-warm-browser-session.png"
+draft: false
+---
+
+I set out to do something that sounds simple: list all the second-hand M.2 NVMe SSDs on a marketplace, sorted by price, so I could buy the cheapest one.
+
+The result was nothing like a clean `requests` + BeautifulSoup script. It was a week-long fight against an anti-bot wall, a warm browser session held together with duct tape, and — the part I didn't expect — a hard lesson about why you cannot just "parallelize" a scraper by throwing more agents at it.
+
+This is that story, told in the order I lived it: the problem, everything I tried that failed, the thing that finally worked, and what I'd do differently.
+
+## The problem, phrased the way someone would actually search it
+
+> "How do I scrape product listings from a site that blocks headless browsers?"
+
+Or, more honestly: **how do you scrape a marketplace that has already made scraping its sole job description?** Shopee doesn't politely serve HTML to `curl`. It detects automation, throws a `/verify` captcha, and silently serves you a page that looks fine but contains nothing.
+
+My requirement list was small:
+
+- Search for "used SSD", filter to "used condition only", filter to M.2 NVMe (not SATA/mSATA), filter to ≥128 GB.
+- Extract every listing's capacity variants and prices.
+- Sort by price ascending.
+- Do it without a `chromedriver` Farm of a hundred headless instances, because I don't own a hundred residential IPs.
+
+Everything after this point is the debugging story.
+
+## What I tried, and why it failed
+
+### Attempt 1: cold headless Chrome via CDP — "just point Chrome's remote debugger at it"
+
+I launched Chrome with `--headless --remote-debugging-port=9222`, connected with the Chrome DevTools Protocol, and navigated to a search URL. It felt clever for about ten seconds.
+
+**Failure:** the page loaded, but every product card was empty. The DOM was a shell. Shopee had fingerprinted the headless browser (no real GPU, no real user-agent fingerprint consistency, `navigator.webdriver` flag) and redirected me into a `/verify` captcha loop I never got past. Cold headless profiles hit the wall immediately.
+
+The lesson is boring but important: **anti-bot systems don't care how good your selectors are if the browser itself is lying about what it is.**
+
+### Attempt 2: parallel subagents — "just spawn N workers"
+
+This was the one I really wanted to work. I have a multi-agent setup. The natural instinct is: spawn five agents, each scraping a different search query, gather the results, merge.
+
+**Failure:** all of them attach to the *same* browser session. They don't get their own browser — they get their own *tab* in one shared Chrome instance, or worse, they all fight over the same single tab. Every `Page.navigate` un-does the previous agent's context. And critically:
+
+1. **The warm session is a shared resource.** There is one Canary profile that isn't flagged. Five agents navigated the same tab in an interleaved order, which is indistinguishable — to Shopee — from one browser doing rapid-fire navigation. That's exactly the behavior that *triggers* the wall.
+2. **Anti-bot pacing is cumulative.** My working setup depends on ~7-second gaps between navigations to look human. Five agents navigating concurrently means five navigations hitting the same session with effectively zero gap.
+
+So the "horizontal scaling" instinct — more workers — is actively hostile to the one thing keeping me alive: looking human. The concurrency ceiling here isn't CPU or memory. It's **one warm browser, one tab at a time.**
+
+### Attempt 3: trusting the "Buy With Voucher" button for the price
+
+Once I had listings rendering, I assumed the buy button's label contained the price. On most marketplaces it does. On this render, the button said `Add To Cart` / `Buy Now`. My regex for "Buy With Voucher … RM…" matched nothing, and I went around in circles for a while before actually reading the DOM.
+
+**Failure:** the price on Shopee product pages isn't in the button. It's rendered as **split spans** — `RM` in one element, `84` in another, `00` in a third — so a naive leaf-text match returns fragments, or a **range** (`RM125.00 - RM155.00`) when the listing has multiple variants.
+
+### Attempt 4: clicking variant options to get an exact price
+
+For a single-axis variant selector (e.g. one dropdown of capacities — 128GB / 256GB / 512GB), clicking one option collapses the price to a single value, and my extraction worked.
+
+**Failure:** for a **two-axis** selector (brand × capacity), clicking the first axis (a brand) locks a price, but adding the second axis (a capacity) *un-collapses* it back to the product-wide range. The panel renders the range of all purchasable combinations, not the exact combination I selected, and the "selected" state is marked by a class that also matches every other option box. There is no reliable way to know which combination I actually locked.
+
+I spent real time on this before accepting the honest answer: **for genuinely two-axis listings, the correct data is the price range, not a guessed per-combination number.** Shipping a fragile override that "sometimes works" would be worse than reporting an honest range.
+
+## The fix — what actually worked
+
+Three pieces, all small, all learned the hard way.
+
+### 1. A warm, human-paced browser session (the foundation)
+
+Don't fight the wall. Use a browser profile that has already logged in and browsed normally — a "warm" profile. Navigate with real 7-second gaps. It is slower and it is the only thing that reliably survives.
+
+```python
+# not real code — the *shape* of it. Navigate, then wait like a human.
+goto_url(target)
+time.sleep(7) # human pacing, non-negotiable
+wait_for_selector(".shopee-search-item-result__item")
+```
+
+### 2. Fail fast, don't time out
+
+Before any scrape, hit the CDP HTTP endpoint and check the session state. If Canary is down or the tab is sitting on `/verify`, abort in ~1 second instead of wasting 40 seconds on a "did not render" timeout.
+
+```python
+import json, urllib.request
+
+def session_healthy():
+ try:
+ tabs = json.load(urllib.request.urlopen("http://127.0.0.1:9222/json"))
+ except Exception:
+ return False
+ for t in tabs:
+ if t.get("type") == "page" and "/verify" in t.get("url", ""):
+ return False # captcha wall — re-warm the session, don't scrape
+ return True
+```
+
+This one change turned "silent 40-second dead-ends" into "instant, actionable failure."
+
+### 3. The right price selector: deepest pure-RM node
+
+The buy button is a lie. The price is the **deepest element whose text is a pure RM value** — a range (`RM125.00 - RM155.00`) before a variant is selected, a single value (`RM84.00`) after. Walk the DOM and take the deepest element whose text matches:
+
+```python
+import re
+
+RM = re.compile(r"^RM\s?[\d,.]+(\s?[-–]\s?RM\s?[\d,.]+)?$")
+
+def find_price(root):
+ best, depth = None, -1
+ for el in root.querySelectorAll("*"):
+ txt = (el.textContent or "").strip()
+ if txt and RM.match(txt) and " " not in txt.split("RM")[0]:
+ d = depth_of(el)
+ if d > depth:
+ best, depth = el, d
+ return best.textContent.strip() if best else None
+```
+
+That's the whole trick. No brittle class names. No guessing. A node whose entire text *is* a price is the price.
+
+## The result — one quantified outcome
+
+From a single warm session, in one serial pass, I extracted **20+ used-NVMe listings with per-variant prices**, filtered to M.2 NVMe ≥128GB, sorted by price. The cheapest real target — a 256GB M.2 NVMe — surfaced at **RM84**, alongside several 256GB options in the RM125–175 range. One command (`sweep.py ...`) now re-checks the whole shortlist and dumps JSONLines.
+
+But the number I'm most glad about is this: **zero captchas.** The warm-session + human-pacing + fail-fast combo survived the entire sweep without tripping `/verify` once.
+
+## What I'd do differently
+
+1. **I'd write the fail-fast health check first**, before a single scraping line. It's thirty seconds of work that saves forty seconds per future failure.
+2. **I'd resist the parallelization instinct day one.** The answer to "the scraper is slow" is "bundle more work into one serial pass" (batch search, `sweep.py` over many URLs), not "more workers." Shared browser state makes concurrency worse, not better.
+3. **I'd accept the two-axis limitation immediately** and report ranges, instead of burning an afternoon on a selector that can't be made reliable.
+4. **I'd stop guessing DOM structure and probe it.** Almost every mistaken assumption (`/i.` vs `-i.` in the URL path, `Buy With Voucher` as the price anchor, "the filter is a real `` checkbox") came from *not reading the actual DOM first*.
+
+## The reusable pieces
+
+All of this became a small, standalone toolkit — each script is one self-contained `python x.py ...` call:
+
+| Script | What it does |
+|---|---|
+| `search.py` | Coarse search → listing cards |
+| `used_search.py` | Clicks the "Used" condition checkbox, verifies `USED_ITEM` in the URL, extracts |
+| `detail.py` | Pulls per-variant prices from a product page (deepest-RM-node method + split-wait) |
+| `sweep.py` | Batch detail over N URLs in one session, dedupes by item-id, writes JSONLines, prints a table |
+| `cdp_common.py` | Shared health check + navigation helpers |
+
+The genuinely reusable lesson isn't the Shopee-specific selectors — it's the *posture*: **assume the page is lying about its DOM, fail fast, and scale by batching, not by parallel workers.**
+
+---
+
+*This is a real debugging session, not a tutorial written after the fact. The scraping toolkit and every one of these failures happened while actually hunting for a cheap SSD — I kept the parts that generalize and cut the parts that only matter to one Malaysian marketplace at one point in time.*
+
+---
+
+## Want this for your business?
+
+I build custom scraping and data-extraction pipelines — warm-session CDP scrapers, bot-wall-aware crawlers, and batch price-monitoring sweeps — as well as full-stack web apps (PHP/CodeIgniter, Java/Spring, React/TypeScript) and self-hosted automation (n8n, Telegram bots, Docker on NAS).
+
+If you need product or price data off a site that fights back, tap through and tell me what you're trying to extract:
+
+- 💬 [WhatsApp](https://wa.me/60127972969) — message me directly
+- ✉️ [Email](mailto:me@hoelee.com?subject=Custom%20scraping%20pipeline) — tell me the site + the data
+- 🌐 [hoelee.com](https://hoelee.com) — see what else I build
\ No newline at end of file
diff --git a/src/content/posts/zh/scraping-bot-walled-marketplace-warm-browser-session.md b/src/content/posts/zh/scraping-bot-walled-marketplace-warm-browser-session.md
new file mode 100644
index 0000000..6c48808
--- /dev/null
+++ b/src/content/posts/zh/scraping-bot-walled-marketplace-warm-browser-session.md
@@ -0,0 +1,165 @@
+---
+title: "用「热浏览器会话」爬取有反爬墙的电商网站"
+description: "我如何用热 headless-Chrome 会话通过 CDP 爬取 Shopee 的反爬商品列表——失败即止的健康检查、分段等待渲染，以及我为什么拒绝并行化处理。"
+pubDate: 2026-09-13
+category: engineering
+tags: ["scraping", "cdp", "headless-chrome", "anti-bot", "shopee", "python"]
+ogImage: "/og/scraping-bot-walled-marketplace-warm-browser-session.png"
+draft: false
+---
+
+我一开始想做一件听起来很简单的事：把一个电商网站上所有的二手 M.2 NVMe SSD 都列出来，按价格排序，好让我买到最便宜的那一个。
+
+结果完全不是一段干净的 `requests` + BeautifulSoup 脚本。它变成了一场长达一周的、跟反爬墙的拉锯战，一个靠胶带勉强粘在一起的热浏览器会话，以及——最让我意外的部分——一个关于「你不能靠堆更多 agent 来『并行化』爬虫」的惨痛教训。
+
+这就是那个故事，按照我经历的顺序来讲：问题、我试过却失败的每一件事、最终奏效的办法，以及下次我会怎么做。
+
+## 问题，用别人真正会搜索的方式表述
+
+> "如何从一个拦截无头浏览器的网站爬取商品列表？"
+
+或者更诚实地说：**当一个电商网站已经把「防爬」当成它唯一的本职工作，你要怎么爬它？** Shopee 不会礼貌地把 HTML 喂给 `curl`。它会检测自动化、抛出一个 `/verify` 验证码，然后静默地给你一个「看起来正常但里面什么都没有」的页面。
+
+我的需求清单很短：
+
+- 搜索「二手 SSD」，只筛选「二手」，只筛选 M.2 NVMe（排除 SATA/mSATA），只筛选 ≥128 GB。
+- 提取每个商品的所有容量规格和价格。
+- 按价格升序排序。
+- 不搞一百个 headless 实例的 `chromedriver` 农场，因为我没有一百个住宅 IP。
+
+从这里开始，全部都是调试故事。
+
+## 我试过的，以及为什么失败
+
+### 尝试 1：通过 CDP 用冷的无头 Chrome——「直接把 Chrome 远程调试开起来」
+
+我用 `--headless --remote-debugging-port=9222` 启动了 Chrome，用 Chrome DevTools Protocol 连上去，然后导航到搜索 URL。大约十秒钟里我觉得自己很聪明。
+
+**失败：** 页面加载了，但每个商品卡片都是空的。DOM 是个空壳。Shopee 指纹识别了这个无头浏览器（没有真正的 GPU、user-agent 指纹不一致、`navigator.webdriver` 标志），把我重定向进了一个我永远过不去的 `/verify` 验证码循环。冷的无头 profile 一上来就撞墙。
+
+教训很无聊但很重要：**如果你的浏览器本身在「身份」上撒谎，选择器写得再好也没用。**
+
+### 尝试 2：并行子 agent——「直接开 N 个 worker」
+
+这是我最想让它成功的方案。我有一套多 agent 的架构。本能反应就是：开五个 agent，每个爬一个不同的搜索词，收集结果，合并。
+
+**失败：** 它们全都连到**同一个**浏览器会话上。它们不会各自拿到一个浏览器——只会各自拿到共享 Chrome 实例里的一个标签页，或者更糟：全都挤在同一个标签页上。每一次 `Page.navigate` 都会把上一个 agent 的上下文抹掉。而且关键在于：
+
+1. **热会话是共享资源。** 只有一个没被标记的 Canary profile。五个 agent 交错访问同一个标签页，在 Shopee 眼里，跟一个浏览器疯狂快速导航没有区别。而这恰恰是**触发**反爬墙的行为。
+2. **反爬的「节奏」是累积的。** 我的可用方案依赖导航之间约 7 秒的间隔来显得像真人。五个 agent 并发导航，就等于五个导航以零间隔打向同一个会话。
+
+所以「横向扩展」的本能——加更多 worker——对我赖以存活的那件事（看起来像真人）是有害的。这里的并发上限不是 CPU 也不是内存，而是**一个热浏览器，一次一个标签页。**
+
+### 尝试 3：靠「Buy With Voucher」按钮来拿价格
+
+一旦列表能渲染出来了，我假设购买按钮的文案里带着价格。大多数电商网站确实如此。但这个渲染出来的按钮写的是 `Add To Cart` / `Buy Now`。我用来匹配「Buy With Voucher … RM…」的正则什么都匹配不到，我在原地打转了好一会儿，才真正去读了 DOM。
+
+**失败：** Shopee 商品页的价格不在按钮里。它被渲染成**拆分的 span**——`RM` 在一个元素里、`84` 在另一个里、`00` 又在另一个里——所以朴素的叶子文本匹配只会抓到碎片，或者当商品有多个规格时，抓到一个**区间**（`RM125.00 - RM155.00`）。
+
+### 尝试 4：点击规格选项来拿精确价格
+
+对于单轴规格选择器（比如一个容量下拉：128GB / 256GB / 512GB），点击一个选项会把价格收敛成单个值，我的提取方法能正常工作。
+
+**失败：** 对于**双轴**选择器（品牌 × 容量），点击第一个轴（某个品牌）会锁定一个价格，但加上第二个轴（某个容量）又把它**解散**回商品全区间。这个面板渲染的是「所有可购买组合」的区间，而不是我选中的那个具体组合；而且「已选中」状态是用一个「同时匹配其他所有选项框」的 class 来标记的。我没法可靠地知道我到底锁定了哪个组合。
+
+我在这个上面花了实打实的时间，最后才接受这个诚实的答案：**对于真正的双轴商品，正确的数据是价格区间，而不是一个瞎猜的「每个组合」数字。** 发布一个「有时候能用」的脆弱覆盖方案，比老实报告一个区间更糟糕。
+
+## 修复——真正奏效的东西
+
+三块，都很小，都是吃了亏才学会的。
+
+### 1. 热的人性化节奏的浏览器会话（地基）
+
+别跟墙硬刚。用一个已经登录过、正常浏览过的浏览器 profile——一个「热」profile。用真实的 7 秒间隔来导航。它更慢，但它是唯一能稳定活下来的方式。
+
+```python
+# 不是真正的代码——是它的「形状」。先导航，然后像真人一样等待。
+goto_url(target)
+time.sleep(7) # 真人节奏，不可妥协
+wait_for_selector(".shopee-search-item-result__item")
+```
+
+### 2. 失败即止，别傻等超时
+
+在任何爬取之前，先打 CDP 的 HTTP 端点，检查会话状态。如果 Canary 挂掉了、或标签页停在 `/verify` 上，就在约 1 秒内中止，而不是在「怎么还不渲染」的超时里浪费 40 秒。
+
+```python
+import json, urllib.request
+
+def session_healthy():
+ try:
+ tabs = json.load(urllib.request.urlopen("http://127.0.0.1:9222/json"))
+ except Exception:
+ return False
+ for t in tabs:
+ if t.get("type") == "page" and "/verify" in t.get("url", ""):
+ return False # 撞到验证码墙——重新热会话，别爬了
+ return True
+```
+
+就这么一个改动，把「静默的 40 秒死胡同」变成了「即时、可执行的失败」。
+
+### 3. 正确的价格选择器：最深的纯 RM 节点
+
+购买按钮是个谎言。价格是**那个文本内容纯粹是一个 RM 值的、最深的元素**——在选中规格之前是一个区间（`RM125.00 - RM155.00`），之后是单个值（`RM84.00`）。遍历 DOM，取文本能匹配上的最深元素：
+
+```python
+import re
+
+RM = re.compile(r"^RM\s?[\d,.]+(\s?[-–]\s?RM\s?[\d,.]+)?$")
+
+def find_price(root):
+ best, depth = None, -1
+ for el in root.querySelectorAll("*"):
+ txt = (el.textContent or "").strip()
+ if txt and RM.match(txt) and " " not in txt.split("RM")[0]:
+ d = depth_of(el)
+ if d > depth:
+ best, depth = el, d
+ return best.textContent.strip() if best else None
+```
+
+这就是全部的诀窍。没有脆弱的 class 名。没有瞎猜。一个「整个文本就是一个价格」的节点，就是价格本身。
+
+## 结果——一个可量化的成果
+
+从单个热会话、一次串行遍历中，我提取了 **20+ 个二手 NVMe 商品及其各规格价格**，筛选到 M.2 NVMe ≥128GB，按价格升序排序。最便宜的真实目标——一块 256GB M.2 NVMe——以 **RM84** 的价格浮出水面，另外还有几块 256GB 的选项在 RM125–175 区间。现在一条命令（`sweep.py ...`）就能重新检查整份候选清单，并输出 JSONLines。
+
+但我最庆幸的数字是这个：**零验证码。** 「热会话 + 人性化节奏 + 失败即止」这套组合，在整次遍历中一次都没触发过 `/verify`。
+
+## 下次我会怎么做
+
+1. **我会一开始就写「失败即止」的健康检查**，在任何爬取代码之前。三十秒的活，换来未来每次失败省四十秒。
+2. **我会从第一天就抵抗并行化的冲动。** 「爬虫太慢」的答案是把更多工作打包进一次串行遍历（批量搜索、对多个 URL 跑 `sweep.py`），而不是「再加 worker」。共享的浏览器状态让并发变得更糟，而不是更好。
+3. **我会立刻接受双轴的限制**，老实报告区间，而不是在一个根本没法可靠化的选择器上烧一个下午。
+4. **我会别再猜 DOM 结构，先去探测它。** 几乎每一个错误假设（URL 路径里 `/i.` vs `-i.`、把 `Buy With Voucher` 当价格锚点、「过滤器是个真正的 `` 复选框」）都来自**没有先读真实的 DOM**。
+
+## 可复用的部分
+
+所有这一切被做成了一个小的独立工具集——每个脚本都是一次自包含的 `python x.py ...` 调用：
+
+| 脚本 | 作用 |
+|---|---|
+| `search.py` | 粗粒度搜索 → 商品卡片 |
+| `used_search.py` | 点击「二手」条件复选框，校验 URL 里的 `USED_ITEM`，再提取 |
+| `detail.py` | 从商品页提取各规格价格（最深 RM 节点法 + 分段等待） |
+| `sweep.py` | 一次会话内对 N 个 URL 做批量详情提取，按 item-id 去重，写 JSONLines，打印表格 |
+| `cdp_common.py` | 共享的健康检查 + 导航辅助 |
+
+真正可复用的教训不是那些 Shopee 专属的选择器——而是**姿态**：**假设页面在 DOM 上撒谎，失败即止，用批量化而不是并行 worker 来扩展。**
+
+---
+
+*这是一次真实的调试过程，不是事后补写的教程。这个爬虫工具集、以及这里的每一次失败，都发生在我真的为了买一块便宜 SSD 而爬取的时候——我保留了能泛化的部分，砍掉了那些只对某个马来西亚电商网站、某个时间点有效的部分。*
+
+---
+
+## 想为你的业务做这个吗？
+
+我做定制爬虫和数据提取管道——热会话 CDP 爬虫、反爬墙感知的采集器、批量价格监控扫描——也做全栈 Web 应用（PHP/CodeIgniter、Java/Spring、React/TypeScript）和自托管自动化（n8n、Telegram 机器人、NAS 上的 Docker）。
+
+如果你需要从某个「会反抗」的网站拿商品或价格数据，直接点进来，告诉我你想提取什么：
+
+- 💬 [WhatsApp](https://wa.me/60127972969) — 直接私信我
+- ✉️ [Email](mailto:me@hoelee.com?subject=Custom%20scraping%20pipeline) — 告诉我网站和数据
+- 🌐 [hoelee.com](https://hoelee.com) — 看看我还做了些什么
\ No newline at end of file