diff --git a/public/og/scraping-bot-walled-marketplace-warm-browser-session.png b/public/og/scraping-bot-walled-marketplace-warm-browser-session.png index 002aac6..863f062 100644 Binary files a/public/og/scraping-bot-walled-marketplace-warm-browser-session.png and b/public/og/scraping-bot-walled-marketplace-warm-browser-session.png differ diff --git a/scripts/og-gen/generate.mjs b/scripts/og-gen/generate.mjs index 7852a43..d5183a8 100644 --- a/scripts/og-gen/generate.mjs +++ b/scripts/og-gen/generate.mjs @@ -120,7 +120,7 @@ const TERMINALS = {
$mount passbolt.php · full fingerprint→ fixed ✓
`, 'scraping-bot-walled-marketplace-warm-browser-session': ` -
$CDP → shopee search "used ssd"
+
$CDP → shopee search "used phone" · client marketplace monitor
 /verify captcha · empty product cards
$warm session · 7s pacing · sweep.py→ 20+ listings ✓
`, }; diff --git a/src/content/posts/scraping-bot-walled-marketplace-warm-browser-session.md b/src/content/posts/scraping-bot-walled-marketplace-warm-browser-session.md index 5b7d987..167b83f 100644 --- a/src/content/posts/scraping-bot-walled-marketplace-warm-browser-session.md +++ b/src/content/posts/scraping-bot-walled-marketplace-warm-browser-session.md @@ -1,6 +1,6 @@ --- title: "Scraping a Bot-Walled Marketplace With a Warm Browser Session" -description: "How I scraped Shopee's bot-walled listings with a warm headless-Chrome session over CDP — fail-fast health checks, split-wait rendering, and why I refused to parallelize it." +description: "Scraping bot-walled marketplace listings for a client's used-electronics business: warm CDP session, fail-fast health checks, split-wait rendering." pubDate: 2026-09-13 category: engineering tags: ["scraping", "cdp", "headless-chrome", "anti-bot", "shopee", "python"] @@ -8,9 +8,9 @@ ogImage: "/og/scraping-bot-walled-marketplace-warm-browser-session.png" draft: false --- -I set out to do something that sounds simple: list all the second-hand M.2 NVMe SSDs on a marketplace, sorted by price, so I could buy the cheapest one. +A client asked me to build something that sounds simple: every day, list the second-hand phones on a big marketplace, sorted by price, so their restocking decisions come from data instead of gut feel. The client resells used electronics for a living — a few hundred ringgit either way on a phone is the whole margin. -The result was nothing like a clean `requests` + BeautifulSoup script. It was a week-long fight against an anti-bot wall, a warm browser session held together with duct tape, and — the part I didn't expect — a hard lesson about why you cannot just "parallelize" a scraper by throwing more agents at it. +The result was nothing like a clean `requests` + BeautifulSoup script. It was a week-long fight against an anti-bot wall, a warm browser session held together with duct tape, and — the part neither of us saw coming — a hard lesson about why you cannot just "parallelize" a scraper by throwing more agents at it. This is that story, told in the order I lived it: the problem, everything I tried that failed, the thing that finally worked, and what I'd do differently. @@ -20,12 +20,12 @@ This is that story, told in the order I lived it: the problem, everything I trie Or, more honestly: **how do you scrape a marketplace that has already made scraping its sole job description?** Shopee doesn't politely serve HTML to `curl`. It detects automation, throws a `/verify` captcha, and silently serves you a page that looks fine but contains nothing. -My requirement list was small: +The client's requirement list was small: -- Search for "used SSD", filter to "used condition only", filter to M.2 NVMe (not SATA/mSATA), filter to ≥128 GB. -- Extract every listing's capacity variants and prices. +- Search for used phones, filter to "used condition only", storage 128 GB and up. +- Extract every listing's storage variants and prices. - Sort by price ascending. -- Do it without a `chromedriver` Farm of a hundred headless instances, because I don't own a hundred residential IPs. +- Do it without a `chromedriver` farm of a hundred headless instances — a reseller's margins don't pay for a hundred residential IPs, and neither do mine. Everything after this point is the debugging story. @@ -54,13 +54,13 @@ So the "horizontal scaling" instinct — more workers — is actively hostile to Once I had listings rendering, I assumed the buy button's label contained the price. On most marketplaces it does. On this render, the button said `Add To Cart` / `Buy Now`. My regex for "Buy With Voucher … RM…" matched nothing, and I went around in circles for a while before actually reading the DOM. -**Failure:** the price on Shopee product pages isn't in the button. It's rendered as **split spans** — `RM` in one element, `84` in another, `00` in a third — so a naive leaf-text match returns fragments, or a **range** (`RM125.00 - RM155.00`) when the listing has multiple variants. +**Failure:** the price on Shopee product pages isn't in the button. It's rendered as **split spans** — `RM` in one element, `850` in another, `00` in a third — so a naive leaf-text match returns fragments, or a **range** (`RM850.00 - RM1150.00`) when the listing has multiple variants. ### Attempt 4: clicking variant options to get an exact price -For a single-axis variant selector (e.g. one dropdown of capacities — 128GB / 256GB / 512GB), clicking one option collapses the price to a single value, and my extraction worked. +For a single-axis variant selector (e.g. one dropdown of storage options — 128GB / 256GB / 512GB), clicking one option collapses the price to a single value, and my extraction worked. -**Failure:** for a **two-axis** selector (brand × capacity), clicking the first axis (a brand) locks a price, but adding the second axis (a capacity) *un-collapses* it back to the product-wide range. The panel renders the range of all purchasable combinations, not the exact combination I selected, and the "selected" state is marked by a class that also matches every other option box. There is no reliable way to know which combination I actually locked. +**Failure:** for a **two-axis** selector (brand × storage), clicking the first axis (a brand) locks a price, but adding the second axis (a storage size) *un-collapses* it back to the product-wide range. The panel renders the range of all purchasable combinations, not the exact combination I selected, and the "selected" state is marked by a class that also matches every other option box. There is no reliable way to know which combination I actually locked. I spent real time on this before accepting the honest answer: **for genuinely two-axis listings, the correct data is the price range, not a guessed per-combination number.** Shipping a fragile override that "sometimes works" would be worse than reporting an honest range. @@ -101,7 +101,7 @@ This one change turned "silent 40-second dead-ends" into "instant, actionable fa ### 3. The right price selector: deepest pure-RM node -The buy button is a lie. The price is the **deepest element whose text is a pure RM value** — a range (`RM125.00 - RM155.00`) before a variant is selected, a single value (`RM84.00`) after. Walk the DOM and take the deepest element whose text matches: +The buy button is a lie. The price is the **deepest element whose text is a pure RM value** — a range (`RM850.00 - RM1150.00`) before a variant is selected, a single value (`RM850.00`) after. Walk the DOM and take the deepest element whose text matches: ```python import re @@ -123,7 +123,7 @@ That's the whole trick. No brittle class names. No guessing. A node whose entire ## The result — one quantified outcome -From a single warm session, in one serial pass, I extracted **20+ used-NVMe listings with per-variant prices**, filtered to M.2 NVMe ≥128GB, sorted by price. The cheapest real target — a 256GB M.2 NVMe — surfaced at **RM84**, alongside several 256GB options in the RM125–175 range. One command (`sweep.py ...`) now re-checks the whole shortlist and dumps JSONLines. +From a single warm session, in one serial pass, I extracted **20+ used-phone listings with per-variant prices**, filtered to 128 GB and up, sorted by price. The cheapest real target — a 128GB model — surfaced at **RM299**, alongside several 128/256GB options in the RM450–880 range. One command (`sweep.py ...`) now re-checks the whole shortlist and dumps JSONLines; the client runs it once a day before restocking. But the number I'm most glad about is this: **zero captchas.** The warm-session + human-pacing + fail-fast combo survived the entire sweep without tripping `/verify` once. @@ -150,7 +150,7 @@ The genuinely reusable lesson isn't the Shopee-specific selectors — it's the * --- -*This is a real debugging session, not a tutorial written after the fact. The scraping toolkit and every one of these failures happened while actually hunting for a cheap SSD — I kept the parts that generalize and cut the parts that only matter to one Malaysian marketplace at one point in time.* +*This is a real debugging session, not a tutorial written after the fact. The scraping toolkit and every one of these failures happened while building it for a client who resells used electronics — I kept the parts that generalize and cut the parts that only matter to one Malaysian marketplace at one point in time.* --- diff --git a/src/content/posts/zh/scraping-bot-walled-marketplace-warm-browser-session.md b/src/content/posts/zh/scraping-bot-walled-marketplace-warm-browser-session.md index 6c48808..4ba3c54 100644 --- a/src/content/posts/zh/scraping-bot-walled-marketplace-warm-browser-session.md +++ b/src/content/posts/zh/scraping-bot-walled-marketplace-warm-browser-session.md @@ -1,6 +1,6 @@ --- title: "用「热浏览器会话」爬取有反爬墙的电商网站" -description: "我如何用热 headless-Chrome 会话通过 CDP 爬取 Shopee 的反爬商品列表——失败即止的健康检查、分段等待渲染，以及我为什么拒绝并行化处理。" +description: "为一位做二手手机生意的客户：爬取有反爬墙的电商商品列表——热 CDP 会话、失败即止的健康检查、分段等待渲染。" pubDate: 2026-09-13 category: engineering tags: ["scraping", "cdp", "headless-chrome", "anti-bot", "shopee", "python"] @@ -8,9 +8,9 @@ ogImage: "/og/scraping-bot-walled-marketplace-warm-browser-session.png" draft: false --- -我一开始想做一件听起来很简单的事：把一个电商网站上所有的二手 M.2 NVMe SSD 都列出来，按价格排序，好让我买到最便宜的那一个。 +一位客户请我做一个听起来很简单的东西：每天把一个大电商平台上的二手手机列出来，按价格排序，好让他的进货决策靠数据而不是靠感觉。这位客户靠倒卖二手电子产品吃饭——一台手机差几百马币就是他的利润。 -结果完全不是一段干净的 `requests` + BeautifulSoup 脚本。它变成了一场长达一周的、跟反爬墙的拉锯战，一个靠胶带勉强粘在一起的热浏览器会话，以及——最让我意外的部分——一个关于「你不能靠堆更多 agent 来『并行化』爬虫」的惨痛教训。 +结果完全不是一段干净的 `requests` + BeautifulSoup 脚本。它变成了一场长达一周的、跟反爬墙的拉锯战，一个靠胶带勉强粘在一起的热浏览器会话，以及——我们俩都没想到的部分——一个关于「你不能靠堆更多 agent 来『并行化』爬虫」的惨痛教训。 这就是那个故事，按照我经历的顺序来讲：问题、我试过却失败的每一件事、最终奏效的办法，以及下次我会怎么做。 @@ -20,12 +20,12 @@ draft: false 或者更诚实地说：**当一个电商网站已经把「防爬」当成它唯一的本职工作，你要怎么爬它？** Shopee 不会礼貌地把 HTML 喂给 `curl`。它会检测自动化、抛出一个 `/verify` 验证码，然后静默地给你一个「看起来正常但里面什么都没有」的页面。 -我的需求清单很短： +客户的需求清单很短： -- 搜索「二手 SSD」，只筛选「二手」，只筛选 M.2 NVMe（排除 SATA/mSATA），只筛选 ≥128 GB。 -- 提取每个商品的所有容量规格和价格。 +- 搜索二手手机，只筛选「二手」成色，存储 128 GB 及以上。 +- 提取每个商品的所有存储规格和价格。 - 按价格升序排序。 -- 不搞一百个 headless 实例的 `chromedriver` 农场，因为我没有一百个住宅 IP。 +- 不搞一百个 headless 实例的 `chromedriver` 农场——倒卖的利润养不起一百个住宅 IP，我的也一样。 从这里开始，全部都是调试故事。 @@ -54,13 +54,13 @@ draft: false 一旦列表能渲染出来了，我假设购买按钮的文案里带着价格。大多数电商网站确实如此。但这个渲染出来的按钮写的是 `Add To Cart` / `Buy Now`。我用来匹配「Buy With Voucher … RM…」的正则什么都匹配不到，我在原地打转了好一会儿，才真正去读了 DOM。 -**失败：** Shopee 商品页的价格不在按钮里。它被渲染成**拆分的 span**——`RM` 在一个元素里、`84` 在另一个里、`00` 又在另一个里——所以朴素的叶子文本匹配只会抓到碎片，或者当商品有多个规格时，抓到一个**区间**（`RM125.00 - RM155.00`）。 +**失败：** Shopee 商品页的价格不在按钮里。它被渲染成**拆分的 span**——`RM` 在一个元素里、`850` 在另一个里、`00` 又在另一个里——所以朴素的叶子文本匹配只会抓到碎片，或者当商品有多个规格时，抓到一个**区间**（`RM850.00 - RM1150.00`）。 ### 尝试 4：点击规格选项来拿精确价格 -对于单轴规格选择器（比如一个容量下拉：128GB / 256GB / 512GB），点击一个选项会把价格收敛成单个值，我的提取方法能正常工作。 +对于单轴规格选择器（比如一个存储下拉：128GB / 256GB / 512GB），点击一个选项会把价格收敛成单个值，我的提取方法能正常工作。 -**失败：** 对于**双轴**选择器（品牌 × 容量），点击第一个轴（某个品牌）会锁定一个价格，但加上第二个轴（某个容量）又把它**解散**回商品全区间。这个面板渲染的是「所有可购买组合」的区间，而不是我选中的那个具体组合；而且「已选中」状态是用一个「同时匹配其他所有选项框」的 class 来标记的。我没法可靠地知道我到底锁定了哪个组合。 +**失败：** 对于**双轴**选择器（品牌 × 存储），点击第一个轴（某个品牌）会锁定一个价格，但加上第二个轴（某个存储规格）又把它**解散**回商品全区间。这个面板渲染的是「所有可购买组合」的区间，而不是我选中的那个具体组合；而且「已选中」状态是用一个「同时匹配其他所有选项框」的 class 来标记的。我没法可靠地知道我到底锁定了哪个组合。 我在这个上面花了实打实的时间，最后才接受这个诚实的答案：**对于真正的双轴商品，正确的数据是价格区间，而不是一个瞎猜的「每个组合」数字。** 发布一个「有时候能用」的脆弱覆盖方案，比老实报告一个区间更糟糕。 @@ -101,7 +101,7 @@ def session_healthy(): ### 3. 正确的价格选择器：最深的纯 RM 节点 -购买按钮是个谎言。价格是**那个文本内容纯粹是一个 RM 值的、最深的元素**——在选中规格之前是一个区间（`RM125.00 - RM155.00`），之后是单个值（`RM84.00`）。遍历 DOM，取文本能匹配上的最深元素： +购买按钮是个谎言。价格是**那个文本内容纯粹是一个 RM 值的、最深的元素**——在选中规格之前是一个区间（`RM850.00 - RM1150.00`），之后是单个值（`RM850.00`）。遍历 DOM，取文本能匹配上的最深元素： ```python import re @@ -123,7 +123,7 @@ def find_price(root): ## 结果——一个可量化的成果 -从单个热会话、一次串行遍历中，我提取了 **20+ 个二手 NVMe 商品及其各规格价格**，筛选到 M.2 NVMe ≥128GB，按价格升序排序。最便宜的真实目标——一块 256GB M.2 NVMe——以 **RM84** 的价格浮出水面，另外还有几块 256GB 的选项在 RM125–175 区间。现在一条命令（`sweep.py ...`）就能重新检查整份候选清单，并输出 JSONLines。 +从单个热会话、一次串行遍历中，我提取了 **20+ 个二手手机商品及其各规格价格**，筛选到 128 GB 及以上，按价格升序排序。最便宜的真实目标——一台 128GB 机型——以 **RM299** 的价格浮出水面，另外还有几台 128/256GB 的选项在 RM450–880 区间。现在一条命令（`sweep.py ...`）就能重新检查整份候选清单并输出 JSONLines；客户每天进货前跑一次。 但我最庆幸的数字是这个：**零验证码。** 「热会话 + 人性化节奏 + 失败即止」这套组合，在整次遍历中一次都没触发过 `/verify`。 @@ -150,7 +150,7 @@ def find_price(root): --- -*这是一次真实的调试过程，不是事后补写的教程。这个爬虫工具集、以及这里的每一次失败，都发生在我真的为了买一块便宜 SSD 而爬取的时候——我保留了能泛化的部分，砍掉了那些只对某个马来西亚电商网站、某个时间点有效的部分。* +*这是一次真实的调试过程，不是事后补写的教程。这个爬虫工具集、以及这里的每一次失败，都发生在为一个做二手电子产品倒卖的客户构建它的过程中——我保留了能泛化的部分，砍掉了那些只对某个马来西亚电商网站、某个时间点有效的部分。* ---