Add per-watch ignored keywords (IgnoredKeywords table)

Silence Telegram alerts for listings whose title contains a keyword configured for that watch, matched case-insensitively as a substring. - New IgnoredKeywords table: search_url (watch key) + keyword (one per row) - load_ignored_keywords() returns {search_url: {lowercase keywords}}, reloaded every notification cycle so UI edits apply immediately - title_matches_keyword() does the case-insensitive substring check - send_pending_notifications() now also pulls search_url on pending records; skip_notify=true + notified=true on a hit (still archived, no Telegram) - Docs updated: AGENTS.md, README.md, DOCUMENTATION.md
2026-09-13 17:50:11 +08:00
parent bcd858106d
commit 9263b91276
4 changed files with 97 additions and 14 deletions
@@ -9,7 +9,9 @@ from the server-rendered `<script type="application/json">` Redux state
 (`SearchListing.listingCards`), dedupes by `product_url` (param-less), archives to a
 NocoDB base, and alerts Telegram `"<title>: N new listings"`. Listings whose
 `seller_name` is in the `IgnoredSellers` table are archived but never alerted
-(`skip_notify=true`). Runs 24/7 as a Docker container on DSM (network
+(`skip_notify=true`). Listings whose **title** contains a keyword listed for their
+watch in the `IgnoredKeywords` table (keyed by `search_url`, case-insensitive) are
+also archived but never alerted. Runs 24/7 as a Docker container on DSM (network
 `bridge_hoelee`, reaches NocoDB at `http://nocodb:10380`).

 ## Iron rules
@@ -22,6 +24,12 @@ NocoDB base, and alerts Telegram `"<title>: N new listings"`. Listings whose
  monitor sets `skip_notify=true` + `notified=true` and does NOT send Telegram.
  The list is reloaded every notification cycle, so UI add/remove takes effect
  immediately.
+- Ignored keywords live in the NocoDB **IgnoredKeywords** table (`search_url` +
+  one `keyword` per row). A listing is silenced when its **title** contains any
+  keyword for its watch (matched by the watch's search URL), case-insensitive
+  substring match. Per-watch, not global: copy the `url` from the desired
+  `Settings` row into `search_url`. Reloaded every cycle, so UI edits take effect
+  immediately.
 - Dedupe key is `product_url` (`https://www.carousell.com.my/p/<id>/`), not the raw
  listing id and never the query-string URL.
 - First run per watch seeds the archive with **no** Telegram alert (`last_checked_at`
@@ -88,6 +88,23 @@ deleting a table is safe; it is recreated on the next start).
 | `first_seen_at` | DateTime (UTC) | when the monitor first captured it |
 | `notified` | Checkbox | false = 待发通知；发完/静默归档后置 true（防重复通知） |

+### `IgnoredSellers` (global seller blocklist)
+
+| Column | Type | Purpose |
+|---|---|---|
+| `seller_name` | SingleLineText | seller username to silence globally (any watch) |
+
+### `IgnoredKeywords` (per-watch title blocklist)
+
+| Column | Type | Purpose |
+|---|---|---|
+| `search_url` | URL | **copy the `url` from a `Settings` row** — which watch this keyword applies to |
+| `keyword` | SingleLineText | if the listing **title** contains this (case-insensitive substring), skip the Telegram alert (still archived) |
+
+One keyword per row; add multiple rows for multiple keywords. A keyword only
+silences listings found by the watch whose `Settings.url` matches `search_url` —
+the same keyword never applies to other watches. Empty rows are ignored.
+
 ### `Settings` (the watch list — you manage this)

 | Column | Type | Purpose |
@@ -114,6 +131,10 @@ deleting a table is safe; it is recreated on the next start).
 - **Afterwards**: new listings are inserted with `notified=false` (pending). At the end
  of each tick the monitor sends every `notified=false` listing **one message each**
  (photo + title/price/condition/seller/url), 1 second apart, then sets `notified=true`.
+- **Silencing**: a pending listing is marked `skip_notify=true` + `notified=true`
+  (no Telegram) if its `seller_name` is in `IgnoredSellers` (global), or if its
+  **title** contains any keyword whose `IgnoredKeywords.search_url` matches the
+  listing's own `search_url` (per-watch, case-insensitive).
 - **Failure handling**: a fetch/parse error on any watch marks that tick failed; the
  container becomes **unhealthy** until the next fully-successful tick. `last_checked_at`
  is only advanced on success, and a hard-failing watch is rate-limited to one attempt
@@ -128,6 +149,8 @@ deleting a table is safe; it is recreated on the next start).
 | Add a keyword | Add a row to `Settings`: `title`, full `url` (open Carousell → search → sort "Recent" → copy URL), `enabled` ✓, `notify` ✓, interval |
 | Remove / pause a watch | Set `enabled` = false (or delete the row) |
 | Stop Telegram pings but keep archiving | Set `notify` = false |
+| Ignore a seller everywhere | Add `seller_name` row in `IgnoredSellers` |
+| Ignore certain keywords for **one watch** | Add row(s) in `IgnoredKeywords`: `search_url` = copy of that watch's `Settings.url`, `keyword` = e.g. `nike` (case-insensitive, matches inside the title) |
 | Change how often it checks | Edit `check_interval_minutes` (5 = every 5 min) |
 | See what's new | Open `Listings`, sort by `first_seen_at` desc |
 | Browse with images | `Listings` grid view — the `image` column renders thumbnails |
@@ -207,4 +230,8 @@ Health file lives at `/data/health.json` inside the container:

 ## 11. Changelog

+- **2026-09-13** 新增 per-watch 忽略关键词：`IgnoredKeywords` 表（`search_url` +
+  `keyword`）。标题命中该 watch 关键词（大小写不敏感子串）时静默归档、不发
+  Telegram（`skip_notify=true`）。`search_url` 需与 `Settings.url` 一致，实现按
+  watch 隔离。
 - **2026-09-08** 通知重构：每商品一条图文消息（title/price/condition/seller/url），归档与通知解耦（`notified` 列 + tick 末尾统一发 + 1s 间隔）。图片改用高清 URL（去 `_progressive_thumbnail`）。condition 归一化（New→Brand new、Used→Used，加第 6 档）。listed_at 加 `active_bump` fallback。修复 Telegram IPv6/DNS 问题（compose `extra_hosts` 钉 IPv4）。bot 换 `@carousellFoundBot`。
@@ -6,7 +6,8 @@ listing to a NocoDB base (with image URL + thumbnail), and alerts Telegram.
 ## How it works

 - `monitor.py` runs in a Docker container on DSM, self-bootstrapping its NocoDB
-  schema (`Listings` + `Settings` + `IgnoredSellers` tables) and looping forever.
+  schema (`Listings` + `Settings` + `IgnoredSellers` + `IgnoredKeywords` tables)
+  and looping forever.
 - Every `TICK_SECONDS` it reads the watch list from the **Settings** table and polls
  each enabled watch's URL on its own `check_interval_minutes`.
 - Dedupe key = `product_url` (param-less listing URL). First run per watch = seed
@@ -28,6 +29,10 @@ listing to a NocoDB base (with image URL + thumbnail), and alerts Telegram.
 **IgnoredSellers** — `seller_name`. Add/remove sellers here to suppress Telegram
 alerts for their listings (still archived, marked `skip_notify=true`).

+**IgnoredKeywords** — `search_url` + `keyword`. Per-watch title blocklist: keyword
+applies only to listings from the watch whose `Settings.url` matches `search_url`;
+case-insensitive substring match against the title. Still archived.
+
 ## Run

 ```bash
@@ -88,6 +88,12 @@ LISTINGS_COLS = [
 IGNORED_SELLERS_COLS = [
    ("seller_name", "SingleLineText"),
 ]
+# 忽略关键词（per-watch）：search_url=Settings 里的 watch URL 原样复制，keyword 单行一条。
+# 大小写不敏感，命中该 watch 的「标题」即跳过通知（仍归档）。
+IGNORED_KEYWORDS_COLS = [
+    ("search_url", "URL"),
+    ("keyword", "SingleLineText"),
+]
 SETTINGS_COLS = [
    ("title", "SingleLineText"),
    ("url", "URL"),
@@ -240,7 +246,8 @@ def bootstrap():
    listings_tid = _ensure_table("Listings", LISTINGS_COLS)
    settings_tid = _ensure_table("Settings", SETTINGS_COLS)
    ignored_sellers_tid = _ensure_table("IgnoredSellers", IGNORED_SELLERS_COLS)
-    return listings_tid, settings_tid, ignored_sellers_tid
+    ignored_keywords_tid = _ensure_table("IgnoredKeywords", IGNORED_KEYWORDS_COLS)
+    return listings_tid, settings_tid, ignored_sellers_tid, ignored_keywords_tid


 # --------------------------------------------------------------------------- #
@@ -373,6 +380,32 @@ def load_ignored_sellers(ignored_sellers_tid):
    return ignored


+def load_ignored_keywords(ignored_keywords_tid):
+    """从 IgnoredKeywords 表读取忽略关键词，返回 {search_url: {小写关键词}}。
+
+    关键词按 watch（search_url）分组；匹配时大小写不敏感。
+    """
+    ignored = {}
+    st, j = nc("GET", f"/api/v2/tables/{ignored_keywords_tid}/records?limit=1000")
+    if st != 200:
+        raise RuntimeError(f"load ignored keywords failed: {j}")
+    for r in j.get("list", []):
+        url = (r.get("search_url") or "").strip()
+        kw = (r.get("keyword") or "").strip().lower()
+        if not url or not kw:
+            continue
+        ignored.setdefault(url, set()).add(kw)
+    return ignored
+
+
+def title_matches_keyword(title, ignored_keywords):
+    """标题命中任一忽略关键词（大小写不敏感的子串匹配）则返回 True。"""
+    if not title or not ignored_keywords:
+        return False
+    t = title.lower()
+    return any(kw in t for kw in ignored_keywords)
+
+
 def load_watches(settings_tid):
    st, j = nc("GET", f"/api/v2/tables/{settings_tid}/records?limit=1000")
    if st != 200:
@@ -491,12 +524,14 @@ def mark_notified(listings_tid, rec_ids):
    nc("PATCH", f"/api/v2/tables/{listings_tid}/records", updates)


-def send_pending_notifications(listings_tid, settings_tid, ignored_sellers_tid):
+def send_pending_notifications(listings_tid, settings_tid, ignored_sellers_tid,
+                               ignored_keywords_tid):
    """tick 末尾统一发：查 notified=false 的记录，逐条发（间隔 1s），发完置 true。

    仅发「其 watch 仍 notify=true」的记录；watch 已关 notify 的则静默置 true。
-    若 seller_name 落在 IgnoredSellers 忽略列表，则置 skip_notify=true + notified=true，
-    不发 Telegram。
+    若 seller_name 落在 IgnoredSellers 忽略列表，或 title 命中该 watch
+    （search_url）在 IgnoredKeywords 里的关键词（大小写不敏感），则置
+    skip_notify=true + notified=true，不发 Telegram。
    """
    # 加载所有 watch 的 notify 开关，key = search_title
    st, j = nc("GET", f"/api/v2/tables/{settings_tid}/records?limit=1000")
@@ -508,11 +543,12 @@ def send_pending_notifications(listings_tid, settings_tid, ignored_sellers_tid):

    # 每轮重新加载忽略列表，中途增删立即生效
    ignored = load_ignored_sellers(ignored_sellers_tid)
+    ignored_kw_by_url = load_ignored_keywords(ignored_keywords_tid)

    # 拉 notified=false 的记录
    st, j = nc("GET", f"/api/v2/tables/{listings_tid}/records"
               f"?limit=1000&fields=Id,title,price,condition,seller_name,"
-               f"product_url,image_url,search_title,notified,skip_notify")
+               f"product_url,image_url,search_title,search_url,notified,skip_notify")
    if st != 200:
        return
    pending = [r for r in j.get("list", []) if not r.get("notified")]
@@ -522,8 +558,11 @@ def send_pending_notifications(listings_tid, settings_tid, ignored_sellers_tid):

    for rec in pending:
        seller = (rec.get("seller_name") or "").strip()
-        if seller in ignored:
-            # 命中忽略列表：标记 skip_notify，静默置 notified，不发
+        title = (rec.get("title") or "").strip()
+        search_url = (rec.get("search_url") or "").strip()
+        kw_for_watch = ignored_kw_by_url.get(search_url, set())
+        if seller in ignored or title_matches_keyword(title, kw_for_watch):
+            # 命中忽略卖家/该 watch 的关键词：标记 skip_notify，静默置 notified，不发
            nc("PATCH", f"/api/v2/tables/{listings_tid}/records",
               [{"Id": rec["Id"], "skip_notify": True, "notified": True}])
            continue
@@ -558,7 +597,8 @@ def write_health(ok, error, extra=None):
 # --------------------------------------------------------------------------- #
 # Main loop
 # --------------------------------------------------------------------------- #
-def run_tick(listings_tid, settings_tid, ignored_sellers_tid, seen, last_run):
+def run_tick(listings_tid, settings_tid, ignored_sellers_tid, ignored_keywords_tid,
+             seen, last_run):
    failures = []
    new_total = 0
    watches = load_watches(settings_tid)
@@ -595,7 +635,8 @@ def run_tick(listings_tid, settings_tid, ignored_sellers_tid, seen, last_run):
        last_run[wid] = now

    # 归档完成后，统一发送待通知的记录（解耦：归档成功才通知）
-    send_pending_notifications(listings_tid, settings_tid, ignored_sellers_tid)
+    send_pending_notifications(listings_tid, settings_tid, ignored_sellers_tid,
+                               ignored_keywords_tid)

    ok = len(failures) == 0
    return ok, ("; ".join(failures) if failures else ""), {
@@ -608,17 +649,19 @@ def main():
        write_health(False, "NOCODB_TOKEN not set")
        sys.exit(2)

-    listings_tid, settings_tid, ignored_sellers_tid = bootstrap()
+    listings_tid, settings_tid, ignored_sellers_tid, ignored_keywords_tid = bootstrap()
    seen = load_seen(listings_tid)
    last_run = {}

    sys.stderr.write(f"ready: listings={listings_tid} settings={settings_tid} "
-                     f"ignored_sellers={ignored_sellers_tid} seen={len(seen)}\n")
+                     f"ignored_sellers={ignored_sellers_tid} "
+                     f"ignored_keywords={ignored_keywords_tid} seen={len(seen)}\n")

    while True:
        try:
            ok, err, extra = run_tick(listings_tid, settings_tid,
-                                      ignored_sellers_tid, seen, last_run)
+                                      ignored_sellers_tid, ignored_keywords_tid,
+                                      seen, last_run)
        except Exception as e:
            ok, err, extra = False, f"tick error: {e}", {}
        write_health(ok, err, extra)