Make IgnoredKeywords.watch a real Link column to Settings

Replace the hand-copied search_url (URL type) with a proper relation:
- watch: LinkToAnotherRecord (belongs-to, many keywords -> one Settings row),
  picked from a dropdown in the UI so a wrong URL can't silently disable a keyword
- search_url: Lookup column that auto-fills Settings.url through the link

_ensure_keyword_watch_link() builds both columns idempotently on bootstrap and
upgrades a legacy URL-typed search_url to Link+Lookup. load_ignored_keywords()
now normalises Lookup values (string or array).
This commit is contained in:
2026-09-13 18:11:51 +08:00
parent 9263b91276
commit 531da933ca
4 changed files with 106 additions and 20 deletions
+7 -6
View File
@@ -24,12 +24,13 @@ also archived but never alerted. Runs 24/7 as a Docker container on DSM (network
monitor sets `skip_notify=true` + `notified=true` and does NOT send Telegram.
The list is reloaded every notification cycle, so UI add/remove takes effect
immediately.
- Ignored keywords live in the NocoDB **IgnoredKeywords** table (`search_url` +
one `keyword` per row). A listing is silenced when its **title** contains any
keyword for its watch (matched by the watch's search URL), case-insensitive
substring match. Per-watch, not global: copy the `url` from the desired
`Settings` row into `search_url`. Reloaded every cycle, so UI edits take effect
immediately.
- Ignored keywords live in the NocoDB **IgnoredKeywords** table: `watch` is a real
**Link column → `Settings`** (pick the watch from a dropdown) and `search_url` is a
**Lookup column** that auto-fills `Settings.url` from that link. A listing is
silenced when its **title** contains any keyword for its watch, case-insensitive
substring match. Per-watch, not global. Reloaded every cycle, so UI edits take
effect immediately. Bootstrap creates both columns and upgrades a legacy URL-typed
`search_url` to Link+Lookup automatically.
- Dedupe key is `product_url` (`https://www.carousell.com.my/p/<id>/`), not the raw
listing id and never the query-string URL.
- First run per watch seeds the archive with **no** Telegram alert (`last_checked_at`
+13 -8
View File
@@ -98,12 +98,18 @@ deleting a table is safe; it is recreated on the next start).
| Column | Type | Purpose |
|---|---|---|
| `search_url` | URL | **copy the `url` from a `Settings` row** — which watch this keyword applies to |
| `watch` | Link → `Settings` | **pick the watch from a dropdown** (belongs-to: many keywords → one watch) — no URL to copy by hand |
| `search_url` | Lookup (`Settings.url`) | read-only, auto-filled from the linked watch — this is what the monitor reads |
| `keyword` | SingleLineText | if the listing **title** contains this (case-insensitive substring), skip the Telegram alert (still archived) |
One keyword per row; add multiple rows for multiple keywords. A keyword only
silences listings found by the watch whose `Settings.url` matches `search_url` —
the same keyword never applies to other watches. Empty rows are ignored.
silences listings found by the watch you linked — the same keyword never applies to
other watches. Rows with no watch or no keyword are ignored.
`watch` is a real NocoDB Link column, so deleting/renaming a watch shows up in the
relation, and there is no hand-copied URL that can silently drift out of sync.
(The monitor's bootstrap also builds these two columns itself, including upgrading a
legacy URL-typed `search_url` to Link+Lookup.)
### `Settings` (the watch list — you manage this)
@@ -150,7 +156,7 @@ the same keyword never applies to other watches. Empty rows are ignored.
| Remove / pause a watch | Set `enabled` = false (or delete the row) |
| Stop Telegram pings but keep archiving | Set `notify` = false |
| Ignore a seller everywhere | Add `seller_name` row in `IgnoredSellers` |
| Ignore certain keywords for **one watch** | Add row(s) in `IgnoredKeywords`: `search_url` = copy of that watch's `Settings.url`, `keyword` = e.g. `nike` (case-insensitive, matches inside the title) |
| Ignore certain keywords for **one watch** | Add row(s) in `IgnoredKeywords`: pick the watch in the `watch` dropdown, `keyword` = e.g. `nike` (case-insensitive, matches inside the title). `search_url` fills itself |
| Change how often it checks | Edit `check_interval_minutes` (5 = every 5 min) |
| See what's new | Open `Listings`, sort by `first_seen_at` desc |
| Browse with images | `Listings` grid view — the `image` column renders thumbnails |
@@ -230,8 +236,7 @@ Health file lives at `/data/health.json` inside the container:
## 11. Changelog
- **2026-09-13** 新增 per-watch 忽略关键词:`IgnoredKeywords` 表(`search_url` +
`keyword`)。标题命中该 watch 关键词(大小写不敏感子串)时静默归档、不发
Telegram(`skip_notify=true`)。`search_url` 需与 `Settings.url` 一致,实现按
watch 隔离。
- **2026-09-13** `IgnoredKeywords.watch` 改成真正的 NocoDB **Link 列**(Many-to-One → `Settings`),`search_url` 改成 **Lookup 列**自动带出 `Settings.url`(UI 下拉选 watch,不用手抄 URL)。bootstrap 会自建这两列,并把旧版 URL 型 `search_url` 自动升级成 Link+Lookup。
- **2026-09-13** 新增 per-watch 忽略关键词:`IgnoredKeywords` 表。标题命中该 watch
关键词(大小写不敏感子串)时静默归档、不发 Telegram(`skip_notify=true`)。
- **2026-09-08** 通知重构:每商品一条图文消息(title/price/condition/seller/url),归档与通知解耦(`notified` 列 + tick 末尾统一发 + 1s 间隔)。图片改用高清 URL(去 `_progressive_thumbnail`)。condition 归一化(New→Brand new、Used→Used,加第 6 档)。listed_at 加 `active_bump` fallback。修复 Telegram IPv6/DNS 问题(compose `extra_hosts` 钉 IPv4)。bot 换 `@carousellFoundBot`。
+5 -3
View File
@@ -29,9 +29,11 @@ listing to a NocoDB base (with image URL + thumbnail), and alerts Telegram.
**IgnoredSellers** — `seller_name`. Add/remove sellers here to suppress Telegram
alerts for their listings (still archived, marked `skip_notify=true`).
**IgnoredKeywords** — `search_url` + `keyword`. Per-watch title blocklist: keyword
applies only to listings from the watch whose `Settings.url` matches `search_url`;
case-insensitive substring match against the title. Still archived.
**IgnoredKeywords** — `watch` (Link → Settings) + `keyword`. Per-watch title
blocklist: pick the watch from a dropdown, add one keyword per row. A keyword only
applies to listings from the linked watch; case-insensitive substring match against
the title. `search_url` is a Lookup column that auto-fills from the linked watch.
Still archived.
## Run
+81 -3
View File
@@ -88,10 +88,10 @@ LISTINGS_COLS = [
IGNORED_SELLERS_COLS = [
("seller_name", "SingleLineText"),
]
# 忽略关键词(per-watch):search_url=Settings 里的 watch URL 原样复制,keyword 单行一条。
# 忽略关键词(per-watch):keyword 单行一条;watch=Link 指向 Settings 记录;
# search_url=Lookup 自动带出被链接 watch 的 Settings.url。
# 大小写不敏感,命中该 watch 的「标题」即跳过通知(仍归档)。
IGNORED_KEYWORDS_COLS = [
("search_url", "URL"),
("keyword", "SingleLineText"),
]
SETTINGS_COLS = [
@@ -247,9 +247,82 @@ def bootstrap():
settings_tid = _ensure_table("Settings", SETTINGS_COLS)
ignored_sellers_tid = _ensure_table("IgnoredSellers", IGNORED_SELLERS_COLS)
ignored_keywords_tid = _ensure_table("IgnoredKeywords", IGNORED_KEYWORDS_COLS)
_ensure_keyword_watch_link(ignored_keywords_tid, settings_tid)
return listings_tid, settings_tid, ignored_sellers_tid, ignored_keywords_tid
def _ensure_keyword_watch_link(kw_tid, settings_tid):
"""IgnoredKeywords 的 watch(Link→Settings) + search_url(Lookup→Settings.url)。
NocoDB 2026.08 的列创建 API 与旧版不同:
- Link 列: uidt="LinkToAnotherRecord", 顶层 parentId/childId/type
- 回读 link 列的 colOptions.fk_column_id(物理外键列)作为 Lookup 的
fk_relation_column_id
- Lookup 列: uidt="Lookup", 顶层 fk_relation_column_id + fk_lookup_column_id
幂等:列已存在就跳过。旧版遗留的 URL 型 search_url 会被替换成 Lookup。
"""
def _kw_meta():
st, m = nc("GET", f"/api/v2/meta/tables/{kw_tid}")
if st != 200:
raise RuntimeError(f"read IgnoredKeywords meta failed: {m}")
return m
m = _kw_meta()
cols = {c.get("title"): c for c in m.get("columns", [])}
# 1. watch: LinkToAnotherRecord -> Settings
link_col = None
if "watch" in cols:
if cols["watch"].get("uidt") != "LinkToAnotherRecord":
nc("DELETE", f"/api/v2/meta/columns/{cols['watch']['id']}")
else:
link_col = cols["watch"]
if link_col is None:
st, out = nc("POST", f"/api/v2/meta/tables/{kw_tid}/columns",
{"title": "watch", "uidt": "LinkToAnotherRecord",
"parentId": settings_tid, "childId": kw_tid,
"type": "bt", "colOptions": {}})
if st != 200:
raise RuntimeError(f"create watch link failed: {out}")
m = _kw_meta()
cols = {c.get("title"): c for c in m.get("columns", [])}
link_col = cols.get("watch")
if link_col is None:
raise RuntimeError("watch link column missing after create")
# 2. search_url: Lookup (auto-pull Settings.url through the link)
if "search_url" in cols and cols["search_url"].get("uidt") == "Lookup":
return # already good
# drop stale non-Lookup search_url (legacy URL column)
if "search_url" in cols:
nc("DELETE", f"/api/v2/meta/columns/{cols['search_url']['id']}")
fk_parent = (link_col.get("colOptions") or {}).get("fk_column_id")
if not fk_parent:
raise RuntimeError("watch link has no fk_column_id")
st, sm = nc("GET", f"/api/v2/meta/tables/{settings_tid}")
if st != 200:
raise RuntimeError(f"read Settings meta failed: {sm}")
url_col = next((c for c in sm.get("columns", [])
if c.get("title") == "url"), None)
if url_col is None:
raise RuntimeError("Settings.url column not found")
st, out = nc("POST", f"/api/v2/meta/tables/{kw_tid}/columns",
{"title": "search_url", "uidt": "Lookup",
"fk_relation_column_id": fk_parent,
"fk_lookup_column_id": url_col["id"],
"colOptions": {}})
if st != 200:
raise RuntimeError(f"create search_url lookup failed: {out}")
# --------------------------------------------------------------------------- #
# Carousell extraction
# --------------------------------------------------------------------------- #
@@ -384,13 +457,18 @@ def load_ignored_keywords(ignored_keywords_tid):
"""从 IgnoredKeywords 表读取忽略关键词,返回 {search_url: {小写关键词}}。
关键词按 watch(search_url)分组;匹配时大小写不敏感。
search_url 是 Lookup 列(经 watch 链接自动带出 Settings.url),
API 可能返回字符串或数组,这里统一归一化。
"""
ignored = {}
st, j = nc("GET", f"/api/v2/tables/{ignored_keywords_tid}/records?limit=1000")
if st != 200:
raise RuntimeError(f"load ignored keywords failed: {j}")
for r in j.get("list", []):
url = (r.get("search_url") or "").strip()
url = r.get("search_url") or ""
if isinstance(url, list): # Lookup 有时返回数组
url = url[0] if url else ""
url = str(url).strip()
kw = (r.get("keyword") or "").strip().lower()
if not url or not kw:
continue