Make IgnoredKeywords.watch a real Link column to Settings
Replace the hand-copied search_url (URL type) with a proper relation: - watch: LinkToAnotherRecord (belongs-to, many keywords -> one Settings row), picked from a dropdown in the UI so a wrong URL can't silently disable a keyword - search_url: Lookup column that auto-fills Settings.url through the link _ensure_keyword_watch_link() builds both columns idempotently on bootstrap and upgrades a legacy URL-typed search_url to Link+Lookup. load_ignored_keywords() now normalises Lookup values (string or array).
This commit is contained in:
@@ -24,12 +24,13 @@ also archived but never alerted. Runs 24/7 as a Docker container on DSM (network
|
||||
monitor sets `skip_notify=true` + `notified=true` and does NOT send Telegram.
|
||||
The list is reloaded every notification cycle, so UI add/remove takes effect
|
||||
immediately.
|
||||
- Ignored keywords live in the NocoDB **IgnoredKeywords** table (`search_url` +
|
||||
one `keyword` per row). A listing is silenced when its **title** contains any
|
||||
keyword for its watch (matched by the watch's search URL), case-insensitive
|
||||
substring match. Per-watch, not global: copy the `url` from the desired
|
||||
`Settings` row into `search_url`. Reloaded every cycle, so UI edits take effect
|
||||
immediately.
|
||||
- Ignored keywords live in the NocoDB **IgnoredKeywords** table: `watch` is a real
|
||||
**Link column → `Settings`** (pick the watch from a dropdown) and `search_url` is a
|
||||
**Lookup column** that auto-fills `Settings.url` from that link. A listing is
|
||||
silenced when its **title** contains any keyword for its watch, case-insensitive
|
||||
substring match. Per-watch, not global. Reloaded every cycle, so UI edits take
|
||||
effect immediately. Bootstrap creates both columns and upgrades a legacy URL-typed
|
||||
`search_url` to Link+Lookup automatically.
|
||||
- Dedupe key is `product_url` (`https://www.carousell.com.my/p/<id>/`), not the raw
|
||||
listing id and never the query-string URL.
|
||||
- First run per watch seeds the archive with **no** Telegram alert (`last_checked_at`
|
||||
|
||||
+13
-8
@@ -98,12 +98,18 @@ deleting a table is safe; it is recreated on the next start).
|
||||
|
||||
| Column | Type | Purpose |
|
||||
|---|---|---|
|
||||
| `search_url` | URL | **copy the `url` from a `Settings` row** — which watch this keyword applies to |
|
||||
| `watch` | Link → `Settings` | **pick the watch from a dropdown** (belongs-to: many keywords → one watch) — no URL to copy by hand |
|
||||
| `search_url` | Lookup (`Settings.url`) | read-only, auto-filled from the linked watch — this is what the monitor reads |
|
||||
| `keyword` | SingleLineText | if the listing **title** contains this (case-insensitive substring), skip the Telegram alert (still archived) |
|
||||
|
||||
One keyword per row; add multiple rows for multiple keywords. A keyword only
|
||||
silences listings found by the watch whose `Settings.url` matches `search_url` —
|
||||
the same keyword never applies to other watches. Empty rows are ignored.
|
||||
silences listings found by the watch you linked — the same keyword never applies to
|
||||
other watches. Rows with no watch or no keyword are ignored.
|
||||
|
||||
`watch` is a real NocoDB Link column, so deleting/renaming a watch shows up in the
|
||||
relation, and there is no hand-copied URL that can silently drift out of sync.
|
||||
(The monitor's bootstrap also builds these two columns itself, including upgrading a
|
||||
legacy URL-typed `search_url` to Link+Lookup.)
|
||||
|
||||
### `Settings` (the watch list — you manage this)
|
||||
|
||||
@@ -150,7 +156,7 @@ the same keyword never applies to other watches. Empty rows are ignored.
|
||||
| Remove / pause a watch | Set `enabled` = false (or delete the row) |
|
||||
| Stop Telegram pings but keep archiving | Set `notify` = false |
|
||||
| Ignore a seller everywhere | Add `seller_name` row in `IgnoredSellers` |
|
||||
| Ignore certain keywords for **one watch** | Add row(s) in `IgnoredKeywords`: `search_url` = copy of that watch's `Settings.url`, `keyword` = e.g. `nike` (case-insensitive, matches inside the title) |
|
||||
| Ignore certain keywords for **one watch** | Add row(s) in `IgnoredKeywords`: pick the watch in the `watch` dropdown, `keyword` = e.g. `nike` (case-insensitive, matches inside the title). `search_url` fills itself |
|
||||
| Change how often it checks | Edit `check_interval_minutes` (5 = every 5 min) |
|
||||
| See what's new | Open `Listings`, sort by `first_seen_at` desc |
|
||||
| Browse with images | `Listings` grid view — the `image` column renders thumbnails |
|
||||
@@ -230,8 +236,7 @@ Health file lives at `/data/health.json` inside the container:
|
||||
|
||||
## 11. Changelog
|
||||
|
||||
- **2026-09-13** 新增 per-watch 忽略关键词:`IgnoredKeywords` 表(`search_url` +
|
||||
`keyword`)。标题命中该 watch 关键词(大小写不敏感子串)时静默归档、不发
|
||||
Telegram(`skip_notify=true`)。`search_url` 需与 `Settings.url` 一致,实现按
|
||||
watch 隔离。
|
||||
- **2026-09-13** `IgnoredKeywords.watch` 改成真正的 NocoDB **Link 列**(Many-to-One → `Settings`),`search_url` 改成 **Lookup 列**自动带出 `Settings.url`(UI 下拉选 watch,不用手抄 URL)。bootstrap 会自建这两列,并把旧版 URL 型 `search_url` 自动升级成 Link+Lookup。
|
||||
- **2026-09-13** 新增 per-watch 忽略关键词:`IgnoredKeywords` 表。标题命中该 watch
|
||||
关键词(大小写不敏感子串)时静默归档、不发 Telegram(`skip_notify=true`)。
|
||||
- **2026-09-08** 通知重构:每商品一条图文消息(title/price/condition/seller/url),归档与通知解耦(`notified` 列 + tick 末尾统一发 + 1s 间隔)。图片改用高清 URL(去 `_progressive_thumbnail`)。condition 归一化(New→Brand new、Used→Used,加第 6 档)。listed_at 加 `active_bump` fallback。修复 Telegram IPv6/DNS 问题(compose `extra_hosts` 钉 IPv4)。bot 换 `@carousellFoundBot`。
|
||||
|
||||
@@ -29,9 +29,11 @@ listing to a NocoDB base (with image URL + thumbnail), and alerts Telegram.
|
||||
**IgnoredSellers** — `seller_name`. Add/remove sellers here to suppress Telegram
|
||||
alerts for their listings (still archived, marked `skip_notify=true`).
|
||||
|
||||
**IgnoredKeywords** — `search_url` + `keyword`. Per-watch title blocklist: keyword
|
||||
applies only to listings from the watch whose `Settings.url` matches `search_url`;
|
||||
case-insensitive substring match against the title. Still archived.
|
||||
**IgnoredKeywords** — `watch` (Link → Settings) + `keyword`. Per-watch title
|
||||
blocklist: pick the watch from a dropdown, add one keyword per row. A keyword only
|
||||
applies to listings from the linked watch; case-insensitive substring match against
|
||||
the title. `search_url` is a Lookup column that auto-fills from the linked watch.
|
||||
Still archived.
|
||||
|
||||
## Run
|
||||
|
||||
|
||||
+81
-3
@@ -88,10 +88,10 @@ LISTINGS_COLS = [
|
||||
IGNORED_SELLERS_COLS = [
|
||||
("seller_name", "SingleLineText"),
|
||||
]
|
||||
# 忽略关键词(per-watch):search_url=Settings 里的 watch URL 原样复制,keyword 单行一条。
|
||||
# 忽略关键词(per-watch):keyword 单行一条;watch=Link 指向 Settings 记录;
|
||||
# search_url=Lookup 自动带出被链接 watch 的 Settings.url。
|
||||
# 大小写不敏感,命中该 watch 的「标题」即跳过通知(仍归档)。
|
||||
IGNORED_KEYWORDS_COLS = [
|
||||
("search_url", "URL"),
|
||||
("keyword", "SingleLineText"),
|
||||
]
|
||||
SETTINGS_COLS = [
|
||||
@@ -247,9 +247,82 @@ def bootstrap():
|
||||
settings_tid = _ensure_table("Settings", SETTINGS_COLS)
|
||||
ignored_sellers_tid = _ensure_table("IgnoredSellers", IGNORED_SELLERS_COLS)
|
||||
ignored_keywords_tid = _ensure_table("IgnoredKeywords", IGNORED_KEYWORDS_COLS)
|
||||
_ensure_keyword_watch_link(ignored_keywords_tid, settings_tid)
|
||||
return listings_tid, settings_tid, ignored_sellers_tid, ignored_keywords_tid
|
||||
|
||||
|
||||
def _ensure_keyword_watch_link(kw_tid, settings_tid):
|
||||
"""IgnoredKeywords 的 watch(Link→Settings) + search_url(Lookup→Settings.url)。
|
||||
|
||||
NocoDB 2026.08 的列创建 API 与旧版不同:
|
||||
- Link 列: uidt="LinkToAnotherRecord", 顶层 parentId/childId/type
|
||||
- 回读 link 列的 colOptions.fk_column_id(物理外键列)作为 Lookup 的
|
||||
fk_relation_column_id
|
||||
- Lookup 列: uidt="Lookup", 顶层 fk_relation_column_id + fk_lookup_column_id
|
||||
|
||||
幂等:列已存在就跳过。旧版遗留的 URL 型 search_url 会被替换成 Lookup。
|
||||
"""
|
||||
|
||||
def _kw_meta():
|
||||
st, m = nc("GET", f"/api/v2/meta/tables/{kw_tid}")
|
||||
if st != 200:
|
||||
raise RuntimeError(f"read IgnoredKeywords meta failed: {m}")
|
||||
return m
|
||||
|
||||
m = _kw_meta()
|
||||
cols = {c.get("title"): c for c in m.get("columns", [])}
|
||||
|
||||
# 1. watch: LinkToAnotherRecord -> Settings
|
||||
link_col = None
|
||||
if "watch" in cols:
|
||||
if cols["watch"].get("uidt") != "LinkToAnotherRecord":
|
||||
nc("DELETE", f"/api/v2/meta/columns/{cols['watch']['id']}")
|
||||
else:
|
||||
link_col = cols["watch"]
|
||||
|
||||
if link_col is None:
|
||||
st, out = nc("POST", f"/api/v2/meta/tables/{kw_tid}/columns",
|
||||
{"title": "watch", "uidt": "LinkToAnotherRecord",
|
||||
"parentId": settings_tid, "childId": kw_tid,
|
||||
"type": "bt", "colOptions": {}})
|
||||
if st != 200:
|
||||
raise RuntimeError(f"create watch link failed: {out}")
|
||||
m = _kw_meta()
|
||||
cols = {c.get("title"): c for c in m.get("columns", [])}
|
||||
link_col = cols.get("watch")
|
||||
|
||||
if link_col is None:
|
||||
raise RuntimeError("watch link column missing after create")
|
||||
|
||||
# 2. search_url: Lookup (auto-pull Settings.url through the link)
|
||||
if "search_url" in cols and cols["search_url"].get("uidt") == "Lookup":
|
||||
return # already good
|
||||
|
||||
# drop stale non-Lookup search_url (legacy URL column)
|
||||
if "search_url" in cols:
|
||||
nc("DELETE", f"/api/v2/meta/columns/{cols['search_url']['id']}")
|
||||
|
||||
fk_parent = (link_col.get("colOptions") or {}).get("fk_column_id")
|
||||
if not fk_parent:
|
||||
raise RuntimeError("watch link has no fk_column_id")
|
||||
|
||||
st, sm = nc("GET", f"/api/v2/meta/tables/{settings_tid}")
|
||||
if st != 200:
|
||||
raise RuntimeError(f"read Settings meta failed: {sm}")
|
||||
url_col = next((c for c in sm.get("columns", [])
|
||||
if c.get("title") == "url"), None)
|
||||
if url_col is None:
|
||||
raise RuntimeError("Settings.url column not found")
|
||||
|
||||
st, out = nc("POST", f"/api/v2/meta/tables/{kw_tid}/columns",
|
||||
{"title": "search_url", "uidt": "Lookup",
|
||||
"fk_relation_column_id": fk_parent,
|
||||
"fk_lookup_column_id": url_col["id"],
|
||||
"colOptions": {}})
|
||||
if st != 200:
|
||||
raise RuntimeError(f"create search_url lookup failed: {out}")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Carousell extraction
|
||||
# --------------------------------------------------------------------------- #
|
||||
@@ -384,13 +457,18 @@ def load_ignored_keywords(ignored_keywords_tid):
|
||||
"""从 IgnoredKeywords 表读取忽略关键词,返回 {search_url: {小写关键词}}。
|
||||
|
||||
关键词按 watch(search_url)分组;匹配时大小写不敏感。
|
||||
search_url 是 Lookup 列(经 watch 链接自动带出 Settings.url),
|
||||
API 可能返回字符串或数组,这里统一归一化。
|
||||
"""
|
||||
ignored = {}
|
||||
st, j = nc("GET", f"/api/v2/tables/{ignored_keywords_tid}/records?limit=1000")
|
||||
if st != 200:
|
||||
raise RuntimeError(f"load ignored keywords failed: {j}")
|
||||
for r in j.get("list", []):
|
||||
url = (r.get("search_url") or "").strip()
|
||||
url = r.get("search_url") or ""
|
||||
if isinstance(url, list): # Lookup 有时返回数组
|
||||
url = url[0] if url else ""
|
||||
url = str(url).strip()
|
||||
kw = (r.get("keyword") or "").strip().lower()
|
||||
if not url or not kw:
|
||||
continue
|
||||
|
||||
Reference in New Issue
Block a user