Drop IgnoredKeywords.search_url column; resolve URL via watch link
The search_url Lookup column is redundant: the watch Link -> Settings row already defines the association. Remove handling for the column: - _ensure_keyword_watch_link() no longer creates a Lookup; deletes any legacy search_url (URL or Lookup type) on bootstrap - load_ignored_keywords(kw_tid, settings_tid) resolves Settings.url via the watch link, one Settings fetch per cycle (no N+1) - send_pending_notifications passes settings_tid through - Add COMPOSE-SETUP.md: detail documentation for the compose stack (topology, compose block-by-block, Dockerfile, healthcheck, deploy paths, troubleshooting); referenced from README + DOCUMENTATION.md
This commit is contained in:
@@ -10,9 +10,9 @@ from the server-rendered `<script type="application/json">` Redux state
|
||||
NocoDB base, and alerts Telegram `"<title>: N new listings"`. Listings whose
|
||||
`seller_name` is in the `IgnoredSellers` table are archived but never alerted
|
||||
(`skip_notify=true`). Listings whose **title** contains a keyword listed for their
|
||||
watch in the `IgnoredKeywords` table (keyed by `search_url`, case-insensitive) are
|
||||
also archived but never alerted. Runs 24/7 as a Docker container on DSM (network
|
||||
`bridge_hoelee`, reaches NocoDB at `http://nocodb:10380`).
|
||||
watch in the `IgnoredKeywords` table (linked to the watch's `Settings` row,
|
||||
case-insensitive) are also archived but never alerted. Runs 24/7 as a Docker
|
||||
container on DSM (network `bridge_hoelee`, reaches NocoDB at `http://nocodb:10380`).
|
||||
|
||||
## Iron rules
|
||||
|
||||
@@ -25,12 +25,11 @@ also archived but never alerted. Runs 24/7 as a Docker container on DSM (network
|
||||
The list is reloaded every notification cycle, so UI add/remove takes effect
|
||||
immediately.
|
||||
- Ignored keywords live in the NocoDB **IgnoredKeywords** table: `watch` is a real
|
||||
**Link column → `Settings`** (pick the watch from a dropdown) and `search_url` is a
|
||||
**Lookup column** that auto-fills `Settings.url` from that link. A listing is
|
||||
**Link column → `Settings`** (pick the watch from a dropdown). A listing is
|
||||
silenced when its **title** contains any keyword for its watch, case-insensitive
|
||||
substring match. Per-watch, not global. Reloaded every cycle, so UI edits take
|
||||
effect immediately. Bootstrap creates both columns and upgrades a legacy URL-typed
|
||||
`search_url` to Link+Lookup automatically.
|
||||
effect immediately. Bootstrap creates the `watch` column and drops any legacy
|
||||
`search_url` column.
|
||||
- Dedupe key is `product_url` (`https://www.carousell.com.my/p/<id>/`), not the raw
|
||||
listing id and never the query-string URL.
|
||||
- First run per watch seeds the archive with **no** Telegram alert (`last_checked_at`
|
||||
|
||||
@@ -0,0 +1,249 @@
|
||||
# COMPOSE-SETUP.md — Carousell Monitor Docker Stack, Explained
|
||||
|
||||
Line-by-line anatomy of the compose stack, how it connects to the rest of the
|
||||
homelab, and the two ways to deploy it. Complementary to `DOCUMENTATION.md`
|
||||
(ops) and `AGENTS.md` (agent entry); this file is the *stack reference*.
|
||||
|
||||
---
|
||||
|
||||
## 1. Topology
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────┐
|
||||
│ DSM host (Synology) │
|
||||
│ │
|
||||
Internet ─ 443 ──► │ DSM nginx reverse proxy │
|
||||
│ │ (not involved for this container — │
|
||||
│ │ it makes outbound calls only) │
|
||||
│ ▼ │
|
||||
│ Docker network bridge_hoelee (external) │
|
||||
│ │ │
|
||||
│ ├── carousell-monitor (this stack) │
|
||||
│ │ │ │
|
||||
│ │ ├─► NocoDB http://nocodb:10380 │
|
||||
│ │ │ (same bridge_hoelee network) │
|
||||
│ │ ├─► Carousell www.carousell.com.my │
|
||||
│ │ │ (public internet, GET search)│
|
||||
│ │ └─► Telegram api.telegram.org │
|
||||
│ │ (pinned IPv4 via extra_hosts)│
|
||||
│ │ │
|
||||
│ └── nocodb container (named "nocodb") │
|
||||
└────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
The monitor is an **outbound-only** worker: it has no inbound port, no web UI,
|
||||
and is never reached through the reverse proxy. All state lives in NocoDB
|
||||
(Listings / Settings / IgnoredSellers / IgnoredKeywords), all alerts go out via
|
||||
Telegram.
|
||||
|
||||
---
|
||||
|
||||
## 2. docker-compose.yml, block by block
|
||||
|
||||
### Top-level services
|
||||
|
||||
```yaml
|
||||
services:
|
||||
carousell-monitor:
|
||||
build: .
|
||||
image: carousell-monitor:latest
|
||||
container_name: carousell-monitor
|
||||
restart: unless-stopped
|
||||
```
|
||||
|
||||
| Key | Meaning |
|
||||
|---|---|
|
||||
| `build: .` | Image is built **locally from this directory** (Dockerfile in repo) — nothing is pulled from a registry |
|
||||
| `image: carousell-monitor:latest` | Local tag for the built image; docker compose will rebuild + retag on `up --build` |
|
||||
| `container_name` | Fixed name = stable DNS name on the network, predictable for health checks and logs |
|
||||
| `restart: unless-stopped` | Survive daemon restarts and DSM reboots; stop it manually to keep it down |
|
||||
|
||||
### Naming vs. images
|
||||
|
||||
The image is **private**: built on DSM, never pushed to Docker Hub or any
|
||||
registry. `image:` is just a local convenience tag (`docker images` shows it; the
|
||||
build context is the repo directory).
|
||||
|
||||
### extra_hosts — the Telegram IPv4 pin
|
||||
|
||||
```yaml
|
||||
extra_hosts:
|
||||
- "api.telegram.org:149.154.166.110"
|
||||
```
|
||||
|
||||
**Why this exists (root cause, do not delete):**
|
||||
|
||||
- The container runs inside `bridge_hoelee`, which has **no IPv6**.
|
||||
- Docker's embedded DNS at `127.0.0.11` may return an **IPv6 AAAA record** for
|
||||
`api.telegram.org`; with no IPv6 route, the connection hangs and Telegram
|
||||
sends fail silently.
|
||||
- Pinning the correct IPv4 (as of 2026-09) in `extra_hosts` short-circuits DNS
|
||||
and makes `api.telegram.org` resolve straight to the IPv4.
|
||||
- ⚠ If Telegram calls start timing out again, **re-verify the current IP**:
|
||||
`nslookup api.telegram.org` / `dig +short api.telegram.org` — Telegram rotates
|
||||
IPs. Update the pin, then redeploy.
|
||||
- Note: the **live** stack on DSM pins a slightly different IP than the repo copy —
|
||||
the live one was fixed during an earlier incident. Treat the repo value as
|
||||
the canonical starting point, and verify before assuming.
|
||||
|
||||
### environment
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
NOCODB_URL: ${NOCODB_URL:-http://nocodb:10380}
|
||||
NOCODB_TOKEN: ${NOCODB_TOKEN}
|
||||
NOCODB_BASE_ID: ${NOCODB_BASE_ID:-poqw1zjw3hnsk37}
|
||||
TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN}
|
||||
TELEGRAM_CHAT_ID: ${TELEGRAM_CHAT_ID}
|
||||
TICK_SECONDS: ${TICK_SECONDS:-60}
|
||||
HEALTH_STALE_SECONDS: ${HEALTH_STALE_SECONDS:-600}
|
||||
TZ: Asia/Kuala_Lumpur
|
||||
```
|
||||
|
||||
| Var | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `NOCODB_URL` | `http://nocodb:10380` | NocoDB REST endpoint. `nocodb` = the NocoDB container's name on `bridge_hoelee` (container DNS). Never use a public hostname here. |
|
||||
| `NOCODB_TOKEN` | *(required)* | Workspace-scoped NocoDB PAT (xref `SECRETS.md`). Never hard-code; comes from the stack env/.env. |
|
||||
| `NOCODB_BASE_ID` | `poqw1zjw3hnsk37` | Base "Carousell" — all four tables live under it. |
|
||||
| `TELEGRAM_BOT_TOKEN` | *(required)* | @carousellFoundBot token (xref `SECRETS.md`). |
|
||||
| `TELEGRAM_CHAT_ID` | *(required)* | `5648309582` — @MrFullStackDev. |
|
||||
| `TICK_SECONDS` | `60` | Scheduler granularity: heartbeat + watch-list reload interval. |
|
||||
| `HEALTH_STALE_SECONDS` | `600` | Docker healthcheck tolerance: if last tick older than this → unhealthy. |
|
||||
| `TZ` | `Asia/Kuala_Lumpur` | Container clock (mostly cosmetic; timestamps are written in UTC deliberately for NocoDB). |
|
||||
|
||||
`${VAR:-default}` syntax: compose substitutes the value from the environment /
|
||||
`.env` file, falling back to the literal default when unset. `${NOCODB_TOKEN}`
|
||||
with **no** default means it's mandatory — compose errors if missing.
|
||||
|
||||
### volumes
|
||||
|
||||
```yaml
|
||||
volumes:
|
||||
- carousell-data:/data
|
||||
```
|
||||
|
||||
Named volume `carousell-data` mounted at `/data`. Inside the container that's:
|
||||
|
||||
- `monitor.py` → `DATA_DIR` default → `/data/health.json` (written each tick, read by the healthcheck)
|
||||
- Nothing else is stored there — NocoDB holds all real state.
|
||||
|
||||
Why a volume and not a bind mount: survives container recreation, no host-path
|
||||
permission issues on DSM ACLs, and it's private to the stack (not exposed to the
|
||||
host filesystem).
|
||||
|
||||
### networks
|
||||
|
||||
```yaml
|
||||
networks:
|
||||
- bridge_hoelee
|
||||
|
||||
networks:
|
||||
bridge_hoelee:
|
||||
external: true
|
||||
```
|
||||
|
||||
`external: true` = the network is **pre-existing** (created by the NocoDB stack,
|
||||
typically). Compose does not create it, just joins it. This is what lets the
|
||||
monitor reach `http://nocodb:10380` by container name instead of an IP that
|
||||
drifts (DSM IP history: 192.168.137.2 → 192.168.1.1 → …).
|
||||
|
||||
---
|
||||
|
||||
## 3. Dockerfile
|
||||
|
||||
```dockerfile
|
||||
FROM python:3.11-alpine
|
||||
|
||||
WORKDIR /app
|
||||
COPY monitor.py healthcheck.py /app/
|
||||
RUN mkdir -p /data
|
||||
|
||||
VOLUME ["/data"]
|
||||
|
||||
HEALTHCHECK --interval=60s --timeout=15s --start-period=120s --retries=3 \
|
||||
CMD python /app/healthcheck.py
|
||||
|
||||
CMD ["python", "-u", "/app/monitor.py"]
|
||||
```
|
||||
|
||||
- `python:3.11-alpine` — tiny, stdlib-only code needs no pip deps → fast builds, small image.
|
||||
- `COPY` bakes the script into the image → a rebuild is how code ships (there is no bind-mount of the repo).
|
||||
- `HEALTHCHECK` runs `healthcheck.py` every 60s: exits 0 iff `/data/health.json` exists, is newer than `HEALTH_STALE_SECONDS`, and `ok == true`.
|
||||
- `python -u` — unbuffered stdout so `docker logs` shows ticks in real time.
|
||||
|
||||
### healthcheck.py logic
|
||||
|
||||
```python
|
||||
age = time.time() - int(h.get("last_run_epoch", 0))
|
||||
if age <= STALE and h.get("ok") is True:
|
||||
sys.exit(0) # healthy
|
||||
sys.exit(1) # unhealthy
|
||||
```
|
||||
|
||||
One failed tick (Carousell 403/429/parse error, NocoDB down) → `ok:false` →
|
||||
container turns **unhealthy** until a fully successful tick. That's the tripwire:
|
||||
Portainer shows it red, `docker inspect` reports it, and you can alert on it.
|
||||
|
||||
---
|
||||
|
||||
## 4. Deployment paths
|
||||
|
||||
### A. On DSM via Portainer stack (current production)
|
||||
|
||||
1. Edit code in the repo (`D:\dev\carousell-monitor`).
|
||||
2. Commit + push to Gitea (`git.hoelee.com/hoelee/carousell-monitor`).
|
||||
3. On DSM, the deploy dir `/volume1/docker/carousell-monitor` is a **manual copy,
|
||||
not a git clone** — copy the changed files there:
|
||||
`cp monitor.py /volume1/docker/carousell-monitor/`
|
||||
(backup the old one first, per the repo's `.bak` convention).
|
||||
4. Rebuild + recreate:
|
||||
```bash
|
||||
sudo /usr/local/bin/docker compose up -d --build
|
||||
```
|
||||
`--build` rebuilds the image from the new `monitor.py`; the NocoDB schema
|
||||
bootstrap and dedupe seeding are idempotent, so a rebuild never duplicates rows.
|
||||
|
||||
### B. Local dev (Windows, against LAN NocoDB)
|
||||
|
||||
```bash
|
||||
NOCODB_URL=http://192.168.1.1:10380 \
|
||||
NOCODB_TOKEN=... TELEGRAM_BOT_TOKEN=... TELEGRAM_CHAT_ID=... \
|
||||
python monitor.py
|
||||
```
|
||||
|
||||
Runs the same loop outside Docker — useful for testing code changes before
|
||||
shipping (uses LAN IP instead of container DNS).
|
||||
⚠ `192.168.1.1` drifts; check `:10380` on the current DSM IP first.
|
||||
|
||||
---
|
||||
|
||||
## 5. What to check when something breaks
|
||||
|
||||
| Symptom | Check |
|
||||
|---|---|
|
||||
| Container `unhealthy` | `sudo docker logs carousell-monitor --tail 100` — read the `error` line |
|
||||
|"No application/json state" | Carousell rate-limited/blocked; raise `check_interval_minutes` in Settings |
|
||||
| No Telegram | `TELEGRAM_BOT_TOKEN`/`TELEGRAM_CHAT_ID` correct? `notify` box on the watch? Telegram IP pin in `extra_hosts` still valid? |
|
||||
| "NOCODB_TOKEN not set" | Stack env missing the token; fix in Portainer stack env, redeploy |
|
||||
| NocoDB unreachable | Is `nocodb` container on `bridge_hoelee`? `sudo docker network inspect bridge_hoelee` |
|
||||
| Timestamps 8h off | They are **UTC by design**; NocoDB display TZ → Asia/Kuala_Lumpur |
|
||||
|
||||
---
|
||||
|
||||
## 6. Files in the repo
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `docker-compose.yml` | Stack definition (this doc) |
|
||||
| `Dockerfile` | Image build + healthcheck |
|
||||
| `monitor.py` | The whole monitor (stdlib only) |
|
||||
| `healthcheck.py` | Docker HEALTHCHECK probe |
|
||||
| `.env.example` | Template for local runs (secrets go in `.env`, gitignored) |
|
||||
| `SECRETS.md` | Credential inventory (private repo) |
|
||||
| `DOCUMENTATION.md` | Deployment & operations manual |
|
||||
| `AGENTS.md` | AI-agent entry point |
|
||||
| `COMPOSE-SETUP.md` | This file |
|
||||
|
||||
---
|
||||
|
||||
*Last reviewed: 2026-09-13 (NocoDB 2026.08.1, Docker Compose v2.20.1 on DSM).*
|
||||
+10
-9
@@ -38,7 +38,8 @@ Nothing is pushed to any image registry — the image is built **privately on DS
|
||||
| `monitor.py` | Main loop: schema bootstrap, fetch/parse, NocoDB IO, Telegram |
|
||||
| `healthcheck.py` | HEALTHCHECK probe (reads `/data/health.json`) |
|
||||
| `.env` | Secrets + tunables (gitignored, NOT in the repo) |
|
||||
| `DOCUMENTATION.md` | This file |
|
||||
| `DOCUMENTATION.md` | Deployment & operations manual |
|
||||
| `COMPOSE-SETUP.md` | Stack anatomy reference: compose file, Dockerfile, networks, deployment paths |
|
||||
|
||||
Source of truth for the code is the **private Gitea repo**
|
||||
`git.hoelee.com/hoelee/carousell-monitor` (local checkout `D:\dev\carousell-monitor`).
|
||||
@@ -99,17 +100,16 @@ deleting a table is safe; it is recreated on the next start).
|
||||
| Column | Type | Purpose |
|
||||
|---|---|---|
|
||||
| `watch` | Link → `Settings` | **pick the watch from a dropdown** (belongs-to: many keywords → one watch) — no URL to copy by hand |
|
||||
| `search_url` | Lookup (`Settings.url`) | read-only, auto-filled from the linked watch — this is what the monitor reads |
|
||||
| `keyword` | SingleLineText | if the listing **title** contains this (case-insensitive substring), skip the Telegram alert (still archived) |
|
||||
|
||||
One keyword per row; add multiple rows for multiple keywords. A keyword only
|
||||
silences listings found by the watch you linked — the same keyword never applies to
|
||||
other watches. Rows with no watch or no keyword are ignored.
|
||||
|
||||
`watch` is a real NocoDB Link column, so deleting/renaming a watch shows up in the
|
||||
relation, and there is no hand-copied URL that can silently drift out of sync.
|
||||
(The monitor's bootstrap also builds these two columns itself, including upgrading a
|
||||
legacy URL-typed `search_url` to Link+Lookup.)
|
||||
`watch` is a real NocoDB Link column: the monitor resolves `Settings.url` through it
|
||||
at load time (one fetch of the Settings table per cycle). There is no hand-copied
|
||||
URL that can silently drift out of sync. (The monitor's bootstrap also builds the
|
||||
`watch` column itself and drops any legacy `search_url` column.)
|
||||
|
||||
### `Settings` (the watch list — you manage this)
|
||||
|
||||
@@ -139,8 +139,9 @@ legacy URL-typed `search_url` to Link+Lookup.)
|
||||
(photo + title/price/condition/seller/url), 1 second apart, then sets `notified=true`.
|
||||
- **Silencing**: a pending listing is marked `skip_notify=true` + `notified=true`
|
||||
(no Telegram) if its `seller_name` is in `IgnoredSellers` (global), or if its
|
||||
**title** contains any keyword whose `IgnoredKeywords.search_url` matches the
|
||||
listing's own `search_url` (per-watch, case-insensitive).
|
||||
**title** contains any keyword whose `IgnoredKeywords.watch` links to the
|
||||
same `Settings` row (resolved via `Settings.url` per cycle, per-watch,
|
||||
case-insensitive).
|
||||
- **Failure handling**: a fetch/parse error on any watch marks that tick failed; the
|
||||
container becomes **unhealthy** until the next fully-successful tick. `last_checked_at`
|
||||
is only advanced on success, and a hard-failing watch is rate-limited to one attempt
|
||||
@@ -236,7 +237,7 @@ Health file lives at `/data/health.json` inside the container:
|
||||
|
||||
## 11. Changelog
|
||||
|
||||
- **2026-09-13** `IgnoredKeywords.watch` 改成真正的 NocoDB **Link 列**(Many-to-One → `Settings`),`search_url` 改成 **Lookup 列**自动带出 `Settings.url`(UI 下拉选 watch,不用手抄 URL)。bootstrap 会自建这两列,并把旧版 URL 型 `search_url` 自动升级成 Link+Lookup。
|
||||
- **2026-09-13** `IgnoredKeywords.watch` = 真正的 NocoDB **Link 列**(Many-to-One → `Settings`),UI 下拉选 watch,不用手抄 URL。`search_url` 列已删除——运行时经 watch 链接 + Settings 表解析出 `.url`(每周期一次拉取)。bootstrap 自建 `watch` 列并清理遗留 `search_url`。
|
||||
- **2026-09-13** 新增 per-watch 忽略关键词:`IgnoredKeywords` 表。标题命中该 watch
|
||||
关键词(大小写不敏感子串)时静默归档、不发 Telegram(`skip_notify=true`)。
|
||||
- **2026-09-08** 通知重构:每商品一条图文消息(title/price/condition/seller/url),归档与通知解耦(`notified` 列 + tick 末尾统一发 + 1s 间隔)。图片改用高清 URL(去 `_progressive_thumbnail`)。condition 归一化(New→Brand new、Used→Used,加第 6 档)。listed_at 加 `active_bump` fallback。修复 Telegram IPv6/DNS 问题(compose `extra_hosts` 钉 IPv4)。bot 换 `@carousellFoundBot`。
|
||||
|
||||
@@ -32,8 +32,7 @@ alerts for their listings (still archived, marked `skip_notify=true`).
|
||||
**IgnoredKeywords** — `watch` (Link → Settings) + `keyword`. Per-watch title
|
||||
blocklist: pick the watch from a dropdown, add one keyword per row. A keyword only
|
||||
applies to listings from the linked watch; case-insensitive substring match against
|
||||
the title. `search_url` is a Lookup column that auto-fills from the linked watch.
|
||||
Still archived.
|
||||
the title. Still archived.
|
||||
|
||||
## Run
|
||||
|
||||
@@ -65,4 +64,5 @@ sudo docker compose up -d --build
|
||||
- `monitor.py` — main loop, schema bootstrap, fetch/parse, NocoDB IO, Telegram.
|
||||
- `healthcheck.py` — Docker HEALTHCHECK probe (`/data/health.json`).
|
||||
- `Dockerfile`, `docker-compose.yml`, `.env.example`.
|
||||
- `COMPOSE-SETUP.md` — stack anatomy: compose file, Dockerfile, networking, deploy paths.
|
||||
- `AGENTS.md` — AI-agent entry. `SECRETS.md` — credentials (private repo).
|
||||
|
||||
+38
-44
@@ -252,15 +252,13 @@ def bootstrap():
|
||||
|
||||
|
||||
def _ensure_keyword_watch_link(kw_tid, settings_tid):
|
||||
"""IgnoredKeywords 的 watch(Link→Settings) + search_url(Lookup→Settings.url)。
|
||||
"""IgnoredKeywords.watch: LinkToAnotherRecord (bt) -> Settings。
|
||||
|
||||
NocoDB 2026.08 的列创建 API 与旧版不同:
|
||||
- Link 列: uidt="LinkToAnotherRecord", 顶层 parentId/childId/type
|
||||
- 回读 link 列的 colOptions.fk_column_id(物理外键列)作为 Lookup 的
|
||||
fk_relation_column_id
|
||||
- Lookup 列: uidt="Lookup", 顶层 fk_relation_column_id + fk_lookup_column_id
|
||||
|
||||
幂等:列已存在就跳过。旧版遗留的 URL 型 search_url 会被替换成 Lookup。
|
||||
只维护 watch Link 列(UI 下拉选词条)。search_url 列已废弃:
|
||||
运行时经 watch 链接 + Settings.url 解析,无需冗余列。
|
||||
NocoDB 2026.08 的建列 API:uidt="LinkToAnotherRecord",顶层
|
||||
parentId/childId/type;幂等,旧版遗留的 search_url(URL 或 Lookup 型)
|
||||
会被删除。
|
||||
"""
|
||||
|
||||
def _kw_meta():
|
||||
@@ -294,34 +292,10 @@ def _ensure_keyword_watch_link(kw_tid, settings_tid):
|
||||
if link_col is None:
|
||||
raise RuntimeError("watch link column missing after create")
|
||||
|
||||
# 2. search_url: Lookup (auto-pull Settings.url through the link)
|
||||
if "search_url" in cols and cols["search_url"].get("uidt") == "Lookup":
|
||||
return # already good
|
||||
|
||||
# drop stale non-Lookup search_url (legacy URL column)
|
||||
# 2. drop legacy search_url column (URL type or Lookup) — no longer used
|
||||
if "search_url" in cols:
|
||||
nc("DELETE", f"/api/v2/meta/columns/{cols['search_url']['id']}")
|
||||
|
||||
fk_parent = (link_col.get("colOptions") or {}).get("fk_column_id")
|
||||
if not fk_parent:
|
||||
raise RuntimeError("watch link has no fk_column_id")
|
||||
|
||||
st, sm = nc("GET", f"/api/v2/meta/tables/{settings_tid}")
|
||||
if st != 200:
|
||||
raise RuntimeError(f"read Settings meta failed: {sm}")
|
||||
url_col = next((c for c in sm.get("columns", [])
|
||||
if c.get("title") == "url"), None)
|
||||
if url_col is None:
|
||||
raise RuntimeError("Settings.url column not found")
|
||||
|
||||
st, out = nc("POST", f"/api/v2/meta/tables/{kw_tid}/columns",
|
||||
{"title": "search_url", "uidt": "Lookup",
|
||||
"fk_relation_column_id": fk_parent,
|
||||
"fk_lookup_column_id": url_col["id"],
|
||||
"colOptions": {}})
|
||||
if st != 200:
|
||||
raise RuntimeError(f"create search_url lookup failed: {out}")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Carousell extraction
|
||||
@@ -453,26 +427,46 @@ def load_ignored_sellers(ignored_sellers_tid):
|
||||
return ignored
|
||||
|
||||
|
||||
def load_ignored_keywords(ignored_keywords_tid):
|
||||
def load_ignored_keywords(ignored_keywords_tid, settings_tid):
|
||||
"""从 IgnoredKeywords 表读取忽略关键词,返回 {search_url: {小写关键词}}。
|
||||
|
||||
关键词按 watch(search_url)分组;匹配时大小写不敏感。
|
||||
search_url 是 Lookup 列(经 watch 链接自动带出 Settings.url),
|
||||
API 可能返回字符串或数组,这里统一归一化。
|
||||
关键词按 watch 分组:watch 是 Link 列(指向 Settings 记录),此处拉一次
|
||||
Settings 表把 Id -> url 建索引,再经 watch.Id 解析出 search_url。
|
||||
匹配时大小写不敏感。
|
||||
"""
|
||||
ignored = {}
|
||||
st, j = nc("GET", f"/api/v2/tables/{ignored_keywords_tid}/records?limit=1000")
|
||||
if st != 200:
|
||||
raise RuntimeError(f"load ignored keywords failed: {j}")
|
||||
|
||||
# 先收集 watch 链接的 Settings 行 Id -> 关键词集合
|
||||
by_watch_id = {} # settings row Id -> set(keywords lower)
|
||||
for r in j.get("list", []):
|
||||
url = r.get("search_url") or ""
|
||||
if isinstance(url, list): # Lookup 有时返回数组
|
||||
url = url[0] if url else ""
|
||||
url = str(url).strip()
|
||||
kw = (r.get("keyword") or "").strip().lower()
|
||||
if not url or not kw:
|
||||
if not kw:
|
||||
continue
|
||||
ignored.setdefault(url, set()).add(kw)
|
||||
watch = r.get("watch") or {}
|
||||
if isinstance(watch, list): # Link 列可能返回对象或数组
|
||||
watch = watch[0] if watch else {}
|
||||
sid = watch.get("Id") if isinstance(watch, dict) else None
|
||||
if sid is None:
|
||||
continue
|
||||
by_watch_id.setdefault(sid, set()).add(kw)
|
||||
|
||||
if not by_watch_id:
|
||||
return ignored
|
||||
|
||||
# 一次拉 Settings,把 Id -> url 解析出来
|
||||
st, s = nc("GET", f"/api/v2/tables/{settings_tid}/records?limit=1000")
|
||||
if st != 200:
|
||||
raise RuntimeError(f"load settings for keywords failed: {s}")
|
||||
url_by_id = {r.get("Id"): (r.get("url") or "").strip()
|
||||
for r in s.get("list", [])}
|
||||
|
||||
for sid, kws in by_watch_id.items():
|
||||
url = url_by_id.get(sid)
|
||||
if url:
|
||||
ignored[url] = kws
|
||||
return ignored
|
||||
|
||||
|
||||
@@ -621,7 +615,7 @@ def send_pending_notifications(listings_tid, settings_tid, ignored_sellers_tid,
|
||||
|
||||
# 每轮重新加载忽略列表,中途增删立即生效
|
||||
ignored = load_ignored_sellers(ignored_sellers_tid)
|
||||
ignored_kw_by_url = load_ignored_keywords(ignored_keywords_tid)
|
||||
ignored_kw_by_url = load_ignored_keywords(ignored_keywords_tid, settings_tid)
|
||||
|
||||
# 拉 notified=false 的记录
|
||||
st, j = nc("GET", f"/api/v2/tables/{listings_tid}/records"
|
||||
|
||||
Reference in New Issue
Block a user