Files

T

hoelee 63bbb2795a Drop IgnoredKeywords.search_url column; resolve URL via watch link

The search_url Lookup column is redundant: the watch Link -> Settings row
already defines the association. Remove handling for the column:
- _ensure_keyword_watch_link() no longer creates a Lookup; deletes any
  legacy search_url (URL or Lookup type) on bootstrap
- load_ignored_keywords(kw_tid, settings_tid) resolves Settings.url via
  the watch link, one Settings fetch per cycle (no N+1)
- send_pending_notifications passes settings_tid through
- Add COMPOSE-SETUP.md: detail documentation for the compose stack
  (topology, compose block-by-block, Dockerfile, healthcheck, deploy paths,
  troubleshooting); referenced from README + DOCUMENTATION.md

2026-09-13 18:22:58 +08:00

3.4 KiB

Raw Blame History

AGENTS.md

Project: Carousell new-listing monitor (Python stdlib, Docker, NocoDB, Telegram).

What it does

monitor.py polls Carousell search URLs (sort_by=3 = recent), extracts listings from the server-rendered <script type="application/json"> Redux state (SearchListing.listingCards), dedupes by product_url (param-less), archives to a NocoDB base, and alerts Telegram "<title>: N new listings". Listings whose seller_name is in the IgnoredSellers table are archived but never alerted (skip_notify=true). Listings whose title contains a keyword listed for their watch in the IgnoredKeywords table (linked to the watch's Settings row, case-insensitive) are also archived but never alerted. Runs 24/7 as a Docker container on DSM (network bridge_hoelee, reaches NocoDB at http://nocodb:10380).

Iron rules

Secrets NEVER in code or compose — only env vars / SECRETS.md (private repo). Operational knobs (enabled / notify / check_interval_minutes) live in the NocoDB Settings table, adjustable from the UI without redeploy.
Ignored sellers live in the NocoDB IgnoredSellers table (one seller_name per row). When a pending listing's seller_name matches an ignored seller, the monitor sets skip_notify=true + notified=true and does NOT send Telegram. The list is reloaded every notification cycle, so UI add/remove takes effect immediately.
Ignored keywords live in the NocoDB IgnoredKeywords table: watch is a real Link column → Settings (pick the watch from a dropdown). A listing is silenced when its title contains any keyword for its watch, case-insensitive substring match. Per-watch, not global. Reloaded every cycle, so UI edits take effect immediately. Bootstrap creates the watch column and drops any legacy search_url column.
Dedupe key is product_url (https://www.carousell.com.my/p/<id>/), not the raw listing id and never the query-string URL.
First run per watch seeds the archive with no Telegram alert (last_checked_at null == unseeded).
Docker HEALTHCHECK: container is unhealthy if the last tick is >10 min old or the last run had a failure (failed extract / 403 / rate-limit).
Image thumbnail: image Attachment field stores the remote URL (NocoDB hotlinks it — media.karousell.com is GCS-backed, Access-Control-Allow-Origin: *). The raw URL is also kept in image_url.

Verified facts (2026-09-02)

Carousell search page: 1.6 MB HTML, state blob ~1.27 MB, ~49 listingCards per load. Card fields: listingID, title, price (e.g. "RM85"), thumbnailURL, seller.username, aboveFold[time_created].timestampContent.seconds.low, belowFold paragraphs where paragraph[1] = condition (Like new/Brand new/...).
https://www.carousell.com.my/p/<id>/ 301-redirects to the canonical slug URL.
NocoDB instance: DSM http://192.168.137.2:10380 (IP drifts; container DNS nocodb:10380 on bridge_hoelee). Base Carousell = poqw1zjw3hnsk37. Workspace token in SECRETS.md. v2 API: records use column titles as JSON keys; Attachment field accepts a JSON string [{"path","mimetype","title"}] and keeps the remote URL (does not re-host).

Build / deploy

Private build on DSM — NO registry (do not push to Docker Hub):

# on DSM (repo cloned to /volume1/docker/carousell-monitor)
cd /volume1/docker/carousell-monitor
sudo docker compose up -d --build