diff --git a/DOCUMENTATION.md b/DOCUMENTATION.md new file mode 100644 index 0000000..36de921 --- /dev/null +++ b/DOCUMENTATION.md @@ -0,0 +1,200 @@ +# carousell-monitor — Deployment & Operations Documentation + +Carousell new-listing monitor: watches Carousell search pages (sorted by *recent*), +archives every listing into a NocoDB base (with image thumbnail), and alerts Telegram +when a genuinely new listing appears. Runs 24/7 as a Docker container on the DSM. + +--- + +## 1. Architecture + +``` +Docker container (carousell-monitor, DSM, network bridge_hoelee) + ├─ every 60 s (TICK_SECONDS) ─ reads watch list from NocoDB Settings table + ├─ per watch (on its own check_interval_minutes) ─ GET the Carousell search URL + │ ├─ parse embedded JSON state → SearchListing.listingCards[] + │ ├─ dedupe by product_url (param-less listing URL) + │ ├─ INSERT new rows into NocoDB Listings table + │ └─ (after first seed) send Telegram ": N new listings" + ├─ writes /data/health.json each tick → Docker HEALTHCHECK + └─ reaches: NocoDB http://nocodb:10380 (container DNS, bridge_hoelee) + Carousell www.carousell.com.my (public internet) + Telegram api.telegram.org (public internet) +``` + +Nothing is pushed to any image registry — the image is built **privately on DSM** +(`build: .`). Secrets come from the `.env` file next to the compose file. + +--- + +## 2. Files in this folder + +| File | Purpose | +|---|---| +| `docker-compose.yml` | Stack definition (build, env, volume, network) | +| `Dockerfile` | python:3.11-alpine + monitor.py + healthcheck.py, HEALTHCHECK | +| `monitor.py` | Main loop: schema bootstrap, fetch/parse, NocoDB IO, Telegram | +| `healthcheck.py` | HEALTHCHECK probe (reads `/data/health.json`) | +| `.env` | Secrets + tunables (gitignored, NOT in the repo) | +| `DOCUMENTATION.md` | This file | + +Source of truth for the code is the **private Gitea repo** +`git.hoelee.com/hoelee/carousell-monitor` (local checkout `D:\dev\carousell-monitor`). +Credentials are documented in `SECRETS.md` there. + +--- + +## 3. Environment variables (`.env`) + +| Var | Value / default | Notes | +|---|---|---| +| `NOCODB_URL` | `http://nocodb:10380` | container DNS on `bridge_hoelee`; LAN form `http://192.168.137.2:10380` | +| `NOCODB_TOKEN` | *(secret)* | workspace-scoped NocoDB PAT | +| `NOCODB_BASE_ID` | `poqw1zjw3hnsk37` | base "Carousell" | +| `TELEGRAM_BOT_TOKEN` | *(secret)* | @HoeleeAgentBot | +| `TELEGRAM_CHAT_ID` | `5648309582` | alert destination | +| `TICK_SECONDS` | `60` | scheduler granularity | +| `HEALTH_STALE_SECONDS` | `600` | healthcheck staleness window | + +**Secrets = env vars (`.env`). Operational knobs = NocoDB Settings table.** +Speed, enable/disable, and notify on/off are all changed from the NocoDB UI — no +redeploy, no `.env` edit. + +--- + +## 4. NocoDB schema (base `Carousell`, id `poqw1zjw3hnsk37`) + +The container bootstraps both tables on startup if they are missing (idempotent — +deleting a table is safe; it is recreated on the next start). + +### `Listings` (archive — every seen listing) + +| Column | Type | Purpose | +|---|---|---| +| `Id` | auto | primary key | +| `product_url` | URL | **unique dedupe key** — `https://www.carousell.com.my/p/<id>/`, no query params | +| `title` | SingleLineText | listing title | +| `price` | Decimal | numeric price, "RM" stripped (e.g. `85.00`) — filterable/sortable | +| `condition` | SingleSelect | Brand new / Like new / Lightly used / Well used / Heavily used | +| `image_url` | URL | raw Carousell thumbnail URL (media.karousell.com) | +| `image` | Attachment | thumbnail — NocoDB hotlinks the URL; renders in grid view | +| `seller_name` | SingleLineText | seller username | +| `seller_url` | URL | `https://www.carousell.com.my/u/<username>/` | +| `search_title` | SingleLineText | which watch found it (denormalized) | +| `search_url` | URL | the watch's search URL | +| `listed_at` | DateTime (UTC) | listing's `time_created` on Carousell | +| `first_seen_at` | DateTime (UTC) | when the monitor first captured it | + +### `Settings` (the watch list — you manage this) + +| Column | Type | Purpose | +|---|---|---| +| `Id` | auto | primary key | +| `title` | SingleLineText | human label, e.g. "Uniform" | +| `url` | URL | full Carousell search URL (with `sort_by=3`) | +| `enabled` | Checkbox | false = paused (skipped entirely) | +| `notify` | Checkbox | false = archive only, no Telegram ping | +| `check_interval_minutes` | Number | per-watch poll interval (default 5) | +| `last_checked_at` | DateTime | null = not yet seeded (first run archives silently) | + +--- + +## 5. Runtime behaviour + +- **Tick loop**: every `TICK_SECONDS` the container reloads the Settings table and + polls each *enabled* watch whose interval has elapsed. +- **Dedupe**: `product_url` is the identity. On startup the container loads every + existing `product_url` from NocoDB into an in-memory set; a listing is "new" only + if its URL is not in that set. +- **First run per watch** (`last_checked_at` is null): seeds the current ~49 listings + as an archive with **no Telegram alert**. This prevents a 98-message flood on setup. +- **Afterwards**: only genuinely-new listings are inserted **and** alerted, as + `"<title>: N new listings"` (one message per watch that had new items, no details). +- **Failure handling**: a fetch/parse error on any watch marks that tick failed; the + container becomes **unhealthy** until the next fully-successful tick. `last_checked_at` + is only advanced on success, and a hard-failing watch is rate-limited to one attempt + per tick so it doesn't hammer the site. + +--- + +## 6. Operations (all from the NocoDB UI — no SSH, no redeploy) + +| Want to… | Do this | +|---|---| +| Add a keyword | Add a row to `Settings`: `title`, full `url` (open Carousell → search → sort "Recent" → copy URL), `enabled` ✓, `notify` ✓, interval | +| Remove / pause a watch | Set `enabled` = false (or delete the row) | +| Stop Telegram pings but keep archiving | Set `notify` = false | +| Change how often it checks | Edit `check_interval_minutes` (5 = every 5 min) | +| See what's new | Open `Listings`, sort by `first_seen_at` desc | +| Browse with images | `Listings` grid view — the `image` column renders thumbnails | + +The URL for a watch must keep `sort_by=3` (recent) so new listings sort first, e.g.: + +``` +https://www.carousell.com.my/search/uniform?addRecent=true&canChangeKeyword=true&includeSuggestions=true&sort_by=3&t-search_query_source=direct_search +``` + +--- + +## 7. Updating the code + +The image is built **on DSM** from this folder. To ship a code change: + +```bash +# 1) edit monitor.py / Dockerfile / compose in the repo (D:\dev\carousell-monitor) +# 2) copy the changed file(s) to this folder, then: +cd /volume1/docker/carousell-monitor +sudo /usr/local/bin/docker compose up -d --build +``` + +`--build` rebuilds the image and recreates the container; the NocoDB schema bootstrap +and dedupe seeding are idempotent, so a rebuild never duplicates rows. + +--- + +## 8. Troubleshooting + +| Symptom | Likely cause | Fix | +|---|---|---| +| Container shows **unhealthy** | last tick failed (403/429/parse) or >10 min stale | `sudo /usr/local/bin/docker logs carousell-monitor --tail 50` — read the `error` | +| "no application/json state found" | Carousell rate-limited / blocked the request | wait; consider raising `check_interval_minutes` | +| No Telegram messages | token/chat wrong, or `notify` off | verify `.env` `TELEGRAM_BOT_TOKEN`; check `notify` checkbox | +| Thumbnails missing in NocoDB | `image` column isn't Attachment, or Carousell blocked hotlink | confirm `image` is Attachment type; `image_url` (URL) always works as a link | +| NocoDB unreachable | DSM IP drifted, or not on `bridge_hoelee` | from DSM: `sudo /usr/local/bin/docker network inspect bridge_hoelee`; NocoDB container must be named `nocodb` | +| `listed_at`/`first_seen_at` look 8 h off | they are **UTC** (NocoDB parses naive datetimes as UTC) | expected — set your NocoDB display timezone to MYT (Asia/Kuala_Lumpur) if you want local times | + +### Handy commands (run on DSM) + +```bash +D="sudo /usr/local/bin/docker" +$D ps --filter name=carousell-monitor # status + health +$D logs carousell-monitor --tail 100 # last log lines +$D inspect carousell-monitor --format '{{.State.Health.Status}}' # health +$D restart carousell-monitor # restart (state is in NocoDB, safe) +``` + +Health file lives at `/data/health.json` inside the container: +`{"last_run_epoch": ..., "ok": true, "error": "", "watch_count": N, "new_this_tick": N}`. + +--- + +## 9. Key identifiers + +| Item | Value | +|---|---| +| Repo (private) | `git.hoelee.com/hoelee/carousell-monitor` | +| Local checkout | `D:\dev\carousell-monitor` | +| DSM deploy dir | `/volume1/docker/carousell-monitor` | +| Container | `carousell-monitor` (network `bridge_hoelee`) | +| NocoDB base | `Carousell` = `poqw1zjw3hnsk37` (workspace `wal4hatt`) | +| Tables | `Listings` + `Settings` (bootstrap finds by title) | +| Telegram | `@HoeleeAgentBot` → chat `5648309582` | + +--- + +## 10. Security notes + +- `.env` contains secrets and is **gitignored** — it exists only on this DSM and is + never committed. Credential inventory: `SECRETS.md` in the repo. +- The image is private (built on DSM, never pushed to Docker Hub). +- NocoDB token is workspace-scoped; regenerate in NocoDB if it leaks and update `.env`.