hoelee
323fd10838
Replace the legacy DSM-dir 'docker compose up -d --build' flow in AGENTS.md, COMPOSE-SETUP.md, DOCUMENTATION.md, README.md with the current deploy: sync build context to /volume1/docker/portainer/compose/240/, rebuild image if code changed, then PUT /api/stacks/240 (compose + env array, never echo masked *** values). Legacy dir marked as such; live tg IP note synced.
78 lines
4.1 KiB
Markdown
78 lines
4.1 KiB
Markdown
# AGENTS.md
|
|
|
|
Project: Carousell new-listing monitor (Python stdlib, Docker, NocoDB, Telegram).
|
|
|
|
## What it does
|
|
|
|
`monitor.py` polls Carousell search URLs (sort_by=3 = recent), extracts listings
|
|
from the server-rendered `<script type="application/json">` Redux state
|
|
(`SearchListing.listingCards`), dedupes by `product_url` (param-less), archives to a
|
|
NocoDB base, and alerts Telegram `"<title>: N new listings"`. Listings whose
|
|
`seller_name` is in the `IgnoredSellers` table are archived but never alerted
|
|
(`skip_notify=true`). Listings whose **title** contains a keyword listed for their
|
|
watch in the `IgnoredKeywords` table (linked to the watch's `Settings` row,
|
|
case-insensitive) are also archived but never alerted. Runs 24/7 as a Docker
|
|
container on DSM (network `bridge_hoelee`, reaches NocoDB at `http://nocodb:10380`).
|
|
|
|
## Iron rules
|
|
|
|
- Secrets NEVER in code or compose — only env vars / `SECRETS.md` (private repo).
|
|
Operational knobs (`enabled` / `notify` / `check_interval_minutes`) live in the
|
|
NocoDB **Settings** table, adjustable from the UI without redeploy.
|
|
- Ignored sellers live in the NocoDB **IgnoredSellers** table (one `seller_name`
|
|
per row). When a pending listing's `seller_name` matches an ignored seller, the
|
|
monitor sets `skip_notify=true` + `notified=true` and does NOT send Telegram.
|
|
The list is reloaded every notification cycle, so UI add/remove takes effect
|
|
immediately.
|
|
- Ignored keywords live in the NocoDB **IgnoredKeywords** table: `watch` is a real
|
|
**Link column → `Settings`** (pick the watch from a dropdown). A listing is
|
|
silenced when its **title** contains any keyword for its watch, case-insensitive
|
|
substring match. Per-watch, not global. Reloaded every cycle, so UI edits take
|
|
effect immediately. Bootstrap creates the `watch` column and drops any legacy
|
|
`search_url` column.
|
|
- Dedupe key is `product_url` (`https://www.carousell.com.my/p/<id>/`), not the raw
|
|
listing id and never the query-string URL.
|
|
- First run per watch seeds the archive with **no** Telegram alert (`last_checked_at`
|
|
null == unseeded).
|
|
- Docker HEALTHCHECK: container is unhealthy if the last tick is >10 min old or the
|
|
last run had a failure (failed extract / 403 / rate-limit).
|
|
- Image thumbnail: `image` Attachment field stores the remote URL (NocoDB hotlinks
|
|
it — media.karousell.com is GCS-backed, `Access-Control-Allow-Origin: *`). The raw
|
|
URL is also kept in `image_url`.
|
|
|
|
## Verified facts (2026-09-02)
|
|
|
|
- Carousell search page: 1.6 MB HTML, state blob ~1.27 MB, ~49 `listingCards` per
|
|
load. Card fields: `listingID`, `title`, `price` (e.g. "RM85"), `thumbnailURL`,
|
|
`seller.username`, `aboveFold[time_created].timestampContent.seconds.low`,
|
|
`belowFold` paragraphs where `paragraph[1]` = condition (Like new/Brand new/...).
|
|
- `https://www.carousell.com.my/p/<id>/` 301-redirects to the canonical slug URL.
|
|
- NocoDB instance: DSM `http://192.168.137.2:10380` (IP drifts; container DNS
|
|
`nocodb:10380` on bridge_hoelee). Base `Carousell` = `poqw1zjw3hnsk37`.
|
|
Workspace token in SECRETS.md. v2 API: records use column *titles* as JSON keys;
|
|
Attachment field accepts a JSON string `[{"path","mimetype","title"}]` and keeps the
|
|
remote URL (does not re-host).
|
|
|
|
## Build / deploy
|
|
|
|
Container is managed by **Portainer stack 240** (standalone; compose + build
|
|
context live at `/volume1/docker/portainer/compose/240/` on DSM). Deploy flow:
|
|
|
|
1. Edit code, commit, push to Gitea (`git.hoelee.com/hoelee/carousell-monitor`).
|
|
2. Sync the build-context files to `/volume1/docker/portainer/compose/240/`
|
|
(`monitor.py`, `healthcheck.py`, `Dockerfile`, `docker-compose.yml`).
|
|
3. If `monitor.py` / `Dockerfile` changed, rebuild the image first (Portainer's
|
|
standalone PUT does **not** rebuild):
|
|
|
|
```bash
|
|
sudo /usr/local/bin/docker build -t carousell-monitor:latest \
|
|
/volume1/docker/portainer/compose/240
|
|
```
|
|
|
|
4. Update stack 240 via the Portainer API: `PUT /api/stacks/240?endpointId=2`
|
|
with the repo `docker-compose.yml` as `stackFileContent` and the current env
|
|
array (see the `portainer-api` skill; ⚠ never echo masked `***` values back).
|
|
|
|
Portainer recreates the container with the new config. Private build on DSM —
|
|
NO registry (do not push to Docker Hub).
|