hoelee/carousell-monitor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Docs overhaul: beginner quick start + all-in-one NocoDB stack

Browse Source 
- README rewritten around a copy-paste quick start (CLI and Portainer),
  verified alert test, day-to-day NocoDB operations, troubleshooting table, FAQ
- new docker-compose.allinone.yml: NocoDB (pinned 2026.09.0, SQLite) + monitor
  on a private network, with healthchecks and the Telegram IPv4 pin documented
- docs/: QUICKSTART-PORTAINER, TELEGRAM-SETUP, NOCODB-SETUP, ARCHITECTURE,
  OPERATIONS, TROUBLESHOOTING (replace DOCUMENTATION.md + COMPOSE-SETUP.md)
- secrets: SECRETS.md is gitignored and untracked; tracked template is
  SECRETS.example.md; real base id / chat id removed from .env.example
- LICENSE (MIT), .gitignore/.dockerignore tidied
- AGENTS.md: layout, iron rules, verification gates; host-specific deploy
  details moved to the gitignored OPS-INTERNAL.md
This commit is contained in:
hoelee
2026-10-06 16:37:59 +08:00
parent 44851a1c29
commit 5d9db48a26
16 changed files with 1354 additions and 647 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.dockerignore
+3 -1
View File
@@ -6,4 +6,6 @@ __pycache__
*.pyc
*.md
SECRETS.md
docker-compose.yml
docker-compose*.yml
LICENSE
test_*.py
.env.example
+66 -11
View File
@@ -1,19 +1,74 @@
# Copy to .env and fill in. All values are required.
# =============================================================================
# carousell-monitor — example environment file
#
# Copy to `.env` and fill in:
# cp .env.example .env
#
# `.env` is gitignored. NEVER commit real values.
# Every value below is read by docker compose (`${VAR}`) and passed into the
# container. Operational knobs (which searches to watch, how often) are NOT
# here — they live in NocoDB's `Settings` table, so you can change them from
# the web UI without a restart.
# =============================================================================
# NocoDB (container DNS when on bridge_hoelee; LAN IP from a desktop):
NOCODB_URL=http://nocodb:10380
NOCODB_BASE_ID=poqw1zjw3hnsk37
# ----------------------------------------------------------------------------
# NocoDB (the archive database) — REQUIRED
# ----------------------------------------------------------------------------
# Where the monitor talks to NocoDB.
# all-in-one stack : http://nocodb:8080 (the compose service name)
# existing NocoDB : the container's name on the shared docker network
NOCODB_URL=http://nocodb:8080
# The base that holds the four tables. Open the base in NocoDB and copy the id
# out of the browser URL:
# http://<host>:8080/dashboard/#/nc/base/<THIS_PART>/...
NOCODB_BASE_ID=
# NocoDB API token (starts with `nc_pat_`). Create it in NocoDB:
# avatar (bottom-left) -> Account Settings -> Tokens -> Create token
# You may leave it blank for the very first start — the monitor refuses to run
# with a clear error until it is set.
NOCODB_TOKEN=
# Telegram alerts (@HoeleeAgentBot):
# ----------------------------------------------------------------------------
# Telegram (where alerts are sent) — REQUIRED
# ----------------------------------------------------------------------------
# 1. Talk to @BotFather in Telegram -> /newbot -> copy the token it gives you.
# 2. Get your numeric chat id from @userinfobot (send it any message).
# For a group: add the bot to the group, send a message, then read chat.id
# from https://api.telegram.org/bot<TOKEN>/getUpdates
TELEGRAM_BOT_TOKEN=
TELEGRAM_CHAT_ID=5648309582
TELEGRAM_CHAT_ID=
# Optional tuning:
# ----------------------------------------------------------------------------
# NocoDB container settings (all-in-one stack only)
# ----------------------------------------------------------------------------
# Signing secret for NocoDB login sessions. Generate one:
# openssl rand -hex 32
NC_AUTH_JWT_SECRET=
# Host port for the NocoDB web UI -> container port 8080.
NOCODB_PORT=8080
# ----------------------------------------------------------------------------
# Optional tuning (defaults are sane; leave blank to use the defaults)
# ----------------------------------------------------------------------------
# How often the monitor wakes up (seconds). This is the heartbeat, not the
# per-search interval — that one is `check_interval_minutes` in NocoDB.
TICK_SECONDS=60
# Minimum pause (seconds) between two Carousell requests inside one tick.
# Prevents a burst of requests when many searches come due at once. 0 disables.
FETCH_GAP_SECONDS=1
HEALTH_STALE_SECONDS=600
# 连续失败几次才发 Telegram 故障告警（去抖）:
ERROR_ALERT_AFTER=3
# 失败 watch 占比达到该值才判定整轮故障（1.0 = 全部失败）:
# A tick that fails every watch marks the container unhealthy. One flaky
# search should not, so the tick is only "failed" when this fraction of the
# watches failed (1.0 = all of them must fail; 0 = never fail).
FAILURE_RATIO_THRESHOLD=1.0
# Send the Telegram failure alert only after this many consecutive failed
# ticks, and one recovery notice when it clears (debounce, avoids spam).
ERROR_ALERT_AFTER=3
# Container clock. Timestamps written to NocoDB are UTC on purpose.
TZ=Asia/Kuala_Lumpur
.gitignore
+7
View File
@@ -1,4 +1,11 @@
.env
.env.local
SECRETS.md
OPS-INTERNAL.md
__pycache__/
*.pyc
*.pyo
.DS_Store
*.bak
*.bak[0-9]
_tmp_*
AGENTS.md
+86 -58
View File
@@ -1,77 +1,105 @@
# AGENTS.md
Project: Carousell new-listing monitor (Python stdlib, Docker, NocoDB, Telegram).
Entry point for AI coding agents working on **carousell-monitor**: a Python-stdlib
monitor that polls Carousell search pages, archives every listing to NocoDB, and
alerts Telegram.
Read this file, then the doc that matches your task. Keep it updated when the
layout or the deployment story changes.
## What it does
`monitor.py` polls Carousell search URLs (sort_by=3 = recent), extracts listings
`monitor.py` polls Carousell search URLs (`sort_by=3` = recent), extracts listings
from the server-rendered `<script type="application/json">` Redux state
(`SearchListing.listingCards`), dedupes by `product_url` (param-less), archives to a
NocoDB base, and alerts Telegram `"<title>: N new listings"`. Listings whose
`seller_name` is in the `IgnoredSellers` table are archived but never alerted
(`skip_notify=true`). Listings whose **title** contains a keyword listed for their
watch in the `IgnoredKeywords` table (linked to the watch's `Settings` row,
case-insensitive) are also archived but never alerted. Runs 24/7 as a Docker
container on DSM (network `bridge_hoelee`, reaches NocoDB at `http://nocodb:10380`).
(`SearchListing.listingCards`), dedupes by `product_url` (param-less), archives to
a NocoDB base, and alerts Telegram `"<title>: N new listings"`.
Silencing: a listing whose `seller_name` is in `IgnoredSellers` is archived but
never alerted (`skip_notify=true`); likewise a listing whose **title** contains a
keyword listed for its watch in `IgnoredKeywords` (a real Link column → `Settings`,
case-insensitive substring).
## Iron rules
- Secrets NEVER in code or compose — only env vars / `SECRETS.md` (private repo).
Operational knobs (`enabled` / `notify` / `check_interval_minutes`) live in the
NocoDB **Settings** table, adjustable from the UI without redeploy.
- Ignored sellers live in the NocoDB **IgnoredSellers** table (one `seller_name`
per row). When a pending listing's `seller_name` matches an ignored seller, the
monitor sets `skip_notify=true` + `notified=true` and does NOT send Telegram.
The list is reloaded every notification cycle, so UI add/remove takes effect
immediately.
- Ignored keywords live in the NocoDB **IgnoredKeywords** table: `watch` is a real
**Link column → `Settings`** (pick the watch from a dropdown). A listing is
silenced when its **title** contains any keyword for its watch, case-insensitive
substring match. Per-watch, not global. Reloaded every cycle, so UI edits take
effect immediately. Bootstrap creates the `watch` column and drops any legacy
`search_url` column.
- Dedupe key is `product_url` (`https://www.carousell.com.my/p/<id>/`), not the raw
listing id and never the query-string URL.
- First run per watch seeds the archive with **no** Telegram alert (`last_checked_at`
null == unseeded).
- Docker HEALTHCHECK: container is unhealthy if the last tick is >10 min old or the
last run had a failure (failed extract / 403 / rate-limit).
- Image thumbnail: `image` Attachment field stores the remote URL (NocoDB hotlinks
it — media.karousell.com is GCS-backed, `Access-Control-Allow-Origin: *`). The raw
URL is also kept in `image_url`.
- Secrets NEVER in code or compose — env vars (`.env`, gitignored) or a stack's
environment. `SECRETS.example.md` is the tracked template; the real inventory is
the gitignored `SECRETS.md`. No default base id, token or chat id may be shipped.
- Operational knobs (`enabled` / `notify` / `check_interval_minutes`) live in the
NocoDB **Settings** table, adjustable from the UI without a redeploy. Only
credentials and tuning belong in the environment.
- Dedupe key is `product_url` (`https://www.carousell.com.my/p/<id>/`), never the
raw listing id and never the query-string URL.
- First run per watch seeds the archive with **no** Telegram alert
(`last_checked_at` null == unseeded).
- Docker HEALTHCHECK: unhealthy when the last tick is older than
`HEALTH_STALE_SECONDS` or the last run failed.
- `image` is an Attachment column storing the remote URL (NocoDB hotlinks it; the
raw URL is also kept in `image_url`).
- Never change `docker-compose.yml` casually: it is the production variant that
joins an existing NocoDB network. The beginner/fresh-install path is
`docker-compose.allinone.yml`.
## Verified facts (2026-09-02)
## Layout
- Carousell search page: 1.6 MB HTML, state blob ~1.27 MB, ~49 `listingCards` per
load. Card fields: `listingID`, `title`, `price` (e.g. "RM85"), `thumbnailURL`,
| File | Role |
|---|---|
| `monitor.py` | the whole loop (stdlib only) |
| `healthcheck.py` | Docker HEALTHCHECK probe over `/data/health.json` |
| `Dockerfile` | python:3.11-alpine, no dependencies |
| `docker-compose.allinone.yml` | NocoDB + monitor (fresh install) |
| `docker-compose.yml` | monitor only, existing NocoDB network (production) |
| `test_pagination.py` | stdlib regression suite — must pass before any commit |
| `docs/` | long-form guides: QUICKSTART-PORTAINER, TELEGRAM-SETUP, NOCODB-SETUP, ARCHITECTURE, OPERATIONS, TROUBLESHOOTING |
| `OPS-INTERNAL.md` | gitignored: this deployment's hosts, paths, stack ids (may not exist) |
## Verified facts (Carousell page shape, 2026-09)
- Search page: ~1.6 MB HTML, state blob ~1.27 MB, ~49 `listingCards` per load.
Card fields: `listingID`, `title`, `price` (e.g. "RM85"), `thumbnailURL`,
`seller.username`, `aboveFold[time_created].timestampContent.seconds.low`,
`belowFold` paragraphs where `paragraph[1]` = condition (Like new/Brand new/...).
`belowFold` paragraphs where `paragraph[1]` = condition.
- `https://www.carousell.com.my/p/<id>/` 301-redirects to the canonical slug URL.
- NocoDB instance: DSM `http://192.168.137.2:10380` (IP drifts; container DNS
`nocodb:10380` on bridge_hoelee). Base `Carousell` = `poqw1zjw3hnsk37`.
Workspace token in SECRETS.md. v2 API: records use column *titles* as JSON keys;
Attachment field accepts a JSON string `[{"path","mimetype","title"}]` and keeps the
remote URL (does not re-host).
- Soft rate limiting returns **HTTP 200 with `listingCards: null`** — treat as a
retryable error, never as "no results".
- NocoDB v2 API: records use column *titles* as JSON keys; an Attachment field
accepts a JSON string `[{"path","mimetype","title"}]` and keeps the remote URL.
## Build / deploy
## Working on the code
Container is managed by **Portainer stack 240** (standalone; compose + build
context live at `/volume1/docker/portainer/compose/240/` on DSM). Deploy flow:
- Stdlib only. Do not add dependencies without a very good reason.
- Run `python test_pagination.py` (exit 0 = pass) and
`python -m pyflakes monitor.py` before committing. pyflakes currently reports one
cosmetic finding (`_send_photo_multipart` has a dead `parts = []` local) — anything
beyond that is yours.
- New env knobs must be added to `monitor.py`, both compose files, `.env.example`
and the tables in `README.md` + `docs/ARCHITECTURE.md` in the same commit.
- Tests must execute the real loader functions, not mocks of them, and must be
mutation-checked (reintroduce the bug, watch the check fail, restore).
- `monitor.py` is checked out with CRLF on Windows; prefer a Python
`read → replace → write(newline='')` over fuzzy patch tools on big blocks.
- Grepping pitfall: a `NameError`/arity error inside a tick does **not** fail at
import. The container keeps running and the only symptom is
`/data/health.json` reporting `ok:false` with a `tick error: …` string.
1. Edit code, commit, push to Gitea (`git.hoelee.com/hoelee/carousell-monitor`).
2. Sync the build-context files to `/volume1/docker/portainer/compose/240/`
(`monitor.py`, `healthcheck.py`, `Dockerfile`, `docker-compose.yml`).
3. If `monitor.py` / `Dockerfile` changed, rebuild the image first (Portainer's
standalone PUT does **not** rebuild):
## Deploying
```bash
sudo /usr/local/bin/docker build -t carousell-monitor:latest \
/volume1/docker/portainer/compose/240
```
Generic:
4. Update stack 240 via the Portainer API: `PUT /api/stacks/240?endpointId=2`
with the repo `docker-compose.yml` as `stackFileContent` and the current env
array (see the `portainer-api` skill; ⚠ never echo masked `***` values back).
```bash
git pull
docker compose -f docker-compose.allinone.yml up -d --build
```
Portainer recreates the container with the new config. Private build on DSM —
NO registry (do not push to Docker Hub).
This project's own deployment (host paths, Portainer stack id, NocoDB instance,
tunnel) is deliberately kept out of the repo: see the gitignored `OPS-INTERNAL.md`
or the operator's `carousell-monitor` skill. Two rules that were learned the hard
way:
1. The **build context is a plain directory** on the host, not a git checkout:
the running container is built from the files copied next to the compose file.
Sync `monitor.py` (and `docker-compose*.yml` / `Dockerfile` when they change)
and then rebuild — a stack PUT alone does not run `--build`.
2. **Compare the live container's environment with the file you are about to
deploy before rebuilding.** Env drift between the compose file, the stack's
stored env and the running container is the single most common cause of a
"fixed" deploy that changes nothing.
COMPOSE-SETUP.md
-266
View File
@@ -1,266 +0,0 @@
# COMPOSE-SETUP.md — Carousell Monitor Docker Stack, Explained
Line-by-line anatomy of the compose stack, how it connects to the rest of the
homelab, and the two ways to deploy it. Complementary to `DOCUMENTATION.md`
(ops) and `AGENTS.md` (agent entry); this file is the *stack reference*.
---
## 1. Topology
```
┌────────────────────────────────────────────────┐
│ DSM host (Synology) │
│ │
Internet ─ 443 ──► │ DSM nginx reverse proxy │
│ │ (not involved for this container — │
│ │ it makes outbound calls only) │
│ ▼ │
│ Docker network bridge_hoelee (external) │
│ │ │
│ ├── carousell-monitor (this stack) │
│ │ │ │
│ │ ├─► NocoDB http://nocodb:10380 │
│ │ │ (same bridge_hoelee network) │
│ │ ├─► Carousell www.carousell.com.my │
│ │ │ (public internet, GET search)│
│ │ └─► Telegram api.telegram.org │
│ │ (pinned IPv4 via extra_hosts)│
│ │ │
│ └── nocodb container (named "nocodb") │
└────────────────────────────────────────────────┘
```
The monitor is an **outbound-only** worker: it has no inbound port, no web UI,
and is never reached through the reverse proxy. All state lives in NocoDB
(Listings / Settings / IgnoredSellers / IgnoredKeywords), all alerts go out via
Telegram.
---
## 2. docker-compose.yml, block by block
### Top-level services
```yaml
services:
carousell-monitor:
build: .
image: carousell-monitor:latest
container_name: carousell-monitor
restart: unless-stopped
```
| Key | Meaning |
|---|---|
| `build: .` | Image is built **locally from this directory** (Dockerfile in repo) — nothing is pulled from a registry |
| `image: carousell-monitor:latest` | Local tag for the built image; docker compose will rebuild + retag on `up --build` |
| `container_name` | Fixed name = stable DNS name on the network, predictable for health checks and logs |
| `restart: unless-stopped` | Survive daemon restarts and DSM reboots; stop it manually to keep it down |
### Naming vs. images
The image is **private**: built on DSM, never pushed to Docker Hub or any
registry. `image:` is just a local convenience tag (`docker images` shows it; the
build context is the repo directory).
### extra_hosts — the Telegram IPv4 pin
```yaml
extra_hosts:
- "api.telegram.org:149.154.166.110"
```
**Why this exists (root cause, do not delete):**
- The container runs inside `bridge_hoelee`, which has **no IPv6**.
- Docker's embedded DNS at `127.0.0.11` may return an **IPv6 AAAA record** for
`api.telegram.org`; with no IPv6 route, the connection hangs and Telegram
sends fail silently.
- Pinning the correct IPv4 (as of 2026-09) in `extra_hosts` short-circuits DNS
and makes `api.telegram.org` resolve straight to the IPv4.
- ⚠ If Telegram calls start timing out again, **re-verify the current IP**:
`nslookup api.telegram.org` / `dig +short api.telegram.org` — Telegram rotates
IPs. Update the pin, then redeploy.
- Note: the **live** stack pins the **same** IP as the repo (synced 2026-09-13).
Treat the repo value as canonical; re-verify on any timeout.
### environment
```yaml
environment:
NOCODB_URL: ${NOCODB_URL:-http://nocodb:10380}
NOCODB_TOKEN: ${NOCODB_TOKEN}
NOCODB_BASE_ID: ${NOCODB_BASE_ID:-poqw1zjw3hnsk37}
TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN}
TELEGRAM_CHAT_ID: ${TELEGRAM_CHAT_ID}
TICK_SECONDS: ${TICK_SECONDS:-60}
FETCH_GAP_SECONDS: ${FETCH_GAP_SECONDS:-1}
HEALTH_STALE_SECONDS: ${HEALTH_STALE_SECONDS:-600}
ERROR_ALERT_AFTER: ${ERROR_ALERT_AFTER:-3}
FAILURE_RATIO_THRESHOLD: ${FAILURE_RATIO_THRESHOLD:-1.0}
TZ: Asia/Kuala_Lumpur
```
| Var | Default | Meaning |
|---|---|---|
| `NOCODB_URL` | `http://nocodb:10380` | NocoDB REST endpoint. `nocodb` = the NocoDB container's name on `bridge_hoelee` (container DNS). Never use a public hostname here. |
| `NOCODB_TOKEN` | *(required)* | Workspace-scoped NocoDB PAT (xref `SECRETS.md`). Never hard-code; comes from the stack env/.env. |
| `NOCODB_BASE_ID` | `poqw1zjw3hnsk37` | Base "Carousell" — all four tables live under it. |
| `TELEGRAM_BOT_TOKEN` | *(required)* | @carousellFoundBot token (xref `SECRETS.md`). |
| `TELEGRAM_CHAT_ID` | *(required)* | `5648309582` — @MrFullStackDev. |
| `TICK_SECONDS` | `60` | Scheduler granularity: heartbeat + watch-list reload interval. |
| `FETCH_GAP_SECONDS` | `1` | Minimum pause (s) between watch URL fetches within one tick — prevents request bursts (default 1; set `0` to disable). |
| `HEALTH_STALE_SECONDS` | `600` | Docker healthcheck tolerance: if last tick older than this → unhealthy. |
| `ERROR_ALERT_AFTER` | `3` | Consecutive failed ticks before a Telegram failure alert fires (debounce). Sent once on entering the failure state and once on recovery. |
| `FAILURE_RATIO_THRESHOLD` | `1.0` | Fraction of watches that must fail for the tick to be reported as failed (`1.0` = all of them; `0` disables the check). |
| `TZ` | `Asia/Kuala_Lumpur` | Container clock (mostly cosmetic; timestamps are written in UTC deliberately for NocoDB). |
`${VAR:-default}` syntax: compose substitutes the value from the environment /
`.env` file, falling back to the literal default when unset. `${NOCODB_TOKEN}`
with **no** default means it's mandatory — compose errors if missing.
### volumes
```yaml
volumes:
- carousell-data:/data
```
Named volume `carousell-data` mounted at `/data`. Inside the container that's:
- `monitor.py` → `DATA_DIR` default → `/data/health.json` (written each tick, read by the healthcheck)
- Nothing else is stored there — NocoDB holds all real state.
Why a volume and not a bind mount: survives container recreation, no host-path
permission issues on DSM ACLs, and it's private to the stack (not exposed to the
host filesystem).
### networks
```yaml
networks:
- bridge_hoelee
networks:
bridge_hoelee:
external: true
```
`external: true` = the network is **pre-existing** (created by the NocoDB stack,
typically). Compose does not create it, just joins it. This is what lets the
monitor reach `http://nocodb:10380` by container name instead of an IP that
drifts (DSM IP history: 192.168.137.2 → 192.168.1.1 → …).
---
## 3. Dockerfile
```dockerfile
FROM python:3.11-alpine
WORKDIR /app
COPY monitor.py healthcheck.py /app/
RUN mkdir -p /data
VOLUME ["/data"]
HEALTHCHECK --interval=60s --timeout=15s --start-period=120s --retries=3 \
CMD python /app/healthcheck.py
CMD ["python", "-u", "/app/monitor.py"]
```
- `python:3.11-alpine` — tiny, stdlib-only code needs no pip deps → fast builds, small image.
- `COPY` bakes the script into the image → a rebuild is how code ships (there is no bind-mount of the repo).
- `HEALTHCHECK` runs `healthcheck.py` every 60s: exits 0 iff `/data/health.json` exists, is newer than `HEALTH_STALE_SECONDS`, and `ok == true`.
- `python -u` — unbuffered stdout so `docker logs` shows ticks in real time.
### healthcheck.py logic
```python
age = time.time() - int(h.get("last_run_epoch", 0))
if age <= STALE and h.get("ok") is True:
sys.exit(0) # healthy
sys.exit(1) # unhealthy
```
One failed tick (Carousell 403/429/parse error, NocoDB down) → `ok:false` →
container turns **unhealthy** until a fully successful tick. That's the tripwire:
Portainer shows it red, `docker inspect` reports it, and you can alert on it.
---
## 4. Deployment paths
### A. On DSM via Portainer stack (current production)
Container is owned by **Portainer stack 240** (standalone). Deploy:
1. Edit code in the repo (`D:\dev\carousell-monitor`), commit, push to Gitea
(`git.hoelee.com/hoelee/carousell-monitor`).
2. Sync the build context — Portainer's project dir holds a full manual copy
(not a git clone): copy changed runtime files to
`/volume1/docker/portainer/compose/240/` (`monitor.py`, `healthcheck.py`,
`Dockerfile`, `docker-compose.yml`).
3. If `monitor.py` / `Dockerfile` changed, rebuild the image — a standalone
stack PUT does **not** run `--build`:
```bash
sudo /usr/local/bin/docker build -t carousell-monitor:latest \
/volume1/docker/portainer/compose/240
```
4. Update the stack via the Portainer API:
`PUT /api/stacks/240?endpointId=2` with the repo `docker-compose.yml` as
`stackFileContent` and the **current env array** (8 entries, incl.
`FETCH_GAP_SECONDS=1`; see the `portainer-api` skill — ⚠ Portainer masks
secret env values, so never echo the masked `***` strings back, and keep the
real Telegram token in `SECRETS.md`). Portainer recreates the container.
The old `/volume1/docker/carousell-monitor` dir is **legacy — do not
`compose up` there anymore**; it only kept around for reference (its `.env` is
masked and stale).
### B. Local dev (Windows, against LAN NocoDB)
```bash
NOCODB_URL=http://192.168.1.1:10380 \
NOCODB_TOKEN=... TELEGRAM_BOT_TOKEN=... TELEGRAM_CHAT_ID=... \
python monitor.py
```
Runs the same loop outside Docker — useful for testing code changes before
shipping (uses LAN IP instead of container DNS).
⚠ `192.168.1.1` drifts; check `:10380` on the current DSM IP first.
---
## 5. What to check when something breaks
| Symptom | Check |
|---|---|
| Container `unhealthy` | `sudo docker logs carousell-monitor --tail 100` — read the `error` line |
|"No application/json state" | Carousell rate-limited/blocked; raise `check_interval_minutes` in Settings |
| No Telegram | `TELEGRAM_BOT_TOKEN`/`TELEGRAM_CHAT_ID` correct? `notify` box on the watch? Telegram IP pin in `extra_hosts` still valid? |
| "NOCODB_TOKEN not set" | Stack env missing the token; fix in Portainer stack env, redeploy |
| NocoDB unreachable | Is `nocodb` container on `bridge_hoelee`? `sudo docker network inspect bridge_hoelee` |
| Timestamps 8h off | They are **UTC by design**; NocoDB display TZ → Asia/Kuala_Lumpur |
---
## 6. Files in the repo
| File | Role |
|---|---|
| `docker-compose.yml` | Stack definition (this doc) |
| `Dockerfile` | Image build + healthcheck |
| `monitor.py` | The whole monitor (stdlib only) |
| `healthcheck.py` | Docker HEALTHCHECK probe |
| `.env.example` | Template for local runs (secrets go in `.env`, gitignored) |
| `SECRETS.md` | Credential inventory (private repo) |
| `DOCUMENTATION.md` | Deployment & operations manual |
| `AGENTS.md` | AI-agent entry point |
| `COMPOSE-SETUP.md` | This file |
---
*Last reviewed: 2026-09-13 (NocoDB 2026.08.1, Docker Compose v2.20.1 on DSM).*
DOCUMENTATION.md
-252
View File
@@ -1,252 +0,0 @@
# carousell-monitor — Deployment & Operations Documentation
Carousell new-listing monitor: watches Carousell search pages (sorted by *recent*),
archives every listing into a NocoDB base (with image thumbnail), and alerts Telegram
when a genuinely new listing appears. Runs 24/7 as a Docker container on the DSM.
---
## 1. Architecture
```
Docker container (carousell-monitor, DSM, network bridge_hoelee)
├─ every 60 s (TICK_SECONDS) ─ reads watch list from NocoDB Settings table
├─ per watch (on its own check_interval_minutes) ─ GET the Carousell search URL
│ ├─ parse embedded JSON state → SearchListing.listingCards[]
│ ├─ dedupe by product_url (param-less listing URL)
│ ├─ INSERT new rows into NocoDB Listings table
│ └─ archive w/ notified flag (first seed = silent; else pending-notify)
├─ end of each tick ─ send ALL pending (notified=false) listings, 1s apart
│ └─ each notice: photo + title/price/condition/seller/url
├─ writes /data/health.json each tick → Docker HEALTHCHECK
└─ reaches: NocoDB http://nocodb:10380 (container DNS, bridge_hoelee)
Carousell www.carousell.com.my (public internet)
Telegram api.telegram.org (public internet)
```
Nothing is pushed to any image registry — the image is built **privately on DSM**
(`build: .`). Secrets come from the `.env` file next to the compose file.
---
## 2. Files in this folder
| File | Purpose |
|---|---|
| `docker-compose.yml` | Stack definition (build, env, volume, network) |
| `Dockerfile` | python:3.11-alpine + monitor.py + healthcheck.py, HEALTHCHECK |
| `monitor.py` | Main loop: schema bootstrap, fetch/parse, NocoDB IO, Telegram |
| `healthcheck.py` | HEALTHCHECK probe (reads `/data/health.json`) |
| `.env` | No longer used — secrets/tunables live in the **Portainer stack env** (stack 240); the legacy DSM-dir `.env` is masked and stale |
| `DOCUMENTATION.md` | Deployment & operations manual |
| `COMPOSE-SETUP.md` | Stack anatomy reference: compose file, Dockerfile, networks, deployment paths |
Source of truth for the code is the **private Gitea repo**
`git.hoelee.com/hoelee/carousell-monitor` (local checkout `D:\dev\carousell-monitor`).
Credentials are documented in `SECRETS.md` there.
---
## 3. Environment variables (`.env`)
| Var | Value / default | Notes |
|---|---|---|
| `NOCODB_URL` | `http://nocodb:10380` | container DNS on `bridge_hoelee`; LAN form `http://192.168.137.2:10380` |
| `NOCODB_TOKEN` | *(secret)* | workspace-scoped NocoDB PAT |
| `NOCODB_BASE_ID` | `poqw1zjw3hnsk37` | base "Carousell" |
| `TELEGRAM_BOT_TOKEN` | *(secret)* | @carousellFoundBot |
| `TELEGRAM_CHAT_ID` | `5648309582` | alert destination |
| `TICK_SECONDS` | `60` | scheduler granularity |
| `FETCH_GAP_SECONDS` | `1` | min pause (s) between watch URL fetches within one tick — anti-burst |
| `HEALTH_STALE_SECONDS` | `600` | healthcheck staleness window |
| `ERROR_ALERT_AFTER` | `3` | consecutive failed ticks before a Telegram failure alert is sent (debounce); a recovery notice is sent when it clears |
| `FAILURE_RATIO_THRESHOLD` | `1.0` | fraction of watches that must fail for the tick to count as failed (`1.0` = all). Shields the healthcheck and alerts from one flaky watch being soft-blocked. |
**Secrets = env vars (`.env`). Operational knobs = NocoDB Settings table.**
Speed, enable/disable, and notify on/off are all changed from the NocoDB UI — no
redeploy, no `.env` edit.
---
## 4. NocoDB schema (base `Carousell`, id `poqw1zjw3hnsk37`)
The container bootstraps both tables on startup if they are missing (idempotent —
deleting a table is safe; it is recreated on the next start).
### `Listings` (archive — every seen listing)
| Column | Type | Purpose |
|---|---|---|
| `Id` | auto | primary key |
| `product_url` | URL | **unique dedupe key** — `https://www.carousell.com.my/p/<id>/`, no query params |
| `title` | SingleLineText | listing title |
| `price` | Decimal | numeric price, "RM" stripped (e.g. `85.00`) — filterable/sortable |
| `condition` | SingleSelect | Brand new / Like new / Lightly used / Well used / Heavily used / Used (归一化自 Carousell 的 New/Used 等写法) |
| `image_url` | URL | 高清图 URL（已去 `_progressive_thumbnail` 后缀） |
| `image` | Attachment | 高清图 — NocoDB hotlinks the URL; renders in grid view |
| `seller_name` | SingleLineText | seller username |
| `seller_url` | URL | `https://www.carousell.com.my/u/<username>/` |
| `search_title` | SingleLineText | which watch found it (denormalized) |
| `search_url` | URL | the watch's search URL |
| `listed_at` | DateTime (UTC) | 上架时间（优先 `time_created`，被顶置商品 fallback `active_bump`） |
| `first_seen_at` | DateTime (UTC) | when the monitor first captured it |
| `notified` | Checkbox | false = 待发通知；发完/静默归档后置 true（防重复通知） |
### `IgnoredSellers` (global seller blocklist)
| Column | Type | Purpose |
|---|---|---|
| `seller_name` | SingleLineText | seller username to silence globally (any watch) |
### `IgnoredKeywords` (per-watch title blocklist)
| Column | Type | Purpose |
|---|---|---|
| `watch` | Link → `Settings` | **pick the watch from a dropdown** (belongs-to: many keywords → one watch) — no URL to copy by hand |
| `keyword` | SingleLineText | if the listing **title** contains this (case-insensitive substring), skip the Telegram alert (still archived) |
One keyword per row; add multiple rows for multiple keywords. A keyword only
silences listings found by the watch you linked — the same keyword never applies to
other watches. Rows with no watch or no keyword are ignored.
`watch` is a real NocoDB Link column: the monitor resolves `Settings.url` through it
at load time (one fetch of the Settings table per cycle). There is no hand-copied
URL that can silently drift out of sync. (The monitor's bootstrap also builds the
`watch` column itself and drops any legacy `search_url` column.)
### `Settings` (the watch list — you manage this)
| Column | Type | Purpose |
|---|---|---|
| `Id` | auto | primary key |
| `title` | SingleLineText | human label, e.g. "Uniform" |
| `url` | URL | full Carousell search URL (with `sort_by=3`) |
| `enabled` | Checkbox | false = paused (skipped entirely) |
| `notify` | Checkbox | false = archive only, no Telegram ping |
| `check_interval_minutes` | Number | per-watch poll interval (default 5) |
| `last_checked_at` | DateTime | null = not yet seeded (first run archives silently) |
---
## 5. Runtime behaviour
- **Tick loop**: every `TICK_SECONDS` the container reloads the Settings table and
polls each *enabled* watch whose interval has elapsed.
- **Dedupe**: `product_url` is the identity. On startup the container loads every
existing `product_url` from NocoDB into an in-memory set; a listing is "new" only
if its URL is not in that set.
- **First run per watch** (`last_checked_at` is null): seeds the current listings as an
archive with `notified=true` (silent, no Telegram). Prevents a flood on setup.
- **Afterwards**: new listings are inserted with `notified=false` (pending). At the end
of each tick the monitor sends every `notified=false` listing **one message each**
(photo + title/price/condition/seller/url), 1 second apart, then sets `notified=true`.
- **Silencing**: a pending listing is marked `skip_notify=true` + `notified=true`
(no Telegram) if its `seller_name` is in `IgnoredSellers` (global), or if its
**title** contains any keyword whose `IgnoredKeywords.watch` links to the
same `Settings` row (resolved via `Settings.url` per cycle, per-watch,
case-insensitive).
- **Failure handling**: a fetch/parse error on any watch marks that tick failed; the
container becomes **unhealthy** until the next fully-successful tick. `last_checked_at`
is only advanced on success, and a hard-failing watch is rate-limited to one attempt
per tick so it doesn't hammer the site.
---
## 6. Operations (all from the NocoDB UI — no SSH, no redeploy)
| Want to… | Do this |
|---|---|
| Add a keyword | Add a row to `Settings`: `title`, full `url` (open Carousell → search → sort "Recent" → copy URL), `enabled` ✓, `notify` ✓, interval |
| Remove / pause a watch | Set `enabled` = false (or delete the row) |
| Stop Telegram pings but keep archiving | Set `notify` = false |
| Ignore a seller everywhere | Add `seller_name` row in `IgnoredSellers` |
| Ignore certain keywords for **one watch** | Add row(s) in `IgnoredKeywords`: pick the watch in the `watch` dropdown, `keyword` = e.g. `nike` (case-insensitive, matches inside the title). `search_url` fills itself |
| Change how often it checks | Edit `check_interval_minutes` (5 = every 5 min) |
| See what's new | Open `Listings`, sort by `first_seen_at` desc |
| Browse with images | `Listings` grid view — the `image` column renders thumbnails |
The URL for a watch must keep `sort_by=3` (recent) so new listings sort first, e.g.:
```
https://www.carousell.com.my/search/uniform?addRecent=true&canChangeKeyword=true&includeSuggestions=true&sort_by=3&t-search_query_source=direct_search
```
---
## 7. Updating the code
The container is owned by **Portainer stack 240**. To ship a code change:
```bash
# 1) edit code in the repo (D:\dev\carousell-monitor), commit, push to Gitea
# 2) sync runtime files to Portainer's build context:
sudo cp monitor.py healthcheck.py Dockerfile docker-compose.yml \
/volume1/docker/portainer/compose/240/
# 3) if monitor.py / Dockerfile changed, rebuild the image (PUT doesn't --build):
sudo /usr/local/bin/docker build -t carousell-monitor:latest \
/volume1/docker/portainer/compose/240
# 4) update stack 240 via Portainer API (PUT /api/stacks/240?endpointId=2,
# repo compose as stackFileContent + current env array — see portainer-api skill;
# ⚠ never echo masked *** values back, real Telegram token is in SECRETS.md)
```
Portainer recreates the container; the NocoDB schema bootstrap and dedupe
seeding are idempotent, so a redeploy never duplicates rows.
---
## 8. Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Container shows **unhealthy** | last tick failed (403/429/parse) or >10 min stale | `sudo /usr/local/bin/docker logs carousell-monitor --tail 50` — read the `error` |
| "no application/json state found" | Carousell rate-limited / blocked the request | wait; consider raising `check_interval_minutes` |
| No Telegram messages | token/chat wrong, or `notify` off | verify `.env` `TELEGRAM_BOT_TOKEN`; check `notify` checkbox |
| Thumbnails missing in NocoDB | `image` column isn't Attachment, or Carousell blocked hotlink | confirm `image` is Attachment type; `image_url` (URL) always works as a link |
| NocoDB unreachable | DSM IP drifted, or not on `bridge_hoelee` | from DSM: `sudo /usr/local/bin/docker network inspect bridge_hoelee`; NocoDB container must be named `nocodb` |
| `listed_at`/`first_seen_at` look 8 h off | they are **UTC** (NocoDB parses naive datetimes as UTC) | expected — set your NocoDB display timezone to MYT (Asia/Kuala_Lumpur) if you want local times |
### Handy commands (run on DSM)
```bash
D="sudo /usr/local/bin/docker"
$D ps --filter name=carousell-monitor # status + health
$D logs carousell-monitor --tail 100 # last log lines
$D inspect carousell-monitor --format '{{.State.Health.Status}}' # health
$D restart carousell-monitor # restart (state is in NocoDB, safe)
```
Health file lives at `/data/health.json` inside the container:
`{"last_run_epoch": ..., "ok": true, "error": "", "watch_count": N, "new_this_tick": N}`.
---
## 9. Key identifiers
| Item | Value |
|---|---|
| Repo (private) | `git.hoelee.com/hoelee/carousell-monitor` |
| Local checkout | `D:\dev\carousell-monitor` |
| Portainer stack | `240` (standalone; compose + build context at `/volume1/docker/portainer/compose/240/` on DSM) |
| Container | `carousell-monitor` (network `bridge_hoelee`) |
| NocoDB base | `Carousell` = `poqw1zjw3hnsk37` (workspace `wal4hatt`) |
| Tables | `Listings` + `Settings` (bootstrap finds by title) |
| Telegram | `@carousellFoundBot` → chat `5648309582` (@MrFullStackDev) |
---
## 10. Security notes
- `.env` contains secrets and is **gitignored** — it exists only on this DSM and is
never committed. Credential inventory: `SECRETS.md` in the repo.
- The image is private (built on DSM, never pushed to Docker Hub).
- NocoDB token is workspace-scoped; regenerate in NocoDB if it leaks and update `.env`.
---
## 11. Changelog
- **2026-09-13** `IgnoredKeywords.watch` = 真正的 NocoDB **Link 列**（Many-to-One → `Settings`），UI 下拉选 watch，不用手抄 URL。`search_url` 列已删除——运行时经 watch 链接 + Settings 表解析出 `.url`（每周期一次拉取）。bootstrap 自建 `watch` 列并清理遗留 `search_url`。
- **2026-09-13** 新增 per-watch 忽略关键词：`IgnoredKeywords` 表。标题命中该 watch
关键词（大小写不敏感子串）时静默归档、不发 Telegram（`skip_notify=true`）。
- **2026-09-08** 通知重构：每商品一条图文消息（title/price/condition/seller/url），归档与通知解耦（`notified` 列 + tick 末尾统一发 + 1s 间隔）。图片改用高清 URL（去 `_progressive_thumbnail`）。condition 归一化（New→Brand new、Used→Used，加第 6 档）。listed_at 加 `active_bump` fallback。修复 Telegram IPv6/DNS 问题（compose `extra_hosts` 钉 IPv4）。bot 换 `@carousellFoundBot`。
LICENSE
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Lee Teong Hoe (Hoelee Enterprise)
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
README.md
+379 -59
View File
@@ -1,73 +1,393 @@
# carousell-monitor
Watches Carousell search pages (sorted by *recent*) for new listings, archives every
listing to a NocoDB base (with image URL + thumbnail), and alerts Telegram.
**Watch Carousell searches, archive every listing into NocoDB, and get a Telegram alert the moment something new appears.**
Self-hosted and free: one docker compose file, no scraping API, no paid service, no account anywhere except your own bot. Built for people who are tired of refreshing search pages all day — sneaker drops, camera gear, used furniture, car parts, uniform lots, whatever you are hunting for.
`Python 3.11 · stdlib only` · `Docker / Portainer` · `NocoDB` · `Telegram` · `MIT`
---
## What it does
- **Watches** any Carousell search URL you give it (sorted by *Recent*), on its own interval.
- **Archives** every listing it sees into NocoDB — title, numeric price, condition, seller, link, and the product photo (grid view shows thumbnails). Deduplicated by product URL, so nothing is stored twice.
- **Alerts** you on Telegram, one photo message per new listing, with the full details and a clickable link.
- **Stays quiet about the past.** The first time it sees a search it archives everything silently, so you are not flooded with 200 messages on setup. Only listings that appear *after* that first pass are alerted.
- **Filters** — mute a seller everywhere, or mute keywords for one specific search (still archived, no alert).
- **Tells you when it breaks.** If the container can no longer fetch, or the loop dies, you get a Telegram failure alert (after N consecutive failures, debounced) and a recovery notice when it works again.
## How it works
- `monitor.py` runs in a Docker container on DSM, self-bootstrapping its NocoDB
schema (`Listings` + `Settings` + `IgnoredSellers` + `IgnoredKeywords` tables)
and looping forever.
- Every `TICK_SECONDS` it reads the watch list from the **Settings** table and polls
each enabled watch's URL on its own `check_interval_minutes`.
- Dedupe key = `product_url` (param-less listing URL). First run per watch = seed
archive only (no Telegram). After that, new listings are archived and alerted as
`"<title>: N new listings"`.
- The container marks itself **unhealthy** (Docker healthcheck) if a tick fails to
extract / gets rate-limited / crashes.
## Schema
**Listings** — `product_url` (unique), `title`, `price` (numeric), `condition`
(SingleSelect), `image_url`, `image` (Attachment → thumbnail), `seller_name`,
`seller_url`, `search_title`, `search_url`, `listed_at`, `first_seen_at`,
`notified`, `skip_notify`.
**Settings** — `title`, `url`, `enabled`, `notify`, `check_interval_minutes`,
`last_checked_at`. Add/remove watches here from the NocoDB UI; no redeploy needed.
**IgnoredSellers** — `seller_name`. Add/remove sellers here to suppress Telegram
alerts for their listings (still archived, marked `skip_notify=true`).
**IgnoredKeywords** — `watch` (Link → Settings) + `keyword`. Per-watch title
blocklist: pick the watch from a dropdown, add one keyword per row. A keyword only
applies to listings from the linked watch; case-insensitive substring match against
the title. Still archived.
## Run
```bash
# local (against LAN NocoDB)
NOCODB_URL=http://192.168.137.2:10380 \
NOCODB_TOKEN=... TELEGRAM_BOT_TOKEN=... TELEGRAM_CHAT_ID=... \
python monitor.py
# docker
docker build -t hoelee/carousell-monitor:latest .
docker compose up -d
```
┌──────────────────────── your machine / NAS / VPS ────────────────────────┐
│ │
│ docker network: carousell (private) │
│ │
│ ┌───────────────────────────┐ ┌────────────────────────────┐ │
│ │ carousell-monitor │ NocoDB │ nocodb │ │
│ │ (python, no open ports) │◄────────►│ (web UI + SQLite, :8080) │ │
│ └───────┬───────────┬───────┘ REST └────────────────────────────┘ │
│ │ │ │
└───────────┼───────────┼──────────────────────────────────────────────────┘
│ │
every tick │ │ new listing ──► Telegram alert (photo + details)
▼ ▼
Carousell search pages api.telegram.org
```
## Deploy (DSM via Portainer stack 240)
The monitor is an **outbound-only worker**: no inbound port, no web UI, nothing to expose to the internet. All state lives in NocoDB; you drive it from the NocoDB UI.
Private build — no registry. The container is owned by **Portainer stack 240**
(standalone; compose + build context at `/volume1/docker/portainer/compose/240/`
on DSM), deployed with secrets passed as stack environment variables.
Each tick it:
1. reads your watch list from the NocoDB `Settings` table,
2. fetches each search page that is due and extracts the embedded listing JSON,
3. inserts anything it has not seen before into `Listings`,
4. sends one Telegram message per still-unnotified listing and marks it notified.
More detail: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md).
---
## Requirements
| | |
|---|---|
| **Docker** | Engine 20.10+ with Compose v2 (`docker compose version`) — or Portainer, if you prefer a GUI |
| **Telegram** | a free account (you create the bot in 60 seconds, see below) |
| **Machine** | anything that stays on: NAS, VPS, Raspberry Pi, mini-PC, old laptop |
| **Skills** | being able to copy-paste commands and edit a text file |
Disk: a few hundred MB (NocoDB + the listings database, which grows slowly). RAM: NocoDB wants ~300 MB, the monitor ~40 MB.
---
## Quick start (CLI)
> Prefer clicking? Jump to [Deploy on Portainer](#deploy-on-portainer-gui) — same result, no shell needed.
### Step 1 — get the code and the env file
```bash
# on DSM — rebuild the image only when monitor.py / Dockerfile changed
sudo /usr/local/bin/docker build -t carousell-monitor:latest \
/volume1/docker/portainer/compose/240
git clone https://github.com/hoelee/carousell-monitor.git
cd carousell-monitor
cp .env.example .env
```
Then update stack 240 via the Portainer API (`PUT /api/stacks/240?endpointId=2`,
repo compose as `stackFileContent` + current env array — see `portainer-api`
skill; ⚠ never echo masked `***` env values back). The old
`/volume1/docker/carousell-monitor` dir is legacy — do not `compose up` there.
### Step 2 — create your Telegram bot
## Files
Open Telegram, talk to [@BotFather](https://t.me/BotFather), send `/newbot`, follow the prompts. You get a token that looks like `8123456789:AAF...`.
- `monitor.py` — main loop, schema bootstrap, fetch/parse, NocoDB IO, Telegram.
- `healthcheck.py` — Docker HEALTHCHECK probe (`/data/health.json`).
- `Dockerfile`, `docker-compose.yml`, `.env.example`.
- `COMPOSE-SETUP.md` — stack anatomy: compose file, Dockerfile, networking, deploy paths.
- `AGENTS.md` — AI-agent entry. `SECRETS.md` — credentials (private repo).
Now message [@userinfobot](https://t.me/userinfobot) and it replies with your numeric id.
Put both into `.env`:
```ini
TELEGRAM_BOT_TOKEN=8123456789:AAF...
TELEGRAM_CHAT_ID=123456789
```
Full walkthrough, including how to alert a **group** instead of yourself: [`docs/TELEGRAM-SETUP.md`](docs/TELEGRAM-SETUP.md).
### Step 3 — start NocoDB and create the base
```bash
docker compose -f docker-compose.allinone.yml up -d nocodb
```
Wait ~30 seconds, then open **http://localhost:8080** (replace `localhost` with your server's address if you are deploying remotely).
1. Create your account (the first account is the admin — use a real email and a real password).
2. Create a **Base**, name it e.g. `Carousell`.
3. Copy the **base id** out of the browser URL — the long id after `/nc/base/`:
```
http://localhost:8080/dashboard/#/nc/base/poqw1zjw3hnsk37/...
^^^^^^^^^^^^^^^ copy this
```
4. Create an **API token**: click your avatar (bottom-left) → *Account Settings* → *Tokens* → *Create token*. Copy it (it starts with `nc_pat_`). NocoDB only shows it once.
Put both into `.env`:
```ini
NOCODB_BASE_ID=poqw1zjw3hnsk37
NOCODB_TOKEN=nc_pat_...
```
While you are there, generate a session secret and put it in `.env` too:
```bash
openssl rand -hex 32 # paste the output into NC_AUTH_JWT_SECRET
```
> The tables themselves are created **by the monitor**, so right now your fresh base is empty and that is expected.
### Step 4 — start the monitor
```bash
docker compose -f docker-compose.allinone.yml up -d
docker compose -f docker-compose.allinone.yml logs --tail 20 carousell-monitor
```
First run builds the image (a minute or so). A healthy start prints one line:
```
ready: listings=... settings=... ignored_sellers=... ignored_keywords=... seen=0
```
It is quiet after that — there is no per-tick logging by design. Progress is visible in NocoDB instead: four tables appear (`Listings`, `Settings`, `IgnoredSellers`, `IgnoredKeywords`), and the container reports `healthy`.
### Step 5 — tell it what to watch
1. Open your NocoDB base → the **`Settings`** table.
2. Add a row:
| title | url | enabled | notify | check_interval_minutes |
|---|---|---|---|---|
| Uniform | *(paste a Carousell search URL)* | ✓ | ✓ | 5 |
3. To get a good URL: search on [carousell.com.my](https://www.carousell.com.my), set the sort to **Recent**, then copy the address bar. Make sure it contains `sort_by=3` — that is what puts the newest listings first:
```
https://www.carousell.com.my/search/uniform?sort_by=3&...
```
That is it. The monitor re-reads `Settings` every tick, so no restart is needed after adding, editing or pausing a search.
### Step 6 — prove the alert works
Open the **`Listings`** table: your first pass has already archived the current listings (silently — no messages, that is correct). Then:
- **Send a test alert**: untick `notified` on any row and save. Within a tick the monitor re-sends that listing to Telegram. That is also the fastest way to debug a silent bot.
- **Watch a real one arrive**: the next genuinely new listing appears in `Listings` and lands in Telegram by itself.
Nothing in Telegram? See [Troubleshooting](#troubleshooting).
---
## Deploy on Portainer (GUI)
Portainer runs the exact same file — it is a normal compose stack. Everything happens in the browser.
1. **Stacks → Add stack**.
2. **Name**: `carousell-monitor`.
3. **Build method**: *Web editor* (or upload `docker-compose.allinone.yml` from this repo).
4. Paste the whole content of [`docker-compose.allinone.yml`](docker-compose.allinone.yml).
5. **Environment variables** — fill in the ones the file references (Portainer lists them for you):
| Variable | Value |
|---|---|
| `NC_AUTH_JWT_SECRET` | output of `openssl rand -hex 32` |
| `NOCODB_TOKEN` | the `nc_pat_…` token |
| `NOCODB_BASE_ID` | the id copied from the base URL |
| `TELEGRAM_BOT_TOKEN` | from @BotFather |
| `TELEGRAM_CHAT_ID` | from @userinfobot |
| `NOCODB_PORT` | optional, default `8080` |
> ⚠️ Portainer **masks secret-looking values you type into this panel** and saves the mask (`***`) with the stack. The credentials then silently stop working on the next stack update. If you plan to edit this stack in Portainer again, put the real values directly in the YAML in the web editor instead of in the env panel.
6. **Deploy the stack.** For a first install you want NocoDB first: after the deploy finishes, open `http://<your-host>:8080`, create the account and the base ([Step 3](#step-3--start-nocodb-and-create-the-base)), then paste the base id + token into the same field(s) and press **Update the stack**.
7. Verify in Portainer: the container list shows `carousell-nocodb` and `carousell-monitor` both **running/healthy**, and the monitor's logs show the `ready:` line.
Managing it afterwards is the same screen: **Stacks → carousell-monitor → Update the stack** (edit YAML/env), **containers → logs/restart**, **volumes** for backups.
Prefer the repository build? *Add stack → Repository* with `https://github.com/hoelee/carousell-monitor` and compose path `docker-compose.allinone.yml`. Note that Portainer's repository stacks do a full `git clone` on every deploy, so the web-editor or upload route is faster for a single file.
---
## Everyday use (all from the NocoDB UI — no SSH, no restart)
Open your base and work in the tables:
| I want to… | Do this |
|---|---|
| Watch another search | Add a row to `Settings`: `title`, full `url` (with `sort_by=3`), `enabled` ✓, `notify` ✓, interval |
| Pause a search | Untick `enabled` (or delete the row) |
| Keep archiving but stop Telegram for a search | Untick `notify` |
| Check more or less often | Edit `check_interval_minutes` (5 = every 5 minutes) |
| Mute a seller everywhere | Add their username to `IgnoredSellers` |
| Mute keywords for **one** search | Add rows to `IgnoredKeywords`: pick the search in the `watch` dropdown, type the `keyword` (e.g. `nike`) |
| See what is new | `Listings`, sorted by `first_seen_at` (newest first) |
| Browse with pictures | `Listings` → grid view; the `image` column renders thumbnails |
| Hide an archived row from the alert queue | tick `notified` on it (pending rows are `notified = false`) |
Keyword matching is a case-insensitive **substring of the title**, and it only applies to the search you linked it to. Ignored sellers and ignored keywords are still archived — they just do not ring your phone.
### Filter fields at a glance
| Table | What it is | Fields you create |
|---|---|---|
| `Settings` | your watch list | `title`, `url`, `enabled`, `notify`, `check_interval_minutes` (the monitor maintains `last_checked_at`) |
| `IgnoredSellers` | global seller blocklist | `seller_name` |
| `IgnoredKeywords` | per-search title blocklist | `watch` (link → `Settings`), `keyword` |
| `Listings` | the archive — written by the monitor | read-only for you, except `notified` |
Full column reference: [`docs/NOCODB-SETUP.md`](docs/NOCODB-SETUP.md).
### What the alert looks like
```
🛒 Seiko 5 SNK809 automatic watch
💰 RM320
📦 Like new
👤 watchguy88
https://www.carousell.com.my/p/seiko-5-snk809-1234567890/
```
plus the listing photo above the text. The monitor downloads the image and uploads it to Telegram, so the alert still works if Carousell later blocks hotlinking.
---
## Configuration
Everything below goes in `.env` (CLI) or in the stack's environment (Portainer). Only the first five are required. The tuning knobs all have working defaults — ignore them until you have a reason.
| Variable | Default | What it does |
|---|---|---|
| `NOCODB_BASE_ID` | — | **required** — the NocoDB base holding the tables |
| `NOCODB_TOKEN` | — | **required** — NocoDB API token (`nc_pat_…`) |
| `TELEGRAM_BOT_TOKEN` | — | **required** — from @BotFather |
| `TELEGRAM_CHAT_ID` | — | **required** — your numeric id, or a group id (negative) |
| `NOCODB_URL` | `http://nocodb:8080` | where the monitor reaches NocoDB |
| `NC_AUTH_JWT_SECRET` | — | NocoDB session secret (all-in-one stack only); `openssl rand -hex 32` |
| `NOCODB_PORT` | `8080` | host port for the NocoDB web UI |
| `TICK_SECONDS` | `60` | how often the loop wakes up and re-reads the watch list |
| `FETCH_GAP_SECONDS` | `1` | minimum pause between two Carousell requests in one tick — keeps a burst of due searches from looking like an attack. `0` disables |
| `FAILURE_RATIO_THRESHOLD` | `1.0` | fraction of watches that must fail for the tick to count as failed. `1.0` = only if everything failed, so one soft-blocked search does not flip the container to unhealthy. `0` = never fail |
| `ERROR_ALERT_AFTER` | `3` | consecutive failed ticks before the Telegram failure alert (debounce); one recovery notice follows when it clears |
| `HEALTH_STALE_SECONDS` | `600` | a tick older than this marks the container unhealthy |
| `TZ` | `Asia/Kuala_Lumpur` | container clock; timestamps written to NocoDB are UTC on purpose |
Want a different schedule per search? That is `check_interval_minutes` in the `Settings` table, not an env var.
### Already running NocoDB? Use the monitor-only file
[`docker-compose.yml`](docker-compose.yml) deploys just the monitor and joins an **existing** Docker network (edit the network name and `NOCODB_URL` to match your NocoDB). That is the setup this project runs in production, next to a NocoDB used by other apps.
---
## Operating it
```bash
# status + health
docker compose -f docker-compose.allinone.yml ps
# logs (quiet unless something is wrong)
docker compose -f docker-compose.allinone.yml logs --tail 100 carousell-monitor
# restart (state lives in NocoDB — safe, nothing is lost)
docker compose -f docker-compose.allinone.yml restart carousell-monitor
# update to a newer revision of this repo
git pull && docker compose -f docker-compose.allinone.yml up -d --build
# stop everything (data stays in the volumes)
docker compose -f docker-compose.allinone.yml down
```
**Is it actually working?** The container writes `/data/health.json` every tick and the Docker HEALTHCHECK reads it:
```bash
docker exec carousell-monitor cat /data/health.json
# {"last_run_epoch": ..., "ok": true, "error": "", "watch_count": 3, "new_this_tick": 0, "failed_watches": 0}
```
`ok: true` and a recent `last_run_epoch` means the loop is alive. `ok: false` with an `error` means it is not fetching — read the error. Backups are the two volumes: `nocodb-data` (all your data) and `carousell-data` (the health file only).
Deeper: [`docs/OPERATIONS.md`](docs/OPERATIONS.md).
---
## Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Container is `unhealthy` | last tick failed (Carousell blocked/rate-limited it, or NocoDB was unreachable) | `docker compose logs --tail 50 carousell-monitor` and read the `error`, then `cat /data/health.json` |
| Log says `NOCODB_TOKEN not set` | `.env` empty or the stack never picked it up | fill it in, then `up -d` again (Portainer: **Update the stack**) |
| `add column … failed` / `create table … failed` on startup | wrong `NOCODB_BASE_ID`, token without access to that base, or NocoDB still starting | check the base id, re-create the token in *that* base's workspace, make sure NocoDB is healthy first |
| No Telegram messages at all | token/chat id wrong, `notify` unticked, or the known Docker/IPv6 hang | untick/retick `notified` on a row to force a test send; check `getMe` with `curl https://api.telegram.org/bot<TOKEN>/getMe`; if the send times out, keep the `api.telegram.org` pin in `extra_hosts` (the all-in-one file already has it) |
| No Telegram for *one* search | `notify` unticked on that row, or the listing's seller/keyword is ignored | check the `Settings`, `IgnoredSellers`, `IgnoredKeywords` tables |
| `no application/json state found (blocked/ratelimited?)` | Carousell served a challenge page instead of results | raise `check_interval_minutes`, keep `FETCH_GAP_SECONDS ≥ 1`, and do not watch dozens of searches at once |
| `listingCards null (soft-block/ratelimit?)` | same thing, softer: Carousell answered 200 with empty state | same as above; the tick is retried, this is not a crash |
| Thumbnails missing in NocoDB | `image` column is not an Attachment column (edited?) | the monitor recreates columns on start — restart the container |
| Timestamps look 8 hours off | they are **UTC by design** | set NocoDB's display timezone to your local zone; the stored values stay UTC |
| NocoDB web UI unreachable | port clash or the container is not up | change `NOCODB_PORT`, or `docker compose ps` / read the NocoDB logs |
Still stuck? [`docs/TROUBLESHOOTING.md`](docs/TROUBLESHOOTING.md) has the full decision tree, including how to tell "not fetching" from "notifying" failures.
---
## FAQ
**Does this scrape or hammer Carousell?**
It performs one plain HTTP GET per search per interval (default every 5 minutes), with a 1-second gap between searches inside a tick, and it de-duplicates everything. It reads the same public search page your browser reads. Be a good citizen: do not set 30-second intervals on 40 searches.
**Can I get alerts for my own listings to test?**
Yes — post something (or edit an existing listing's title/price: an edit sometimes re-surfaces it), or simply untick `notified` on a row to replay an alert.
**Can I watch a category page or a seller's page instead of a search?**
Any Carousell page that renders the same listing grid works. Search URLs are what is tested.
**Multiple people / multiple searches?**
Everything is one monitor loop, one NocoDB base, one Telegram chat id. Add as many `Settings` rows as you like. For a second Telegram destination, run a second stack with its own bot and base.
**Do I need a reverse proxy / HTTPS?**
No — the monitor has no inbound port. If you want the NocoDB UI reachable from outside your LAN, put it behind your reverse proxy of choice (Synology/nginx/Traefik/Caddy) and keep the token out of the URL.
**Does it work outside Malaysia?**
It is written against `carousell.com.my` (Malaysia); the `.my` endpoints and the `RM` price format are baked in. Other Carousell country sites use the same page structure — change the two URL templates in `monitor.py` (`PRODUCT_URL_TMPL`, `SELLER_URL_TMPL`) and the search URL you paste into `Settings`.
**How do I upgrade NocoDB?**
Change the image tag in `docker-compose.allinone.yml`, `docker compose -f docker-compose.allinone.yml up -d`, and let it migrate. Back up `nocodb-data` first. The monitor only supports the meta API of the versions pinned in that file.
**Tests?**
```bash
python test_pagination.py # stdlib only, no network; exit 0 = pass
```
---
## Project layout
```
monitor.py the whole monitor (stdlib only): bootstrap, fetch, NocoDB IO, Telegram
healthcheck.py Docker HEALTHCHECK probe (reads /data/health.json)
Dockerfile python:3.11-alpine, no dependencies, ~60 MB
docker-compose.allinone.yml NocoDB + monitor — start here
docker-compose.yml monitor only, joining an existing NocoDB on a shared network
.env.example every setting, explained
test_pagination.py stdlib regression tests (run: python test_pagination.py)
docs/ the long-form guides
AGENTS.md notes for AI coding agents working on this repo
SECRETS.example.md credential template (the real values stay out of git)
```
### Guides
| Doc | Read it when |
|---|---|
| [`docs/QUICKSTART-PORTAINER.md`](docs/QUICKSTART-PORTAINER.md) | you want the click-by-click Portainer version |
| [`docs/TELEGRAM-SETUP.md`](docs/TELEGRAM-SETUP.md) | you need the bot token or a group chat id |
| [`docs/NOCODB-SETUP.md`](docs/NOCODB-SETUP.md) | you want to understand the tables before you fill them |
| [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) | you want to modify the code, or understand the compose file line by line |
| [`docs/OPERATIONS.md`](docs/OPERATIONS.md) | something is wrong, or it is time to back up / upgrade |
| [`docs/TROUBLESHOOTING.md`](docs/TROUBLESHOOTING.md) | you have a symptom and want the decision tree |
---
## Contributing
Issues and pull requests are welcome. Two house rules before you send a patch:
1. **Open an issue first for anything beyond a typo** — this is a small, opinionated tool and the maintainer would rather agree on the shape before you write it.
2. **Do not break the tests.** `python test_pagination.py` must pass, and new behaviour wants a check added to it.
Please never commit credentials, host names or personal URLs. The repo intentionally ships no default base id, token or chat id.
## Credits
Written and maintained by **Lee Teong Hoe** ([Mr Hoelee](https://hoelee.com)) — Hoelee Enterprise, Malaysia.
Built on the shoulders of [NocoDB](https://nocodb.com) and the [Telegram Bot API](https://core.telegram.org/bots/api).
## License
[MIT](LICENSE) — do what you want, no warranty.
SECRETS.example.md
+24
View File
@@ -0,0 +1,24 @@
# SECRETS.example.md — credential template
Copy to `SECRETS.md` (gitignored) or keep the values in your own secret store.
**Never commit real credentials** — the tracked repo carries only this template.
| Var | Value / source |
|---|---|
| `NOCODB_URL` | `http://nocodb:8080` (all-in-one) or your existing NocoDB URL |
| `NOCODB_BASE_ID` | NocoDB → open the base → copy the id from the URL |
| `NOCODB_TOKEN` | NocoDB → Account Settings → Tokens → Create token (`nc_pat_…`) |
| `TELEGRAM_BOT_TOKEN` | @BotFather → `/newbot` → token |
| `TELEGRAM_CHAT_ID` | @userinfobot, or `getUpdates` for a group |
| `NC_AUTH_JWT_SECRET` | `openssl rand -hex 32` (all-in-one stack only) |
## Where the values are used
| Store | Used by | Committed? |
|---|---|---|
| `.env` (repo root) | `docker compose` local runs | no (gitignored) |
| Portainer stack environment / stack file | Portainer deploys | no (lives on the Portainer host) |
| `SECRETS.md` | human inventory | no (gitignored) |
If a token ever leaks, regenerate it: NocoDB → Account Settings → Tokens
(revoke + create), Telegram → @BotFather → `/revoke`.
docker-compose.allinone.yml
+109
View File
@@ -0,0 +1,109 @@
# =============================================================================
# carousell-monitor — ALL-IN-ONE stack: NocoDB + the monitor
#
# Use this file when you do NOT already run NocoDB.
# If you already have a NocoDB, use docker-compose.yml instead (monitor only).
#
# These two containers talk to each other over the private docker network
# `carousell`; only the NocoDB web UI is published to the host.
#
# CLI quick start
# ---------------
# 1. cp .env.example .env # then edit .env
# 2. docker compose -f docker-compose.allinone.yml up -d nocodb
# 3. open http://<host>:8080 -> create your account, create a base,
# then put the base id + an API token into .env
# 4. docker compose -f docker-compose.allinone.yml up -d
# 5. follow docs/QUICKSTART-PORTAINER.md (same steps, GUI edition)
#
# Portainer
# ---------
# Stacks -> Add stack -> Web editor -> paste this whole file -> fill in the
# environment variables -> Deploy the stack.
# (Recommended: replace the ${...} placeholders with real values in the
# editor. Portainer masks secret-looking values that you type into its
# "Environment variables" panel, which can break a later stack update.)
#
# Upgrading NocoDB: change the image tag below, then
# docker compose -f docker-compose.allinone.yml up -d
# NocoDB migrates its own schema on start. Back up the volume first.
# =============================================================================
services:
# --------------------------------------------------------------------------
# NocoDB — the archive database + web UI (tables, grid view, image previews).
# --------------------------------------------------------------------------
nocodb:
# Verified working with this monitor: 2026.08.x – 2026.09.0.
# NocoDB's meta API (used to bootstrap the tables) changes between majors,
# so upgrade this tag deliberately, not blindly to :latest.
image: nocodb/nocodb:2026.09.0
container_name: carousell-nocodb
restart: unless-stopped
environment:
# NocoDB listens on 8080 inside the container.
PORT: "8080"
# Signs login sessions. Generate with: openssl rand -hex 32
NC_AUTH_JWT_SECRET: ${NC_AUTH_JWT_SECRET}
# Self-hosted defaults: no telemetry, no local webhooks.
NC_DISABLE_TELE: "true"
NC_ALLOW_LOCAL_HOOKS: "false"
volumes:
# SQLite database + uploaded attachments live here. Back this up.
- nocodb-data:/usr/app/data
ports:
# Web UI: http://<host>:8080 (change the left side if 8080 is taken)
- "${NOCODB_PORT:-8080}:8080"
healthcheck:
test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:8080/api/v1/health >/dev/null 2>&1 || exit 1"]
interval: 30s
timeout: 10s
retries: 5
start_period: 60s
networks:
- carousell
# --------------------------------------------------------------------------
# carousell-monitor — polls Carousell, archives to NocoDB, alerts Telegram.
# No inbound port, no web UI: outbound calls only.
# --------------------------------------------------------------------------
carousell-monitor:
build: .
image: carousell-monitor:latest
container_name: carousell-monitor
restart: unless-stopped
depends_on:
nocodb:
condition: service_healthy
# Some docker networks have no IPv6 while api.telegram.org resolves to an
# IPv6 (AAAA) address first, so the send hangs and the alert is silently
# lost. Pinning the IPv4 record fixes it. Refresh the IP occasionally with
# dig +short api.telegram.org
# If your host has working IPv6, you can delete these two lines.
extra_hosts:
- "api.telegram.org:149.154.166.110"
environment:
# Service name of the NocoDB container on the `carousell` network.
NOCODB_URL: http://nocodb:8080
NOCODB_TOKEN: ${NOCODB_TOKEN}
NOCODB_BASE_ID: ${NOCODB_BASE_ID}
TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN}
TELEGRAM_CHAT_ID: ${TELEGRAM_CHAT_ID}
TICK_SECONDS: ${TICK_SECONDS:-60}
FETCH_GAP_SECONDS: ${FETCH_GAP_SECONDS:-1}
HEALTH_STALE_SECONDS: ${HEALTH_STALE_SECONDS:-600}
ERROR_ALERT_AFTER: ${ERROR_ALERT_AFTER:-3}
FAILURE_RATIO_THRESHOLD: ${FAILURE_RATIO_THRESHOLD:-1.0}
TZ: ${TZ:-Asia/Kuala_Lumpur}
volumes:
# /data/health.json — the Docker HEALTHCHECK reads it each tick.
- carousell-data:/data
networks:
- carousell
volumes:
nocodb-data:
carousell-data:
networks:
carousell:
docs/ARCHITECTURE.md
+141
View File
@@ -0,0 +1,141 @@
# Architecture
How the pieces fit together, and what every line of the compose file and Dockerfile is doing. Read this before changing the code.
---
## 1. Runtime
```
carousell-monitor container (python:3.11-alpine, no ports, no web UI)
│
├─ once at start ─ bootstrap(): create/fix the NocoDB tables, load every known
│ product_url into an in-memory seen-set
│
└─ every TICK_SECONDS (default 60s):
├─ read the Settings table (watch list)
├─ per watch whose check_interval_minutes has elapsed:
│ GET the Carousell search page
│ parse the embedded JSON state → SearchListing.listingCards[]
│ keep only URLs not in seen
│ INSERT them into Listings
│ first pass for a watch (last_checked_at empty) → notified = true (silent seed)
│ afterwards → notified = false (queued)
│ advance last_checked_at, then sleep FETCH_GAP_SECONDS
├─ send every queued listing (notified = false), one Telegram message each,
│ applying the ignore filters, then mark it notified
└─ write /data/health.json → the Docker HEALTHCHECK reads it
```
External calls: **Carousell** (search pages, and listing photos when alerting), **NocoDB** (REST on the private network), **Telegram** (Bot API). Nothing calls in.
## 2. The files
| File | Role |
|---|---|
| `monitor.py` | everything: config, HTTP, NocoDB schema bootstrap, extraction, IO, Telegram, health, alerting |
| `healthcheck.py` | Docker HEALTHCHECK probe — exits 0 only if `health.json` is fresh **and** `ok: true` |
| `Dockerfile` | `python:3.11-alpine`, copies the two scripts, declares the healthcheck |
| `docker-compose.allinone.yml` | NocoDB + monitor, private `carousell` network |
| `docker-compose.yml` | monitor only, joins an existing NocoDB network (production variant) |
| `test_pagination.py` | regression suite (stdlib, no network): NocoDB paging, the Telegram HTTP verb, failure thresholds |
| `.env.example` | the settings, documented |
## 3. Why it is built this way
**Two containers minimum.** NocoDB holds all state; the monitor is disposable. You can delete the monitor container, rebuild it, or change `monitor.py` and nothing is lost — the seen-set is rebuilt from `Listings` on start.
**Archive and notify are decoupled.** A listing is inserted with `notified = false`; a later pass sends it and flips the flag. So a Telegram outage or a wrong token never loses a listing — the queue just drains later. It also means "no alert" and "no data" are distinguishable failures.
**Filters are data, not config.** Ignored sellers and keywords live in NocoDB tables that are re-read every cycle, so muting something is a UI action, not a redeploy. Operational knobs (which searches, how often, whether to notify) are in `Settings`; only credentials and tuning live in the environment.
**First pass is silent.** `last_checked_at` empty means "never seeded": the current listings are archived with `notified = true`. Otherwise the first start would fire hundreds of messages.
**Dedupe on the canonical product URL**, `https://www.carousell.com.my/p/<id>/` — never the search URL and never the raw listing id alone. The search page's own links carry tracking parameters that change between loads.
**The failure threshold matters.** `FAILURE_RATIO_THRESHOLD` (default `1.0`) means a tick only counts as failed if *every* watch failed. Carousell soft-blocks individual searches from time to time; without this, one flaky search would mark the container unhealthy and fire a failure alert every tick.
**Telegram failures are alerted on their own edge.** `ERROR_ALERT_AFTER` consecutive failed ticks send one alert, and clearing the failure state sends one recovery notice — a state machine debounce, not a per-tick message.
## 4. The compose file, block by block
### `docker-compose.allinone.yml`
```yaml
services:
nocodb:
image: nocodb/nocodb:2026.09.0 # pinned on purpose; the monitor targets this meta API
volumes:
- nocodb-data:/usr/app/data # SQLite db + attachments
ports:
- "${NOCODB_PORT:-8080}:8080" # only the web UI is published
healthcheck:
test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:8080/api/v1/health >/dev/null 2>&1 || exit 1"]
carousell-monitor:
build: . # built locally from this repo — no registry involved
depends_on:
nocodb:
condition: service_healthy # do not start before the database answers
extra_hosts:
- "api.telegram.org:149.154.166.110"
volumes:
- carousell-data:/data # health.json, alert_state.json
networks: [carousell]
```
| Key | Why |
|---|---|
| `build: .` | the image is built from this directory; nothing is pulled from a registry |
| `depends_on: service_healthy` | the monitor bootstraps the schema at startup, so NocoDB must be up first |
| `extra_hosts` | pins `api.telegram.org` to its IPv4. On a Docker network without IPv6, the embedded DNS can hand back an AAAA record and the send hangs with no error — the alert is lost silently. Refresh the IP with `dig +short api.telegram.org` if sends start timing out; delete the two lines if your host has working IPv6 |
| named volumes | survive container recreation and `down`; no host-path/ACL problems |
| private network | only `nocodb` publishes a port, and the monitor resolves it as `http://nocodb:8080` — no IPs anywhere |
### `docker-compose.yml` (existing NocoDB)
The same monitor service, minus NocoDB, joining a network that already exists:
```yaml
networks:
- bridge_hoelee # ← your existing network name
networks:
bridge_hoelee:
external: true # created by the NocoDB stack, not by this one
```
Point `NOCODB_URL` at the existing container's name on that network (e.g. `http://nocodb:10380` if it listens on a non-default port).
## 5. The Dockerfile
```dockerfile
FROM python:3.11-alpine
WORKDIR /app
COPY monitor.py healthcheck.py /app/
RUN mkdir -p /data
VOLUME ["/data"]
HEALTHCHECK --interval=60s --timeout=15s --start-period=120s --retries=3 \
CMD python /app/healthcheck.py
CMD ["python", "-u", "/app/monitor.py"]
```
- **stdlib only** — no `pip install`, so the image is ~60 MB and builds in seconds.
- **Code is baked in** — shipping a change means rebuilding (`docker compose up -d --build`), not restarting.
- **`python -u`** — unbuffered output so logs appear immediately.
- **`start-period=120s`** — the first tick includes the schema bootstrap; give it room before Docker starts reporting health.
## 6. Extraction details (why it is fragile)
Carousell server-renders the search results into a `<script type="application/json">` blob; the monitor parses the largest one and reads `SearchListing.listingCards[]`. Each card carries `listingID`, `title`, `thumbnailURL`, `seller.username`, the condition as a paragraph, and timestamps under `aboveFold` (`time_created`, or `active_bump` for bumped listings).
Consequences worth knowing before you patch it:
- **HTTP 200 does not mean success.** Under soft rate limiting Carousell returns 200 with `listingCards: null`. That is a retryable condition, not an empty result set — the code raises a named error for it.
- **No `application/json` blob at all** means a challenge/blocked page. Same treatment.
- **Ad cards exist** (`listingID = 0`) and are skipped rather than archived.
- **Thumbnail URLs carry a `_progressive_thumbnail` suffix** that is stripped to get the full-size image.
## 7. Scaling and limits
One process polls everything. That is comfortable for tens of searches at 5-minute intervals. If you need more, raise `check_interval_minutes` rather than lowering `TICK_SECONDS`, and keep `FETCH_GAP_SECONDS ≥ 1` — the point of the gap is that your traffic never looks like a burst.
docs/NOCODB-SETUP.md
+101
View File
@@ -0,0 +1,101 @@
# NocoDB setup
The monitor stores everything in [NocoDB](https://nocodb.com) — an open-source Airtable alternative. It gives you the archive (with thumbnails), the watch-list UI, and the filter tables, and it is the only interface you need day to day.
The **base** and the **API token** are created by you; the **four tables** are created by the monitor on its first start.
---
## 1. First run of NocoDB
Start it (all-in-one stack):
```bash
docker compose -f docker-compose.allinone.yml up -d nocodb
```
Open `http://<host>:8080`, then:
1. **Create the admin account.** First click wins — pick a real password, there is no password reset without mail configured.
2. **Create a Base** (`+ New Base`), named e.g. `Carousell`. A base is a database; everything this project needs lives inside it.
3. Do not bother creating tables by hand. The monitor does that.
## 2. Collect the two values
**Base id** — open the base and read the browser URL:
```
http://localhost:8080/dashboard/#/nc/base/poqw1zjw3hnsk37/...
^^^^^^^^^^^^^^^ NOCODB_BASE_ID
```
**API token** — click your avatar (bottom-left) → **Account Settings** → **Tokens** → **Create token**.
- Give it a name (`carousell-monitor`), no expiry (or an expiry you will remember to renew).
- Copy the value immediately — it starts with `nc_pat_` and NocoDB will not show it again. That is `NOCODB_TOKEN`.
If your NocoDB is multi-workspace, create the token inside the workspace that owns the base.
## 3. The tables the monitor creates
Restart the monitor after filling in the token; `Listings`, `Settings`, `IgnoredSellers` and `IgnoredKeywords` appear in the base. Creating them is idempotent — restarting never duplicates a table or a row.
### `Settings` — your watch list (you edit this)
| Column | Type | Notes |
|---|---|---|
| `title` | text | label shown in alerts and used to link filters |
| `url` | URL | the full Carousell search URL, **must contain `sort_by=3`** |
| `enabled` | checkbox | untick to pause a search |
| `notify` | checkbox | untick to archive without alerting |
| `check_interval_minutes` | number | how often this search is polled (default 5) |
| `last_checked_at` | datetime | maintained by the monitor; empty = never seeded yet |
### `Listings` — the archive (written by the monitor)
| Column | Type | Notes |
|---|---|---|
| `product_url` | URL | dedupe key, `https://www.carousell.com.my/p/<id>/`, no query string |
| `title` | text | listing title |
| `price` | decimal | numeric, `RM` stripped (`85.00`) so you can sort and filter |
| `condition` | select | Brand new / Like new / Lightly used / Well used / Heavily used / Used |
| `image_url` | URL | the full-size photo URL |
| `image` | attachment | same photo as an attachment — renders as a thumbnail in grid view |
| `seller_name` / `seller_url` | text / URL | who is selling |
| `search_title` / `search_url` | text / URL | which watch found it |
| `listed_at` | datetime (UTC) | when the seller posted it |
| `first_seen_at` | datetime (UTC) | when the monitor first saw it |
| `notified` | checkbox | `false` = still queued for an alert; `true` = sent or deliberately silenced |
| `skip_notify` | checkbox | set when a filter matched (see below) |
### `IgnoredSellers` — mute a seller everywhere
One row per seller, column `seller_name` = the Carousell username. Their listings stay in the archive (`skip_notify = true`) but never reach Telegram.
### `IgnoredKeywords` — mute words for one search
| Column | Type | Notes |
|---|---|---|
| `watch` | link → `Settings` | **pick the search from the dropdown**; keywords only apply to it |
| `keyword` | text | case-insensitive substring match against the listing **title** |
Both filter tables are reloaded every tick, so edits take effect within a minute without a restart.
## 4. Getting a good search URL
1. Search on [carousell.com.my](https://www.carousell.com.my).
2. Set the sort to **Recent** (the newest listing must come first — the monitor only sees what the first page shows).
3. Copy the address bar into the `url` column. It should look like:
```
https://www.carousell.com.my/search/mechanical-keyboard?addRecent=true&canChangeKeyword=true&includeSuggestions=true&sort_by=3&t-search_query_source=direct_search
```
Tips: keep searches specific (a specific model, a narrow category). Each search returns roughly the first page of results; a very broad search whose first page turns over slowly will miss the churn below the fold.
## 5. Quality-of-life
- **Images**: switch `Listings` to **grid view** (`Fields` → include `image`) for a thumbnail wall of everything found.
- **Timezones**: stored timestamps are UTC on purpose. Change your display timezone in NocoDB's account settings if you want local times in the UI.
- **Filters/sorts**: `price` is numeric, so `price < 200 AND condition = 'Like new'` works.
- **Backups**: everything is in the `nocodb-data` volume. NocoDB also has its own export (base → `…` → Export), which is the friendlier thing to keep off-box.
docs/OPERATIONS.md
+124
View File
@@ -0,0 +1,124 @@
# Operations
Running it, watching it, backing it up, upgrading it.
---
## Health at a glance
The container writes `/data/health.json` at the end of every tick, and the Docker HEALTHCHECK (`healthcheck.py`) reads it every 60 s:
```bash
docker exec carousell-monitor cat /data/health.json
```
```json
{
"last_run_epoch": 1791275045,
"ok": true,
"error": "",
"watch_count": 3,
"new_this_tick": 0,
"failed_watches": 0
}
```
| Field | Meaning |
|---|---|
| `ok` | the last tick completed without exceeding the failure threshold |
| `error` | empty, or which watches failed and why (`[partial 1/3] …` when the ratio threshold absorbed it) |
| `last_run_epoch` | when the tick ended (Unix seconds, UTC) |
| `watch_count` | how many enabled watches were read from `Settings` |
| `new_this_tick` | listings fetched this tick that were not already known — **not** how many alerts were sent |
Two traps worth internalising:
- **`ok: true` only proves the fetch loop ran.** Notification failures are retried on the next tick rather than reported, so an archive that fills up happily can still be silent in Telegram. To check the notify path, untick `notified` on a row and watch it flip back to `true` (that flip only happens after Telegram answered 200).
- **`new_this_tick: 0` is not evidence of anything** — it counts fetched rows, not delivered messages.
The container's own health is the other half:
```bash
docker compose -f docker-compose.allinone.yml ps # State / Health column
docker inspect --format '{{.State.Health.Status}}' carousell-monitor
```
`unhealthy` = the last tick failed or is older than `HEALTH_STALE_SECONDS` (600 s).
## Logs
```bash
docker compose -f docker-compose.allinone.yml logs -f --tail 100 carousell-monitor
```
The loop is deliberately quiet: one `ready:` line at startup, then nothing unless something fails. Per-watch failures are logged to stderr, and hard failures appear in `health.json`. If you are debugging "why no alert", logs are the wrong place — use the decision tree in [TROUBLESHOOTING.md](TROUBLESHOOTING.md).
## Alerting on the monitor itself
`ERROR_ALERT_AFTER` (default 3) consecutive failed ticks trigger one Telegram message:
```
🚨 carousell-monitor 故障
连续失败 3 次
错误: Uniform: carousell fetch HTTP 403
容器将标记为 unhealthy
```
and one recovery message when the next good tick arrives. The debounce state lives in `/data/alert_state.json`, so a container restart does not re-fire an alert you already saw.
The alert strings are Chinese in the current code (`monitor.py` → `alert_on_health()`); change the two `msg = (…)` literals if you want English.
## Backups
| What | Where | How |
|---|---|---|
| All listings, settings and filters | volume `nocodb-data` | stop the stack, tar the volume; or use NocoDB's own **Export base** |
| Health + alert state | volume `carousell-data` | disposable — do not bother |
| This repo's config | `.env` / the Portainer stack file | keep a copy in your password manager |
Nothing else is stateful. NocoDB is the single source of truth.
## Upgrading
**The monitor** (code change in this repo):
```bash
git pull
docker compose -f docker-compose.allinone.yml up -d --build
```
The rebuild re-creates the container. The schema bootstrap and the seen-set are idempotent, so nothing is duplicated.
**NocoDB** — change the image tag in the compose file and:
```bash
docker compose -f docker-compose.allinone.yml up -d
```
NocoDB migrates its own database on start. Back up `nocodb-data` first. The monitor's schema bootstrap talks to NocoDB's **meta API**, which does change between majors: the tag in `docker-compose.allinone.yml` is the version range this code is verified against, so upgrade it deliberately (and check the table columns in the UI afterwards).
**Portainer users:** same thing through the UI — edit the stack file, *Update the stack*. Be careful with `Re-pull image` on a stack whose env values were entered through Portainer's panel (see the README's masking warning).
## Tests
```bash
python test_pagination.py # stdlib only, no network, exit 0 = pass
```
Covers the things that have actually broken: NocoDB paging past 1000 rows, the Telegram HTTP verb bug, the failure-ratio threshold maths, and the fetch-gap timing.
## Tuning notes
| Goal | Change |
|---|---|
| More search coverage | add rows to `Settings`, keep `check_interval_minutes` ≥ 5 |
| React faster | lower `check_interval_minutes` (not `TICK_SECONDS` below ~30 s) |
| Be gentler on Carousell | raise `check_interval_minutes`, keep `FETCH_GAP_SECONDS` ≥ 1 |
| Alert inbox too noisy | set `notify = false` on a watch, or add entries to `IgnoredKeywords` / `IgnoredSellers` |
| Fewer failure alerts | raise `ERROR_ALERT_AFTER`, or let `FAILURE_RATIO_THRESHOLD` stay at `1.0` |
## Data safety rules
- `.env` and your NocoDB token are credentials. Never commit them, never paste them into an issue.
- The NocoDB token is scoped to a workspace: regenerate it in NocoDB and update the stack if it leaks.
- The monitor never deletes or modifies archived listings, apart from the `notified` / `skip_notify` flags. Cleaning up the archive is your job — NocoDB's grid view deletes rows fine.
docs/QUICKSTART-PORTAINER.md
+106
View File
@@ -0,0 +1,106 @@
# Quick start on Portainer (click-by-click)
For a fresh machine, no shell needed. Portainer runs the same compose file as the CLI flow — see the [README](../README.md) for that version.
**Before you start:** Portainer must already be up and connected to a Docker endpoint, and Portainer's own container needs access to the Docker socket (the standard install does). NocoDB will need one free host port — `8080` by default.
---
## 1. Create the NocoDB container first
Because you cannot paste the base id and token until NocoDB exists, do this in two deploys.
1. **Stacks → Add stack**
2. **Name:** `carousell-monitor`
3. **Build method:** *Web editor*
4. Paste the **first service only** for now — the `nocodb:` block from [`docker-compose.allinone.yml`](../docker-compose.allinone.yml) plus the closing `volumes:` / `networks:` sections:
```yaml
services:
nocodb:
image: nocodb/nocodb:2026.09.0
container_name: carousell-nocodb
restart: unless-stopped
environment:
PORT: "8080"
NC_AUTH_JWT_SECRET: <paste output of: openssl rand -hex 32>
NC_DISABLE_TELE: "true"
NC_ALLOW_LOCAL_HOOKS: "false"
volumes:
- nocodb-data:/usr/app/data
ports:
- "8080:8080"
healthcheck:
test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:8080/api/v1/health >/dev/null 2>&1 || exit 1"]
interval: 30s
timeout: 10s
retries: 5
start_period: 60s
networks:
- carousell
volumes:
nocodb-data:
networks:
carousell:
```
5. **Deploy the stack.** Wait for `carousell-nocodb` to show as running/healthy.
## 2. Create the base and collect the two values
Open `http://<your-host>:8080`:
1. Create the admin account.
2. Create a base (`+ New Base`) — call it `Carousell`.
3. Copy the **base id** from the URL (`…/nc/base/<BASE_ID>/…`).
4. Avatar (bottom-left) → **Account Settings → Tokens → Create token**, and copy the `nc_pat_…` value. It is shown once.
## 3. Add the monitor to the same stack
1. Get the Telegram values first ([docs/TELEGRAM-SETUP.md](TELEGRAM-SETUP.md)) — bot token from @BotFather, chat id from @userinfobot.
2. Back in Portainer: **Stacks → carousell-monitor → Editor** tab.
3. Replace the whole file with the content of [`docker-compose.allinone.yml`](../docker-compose.allinone.yml), **with the real values written in directly** instead of `${...}` placeholders:
```yaml
environment:
NOCODB_URL: http://nocodb:8080
NOCODB_TOKEN: nc_pat_your_token_here
NOCODB_BASE_ID: your_base_id_here
TELEGRAM_BOT_TOKEN: 8123456789:AAF...
TELEGRAM_CHAT_ID: 123456789
```
and the same for `NC_AUTH_JWT_SECRET` / `NC_DISABLE_TELE` in the `nocodb` service.
> **Why inline values instead of Portainer's environment panel?** Portainer CE masks secret-looking values that are entered through the UI and stores the mask with the stack. On the next stack update the container receives `***` and silently stops authenticating. Editing the YAML keeps the real value on your Portainer host, which is where a stack file's secrets live anyway.
4. **Update the stack.** Portainer recreates the services that changed and adds `carousell-monitor`.
5. **Containers → carousell-monitor → Logs**: expect one line starting with `ready: listings=… settings=… seen=0`.
## 4. Start using it
In NocoDB, open the base — four tables are now present. Add your first watch to **`Settings`** (see [docs/NOCODB-SETUP.md](NOCODB-SETUP.md#3-the-tables-the-monitor-creates)) and give it a minute. Then untick `notified` on any `Listings` row to force a test alert.
## 5. Everyday maintenance in Portainer
| Task | Where |
|---|---|
| Change a search / filters | NocoDB UI — nothing to redeploy |
| Change credentials or tuning | **Stacks → carousell-monitor → Editor** → edit YAML → **Update the stack** |
| See why it is unhealthy | **Containers → carousell-monitor → Logs**, then the `Listings`/`Settings` tables for real progress |
| Restart | **Containers → carousell-monitor → Restart** (state is in NocoDB, nothing is lost) |
| Update this project | pull the new code on the host, then **Editor** → *Update the stack* with `Re-pull image`/rebuild enabled, or `docker compose up -d --build` on the CLI |
| Back up | **Volumes → nocodb-data** (all data) — or NocoDB's own base export |
## 6. Optional: deploy from the repository instead
**Stacks → Add stack → Repository**, with:
| Field | Value |
|---|---|
| Repository URL | `https://github.com/hoelee/carousell-monitor` |
| Reference | `refs/heads/main` |
| Compose path | `docker-compose.allinone.yml` |
Set `NOCODB_BASE_ID` / `NOCODB_TOKEN` / `TELEGRAM_*` / `NC_AUTH_JWT_SECRET` in the environment panel for this one (they are not in git). Be aware of the masking caveat above, and that Portainer clones the repository on every deploy.
docs/TELEGRAM-SETUP.md
+71
View File
@@ -0,0 +1,71 @@
# Telegram setup
Two values are needed: a **bot token** (who sends) and a **chat id** (where it sends). Both are free and take about a minute.
---
## 1. Create the bot
1. Open Telegram and search for **@BotFather** (the one with a blue checkmark).
2. Send `/newbot`.
3. It asks for a **name** — anything, e.g. `My Carousell Alerts`.
4. It asks for a **username** — must be unique and end in `bot`, e.g. `mycarousell_alerts_bot`.
5. It replies with a token like:
```
Use this token to access the HTTP API:
8123456789:AAF7xK3nQw8_your_token_here_9dZ
```
That whole string is `TELEGRAM_BOT_TOKEN`. Treat it like a password — anyone holding it can send and read messages as your bot.
Optional, but nice: `/setdescription` and `/setuserpic` to make the alert messages look intentional.
## 2. Get your chat id (alert yourself)
1. **Send your new bot a message** — click the link BotFather gave you and say `hi`. This step matters: until you have messaged the bot, it is not allowed to message you, and sends fail with `chat not found`.
2. Message **@userinfobot**. It replies with your id:
```
Id: 123456789
```
That number is `TELEGRAM_CHAT_ID`.
## 3. Or alert a group
1. Create the group (or use an existing one) and **add your bot** to it as a member.
2. Send a message in the group (any text).
3. Read the group's chat id:
```bash
curl -s "https://api.telegram.org/bot<YOUR_TOKEN>/getUpdates"
```
Look for `"chat":{"id":-1001234567890,"title":"..."}`. Group ids are **negative** and usually start with `-100`. Use that number as `TELEGRAM_CHAT_ID`.
If `getUpdates` returns `{"ok":true,"result":[]}`, the bot has not seen any message yet — send another one in the group and retry.
## 4. Verify before you blame the monitor
```bash
# who am I?
curl -s "https://api.telegram.org/bot<YOUR_TOKEN>/getMe"
# send a test message
curl -s -X POST "https://api.telegram.org/bot<YOUR_TOKEN>/sendMessage" \
-d chat_id=<YOUR_CHAT_ID> -d text="hello from carousell-monitor"
```
Expected: `{"ok":true,...}` and a message on your phone. If this works, the monitor's credentials are right and anything missing is a monitor-side issue (filters, `notify` checkbox, pending queue).
| Error | Meaning |
|---|---|
| `401 Unauthorized` | token is wrong, or it was revoked in BotFather |
| `400 chat not found` | wrong chat id, or you never messaged the bot / the bot is not in the group |
| `403 bot was blocked by the user` | you blocked the bot — unblock it |
| the `curl` hangs forever | DNS/IPv6 trouble. The compose files pin `api.telegram.org` to its IPv4 address in `extra_hosts`; refresh that IP with `dig +short api.telegram.org` |
## 5. Rotate it later
If the token leaks: @BotFather → `/revoke` → pick the bot → you get a new token. Update `.env` (or the Portainer stack environment) and restart the monitor.
docs/TROUBLESHOOTING.md
+116
View File
@@ -0,0 +1,116 @@
# Troubleshooting
A symptom, a cause, a fix — plus the decision tree to run when the symptom is the vague one: *"it stopped alerting me"*.
---
## 1. The decision tree for "no alert"
Four different failures look identical from the outside. Find which stage breaks before changing anything.
**Stage 0 — is it running at all?**
```bash
docker compose -f docker-compose.allinone.yml ps
docker exec carousell-monitor cat /data/health.json
docker exec carousell-monitor cat /data/alert_state.json # fail_streak, alerted
```
- `ok: false` → the fetch loop is dead. Read `error` and jump to §2.
- `last_run_epoch` older than a couple of ticks → the loop is stuck; check the logs for a traceback.
- `No such file` → the container never completed a tick (bad credentials most likely — see `NOCODB_TOKEN not set`, §2).
**Stage 1 — is it fetching?**
Open `Settings` and check `last_checked_at` on your watch. It should advance every `check_interval_minutes`. If it never advances, the watch is disabled (`enabled` unticked), the interval is huge, or every fetch is failing.
**Stage 2 — is it archiving?**
Open `Listings`, sort by `first_seen_at` descending. New rows appearing means fetch + parse + NocoDB writes all work. If `last_checked_at` advances but `Listings` stays empty, you are the victim of an over-narrow search, not a bug — nothing new has appeared.
**Stage 3 — is it notifying?**
This is where the archive can look healthy while Telegram is silent. Sort `Listings` by `notified`:
- Rows with `notified = false` **and** `skip_notify = false` that stay false → the send is failing (see §2 transport rows) or the monitor is stuck.
- Rows with `skip_notify = true` → a filter matched, by design. Check `IgnoredSellers` / `IgnoredKeywords` and the watch's `notify` checkbox.
**Stage 4 — is the transport sane?**
From inside the container, prove the credentials and egress in one shot:
```bash
docker exec carousell-monitor sh -c '
wget -qO- "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/getMe"'
```
`{"ok":true,...}` means the token is good and the container can reach Telegram. Then send a real message:
```bash
docker exec carousell-monitor sh -c '
wget -qO- --post-data="chat_id=$TELEGRAM_CHAT_ID&text=test" \
"https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/sendMessage"'
```
If `getMe` works but the send does not, it is the chat id (a bot cannot open a conversation with you before you have messaged it — see [TELEGRAM-SETUP.md](TELEGRAM-SETUP.md)).
**Stage 5 — force a notification.**
In NocoDB, untick `notified` on any listing row. Within one tick the monitor re-sends it and flips the flag back to `true`. The flip **is** the proof of delivery: it only happens after Telegram returns HTTP 200.
---
## 2. Symptom → cause → fix
| Symptom | Likely cause | Fix |
|---|---|---|
| `NOCODB_TOKEN not set` in the logs, container exits | `.env` missing the token, or the stack did not pick it up | fill it in and re-create the container (`up -d`, or Portainer *Update the stack*) |
| `list tables failed: HTTP 401/403` | token wrong, expired, or from another workspace | create a token inside the workspace that owns the base |
| `list tables failed: HTTP 404` | wrong `NOCODB_BASE_ID` | re-copy the id from the base URL |
| `create table … failed` / `add column … failed` | the token can read but not write, or the base was deleted | verify with a write test (create a scratch table in the UI), check the base exists |
| `connection refused` to `nocodb` | the monitor is not on the same Docker network, or the host name is wrong | both services must share the network in the compose file; `NOCODB_URL` must use the service/container name, not `localhost` |
| `no application/json state found (blocked/ratelimited?)` | Carousell returned a challenge or error page | raise `check_interval_minutes`, keep `FETCH_GAP_SECONDS ≥ 1`, reduce the number of watches; the watch retries next interval |
| `listingCards null (soft-block/ratelimit?)` | Carousell answered 200 with an empty state blob | same as above — this is rate limiting, not a bug |
| `tick error: …` and the container goes unhealthy | an exception escaped the tick (NocoDB error, unexpected page shape) | read the message; the loop keeps running and retries |
| `telegram sendMessage failed: 400` in the logs | the request was malformed — historically a code bug where the Telegram **method name** was used as the HTTP verb. Fixed; keep `test_pagination.py` passing | run the test suite |
| Telegram sends hang and time out | Docker DNS handing back an IPv6 (AAAA) address on a network with no IPv6 | keep/refresh the `extra_hosts` pin (`dig +short api.telegram.org`) |
| `chat not found` (400) | you never started the bot; or the group id is wrong | message the bot once; re-read the id from `getUpdates` |
| Archive filling up, Telegram silent | see the tree in §1 — usually the `notify` checkbox, a filter match, or a failed send that is being retried |
| Every listing alerted twice | you reset `notified`, or two monitors point at the same base | untick only once; run one monitor per base |
| Hundreds of messages right after setup | the watch's `last_checked_at` was not empty (it was seeded before) | expected on a re-seed; delete `last_checked_at` only when you *want* a silent re-seed |
| Thumbnails broken in the NocoDB grid | the `image` column is not an Attachment column | restart the monitor — the bootstrap recreates missing columns |
| Timestamps 8 hours off | stored as UTC by design | change the display timezone in NocoDB |
| Container healthy but nothing happens | no enabled watches, or all of them fall outside their interval | add/enable a row in `Settings`, lower `check_interval_minutes` |
---
## 3. Useful commands
```bash
# status
docker compose -f docker-compose.allinone.yml ps
# follow the logs
docker compose -f docker-compose.allinone.yml logs -f --tail 100 carousell-monitor
# health + alert state as the container sees them
docker exec carousell-monitor cat /data/health.json
docker exec carousell-monitor cat /data/alert_state.json
# what credentials did it actually receive?
docker exec carousell-monitor sh -c 'env | grep -E "NOCODB|TELEGRAM|TICK|FAILURE"'
# is NocoDB reachable from the monitor's network?
docker exec carousell-monitor sh -c 'wget -qO- http://nocodb:8080/api/v1/health'
# restart / rebuild / stop
docker compose -f docker-compose.allinone.yml restart carousell-monitor
docker compose -f docker-compose.allinone.yml up -d --build
docker compose -f docker-compose.allinone.yml down
```
---
## 4. Reporting a bug
Include: the `health.json` contents, the last ~50 log lines, the compose file with every credential replaced by `***`, and your NocoDB version (`…/api/v1/version`). Do not paste tokens, chat ids or your base id.