Drop IgnoredKeywords.search_url column; resolve URL via watch link
The search_url Lookup column is redundant: the watch Link -> Settings row already defines the association. Remove handling for the column: - _ensure_keyword_watch_link() no longer creates a Lookup; deletes any legacy search_url (URL or Lookup type) on bootstrap - load_ignored_keywords(kw_tid, settings_tid) resolves Settings.url via the watch link, one Settings fetch per cycle (no N+1) - send_pending_notifications passes settings_tid through - Add COMPOSE-SETUP.md: detail documentation for the compose stack (topology, compose block-by-block, Dockerfile, healthcheck, deploy paths, troubleshooting); referenced from README + DOCUMENTATION.md
This commit is contained in:
@@ -0,0 +1,249 @@
|
||||
# COMPOSE-SETUP.md — Carousell Monitor Docker Stack, Explained
|
||||
|
||||
Line-by-line anatomy of the compose stack, how it connects to the rest of the
|
||||
homelab, and the two ways to deploy it. Complementary to `DOCUMENTATION.md`
|
||||
(ops) and `AGENTS.md` (agent entry); this file is the *stack reference*.
|
||||
|
||||
---
|
||||
|
||||
## 1. Topology
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────┐
|
||||
│ DSM host (Synology) │
|
||||
│ │
|
||||
Internet ─ 443 ──► │ DSM nginx reverse proxy │
|
||||
│ │ (not involved for this container — │
|
||||
│ │ it makes outbound calls only) │
|
||||
│ ▼ │
|
||||
│ Docker network bridge_hoelee (external) │
|
||||
│ │ │
|
||||
│ ├── carousell-monitor (this stack) │
|
||||
│ │ │ │
|
||||
│ │ ├─► NocoDB http://nocodb:10380 │
|
||||
│ │ │ (same bridge_hoelee network) │
|
||||
│ │ ├─► Carousell www.carousell.com.my │
|
||||
│ │ │ (public internet, GET search)│
|
||||
│ │ └─► Telegram api.telegram.org │
|
||||
│ │ (pinned IPv4 via extra_hosts)│
|
||||
│ │ │
|
||||
│ └── nocodb container (named "nocodb") │
|
||||
└────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
The monitor is an **outbound-only** worker: it has no inbound port, no web UI,
|
||||
and is never reached through the reverse proxy. All state lives in NocoDB
|
||||
(Listings / Settings / IgnoredSellers / IgnoredKeywords), all alerts go out via
|
||||
Telegram.
|
||||
|
||||
---
|
||||
|
||||
## 2. docker-compose.yml, block by block
|
||||
|
||||
### Top-level services
|
||||
|
||||
```yaml
|
||||
services:
|
||||
carousell-monitor:
|
||||
build: .
|
||||
image: carousell-monitor:latest
|
||||
container_name: carousell-monitor
|
||||
restart: unless-stopped
|
||||
```
|
||||
|
||||
| Key | Meaning |
|
||||
|---|---|
|
||||
| `build: .` | Image is built **locally from this directory** (Dockerfile in repo) — nothing is pulled from a registry |
|
||||
| `image: carousell-monitor:latest` | Local tag for the built image; docker compose will rebuild + retag on `up --build` |
|
||||
| `container_name` | Fixed name = stable DNS name on the network, predictable for health checks and logs |
|
||||
| `restart: unless-stopped` | Survive daemon restarts and DSM reboots; stop it manually to keep it down |
|
||||
|
||||
### Naming vs. images
|
||||
|
||||
The image is **private**: built on DSM, never pushed to Docker Hub or any
|
||||
registry. `image:` is just a local convenience tag (`docker images` shows it; the
|
||||
build context is the repo directory).
|
||||
|
||||
### extra_hosts — the Telegram IPv4 pin
|
||||
|
||||
```yaml
|
||||
extra_hosts:
|
||||
- "api.telegram.org:149.154.166.110"
|
||||
```
|
||||
|
||||
**Why this exists (root cause, do not delete):**
|
||||
|
||||
- The container runs inside `bridge_hoelee`, which has **no IPv6**.
|
||||
- Docker's embedded DNS at `127.0.0.11` may return an **IPv6 AAAA record** for
|
||||
`api.telegram.org`; with no IPv6 route, the connection hangs and Telegram
|
||||
sends fail silently.
|
||||
- Pinning the correct IPv4 (as of 2026-09) in `extra_hosts` short-circuits DNS
|
||||
and makes `api.telegram.org` resolve straight to the IPv4.
|
||||
- ⚠ If Telegram calls start timing out again, **re-verify the current IP**:
|
||||
`nslookup api.telegram.org` / `dig +short api.telegram.org` — Telegram rotates
|
||||
IPs. Update the pin, then redeploy.
|
||||
- Note: the **live** stack on DSM pins a slightly different IP than the repo copy —
|
||||
the live one was fixed during an earlier incident. Treat the repo value as
|
||||
the canonical starting point, and verify before assuming.
|
||||
|
||||
### environment
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
NOCODB_URL: ${NOCODB_URL:-http://nocodb:10380}
|
||||
NOCODB_TOKEN: ${NOCODB_TOKEN}
|
||||
NOCODB_BASE_ID: ${NOCODB_BASE_ID:-poqw1zjw3hnsk37}
|
||||
TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN}
|
||||
TELEGRAM_CHAT_ID: ${TELEGRAM_CHAT_ID}
|
||||
TICK_SECONDS: ${TICK_SECONDS:-60}
|
||||
HEALTH_STALE_SECONDS: ${HEALTH_STALE_SECONDS:-600}
|
||||
TZ: Asia/Kuala_Lumpur
|
||||
```
|
||||
|
||||
| Var | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `NOCODB_URL` | `http://nocodb:10380` | NocoDB REST endpoint. `nocodb` = the NocoDB container's name on `bridge_hoelee` (container DNS). Never use a public hostname here. |
|
||||
| `NOCODB_TOKEN` | *(required)* | Workspace-scoped NocoDB PAT (xref `SECRETS.md`). Never hard-code; comes from the stack env/.env. |
|
||||
| `NOCODB_BASE_ID` | `poqw1zjw3hnsk37` | Base "Carousell" — all four tables live under it. |
|
||||
| `TELEGRAM_BOT_TOKEN` | *(required)* | @carousellFoundBot token (xref `SECRETS.md`). |
|
||||
| `TELEGRAM_CHAT_ID` | *(required)* | `5648309582` — @MrFullStackDev. |
|
||||
| `TICK_SECONDS` | `60` | Scheduler granularity: heartbeat + watch-list reload interval. |
|
||||
| `HEALTH_STALE_SECONDS` | `600` | Docker healthcheck tolerance: if last tick older than this → unhealthy. |
|
||||
| `TZ` | `Asia/Kuala_Lumpur` | Container clock (mostly cosmetic; timestamps are written in UTC deliberately for NocoDB). |
|
||||
|
||||
`${VAR:-default}` syntax: compose substitutes the value from the environment /
|
||||
`.env` file, falling back to the literal default when unset. `${NOCODB_TOKEN}`
|
||||
with **no** default means it's mandatory — compose errors if missing.
|
||||
|
||||
### volumes
|
||||
|
||||
```yaml
|
||||
volumes:
|
||||
- carousell-data:/data
|
||||
```
|
||||
|
||||
Named volume `carousell-data` mounted at `/data`. Inside the container that's:
|
||||
|
||||
- `monitor.py` → `DATA_DIR` default → `/data/health.json` (written each tick, read by the healthcheck)
|
||||
- Nothing else is stored there — NocoDB holds all real state.
|
||||
|
||||
Why a volume and not a bind mount: survives container recreation, no host-path
|
||||
permission issues on DSM ACLs, and it's private to the stack (not exposed to the
|
||||
host filesystem).
|
||||
|
||||
### networks
|
||||
|
||||
```yaml
|
||||
networks:
|
||||
- bridge_hoelee
|
||||
|
||||
networks:
|
||||
bridge_hoelee:
|
||||
external: true
|
||||
```
|
||||
|
||||
`external: true` = the network is **pre-existing** (created by the NocoDB stack,
|
||||
typically). Compose does not create it, just joins it. This is what lets the
|
||||
monitor reach `http://nocodb:10380` by container name instead of an IP that
|
||||
drifts (DSM IP history: 192.168.137.2 → 192.168.1.1 → …).
|
||||
|
||||
---
|
||||
|
||||
## 3. Dockerfile
|
||||
|
||||
```dockerfile
|
||||
FROM python:3.11-alpine
|
||||
|
||||
WORKDIR /app
|
||||
COPY monitor.py healthcheck.py /app/
|
||||
RUN mkdir -p /data
|
||||
|
||||
VOLUME ["/data"]
|
||||
|
||||
HEALTHCHECK --interval=60s --timeout=15s --start-period=120s --retries=3 \
|
||||
CMD python /app/healthcheck.py
|
||||
|
||||
CMD ["python", "-u", "/app/monitor.py"]
|
||||
```
|
||||
|
||||
- `python:3.11-alpine` — tiny, stdlib-only code needs no pip deps → fast builds, small image.
|
||||
- `COPY` bakes the script into the image → a rebuild is how code ships (there is no bind-mount of the repo).
|
||||
- `HEALTHCHECK` runs `healthcheck.py` every 60s: exits 0 iff `/data/health.json` exists, is newer than `HEALTH_STALE_SECONDS`, and `ok == true`.
|
||||
- `python -u` — unbuffered stdout so `docker logs` shows ticks in real time.
|
||||
|
||||
### healthcheck.py logic
|
||||
|
||||
```python
|
||||
age = time.time() - int(h.get("last_run_epoch", 0))
|
||||
if age <= STALE and h.get("ok") is True:
|
||||
sys.exit(0) # healthy
|
||||
sys.exit(1) # unhealthy
|
||||
```
|
||||
|
||||
One failed tick (Carousell 403/429/parse error, NocoDB down) → `ok:false` →
|
||||
container turns **unhealthy** until a fully successful tick. That's the tripwire:
|
||||
Portainer shows it red, `docker inspect` reports it, and you can alert on it.
|
||||
|
||||
---
|
||||
|
||||
## 4. Deployment paths
|
||||
|
||||
### A. On DSM via Portainer stack (current production)
|
||||
|
||||
1. Edit code in the repo (`D:\dev\carousell-monitor`).
|
||||
2. Commit + push to Gitea (`git.hoelee.com/hoelee/carousell-monitor`).
|
||||
3. On DSM, the deploy dir `/volume1/docker/carousell-monitor` is a **manual copy,
|
||||
not a git clone** — copy the changed files there:
|
||||
`cp monitor.py /volume1/docker/carousell-monitor/`
|
||||
(backup the old one first, per the repo's `.bak` convention).
|
||||
4. Rebuild + recreate:
|
||||
```bash
|
||||
sudo /usr/local/bin/docker compose up -d --build
|
||||
```
|
||||
`--build` rebuilds the image from the new `monitor.py`; the NocoDB schema
|
||||
bootstrap and dedupe seeding are idempotent, so a rebuild never duplicates rows.
|
||||
|
||||
### B. Local dev (Windows, against LAN NocoDB)
|
||||
|
||||
```bash
|
||||
NOCODB_URL=http://192.168.1.1:10380 \
|
||||
NOCODB_TOKEN=... TELEGRAM_BOT_TOKEN=... TELEGRAM_CHAT_ID=... \
|
||||
python monitor.py
|
||||
```
|
||||
|
||||
Runs the same loop outside Docker — useful for testing code changes before
|
||||
shipping (uses LAN IP instead of container DNS).
|
||||
⚠ `192.168.1.1` drifts; check `:10380` on the current DSM IP first.
|
||||
|
||||
---
|
||||
|
||||
## 5. What to check when something breaks
|
||||
|
||||
| Symptom | Check |
|
||||
|---|---|
|
||||
| Container `unhealthy` | `sudo docker logs carousell-monitor --tail 100` — read the `error` line |
|
||||
|"No application/json state" | Carousell rate-limited/blocked; raise `check_interval_minutes` in Settings |
|
||||
| No Telegram | `TELEGRAM_BOT_TOKEN`/`TELEGRAM_CHAT_ID` correct? `notify` box on the watch? Telegram IP pin in `extra_hosts` still valid? |
|
||||
| "NOCODB_TOKEN not set" | Stack env missing the token; fix in Portainer stack env, redeploy |
|
||||
| NocoDB unreachable | Is `nocodb` container on `bridge_hoelee`? `sudo docker network inspect bridge_hoelee` |
|
||||
| Timestamps 8h off | They are **UTC by design**; NocoDB display TZ → Asia/Kuala_Lumpur |
|
||||
|
||||
---
|
||||
|
||||
## 6. Files in the repo
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `docker-compose.yml` | Stack definition (this doc) |
|
||||
| `Dockerfile` | Image build + healthcheck |
|
||||
| `monitor.py` | The whole monitor (stdlib only) |
|
||||
| `healthcheck.py` | Docker HEALTHCHECK probe |
|
||||
| `.env.example` | Template for local runs (secrets go in `.env`, gitignored) |
|
||||
| `SECRETS.md` | Credential inventory (private repo) |
|
||||
| `DOCUMENTATION.md` | Deployment & operations manual |
|
||||
| `AGENTS.md` | AI-agent entry point |
|
||||
| `COMPOSE-SETUP.md` | This file |
|
||||
|
||||
---
|
||||
|
||||
*Last reviewed: 2026-09-13 (NocoDB 2026.08.1, Docker Compose v2.20.1 on DSM).*
|
||||
Reference in New Issue
Block a user