carousell-monitor

Watch Carousell searches, archive every listing into NocoDB, and get a Telegram alert the moment something new appears.

Self-hosted and free: one docker compose file, no scraping API, no paid service, no account anywhere except your own bot. Built for people who are tired of refreshing search pages all day — sneaker drops, camera gear, used furniture, car parts, uniform lots, whatever you are hunting for.

Python 3.11 · stdlib only · Docker / Portainer · NocoDB · Telegram · MIT

---

What it does

• Watches any Carousell search URL you give it (sorted by Recent), on its own interval.
• Archives every listing it sees into NocoDB — title, numeric price, condition, seller, link, and the product photo (grid view shows thumbnails). Deduplicated by product URL, so nothing is stored twice.
• Alerts you on Telegram, one photo message per new listing, with the full details and a clickable link.
• Stays quiet about the past. The first time it sees a search it archives everything silently, so you are not flooded with 200 messages on setup. Only listings that appear after that first pass are alerted.
• Filters — mute a seller everywhere, or mute keywords for one specific search (still archived, no alert).
• Tells you when it breaks. If the container can no longer fetch, or the loop dies, you get a Telegram failure alert (after N consecutive failures, debounced) and a recovery notice when it works again.

How it works

        ┌──────────────────────── your machine / NAS / VPS ────────────────────────┐
        │                                                                          │
        │   docker network: carousell (private)                                    │
        │                                                                          │
        │   ┌───────────────────────────┐          ┌────────────────────────────┐  │
        │   │ carousell-monitor         │  NocoDB  │ nocodb                     │  │
        │   │ (python, no open ports)   │◄────────►│ (web UI + SQLite, :8080)   │  │
        │   └───────┬───────────┬───────┘  REST    └────────────────────────────┘  │
        │           │           │                                                  │
        └───────────┼───────────┼──────────────────────────────────────────────────┘
                    │           │
        every tick  │           │  new listing  ──►  Telegram alert (photo + details)
                    ▼           ▼
        Carousell search pages   api.telegram.org

The monitor is an outbound-only worker: no inbound port, no web UI, nothing to expose to the internet. All state lives in NocoDB; you drive it from the NocoDB UI.

Each tick it:

1. reads your watch list from the NocoDB Settings table,
2. fetches each search page that is due and extracts the embedded listing JSON,
3. inserts anything it has not seen before into Listings,
4. sends one Telegram message per still-unnotified listing and marks it notified.

More detail: docs/ARCHITECTURE.md.

---

Requirements

Docker	Engine 20.10+ with Compose v2 (docker compose version) — or Portainer, if you prefer a GUI
Telegram	a free account (you create the bot in 60 seconds, see below)
Machine	anything that stays on: NAS, VPS, Raspberry Pi, mini-PC, old laptop
Skills	being able to copy-paste commands and edit a text file

Disk: a few hundred MB (NocoDB + the listings database, which grows slowly). RAM: NocoDB wants ~300 MB, the monitor ~40 MB.

---

Quick start (CLI)

Prefer clicking? Jump to Deploy on Portainer — same result, no shell needed.

Step 1 — get the code and the env file

git clone https://github.com/hoelee/carousell-monitor.git
cd carousell-monitor
cp .env.example .env

Step 2 — create your Telegram bot

Open Telegram, talk to @BotFather, send /newbot, follow the prompts. You get a token that looks like 8123456789:AAF....

Now message @userinfobot and it replies with your numeric id.

Put both into .env:

TELEGRAM_BOT_TOKEN=8123456789:AAF...
TELEGRAM_CHAT_ID=123456789

Full walkthrough, including how to alert a group instead of yourself: docs/TELEGRAM-SETUP.md.

Step 3 — start NocoDB and create the base

docker compose -f docker-compose.allinone.yml up -d nocodb

Wait ~30 seconds, then open http://localhost:8080 (replace localhost with your server's address if you are deploying remotely).

1. Create your account (the first account is the admin — use a real email and a real password).

2. Create a Base, name it e.g. Carousell.

3. Copy the base id out of the browser URL — the long id after /nc/base/:

   http://localhost:8080/dashboard/#/nc/base/poqw1zjw3hnsk37/...
                                            ^^^^^^^^^^^^^^^ copy this
   

4. Create an API token: click your avatar (bottom-left) → Account Settings → Tokens → Create token. Copy it (it starts with nc_pat_). NocoDB only shows it once.

Put both into .env:

NOCODB_BASE_ID=poqw1zjw3hnsk37
NOCODB_TOKEN=nc_pat_...

While you are there, generate a session secret and put it in .env too:

openssl rand -hex 32     # paste the output into NC_AUTH_JWT_SECRET

The tables themselves are created by the monitor, so right now your fresh base is empty and that is expected.

Step 4 — start the monitor

docker compose -f docker-compose.allinone.yml up -d
docker compose -f docker-compose.allinone.yml logs --tail 20 carousell-monitor

First run builds the image (a minute or so). A healthy start prints one line:

ready: listings=... settings=... ignored_sellers=... ignored_keywords=... seen=0

It is quiet after that — there is no per-tick logging by design. Progress is visible in NocoDB instead: four tables appear (Listings, Settings, IgnoredSellers, IgnoredKeywords), and the container reports healthy.

Step 5 — tell it what to watch

1. Open your NocoDB base → the Settings table.

2. Add a row:

   title	url	enabled	notify	check_interval_minutes
   Uniform	(paste a Carousell search URL)	✓	✓	5

3. To get a good URL: search on carousell.com.my, set the sort to Recent, then copy the address bar. Make sure it contains sort_by=3 — that is what puts the newest listings first:

   https://www.carousell.com.my/search/uniform?sort_by=3&...
   

That is it. The monitor re-reads Settings every tick, so no restart is needed after adding, editing or pausing a search.

Step 6 — prove the alert works

Open the Listings table: your first pass has already archived the current listings (silently — no messages, that is correct). Then:

• Send a test alert: untick notified on any row and save. Within a tick the monitor re-sends that listing to Telegram. That is also the fastest way to debug a silent bot.
• Watch a real one arrive: the next genuinely new listing appears in Listings and lands in Telegram by itself.

Nothing in Telegram? See Troubleshooting.

---

Deploy on Portainer (GUI)

Portainer runs the same file — it is a normal compose stack. One catch first: the monitor's image is built from this repository (build: .), and Portainer's Web editor / Upload options have no build context, so they fail with:

compose build operation failed: failed to read dockerfile:
open /volume1/@docker/tmp/…/Dockerfile: no such file or directory

So pick one of these two routes — the first needs no shell at all.

Route 1 — deploy from the Git repository (recommended)

Portainer clones the repo itself, so build: . resolves normally.

1. Stacks → Add stack → Repository.

2. Name: carousell-monitor.

3. Repository URL: https://github.com/hoelee/carousell-monitor · Reference: refs/heads/main · Compose path: docker-compose.allinone.yml.

4. Environment variables — add these (Portainer lists the ones the file references):

   Variable	Value
   NC_AUTH_JWT_SECRET	output of openssl rand -hex 32
   TELEGRAM_BOT_TOKEN	from @BotFather
   TELEGRAM_CHAT_ID	from @userinfobot
   NOCODB_PORT	optional, default 8080

   Leave NOCODB_TOKEN and NOCODB_BASE_ID empty for now — you cannot have them until NocoDB is running.

5. Deploy the stack. carousell-nocodb comes up healthy; carousell-monitor will restart in a short loop logging NOCODB_TOKEN not set. That is expected — stop that container for now if the noise bothers you.

6. Open http://<your-host>:8080 and follow Step 3: create the account, create a base, copy the base id, create an API token.

7. Back in Portainer: Stacks → carousell-monitor → Editor (or Environment variables) and fill in NOCODB_BASE_ID + NOCODB_TOKEN, then Update the stack. The monitor starts its real loop.

   ⚠️ Portainer masks secret-looking values that you type into its environment panel and stores the mask (***) with the stack, so a value can silently stop working after a later stack update. If you hit that, put the real values directly in the YAML in the Editor tab instead of the panel.

8. Verify: Containers shows carousell-nocodb and carousell-monitor healthy, and the monitor's log shows the ready: … line.

Route 2 — Web editor, with the image built by hand first

Use this if you cannot give Portainer access to your Git host, or you want the stack file to be self-contained.

# on the Docker host (or anywhere that can reach its daemon)
git clone https://github.com/hoelee/carousell-monitor.git
cd carousell-monitor
docker build -t carousell-monitor:latest .

Then Stacks → Add stack → Web editor, paste docker-compose.allinone.yml and delete the build: . line from the monitor service — Portainer's web-editor stack directory contains no Dockerfile, so a build: entry can only fail there. Fill in the same variables as above, deploy NocoDB first, then add the base id and token and Update the stack.

Afterwards, updating the code means: git pull && docker build -t carousell-monitor:latest . on the host, then Update the stack in Portainer.

Managing it afterwards

Task	Where
Change a search / filters	NocoDB UI — nothing to redeploy
Change credentials or tuning	Stacks → carousell-monitor → Editor (or Environment variables) → Update the stack
See logs	Containers → carousell-monitor → Logs
Restart	Containers → carousell-monitor → Restart (state lives in NocoDB — nothing is lost)
Back up	Volumes → nocodb-data — that volume is your data
Upgrade NocoDB	change the image tag in the Editor → Update the stack (back up the volume first)

---

Everyday use (all from the NocoDB UI — no SSH, no restart)

Open your base and work in the tables:

I want to…	Do this
Watch another search	Add a row to Settings: title, full url (with sort_by=3), enabled ✓, notify ✓, interval
Pause a search	Untick enabled (or delete the row)
Keep archiving but stop Telegram for a search	Untick notify
Check more or less often	Edit check_interval_minutes (5 = every 5 minutes)
Mute a seller everywhere	Add their username to IgnoredSellers
Mute keywords for one search	Add rows to IgnoredKeywords: pick the search in the watch dropdown, type the keyword (e.g. nike)
See what is new	Listings, sorted by first_seen_at (newest first)
Browse with pictures	Listings → grid view; the image column renders thumbnails
Hide an archived row from the alert queue	tick notified on it (pending rows are notified = false)

Keyword matching is a case-insensitive substring of the title, and it only applies to the search you linked it to. Ignored sellers and ignored keywords are still archived — they just do not ring your phone.

Filter fields at a glance

Table	What it is	Fields you create
Settings	your watch list	title, url, enabled, notify, check_interval_minutes (the monitor maintains last_checked_at)
IgnoredSellers	global seller blocklist	seller_name
IgnoredKeywords	per-search title blocklist	watch (link → Settings), keyword
Listings	the archive — written by the monitor	read-only for you, except notified

Full column reference: docs/NOCODB-SETUP.md.

What the alert looks like

🛒 Seiko 5 SNK809 automatic watch
💰 RM320
📦 Like new
👤 watchguy88
https://www.carousell.com.my/p/seiko-5-snk809-1234567890/

plus the listing photo above the text. The monitor downloads the image and uploads it to Telegram, so the alert still works if Carousell later blocks hotlinking.

---

Configuration

Everything below goes in .env (CLI) or in the stack's environment (Portainer). Only the first five are required. The tuning knobs all have working defaults — ignore them until you have a reason.

Variable	Default	What it does
NOCODB_BASE_ID	—	required — the NocoDB base holding the tables
NOCODB_TOKEN	—	required — NocoDB API token (nc_pat_…)
TELEGRAM_BOT_TOKEN	—	required — from @BotFather
TELEGRAM_CHAT_ID	—	required — your numeric id, or a group id (negative)
NOCODB_URL	http://nocodb:8080	where the monitor reaches NocoDB
NC_AUTH_JWT_SECRET	—	NocoDB session secret (all-in-one stack only); openssl rand -hex 32
NOCODB_PORT	8080	host port for the NocoDB web UI
TICK_SECONDS	60	how often the loop wakes up and re-reads the watch list
FETCH_GAP_SECONDS	1	minimum pause between two Carousell requests in one tick — keeps a burst of due searches from looking like an attack. 0 disables
FAILURE_RATIO_THRESHOLD	1.0	fraction of watches that must fail for the tick to count as failed. 1.0 = only if everything failed, so one soft-blocked search does not flip the container to unhealthy. 0 = never fail
ERROR_ALERT_AFTER	3	consecutive failed ticks before the Telegram failure alert (debounce); one recovery notice follows when it clears
HEALTH_STALE_SECONDS	600	a tick older than this marks the container unhealthy
TZ	Asia/Kuala_Lumpur	container clock; timestamps written to NocoDB are UTC on purpose

Want a different schedule per search? That is check_interval_minutes in the Settings table, not an env var.

Already running NocoDB? Use the monitor-only file

docker-compose.yml deploys just the monitor and joins an existing Docker network (edit the network name and NOCODB_URL to match your NocoDB). That is the setup this project runs in production, next to a NocoDB used by other apps.

---

Operating it

# status + health
docker compose -f docker-compose.allinone.yml ps

# logs (quiet unless something is wrong)
docker compose -f docker-compose.allinone.yml logs --tail 100 carousell-monitor

# restart (state lives in NocoDB — safe, nothing is lost)
docker compose -f docker-compose.allinone.yml restart carousell-monitor

# update to a newer revision of this repo
git pull && docker compose -f docker-compose.allinone.yml up -d --build

# stop everything (data stays in the volumes)
docker compose -f docker-compose.allinone.yml down

Is it actually working? The container writes /data/health.json every tick and the Docker HEALTHCHECK reads it:

docker exec carousell-monitor cat /data/health.json
# {"last_run_epoch": ..., "ok": true, "error": "", "watch_count": 3, "new_this_tick": 0, "failed_watches": 0}

ok: true and a recent last_run_epoch means the loop is alive. ok: false with an error means it is not fetching — read the error. Backups are the two volumes: nocodb-data (all your data) and carousell-data (the health file only).

Deeper: docs/OPERATIONS.md.

---

Troubleshooting

Symptom	Likely cause	Fix
Container is unhealthy	last tick failed (Carousell blocked/rate-limited it, or NocoDB was unreachable)	docker compose logs --tail 50 carousell-monitor and read the error, then cat /data/health.json
Log says NOCODB_TOKEN not set	.env empty or the stack never picked it up	fill it in, then up -d again (Portainer: Update the stack)
add column … failed / create table … failed on startup	wrong NOCODB_BASE_ID, token without access to that base, or NocoDB still starting	check the base id, re-create the token in that base's workspace, make sure NocoDB is healthy first
No Telegram messages at all	token/chat id wrong, notify unticked, or the known Docker/IPv6 hang	untick/retick notified on a row to force a test send; check getMe with curl https://api.telegram.org/bot<TOKEN>/getMe; if the send times out, keep the api.telegram.org pin in extra_hosts (the all-in-one file already has it)
No Telegram for one search	notify unticked on that row, or the listing's seller/keyword is ignored	check the Settings, IgnoredSellers, IgnoredKeywords tables
no application/json state found (blocked/ratelimited?)	Carousell served a challenge page instead of results	raise check_interval_minutes, keep FETCH_GAP_SECONDS ≥ 1, and do not watch dozens of searches at once
listingCards null (soft-block/ratelimit?)	same thing, softer: Carousell answered 200 with empty state	same as above; the tick is retried, this is not a crash
Thumbnails missing in NocoDB	image column is not an Attachment column (edited?)	the monitor recreates columns on start — restart the container
Timestamps look 8 hours off	they are UTC by design	set NocoDB's display timezone to your local zone; the stored values stay UTC
NocoDB web UI unreachable	port clash or the container is not up	change NOCODB_PORT, or docker compose ps / read the NocoDB logs

Still stuck? docs/TROUBLESHOOTING.md has the full decision tree, including how to tell "not fetching" from "notifying" failures.

---

FAQ

Does this scrape or hammer Carousell? It performs one plain HTTP GET per search per interval (default every 5 minutes), with a 1-second gap between searches inside a tick, and it de-duplicates everything. It reads the same public search page your browser reads. Be a good citizen: do not set 30-second intervals on 40 searches.

Can I get alerts for my own listings to test? Yes — post something (or edit an existing listing's title/price: an edit sometimes re-surfaces it), or simply untick notified on a row to replay an alert.

Can I watch a category page or a seller's page instead of a search? Any Carousell page that renders the same listing grid works. Search URLs are what is tested.

Multiple people / multiple searches? Everything is one monitor loop, one NocoDB base, one Telegram chat id. Add as many Settings rows as you like. For a second Telegram destination, run a second stack with its own bot and base.

Do I need a reverse proxy / HTTPS? No — the monitor has no inbound port. If you want the NocoDB UI reachable from outside your LAN, put it behind your reverse proxy of choice (Synology/nginx/Traefik/Caddy) and keep the token out of the URL.

Does it work outside Malaysia? It is written against carousell.com.my (Malaysia); the .my endpoints and the RM price format are baked in. Other Carousell country sites use the same page structure — change the two URL templates in monitor.py (PRODUCT_URL_TMPL, SELLER_URL_TMPL) and the search URL you paste into Settings.

How do I upgrade NocoDB? Change the image tag in docker-compose.allinone.yml, docker compose -f docker-compose.allinone.yml up -d, and let it migrate. Back up nocodb-data first. The monitor only supports the meta API of the versions pinned in that file.

Tests?

python test_pagination.py    # stdlib only, no network; exit 0 = pass

---

Project layout

monitor.py                     the whole monitor (stdlib only): bootstrap, fetch, NocoDB IO, Telegram
healthcheck.py                 Docker HEALTHCHECK probe (reads /data/health.json)
Dockerfile                     python:3.11-alpine, no dependencies, ~60 MB
docker-compose.allinone.yml    NocoDB + monitor — start here
docker-compose.yml             monitor only, joining an existing NocoDB on a shared network
.env.example                   every setting, explained
test_pagination.py             stdlib regression tests (run: python test_pagination.py)
docs/                          the long-form guides
AGENTS.md                      notes for AI coding agents working on this repo
SECRETS.example.md             credential template (the real values stay out of git)

Guides

Doc	Read it when
docs/QUICKSTART-PORTAINER.md	you want the click-by-click Portainer version
docs/TELEGRAM-SETUP.md	you need the bot token or a group chat id
docs/NOCODB-SETUP.md	you want to understand the tables before you fill them
docs/ARCHITECTURE.md	you want to modify the code, or understand the compose file line by line
docs/OPERATIONS.md	something is wrong, or it is time to back up / upgrade
docs/TROUBLESHOOTING.md	you have a symptom and want the decision tree

---

Contributing

Issues and pull requests are welcome. Two house rules before you send a patch:

1. Open an issue first for anything beyond a typo — this is a small, opinionated tool and the maintainer would rather agree on the shape before you write it.
2. Do not break the tests. python test_pagination.py must pass, and new behaviour wants a check added to it.

Please never commit credentials, host names or personal URLs. The repo intentionally ships no default base id, token or chat id.

Credits

Written and maintained by Lee Teong Hoe (Mr Hoelee) — Hoelee Enterprise, Malaysia. Built on the shoulders of NocoDB and the Telegram Bot API.

License

MIT — do what you want, no warranty.