Two problems found while verifying the pagination fix on the live stack:
1. fetch_listings() did state['SearchListing']['listingCards'] with no guard.
Carousell serves HTTP 200 with listingCards=null under soft rate limiting,
so the tick died with TypeError: 'NoneType' object is not iterable ->
ok:false -> container unhealthy, repeatedly.
2. run_tick() set ok = (no failures at all), so a single soft-blocked watch
out of 7 marked the whole monitor failed. That flaps on transient blocks
and (now that failures alert) would spam Telegram.
- fetch_listings: explicit null check -> clear retryable RuntimeError
- add FAILURE_RATIO_THRESHOLD (default 1.0 = all watches must fail); partial
failures are reported as '[partial n/N] ...' without failing the tick
- health gains failed_watches
- wire the knob into compose/.env.example/DOCUMENTATION/COMPOSE-SETUP
- 15 new checks: null vs empty cards, real card still parses, threshold edges
The previous commit renamed the record list to 'rows' but missed the loop in
load_ignored_keywords, leaving a bare 'j' -> 'tick error: name j is not
defined' every tick. The loop died, health.json went ok:false, the container
reported unhealthy, and alert_on_health fired against a live failure.
- fix the stale reference
- use real emoji instead of literal backslash escapes in alert text
- test: exercise the real loaders end-to-end against a fake NocoDB so a
missed rename is caught (mutation-verified: reintroducing the bug fails 3
checks)
send_pending_notifications() read Listings with ?limit=1000 and no paging.
Once the table passed 1000 rows the newest records (highest Id, at the tail)
fell outside page 1, so they were never sent and never marked notified ->
notifications silently dead while health.json stayed ok:true. Found live
2026-09-22: Id 1007-1041 (35 rows, ~29h of listings) never alerted.
- add nc_list_all(): offset-paged full-table read
- use it for listings pending, seen, watches, settings, both ignore lists
- add alert_on_health(): Telegram failure alert debounced over
ERROR_ALERT_AFTER consecutive failed ticks, plus a recovery notice
(container already reported unhealthy via healthcheck.py on ok:false)
- new env knob ERROR_ALERT_AFTER wired into compose/.env.example/docs
- test_pagination.py: 30 checks incl. the page-2 regression and alert edges
hoelee