Harden against Carousell soft-blocks; stop one flaky watch failing the tick

Two problems found while verifying the pagination fix on the live stack:

1. fetch_listings() did state['SearchListing']['listingCards'] with no guard.
   Carousell serves HTTP 200 with listingCards=null under soft rate limiting,
   so the tick died with TypeError: 'NoneType' object is not iterable ->
   ok:false -> container unhealthy, repeatedly.
2. run_tick() set ok = (no failures at all), so a single soft-blocked watch
   out of 7 marked the whole monitor failed. That flaps on transient blocks
   and (now that failures alert) would spam Telegram.

- fetch_listings: explicit null check -> clear retryable RuntimeError
- add FAILURE_RATIO_THRESHOLD (default 1.0 = all watches must fail); partial
  failures are reported as '[partial n/N] ...' without failing the tick
- health gains failed_watches
- wire the knob into compose/.env.example/DOCUMENTATION/COMPOSE-SETUP
- 15 new checks: null vs empty cards, real card still parses, threshold edges

This commit is contained in:

hoelee

2026-09-22 03:52:14 +08:00

parent b88f6d0963

commit f99a0d878e

6 changed files with 122 additions and 4 deletions

									
										.env.example
									
		+2
		
												View File
												
				@@ -15,3 +15,5 @@ FETCH_GAP_SECONDS=1

				HEALTH_STALE_SECONDS=600

				# 连续失败几次才发 Telegram 故障告警（去抖）:

				ERROR_ALERT_AFTER=3

				# 失败 watch 占比达到该值才判定整轮故障（1.0 = 全部失败）:

				FAILURE_RATIO_THRESHOLD=1.0