traefik-botfilter/README.md

Traefik Bot Filter

traefik-botfilter is a dependency-free Traefik middleware plugin for public sites that are receiving scanners or unsophisticated HTTP floods. It blocks known scan paths before they reach the upstream and keeps a bounded, local per-client score cache for temporary bans.

It is deliberately a request filter, not a bot-management service. A distributed attacker that sends valid browser-like requests from many IPs needs an upstream CDN/WAF or firewall as well.

What it does

• Immediately rejects and temporarily bans configured scanner paths, file extensions, and User-Agent tokens.
• Optionally requires User-Agent, Accept, and Host. A failed required check is immediately cached as a temporary ban, so repeated requests do not reach Kiwix.
• Adds scores in a configurable sliding window. Defaults implement the stated model: empty UA +40, missing Accept +20, bad path +50, blocked UA +80, first request directly to /content/ +15, and an upstream 404 +40.
• Bans when the score reaches scoreThreshold (default 100). A blocked scan path always bans immediately, independently of the score threshold.
• Applies conservative browser plausibility checks. It detects internally inconsistent browser User-Agents, but does not claim to distinguish every automation client from a real browser. That requires a JavaScript challenge at a CDN/WAF layer.
• Bounds memory with maxTrackedIPs (default 50,000) and maxScoreEventsPerIP (default 16), with lock-safe access and no cleanup goroutine.
• Never trusts X-Forwarded-For unless clientIPHeader is explicitly set and the TCP peer is in trustedProxyCIDRs.

Important fixes before enabling it

Your Kiwix service currently publishes 6001:8080. Requests to that port go straight to Kiwix and bypass this middleware, rate limiting, and Traefik access controls. Remove this line unless you need a separate protected private listener:

kiwix:
  # ports:
  #   - "6001:8080"

Also narrow the current error middleware. 400-599 makes every ordinary Kiwix 404 and upstream 502 call host.docker.internal:44440, which adds work and obscures the original failure. Keep it only for rate-limit responses:

cp-ratelimit-errorpages:
  errors:
    status:
      - "429-429"
    service: srv-error
    query: "/{status}.html"

The access log supplied with this request contains 57,141 requests with an empty User-Agent ("-"), including thousands of 404 and 502 responses. With requireUserAgent: true, those requests are rejected at Traefik before they consume Kiwix CPU.

Configuration

Add the following to the dynamic file provider configuration. The middleware must be attached to each public Kiwix router.

http:
  middlewares:
    botfilter:
      plugin:
        botfilter:
          statusCode: 403

          requireUserAgent: true
          requireAccept: true
          requireHost: true
          browserValidation: true

          whitelistCIDRs:
            - 192.168.0.0/16
            - 10.0.0.0/8

          temporaryBanMinutes: 15
          scoreThreshold: 100
          scoreWindowMinutes: 15
          maxTrackedIPs: 50000
          maxScoreEventsPerIP: 16

          blockedUserAgents:
            - curl
            - wget
            - python
            - Go-http-client
            - masscan
            - sqlmap
            - zgrab
            - nikto

          blockedPaths:
            - /.env
            - /.git
            - /wp-login.php
            - /xmlrpc.php
            - /phpmyadmin

          blockedExtensions:
            - .env
            - .bak
            - .zip

          # This weak signal contributes only 15 points. Direct links to a
          # Kiwix article remain possible; they are not banned by themselves.
          randomArticlePatterns:
            - /content/

          # Leave both unset when Traefik accepts traffic directly. If a CDN
          # or load balancer is in front, configure its exact source CIDRs and
          # ensure it overwrites this header.
          # clientIPHeader: X-Forwarded-For
          # trustedProxyCIDRs:
          #   - 203.0.113.0/24

          # Do not turn this on during an attack unless short diagnostics are
          # needed: per-request disk logging can itself become expensive.
          logBlockedRequests: false

Apply it before cp-ratelimit so rejected requests do not consume the rate-limiter's work. Keep the narrowed errors middleware as the outer wrapper for the rate limiter:

http:
  routers:
    rtr-default-wiki:
      # ... existing rule/service fields ...
      middlewares:
        - cp-ratelimit-errorpages
        - botfilter
        - cp-ratelimit

Use the same middleware list on rtr-wiki and rtr-yes if they expose Kiwix.

Install in Traefik

Production: remote plugin

The module name in .traefik.yml is intentionally the one requested here. Create the public repository github.com/hoelee/traefik-botfilter, push this directory, and create the immutable Git tag v0.1.0. Then put this in the static Traefik configuration (traefik.yml), not dynamic.yml:

experimental:
  plugins:
    botfilter:
      moduleName: github.com/hoelee/traefik-botfilter
      version: v0.1.0

Restart Traefik after changing static plugin configuration. Do not retag an existing version; publish v0.1.1 for later changes.

Test locally first

Traefik local plugins need the Go module at the exact module path below /plugins-local/src. On the Synology host, copy or clone this repository to:

/volume1/docker/traefik-plugins/src/github.com/hoelee/traefik-botfilter

Add this readonly mount to the Traefik service:

volumes:
  - /volume1/docker/traefik-plugins:/plugins-local:ro

And use this static configuration instead of the remote plugins block:

experimental:
  localPlugins:
    botfilter:
      moduleName: github.com/hoelee/traefik-botfilter

Restart Traefik and confirm its startup log says that the botfilter plugin loaded before exposing the public router.

Option reference

Option	Default	Meaning
statusCode	403	HTTP status for rejected requests (400–599).
temporaryBanMinutes	15	In-memory ban duration.
scoreThreshold	100	Score at which a client is banned.
scoreWindowMinutes	15	Sliding window for score events.
maxTrackedIPs	50000	Hard maximum size of the per-IP state map.
maxScoreEventsPerIP	16	Bound on score events retained per client.
requireUserAgent	false	Immediately ban missing User-Agent requests.
requireAccept	false	Immediately ban missing Accept requests.
requireHost	false	Immediately ban missing Host requests.
browserValidation	false	Score implausible Mozilla-family header combinations.
whitelistCIDRs	none	Clients that bypass all checks and cache updates.
clientIPHeader	empty	Optional header used only from trustedProxyCIDRs.
trustedProxyCIDRs	none	TCP peer ranges allowed to supply the client-IP header.
randomArticlePatterns	/content/	First-request path prefixes that add randomArticleScore.
logBlockedRequests	false	Opt-in rejected-request logging.

All listed score fields are configurable: emptyUserAgentScore, missingAcceptScore, blockedUserAgentScore, badPathScore, randomArticleScore, notFoundScore, and fakeBrowserScore. Set a score to 0 to disable that one signal; scanner-path and required-header rejections still ban immediately.

Operational limits and recommended defences

This plugin protects Kiwix from the malformed-header/scanner pattern in the log, but it cannot stop a botnet that rotates IPs and perfectly imitates browser headers. For that case:

1. Put the hostname behind a CDN/WAF with bot challenge and request-rate rules.
2. Firewall the NAS so public clients cannot reach Kiwix's port 6001 or any Traefik entry point other than the intended public port.
3. Do not expose the NAS origin address in DNS or other services; otherwise attackers can bypass the CDN.
4. Keep Traefik access logs sampled or rotate them quickly during an incident. Per-request synchronous disk logging becomes material at flood volume.
5. deploy.resources is commonly ignored by non-Swarm Docker Compose. Verify resource limits with docker inspect on the NAS rather than assuming the deploy block limits CPU.

The cache is intentionally local to a Traefik process and is reset on container restart. That is appropriate for a low-overhead edge filter; use a CDN/WAF or shared store if bans must survive restarts or be shared by multiple Traefik replicas.

Development

go test ./...
go vet ./...

The implementation only uses the Go standard library, which reduces plugin startup and supply-chain risk.