255 lines
8.8 KiB
Markdown
255 lines
8.8 KiB
Markdown
# Traefik Bot Filter
|
||
|
||
`traefik-botfilter` is a dependency-free Traefik middleware plugin for public
|
||
sites that are receiving scanners or unsophisticated HTTP floods. It blocks
|
||
known scan paths before they reach the upstream and keeps a bounded, local
|
||
per-client score cache for temporary bans.
|
||
|
||
It is deliberately a **request filter**, not a bot-management service. A
|
||
distributed attacker that sends valid browser-like requests from many IPs
|
||
needs an upstream CDN/WAF or firewall as well.
|
||
|
||
## What it does
|
||
|
||
- Immediately rejects and temporarily bans configured scanner paths, file
|
||
extensions, and User-Agent tokens.
|
||
- Optionally requires `User-Agent`, `Accept`, and `Host`. A failed required
|
||
check is immediately cached as a temporary ban, so repeated requests do not
|
||
reach Kiwix.
|
||
- Adds scores in a configurable sliding window. Defaults implement the stated
|
||
model: empty UA `+40`, missing Accept `+20`, bad path `+50`, blocked UA
|
||
`+80`, first request directly to `/content/` `+15`, and an upstream `404`
|
||
`+40`.
|
||
- Bans when the score reaches `scoreThreshold` (default `100`). A blocked scan
|
||
path always bans immediately, independently of the score threshold.
|
||
- Applies conservative browser plausibility checks. It detects internally
|
||
inconsistent browser User-Agents, but does not claim to distinguish every
|
||
automation client from a real browser. That requires a JavaScript challenge
|
||
at a CDN/WAF layer.
|
||
- Bounds memory with `maxTrackedIPs` (default `50,000`) and
|
||
`maxScoreEventsPerIP` (default `16`), with lock-safe access and no cleanup
|
||
goroutine.
|
||
- Never trusts `X-Forwarded-For` unless `clientIPHeader` is explicitly set
|
||
**and** the TCP peer is in `trustedProxyCIDRs`.
|
||
|
||
## Important fixes before enabling it
|
||
|
||
Your Kiwix service currently publishes `6001:8080`. Requests to that port go
|
||
straight to Kiwix and bypass this middleware, rate limiting, and Traefik
|
||
access controls. Remove this line unless you need a separate protected private
|
||
listener:
|
||
|
||
```yaml
|
||
kiwix:
|
||
# ports:
|
||
# - "6001:8080"
|
||
```
|
||
|
||
Also narrow the current error middleware. `400-599` makes every ordinary
|
||
Kiwix 404 and upstream 502 call `host.docker.internal:44440`, which adds work
|
||
and obscures the original failure. Keep it only for rate-limit responses:
|
||
|
||
```yaml
|
||
cp-ratelimit-errorpages:
|
||
errors:
|
||
status:
|
||
- "429-429"
|
||
service: srv-error
|
||
query: "/{status}.html"
|
||
```
|
||
|
||
The access log supplied with this request contains 57,141 requests with an
|
||
empty User-Agent (`"-"`), including thousands of `404` and `502` responses.
|
||
With `requireUserAgent: true`, those requests are rejected at Traefik before
|
||
they consume Kiwix CPU.
|
||
|
||
## Configuration
|
||
|
||
Add the following to the dynamic file provider configuration. The middleware
|
||
must be attached to each public Kiwix router.
|
||
|
||
```yaml
|
||
http:
|
||
middlewares:
|
||
botfilter:
|
||
plugin:
|
||
botfilter:
|
||
statusCode: 403
|
||
|
||
requireUserAgent: true
|
||
requireAccept: true
|
||
requireHost: true
|
||
browserValidation: true
|
||
|
||
whitelistCIDRs:
|
||
- 192.168.0.0/16
|
||
- 10.0.0.0/8
|
||
|
||
temporaryBanMinutes: 15
|
||
scoreThreshold: 100
|
||
scoreWindowMinutes: 15
|
||
maxTrackedIPs: 50000
|
||
maxScoreEventsPerIP: 16
|
||
|
||
blockedUserAgents:
|
||
- curl
|
||
- wget
|
||
- python
|
||
- Go-http-client
|
||
- masscan
|
||
- sqlmap
|
||
- zgrab
|
||
- nikto
|
||
|
||
blockedPaths:
|
||
- /.env
|
||
- /.git
|
||
- /wp-login.php
|
||
- /xmlrpc.php
|
||
- /phpmyadmin
|
||
|
||
blockedExtensions:
|
||
- .env
|
||
- .bak
|
||
- .zip
|
||
|
||
# This weak signal contributes only 15 points. Direct links to a
|
||
# Kiwix article remain possible; they are not banned by themselves.
|
||
randomArticlePatterns:
|
||
- /content/
|
||
|
||
# Leave both unset when Traefik accepts traffic directly. If a CDN
|
||
# or load balancer is in front, configure its exact source CIDRs and
|
||
# ensure it overwrites this header.
|
||
# clientIPHeader: X-Forwarded-For
|
||
# trustedProxyCIDRs:
|
||
# - 203.0.113.0/24
|
||
|
||
# Do not turn this on during an attack unless short diagnostics are
|
||
# needed: per-request disk logging can itself become expensive.
|
||
logBlockedRequests: false
|
||
```
|
||
|
||
Apply it before `cp-ratelimit` so rejected requests do not consume the
|
||
rate-limiter's work. Keep the narrowed errors middleware as the outer wrapper
|
||
for the rate limiter:
|
||
|
||
```yaml
|
||
http:
|
||
routers:
|
||
rtr-default-wiki:
|
||
# ... existing rule/service fields ...
|
||
middlewares:
|
||
- cp-ratelimit-errorpages
|
||
- botfilter
|
||
- cp-ratelimit
|
||
```
|
||
|
||
Use the same middleware list on `rtr-wiki` and `rtr-yes` if they expose Kiwix.
|
||
|
||
## Install in Traefik
|
||
|
||
### Production: remote plugin
|
||
|
||
The module name in `.traefik.yml` is intentionally the one requested here.
|
||
Create the public repository `github.com/hoelee/traefik-botfilter`, push this
|
||
directory, and create the immutable Git tag `v0.1.0`. Then put this in the
|
||
**static** Traefik configuration (`traefik.yml`), not `dynamic.yml`:
|
||
|
||
```yaml
|
||
experimental:
|
||
plugins:
|
||
botfilter:
|
||
moduleName: github.com/hoelee/traefik-botfilter
|
||
version: v0.1.0
|
||
```
|
||
|
||
Restart Traefik after changing static plugin configuration. Do not retag an
|
||
existing version; publish `v0.1.1` for later changes.
|
||
|
||
### Test locally first
|
||
|
||
Traefik local plugins need the Go module at the exact module path below
|
||
`/plugins-local/src`. On the Synology host, copy or clone this repository to:
|
||
|
||
```text
|
||
/volume1/docker/traefik-plugins/src/github.com/hoelee/traefik-botfilter
|
||
```
|
||
|
||
Add this readonly mount to the Traefik service:
|
||
|
||
```yaml
|
||
volumes:
|
||
- /volume1/docker/traefik-plugins:/plugins-local:ro
|
||
```
|
||
|
||
And use this static configuration instead of the remote `plugins` block:
|
||
|
||
```yaml
|
||
experimental:
|
||
localPlugins:
|
||
botfilter:
|
||
moduleName: github.com/hoelee/traefik-botfilter
|
||
```
|
||
|
||
Restart Traefik and confirm its startup log says that the `botfilter` plugin
|
||
loaded before exposing the public router.
|
||
|
||
## Option reference
|
||
|
||
| Option | Default | Meaning |
|
||
| --- | ---: | --- |
|
||
| `statusCode` | `403` | HTTP status for rejected requests (`400`–`599`). |
|
||
| `temporaryBanMinutes` | `15` | In-memory ban duration. |
|
||
| `scoreThreshold` | `100` | Score at which a client is banned. |
|
||
| `scoreWindowMinutes` | `15` | Sliding window for score events. |
|
||
| `maxTrackedIPs` | `50000` | Hard maximum size of the per-IP state map. |
|
||
| `maxScoreEventsPerIP` | `16` | Bound on score events retained per client. |
|
||
| `requireUserAgent` | `false` | Immediately ban missing User-Agent requests. |
|
||
| `requireAccept` | `false` | Immediately ban missing Accept requests. |
|
||
| `requireHost` | `false` | Immediately ban missing Host requests. |
|
||
| `browserValidation` | `false` | Score implausible Mozilla-family header combinations. |
|
||
| `whitelistCIDRs` | none | Clients that bypass all checks and cache updates. |
|
||
| `clientIPHeader` | empty | Optional header used only from `trustedProxyCIDRs`. |
|
||
| `trustedProxyCIDRs` | none | TCP peer ranges allowed to supply the client-IP header. |
|
||
| `randomArticlePatterns` | `/content/` | First-request path prefixes that add `randomArticleScore`. |
|
||
| `logBlockedRequests` | `false` | Opt-in rejected-request logging. |
|
||
|
||
All listed score fields are configurable: `emptyUserAgentScore`,
|
||
`missingAcceptScore`, `blockedUserAgentScore`, `badPathScore`,
|
||
`randomArticleScore`, `notFoundScore`, and `fakeBrowserScore`. Set a score to
|
||
`0` to disable that one signal; scanner-path and required-header rejections
|
||
still ban immediately.
|
||
|
||
## Operational limits and recommended defences
|
||
|
||
This plugin protects Kiwix from the malformed-header/scanner pattern in the
|
||
log, but it cannot stop a botnet that rotates IPs and perfectly imitates
|
||
browser headers. For that case:
|
||
|
||
1. Put the hostname behind a CDN/WAF with bot challenge and request-rate rules.
|
||
2. Firewall the NAS so public clients cannot reach Kiwix's port `6001` or any
|
||
Traefik entry point other than the intended public port.
|
||
3. Do not expose the NAS origin address in DNS or other services; otherwise
|
||
attackers can bypass the CDN.
|
||
4. Keep Traefik access logs sampled or rotate them quickly during an incident.
|
||
Per-request synchronous disk logging becomes material at flood volume.
|
||
5. `deploy.resources` is commonly ignored by non-Swarm Docker Compose. Verify
|
||
resource limits with `docker inspect` on the NAS rather than assuming the
|
||
`deploy` block limits CPU.
|
||
|
||
The cache is intentionally local to a Traefik process and is reset on
|
||
container restart. That is appropriate for a low-overhead edge filter; use a
|
||
CDN/WAF or shared store if bans must survive restarts or be shared by multiple
|
||
Traefik replicas.
|
||
|
||
## Development
|
||
|
||
```text
|
||
go test ./...
|
||
go vet ./...
|
||
```
|
||
|
||
The implementation only uses the Go standard library, which reduces plugin
|
||
startup and supply-chain risk.
|