Initial version
This commit is contained in:
@@ -0,0 +1,254 @@
|
||||
# Traefik Bot Filter
|
||||
|
||||
`traefik-botfilter` is a dependency-free Traefik middleware plugin for public
|
||||
sites that are receiving scanners or unsophisticated HTTP floods. It blocks
|
||||
known scan paths before they reach the upstream and keeps a bounded, local
|
||||
per-client score cache for temporary bans.
|
||||
|
||||
It is deliberately a **request filter**, not a bot-management service. A
|
||||
distributed attacker that sends valid browser-like requests from many IPs
|
||||
needs an upstream CDN/WAF or firewall as well.
|
||||
|
||||
## What it does
|
||||
|
||||
- Immediately rejects and temporarily bans configured scanner paths, file
|
||||
extensions, and User-Agent tokens.
|
||||
- Optionally requires `User-Agent`, `Accept`, and `Host`. A failed required
|
||||
check is immediately cached as a temporary ban, so repeated requests do not
|
||||
reach Kiwix.
|
||||
- Adds scores in a configurable sliding window. Defaults implement the stated
|
||||
model: empty UA `+40`, missing Accept `+20`, bad path `+50`, blocked UA
|
||||
`+80`, first request directly to `/content/` `+15`, and an upstream `404`
|
||||
`+40`.
|
||||
- Bans when the score reaches `scoreThreshold` (default `100`). A blocked scan
|
||||
path always bans immediately, independently of the score threshold.
|
||||
- Applies conservative browser plausibility checks. It detects internally
|
||||
inconsistent browser User-Agents, but does not claim to distinguish every
|
||||
automation client from a real browser. That requires a JavaScript challenge
|
||||
at a CDN/WAF layer.
|
||||
- Bounds memory with `maxTrackedIPs` (default `50,000`) and
|
||||
`maxScoreEventsPerIP` (default `16`), with lock-safe access and no cleanup
|
||||
goroutine.
|
||||
- Never trusts `X-Forwarded-For` unless `clientIPHeader` is explicitly set
|
||||
**and** the TCP peer is in `trustedProxyCIDRs`.
|
||||
|
||||
## Important fixes before enabling it
|
||||
|
||||
Your Kiwix service currently publishes `6001:8080`. Requests to that port go
|
||||
straight to Kiwix and bypass this middleware, rate limiting, and Traefik
|
||||
access controls. Remove this line unless you need a separate protected private
|
||||
listener:
|
||||
|
||||
```yaml
|
||||
kiwix:
|
||||
# ports:
|
||||
# - "6001:8080"
|
||||
```
|
||||
|
||||
Also narrow the current error middleware. `400-599` makes every ordinary
|
||||
Kiwix 404 and upstream 502 call `host.docker.internal:44440`, which adds work
|
||||
and obscures the original failure. Keep it only for rate-limit responses:
|
||||
|
||||
```yaml
|
||||
cp-ratelimit-errorpages:
|
||||
errors:
|
||||
status:
|
||||
- "429-429"
|
||||
service: srv-error
|
||||
query: "/{status}.html"
|
||||
```
|
||||
|
||||
The access log supplied with this request contains 57,141 requests with an
|
||||
empty User-Agent (`"-"`), including thousands of `404` and `502` responses.
|
||||
With `requireUserAgent: true`, those requests are rejected at Traefik before
|
||||
they consume Kiwix CPU.
|
||||
|
||||
## Configuration
|
||||
|
||||
Add the following to the dynamic file provider configuration. The middleware
|
||||
must be attached to each public Kiwix router.
|
||||
|
||||
```yaml
|
||||
http:
|
||||
middlewares:
|
||||
botfilter:
|
||||
plugin:
|
||||
botfilter:
|
||||
statusCode: 403
|
||||
|
||||
requireUserAgent: true
|
||||
requireAccept: true
|
||||
requireHost: true
|
||||
browserValidation: true
|
||||
|
||||
whitelistCIDRs:
|
||||
- 192.168.0.0/16
|
||||
- 10.0.0.0/8
|
||||
|
||||
temporaryBanMinutes: 15
|
||||
scoreThreshold: 100
|
||||
scoreWindowMinutes: 15
|
||||
maxTrackedIPs: 50000
|
||||
maxScoreEventsPerIP: 16
|
||||
|
||||
blockedUserAgents:
|
||||
- curl
|
||||
- wget
|
||||
- python
|
||||
- Go-http-client
|
||||
- masscan
|
||||
- sqlmap
|
||||
- zgrab
|
||||
- nikto
|
||||
|
||||
blockedPaths:
|
||||
- /.env
|
||||
- /.git
|
||||
- /wp-login.php
|
||||
- /xmlrpc.php
|
||||
- /phpmyadmin
|
||||
|
||||
blockedExtensions:
|
||||
- .env
|
||||
- .bak
|
||||
- .zip
|
||||
|
||||
# This weak signal contributes only 15 points. Direct links to a
|
||||
# Kiwix article remain possible; they are not banned by themselves.
|
||||
randomArticlePatterns:
|
||||
- /content/
|
||||
|
||||
# Leave both unset when Traefik accepts traffic directly. If a CDN
|
||||
# or load balancer is in front, configure its exact source CIDRs and
|
||||
# ensure it overwrites this header.
|
||||
# clientIPHeader: X-Forwarded-For
|
||||
# trustedProxyCIDRs:
|
||||
# - 203.0.113.0/24
|
||||
|
||||
# Do not turn this on during an attack unless short diagnostics are
|
||||
# needed: per-request disk logging can itself become expensive.
|
||||
logBlockedRequests: false
|
||||
```
|
||||
|
||||
Apply it before `cp-ratelimit` so rejected requests do not consume the
|
||||
rate-limiter's work. Keep the narrowed errors middleware as the outer wrapper
|
||||
for the rate limiter:
|
||||
|
||||
```yaml
|
||||
http:
|
||||
routers:
|
||||
rtr-default-wiki:
|
||||
# ... existing rule/service fields ...
|
||||
middlewares:
|
||||
- cp-ratelimit-errorpages
|
||||
- botfilter
|
||||
- cp-ratelimit
|
||||
```
|
||||
|
||||
Use the same middleware list on `rtr-wiki` and `rtr-yes` if they expose Kiwix.
|
||||
|
||||
## Install in Traefik
|
||||
|
||||
### Production: remote plugin
|
||||
|
||||
The module name in `.traefik.yml` is intentionally the one requested here.
|
||||
Create the public repository `github.com/hoelee/traefik-botfilter`, push this
|
||||
directory, and create the immutable Git tag `v0.1.0`. Then put this in the
|
||||
**static** Traefik configuration (`traefik.yml`), not `dynamic.yml`:
|
||||
|
||||
```yaml
|
||||
experimental:
|
||||
plugins:
|
||||
botfilter:
|
||||
moduleName: github.com/hoelee/traefik-botfilter
|
||||
version: v0.1.0
|
||||
```
|
||||
|
||||
Restart Traefik after changing static plugin configuration. Do not retag an
|
||||
existing version; publish `v0.1.1` for later changes.
|
||||
|
||||
### Test locally first
|
||||
|
||||
Traefik local plugins need the Go module at the exact module path below
|
||||
`/plugins-local/src`. On the Synology host, copy or clone this repository to:
|
||||
|
||||
```text
|
||||
/volume1/docker/traefik-plugins/src/github.com/hoelee/traefik-botfilter
|
||||
```
|
||||
|
||||
Add this readonly mount to the Traefik service:
|
||||
|
||||
```yaml
|
||||
volumes:
|
||||
- /volume1/docker/traefik-plugins:/plugins-local:ro
|
||||
```
|
||||
|
||||
And use this static configuration instead of the remote `plugins` block:
|
||||
|
||||
```yaml
|
||||
experimental:
|
||||
localPlugins:
|
||||
botfilter:
|
||||
moduleName: github.com/hoelee/traefik-botfilter
|
||||
```
|
||||
|
||||
Restart Traefik and confirm its startup log says that the `botfilter` plugin
|
||||
loaded before exposing the public router.
|
||||
|
||||
## Option reference
|
||||
|
||||
| Option | Default | Meaning |
|
||||
| --- | ---: | --- |
|
||||
| `statusCode` | `403` | HTTP status for rejected requests (`400`–`599`). |
|
||||
| `temporaryBanMinutes` | `15` | In-memory ban duration. |
|
||||
| `scoreThreshold` | `100` | Score at which a client is banned. |
|
||||
| `scoreWindowMinutes` | `15` | Sliding window for score events. |
|
||||
| `maxTrackedIPs` | `50000` | Hard maximum size of the per-IP state map. |
|
||||
| `maxScoreEventsPerIP` | `16` | Bound on score events retained per client. |
|
||||
| `requireUserAgent` | `false` | Immediately ban missing User-Agent requests. |
|
||||
| `requireAccept` | `false` | Immediately ban missing Accept requests. |
|
||||
| `requireHost` | `false` | Immediately ban missing Host requests. |
|
||||
| `browserValidation` | `false` | Score implausible Mozilla-family header combinations. |
|
||||
| `whitelistCIDRs` | none | Clients that bypass all checks and cache updates. |
|
||||
| `clientIPHeader` | empty | Optional header used only from `trustedProxyCIDRs`. |
|
||||
| `trustedProxyCIDRs` | none | TCP peer ranges allowed to supply the client-IP header. |
|
||||
| `randomArticlePatterns` | `/content/` | First-request path prefixes that add `randomArticleScore`. |
|
||||
| `logBlockedRequests` | `false` | Opt-in rejected-request logging. |
|
||||
|
||||
All listed score fields are configurable: `emptyUserAgentScore`,
|
||||
`missingAcceptScore`, `blockedUserAgentScore`, `badPathScore`,
|
||||
`randomArticleScore`, `notFoundScore`, and `fakeBrowserScore`. Set a score to
|
||||
`0` to disable that one signal; scanner-path and required-header rejections
|
||||
still ban immediately.
|
||||
|
||||
## Operational limits and recommended defences
|
||||
|
||||
This plugin protects Kiwix from the malformed-header/scanner pattern in the
|
||||
log, but it cannot stop a botnet that rotates IPs and perfectly imitates
|
||||
browser headers. For that case:
|
||||
|
||||
1. Put the hostname behind a CDN/WAF with bot challenge and request-rate rules.
|
||||
2. Firewall the NAS so public clients cannot reach Kiwix's port `6001` or any
|
||||
Traefik entry point other than the intended public port.
|
||||
3. Do not expose the NAS origin address in DNS or other services; otherwise
|
||||
attackers can bypass the CDN.
|
||||
4. Keep Traefik access logs sampled or rotate them quickly during an incident.
|
||||
Per-request synchronous disk logging becomes material at flood volume.
|
||||
5. `deploy.resources` is commonly ignored by non-Swarm Docker Compose. Verify
|
||||
resource limits with `docker inspect` on the NAS rather than assuming the
|
||||
`deploy` block limits CPU.
|
||||
|
||||
The cache is intentionally local to a Traefik process and is reset on
|
||||
container restart. That is appropriate for a low-overhead edge filter; use a
|
||||
CDN/WAF or shared store if bans must survive restarts or be shared by multiple
|
||||
Traefik replicas.
|
||||
|
||||
## Development
|
||||
|
||||
```text
|
||||
go test ./...
|
||||
go vet ./...
|
||||
```
|
||||
|
||||
The implementation only uses the Go standard library, which reduces plugin
|
||||
startup and supply-chain risk.
|
||||
Reference in New Issue
Block a user