It is deliberately a **request filter**, not a bot-management service. A
distributed attacker that sends valid browser-like requests from many IPs
needs an upstream CDN/WAF or firewall as well.
`traefik-botfilter` is a dependency-free Traefik middleware plugin that rejects common scanner requests and temporarily bans suspicious clients with bounded, in-memory per-IP scoring. It is a lightweight first layer of protection, not a replacement for a WAF.
## What it does
## Features
-Immediately rejects and temporarily bans configured scanner paths, file
extensions, and User-Agent tokens.
-Optionally requires `User-Agent`, `Accept`, and `Host`. A failed required
check is immediately cached as a temporary ban, so repeated requests do not
reach Kiwix.
-Adds scores in a configurable sliding window. Defaults implement the stated
model: empty UA `+40`, missing Accept `+20`, bad path `+50`, blocked UA
`+80`, first request directly to `/content/``+15`, and an upstream `404`
`+40`.
- Bans when the score reaches `scoreThreshold` (default `100`). A blocked scan
path always bans immediately, independently of the score threshold.
- Applies conservative browser plausibility checks. It detects internally
inconsistent browser User-Agents, but does not claim to distinguish every
automation client from a real browser. That requires a JavaScript challenge
at a CDN/WAF layer.
- Bounds memory with `maxTrackedIPs` (default `50,000`) and
`maxScoreEventsPerIP` (default `16`), with lock-safe access and no cleanup
goroutine.
- Never trusts `X-Forwarded-For` unless `clientIPHeader` is explicitly set
**and** the TCP peer is in `trustedProxyCIDRs`.
-Blocks configured paths, extensions, and User-Agent substrings before the request reaches the backend.
- Optionally requires `User-Agent`, `Accept`, and `Host` headers.
- Scores suspicious requests and backend `404` responses, then applies a temporary ban.
- Supports CIDR allowlists and trusted proxy client-IP headers.
-Uses only the Go standard library and keeps its per-client cache bounded.
## Important fixes before enabling it
## Install From The Plugin Catalog
Your Kiwix service currently publishes `6001:8080`. Requests to that port go
straight to Kiwix and bypass this middleware, rate limiting, and Traefik
access controls. Remove this line unless you need a separate protected private
listener:
After a release tag has been accepted by the catalog, declare the plugin in Traefik's static configuration:
```yaml
kiwix:
# ports:
# - "6001:8080"
experimental:
plugins:
botfilter:
moduleName:github.com/hoelee/traefik-botfilter
version:v0.1.1
```
Also narrow the current error middleware. `400-599` makes every ordinary
Kiwix 404 and upstream 502 call `host.docker.internal:44440`, which adds work
and obscures the original failure. Keep it only for rate-limit responses:
```yaml
cp-ratelimit-errorpages:
errors:
status:
- "429-429"
service:srv-error
query:"/{status}.html"
```
The access log supplied with this request contains 57,141 requests with an
empty User-Agent (`"-"`), including thousands of `404` and `502` responses.
With `requireUserAgent: true`, those requests are rejected at Traefik before
they consume Kiwix CPU.
## Configuration
Add the following to the dynamic file provider configuration. The middleware
must be attached to each public Kiwix router.
Declare a middleware in dynamic configuration and attach it to a router:
```yaml
http:
@@ -74,116 +34,42 @@ http:
botfilter:
plugin:
botfilter:
statusCode:403
requireUserAgent:true
requireAccept:true
requireHost:true
browserValidation:true
whitelistCIDRs:
- 192.168.0.0/16
- 10.0.0.0/8
temporaryBanMinutes:15
scoreThreshold:100
scoreWindowMinutes:15
maxTrackedIPs:50000
maxScoreEventsPerIP:16
blockedUserAgents:
- curl
- wget
- python
- Go-http-client
- masscan
- sqlmap
- zgrab
- nikto
blockedPaths:
- /.env
- /.git
- /wp-login.php
- /xmlrpc.php
- /phpmyadmin
blockedUserAgents:
- sqlmap
- nikto
blockedExtensions:
- .env
- .bak
- .zip
# This weak signal contributes only 15 points. Direct links to a
# Kiwix article remain possible; they are not banned by themselves.
randomArticlePatterns:
- /content/
# Leave both unset when Traefik accepts traffic directly. If a CDN
# or load balancer is in front, configure its exact source CIDRs and
# ensure it overwrites this header.
# clientIPHeader: X-Forwarded-For
# trustedProxyCIDRs:
# - 203.0.113.0/24
# Do not turn this on during an attack unless short diagnostics are
# needed: per-request disk logging can itself become expensive.
logBlockedRequests:false
```
Apply it before `cp-ratelimit` so rejected requests do not consume the
rate-limiter's work. Keep the narrowed errors middleware as the outer wrapper
for the rate limiter:
```yaml
http:
routers:
rtr-default-wiki:
# ... existing rule/service fields ...
app:
rule:Host(`app.example.com`)
middlewares:
- cp-ratelimit-errorpages
- botfilter
- cp-ratelimit
service:app
```
Use the same middleware list on `rtr-wiki` and `rtr-yes` if they expose Kiwix.
Plugin names have two distinct roles above: `botfilter` is the local Traefik plugin identifier, while `github.com/hoelee/traefik-botfilter` is its module path. Keep the identifier the same under `experimental.plugins` and `http.middlewares.<name>.plugin`.
## Install in Traefik
## Local Development
### Production: remote plugin
The module name in `.traefik.yml` is intentionally the one requested here.
Create the public repository `github.com/hoelee/traefik-botfilter`, push this
directory, and create the immutable Git tag `v0.1.0`. Then put this in the
**static** Traefik configuration (`traefik.yml`), not `dynamic.yml`:
```yaml
experimental:
plugins:
botfilter:
moduleName:github.com/hoelee/traefik-botfilter
version:v0.1.0
```
Restart Traefik after changing static plugin configuration. Do not retag an
existing version; publish `v0.1.1` for later changes.
### Test locally first
Traefik local plugins need the Go module at the exact module path below
`/plugins-local/src`. On the Synology host, copy or clone this repository to:
Traefik local plugins must be placed under a directory matching the module path:
And use this static configuration instead of the remote `plugins` block:
Use `localPlugins` instead of `plugins` in static configuration:
```yaml
experimental:
@@ -192,63 +78,93 @@ experimental:
moduleName:github.com/hoelee/traefik-botfilter
```
Restart Traefik and confirm its startup log says that the `botfilter` plugin
loaded before exposing the public router.
## Example
## Option reference
[`example/traefik.yml`](example/traefik.yml) enables the local plugin and reads [`example/dynamic.yml`](example/dynamic.yml). The dynamic configuration protects `example.localhost` and forwards accepted requests to a backend listening on `127.0.0.1:8081`.
| Option | Default | Meaning |
| --- | ---: | --- |
| `statusCode` | `403` | HTTP status for rejected requests (`400`–`599`). |
| `temporaryBanMinutes` | `15` | In-memory ban duration. |
Start a test backend, for example:
```bash
go run github.com/traefik/whoami@latest --port 8081
```
Start Traefik with the example configuration, making the repository available at `/plugins-local/src/github.com/hoelee/traefik-botfilter` in the Traefik process or container. Then test it:
`randomArticleScore`, `notFoundScore`, and `fakeBrowserScore`. Set a score to
`0` to disable that one signal; scanner-path and required-header rejections
still ban immediately.
Score options are `emptyUserAgentScore` (40), `missingAcceptScore` (20), `blockedUserAgentScore` (80), `badPathScore` (50), `randomArticleScore` (15), `notFoundScore` (40), and `fakeBrowserScore` (40). Set a score to `0` to disable that signal.
## Operational limits and recommended defences
## Proxy Configuration
This plugin protects Kiwix from the malformed-header/scanner pattern in the
log, but it cannot stop a botnet that rotates IPs and perfectly imitates
browser headers. For that case:
Only configure `clientIPHeader` when Traefik receives traffic from a proxy you trust. Also set `trustedProxyCIDRs`; the plugin ignores the header for all other remote addresses to prevent clients from choosing their own cache identity.
1. Put the hostname behind a CDN/WAF with bot challenge and request-rate rules.
2. Firewall the NAS so public clients cannot reach Kiwix's port `6001` or any
Traefik entry point other than the intended public port.
3. Do not expose the NAS origin address in DNS or other services; otherwise
attackers can bypass the CDN.
4. Keep Traefik access logs sampled or rotate them quickly during an incident.
Per-request synchronous disk logging becomes material at flood volume.
5.`deploy.resources` is commonly ignored by non-Swarm Docker Compose. Verify
resource limits with `docker inspect` on the NAS rather than assuming the
`deploy` block limits CPU.
```yaml
clientIPHeader:X-Forwarded-For
trustedProxyCIDRs:
- 10.0.0.0/8
```
The cache is intentionally local to a Traefik process and is reset on
container restart. That is appropriate for a low-overhead edge filter; use a
CDN/WAF or shared store if bans must survive restarts or be shared by multiple
Traefik replicas.
Ensure the trusted proxy overwrites, rather than appends untrusted values to, that header.
## Publishing To The Catalog
The repository must be public, have the `traefik-plugin` GitHub topic, contain a valid root `.traefik.yml` manifest, and have an annotated or lightweight Git tag for each release. Push a new semantic-version tag after merging the package-name fix, for example:
```bash
git tag v1.0.1
git push origin v1.0.1
```
The package is intentionally named `traefik_botfilter`: Traefik's Yaegi loader derives that Go identifier from the final module path segment, replacing `-` with `_`. This must remain aligned with `github.com/hoelee/traefik-botfilter` for `CreateConfig` and `New` to load from the catalog.
## Development
```text
```bash
go test ./...
go vet ./...
gofmt -w *.go
```
The implementation only uses the Go standard library, which reduces plugin
startup and supply-chain risk.
## Limitations
This plugin reduces opportunistic scans and low-effort bot traffic. It does not reliably protect against distributed botnets, residential proxies, sophisticated browser automation, or application vulnerabilities. Pair it with rate limiting, a CDN or WAF, TLS, and application-level security controls for Internet-facing services.
## Contributing
Issues and pull requests are welcome. Include the Traefik version, plugin version, relevant configuration, and logs when reporting a problem.
## License
This project is licensed under the [Apache-2.0 license](LICENSE).
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
hoelee