diff --git a/README.md b/README.md index 6c3a688..6b4ad78 100644 --- a/README.md +++ b/README.md @@ -1,72 +1,64 @@ # Traefik Bot Filter -`traefik-botfilter` is a dependency-free Traefik middleware plugin for public -sites that are receiving scanners or unsophisticated HTTP floods. It blocks -known scan paths before they reach the upstream and keeps a bounded, local -per-client score cache for temporary bans. +> A lightweight Traefik middleware plugin that blocks scanners, malformed HTTP requests, and suspicious clients before they reach your backend. -It is deliberately a **request filter**, not a bot-management service. A -distributed attacker that sends valid browser-like requests from many IPs -needs an upstream CDN/WAF or firewall as well. +![Go](https://img.shields.io/badge/Go-1.20+-00ADD8?logo=go) +![Traefik](https://img.shields.io/badge/Traefik-v3.x-24A1C1?logo=traefikproxy) +![License](https://img.shields.io/github/license/hoelee/traefik-botfilter) +![GitHub Release](https://img.shields.io/github/v/release/hoelee/traefik-botfilter) -## What it does +`traefik-botfilter` is a dependency-free Traefik middleware plugin designed to stop common Internet scans, malformed HTTP requests, and low-quality bots before they reach your applications. -- Immediately rejects and temporarily bans configured scanner paths, file - extensions, and User-Agent tokens. -- Optionally requires `User-Agent`, `Accept`, and `Host`. A failed required - check is immediately cached as a temporary ban, so repeated requests do not - reach Kiwix. -- Adds scores in a configurable sliding window. Defaults implement the stated - model: empty UA `+40`, missing Accept `+20`, bad path `+50`, blocked UA - `+80`, first request directly to `/content/` `+15`, and an upstream `404` - `+40`. -- Bans when the score reaches `scoreThreshold` (default `100`). A blocked scan - path always bans immediately, independently of the score threshold. -- Applies conservative browser plausibility checks. It detects internally - inconsistent browser User-Agents, but does not claim to distinguish every - automation client from a real browser. That requires a JavaScript challenge - at a CDN/WAF layer. -- Bounds memory with `maxTrackedIPs` (default `50,000`) and - `maxScoreEventsPerIP` (default `16`), with lock-safe access and no cleanup - goroutine. -- Never trusts `X-Forwarded-For` unless `clientIPHeader` is explicitly set - **and** the TCP peer is in `trustedProxyCIDRs`. +Unlike a traditional Web Application Firewall (WAF), Bot Filter focuses on lightweight request validation and heuristic scoring. It requires **no Redis, database, or external services**, making it suitable for self-hosted environments, home labs, and production deployments. -## Important fixes before enabling it +## Features -Your Kiwix service currently publishes `6001:8080`. Requests to that port go -straight to Kiwix and bypass this middleware, rate limiting, and Traefik -access controls. Remove this line unless you need a separate protected private -listener: +- 🚫 Block common scanner paths (`/.env`, `/wp-login.php`, `/phpmyadmin`, etc.) +- 🤖 Detect and reject known bot User-Agents +- 📄 Require valid HTTP request headers +- 🧠 Configurable heuristic scoring system +- ⛔ Temporary in-memory IP banning +- 🌐 CIDR whitelist support +- 🔒 Browser header validation +- ⚡ Dependency-free (Go standard library only) +- 💾 Bounded memory usage +- 🔄 Reverse proxy aware +- 📦 No Redis or database required + +--- + +# Quick Start + +## 1. Enable the plugin + +### Local Plugin (Development) ```yaml -kiwix: - # ports: - # - "6001:8080" +experimental: + localPlugins: + botfilter: + moduleName: github.com/hoelee/traefik-botfilter ``` -Also narrow the current error middleware. `400-599` makes every ordinary -Kiwix 404 and upstream 502 call `host.docker.internal:44440`, which adds work -and obscures the original failure. Keep it only for rate-limit responses: +### Plugin Catalog (Future) + +After the plugin is available in the official Traefik Plugin Catalog: ```yaml -cp-ratelimit-errorpages: - errors: - status: - - "429-429" - service: srv-error - query: "/{status}.html" +experimental: + plugins: + botfilter: + moduleName: github.com/hoelee/traefik-botfilter + version: v1.0.0 ``` -The access log supplied with this request contains 57,141 requests with an -empty User-Agent (`"-"`), including thousands of `404` and `502` responses. -With `requireUserAgent: true`, those requests are rejected at Traefik before -they consume Kiwix CPU. +> **Note** +> +> The Plugin Catalog installation only works after this repository has been indexed by the official Traefik Plugin Catalog. -## Configuration +--- -Add the following to the dynamic file provider configuration. The middleware -must be attached to each public Kiwix router. +## 2. Configure the middleware ```yaml http: @@ -74,32 +66,32 @@ http: botfilter: plugin: botfilter: + statusCode: 403 requireUserAgent: true requireAccept: true requireHost: true + browserValidation: true - whitelistCIDRs: - - 192.168.0.0/16 - - 10.0.0.0/8 - temporaryBanMinutes: 15 + scoreThreshold: 100 scoreWindowMinutes: 15 - maxTrackedIPs: 50000 - maxScoreEventsPerIP: 16 + + whitelistCIDRs: + - 127.0.0.1/32 + - 192.168.0.0/16 blockedUserAgents: - curl - wget - python - Go-http-client - - masscan - sqlmap - - zgrab - nikto + - masscan blockedPaths: - /.env @@ -113,77 +105,395 @@ http: - .bak - .zip - # This weak signal contributes only 15 points. Direct links to a - # Kiwix article remain possible; they are not banned by themselves. randomArticlePatterns: - /content/ - # Leave both unset when Traefik accepts traffic directly. If a CDN - # or load balancer is in front, configure its exact source CIDRs and - # ensure it overwrites this header. - # clientIPHeader: X-Forwarded-For - # trustedProxyCIDRs: - # - 203.0.113.0/24 - - # Do not turn this on during an attack unless short diagnostics are - # needed: per-request disk logging can itself become expensive. logBlockedRequests: false ``` -Apply it before `cp-ratelimit` so rejected requests do not consume the -rate-limiter's work. Keep the narrowed errors middleware as the outer wrapper -for the rate limiter: +--- + +## 3. Attach the middleware ```yaml http: routers: - rtr-default-wiki: - # ... existing rule/service fields ... + + website: + rule: Host(`example.com`) + service: website + middlewares: - - cp-ratelimit-errorpages - botfilter - - cp-ratelimit ``` -Use the same middleware list on `rtr-wiki` and `rtr-yes` if they expose Kiwix. +--- -## Install in Traefik +# Installation -### Production: remote plugin +## Local Plugin -The module name in `.traefik.yml` is intentionally the one requested here. -Create the public repository `github.com/hoelee/traefik-botfilter`, push this -directory, and create the immutable Git tag `v0.1.0`. Then put this in the -**static** Traefik configuration (`traefik.yml`), not `dynamic.yml`: +Clone the repository into Traefik's local plugin directory. + +``` +plugins-local/ +└── src/ + └── github.com/ + └── hoelee/ + └── traefik-botfilter/ + ├── .traefik.yml + ├── go.mod + ├── config.go + ├── botfilter.go + └── ... +``` + +Docker Compose: + +```yaml +services: + + traefik: + + image: traefik:v3.5 + + restart: unless-stopped + + volumes: + + - /var/run/docker.sock:/var/run/docker.sock:ro + + - ./traefik.yml:/etc/traefik/traefik.yml:ro + + - ./dynamic.yml:/etc/traefik/dynamic.yml:ro + + - ./plugins-local:/plugins-local:ro +``` + +Restart Traefik after copying the plugin source. + +--- + +# Example Configuration + +## traefik.yml ```yaml experimental: - plugins: + + localPlugins: + botfilter: + moduleName: github.com/hoelee/traefik-botfilter - version: v0.1.0 ``` -Restart Traefik after changing static plugin configuration. Do not retag an -existing version; publish `v0.1.1` for later changes. +--- -### Test locally first - -Traefik local plugins need the Go module at the exact module path below -`/plugins-local/src`. On the Synology host, copy or clone this repository to: - -```text -/volume1/docker/traefik-plugins/src/github.com/hoelee/traefik-botfilter -``` - -Add this readonly mount to the Traefik service: +## dynamic.yml ```yaml -volumes: - - /volume1/docker/traefik-plugins:/plugins-local:ro +http: + middlewares: + botfilter: + plugin: + botfilter: + statusCode: 403 + requireUserAgent: true + requireAccept: true + requireHost: true + browserValidation: true + temporaryBanMinutes: 15 + scoreThreshold: 100 + scoreWindowMinutes: 15 + whitelistCIDRs: + - 127.0.0.1/32 + - 192.168.0.0/16 + blockedUserAgents: + - curl + - wget + - python + - Go-http-client + blockedPaths: + - /.env + - /.git + - /wp-login.php + - /xmlrpc.php + blockedExtensions: + - .env + - .bak + - .zip + logBlockedRequests: false ``` -And use this static configuration instead of the remote `plugins` block: +# Configuration Reference + +| Option | Default | Description | +|---------|:-------:|-------------| +| `statusCode` | `403` | HTTP status code returned for blocked requests. | +| `requireUserAgent` | `false` | Reject requests without a `User-Agent` header. | +| `requireAccept` | `false` | Reject requests without an `Accept` header. | +| `requireHost` | `false` | Reject requests without a `Host` header. | +| `browserValidation` | `false` | Detect inconsistent browser headers. | +| `temporaryBanMinutes` | `15` | Temporary in-memory IP ban duration. | +| `scoreThreshold` | `100` | Score required before an IP is temporarily banned. | +| `scoreWindowMinutes` | `15` | Sliding window used for score accumulation. | +| `maxTrackedIPs` | `50000` | Maximum number of IP addresses stored in memory. | +| `maxScoreEventsPerIP` | `16` | Maximum scoring events stored per client. | +| `whitelistCIDRs` | None | Clients that bypass all filtering. | +| `blockedUserAgents` | None | User-Agent substrings immediately rejected. | +| `blockedPaths` | None | URL paths that are immediately rejected. | +| `blockedExtensions` | None | Dangerous file extensions to reject. | +| `randomArticlePatterns` | `/content/` | URL prefixes that contribute to suspicion scoring. | +| `clientIPHeader` | Empty | Header containing the real client IP when behind trusted proxies. | +| `trustedProxyCIDRs` | None | Trusted proxies allowed to provide `clientIPHeader`. | +| `logBlockedRequests` | `false` | Log rejected requests to the Traefik log. | + +## Score Configuration + +The following values determine how much suspicion is added for different request characteristics. + +| Option | Default | +|---------|---------| +| `emptyUserAgentScore` | `40` | +| `missingAcceptScore` | `20` | +| `blockedUserAgentScore` | `80` | +| `badPathScore` | `50` | +| `randomArticleScore` | `15` | +| `notFoundScore` | `40` | +| `fakeBrowserScore` | `40` | + +Setting any score to `0` disables that individual signal without disabling the rest of the protection. + +--- + +# How It Works + +Bot Filter combines immediate blocking rules with a lightweight heuristic scoring engine. + +```text +Incoming Request + │ + ▼ +Required Header Validation + │ + ▼ +Blocked Path Detection + │ + ▼ +Blocked User-Agent Detection + │ + ▼ +Browser Validation + │ + ▼ +Heuristic Scoring + │ + ▼ +Score ≥ Threshold ? + ┌────┴────┐ + │ │ + ▼ ▼ +Reject Forward +403 Backend +``` + +Each client accumulates a temporary suspicion score. + +When the score reaches the configured threshold, the client is temporarily banned for the configured duration. + +The score automatically expires using a sliding time window, allowing legitimate users to recover without manual intervention. + +--- + +# Performance + +Bot Filter is intentionally lightweight. + +| Item | Value | +|------|------:| +| Dependencies | None | +| External Services | None | +| Redis | No | +| Database | No | +| Background Workers | None | +| Thread Safe | Yes | +| Memory Usage | Bounded | +| Reverse Proxy Support | Yes | + +The plugin only uses the Go standard library and stores a bounded amount of per-client state. + +--- + +# Limitations + +Bot Filter is **not** intended to replace a full Web Application Firewall (WAF). + +It is designed to eliminate: + +- Internet scanners +- Opportunistic bots +- Malformed HTTP requests +- Common exploit probes +- Low-quality scraping traffic + +It cannot reliably stop: + +- Large botnets +- Residential proxy networks +- Human-assisted attacks +- Browser automation that perfectly mimics legitimate traffic + +For those scenarios, combine Bot Filter with: + +- Cloudflare +- Traefik Rate Limit +- Fail2Ban +- Reverse Proxy Firewalls +- CDN Bot Protection + +--- + +# Examples + +This directory contains working examples for running **Traefik Bot Filter**. + +## Files + +| File | Description | +|------|-------------| +| `docker-compose.yml` | Complete Docker Compose example using Traefik and the local plugin. | +| `traefik.yml` | Static Traefik configuration. | +| `dynamic.yml` | Dynamic configuration containing the Bot Filter middleware. | + +--- + +## Quick Start + +Clone this repository: + +```bash +git clone https://github.com/hoelee/traefik-botfilter.git +``` + +Copy the plugin into Traefik's local plugin directory: + +```text +plugins-local/ +└── src/ + └── github.com/ + └── hoelee/ + └── traefik-botfilter/ +``` + +Start the example: + +```bash +docker compose up -d +``` + +Open the dashboard: + +``` +http://localhost:8080/dashboard/ +``` + +Example service: + +``` +http://localhost/ +``` + +--- + +## Middleware Flow + +``` +Internet + │ + ▼ +Traefik + │ + ▼ +Bot Filter + │ + ▼ +Backend Service +``` + +--- + +## Plugin Configuration + +The middleware is configured in **dynamic.yml**. + +The plugin is enabled in **traefik.yml**. + +--- + +## Testing + +### Normal Browser + +``` +curl http://localhost/ +``` + +Expected: + +``` +HTTP/1.1 200 OK +``` + +--- + +### Missing User-Agent + +``` +curl -H "User-Agent:" http://localhost/ +``` + +Expected: + +``` +HTTP/1.1 403 Forbidden +``` + +--- + +### Blocked Path + +``` +curl http://localhost/.env +``` + +Expected: + +``` +HTTP/1.1 403 Forbidden +``` + +--- + +### Blocked User-Agent + +``` +curl -A "sqlmap" http://localhost/ +``` + +Expected: + +``` +HTTP/1.1 403 Forbidden +``` + +--- + +## Notes + +These examples use the **local plugin** loader. + +After the plugin is published in the official Traefik Plugin Catalog, replace: ```yaml experimental: @@ -192,63 +502,135 @@ experimental: moduleName: github.com/hoelee/traefik-botfilter ``` -Restart Traefik and confirm its startup log says that the `botfilter` plugin -loaded before exposing the public router. +with: -## Option reference +```yaml +experimental: + plugins: + botfilter: + moduleName: github.com/hoelee/traefik-botfilter + version: v1.0.0 +``` -| Option | Default | Meaning | -| --- | ---: | --- | -| `statusCode` | `403` | HTTP status for rejected requests (`400`–`599`). | -| `temporaryBanMinutes` | `15` | In-memory ban duration. | -| `scoreThreshold` | `100` | Score at which a client is banned. | -| `scoreWindowMinutes` | `15` | Sliding window for score events. | -| `maxTrackedIPs` | `50000` | Hard maximum size of the per-IP state map. | -| `maxScoreEventsPerIP` | `16` | Bound on score events retained per client. | -| `requireUserAgent` | `false` | Immediately ban missing User-Agent requests. | -| `requireAccept` | `false` | Immediately ban missing Accept requests. | -| `requireHost` | `false` | Immediately ban missing Host requests. | -| `browserValidation` | `false` | Score implausible Mozilla-family header combinations. | -| `whitelistCIDRs` | none | Clients that bypass all checks and cache updates. | -| `clientIPHeader` | empty | Optional header used only from `trustedProxyCIDRs`. | -| `trustedProxyCIDRs` | none | TCP peer ranges allowed to supply the client-IP header. | -| `randomArticlePatterns` | `/content/` | First-request path prefixes that add `randomArticleScore`. | -| `logBlockedRequests` | `false` | Opt-in rejected-request logging. | +--- -All listed score fields are configurable: `emptyUserAgentScore`, -`missingAcceptScore`, `blockedUserAgentScore`, `badPathScore`, -`randomArticleScore`, `notFoundScore`, and `fakeBrowserScore`. Set a score to -`0` to disable that one signal; scanner-path and required-header rejections -still ban immediately. +# Best Practices -## Operational limits and recommended defences +For production deployments: -This plugin protects Kiwix from the malformed-header/scanner pattern in the -log, but it cannot stop a botnet that rotates IPs and perfectly imitates -browser headers. For that case: +- Place Bot Filter before Rate Limit middleware. +- Protect the origin server from direct Internet access. +- Use HTTPS only. +- Enable access logs only when required. +- Keep Traefik updated. +- Use a CDN or WAF for Internet-facing services. -1. Put the hostname behind a CDN/WAF with bot challenge and request-rate rules. -2. Firewall the NAS so public clients cannot reach Kiwix's port `6001` or any - Traefik entry point other than the intended public port. -3. Do not expose the NAS origin address in DNS or other services; otherwise - attackers can bypass the CDN. -4. Keep Traefik access logs sampled or rotate them quickly during an incident. - Per-request synchronous disk logging becomes material at flood volume. -5. `deploy.resources` is commonly ignored by non-Swarm Docker Compose. Verify - resource limits with `docker inspect` on the NAS rather than assuming the - `deploy` block limits CPU. +Recommended middleware order: -The cache is intentionally local to a Traefik process and is reset on -container restart. That is appropriate for a low-overhead edge filter; use a -CDN/WAF or shared store if bans must survive restarts or be shared by multiple -Traefik replicas. +```yaml +middlewares: + - botfilter + - ratelimit + - headers + - compress +``` -## Development +--- -```text +# Roadmap + +## v1.0 + +- Request validation +- User-Agent filtering +- Browser validation +- Path filtering +- Temporary IP bans +- Heuristic scoring + +## v1.1 + +- Regular expression matching +- Prometheus metrics +- Configurable response body +- IPv6 optimizations +- Better browser fingerprint validation + +## v2.0 + +- Redis shared cache +- Multi-instance synchronization +- Dashboard statistics +- ASN filtering +- GeoIP filtering +- Optional CAPTCHA integration + +--- + +# Development + +Clone the repository: + +```bash +git clone https://github.com/hoelee/traefik-botfilter.git +cd traefik-botfilter +``` + +Run tests: + +```bash go test ./... +``` + +Run static analysis: + +```bash go vet ./... ``` -The implementation only uses the Go standard library, which reduces plugin -startup and supply-chain risk. +Format source code: + +```bash +go fmt ./... +``` + +--- + +# Contributing + +Contributions are welcome. + +If you discover a bug, have a feature request, or would like to improve the plugin, please open an Issue or Pull Request. + +Please include: + +- Traefik version +- Go version +- Plugin version +- Example configuration +- Relevant logs + +--- + +# License + +This project is licensed under the MIT License. + +See the [LICENSE](LICENSE) file for details. + +--- + +# Acknowledgements + +- Traefik Labs +- Go Community +- Contributors and users of the project + +--- + +## Star the Project + +If this plugin helps protect your services, please consider giving the repository a ⭐ on GitHub. + +It helps others discover the project and supports future development. +