637 lines
12 KiB
Markdown
637 lines
12 KiB
Markdown
# Traefik Bot Filter
|
|
|
|
> A lightweight Traefik middleware plugin that blocks scanners, malformed HTTP requests, and suspicious clients before they reach your backend.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
`traefik-botfilter` is a dependency-free Traefik middleware plugin designed to stop common Internet scans, malformed HTTP requests, and low-quality bots before they reach your applications.
|
|
|
|
Unlike a traditional Web Application Firewall (WAF), Bot Filter focuses on lightweight request validation and heuristic scoring. It requires **no Redis, database, or external services**, making it suitable for self-hosted environments, home labs, and production deployments.
|
|
|
|
## Features
|
|
|
|
- 🚫 Block common scanner paths (`/.env`, `/wp-login.php`, `/phpmyadmin`, etc.)
|
|
- 🤖 Detect and reject known bot User-Agents
|
|
- 📄 Require valid HTTP request headers
|
|
- 🧠 Configurable heuristic scoring system
|
|
- ⛔ Temporary in-memory IP banning
|
|
- 🌐 CIDR whitelist support
|
|
- 🔒 Browser header validation
|
|
- ⚡ Dependency-free (Go standard library only)
|
|
- 💾 Bounded memory usage
|
|
- 🔄 Reverse proxy aware
|
|
- 📦 No Redis or database required
|
|
|
|
---
|
|
|
|
# Quick Start
|
|
|
|
## 1. Enable the plugin
|
|
|
|
### Local Plugin (Development)
|
|
|
|
```yaml
|
|
experimental:
|
|
localPlugins:
|
|
botfilter:
|
|
moduleName: github.com/hoelee/traefik-botfilter
|
|
```
|
|
|
|
### Plugin Catalog (Future)
|
|
|
|
After the plugin is available in the official Traefik Plugin Catalog:
|
|
|
|
```yaml
|
|
experimental:
|
|
plugins:
|
|
botfilter:
|
|
moduleName: github.com/hoelee/traefik-botfilter
|
|
version: v1.0.0
|
|
```
|
|
|
|
> **Note**
|
|
>
|
|
> The Plugin Catalog installation only works after this repository has been indexed by the official Traefik Plugin Catalog.
|
|
|
|
---
|
|
|
|
## 2. Configure the middleware
|
|
|
|
```yaml
|
|
http:
|
|
middlewares:
|
|
botfilter:
|
|
plugin:
|
|
botfilter:
|
|
|
|
statusCode: 403
|
|
|
|
requireUserAgent: true
|
|
requireAccept: true
|
|
requireHost: true
|
|
|
|
browserValidation: true
|
|
|
|
temporaryBanMinutes: 15
|
|
|
|
scoreThreshold: 100
|
|
scoreWindowMinutes: 15
|
|
|
|
whitelistCIDRs:
|
|
- 127.0.0.1/32
|
|
- 192.168.0.0/16
|
|
|
|
blockedUserAgents:
|
|
- curl
|
|
- wget
|
|
- python
|
|
- Go-http-client
|
|
- sqlmap
|
|
- nikto
|
|
- masscan
|
|
|
|
blockedPaths:
|
|
- /.env
|
|
- /.git
|
|
- /wp-login.php
|
|
- /xmlrpc.php
|
|
- /phpmyadmin
|
|
|
|
blockedExtensions:
|
|
- .env
|
|
- .bak
|
|
- .zip
|
|
|
|
randomArticlePatterns:
|
|
- /content/
|
|
|
|
logBlockedRequests: false
|
|
```
|
|
|
|
---
|
|
|
|
## 3. Attach the middleware
|
|
|
|
```yaml
|
|
http:
|
|
routers:
|
|
|
|
website:
|
|
rule: Host(`example.com`)
|
|
service: website
|
|
|
|
middlewares:
|
|
- botfilter
|
|
```
|
|
|
|
---
|
|
|
|
# Installation
|
|
|
|
## Local Plugin
|
|
|
|
Clone the repository into Traefik's local plugin directory.
|
|
|
|
```
|
|
plugins-local/
|
|
└── src/
|
|
└── github.com/
|
|
└── hoelee/
|
|
└── traefik-botfilter/
|
|
├── .traefik.yml
|
|
├── go.mod
|
|
├── config.go
|
|
├── botfilter.go
|
|
└── ...
|
|
```
|
|
|
|
Docker Compose:
|
|
|
|
```yaml
|
|
services:
|
|
|
|
traefik:
|
|
|
|
image: traefik:v3.5
|
|
|
|
restart: unless-stopped
|
|
|
|
volumes:
|
|
|
|
- /var/run/docker.sock:/var/run/docker.sock:ro
|
|
|
|
- ./traefik.yml:/etc/traefik/traefik.yml:ro
|
|
|
|
- ./dynamic.yml:/etc/traefik/dynamic.yml:ro
|
|
|
|
- ./plugins-local:/plugins-local:ro
|
|
```
|
|
|
|
Restart Traefik after copying the plugin source.
|
|
|
|
---
|
|
|
|
# Example Configuration
|
|
|
|
## traefik.yml
|
|
|
|
```yaml
|
|
experimental:
|
|
|
|
localPlugins:
|
|
|
|
botfilter:
|
|
|
|
moduleName: github.com/hoelee/traefik-botfilter
|
|
```
|
|
|
|
---
|
|
|
|
## dynamic.yml
|
|
|
|
```yaml
|
|
http:
|
|
middlewares:
|
|
botfilter:
|
|
plugin:
|
|
botfilter:
|
|
statusCode: 403
|
|
requireUserAgent: true
|
|
requireAccept: true
|
|
requireHost: true
|
|
browserValidation: true
|
|
temporaryBanMinutes: 15
|
|
scoreThreshold: 100
|
|
scoreWindowMinutes: 15
|
|
whitelistCIDRs:
|
|
- 127.0.0.1/32
|
|
- 192.168.0.0/16
|
|
blockedUserAgents:
|
|
- curl
|
|
- wget
|
|
- python
|
|
- Go-http-client
|
|
blockedPaths:
|
|
- /.env
|
|
- /.git
|
|
- /wp-login.php
|
|
- /xmlrpc.php
|
|
blockedExtensions:
|
|
- .env
|
|
- .bak
|
|
- .zip
|
|
logBlockedRequests: false
|
|
```
|
|
|
|
# Configuration Reference
|
|
|
|
| Option | Default | Description |
|
|
|---------|:-------:|-------------|
|
|
| `statusCode` | `403` | HTTP status code returned for blocked requests. |
|
|
| `requireUserAgent` | `false` | Reject requests without a `User-Agent` header. |
|
|
| `requireAccept` | `false` | Reject requests without an `Accept` header. |
|
|
| `requireHost` | `false` | Reject requests without a `Host` header. |
|
|
| `browserValidation` | `false` | Detect inconsistent browser headers. |
|
|
| `temporaryBanMinutes` | `15` | Temporary in-memory IP ban duration. |
|
|
| `scoreThreshold` | `100` | Score required before an IP is temporarily banned. |
|
|
| `scoreWindowMinutes` | `15` | Sliding window used for score accumulation. |
|
|
| `maxTrackedIPs` | `50000` | Maximum number of IP addresses stored in memory. |
|
|
| `maxScoreEventsPerIP` | `16` | Maximum scoring events stored per client. |
|
|
| `whitelistCIDRs` | None | Clients that bypass all filtering. |
|
|
| `blockedUserAgents` | None | User-Agent substrings immediately rejected. |
|
|
| `blockedPaths` | None | URL paths that are immediately rejected. |
|
|
| `blockedExtensions` | None | Dangerous file extensions to reject. |
|
|
| `randomArticlePatterns` | `/content/` | URL prefixes that contribute to suspicion scoring. |
|
|
| `clientIPHeader` | Empty | Header containing the real client IP when behind trusted proxies. |
|
|
| `trustedProxyCIDRs` | None | Trusted proxies allowed to provide `clientIPHeader`. |
|
|
| `logBlockedRequests` | `false` | Log rejected requests to the Traefik log. |
|
|
|
|
## Score Configuration
|
|
|
|
The following values determine how much suspicion is added for different request characteristics.
|
|
|
|
| Option | Default |
|
|
|---------|---------|
|
|
| `emptyUserAgentScore` | `40` |
|
|
| `missingAcceptScore` | `20` |
|
|
| `blockedUserAgentScore` | `80` |
|
|
| `badPathScore` | `50` |
|
|
| `randomArticleScore` | `15` |
|
|
| `notFoundScore` | `40` |
|
|
| `fakeBrowserScore` | `40` |
|
|
|
|
Setting any score to `0` disables that individual signal without disabling the rest of the protection.
|
|
|
|
---
|
|
|
|
# How It Works
|
|
|
|
Bot Filter combines immediate blocking rules with a lightweight heuristic scoring engine.
|
|
|
|
```text
|
|
Incoming Request
|
|
│
|
|
▼
|
|
Required Header Validation
|
|
│
|
|
▼
|
|
Blocked Path Detection
|
|
│
|
|
▼
|
|
Blocked User-Agent Detection
|
|
│
|
|
▼
|
|
Browser Validation
|
|
│
|
|
▼
|
|
Heuristic Scoring
|
|
│
|
|
▼
|
|
Score ≥ Threshold ?
|
|
┌────┴────┐
|
|
│ │
|
|
▼ ▼
|
|
Reject Forward
|
|
403 Backend
|
|
```
|
|
|
|
Each client accumulates a temporary suspicion score.
|
|
|
|
When the score reaches the configured threshold, the client is temporarily banned for the configured duration.
|
|
|
|
The score automatically expires using a sliding time window, allowing legitimate users to recover without manual intervention.
|
|
|
|
---
|
|
|
|
# Performance
|
|
|
|
Bot Filter is intentionally lightweight.
|
|
|
|
| Item | Value |
|
|
|------|------:|
|
|
| Dependencies | None |
|
|
| External Services | None |
|
|
| Redis | No |
|
|
| Database | No |
|
|
| Background Workers | None |
|
|
| Thread Safe | Yes |
|
|
| Memory Usage | Bounded |
|
|
| Reverse Proxy Support | Yes |
|
|
|
|
The plugin only uses the Go standard library and stores a bounded amount of per-client state.
|
|
|
|
---
|
|
|
|
# Limitations
|
|
|
|
Bot Filter is **not** intended to replace a full Web Application Firewall (WAF).
|
|
|
|
It is designed to eliminate:
|
|
|
|
- Internet scanners
|
|
- Opportunistic bots
|
|
- Malformed HTTP requests
|
|
- Common exploit probes
|
|
- Low-quality scraping traffic
|
|
|
|
It cannot reliably stop:
|
|
|
|
- Large botnets
|
|
- Residential proxy networks
|
|
- Human-assisted attacks
|
|
- Browser automation that perfectly mimics legitimate traffic
|
|
|
|
For those scenarios, combine Bot Filter with:
|
|
|
|
- Cloudflare
|
|
- Traefik Rate Limit
|
|
- Fail2Ban
|
|
- Reverse Proxy Firewalls
|
|
- CDN Bot Protection
|
|
|
|
---
|
|
|
|
# Examples
|
|
|
|
This directory contains working examples for running **Traefik Bot Filter**.
|
|
|
|
## Files
|
|
|
|
| File | Description |
|
|
|------|-------------|
|
|
| `docker-compose.yml` | Complete Docker Compose example using Traefik and the local plugin. |
|
|
| `traefik.yml` | Static Traefik configuration. |
|
|
| `dynamic.yml` | Dynamic configuration containing the Bot Filter middleware. |
|
|
|
|
---
|
|
|
|
## Quick Start
|
|
|
|
Clone this repository:
|
|
|
|
```bash
|
|
git clone https://github.com/hoelee/traefik-botfilter.git
|
|
```
|
|
|
|
Copy the plugin into Traefik's local plugin directory:
|
|
|
|
```text
|
|
plugins-local/
|
|
└── src/
|
|
└── github.com/
|
|
└── hoelee/
|
|
└── traefik-botfilter/
|
|
```
|
|
|
|
Start the example:
|
|
|
|
```bash
|
|
docker compose up -d
|
|
```
|
|
|
|
Open the dashboard:
|
|
|
|
```
|
|
http://localhost:8080/dashboard/
|
|
```
|
|
|
|
Example service:
|
|
|
|
```
|
|
http://localhost/
|
|
```
|
|
|
|
---
|
|
|
|
## Middleware Flow
|
|
|
|
```
|
|
Internet
|
|
│
|
|
▼
|
|
Traefik
|
|
│
|
|
▼
|
|
Bot Filter
|
|
│
|
|
▼
|
|
Backend Service
|
|
```
|
|
|
|
---
|
|
|
|
## Plugin Configuration
|
|
|
|
The middleware is configured in **dynamic.yml**.
|
|
|
|
The plugin is enabled in **traefik.yml**.
|
|
|
|
---
|
|
|
|
## Testing
|
|
|
|
### Normal Browser
|
|
|
|
```
|
|
curl http://localhost/
|
|
```
|
|
|
|
Expected:
|
|
|
|
```
|
|
HTTP/1.1 200 OK
|
|
```
|
|
|
|
---
|
|
|
|
### Missing User-Agent
|
|
|
|
```
|
|
curl -H "User-Agent:" http://localhost/
|
|
```
|
|
|
|
Expected:
|
|
|
|
```
|
|
HTTP/1.1 403 Forbidden
|
|
```
|
|
|
|
---
|
|
|
|
### Blocked Path
|
|
|
|
```
|
|
curl http://localhost/.env
|
|
```
|
|
|
|
Expected:
|
|
|
|
```
|
|
HTTP/1.1 403 Forbidden
|
|
```
|
|
|
|
---
|
|
|
|
### Blocked User-Agent
|
|
|
|
```
|
|
curl -A "sqlmap" http://localhost/
|
|
```
|
|
|
|
Expected:
|
|
|
|
```
|
|
HTTP/1.1 403 Forbidden
|
|
```
|
|
|
|
---
|
|
|
|
## Notes
|
|
|
|
These examples use the **local plugin** loader.
|
|
|
|
After the plugin is published in the official Traefik Plugin Catalog, replace:
|
|
|
|
```yaml
|
|
experimental:
|
|
localPlugins:
|
|
botfilter:
|
|
moduleName: github.com/hoelee/traefik-botfilter
|
|
```
|
|
|
|
with:
|
|
|
|
```yaml
|
|
experimental:
|
|
plugins:
|
|
botfilter:
|
|
moduleName: github.com/hoelee/traefik-botfilter
|
|
version: v1.0.0
|
|
```
|
|
|
|
---
|
|
|
|
# Best Practices
|
|
|
|
For production deployments:
|
|
|
|
- Place Bot Filter before Rate Limit middleware.
|
|
- Protect the origin server from direct Internet access.
|
|
- Use HTTPS only.
|
|
- Enable access logs only when required.
|
|
- Keep Traefik updated.
|
|
- Use a CDN or WAF for Internet-facing services.
|
|
|
|
Recommended middleware order:
|
|
|
|
```yaml
|
|
middlewares:
|
|
- botfilter
|
|
- ratelimit
|
|
- headers
|
|
- compress
|
|
```
|
|
|
|
---
|
|
|
|
# Roadmap
|
|
|
|
## v1.0
|
|
|
|
- Request validation
|
|
- User-Agent filtering
|
|
- Browser validation
|
|
- Path filtering
|
|
- Temporary IP bans
|
|
- Heuristic scoring
|
|
|
|
## v1.1
|
|
|
|
- Regular expression matching
|
|
- Prometheus metrics
|
|
- Configurable response body
|
|
- IPv6 optimizations
|
|
- Better browser fingerprint validation
|
|
|
|
## v2.0
|
|
|
|
- Redis shared cache
|
|
- Multi-instance synchronization
|
|
- Dashboard statistics
|
|
- ASN filtering
|
|
- GeoIP filtering
|
|
- Optional CAPTCHA integration
|
|
|
|
---
|
|
|
|
# Development
|
|
|
|
Clone the repository:
|
|
|
|
```bash
|
|
git clone https://github.com/hoelee/traefik-botfilter.git
|
|
cd traefik-botfilter
|
|
```
|
|
|
|
Run tests:
|
|
|
|
```bash
|
|
go test ./...
|
|
```
|
|
|
|
Run static analysis:
|
|
|
|
```bash
|
|
go vet ./...
|
|
```
|
|
|
|
Format source code:
|
|
|
|
```bash
|
|
go fmt ./...
|
|
```
|
|
|
|
---
|
|
|
|
# Contributing
|
|
|
|
Contributions are welcome.
|
|
|
|
If you discover a bug, have a feature request, or would like to improve the plugin, please open an Issue or Pull Request.
|
|
|
|
Please include:
|
|
|
|
- Traefik version
|
|
- Go version
|
|
- Plugin version
|
|
- Example configuration
|
|
- Relevant logs
|
|
|
|
---
|
|
|
|
# License
|
|
|
|
This project is licensed under the MIT License.
|
|
|
|
See the [LICENSE](LICENSE) file for details.
|
|
|
|
---
|
|
|
|
# Acknowledgements
|
|
|
|
- Traefik Labs
|
|
- Go Community
|
|
- Contributors and users of the project
|
|
|
|
---
|
|
|
|
## Star the Project
|
|
|
|
If this plugin helps protect your services, please consider giving the repository a ⭐ on GitHub.
|
|
|
|
It helps others discover the project and supports future development.
|
|
|