Brute Force Protection: fail2ban, CAPTCHA, and Rate Limiting

Brute force attacks — automated bots guessing passwords — remain one of the most common attack vectors. Every day, a typical VPS server receives thousands of password-guessing attempts against SSH and web forms. Layered defense makes such attacks unprofitable.

fail2ban: Automatic Attacker Blocking

apt install fail2ban -y

# /etc/fail2ban/jail.local
[DEFAULT]
bantime  = 3600     # Ban for 1 hour
maxretry = 5        # After 5 failed attempts
findtime = 600      # Within 10 minutes

[sshd]
enabled  = true
port     = ssh,2222
logpath  = /var/log/auth.log

[nginx-http-auth]
enabled  = true
logpath  = /var/log/nginx/error.log

[nginx-limit-req]
enabled  = true
logpath  = /var/log/nginx/error.log

systemctl enable --now fail2ban

Managing fail2ban

fail2ban-client status sshd
fail2ban-client set sshd unbanip 1.2.3.4
fail2ban-client set sshd banip 5.6.7.8
grep "Ban " /var/log/fail2ban.log | tail -20

Rate Limiting in Nginx

http {
    limit_req_zone $binary_remote_addr zone=login:10m rate=1r/m;
    limit_req_zone $binary_remote_addr zone=api:10m rate=30r/m;
}
server {
    location /wp-login.php {
        limit_req zone=login burst=3 nodelay;
        limit_req_status 429;
    }
}

Geoblocking with GeoIP2

apt install libnginx-mod-http-geoip2 -y

http {
    geoip2 /var/lib/GeoIP/GeoLite2-Country.mmdb {
        $geoip2_data_country_code country iso_code;
    }
    map $geoip2_data_country_code $allowed_country {
        default 0;
        UA 1; RU 1; PL 1; DE 1;
    }
}
server {
    if ($allowed_country = 0) { return 403; }
}

Protecting the Login Form with CAPTCHA

For WordPress, use the Cloudflare Turnstile or hCaptcha plugin. For custom forms:

// PHP: hCaptcha verification
$response = file_get_contents('https://hcaptcha.com/siteverify', false,
    stream_context_create(['http' => ['method' => 'POST',
    'content' => http_build_query([
        'secret' => 'YOUR_SECRET_KEY',
        'response' => $_POST['h-captcha-response']
    ])]]));
$result = json_decode($response);
if (!$result->success) { die('CAPTCHA failed'); }

echo "ignoreip = 127.0.0.1/8 YOUR_IP" >> /etc/fail2ban/jail.local

Common Protection Setup Errors

Layers of Protection: What Guards Against What

When One Layer Isn't Enough

• The attack comes from thousands of different IPs at once (a botnet) — fail2ban is powerless here, you need protection at the provider or Cloudflare level
• The passwords are genuinely weak — no brute-force defense replaces switching to strong passwords and SSH keys
• In a targeted attack on one site, CAPTCHA can be bypassed by paid solving services — add behavioral analysis (Cloudflare Turnstile accounts for this)
• Rate limiting is only configured on Nginx while the application listens on a port directly, bypassing the proxy

Verifying the Result

# Current status of the sshd jail
fail2ban-client status sshd

# Simulate exceeding the request limit
for i in $(seq 1 10); do curl -s -o /dev/null -w "%{http_code}\n" https://example.com/wp-login.php; done

If the loop above never returns a 429, the limit_req rule isn't being applied to that location — check the Nginx config with nginx -T. Related: basic fail2ban setup and DDoS protection with Nginx and Cloudflare.

fail2ban on Different Distros: Where to Find the Logs

The path to the system authentication log differs between distributions, and that's the most common reason a copied jail.local "doesn't work" after moving it from another server. Ubuntu and Debian log SSH login attempts to /var/log/auth.log, while RHEL-based systems (CentOS, AlmaLinux, Rocky Linux) use /var/log/secure. Before starting fail2ban on a new server, always confirm the actual path with ls -la /var/log/ | grep -E "auth|secure" instead of copying the config blindly.

Frequently asked questions

Will fail2ban help if an attack comes from thousands of different IP addresses at once?

Practically no — fail2ban blocks one address once it crosses a threshold of failed attempts, but in a distributed attack from thousands of different IPs, each address only manages a couple of attempts and never crosses it. Protection needs to sit higher up — the provider's side or a service like Cloudflare, seeing the whole traffic picture, not one server's logs.

How do I avoid locking myself out while setting up this protection?

Before tightening the rules, add your own IP address to fail2ban's exception list in advance — that way the service won't ban it along with everyone else during checks and tests. If access does disappear anyway, usually it's enough to wait out the ban duration or lift it through another access channel, such as the provider's console.

Does CAPTCHA stop automated bots completely?

Not completely — in a targeted attack on a specific site, CAPTCHA can be bypassed using paid third-party solving services that get a real human to solve the puzzle on the other end. So CAPTCHA is best treated as one layer of defense rather than a final solution, worth pairing with behavioral analysis or combining with rate limiting and fail2ban.

Why do legitimate users sometimes get a too-many-requests error after rate limiting is set up?

Usually the cause is that the rate or burst value is set stricter than the form's real traffic pattern — a user accidentally submits the form twice in a row, or the browser retries a request over a slow connection. The fix: revisit the limiting zone's parameters against real usage scenarios rather than lowering the block threshold at random.