Problem

You’ve users that you want to ban / block from your web servers using Fail2Ban but your web servers are behind a proxy so all traffic appears to be coming from the proxy server IP/Interface.

Ideally you should have learning rules on your proxy or load balancer where you can automatically filter/rate/take actions, but this may not always be the case.

Most people that come across this problem get as far as the first part of the solution but then are baffled when they use Fail2Ban and it still doesn’t work.

For the rest of this post I’ll refer to the Proxy/Load Balancer simply as the LB, mainly as this solution was developed for three web servers behind a load balancer.

Modify Your LB

Ensure your LB is set up to add the http header “X-Forwarded-For”.

How you enable this will depend on your LB and is outside the scope of this post.

Modify Your Web Server Apache Configs

Specifically around logging.  We want to ensure that Fail2Ban can identify the “correct” IP in order to successfully ban it.

I usually use a custom log format and add the X-Forwarded-For details at the end in an easy to identify block, this appears in the logs at the end as “[XF www.xxx.yyy.zzz]”

To do this I use the following directives in the vhost definition in Apache:

CustomLog /path/to/logs/access_log "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\" \"[XF %{X-Forwarded-For}i]\""

You’ll then start seeing the logs as follows:

192.168.1.2 - - [29/Sep/2014:10:56:31 +0100] "POST /login.php HTTP/1.1" 200 15 "-" "curl/7.19.7 " "[XF 10.10.1.1]"

So what we’re interested in here is that the request is coming from the external IP 10.10.1.1 (yes, I know that’s private address space but for this post I’m not going to use “real” IP addresses)

Create Your Fail2Ban Filter Recipe

There’s three parts of this, the filter, the “jail” and the custom IPTable action.

Filter

Create your filter in fail2ban/filter.d accordingly:

# Fail2Ban configuration file
#
# Author: Centos.Tips
#
# $Revision: 1$
#

[Definition]

# Option:  failregex
# Notes.:  Regexp to catch Apache brute force login attempts (using X-Forwarded-For)
# Values:  TEXT
#
failregex = POST .*/login.php.*\[XF <HOST>

# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
#
ignoreregex =

Action

Create your custom IPTables action in fail2ban/action.d accordingly:

# Fail2Ban configuration file
#
# Author: Centos.Tips
#
#

[INCLUDES]

before = iptables-blocktype.conf

[Definition]

# Option:  actionstart
# Notes.:  command executed once at the start of Fail2Ban.
# Values:  CMD
#
actionstart = iptables -N fail2ban-<name>
              iptables -A fail2ban-<name> -j RETURN
              iptables -I <chain> -p <protocol> --dport <port> -j fail2ban-<name>

# Option:  actionstop
# Notes.:  command executed once at the end of Fail2Ban
# Values:  CMD
#
actionstop = iptables -D <chain> -p <protocol> --dport <port> -j fail2ban-<name>
             iptables -F fail2ban-<name>
             iptables -X fail2ban-<name>

# Option:  actioncheck
# Notes.:  command executed once before each actionban command
# Values:  CMD
#
actioncheck = iptables -n -L <chain> | grep -q 'fail2ban-<name>[ \t]'

# Option:  actionban
# Notes.:  command executed when banning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionban = iptables -I fail2ban-<name> 1 -p tcp --dport 80 -m string --algo bm --string 'X-Forwarded-For: <ip>' -j DROP

# Option:  actionunban
# Notes.:  command executed when unbanning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionunban = iptables -D fail2ban-<name> -p tcp --dport 80 -m string --algo bm --string 'X-Forwarded-For: <ip>' -j DROP

[Init]

# Default name of the chain
#
name = default

# Option:  port
# Notes.:  specifies port to monitor
# Values:  [ NUM | STRING ]  Default:
#
port = http

# Option:  protocol
# Notes.:  internally used by config reader for interpolations.
# Values:  [ tcp | udp | icmp | all ] Default: tcp
#
protocol = tcp

# Option:  chain
# Notes    specifies the iptables chain to which the fail2ban rules should be
#          added
# Values:  STRING  Default: INPUT
chain = INPUT

The difference between this and a normal IPTables DROP is that we have to do packet inspection  where we look for the X-Forwarded-For and the relevant IP address in the packet before we drop it.  This should generally work but your mileage may vary.

Jail

Now you can mix together your recipe and stick it in your fail2ban/jail.local file

[apache-proxy]
enabled = true
filter = apache-proxy
action = iptables-proxy[name = apache-proxy, port = http, protocol = tcp]
         sendmail-whois[name=LoginDetect, dest=yourname@yourdomain.com, sender=fail2ban@yourdomain.com, sendername="Fail2Ban"]
port = http
logpath = /path/to/your/access_log
maxretry = 5
findtime = 60
bantime = 900

So here we’ll allow 5 login attempts in a minute, after that you’re blocked from the server for 15 minutes.

Reload fail2ban and you’re done!