fail2ban + Nginx
In this system fail2ban is supposed to parse nginx
logs (customized) for 404 and 403 status codes and add iptables rules
to block IPs on the network layer from which excessive 404 and 403
are coming up.
Under a DDOS, because of the verity of IPs
available, the frequency of banning and unbanning will be large, as a
result there the iptables command will run too many times, resulting
in an overhead. A system has been created to prevent this overhead
even when there are 1000s of Ips being banned and unbanned.
Objective is to prevent overload of the
application, brute force attacks by sending frequent failed
authentication requests. 404s have also been taken care of to prevent
path discovery apart from the same reasons as previously stated.
Architecture
Instead of the banning iptables being run directly
by fail2ban, it's indirectly executed by a bash script on a cron job
which runs a single iptables command to ban/unban any no. of IPs in
bulk.
fail2ban runs as an unprivileged user, writes to
files containing the IPs to be banned/unbanned which the script
parses and bans/unbans them in bulk using a single execution of
iptables command.
Implementation
Since this is done for testing purposes on a
minimal local system (Gentoo) which runs a custom kernel (no iptables
FILTER table support), a Debian VM will be created which will contain
the actual implementation of the project.
Hits to the VM will be done from the base machine.
Prepare VM --
$ cat /etc/gentoo-release
Gentoo Base System release 2.3
Create rootfs image from template --
qemu-img create -f qcow2 -o
cluster_size=512,lazy_refcounts=on,backing_file=Debian8NetworkedSSHRepoPackagesEnhancedUpdate.qcow
Debian8NetworkedSSHRepoPackagesEnhancedUpdate_fail2ban.qcow 20G
Load KVM modules (not loaded because of minimum
and highly customized OS) --
modprobe kvm_intel
Create tap device veth for the VM to connect to
the base machine --
modprobe tun;ip tuntap add mode tap veth
Assign ipv6 and ipv4 addresses on a temporary
basis --
ip a add fc00::1:1/112 dev veth;ip link set
dev veth up
ip a add 192.168.3.1/24 dev veth
Enable KSM --
echo 1 > /sys/kernel/mm/ksm/run
echo 30000 >
/sys/kernel/mm/ksm/sleep_millisecs
Start VM –
qemu-system-x86_64 -machine
accel=kvm,kernel_irqchip=on,mem-merge=on -drive
file=/home/de/large/VM_images/Debian8NetworkedSSHRepoPackagesEnhancedUpdate_fail2ban.qcow,id=centos,if=ide,media=disk,cache=unsafe,aio=threads,index=0
-vnc :1 -device e1000,id=ethnet,vlan=0 -net
tap,ifname=veth,script=no,downscript=no,vlan=0 -m 512 -smp 4
-daemonize -device e1000,id=inet,vlan=1,mac=52:54:0F:12:34:57 -net
user,id=internet,net=192.168.2.0/24,vlan=1
Login to the VM –
$ ssh root@fc00::1:2
root@fc00::1:2's password:
The programs included with the Debian
GNU/Linux system are free software;
the exact distribution terms for each program
are described in the
individual files in
/usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO
WARRANTY, to the extent
permitted by applicable law.
Last login: Thu Nov 17 12:03:07 2016 from
fc00::1:1
root@LINUXADMIN:~#
Configure ipv4 address for the VM
In /etc/network/interfaces –
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
allow-hotplug eth0
iface eth0 inet6 static
address fc00::1:2
netmask 112
#gateway fc00::1
# dns-* options are implemented by the
resolvconf package, if installed
#dns-nameservers fc00::1
#dns-search LinuxAdmin
iface eth0 inet static
address 192.168.3.2
netmask 24
auto eth1
iface eth1 inet dhcp
Bring up the changes via console –
ifdown eth0; ifup eth0
Setup nginx –
This setup is just for testing.
aptitude install nginx
The following NEW packages will be installed:
fontconfig-config{a} fonts-dejavu-core{a}
geoip-database{a} libfontconfig1{a} libgd3{a} libgeoip1{a}
libjbig0{a}
libjpeg62-turbo{a} libtiff5{a} libvpx1{a}
libxml2{a} libxpm4{a} libxslt1.1{a} nginx nginx-common{a}
nginx-full{a}
sgml-base{a} xml-core{a}
0 packages upgraded, 18 newly installed, 0 to
remove and 27 not upgraded.
Need to get 6,076 kB of archives. After
unpacking 16.7 MB will be used.
Do you want to continue? [Y/n/?]
systemctl enable nginx
Synchronizing state for nginx.service with
sysvinit using update-rc.d...
Executing /usr/sbin/update-rc.d nginx
defaults
Executing /usr/sbin/update-rc.d nginx enable
root@LINUXADMIN:~# systemctl start nginx
Setup virtualhost –
rm /etc/nginx/sites-enabled/default
Create /etc/nginx/conf.d/default.conf
server {
listen *:8080;
root /home/docroot;
}
Setup custom log format for nginx as per
requirement, tune it as per VM specs –
user www-data;
worker_processes 1;
pid /run/nginx.pid;
events {
worker_connections 768;
# multi_accept on;
}
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type
application/octet-stream;
##
# Logging Settings
##
log_format custom "[$time_local]
$remote_addr $status $request";
access_log /var/log/nginx/access.log
custom;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css
application/json application/javascript text/xml application/xml
application/xml+rss text/javascript;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
Test nginx and start –
nginx -t
nginx: the configuration file
/etc/nginx/nginx.conf syntax is ok
nginx: configuration file
/etc/nginx/nginx.conf test is successful
systemctl start nginx
Setup fail2ban –
aptitude install fail2ban
The following NEW packages will be installed:
fail2ban file{a} libmagic1{a}
libpython-stdlib{a} libpython2.7-minimal{a} libpython2.7-stdlib{a}
mime-support{a}
python{a} python-minimal{a}
python-pyinotify{a} python2.7{a} python2.7-minimal{a} whois{a}
0 packages upgraded, 13 newly installed, 0 to
remove and 0 not upgraded.
Need to get 4,687 kB of archives. After
unpacking 20.6 MB will be used.
Do you want to continue? [Y/n/?]
Configure fail2ban to start as unprivileged user –
mkdir /var/fail2ban
useradd -G adm fail2ban
chown fail2ban /var/fail2ban
Group adm is to allow fail2ban to read nginx
access logs.
Allow fail2ban user to write logs –
chown fail2ban /var/log/fail2ban.log
Modify fail2ban logrotation config to create new
empty log files with the correct permission –
/var/log/fail2ban.log {
weekly
rotate 4
compress
delaycompress
missingok
postrotate
fail2ban-client flushlogs 1>/dev/null
endscript
# If fail2ban runs as non-root it still
needs to have write access
# to logfiles.
# create 640 fail2ban adm
create 640 fail2ban adm
}
Create /etc/fail2ban/fail2ban.local to make
changes to allow running as the unprivileged user –
[Definition]
socket = /var/fail2ban/fail2ban.sock
pidfile = /var/fail2ban/fail2ban.pid
Make changes to /etc/default/fail2ban –
FAIL2BAN_USER="fail2ban"
Start and enable fail2ban –
systemctl start fail2ban
systemctl enable fail2ban
Synchronizing state for fail2ban.service with
sysvinit using update-rc.d...
Executing /usr/sbin/update-rc.d fail2ban
defaults
Executing /usr/sbin/update-rc.d fail2ban
enable
Create actions –
cat /etc/fail2ban/action.d/nginx.local
[Definition]
actionban = echo -n <ip>, >>
/var/fail2ban/ban
actionunban = echo -n <ip>, >>
/var/fail2ban/unban
As stated before, these actions append to a file
containing the IPs to be banned/unbanned as CSV values (that's why >>
has been used).
Create filters –
cat /etc/fail2ban/filter.d/nginx40{3,4}.local
[Definition]
failregex = ^\[ \+0530\] <HOST> 403 .*$
[Definition]
failregex = ^\[ \+0530\] <HOST> 404 .*$
The anchors (^, $) specify that the whole log has
been considered.
Create the jail –
cat /etc/fail2ban/jail.local
[nginx_403]
filter = nginx403
logpath = /var/log/nginx/access.log
action = nginx
findtime = 30
maxretry = 5
bantime = 300
usedns = no
enabled = true
[nginx_404]
filter = nginx404
logpath = /var/log/nginx/access.log
action = nginx
findtime = 30
maxretry = 50
bantime = 120
usedns = no
enabled = true
[ssh]
enabled = false
Since ssh service was not a part of the project,
but enabled in fail2ban by default on Debian, it has been disabled
here.
Make fail2ban read the changes and verify status
of jails –
fail2ban-client reload
fail2ban-client status
Status
|- Number of jail: 2
`- Jail list: nginx_404, nginx_403
Create iptables scripts to read files
/var/fail2ban/ban, /var/fail2ban/unban and add iptables rules.
cat /usr/bin/fail2ban_iptables.sh
#! /bin/bash
PATH="$PATH:/sbin"
if test -e /var/fail2ban/ban
then
iptables -A INPUT -s `cat
/var/fail2ban/ban | sed s/,$//` -j DROP
rm /var/fail2ban/ban
fi
if test -e /var/fail2ban/unban
then
iptables -D INPUT -s `cat
/var/fail2ban/unban | sed s/,$//` -j DROP
rm /var/fail2ban/unban
fi
Changes to PATH environment variables are there
since cron has a very minimal set of executable search paths.
Fix permissions of the file –
chmod 744 /usr/bin/fail2ban_iptables.sh
Make a cron job to execute the script as root –
root@LINUXADMIN:~# crontab -l | grep -v ^\#
* * * * * /usr/bin/fail2ban_iptables.sh
Testing –
2016-11-17 12:54:57,403
fail2ban.actions[1500]: WARNING [nginx_404] Ban 192.168.3.1
cat /var/fail2ban/ban
192.168.3.1,
After some time (once cron job runs) –
iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source
destination
DROP all -- 192.168.3.1
anywhere
Chain FORWARD (policy ACCEPT)
target prot opt source
destination
Chain OUTPUT (policy ACCEPT)
target prot opt source
destination
The same client on hitting the server –
wget --timeout 5 http://192.168.3.2/xyzz
--2016-11-17 12:55:04--
http://192.168.3.2/xyzz
Connecting to 192.168.3.2:80... failed:
Connection timed out.
Retrying.
--2016-11-17 12:55:10-- (try: 2)
http://192.168.3.2/xyzz
Connecting to 192.168.3.2:80... failed:
Connection timed out.
Retrying.
--2016-11-17 12:55:15-- (try: 3)
http://192.168.3.2/xyzz
Connecting to 192.168.3.2:80... failed:
Connection timed out.
Retrying.
--2016-11-17 12:55:22-- (try: 4)
http://192.168.3.2/xyzz
Connecting to 192.168.3.2:80... failed:
Connection timed out.
Retrying.
After 2 minutes –
2016-11-17 12:56:57,541
fail2ban.actions[1500]: WARNING [nginx_404] Unban 192.168.3.1
cat /var/fail2ban/unban
192.168.3.1,
After some time (once cron job runs) –
Chain INPUT (policy ACCEPT)
target prot opt source
destination
Chain FORWARD (policy ACCEPT)
target prot opt source
destination
Chain OUTPUT (policy ACCEPT)
target prot opt source
destination
