ipset, geoip, bogon, abuse ips and automation

General questions.
Post Reply
troll-op
Posts: 12
Joined: September 23rd, 2010, 8:35 am
Location: South Africa

ipset, geoip, bogon, abuse ips and automation

Post by troll-op » January 10th, 2017, 10:52 pm

Warning, I don’t write code or do creative thinking with I/O… I have a hammer and not afraid to use it. >:D
Having said that, this project is a personal one, useful, but should probably for now not be used on a productive server, and network environment. Sandbox, and Pentests only… all that was so no one wants to lynch me later.

Some time ago I posted a “ductape” solution to the geoip block question, before the geoip block became part of ipFire and before bad countries was a firewall group block option. If interested you can find the old bedtime story here viewtopic.php?t=10858#p73914

In short it was a null route (blackhole) entry in /etc/sysconfig/rc.local that would block bidirectional chatter on the WAN port. Even though it did the trick, it was a pain in the ath to maintain the lists, as I’m not the most brightest crayon when it comes to scripting or automation.

On the other hand, I’m a bit of a lazy bastard and don’t like doing things manually, when I can train the silicon box to do it for me. So instead of ip route add blackhole xyz, I’ve looked at ipset a bit more closely. The advantage of using ipset compared to most other options, is no matter how large your block lists become, you will not run out of resources, and you keep your iptables light and nimble. ;)

I’m sure that there are more elegant ways of doing this, but there is method to my madness, and it works… with a few nigglies that need to be iron out. I’m 100% sure I’ve buggard up something, please feel free to give pointers and corrections.

For this example we will block China, and use the bad ip list published by FireHol.

Step 1:
You will need to create network sets that will be used to populate with IPs (single and CIDR) to be blocked, dropped, rejected, or what have you.

Code: Select all

ipset create blacklist_china hash:net
ipset create firehol_level1 hash:net
You will need to make sure that after a reboot, things get remembered so a

Code: Select all

ipset save > /etc/ipset/ipset.conf
will help that along.
If you know type ipset -n list you should see your two lists names pop up.

Step 2:
You need to add two scripts to /etc/sysconfig that will populate the sets with the relevant ip sets. Or at least that's where I placed them.

Code: Select all

vi /etc/sysconfig/blacklist_china.sh

Code: Select all

#!/bin/bash

NETGROUP="blacklist_china"

>/tmp/block-cn
curl -s http://www.ipdeny.com/ipblocks/data/countries/cn.zone | grep '^[0-9]' | sed -e 's/;.*//' | sort | uniq >> /tmp/block-cn

sudo ipset -q -L $NETGROUP > /dev/null 2>&1
if [ "$?" != 0 ]; then
   echo "firewall network group $NETGROUP doesn't exist yet"
   exit 1
fi

NEWGROUP=$NETGROUP-$$
sudo ipset create $NEWGROUP hash:net
if [ "$?" != 0 ]; then
  echo "There was an error trying to create temporary set"
  exit 1
fi

count=0;
for i in `cat /tmp/block-cn`;
do
  sudo ipset -q -A $NEWGROUP $i
  if [ "$?" != 0 ]; then
     echo "There was an error trying to add $i"
     exit 1
  fi
  let "count++"
done
sudo ipset swap $NEWGROUP $NETGROUP
if [ "$?" != 0 ]; then
  echo "There was an error trying to swap temporary set"
  exit 1
fi
sudo ipset destroy $NEWGROUP
rm /tmp/block-cn
echo Added $count entries to $NETGROUP;
exit 0
and your second script

Code: Select all

vi /etc/sysconfig/firehol_level1.sh

Code: Select all

#!/bin/bash

NETGROUP="firehol_level1"

>/tmp/block
curl -s https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level1.netset | grep '^[0-9]' | sed -e 's/;.*//' | sort | uniq >> /tmp/block

sudo ipset -q -L $NETGROUP > /dev/null 2>&1
if [ "$?" != 0 ]; then
   echo "firewall network group $NETGROUP doesn't exist yet"
   exit 1
fi

NEWGROUP=$NETGROUP-$$
sudo ipset create $NEWGROUP hash:net
if [ "$?" != 0 ]; then
  echo "There was an error trying to create temporary set"
  exit 1
fi

count=0;
for i in `cat /tmp/block`;
do
  sudo ipset -q -A $NEWGROUP $i
  if [ "$?" != 0 ]; then
     echo "There was an error trying to add $i"
     exit 1
  fi
  let "count++"
done
sudo ipset swap $NEWGROUP $NETGROUP
if [ "$?" != 0 ]; then
  echo "There was an error trying to swap temporary set"
  exit 1
fi
sudo ipset destroy $NEWGROUP
rm /tmp/block
echo Added $count entries to $NETGROUP;
exit 0
Change them to executable

Code: Select all

cd /etc/sysconfig
chmod +x blacklist_china.sh firehol_level1.sh
At this point you could run the scripts to populate the sets, but whats the point if they cannot be used yet? So let’s skip that for now…

Step 3:
You want to make the following changes to rc.local and firewall.local
And this is probably where things could be more elegant….

Code: Select all

vi /etc/sysconfig/rc.local
I’m not going to post the whole file, as I’ve mucked about with it and have the old stuff still # out in it, but if you have a brand spanking new one, you want to add the following lines in…

Code: Select all

#Restart and restore ipset lists
ipset restore < /etc/ipset/ipset.conf
Now you could also add in

Code: Select all

# Refresh bad IP lists
/etc/sysconfig/firehol_level1.sh
/etc/sysconfig/blacklist_china.sh
...but I prefer letting crontab execute them, and not have them run at boot time. Your choice.
Next you need to tell the firewall what to do with these new VIP lists, the bouncer after all needs instructions when facial recognition fails. :D

Code: Select all

vi /etc/sysconfig/firewall.local
This file should be stock standard for most, unless you have edited it in the past, so here is what mine looks like, after I panelbeated it.

Code: Select all

#!/bin/sh
# Used for private firewall rules

# See how we were called.
case "$1" in
  start)
        ## add your 'start' rules here
        # IPSET FW entries in start
        # IPSET add rules for CIDR list
        /sbin/iptables -I CUSTOMFORWARD -i red0 -m set --match-set firehol_level1 dst -j DROP
        /sbin/iptables -I CUSTOMINPUT -i red0 -m set --match-set firehol_level1 src -j DROP
        /sbin/iptables -I CUSTOMOUTPUT -i red0 -m set --match-set firehol_level1 dst -j DROP
        /sbin/iptables -I CUSTOMFORWARD -i red0 -m set --match-set blacklist_china dst -j DROP
        /sbin/iptables -I CUSTOMINPUT -i red0 -m set --match-set blacklist_china src -j DROP
        /sbin/iptables -I CUSTOMOUTPUT -i red0 -m set --match-set blacklist_china dst -j DROP 
        ;;
  stop)
        ## add your 'stop' rules here
        # IPSET flushing related chains
        /sbin/iptable -F CUSTOMFORWARD
        /sbin/iptable -F CUSTOMINPUT
        /sbin/iptable -F CUSTOMOUTPUT
        ;;
  reload)
        $0 stop
        $0 start
        ## add your 'reload' rules here
        ;;
  *)
        echo "Usage: $0 {start|stop|reload}"
        ;;
esac

Please note there is one blank line after esac, this seems to be important for some obscure reason. :o
You don’t need the -i red0, but as the firehol_level1 list also contains BOGON IP, and I don’t feel like removing some to accommodate the network layouts I deal with, I thought it best to let iptables know that this is for external abuse only and not the LAN or VPN connections.

Step 4
If you don’t want the scripts to update the lists you can skip this part, but I highly recommend you do this, otherwise what’s the point of automating things, wouldn’t you say? ???

Copy or move your scripts from /etc/sysconfig -> one of the cron folders. My recommendation would be to the daily one, as the country IPs don’t change that often, and we don’t want to overload the FireHOL site with hourly requests when daily will do.
So your target folder is /etc/fcron.daily/

Step 5
You can now wait for the scripts to run on their own… or if you are an impatient little padawan 8)

Code: Select all

/etc/sysconfig/firehol_level1.sh
/etc/sysconfig/blacklist_china.sh
It will take some time for the lists to download, shuffle and mix, before the deal.
All depends on your machine resources, and connection speed.
But once done you can test and see for example instead of

Code: Select all

ipset -n list
try

Code: Select all

ipset -t list
this should give you a bit of nice useless information… just kidding. The two listed sets should be different in size, if not you know something went wrong. Provided you don’t block your ipFire from wget things or internet access you should be fine though.

You can also test and see if things are all above board, and good to go, for example

Code: Select all

ipset test blacklist_china 58.83.128.25
or

Code: Select all

ipset test firehol_level1 192.168.1.1
should tell you either something is or is NOT in a set. For the above they should both say
xyz is in set NAME (although the china one may change in future).

Right that’s it. Go forth and multiple. ;D
Constructive feedback, is most appreciated. :)

PS the lists for the scripts can obviously be modded to block other countries etc.
The sources I used, if unclear, are:
FireHol http://iplists.firehol.org/
IP Deny http://www.ipdeny.com/ipblocks/

Good alternatives, but are included in firewall
Ransomeware https://ransomwaretracker.abuse.ch/blocklist/
Zeus Trackre https://zeustracker.abuse.ch/
SSLBL https://sslbl.abuse.ch/
Feodo Tracker https://feodotracker.abuse.ch/
MLBL https://zeltser.com/malicious-ip-blocklists/

Really, the lists are endless, so I'm stopping with those. The first two should however do to cover most as firehol includes a good number of these, and created a combined (duplicates removed) list.

troll-op
Posts: 12
Joined: September 23rd, 2010, 8:35 am
Location: South Africa

Re: ipset, geoip, bogon, abuse ips and automation

Post by troll-op » January 10th, 2017, 11:01 pm

I should mention that the above is not purely my work. I've scrounged information together from quite a few sites, including here, UBNT, and other open source projects. I hammered them together into something that actually works. Yip I'm surprised as you are. ^-^

I would like to thank all those that contributed with their posts, but I never took note of the authors, which I now feel sorry about.

So thanks to all who provided their greymatter, and those who probably still will in the near future :)

User avatar
Roberto Peña
Posts: 761
Joined: July 16th, 2014, 3:56 pm
Location: Bilbao (España)
Contact:

Re: ipset, geoip, bogon, abuse ips and automation

Post by Roberto Peña » January 19th, 2017, 7:57 am

Thanks troll-op. I find it a very good contribution.

I will try and comment if I encounter any problem or something to improve.

Greetings.
Image
Image

╔════════════════════════════════════════════════╗
Donate to improve IPFire: https://www.ipfire.org/donate
╚════════════════════════════════════════════════╝

ummeegge
Community Developer
Community Developer
Posts: 5001
Joined: October 9th, 2010, 10:00 am

Re: ipset, geoip, bogon, abuse ips and automation

Post by ummeegge » January 19th, 2017, 11:06 am

Good job troll-op,
as a beside one...
Regarding your lists from firehol, we have had Costa (developer from Firehol) in the forum and he suggested to not use the lists from the Github repo causing the update cycles
I strongly suggest not to use files downloaded from the github repo, in production systems. They might not be updated regularly.
<-- is from --> viewtopic.php?f=50&t=15124#p91681 .

as another one :P , longer time ago i have had a list where 192.168/16 addresses appears and my network matched fully into it, after loooong time searches (use also FORWARD chains and IPFire on red is behind another router) i found that problem and tried to prevent those accidents via script (sorted out OpenVPN, DNS, all IPFire network addresses) but i know you know about that problem ;) but may an idea to think again about it ?!...

and another one :D ,
if you are interested in iprange or update-ipsets you can find a 32 bit version in here --> http://people.ipfire.org/~ummeegge/ipset/ , if you need 64 bit binaries say something O0 .

Cheers,

UE

EDIT:
--> http://wiki.ipfire.org/en/configuration/firewall/ipset <-- also an informative one in my opinion ^-^ .
Image
Image

troll-op
Posts: 12
Joined: September 23rd, 2010, 8:35 am
Location: South Africa

Re: ipset, geoip, bogon, abuse ips and automation

Post by troll-op » January 24th, 2017, 7:58 am

Hi ummeegge, and Roberto
Thanks for the feedback. I'll have a look at your script as well.
Regarding the Firehol script, I found this out as well, was already formulating me “fix” or work around for it, see below :)

Side note it's only been running for a few days, but seems stable and has not blocked anything it was not supposed to. So no issues with the LAN or VPN. Still testing ...early days

8< —— Snip delayed “fix” begin——

As predicted there is a flaw in my approach.
The firehol_level1 contains the full bogon list, and all are treated to the same DROP approach. This has caused weird and wonderful problems inside the LAN.

I’ve taken the information from their site, and split up the whole listing, using all except for fullbogon, and created their individual scripts.
Advantage, a bit more control, disadvantage, might contain duplicate IPs, but this should not be the end of the world.

To correct the above you would need to flush, and delete the set.

Code: Select all

ipset flush firehol_level1
ipset destroy firehol_level1
mv /etc/ipset/ipset.conf /etc/ipset/ipset.conf.old
reboot
If you now do a

Code: Select all

ipset -n list
You should only have the CN set, or if you did not use it, blank.

First create the new set names.

Code: Select all

ipset create bambenek_c2 hash:ip
ipset create bruteforcelogin hash:net
ipset create dshield_1d hash:net
ipset create feodo hash:ip
ipset create palevo hash:ip
ipset create spamhaus_drop hash:net
ipset create spamhaus_edrop hash:net
ipset create sslbl hash:ip
ipset create zeus_badips hash:ip
Please note that the hash is not always net, but in some cases ip.
I’ve also added brutforcelogin as a new option from https://www.blocklist.de/en/export.html
just because I can :P
Actually it’s to show an alternative, and what changes in the script. I’m sure it was not needed, but for those still new to this, it could be helpful in understanding how things hang together.

If you now do a

Code: Select all

ipset -n list
You should should get
bambenek_c2
bruteforcelogin
dshield_1d
feodo
palevo
spamhaus_drop
spamhaus_edrop
sslbl
zeus_badips
as a response.

Next save the new sets,

Code: Select all

ipset save > /etc/ipset/ipset.conf
Now create the scripts in /etc/fcron.daily/

Code: Select all

vi bambenek_c2.sh

Code: Select all

#!/bin/bash

NETGROUP="bambenek_c2"

>/tmp/block-bambenek_c2
curl -s https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/bambenek_c2.ipset | grep '^[0-9]' | sed -e 's/;.*//' | sort | uniq >> /tmp/block-bambenek_c2

sudo ipset -q -L $NETGROUP > /dev/null 2>&1
if [ "$?" != 0 ]; then
   echo "firewall network group $NETGROUP doesn't exist yet"
   exit 1
fi

NEWGROUP=$NETGROUP-$$
sudo ipset create $NEWGROUP hash:ip
if [ "$?" != 0 ]; then
  echo "There was an error trying to create temporary set"
  exit 1
fi

count=0;
for i in `cat /tmp/block-bambenek_c2`;
do
  sudo ipset -q -A $NEWGROUP $i
  if [ "$?" != 0 ]; then
     echo "There was an error trying to add $i"
     exit 1
  fi
  let "count++"
done
sudo ipset swap $NEWGROUP $NETGROUP
if [ "$?" != 0 ]; then
  echo "There was an error trying to swap temporary set"
  exit 1
fi
sudo ipset destroy $NEWGROUP
rm /tmp/block-bambenek_c2
echo Added $count entries to $NETGROUP;
exit 0

Code: Select all

vi bruteforcelogin.sh

Code: Select all

#!/bin/bash

NETGROUP="bruteforcelogin"

>/tmp/bruteforcelogin
curl -s https://lists.blocklist.de/lists/bruteforcelogin.txt | grep '^[0-9]' | sed -e 's/;.*//' | sort | uniq >> /tmp/bruteforcelogin

sudo ipset -q -L $NETGROUP > /dev/null 2>&1
if [ "$?" != 0 ]; then
   echo "firewall network group $NETGROUP doesn't exist yet"
   exit 1
fi

NEWGROUP=$NETGROUP-$$
sudo ipset create $NEWGROUP hash:net
if [ "$?" != 0 ]; then
  echo "There was an error trying to create temporary set"
  exit 1
fi

count=0;
for i in `cat /tmp/bruteforcelogin`;
do
  sudo ipset -q -A $NEWGROUP $i
  if [ "$?" != 0 ]; then
     echo "There was an error trying to add $i"
     exit 1
  fi
  let "count++"
done
sudo ipset swap $NEWGROUP $NETGROUP
if [ "$?" != 0 ]; then
  echo "There was an error trying to swap temporary set"
  exit 1
fi
sudo ipset destroy $NEWGROUP
rm /tmp/bruteforcelogin
echo Added $count entries to $NETGROUP;
exit 0

Code: Select all

vi dshield_1d.sh

Code: Select all

#!/bin/bash

NETGROUP="dshield_1d"

>/tmp/block-dshield_1d
curl -s https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/dshield_1d.netset | grep '^[0-9]' | sed -e 's/;.*//' | sort | uniq >> /tmp/block-dshield_1d

sudo ipset -q -L $NETGROUP > /dev/null 2>&1
if [ "$?" != 0 ]; then
   echo "firewall network group $NETGROUP doesn't exist yet"
   exit 1
fi

NEWGROUP=$NETGROUP-$$
sudo ipset create $NEWGROUP hash:net
if [ "$?" != 0 ]; then
  echo "There was an error trying to create temporary set"
  exit 1
fi

count=0;
for i in `cat /tmp/block-dshield_1d`;
do
  sudo ipset -q -A $NEWGROUP $i
  if [ "$?" != 0 ]; then
     echo "There was an error trying to add $i"
     exit 1
  fi
  let "count++"
done
sudo ipset swap $NEWGROUP $NETGROUP
if [ "$?" != 0 ]; then
  echo "There was an error trying to swap temporary set"
  exit 1
fi
sudo ipset destroy $NEWGROUP
rm /tmp/block-dshield_1d
echo Added $count entries to $NETGROUP;
exit 0

Code: Select all

vi feodo.sh

Code: Select all

#!/bin/bash

NETGROUP="feodo"

>/tmp/block-feodo
curl -s https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/feodo.ipset | grep '^[0-9]' | sed -e 's/;.*//' | sort | uniq >> /tmp/block-feodo

sudo ipset -q -L $NETGROUP > /dev/null 2>&1
if [ "$?" != 0 ]; then
   echo "firewall network group $NETGROUP doesn't exist yet"
   exit 1
fi

NEWGROUP=$NETGROUP-$$
sudo ipset create $NEWGROUP hash:ip
if [ "$?" != 0 ]; then
  echo "There was an error trying to create temporary set"
  exit 1
fi

count=0;
for i in `cat /tmp/block-feodo`;
do
  sudo ipset -q -A $NEWGROUP $i
  if [ "$?" != 0 ]; then
     echo "There was an error trying to add $i"
     exit 1
  fi
  let "count++"
done
sudo ipset swap $NEWGROUP $NETGROUP
if [ "$?" != 0 ]; then
  echo "There was an error trying to swap temporary set"
  exit 1
fi
sudo ipset destroy $NEWGROUP
rm /tmp/block-feodo
echo Added $count entries to $NETGROUP;
exit 0

Code: Select all

vi palevo.sh

Code: Select all

#!/bin/bash

NETGROUP="palevo"

>/tmp/block-palevo
curl -s https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/palevo.ipset | grep '^[0-9]' | sed -e 's/;.*//' | sort | uniq >> /tmp/block-palevo

sudo ipset -q -L $NETGROUP > /dev/null 2>&1
if [ "$?" != 0 ]; then
   echo "firewall network group $NETGROUP doesn't exist yet"
   exit 1
fi

NEWGROUP=$NETGROUP-$$
sudo ipset create $NEWGROUP hash:ip
if [ "$?" != 0 ]; then
  echo "There was an error trying to create temporary set"
  exit 1
fi

count=0;
for i in `cat /tmp/block-palevo`;
do
  sudo ipset -q -A $NEWGROUP $i
  if [ "$?" != 0 ]; then
     echo "There was an error trying to add $i"
     exit 1
  fi
  let "count++"
done
sudo ipset swap $NEWGROUP $NETGROUP
if [ "$?" != 0 ]; then
  echo "There was an error trying to swap temporary set"
  exit 1
fi
sudo ipset destroy $NEWGROUP
rm /tmp/block-palevo
echo Added $count entries to $NETGROUP;
exit 0

Code: Select all

vi spamhaus_drop.sh

Code: Select all

#!/bin/bash

NETGROUP="spamhaus_drop"

>/tmp/block-spamhaus_drop
curl -s https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/spamhaus_drop.netset | grep '^[0-9]' | sed -e 's/;.*//' | sort | uniq >> /tmp/block-spamhaus_drop

sudo ipset -q -L $NETGROUP > /dev/null 2>&1
if [ "$?" != 0 ]; then
   echo "firewall network group $NETGROUP doesn't exist yet"
   exit 1
fi

NEWGROUP=$NETGROUP-$$
sudo ipset create $NEWGROUP hash:net
if [ "$?" != 0 ]; then
  echo "There was an error trying to create temporary set"
  exit 1
fi

count=0;
for i in `cat /tmp/block-spamhaus_drop`;
do
  sudo ipset -q -A $NEWGROUP $i
  if [ "$?" != 0 ]; then
     echo "There was an error trying to add $i"
     exit 1
  fi
  let "count++"
done
sudo ipset swap $NEWGROUP $NETGROUP
if [ "$?" != 0 ]; then
  echo "There was an error trying to swap temporary set"
  exit 1
fi
sudo ipset destroy $NEWGROUP
rm /tmp/block-spamhaus_drop
echo Added $count entries to $NETGROUP;
exit 0

Code: Select all

vi spamhaus_edrop.sh

Code: Select all

#!/bin/bash

NETGROUP="spamhaus_edrop"

>/tmp/block-spamhaus_edrop
curl -s https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/spamhaus_edrop.netset | grep '^[0-9]' | sed -e 's/;.*//' | sort | uniq >> /tmp/block-spamhaus_edrop

sudo ipset -q -L $NETGROUP > /dev/null 2>&1
if [ "$?" != 0 ]; then
   echo "firewall network group $NETGROUP doesn't exist yet"
   exit 1
fi

NEWGROUP=$NETGROUP-$$
sudo ipset create $NEWGROUP hash:net
if [ "$?" != 0 ]; then
  echo "There was an error trying to create temporary set"
  exit 1
fi

count=0;
for i in `cat /tmp/block-spamhaus_edrop`;
do
  sudo ipset -q -A $NEWGROUP $i
  if [ "$?" != 0 ]; then
     echo "There was an error trying to add $i"
     exit 1
  fi
  let "count++"
done
sudo ipset swap $NEWGROUP $NETGROUP
if [ "$?" != 0 ]; then
  echo "There was an error trying to swap temporary set"
  exit 1
fi
sudo ipset destroy $NEWGROUP
rm /tmp/block-spamhaus_edrop
echo Added $count entries to $NETGROUP;
exit 0

Code: Select all

vi sslbl.sh

Code: Select all

#!/bin/bash

NETGROUP="sslbl"

>/tmp/block-sslbl
curl -s https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/sslbl.ipset | grep '^[0-9]' | sed -e 's/;.*//' | sort | uniq >> /tmp/block-sslbl

sudo ipset -q -L $NETGROUP > /dev/null 2>&1
if [ "$?" != 0 ]; then
   echo "firewall network group $NETGROUP doesn't exist yet"
   exit 1
fi

NEWGROUP=$NETGROUP-$$
sudo ipset create $NEWGROUP hash:ip
if [ "$?" != 0 ]; then
  echo "There was an error trying to create temporary set"
  exit 1
fi

count=0;
for i in `cat /tmp/block-sslbl`;
do
  sudo ipset -q -A $NEWGROUP $i
  if [ "$?" != 0 ]; then
     echo "There was an error trying to add $i"
     exit 1
  fi
  let "count++"
done
sudo ipset swap $NEWGROUP $NETGROUP
if [ "$?" != 0 ]; then
  echo "There was an error trying to swap temporary set"
  exit 1
fi
sudo ipset destroy $NEWGROUP
rm /tmp/block-sslbl
echo Added $count entries to $NETGROUP;
exit 0

Code: Select all

vi zeus_badips.sh

Code: Select all

#!/bin/bash

NETGROUP="zeus_badips"

>/tmp/block-zeus_badips
curl -s https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/zeus_badips.ipset | grep '^[0-9]' | sed -e 's/;.*//' | sort | uniq >> /tmp/block-zeus_badips

sudo ipset -q -L $NETGROUP > /dev/null 2>&1
if [ "$?" != 0 ]; then
   echo "firewall network group $NETGROUP doesn't exist yet"
   exit 1
fi

NEWGROUP=$NETGROUP-$$
sudo ipset create $NEWGROUP hash:ip
if [ "$?" != 0 ]; then
  echo "There was an error trying to create temporary set"
  exit 1
fi

count=0;
for i in `cat /tmp/block-zeus_badips`;
do
  sudo ipset -q -A $NEWGROUP $i
  if [ "$?" != 0 ]; then
     echo "There was an error trying to add $i"
     exit 1
  fi
  let "count++"
done
sudo ipset swap $NEWGROUP $NETGROUP
if [ "$?" != 0 ]; then
  echo "There was an error trying to swap temporary set"
  exit 1
fi
sudo ipset destroy $NEWGROUP
rm /tmp/block-zeus_badips
echo Added $count entries to $NETGROUP;
exit 0
Make the scripts executalble

Code: Select all

chmod +x *.sh
should do the trick if you are in the same folder you saved them in.

Once the scripts are in place edit /etc/sysconfig/firewall.local

Code: Select all

#!/bin/sh
# Used for private firewall rules

# See how we were called.
case "$1" in
  start)
        ## add your 'start' rules here
        # IPSET FW entries in start
        # IPSET add rules for CIDR list
        /sbin/iptables -I CUSTOMFORWARD -i red0 -m set --match-set bambenek_c2 dst -j DROP
        /sbin/iptables -I CUSTOMINPUT -i red0 -m set --match-set bambenek_c2 src -j DROP
        /sbin/iptables -I CUSTOMOUTPUT -i red0 -m set --match-set bambenek_c2 dst -j DROP
        /sbin/iptables -I CUSTOMFORWARD -i red0 -m set --match-set bruteforcelogin dst -j DROP
        /sbin/iptables -I CUSTOMINPUT -i red0 -m set --match-set bruteforcelogin src -j DROP
        /sbin/iptables -I CUSTOMOUTPUT -i red0 -m set --match-set bruteforcelogin dst -j DROP
        /sbin/iptables -I CUSTOMFORWARD -i red0 -m set --match-set dshield_1d dst -j DROP
        /sbin/iptables -I CUSTOMINPUT -i red0 -m set --match-set dshield_1d src -j DROP
        /sbin/iptables -I CUSTOMOUTPUT -i red0 -m set --match-set dshield_1d dst -j DROP
        /sbin/iptables -I CUSTOMFORWARD -i red0 -m set --match-set feodo dst -j DROP
        /sbin/iptables -I CUSTOMINPUT -i red0 -m set --match-set feodo src -j DROP
        /sbin/iptables -I CUSTOMOUTPUT -i red0 -m set --match-set feodo dst -j DROP
        /sbin/iptables -I CUSTOMFORWARD -i red0 -m set --match-set palevo dst -j DROP
        /sbin/iptables -I CUSTOMINPUT -i red0 -m set --match-set palevo src -j DROP
        /sbin/iptables -I CUSTOMOUTPUT -i red0 -m set --match-set palevo dst -j DROP
        /sbin/iptables -I CUSTOMFORWARD -i red0 -m set --match-set spamhaus_drop dst -j DROP
        /sbin/iptables -I CUSTOMINPUT -i red0 -m set --match-set spamhaus_drop src -j DROP
        /sbin/iptables -I CUSTOMOUTPUT -i red0 -m set --match-set spamhaus_drop dst -j DROP
        /sbin/iptables -I CUSTOMFORWARD -i red0 -m set --match-set spamhaus_edrop dst -j DROP
        /sbin/iptables -I CUSTOMINPUT -i red0 -m set --match-set spamhaus_edrop src -j DROP
        /sbin/iptables -I CUSTOMOUTPUT -i red0 -m set --match-set spamhaus_edrop dst -j DROP
        /sbin/iptables -I CUSTOMFORWARD -i red0 -m set --match-set sslbl dst -j DROP
        /sbin/iptables -I CUSTOMINPUT -i red0 -m set --match-set sslbl src -j DROP
        /sbin/iptables -I CUSTOMOUTPUT -i red0 -m set --match-set sslbl dst -j DROP
        /sbin/iptables -I CUSTOMFORWARD -i red0 -m set --match-set zeus_badips dst -j DROP
        /sbin/iptables -I CUSTOMINPUT -i red0 -m set --match-set zeus_badips src -j DROP
        /sbin/iptables -I CUSTOMOUTPUT -i red0 -m set --match-set zeus_badips dst -j DROP
        # Bad Country CIDR lists
        #/sbin/iptables -I CUSTOMFORWARD -i red0 -m set --match-set blacklist_china dst -j DROP
        #/sbin/iptables -I CUSTOMINPUT -i red0 -m set --match-set blacklist_china src -j DROP
        #/sbin/iptables -I CUSTOMOUTPUT -i red0 -m set --match-set blacklist_china dst -j DROP 
        ;;
  stop)
        ## add your 'stop' rules here
        # IPSET flushing related chains
        /sbin/iptable -F CUSTOMFORWARD
        /sbin/iptable -F CUSTOMINPUT
        /sbin/iptable -F CUSTOMOUTPUT
        ;;
  reload)
        $0 stop
        $0 start
        ## add your 'reload' rules here
        ;;
  *)
        echo "Usage: $0 {start|stop|reload}"
        ;;
esac

And if you still would like to block BOGON, then I recommend to edit /etc/sysconfig/rc.local

Code: Select all

# Blackhole route entries - BOGON aggregated 
# source - https://www.team-cymru.org/Services/Bogons/bogon-bn-agg.txt
# DISABLE IF FireHOL SCRIPT IS ENABLED
ip route add blackhole 0.0.0.0/8
ip route add blackhole 10.0.0.0/8
ip route add blackhole 100.64.0.0/10
ip route add blackhole 127.0.0.0/8
ip route add blackhole 169.254.0.0/16
ip route add blackhole 172.16.0.0/12
ip route add blackhole 192.0.0.0/24
ip route add blackhole 192.0.2.0/24
ip route add blackhole 192.168.0.0/16
ip route add blackhole 198.18.0.0/15
ip route add blackhole 198.51.100.0/24
ip route add blackhole 203.0.113.0/24
ip route add blackhole 224.0.0.0/3
For me internal issues arose from the bogon listing in the firehol_level1, and the way the iptables handled it. Not having them in the list, and being able to null route them instead fixed the issue.

If you moved the scripts into fcron.daily or what-have-you it will run by itself, but you can always run it manually if you are an impatient padawan.

8< —— Snip delayed “fix” end——

Ok, not sure if I missed something now... Kindly let me know if you find a flaw or hiccup in my little frankenstein. ;D

EDIT: fixed some errors in the .sh files and brutforce set
Last edited by troll-op on January 24th, 2017, 10:14 pm, edited 1 time in total.

ummeegge
Community Developer
Community Developer
Posts: 5001
Joined: October 9th, 2010, 10:00 am

Re: ipset, geoip, bogon, abuse ips and automation

Post by ummeegge » January 24th, 2017, 6:05 pm

Heyho,
troll-op wrote:Please note that the hash is not always net, but in some cases ip.
have had that too, sometimes there where some CIDRs in the IP lists and some IPs in the CIDR lists, as an idea for this...
You can possibly download all that lists and with a little RegEX HongKongFU and grep (let´s say, use the force luke) you can put them in two directory categories (CIDRs and IPs), and in a double shot for this one -->
troll-op wrote: disadvantage, might contain duplicate IPs, but this should not be the end of the world.
you can sort doubles (-u option for sort <--> uniq) out and bring them in a list structure with a command like this e.g.
For CIDRs:

Code: Select all

# grep CIDRs and sort and make them uniq
cat ${ALL_LISTS} | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}/[0-9]{1,2}" | sort -nu > ${CIDRDIRECTORY}
For IPs:

Code: Select all

# grep IPs and sort and make them uniq
cat ${ALL_LISTS} | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" | sort -nu > ${IPDIRECTORY};
which would then also mean, check all lists for both.

Another advantage might be that you have from now on only two IPSets (CIDR and IP) (RANGEs are another one in al lot of lists but this might be another theme) which means if you block FORWARD, OUTGOING and INPUT (good idea in my opinion) you need from now on 6 lines in firewall.local for the start and 3 lines (for flushing the chains as you did it) in the stop section to bring it all together.
<-- i think you will need to reload the firewall.local after every fcron update...

May all addresses can then also be downloaded/sorted/integrated over one script ?!
Spoiling all that data then through another filter of possible address conflicts might possibly be not that hard ?!?!
... let me say nice work what you did until now and thanks for publishing your ideas here for others 8) .

Cheers,

UE
Image
Image

troll-op
Posts: 12
Joined: September 23rd, 2010, 8:35 am
Location: South Africa

Re: ipset, geoip, bogon, abuse ips and automation

Post by troll-op » January 24th, 2017, 10:20 pm

Howzit ummeegge

I’ve had a look at your links… had I found them, especially the update-ipsets I would probably not have bothered with all of this. You clearly understand this a bit better than I do. Is there any need for all I’ve done, or did I just saddle the horse from the otherside? :o

It all started with me panel-beating my Ubiquiti USGW to accept some block-lists, thinking null routes are to cumbrous and resource hungry, discovering ipset, and it took on a life of its own from there. Porting it to ipFire seemed like the next logic step.

I understand why you wish to shrink the number of sets down to two, seems like a good approach. The only problem I see with it is, if there is a hiccup with one of the resource downloads, that tracing it will become difficult.
For example my bogon LAN issue was clear that 192.168.0.0/16 was the problem, and it was contained in the initial firehol_level1 set. This lead me to break it down to all the make up sets, as listed on the FireHOL site, excluding the fullbogon list by Team Cymru. Was this all in one packaged set net or ip, it would have been difficult to figure out which source was to blame.
… am I over thinking or misunderstanding what you are trying to tell me with the CIDR and IP split? ???

Would the /etc/sysconfig/firewall.local reload need to happen after each cron update? Or only the initial one?
I've not had any problems on the machine running all this, have not restarted or reloaded the firewall.local. How would I know if the latest lists are being used?

Side note:
I’ve fixed a few spelling mistakes in the above post and also a minor balls-up in the .sh scripts.
All contained

Code: Select all

sudo ipset create $NEWGROUP hash:net
where in some cases its supposed to be

Code: Select all

sudo ipset create $NEWGROUP hash:ip
Also

Code: Select all

ipset create bruteforcelogin hash:ip
is actually

Code: Select all

ipset create bruteforcelogin hash:net 
or it can also be hash:ip,net
I fixed it on my machine, and for some obscure reason did not fix my notes. Yip my bad ::)

ummeegge
Community Developer
Community Developer
Posts: 5001
Joined: October 9th, 2010, 10:00 am

Re: ipset, geoip, bogon, abuse ips and automation

Post by ummeegge » January 26th, 2017, 3:12 pm

cruezi well troll-op,
troll-op wrote: I’ve had a look at your links… had I found them, especially the update-ipsets I would probably not have bothered with all of this. You clearly understand this a bit better than I do. Is there any need for all I’ve done, or did I just saddle the horse from the otherside? :o
i don´t think so, there are always different ways to Rom and it is always nice to fiddle out some new stuff.
troll-op wrote: It all started with me panel-beating my Ubiquiti USGW to accept some block-lists, thinking null routes are to cumbrous and resource hungry, discovering ipset, and it took on a life of its own from there. Porting it to ipFire seemed like the next logic step.
Well done.
troll-op wrote:I understand why you wish to shrink the number of sets down to two, seems like a good approach. The only problem I see with it is, if there is a hiccup with one of the resource downloads, that tracing it will become difficult.
For example my bogon LAN issue was clear that 192.168.0.0/16 was the problem, and it was contained in the initial firehol_level1 set. This lead me to break it down to all the make up sets, as listed on the FireHOL site, excluding the fullbogon list by Team Cymru. Was this all in one packaged set net or ip, it would have been difficult to figure out which source was to blame.
… am I over thinking or misunderstanding what you are trying to tell me with the CIDR and IP split? ???
OK let´s step into the rabbit hole :D . Do we need private IP ranges in general in our lists, also there is not really the need to block our DNS, OpenVPN´s, localhost, broad- or multicast nor is it a nice one to have 0.0.0.0/8 in a firewall rule. To sort the static ones out we can use sed , to sort the dynamic ones out we need to burrow a little in the settings.
How does it looks like ?
Dynamic ones may like this ?

Code: Select all

SET="/var/ipfire/ethernet/settings";
OVPNSUB="/var/ipfire/ovpn/server.conf";


## Investigate system addresses to prevent potential blocks
# LAN, WLAN, DMZ, DNS and OpenVPN addresses
USEDADDRESSES=$(awk -F'=' '/GREEN_ADDRESS/ || /BLUE_ADDRESS/ || /RED_ADDRESS/ || /ORANGE_ADDRESS/ { print $2 }' ${SET} | cut -d'.' -f1,2 && \
awk -F'=' '/DNS1=/ || /DNS2=/ { print $2 }' ${SET} && \
awk '/server / || /route / { print $2 }' ${OVPNSUB} | sed 's/.0$//g')
OWNAD=$(echo "${USEDADDRESSES}" | tr ' ' '\n' | sort -nu);

for i in ${OWNAD}; do
    sed -i "/${i}/d" ${CIDRLIST} ${IPLIST};
done
Static ones may like this ?

Code: Select all

sed -i -r -e '/^(10\.|127\.|172\.16\.|192\.168\.|224\.0|0\.0\.0\.)/d' ${CIDRLIST} ${IPLIST};
so we´ve done a double check to prevent the block of our addresses.
A long story short, i used your addresses for the following script. This is only the downloader:

Code: Select all

#!/bin/bash -

#
# Downloader for blocklists.
# Searches for CIDRs and IPs and sorts system addresses and private IP ranges out.
# Writes lists to /tmp/lists
#
# $Author: ummeegge $date: 26.01.2017
###################################################################################
#


set -x

## Locations
LISTS="/tmp/lists";
CIDRLIST="${LISTS}/cidrs";
IPLIST="${LISTS}/ips"
SET="/var/ipfire/ethernet/settings";
OVPNSUB="/var/ipfire/ovpn/server.conf";


## Investigate system addresses to prevent potential blocks
# LAN, WLAN, DMZ, DNS and OpenVPN addresses
USEDADDRESSES=$(awk -F'=' '/GREEN_ADDRESS/ || /BLUE_ADDRESS/ || /RED_ADDRESS/ || /ORANGE_ADDRESS/ { print $2 }' ${SET} | cut -d'.' -f1,2 && \
awk -F'=' '/DNS1=/ || /DNS2=/ { print $2 }' ${SET} && \
awk '/server / || /route / { print $2 }' ${OVPNSUB} | sed 's/.0$//g')
OWNAD=$(echo "${USEDADDRESSES}" | tr ' ' '\n' | sort -nu);

## Addresses
# For IPs
IPURLS="
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/bambenek_c2.ipset \
https://lists.blocklist.de/lists/bruteforcelogin.txt \
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/feodo.blocklist-ipsets \
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/palevo.ipset \
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/sslbl.ipset \
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/zeus_badips.ipset \
https://ransomwaretracker.abuse.ch/downloads/RW_IPBL.txt \
https://ransomwaretracker.abuse.ch/downloads/TC_PS_IPBL.txt \
https://ransomwaretracker.abuse.ch/downloads/LY_C2_IPBL.txt \
https://ransomwaretracker.abuse.ch/downloads/LY_PS_IPBL.txt \
https://ransomwaretracker.abuse.ch/downloads/TL_C2_IPBL.txt \
https://ransomwaretracker.abuse.ch/downloads/TL_PS_IPBL.txt \
https://zeustracker.abuse.ch/blocklist.php?download=ipblocklist \
https://sslbl.abuse.ch/blacklist/sslipblacklist_aggressive.csv \
https://sslbl.abuse.ch/blacklist/sslipblacklist.csv \
https://sslbl.abuse.ch/blacklist/sslipblacklist.rules \
https://sslbl.abuse.ch/blacklist/sslipblacklist_aggressive.rules \
https://hosts-file.net/rss.asp \
http://malc0de.com/rss/ \
https://www.scumware.org/ \
https://feodotracker.abuse.ch \
https://www.dshield.org/ipsascii.html
";
# For CIDRs
CIDRURLS="
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/dshield_1d.netset \
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/spamhaus_drop.netset \
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/spamhaus_edrop.netset \
https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level1.netset \
https://zeustracker.abuse.ch/blocklist.php?download=squidip \
http://www.ipdeny.com/ipblocks/data/countries/cn.zone
";

### Left out
## Private ip addresses will be deleted via script
#https://www.team-cymru.org/Services/Bogons/bogon-bn-agg.txt
## Does all countries makes sense ?
#http://www.ipdeny.com/ipblocks \
## URLs and Hosts are included use only IPs
#https://ransomwaretracker.abuse.ch/blocklist
##IPs and domains mixed withanother
#https://zeustracker.abuse.ch
## Lots of different stuff in it
#https://sslbl.abuse.ch
##Lots of different stuff in it
#https://zeltser.com/malicious-ip-blocklists \

## Main part
mkdir ${LISTS}
cd ${LISTS}
# Get CIDRS friendly
wget --user-agent "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:11.0) Gecko/20100101 Firefox/11.0" \
-S -t 3 -T 10 -O- ${CIDRURLS} --no-check-certificate | \
grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}/[0-9]{1,2}" | sort -nu > ${CIDRLIST};

# Get IPs
wget --user-agent "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:11.0) Gecko/20100101 Firefox/11.0" \
-S -t 3 -T 10 -O- ${IPURLS} --no-check-certificate | \
grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" | sed '/\.0$/d' | sort -nu > ${IPLIST};

sed -i -r -e '/^(10\.|127\.|172\.16\.|192\.168\.|224\.0|0\.0\.0\.)/d' ${CIDRLIST} ${IPLIST};
for i in ${OWNAD}; do
    sed -i "/${i}/d" ${CIDRLIST} ${IPLIST};
done

# End script

Debugger is still on, wget has become a little friendlier, lists won´t be downloaded but redirected to STDOUT. CIDR and IP lists are separated but should be expandable. All lists can be found under /tmp/lists .
troll-op wrote: Would the /etc/sysconfig/firewall.local reload need to happen after each cron update? Or only the initial one?
I've not had any problems on the machine running all this, have not restarted or reloaded the firewall.local. How would I know if the latest lists are being used?
Hmm, if i try e.g. to destroy my sets, i get this one:

Code: Select all

-> ipset destroy
ipset v6.29: Set cannot be destroyed: it is in use by a kernel component
which seems clear since the old configuration is stored mostly hash based in the FW, somewhere a reload is needed after an update ;) .

Left some comments to the URLs in the script.

Greetings,

UE
Image
Image

Post Reply