IPset for IPFire

Help on building IPFire & Feature Requests
ummeegge
Community Developer
Community Developer
Posts: 4990
Joined: October 9th, 2010, 10:00 am

IPset for IPFire

Post by ummeegge » October 23rd, 2015, 7:53 am

EDIT: IPset is meanwhile in IPFire Core 95 integrated so there is no need to install it manually like below described. This includes only the binary and installation directory the update script and the other integrations aren´t included in the Core update and currently only here available.

Hi all,
at the first i wanted to introduce a tool named IPset --> http://ipset.netfilter.org/ which is a companion application for IPTables. It allows you to setup rules to quickly and easily block a set of IP addresses, among other things.
And secondly what are you thinking about this feature for IPFire ?

An interessting Site in my opinion for this theme can be found in here --> http://iplists.firehol.org/ .

Greetings,

UE
Image
Image

jawz101
Posts: 2
Joined: October 26th, 2015, 4:37 pm

Re: IPset for IPFire

Post by jawz101 » October 26th, 2015, 7:04 pm

Thanks. That FireHOL website is pretty damn awesome. It's nice to see someone has taken an analytical approach to the maintenance of all of these publicly available blocklists.

https://github.com/firehol

ummeegge
Community Developer
Community Developer
Posts: 4990
Joined: October 9th, 2010, 10:00 am

Re: IPset for IPFire

Post by ummeegge » October 27th, 2015, 9:58 am

Hi jawz101,
jawz101 wrote:Thanks. That FireHOL website is pretty damn awesome.
i think so, very good sorted but also infos about list updates and some more goodies :) .

But IPset is also a nifty tool. Have compiled it for IPFire now, wrote a little daily update script for it, included meanwhile 22 lists with currently 10062 IPs and 588 CIDRs and in puncto performance it makes a good impression for me. Also there would be no way to integrate this mass of IPs/CIDRs via IPTables directly without a subterraneously speed performance.
A nice site for performance test with and without IPset (and some more interessting infos) can also be found in here --> https://www.dbsysnet.com/2016/03/mass-b ... with-ipset .

UE

EDIT: Renewed link
Image
Image

Garp
Posts: 127
Joined: July 8th, 2014, 7:38 am
Location: The Netherlands
Contact:

Re: IPset for IPFire

Post by Garp » October 29th, 2015, 3:59 pm

Hi UE,

Can i help test it, in any way? I'm no application coder myself, so i cannot write this myself. May/Can i use your application?
Image
Provide some additional protection for the clients on your network in a few easy steps: viewtopic.php?f=27&t=12122&p=78219#p78219

ummeegge
Community Developer
Community Developer
Posts: 4990
Joined: October 9th, 2010, 10:00 am

Re: IPset for IPFire

Post by ummeegge » October 29th, 2015, 5:38 pm

Hi Garp,
Garp wrote:Can i help test it, in any way? I'm no application coder myself, so i cannot write this myself.
are you a little familar with the console/ssh ? If so, feel free to go for some testings. You can find IPset in here --> http://people.ipfire.org/~ummeegge/ipset/ <-- please check the sha256 sum.
Installation:
wget or what ever it to /opt/pakfire/tmp unpack it with a

Code: Select all

tar xvf ipset-6.26-1.ipfire
and install it with a

Code: Select all

./install.sh
save the output so it is easier to uninstall it...
checkout if the appropriate kernel module is loaded

Code: Select all

lsmod | grep ip_set
if there is something presant you are good to go, otherweise load it with a

Code: Select all

modprobe ip_set
(above explanation do not include a reboot!!! <-- Please check the IPset man/homepage)
...

May it is a good idea that you go for a try on your own ways for the first ::) ??, otherwise there are also some scripts out there which might accelerate further intends.
Have also one here.. still in testing mode ;) .

Please use testing systems for this.

Greetings,

UE
Image
Image

Garp
Posts: 127
Joined: July 8th, 2014, 7:38 am
Location: The Netherlands
Contact:

Re: IPset for IPFire

Post by Garp » October 29th, 2015, 6:34 pm

Ok, thx.

I use IPFire as my home router. What would be the risk in using this addon in a home situation? Will it break anything?
Image
Provide some additional protection for the clients on your network in a few easy steps: viewtopic.php?f=27&t=12122&p=78219#p78219

ummeegge
Community Developer
Community Developer
Posts: 4990
Joined: October 9th, 2010, 10:00 am

Re: IPset for IPFire

Post by ummeegge » October 29th, 2015, 6:43 pm

You are welcome,
Garp wrote: What would be the risk in using this addon in a home situation? Will it break anything?
we are in development state , it could break anything anytimes.

greetings,

UE
Image
Image

ummeegge
Community Developer
Community Developer
Posts: 4990
Joined: October 9th, 2010, 10:00 am

Re: IPset for IPFire

Post by ummeegge » October 31st, 2015, 7:53 am

First idea for an daily update script:

Please use only testing systems for this.

- Don´t forget to make it executable with a

Code: Select all

chmod +x ipset_updater.sh
- Debugger is still activated for testing
- After testing it can be placed under /etc/fcron.daily/ <-- 'set -x' can be deleted after testings.
- Blocks currently FORWARD, INPUT and OUTPUT (also IPFire itself) for all ports and protocols. Block works with REJECT not DROP. IPs/CIDRs will be rejected as source and destination.
- IPTable rules will be made automatically by the script. Rules are located under /etc/sysconfig/firewall.local .
- Two sets will be created. One set is for IPs only and the other for CIDRs. A counter for bytes and packets are integrated too.
- Script contains currently 22 lists which have today 568 CIDRs and 11461 IPs.
- There is currently no IPset un- or installer but this should be no problem if needed.
- LAN, WAN, WLAN, DMZ, DNS1-2 and OpenVPN addresses will be automatically investigated and if presant in one of the list it will be deleted so yourself shouldn´t be locked out <-- got this problem with 192.168.0.0/16.
- restore command are placed under /etc/sysconfig/rc.local, to reactivate IPset after a reboot.
- ip_set kernel module will be loaded if not already done.
- If byte and packets counter are not 0, the appropriate entries will be written to /etc/ipset/counterlist_ipset, this works on two machines but on a third the counter do not work, don´t no why currently.
- All lists are located under /tmp/ipset which won´t be deleted in testing phase.
- Every update will be logged into syslog.
- Attention: If you use Tor cause a Tor blocklist is also integrated and it won´t work anymore if you use this list. Delete the "https://check.torproject.org/cgi-bin/To ... ip=1.1.1.1" line in the "## Blacklist addresses" section to prevent a Tor block.

Code: Select all

#!/bin/bash -

set -x
 
#
# Update script example for blacklist update in IPset.
# Includes FW rule integration,  configuration entries, and restore command for system restart.
# ummeegge[at]ipfire.org $date 01.01.2016
#################################################################
#

## Locations
CONFDIR="/etc/ipset";
CONF="${CONFDIR}/ipset.conf";
COUNTER="${CONFDIR}/counterlist_ipset";
IPSET="/usr/sbin/ipset";
DIR="/tmp/ipset";
DWNLOG="/tmp/ipset/dwnload.log";
CIDRLIST="${DIR}/CIDRLIST";
IPLIST="${DIR}/IPLIST";
SETCIDR="cidrlist";
SETIP="iplist";
FWL="/etc/sysconfig/firewall.local";
SET="/var/ipfire/ethernet/settings";
OVPNSUB="/var/ipfire/ovpn/server.conf";
RC="/etc/sysconfig/rc.local";

#----------------------------------------------------------------------------------------------------------------

## Investigate system addresses to prevent potential blocks
# LAN, WLAN, DMZ, DNS and OpenVPN addresses
USEDADDRESSES=$(awk -F'=' '/GREEN_ADDRESS/ || /BLUE_ADDRESS/ || /RED_ADDRESS/ || /ORANGE_ADDRESS/ { print $2 }' \
${SET} | cut -d'.' -f1,2 && \
awk -F'=' '/DNS1=/ || /DNS2=/ { print $2 }' ${SET} && \
awk '/server / || /route / { print $2 }' ${OVPNSUB} | sed 's/.0$//g')
OWNAD=$(echo "${USEDADDRESSES}" | tr ' ' '\n' | sort -nu);

## FW functions
# Searcher
# Start
STARTFW=$(grep "\${IPSET}" ${FWL} | echo $?);
# Stop
STOPFW=$(grep "\${IPSET}" ${FWL} | echo $?);

fwadd_funct() {
  if [ "${STARTFW}" -eq 0 ]; then
   sed -i '/# Used for private firewall rules/ a\IPSET="\/sbin\/iptables"' ${FWL};
   sed -i "/## add your 'start' rules here/ a\ \
       # IPSET FW entries in start\n \
       # IPSET add rules for CIDR list\n \
       \${IPSET} -I CUSTOMFORWARD -m set --match-set ${SETCIDR} dst -j REJECT\n \
       \${IPSET} -I CUSTOMINPUT -m set --match-set ${SETCIDR} src -j REJECT\n \
       \${IPSET} -I CUSTOMOUTPUT -m set --match-set ${SETCIDR} dst -j REJECT\n \
       # IPSET rules for ip list\n \
       \${IPSET} -I CUSTOMFORWARD -m set --match-set ${SETIP} dst -j REJECT\n \
       \${IPSET} -I CUSTOMINPUT -m set --match-set ${SETIP} src -j REJECT\n \
       \${IPSET} -I CUSTOMOUTPUT -m set --match-set ${SETIP} dst -j REJECT" ${FWL};
       # Add stop rules
   sed -i "/## add your 'stop' rules here/ a\ \
       # IPSET flushing related chains\n \
       \${IPSET} -F CUSTOMFORWARD\n \
       \${IPSET} -F CUSTOMINPUT\n \
       \${IPSET} -F CUSTOMOUTPUT" ${FWL};
  fi
}

fwstop_funct() {
  if [ "${STOPFW}" -eq 0 ]; then
   # Delete IPset related entries
   sed -i -e "/\${IPSET}.*/d" -e "/# IPSET.*/d" -e "/IPSET.*/d" ${FWL};
  fi
}

#----------------------------------------------------------------------------------------------------------------

# Check for installation
if [[ ! -e "${IPSET}" ]]; then
   echo "Haven´t found an IPset installation on this system, need to quit... ";
   exit 1;
fi

# Create appropriate sets with counter if not already done
if [[ -z "$(ipset -n list | grep ${SETCIDR})" ]]; then
   ipset create ${SETCIDR} hash:net counters;
fi
if [[ -z "$(ipset -n list | grep ${SETIP})" ]]; then
   ipset create ${SETIP} hash:ip counters;
fi

## Check for file, rc.local entry, dirs and module
if [[ ! -d "${CONFDIR}" ]]; then
   mkdir ${CONFDIR};
elif [[ ! -e "${COUNTER}" ]]; then
   touch ${COUNTER};
elif [[ -z "$(lsmod | grep ip_set)" ]]; then
   modprobe ip_set;
elif [[ -z "$(grep 'ipset' ${RC})" ]]; then
      echo "${IPSET} restore < ${CONF} && /etc/sysconfig/firewall.local reload;" >> ${RC};
fi

#----------------------------------------------------------------------------------------------------------------

## Blacklist addresses
URLS="https://rules.emergingthreats.net/blockrules/emerging-botcc.rules \
https://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=1.1.1.1 \
https://danger.rulez.sk/projects/bruteforceblocker/blist.php \
https://zeustracker.abuse.ch/blocklist.php?download=ipblocklist \
https://zeustracker.abuse.ch/blocklist.php?download=squidip \
https://www.spamhaus.org/drop/drop.lasso \
http://cinsscore.com/list/ci-badguys.txt \
https://www.openbl.org/lists/base.txt \
https://autoshun.org/files/shunlist.csv \
https://lists.blocklist.de/lists/all.txt \
https://feeds.dshield.org/top10-2.txt \
https://palevotracker.abuse.ch/blocklists.php?download=ipblocklist \
https://zeustracker.abuse.ch/blocklist.php?download=badips";

#----------------------------------------------------------------------------------------------------------------
# Start processing

# Check if entries exist
if [[ -n "$(ipset list | tail -1)" ]]; then
   # Prepare firewall.local
   fwstop_funct;
   ${FWL} reload;
   # Write to counter list before flushing the sets if packet counter ! 0
   if [[ -n $(ipset list | sed -e '/packets 0/d' -e '/^[^0-9]/d' -e '/^$/d') ]]; then
       echo -e "\e[31m$(date)\e[0m" >> ${COUNTER} \
       && ipset list | sed -e '/packets 0/d' -e '/^[^0-9]/d' -e '/^$/d' >> ${COUNTER};
   else
       echo -e "\e[31m$(date)\e[0m" >> ${COUNTER};
       echo "No entries today" >> ${COUNTER};
   fi
   # Flushing existing sets
   ipset flush ${SETCIDR};
   ipset flush ${SETIP};
fi

# Check for installation directory
if [ -d "${DIR}" ]; then
   rm -rf ${DIR};
   mkdir ${DIR};
else
   mkdir ${DIR};
fi

#################################################################################################################
# Donwload and process list(s)
cd ${DIR} || exit 1;
wget -S -N -t 3 -T 10 -o "${DWNLOG}" ${URLS} --no-check-certificate;
############################################ Get all IPs ########################################################
# grep IPs and sort and make them uniq
cat * | \
grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" | sort -nu > ${IPLIST};
########################################### Get all CIDRs #######################################################
# grep CIDRs and sort and make them uniq
cat * | \
grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}/[0-9]{1,2}" | \
sort -nu > ${CIDRLIST}
#################################################################################################################
# Clean up and sort LAN, WLAN, DNS and OpenVPN addresses out
sed -i -e 's/^M//g' -e '/#/d' -e '/0.0.0.0.*/d' ${CIDRLIST} ${IPLIST};
for i in ${OWNAD}; do
    sed -i "/${i}/d" ${CIDRLIST} ${IPLIST};
done

# Introduce new content to IPset and add downloaded lists if needed
for l in $(cat ${CIDRLIST}); do ipset --add ${SETCIDR} "${l}"; done;
for l in $(cat ${IPLIST}); do ipset --add ${SETIP} "${l}"; done;
# Save the new lists
ipset save > ${CONF};
logger -t ipset "IPset: has updated blacklist";

fwadd_funct;
${FWL} reload;

exit 0

# End script

Feedback and testings might be nice.

Greetings,

UE

EDIT(s):
- Fixed bug for list directory creation.
- Fixed echo command for timestamp in 'counterlist_ipset ' <-- byte and packet counter do not work on every machine, haven´t find the problem until now.
- Deleted honeypot lists cause they are currently off.
- Deleted 'firehol_level1.netset' cause content it is already in other lists but also of its mixture of IPs and CIDRs.
- Fixed rc.local command.
21.11.15
- Fixed initial set creation.
- Set appropriate src and dst for FW rule chains.
- Added 'No entries today' in counter_list if no blacklistet addresses where recognized by byte and packet counter.
07.12.15
- Changed wget command. Reduced connections retries to 3, added timeout to 10 sec. to prevent long time loops if an URL isn´t reachable. Added time stamping for dwnload.log. Prints now "HTTP request" to dwnload.log. Prints now also "Content-Security-Policy" to dwnload.log.
01.01.2015
- Deleted Github addresses from Firehol regarding Costas hint, provide instead update-ipsets from Firehol project --> https://forum.ipfire.org/viewtopic.php? ... =15#p93409 which serves all Firehol lists.
- Changed some smaller things in the script (syntax).
02.12.2016
- Added firewall.local reload in rc.local.
Image
Image

Garp
Posts: 127
Joined: July 8th, 2014, 7:38 am
Location: The Netherlands
Contact:

Re: IPset for IPFire

Post by Garp » October 31st, 2015, 9:05 am

I will test when i find the time to make a full backup of my SD card that i run IPFire on.

When going through your script, i notice that you use a lot of sources. I was under the impression that the FireHole source should be 'suffient' as a source because then do the combining.

What is the reason that you added the other sources?
Image
Provide some additional protection for the clients on your network in a few easy steps: viewtopic.php?f=27&t=12122&p=78219#p78219

ummeegge
Community Developer
Community Developer
Posts: 4990
Joined: October 9th, 2010, 10:00 am

Re: IPset for IPFire

Post by ummeegge » October 31st, 2015, 9:49 am

Goedemiddag Garp,
Garp wrote: When going through your script, i notice that you use a lot of sources. I was under the impression that the FireHole source should be 'suffient' as a source because then do the combining.

What is the reason that you added the other sources?
different types of reaserches (u.a. honeypots) delivers different results, so in my opinion a better variation. A lot of IPs occurs double in this lists but the script sorts them out and makes them uniqe <-- IPset makes this job too but better to make this before.
The Firehol lists are very good but lists like e.g. this --> https://raw.githubusercontent.com/ktsao ... el1.netset brought some problems. In CIDR section was something i couldn´t work out until now which blocks browser traffic so i left the CIDRs out from this list.

Also everybody needs to check his own setup for his usage, for example i do not need the whole Spamhouse section in my environment so i left it out. Even the lists of the script are hold nearly general, please investigate the lists for your own purposes and add or delete what you need or you don´t need. The script is an example...

The grep commands to separate CIDRs and IPs from the rest should work in general, so you should be able to use it for your own lists too.
Garp wrote:I will test when i find the time to make a full backup of my SD card that i run IPFire on.
All right.

UE
Image
Image

ummeegge
Community Developer
Community Developer
Posts: 4990
Joined: October 9th, 2010, 10:00 am

Re: IPset for IPFire

Post by ummeegge » November 7th, 2015, 5:42 am

Hi all,
have fixed somethings on the script --> https://forum.ipfire.org/viewtopic.php? ... 514#p91514 <-- which can be overviewed at the bottom in the 'EDIT(s)' section.

Also if someone is interessted in howto build IPset, take a look in here --> http://git.ipfire.org/?p=ipfire-2.x.git ... 9168399b10 .
Also IPFire will deliver IPset with Core 95, a wiki comes in the next view days, will announce it here if it is ready.

Greetings,

UE
Image
Image

Garp
Posts: 127
Joined: July 8th, 2014, 7:38 am
Location: The Netherlands
Contact:

Re: IPset for IPFire

Post by Garp » November 7th, 2015, 10:08 am

Great news, thx!
Image
Provide some additional protection for the clients on your network in a few easy steps: viewtopic.php?f=27&t=12122&p=78219#p78219

ktsaou
Posts: 5
Joined: November 8th, 2015, 9:25 am

Re: IPset for IPFire

Post by ktsaou » November 8th, 2015, 9:34 am

Hi all,

sorry for intervening, I just got a notification from mention.net you are referring to iplists.firehol.org.

I strongly suggest not to use files downloaded from the github repo, in production systems. They might not be updated regularly.

I have added a wiki where I explain the right way of updating all these IP lists. Check it here: https://github.com/firehol/blocklist-ipsets/wiki

I don't know if you can use my method with ipfire, but anyway, I thought I should at least let you know.

Thanks!

ummeegge
Community Developer
Community Developer
Posts: 4990
Joined: October 9th, 2010, 10:00 am

Re: IPset for IPFire

Post by ummeegge » November 9th, 2015, 5:08 pm

Hi ktsaou and thanks for your intervention and your feedback which is very nice.
IPset will be delivered in IPFires next Core update but without Firehol which i think is a requirement for the usage of the 'update-ipsets' command, nevertheless i have build it for IPFire too and will go in media res with it.
ktsaou wrote: I strongly suggest not to use files downloaded from the github repo, in production systems. They might not be updated regularly.
I´am not hundred procents clear of it but does for example an 'update-ipsets enable dshield' do not deliver the data from here --> https://github.com/firehol/blocklist-ip ... eld.netset ? In that case all the actual lists are probably located in here --> https://github.com/firehol/blocklist-ipsets/ ? May if i´am wrong please correct me.

The script above should do similar things but at this time not so comfortable like Firehol it does, may it will becomes better as time goes by or even may Firehol belongs sometimes also in to the distribution, will see.

So let me say a big thanks to your work in the firehol.org platform which is very usful but also your prompt support in this topic.

Greetings and keep up the good work :) ,

UE
Image
Image

ummeegge
Community Developer
Community Developer
Posts: 4990
Joined: October 9th, 2010, 10:00 am

Re: IPset for IPFire

Post by ummeegge » November 11th, 2015, 8:42 am

Have made now the IPset wiki which can be found in here --> https://wiki.ipfire.org/en/configuration/firewall/ipset .
It might be great if you can go for correction readings. If you think there is something missing or not wright let it me know.

Greetings,

UE
Image
Image

Locked