Intrusion Prevention System - core 131

General questions.
ipfireuser5150
Posts: 32
Joined: May 18th, 2019, 5:28 pm

Re: Intrusion Prevention System - core 131

Post by ipfireuser5150 » May 23rd, 2019, 10:34 am

H&M wrote:
May 23rd, 2019, 4:07 am
Did anybody of you found a way to use both ET and Talos?
With the previous Snort IPS, this was possible. I couldn't figure out how to do it with this new system. I'd load the ET rulesets, then when I tried to load the Snort Community rules (it's just one rule and should show up as a single checkbox), it wouldn't appear.

Mentalic
Posts: 31
Joined: April 14th, 2018, 2:51 pm

Re: Intrusion Prevention System - core 131

Post by Mentalic » May 23rd, 2019, 2:46 pm

ipfireuser5150 wrote:
May 23rd, 2019, 10:34 am
H&M wrote:
May 23rd, 2019, 4:07 am
Did anybody of you found a way to use both ET and Talos?
With the previous Snort IPS, this was possible. I couldn't figure out how to do it with this new system. I'd load the ET rulesets, then when I tried to load the Snort Community rules (it's just one rule and should show up as a single checkbox), it wouldn't appear.
Asked early on about this and only one ruleset at a time can be selected.
Image
Image

Flash1232
Posts: 30
Joined: April 29th, 2012, 2:50 pm

Re: Intrusion Prevention System - core 131

Post by Flash1232 » May 23rd, 2019, 5:06 pm

Well but folks it can't really be true that the only option to use IPS is to switch the DNS away from Cloudflare ??? ...
Should we submit a bug or what do we do?

Mentalic
Posts: 31
Joined: April 14th, 2018, 2:51 pm

Re: Intrusion Prevention System - core 131

Post by Mentalic » May 23rd, 2019, 7:03 pm

dilse wrote:
May 22nd, 2019, 7:35 pm
Thanks for the that explanation, makes sense.

Agreed, 108MB (Talos) compared to 2.3MB (EM), big difference. Now I'll switch back to Talos :)
One thing I noticed about the Talos rules are that while there large files they have very few actual sub-rules selected if you hit the "show" target. So you have to check and select the sub rules or there's nearly nothing actually monitored.
Image
Image

bloater99
Posts: 482
Joined: October 13th, 2014, 3:47 pm

Re: Intrusion Prevention System - core 131

Post by bloater99 » May 23rd, 2019, 9:38 pm

Mentalic wrote:
May 23rd, 2019, 2:46 pm
ipfireuser5150 wrote:
May 23rd, 2019, 10:34 am
H&M wrote:
May 23rd, 2019, 4:07 am
Did anybody of you found a way to use both ET and Talos?
With the previous Snort IPS, this was possible. I couldn't figure out how to do it with this new system. I'd load the ET rulesets, then when I tried to load the Snort Community rules (it's just one rule and should show up as a single checkbox), it wouldn't appear.
Asked early on about this and only one ruleset at a time can be selected.
Hopefully this is a temporary limitation that will gradually be implemented, since this would be considered a feature downgrade from the Snort-based IPS. I have not upgraded my production IPFire to 131 yet. I did upgrade my personal IPFire, but don't use IPS at home due to slow cpu and limited (2G) ram.
Image

Image

dnl
Posts: 375
Joined: June 28th, 2013, 11:03 am

Re: Intrusion Prevention System - core 131

Post by dnl » May 27th, 2019, 4:05 am

bloater99 wrote:
May 23rd, 2019, 9:38 pm
Mentalic wrote:
May 23rd, 2019, 2:46 pm
ipfireuser5150 wrote:
May 23rd, 2019, 10:34 am

With the previous Snort IPS, this was possible. I couldn't figure out how to do it with this new system. I'd load the ET rulesets, then when I tried to load the Snort Community rules (it's just one rule and should show up as a single checkbox), it wouldn't appear.
Asked early on about this and only one ruleset at a time can be selected.
Hopefully this is a temporary limitation that will gradually be implemented, since this would be considered a feature downgrade from the Snort-based IPS. I have not upgraded my production IPFire to 131 yet. I did upgrade my personal IPFire, but don't use IPS at home due to slow cpu and limited (2G) ram.
+1

I agree. While the Suricata implementation is much better than the old Snort IDS, some major features have been overlooked.
IPFire 2.x (Latest Update) on x86_64 Intel Bay Trail CPU, 4GiB RAM, RED + GREEN + BLUE + ORANGE

MysteryGoat
Posts: 6
Joined: May 18th, 2019, 9:47 pm

Re: Intrusion Prevention System - core 131

Post by MysteryGoat » May 28th, 2019, 3:17 am

So after spending a couple of days on Talos I still have no logs. I'm curious if it's an issue of it not working correctly or if it's because not enough individual rules are selected. Personally I'm switching back to ET because even if it is the latter, I don't want to go through all of those rules to figure out which ones I need and don't. I'm sure that might be something I SHOULD do but for just a small home network I'm not going through that much trouble.

User avatar
H&M
Posts: 467
Joined: May 29th, 2014, 9:38 pm
Location: Europe

Re: Intrusion Prevention System - core 131

Post by H&M » June 9th, 2019, 4:31 pm

Hello all,

I managed to start suricata with both Emerging and Talos Rules.

At this moment process is manual - just to test it is working:

1. I've created 2 rules settings files - one for Talos, one for Emerging:

Code: Select all

ls -l /var/ipfire/suricata/rules-settings.*
-rw-r--r-- 1 nobody nobody 91 Jun  9 18:26 /var/ipfire/suricata/rules-settings.emerging
-rw-r--r-- 1 nobody nobody 93 Jun  9 15:31 /var/ipfire/suricata/rules-settings.talos
2. I 've run the update with each of them and copying the files brought by each update to one temp folder

Talos rules files were copied to /tmp/rules/talos
Emerging rules were copied to /tmp/rules/emerging

I also created a list of rules files with this script after each download (Emerging and Talos)

Code: Select all

cat ~/get_suricata_rules.sh
ls -1 /var/lib/suricata/*.rules |awk -F/ '{print " - "  $5}'
Like this:

After running Talos update I dumped the list of files to a file:

Code: Select all

~/get_suricata_rules.sh > ~/talos.rules
Same after running Emerging update:

Code: Select all

~/get_suricata_rules.sh > ~/emerging.rules
3. After each download I also put all rules files in /var/lib/suricata

Code: Select all

mv -f /tmp/rules/talos/*.rules /var/lib/suricata
mv -f /tmp/rules/emerging/*.rules /var/lib/suricata
chown nobody:nobody /var/lib/suricata/*.rules
4. Then update the list of rules files to be used by suricata at startup - the file needs to be empty (as delivered by installer, no rule activated).
I made a copy of it after fresh start - I use this empty one.

Code: Select all

cat ~/talos.rules  >> /var/ipfire/suricata/suricata-used-rulefiles.yaml
cat ~/emerging.rules  >> /var/ipfire/suricata/suricata-used-rulefiles.yaml


5.At this point suricata can be started:

Code: Select all

/etc/init.d/suricata start

Now I have in /var/ipfire/suricata/suricata-used-rulefiles.yaml both Emerging and Talos rules activated, and the rules files are also located in /var/lib/suricata

After this Suricata is consuming a whopping 1GB of RAM....
And of course the ids.cgi page list files from Both Emerging and Talos
Suricata Memory.PNG
ET Rules.PNG
Talos_rules.PNG

Late edit: here is a script doing all above except creation of the Emerging and Talos setting files
Is tested only twice - use on your own risk.

Code: Select all


#!/bin/sh
#########################################################################
#Suricata - use both ET and Talos rules                                 #
#Created by H&M                                                                       #
#Version 1.0 09.06.2019                                                 #
#########################################################################

#Variables definition
suricata_settings="/var/ipfire/suricata"
#You need these files to be created manually by you
ET_rules_settings=$suricata_settings/rules-settings.emerging 
Talos_rules_settings=$suricata_settings/rules-settings.talos
#Other variables
rules_settings=$suricata_settings/rules-settings
rules_folder="/var/lib/suricata"
TMP="/tmp/rules"
ET_TMP=$TMP/emerging
ET_rules=$ET_TMP/emerging
Talos_TMP=$TMP/talos
Talos_rules=$Talos_TMP/talos
suricata_rulefiles=$suricata_settings/suricata-used-rulefiles.yaml
empty_rulefiles=$suricata_settings/suricata-used-rulefiles.yaml.orig

#Stop Suricata - oinkmaster will be faster. If you prefer to run update with suricata on, comment next line
suricatactrl stop

#Some checks and folder creation
if [[ ! -d "$ET_TMP" ]]; then  mkdir -p $ET_TMP; fi
if [[ ! -d "$Talos_TMP" ]]; then mkdir -p $Talos_TMP; fi
chown -R nobody:nobody $TMP

#Emerging Threat
cp -f $ET_rules_settings $rules_settings
/usr/local/bin/update-ids-ruleset
mv -f $rules_folder/*.rules $ET_TMP
ls -1 $ET_TMP/*.rules |awk -F/ '{print " - "  $5}' > $ET_rules

#Talos
cp -f $Talos_rules_settings $rules_settings
/usr/local/bin/update-ids-ruleset
mv -f $rules_folder/*.rules $Talos_TMP
ls -1 $Talos_TMP/*.rules |awk -F/ '{print " - "  $5}' > $Talos_rules


#Put rules files in the place
mv -f $Talos_TMP/*.rules $rules_folder
mv -f $ET_TMP/*.rules $rules_folder


#Activate rules in /var/ipfire/suricata/suricata-used-rulefiles.yaml
cp -f $empty_rulefiles $suricata_rulefiles
cat $Talos_rules >> $suricata_rulefiles
cat $ET_rules  >> $suricata_rulefiles
chown  -R nobody:nobody  $suricata_settings

#Start suricata
suricatactrl start

User avatar
H&M
Posts: 467
Joined: May 29th, 2014, 9:38 pm
Location: Europe

Re: Intrusion Prevention System - core 131

Post by H&M » June 11th, 2019, 6:01 pm

After 48 hours running suricata with both ET and Talos rules, I got first hit from a Talos rules:

Code: Select all

 cat /var/log/suricata/fast.log | awk  '{sub(/.*\[1:/,"",$SID);print $2}' | grep "^[^EG]"
MALWARE-CNC
 cat /var/log/suricata/fast.log | grep MALWARE
06/11/2019-09:31:52.462414  [Drop] [**] [1:31136:2] MALWARE-CNC Win.Trojan.ZeroAccess inbound connection [**] [Classification: A Network Trojan was Detected] [Priority: 1] {UDP} 151.53.0.46:1024 -> a.b.c.d:16464
All the rest hits are from ET rules: 2912 hits that contains in the rule description either ET, either GPL

Code: Select all

cat /var/log/suricata/fast.log | awk  '{sub(/.*\[1:/,"",$SID);print $2}' | grep "[^M]" |wc -l
2912

cat /var/log/suricata/fast.log | awk  '{sub(/.*\[1:/,"",$SID);print $0}'
2101411:12] GPL SNMP public access udp [**] [Classification: Attempted Information Leak] [Priority: 2] {UDP} 92.118.161.17:64941 -> 79.115.162.42:161
2402000:5203] ET DROP Dshield Block Listed Source group 1 [**] [Classification: Misc Attack] [Priority: 2] {TCP} 185.176.27.178:50323 -> a.b.c.d:5908
2008578:4] ET SCAN Sipvicious Scan [**] [Classification: Attempted Information Leak] [Priority: 2] {UDP} 77.247.108.113:5103 -> a.b.c.d:5060
So suricata it's working with both Talos and ET rules ...

User avatar
cbrown
Posts: 41
Joined: December 29th, 2017, 11:54 pm
Location: Texas

Re: Intrusion Prevention System - core 131

Post by cbrown » June 26th, 2019, 2:34 pm

A sweet bit of hacking there, H&M ;D
Any chance this capability would make it into IPFire 2.x in the next few months?

-cb
Image

User avatar
H&M
Posts: 467
Joined: May 29th, 2014, 9:38 pm
Location: Europe

Re: Intrusion Prevention System - core 131

Post by H&M » June 27th, 2019, 4:39 am

Can't say.
I am not a developer....

And indeed what I did was close to "hack" the normal update process designed by developers.

Happy to report that is working fine since then...

Hellfire
Posts: 695
Joined: November 8th, 2015, 8:54 am

Re: Intrusion Prevention System - core 131

Post by Hellfire » June 27th, 2019, 10:47 am

H&M wrote:
June 11th, 2019, 6:01 pm
After 48 hours running suricata with both ET and Talos rules, I got first hit from a Talos rules:

Code: Select all

 cat /var/log/suricata/fast.log | awk  '{sub(/.*\[1:/,"",$SID);print $2}' | grep "^[^EG]"
MALWARE-CNC
 cat /var/log/suricata/fast.log | grep MALWARE
06/11/2019-09:31:52.462414  [Drop] [**] [1:31136:2] MALWARE-CNC Win.Trojan.ZeroAccess inbound connection [**] [Classification: A Network Trojan was Detected] [Priority: 1] {UDP} 151.53.0.46:1024 -> a.b.c.d:16464
So does that mean that I've to search logs manually to find any hits?
Image

User avatar
H&M
Posts: 467
Joined: May 29th, 2014, 9:38 pm
Location: Europe

Re: Intrusion Prevention System - core 131

Post by H&M » June 27th, 2019, 5:28 pm

Yes,

I got 2900 hits from ET rules and only one from Talos in 48 hours.

The filter you quoted does that: eliminates all ET hits becaute these are always starting with either ET, either G...

Hope it helps.
H&M

TimF
Posts: 83
Joined: June 10th, 2017, 7:27 pm

Re: Intrusion Prevention System - core 131

Post by TimF » June 27th, 2019, 6:29 pm

Probably the ET hits were one of the IP Address blacklists. These packets are often port scans and would be blocked by the default input policy anyway. The lists are:

BOTCC, CIARMY, COMPROMISED-IPS, DROP and DSHIELD.

If any of these rules is hit it's because the IP Address is recognised as a potential attacker.

The Talos VRT rulesets don't have any equivalent rules, which is why there are fewer hits; these rules are only looking for traffic that is a potential attack.

Note that the GPL rules are a subset of the Talos VRT community rules, and can also be found in the Talos VRT Registered/Subscribed ruleset.

User avatar
H&M
Posts: 467
Joined: May 29th, 2014, 9:38 pm
Location: Europe

Re: Intrusion Prevention System - core 131

Post by H&M » June 27th, 2019, 7:14 pm

There could be another factor that influence low hits from Talos: many files from Talos are empty, no rules at all in them.

Ok, I use community rules but stil.
I did a count back sone tine and I had a total of +35000 rules actives and another 25000 inactive (lines in rules files start with # )

Talos rules were merely 10% of active ones...
I hope that I did the awk filter right...
If above ratio is correct, then it can also contribute to low number of hits.


Last thing: I have a big GeoIP filtering in place - less than 3 countries allowed in. So IPS gets little number of packets, vast majority are blocked by GeoIP chain.
This chain stops all netscan attempts from so many entities that does that regularly...

Hope it helps,
H&M

Post Reply