IDS Rule updater - with rule state persistance

General questions.
JonM
Posts: 83
Joined: August 4th, 2017, 5:49 pm
Location: US

Re: IDS Rule updater - with rule state persistance

Post by JonM » August 27th, 2018, 5:18 pm

TimF wrote:
August 27th, 2018, 1:50 pm
The entry for snort-update.pl should have been removed by the installer - I've corrected it.
Is the snort-update.pl to be removed? or ids-update.pl?

EDIT: I found this line in crontab, should it be removed?

Code: Select all

[root@ipfire ~]# fcrontab -l
...
# Update snort rules
#%hourly,random 3-29 /var/ipfire/snort/update.sh
...
Production:
Image

Testing Raspi 3B+:
Image

Drexbengel48
Posts: 6
Joined: June 12th, 2017, 4:50 am
Location: Berlin

Re: IDS Rule updater - with rule state persistance

Post by Drexbengel48 » August 28th, 2018, 2:37 am

TimF wrote:
August 17th, 2018, 3:21 pm
I've now uploaded a new version. I'm not entirely sure the installer will work correctly, so it's on a branch at the moment. You can find it at:

https://github.com/timfprogs/ipfidsupdate/tree/version3

The major change is in the handling of community rules. While it's true that the Talos VRT rules contain a version of the community rules, for the registered ruleset this is a month out of date, so the script will now update the community rules if the VRT ruleset is in use, and will ensure that only the rule in the community ruleset are used where the rule is found in both rulesets. This should ensure that the latest version of the rule is in use.

The full changelist:
  • Added language files for French, German and Spanish. Unfortunately they're machine translated so I expect some errors.
Hi TimF,
can you please correct the lines in

https://github.com/timfprogs/ipfidsupda ... date.de.pl

Code: Select all

62 'idsupdate daily' => 'Daily',
63 'idsupdate weekly' => 'Täglich',

THX!!!

Greetings
Drexbengel48
Image

TimF
Posts: 60
Joined: June 10th, 2017, 7:27 pm

Re: IDS Rule updater - with rule state persistance

Post by TimF » August 28th, 2018, 7:14 pm

@JonM snort-update.pl should be removed if you're using the latest version of the script.

I'm not sure where the other line came from, but it should be able to be removed.

@Drexbengel48 I've edited the file - hopefully correctly.

xPliZit_xs
Posts: 127
Joined: May 31st, 2014, 8:22 pm

Re: IDS Rule updater - with rule state persistance

Post by xPliZit_xs » August 30th, 2018, 9:32 pm

Hi TimF,

the new version is working for me using core123 again!

Thank you.
xPliZit_xs

Image

xPliZit_xs
Posts: 127
Joined: May 31st, 2014, 8:22 pm

Re: IDS Rule updater - with rule state persistance

Post by xPliZit_xs » September 30th, 2018, 1:54 am

Hello TimF,

i am testing core 124 and the rule updater seems not to work anymore.
The services and Logs menu are missing.
This is just a head's up.

This is the install procedure output from the shell, maybe you can see whats wrong there already.


===================================================================================================
[root@ipfire ~]# wget https://github.com/timfprogs/ipfidsupda ... supdate.sh
--2018-09-29 21:48:04-- https://github.com/timfprogs/ipfidsupda ... supdate.sh
Resolving github.com... 192.30.253.113, 192.30.253.112
Connecting to github.com|192.30.253.113|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://raw.githubusercontent.com/timfp ... supdate.sh [following]
--2018-09-29 21:48:04-- https://raw.githubusercontent.com/timfp ... supdate.sh
Resolving raw.githubusercontent.com... 151.101.128.133, 151.101.192.133, 151.101.0.133, ...
Connecting to raw.githubusercontent.com|151.101.128.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4439 (4.3K) [text/plain]
Saving to: 'install-idsupdate.sh'

install-idsupdate.sh 100%[=========================================>] 4.33K --.-KB/s in 0s

2018-09-29 21:48:04 (202 MB/s) - 'install-idsupdate.sh' saved [4439/4439]

[root@ipfire ~]# chmod +x install-idsupdate.sh
[root@ipfire ~]# ./install-idsupdate.sh
read old settings
Check for new version
--2018-09-29 21:48:20-- https://github.com/timfprogs/ipfidsupda ... n3/VERSION
Resolving github.com... 192.30.253.112, 192.30.253.113
Connecting to github.com|192.30.253.112|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://raw.githubusercontent.com/timfp ... n3/VERSION [following]
--2018-09-29 21:48:20-- https://raw.githubusercontent.com/timfp ... n3/VERSION
Resolving raw.githubusercontent.com... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...
Connecting to raw.githubusercontent.com|151.101.0.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2 [text/plain]
Saving to: 'VERSION'

VERSION 100%[=========================================>] 2 --.-KB/s in 0s

2018-09-29 21:48:20 (757 KB/s) - 'VERSION' saved [2/2]

2018-09-29 21:48:20 INFO listing root's fcrontab
2018-09-29 21:48:20 INFO installing file /root/fcrontab_old for user root
Modifications will be taken into account right now.
[root@ipfire ~]
==========================================================================================================================
xPliZit_xs

Image

TimF
Posts: 60
Joined: June 10th, 2017, 7:27 pm

Re: IDS Rule updater - with rule state persistance

Post by TimF » October 5th, 2018, 5:30 pm

Hello,

There's some code in the installer that is meant to stop it downloading files unless they belong to a newer version than the version that's installed. That's probably the reason the installer seems not to work.

I've removed it because I don't think it's really necessary.

xPliZit_xs
Posts: 127
Joined: May 31st, 2014, 8:22 pm

Re: IDS Rule updater - with rule state persistance

Post by xPliZit_xs » October 5th, 2018, 8:40 pm

Nice! Going to try it again.
xPliZit_xs

Image

xPliZit_xs
Posts: 127
Joined: May 31st, 2014, 8:22 pm

Re: IDS Rule updater - with rule state persistance

Post by xPliZit_xs » October 7th, 2018, 6:53 pm

Hello,

it seems to install with core124 correctly if you download the correct github branch haha.

FYI:
USE: wget https://github.com/timfprogs/ipfidsupda ... supdate.sh

DONT USE: wget https://github.com/timfprogs/ipfidsupda ... supdate.sh

have to wait till it updates automatically...

regards
xPliZit_xs

Image

TimF
Posts: 60
Joined: June 10th, 2017, 7:27 pm

Re: IDS Rule updater - with rule state persistance

Post by TimF » October 12th, 2018, 3:01 pm

The version 3 changes works, so I merged them onto the master branch and I modified the updater on that branch. So the version 3 branch is now obsolete.

dnl
Posts: 336
Joined: June 28th, 2013, 11:03 am

Re: IDS Rule updater - with rule state persistance

Post by dnl » November 6th, 2018, 9:06 am

Hi TimF,

Thanks for this, I've been busy and haven't read the forums in a very long time. Looks like I've missed out!

I've just updated the IPFire Security Hardening guide to reference the first post in this thread, rather than the old "snortupdate.pl" script written by Kick@ass, H&M and gitarman94 as that script doesn't seem to be maintained any longer and appears to have less features than yours.

I've been really annoyed by getting thousands of IDS hits daily, when the vast majority are just from blocklists. So I'll check out your blocklist addon also!


Have you spoken to one of the core developers about having your addons packaged as official IPFire addons? I'd rather not automate the download and execution of shell scripts from a remote website.

Thanks!
Image

User avatar
raffe
Posts: 14
Joined: August 20th, 2018, 8:40 am

Re: IDS Rule updater - with rule state persistance

Post by raffe » November 6th, 2018, 10:10 am

Hi! Thanks for an excellent script! I think I soon have tweaked the choices of rules and flowbits so I can start thinking about blocklists :)

But before that, I have one question about the "Default policy" setting in https://ipfire:444/cgi-bin/idsupdate.cgi. I can choose Connectivity, Balanced, Security and Max-Detect.

What do these settings really do? Do they set or change some rules? If so, how will the settings co-exist with the rules I have manually chosen? Or is this setting only activated and used during the automatic update?

With best regards
raffe
Image

TimF
Posts: 60
Joined: June 10th, 2017, 7:27 pm

Re: IDS Rule updater - with rule state persistance

Post by TimF » November 7th, 2018, 8:57 pm

The settings affect new rules. The script will evaluate new rules against your selected policy and will enable the rule if it's in the selected policy or disable it if not. The default policy is 'Balanced' - this is what you would get if you just downloaded the rule files. It doesn't affect your existing rule selections.

In addition it will warn you of changes to rules that you've enabled (or disabled) and would normally be disabled (or enabled) in your policy. This is in case the reason that you enabled (or disabled) the rule is no longer valid.

Finally, if you select 'Apply policy changes' it will enable or disable rules if their policy changes. So, for example, if you've got a rule selected that is in the balanced policy, and that is your selected policy, it will disable the rule if the policy of the rule changes to 'Security'. This is very rare.

User avatar
raffe
Posts: 14
Joined: August 20th, 2018, 8:40 am

Re: IDS Rule updater - with rule state persistance

Post by raffe » November 8th, 2018, 9:22 am

Thanks for the answer!
dnl wrote:
November 6th, 2018, 9:06 am
Have you spoken to one of the core developers about having your addons packaged as official IPFire addons?
I agree with DNL, this is so good and important that it should really be included as an official addon! 8)

Or are you waiting for this? viewtopic.php?f=27&t=8323&start=75#p120129
Image

dnl
Posts: 336
Joined: June 28th, 2013, 11:03 am

Re: IDS Rule updater - with rule state persistance

Post by dnl » November 8th, 2018, 9:54 am

raffe wrote:
November 8th, 2018, 9:22 am
I agree with DNL, this is so good and important that it should really be included as an official addon! 8)

Or are you waiting for this? viewtopic.php?f=27&t=8323&start=75#p120129
Even if the Suricata feature comes with automatic updates, I still like the idea of moving the blocklist rules out.

So TimF can you please ask the core developers about including your other add-on? https://github.com/timfprogs/ipfblocklist
Image

Hellfire
Posts: 580
Joined: November 8th, 2015, 8:54 am

Re: IDS Rule updater - with rule state persistance

Post by Hellfire » November 16th, 2018, 6:49 pm

Hi,

I've got a couple of questions, although some have already been asked above, there are still some (if not all) settings that are unclear to me at the moment:

I will start from the beginning:

1) If I configure IDS-Update for performing automatic updates, which rule sets will it download? All available sets from the drop down list configured at IDS: https://ipfire:444/cgi-bin/ids.cgi or just the one and only that is selected and saved.
I guess the later is true because both pages (IDS and IDS update) show exactly the same timestamp for the latest ruleset update, am I correct?

2) Although some explanations have been given für option "Default policy". Does the updater automatically choose which rules are best for me according to this option? What exactly happens when selecting:
Connectivity:
Balanced:
Security:
Max-Detect:
in respect to my already checked or unchecked rules from https://ipfire:444/cgi-bin/ids.cgi ?

3) Enabled live updates: What does this option mean? I thought checking "Enable automatic updates" already does the job of automatic ruleset updates?

4) Apply policy changes: You stated
"it will enable or disable rules if their policy changes. So, for example, if you've got a rule selected that is in the balanced policy, and that is your selected policy, it will disable the rule if the policy of the rule changes to 'Security'. This is very rare"
I don't get the exact meaning of this quoted sentence. Does it mean, if I make any changes to option 2), the updater will automatically switch on/off rules that are not found within the new "Default policy"? This again raises the question where can I find the appropriate rules/rulesets behind Connectivity, Balanced, Security, Max-Detect?

5) Last but not least: Do I still have to check or unecheck those rules at https://ipfire:444/cgi-bin/ids.cgi for IDS updater to work correctly? Does this apply too for sub rules? Or is this job done by the updater according to the option chosen at 2)

Thanks for reading,
Michael
Image

Post Reply