IDS Rule updater - with rule state persistance

[TimF]

I've now got a script running that will not only download Snort rule updates automatically, but will also persist the state of existing rules. So if you want to enable all the rules and still have them enabled after an update, you can now do this (but don't - it's a really bad idea to enable all the rules). It also includes more checks than the previous script, adds a log page so you can see what's going on and can email you when it does an update.

The caveat is that doing a manual update will reset the state of the rules; it's only automatic updates that will persist the state.

I've got it running on two machines and it seems to be working, but it should still be considered to be experimental. If you want to try it then go to:

https://github.com/timfprogs/ipfidsupdate

Make sure you read the instructions and especially the notes.

[Roberto Peña]

Good afternoon TimF.

It looks good. But when I install it, it gives me the following error:

VERSION 100%[===================>] 2 --.-KB/s in 0s

2018-06-26 20:56:00 (139 KB/s) - 'VERSION' saved [2/2]

./install-idsupdate.sh: line 96: /2: syntax error: operand expected (error token is "/2")

---------------------------------------------------------------------

The system can check for an update to the rule files at a number
of different rates: Hourly, Daily or Weekly. It will check for

Nor do I see any new page as it puts on the GitHub.

Does the new page appear to you?

Greetings.

[Roberto Peña]

I'm sorry to say that by following the steps described, it has not been installed correctly.

I had to create or copy files by hand.

I have consulted this file and I have known what I had to copy:

Code: Select all

snort-update.pl /usr/local/bin root.root 0755  ---> Copy by hand.
ids-update.en.pl /var/ipfire/addon-lang root.root 0004 ---> Copy by hand.
idsflowbits.cgi /srv/web/ipfire/cgi-bin root.root 0755 ---> Copy by hand.
idsupdate.dat /srv/web/ipfire/cgi-bin/logs.cgi root.root 0755 ---> Copy by hand.
EX-idsupdate.menu /var/ipfire/menu.d nobody.nobody 644 ---> Copy by hand.
install-idsupdate.sh . root.root 0755

If you need more information, do not hesitate to ask me.

Greetings and good work.

[Roberto Peña]

Another thing that I have seen is that it sends the emails without subject. It would be interesting if there was an subject in the mail.

Greetings.

[TimF]

I think I've fixed the problem - it was reading a null string for the downlink speed from the QOS settings and not handling it properly.

The lack of the log page and empty email subject is due to the language cache not being updated (the last thing the installer does). Running

Code: Select all

update-lang-cache

from the command line should fix this.

(Both the boxes I've got running the script have just sent me emails saying they've installed updates)

[Roberto Peña]

Thanks for answering.

Now it works correctly. When there is an improvement, post it.

Greetings.

[Drexbengel48]

Hi TimF,

looks really nice, thanx for your work!

Greetings!

[xPliZit_xs]

Hello TimF,

this is working great so far!
Thanks for making it available to the community.

regards

[Deepcuts]

At the moment snort-update.pl does not get copied to /usr/local/bin most likely because it is not downloaded as stated (The installer will download the files and install them in the correct places)
Did not have too much time to look into this, but it is not working by only downloading the installer.

[TimF]

Hopefully it's fixed now.

A minor problem with the code which is meant to stop downloading files if the latest version is already installed.

[TimF]

I've now uploaded a new version. I'm not entirely sure the installer will work correctly, so it's on a branch at the moment. You can find it at:

https://github.com/timfprogs/ipfidsupdate/tree/version3

The major change is in the handling of community rules. While it's true that the Talos VRT rules contain a version of the community rules, for the registered ruleset this is a month out of date, so the script will now update the community rules if the VRT ruleset is in use, and will ensure that only the rule in the community ruleset are used where the rule is found in both rulesets. This should ensure that the latest version of the rule is in use.

The full changelist:

• Updated processing of community ruleset.
• Internal storage rearranged to use less memory.
• New WUI page added under 'Services' to configure the updater - the installer no longer asks questions.
• Added language files for French, German and Spanish. Unfortunately they're machine translated so I expect some errors.
• Summary of updates added to daily log summary.
• Check that all expected Snort processes are running is now more robust.
• Some name changes to prepare for the switch from Snort to Suricata.
• Drops privileges for the most critical sections of processing updates.
• Changed mechanism for deciding when to check updates.

Because of the name changes and the change to the timing the installer has to move some files about. It should do this correctly now (some of my tests didn't work fully), but you can check the following to make sure:

• fcrontab -l has a reference to ids-update.pl near the end, and not one to snort-update.pl
• There's no directory /var/ipfire/snortupdate
• /var/ipfire/idsupdate and /var/ipfire/idsupdate/settings are owned by nobody
• The rule files in /var/tmp are owned by nobody (but the backup is owned by root).

[xPliZit_xs]

Hi TimF,

very nice.
I am having an issue with the "previous" version on core 123, its not working there anymore.
Must the "old" script/install uninstalled first?
However i am trying to install this new version now and see how it goes.

One more question,
Quote: Some name changes to prepare for the switch from Snort to Suricata.

Who or what is preparing to switch to Suricata?
I would be interested to use it as well since its using more than one core.

thx

[TimF]

Hi xPliZit_xs,

Have you any idea why it's stopped working in 123? If not a couple of things to check -

• Have a look at the crontab - log in as root and run fcrontab -l (lower case L). There should be a reference to snort-update.pl (ids-update.pl for the new version), probably near the bottom. A possibility is that core update 123 has replaced the crontab, removing this line.
• If the entry is in the crontab, try running the rule update script from the command line /usr/local/bin/snort-update.pl and see if that gives any errors.

Updating to the new version should not require the old version to be removed - the installer should rename, move files about and change permissions as necessary. However I'm not entirely sure of this, which is why it's on a separate branch. I believe that I've fixed all the problems that came up installing it on my machines, but there could be additional problems on a different set up. It's a good ideas to do the checks at the end of my previous post.

If the entry has gone from the crontab and you don't want to update to the new version, you should just be able to run the old installer.

The switch from Snort to Suricata is something the developers are working on. It was planned for IPFire V3, but they've decided to also implement it in V2 - it's ability to use multiple cores is, I think, one reason for the change. I can't tell you any more than that - all I've seen is a couple of entries in bugzilla and a couple of messages on the mailing list.

[xPliZit_xs]

Hi,

i recently migrated from bare metal ipfire to a virtualized environment and use the ipfire backup to restore my data.
Then i installed the snort updater and since then i don't saw it working again. Reinstalled it multiple times but no luck.

With the new version of the IDS updater i have not seen an update yet:
Last rule update was Fri Aug 24 17:09:57 2018 according the GUI.
Perhaps they don't update rules during the weekend.

This is at the end of the fcrontab file:
# Snort rule update
%nightly,nice(1),random(true),serialonce(true) 15-45 23-4 /usr/local/bin/snort-update.pl

# Snort rule update
%hourly,nice(1),random,serialonce(true) 6-16 /usr/local/bin/ids-update.pl


running it manually gives this:
[root@ipfire bin]# ./ids-update.pl
(6) Starting Snort update check
(7) Connection and disk space checks OK
(7) Reading Oinkmaster configuration
(7) Reading classification file /etc/snort/rules/classification.config
(7) Reading classification file /etc/snort/rules/EMERGING_THREATS_classification.config
(7) Check for Emerging Threats Open update
(7) Versions: Old c2b9efcdc00f799204598d9efcc77f82, new c2b9efcdc00f799204598d9efcc77f82
(6) No updates available
(6) Checking that Snort is running correctly

That looks OK i guess.

Assume that it should now work, i can probably remove the entry for the snortupdate since its outdated.
Thanks for your help.

regards

[TimF]

The entry for snort-update.pl should have been removed by the installer - I've corrected it.

The output from running it looks OK. The Emerging Threats rules are updated around midnight (UK time) on weekdays so the true test that it's working OK should come tonight. Hopefully tomorrow you'll be able to see the evidence that the rules have been updated in Services > IDS Update, Logs > IDS Update Logs, and Logs > Log Summary.