IDS Rule updater - with rule state persistance
IDS Rule updater - with rule state persistance
I've now got a script running that will not only download Snort rule updates automatically, but will also persist the state of existing rules. So if you want to enable all the rules and still have them enabled after an update, you can now do this (but don't - it's a really bad idea to enable all the rules). It also includes more checks than the previous script, adds a log page so you can see what's going on and can email you when it does an update.
The caveat is that doing a manual update will reset the state of the rules; it's only automatic updates that will persist the state.
I've got it running on two machines and it seems to be working, but it should still be considered to be experimental. If you want to try it then go to:
https://github.com/timfprogs/ipfidsupdate
Make sure you read the instructions and especially the notes.
The caveat is that doing a manual update will reset the state of the rules; it's only automatic updates that will persist the state.
I've got it running on two machines and it seems to be working, but it should still be considered to be experimental. If you want to try it then go to:
https://github.com/timfprogs/ipfidsupdate
Make sure you read the instructions and especially the notes.
- Roberto Peña
- Posts: 761
- Joined: July 16th, 2014, 3:56 pm
- Location: Bilbao (España)
- Contact:
Re: IDS Rule updater - with rule state persistance
Good afternoon TimF.
It looks good. But when I install it, it gives me the following error:
Does the new page appear to you?
Greetings.
It looks good. But when I install it, it gives me the following error:
Nor do I see any new page as it puts on the GitHub.VERSION 100%[===================>] 2 --.-KB/s in 0s
2018-06-26 20:56:00 (139 KB/s) - 'VERSION' saved [2/2]
./install-idsupdate.sh: line 96: /2: syntax error: operand expected (error token is "/2")
---------------------------------------------------------------------
The system can check for an update to the rule files at a number
of different rates: Hourly, Daily or Weekly. It will check for
Does the new page appear to you?
Greetings.


╔════════════════════════════════════════════════╗
║ Donate to improve IPFire: https://www.ipfire.org/donate ║
╚════════════════════════════════════════════════╝
- Roberto Peña
- Posts: 761
- Joined: July 16th, 2014, 3:56 pm
- Location: Bilbao (España)
- Contact:
Re: IDS Rule updater - with rule state persistance
I'm sorry to say that by following the steps described, it has not been installed correctly.
I had to create or copy files by hand.
I have consulted this file and I have known what I had to copy:
If you need more information, do not hesitate to ask me.
Greetings and good work.
I had to create or copy files by hand.
I have consulted this file and I have known what I had to copy:
Code: Select all
snort-update.pl /usr/local/bin root.root 0755 ---> Copy by hand.
ids-update.en.pl /var/ipfire/addon-lang root.root 0004 ---> Copy by hand.
idsflowbits.cgi /srv/web/ipfire/cgi-bin root.root 0755 ---> Copy by hand.
idsupdate.dat /srv/web/ipfire/cgi-bin/logs.cgi root.root 0755 ---> Copy by hand.
EX-idsupdate.menu /var/ipfire/menu.d nobody.nobody 644 ---> Copy by hand.
install-idsupdate.sh . root.root 0755
Greetings and good work.

Last edited by Roberto Peña on June 26th, 2018, 8:05 pm, edited 2 times in total.


╔════════════════════════════════════════════════╗
║ Donate to improve IPFire: https://www.ipfire.org/donate ║
╚════════════════════════════════════════════════╝
- Roberto Peña
- Posts: 761
- Joined: July 16th, 2014, 3:56 pm
- Location: Bilbao (España)
- Contact:
Re: IDS Rule updater - with rule state persistance
Another thing that I have seen is that it sends the emails without subject. It would be interesting if there was an subject in the mail.
Greetings.
Greetings.


╔════════════════════════════════════════════════╗
║ Donate to improve IPFire: https://www.ipfire.org/donate ║
╚════════════════════════════════════════════════╝
Re: IDS Rule updater - with rule state persistance
I think I've fixed the problem - it was reading a null string for the downlink speed from the QOS settings and not handling it properly.
The lack of the log page and empty email subject is due to the language cache not being updated (the last thing the installer does). Running from the command line should fix this.
(Both the boxes I've got running the script have just sent me emails saying they've installed updates)
The lack of the log page and empty email subject is due to the language cache not being updated (the last thing the installer does). Running
Code: Select all
update-lang-cache
(Both the boxes I've got running the script have just sent me emails saying they've installed updates)
- Roberto Peña
- Posts: 761
- Joined: July 16th, 2014, 3:56 pm
- Location: Bilbao (España)
- Contact:
Re: IDS Rule updater - with rule state persistance
Thanks for answering.
Now it works correctly. When there is an improvement, post it.
Greetings.
Now it works correctly. When there is an improvement, post it.
Greetings.


╔════════════════════════════════════════════════╗
║ Donate to improve IPFire: https://www.ipfire.org/donate ║
╚════════════════════════════════════════════════╝
-
- Posts: 6
- Joined: June 12th, 2017, 4:50 am
- Location: Berlin
Re: IDS Rule updater - with rule state persistance
Hi TimF,
looks really nice, thanx for your work!
Greetings!
looks really nice, thanx for your work!
Greetings!

-
- Posts: 127
- Joined: May 31st, 2014, 8:22 pm
Re: IDS Rule updater - with rule state persistance
Hello TimF,
this is working great so far!
Thanks for making it available to the community.
regards
this is working great so far!
Thanks for making it available to the community.
regards
Re: IDS Rule updater - with rule state persistance
At the moment snort-update.pl does not get copied to /usr/local/bin most likely because it is not downloaded as stated (The installer will download the files and install them in the correct places)
Did not have too much time to look into this, but it is not working by only downloading the installer.
Did not have too much time to look into this, but it is not working by only downloading the installer.
Re: IDS Rule updater - with rule state persistance
Hopefully it's fixed now.
A minor problem with the code which is meant to stop downloading files if the latest version is already installed.
A minor problem with the code which is meant to stop downloading files if the latest version is already installed.
Re: IDS Rule updater - with rule state persistance
I've now uploaded a new version. I'm not entirely sure the installer will work correctly, so it's on a branch at the moment. You can find it at:
https://github.com/timfprogs/ipfidsupdate/tree/version3
The major change is in the handling of community rules. While it's true that the Talos VRT rules contain a version of the community rules, for the registered ruleset this is a month out of date, so the script will now update the community rules if the VRT ruleset is in use, and will ensure that only the rule in the community ruleset are used where the rule is found in both rulesets. This should ensure that the latest version of the rule is in use.
The full changelist:
https://github.com/timfprogs/ipfidsupdate/tree/version3
The major change is in the handling of community rules. While it's true that the Talos VRT rules contain a version of the community rules, for the registered ruleset this is a month out of date, so the script will now update the community rules if the VRT ruleset is in use, and will ensure that only the rule in the community ruleset are used where the rule is found in both rulesets. This should ensure that the latest version of the rule is in use.
The full changelist:
- Updated processing of community ruleset.
- Internal storage rearranged to use less memory.
- New WUI page added under 'Services' to configure the updater - the installer no longer asks questions.
- Added language files for French, German and Spanish. Unfortunately they're machine translated so I expect some errors.
- Summary of updates added to daily log summary.
- Check that all expected Snort processes are running is now more robust.
- Some name changes to prepare for the switch from Snort to Suricata.
- Drops privileges for the most critical sections of processing updates.
- Changed mechanism for deciding when to check updates.
- fcrontab -l has a reference to ids-update.pl near the end, and not one to snort-update.pl
- There's no directory /var/ipfire/snortupdate
- /var/ipfire/idsupdate and /var/ipfire/idsupdate/settings are owned by nobody
- The rule files in /var/tmp are owned by nobody (but the backup is owned by root).
-
- Posts: 127
- Joined: May 31st, 2014, 8:22 pm
Re: IDS Rule updater - with rule state persistance
Hi TimF,
very nice.
I am having an issue with the "previous" version on core 123, its not working there anymore.
Must the "old" script/install uninstalled first?
However i am trying to install this new version now and see how it goes.
One more question,
Quote: Some name changes to prepare for the switch from Snort to Suricata.
Who or what is preparing to switch to Suricata?
I would be interested to use it as well since its using more than one core.
thx
very nice.
I am having an issue with the "previous" version on core 123, its not working there anymore.
Must the "old" script/install uninstalled first?
However i am trying to install this new version now and see how it goes.
One more question,
Quote: Some name changes to prepare for the switch from Snort to Suricata.
Who or what is preparing to switch to Suricata?
I would be interested to use it as well since its using more than one core.
thx
Re: IDS Rule updater - with rule state persistance
Hi xPliZit_xs,
Have you any idea why it's stopped working in 123? If not a couple of things to check -
If the entry has gone from the crontab and you don't want to update to the new version, you should just be able to run the old installer.
The switch from Snort to Suricata is something the developers are working on. It was planned for IPFire V3, but they've decided to also implement it in V2 - it's ability to use multiple cores is, I think, one reason for the change. I can't tell you any more than that - all I've seen is a couple of entries in bugzilla and a couple of messages on the mailing list.
Have you any idea why it's stopped working in 123? If not a couple of things to check -
- Have a look at the crontab - log in as root and run fcrontab -l (lower case L). There should be a reference to snort-update.pl (ids-update.pl for the new version), probably near the bottom. A possibility is that core update 123 has replaced the crontab, removing this line.
- If the entry is in the crontab, try running the rule update script from the command line /usr/local/bin/snort-update.pl and see if that gives any errors.
If the entry has gone from the crontab and you don't want to update to the new version, you should just be able to run the old installer.
The switch from Snort to Suricata is something the developers are working on. It was planned for IPFire V3, but they've decided to also implement it in V2 - it's ability to use multiple cores is, I think, one reason for the change. I can't tell you any more than that - all I've seen is a couple of entries in bugzilla and a couple of messages on the mailing list.
-
- Posts: 127
- Joined: May 31st, 2014, 8:22 pm
Re: IDS Rule updater - with rule state persistance
Hi,
i recently migrated from bare metal ipfire to a virtualized environment and use the ipfire backup to restore my data.
Then i installed the snort updater and since then i don't saw it working again. Reinstalled it multiple times but no luck.
With the new version of the IDS updater i have not seen an update yet:
Last rule update was Fri Aug 24 17:09:57 2018 according the GUI.
Perhaps they don't update rules during the weekend.
This is at the end of the fcrontab file:
# Snort rule update
%nightly,nice(1),random(true),serialonce(true) 15-45 23-4 /usr/local/bin/snort-update.pl
# Snort rule update
%hourly,nice(1),random,serialonce(true) 6-16 /usr/local/bin/ids-update.pl
running it manually gives this:
[root@ipfire bin]# ./ids-update.pl
(6) Starting Snort update check
(7) Connection and disk space checks OK
(7) Reading Oinkmaster configuration
(7) Reading classification file /etc/snort/rules/classification.config
(7) Reading classification file /etc/snort/rules/EMERGING_THREATS_classification.config
(7) Check for Emerging Threats Open update
(7) Versions: Old c2b9efcdc00f799204598d9efcc77f82, new c2b9efcdc00f799204598d9efcc77f82
(6) No updates available
(6) Checking that Snort is running correctly
That looks OK i guess.
Assume that it should now work, i can probably remove the entry for the snortupdate since its outdated.
Thanks for your help.
regards
i recently migrated from bare metal ipfire to a virtualized environment and use the ipfire backup to restore my data.
Then i installed the snort updater and since then i don't saw it working again. Reinstalled it multiple times but no luck.
With the new version of the IDS updater i have not seen an update yet:
Last rule update was Fri Aug 24 17:09:57 2018 according the GUI.
Perhaps they don't update rules during the weekend.
This is at the end of the fcrontab file:
# Snort rule update
%nightly,nice(1),random(true),serialonce(true) 15-45 23-4 /usr/local/bin/snort-update.pl
# Snort rule update
%hourly,nice(1),random,serialonce(true) 6-16 /usr/local/bin/ids-update.pl
running it manually gives this:
[root@ipfire bin]# ./ids-update.pl
(6) Starting Snort update check
(7) Connection and disk space checks OK
(7) Reading Oinkmaster configuration
(7) Reading classification file /etc/snort/rules/classification.config
(7) Reading classification file /etc/snort/rules/EMERGING_THREATS_classification.config
(7) Check for Emerging Threats Open update
(7) Versions: Old c2b9efcdc00f799204598d9efcc77f82, new c2b9efcdc00f799204598d9efcc77f82
(6) No updates available
(6) Checking that Snort is running correctly
That looks OK i guess.
Assume that it should now work, i can probably remove the entry for the snortupdate since its outdated.
Thanks for your help.
regards
Re: IDS Rule updater - with rule state persistance
The entry for snort-update.pl should have been removed by the installer - I've corrected it.
The output from running it looks OK. The Emerging Threats rules are updated around midnight (UK time) on weekdays so the true test that it's working OK should come tonight. Hopefully tomorrow you'll be able to see the evidence that the rules have been updated in Services > IDS Update, Logs > IDS Update Logs, and Logs > Log Summary.
The output from running it looks OK. The Emerging Threats rules are updated around midnight (UK time) on weekdays so the true test that it's working OK should come tonight. Hopefully tomorrow you'll be able to see the evidence that the rules have been updated in Services > IDS Update, Logs > IDS Update Logs, and Logs > Log Summary.