IDS Rule updater - with rule state persistance

General questions.
TimF
Posts: 60
Joined: June 10th, 2017, 7:27 pm

Re: IDS Rule updater - with rule state persistance

Post by TimF » November 17th, 2018, 6:15 pm

Hi Michael,

1) It will update any rulesets that you've previously downloaded, not just the currently selected one. (It looks at the rulefiles in /etc/snort/rules - community.rules is the community rules, emerging-*.rules is Emerging Threats, anything else is Talos VRT). This means that any rules listed in the Intrusion Detection WUI will be kept up to date.

2) For new rules the updater will choose to enable or disable the rule based on the selected default policy. Rules you've previously checked or unchecked in the Intrusion Detection WUI will be left in the checked or unchecked state you selected unless:
  • You've checked 'Apply policy changes' and
  • An updated version of the rule has a different policy.
In this case the rule will be enabled or disabled depending on the chosen policy.

The policies Connectivity -> Balanced -> Security -> Max-Detect have increasing numbers of rules enabled, with 'Connectivity' having the least and 'Max-Detect' the most. Each policy includes all the rules from the lower policies. The updater works out which policy a rule belongs to based on information in the rule. If you just use the WUI to manually download an update it implicitly uses the 'Balanced' policy.

3) Enable live update affects how the updated rules are applied. The default (and the method used by a manual update) is to stop all the instances on Snort and then to re-start them with the new rules. The problem with this is that your network will not be protected by the Intrusion Detection rules during this period, which will probably be a few minutes.

If you select 'Enable live update' the updater will tell Snort to re-read the rules without stopping, which means you will be protected throughout the update, however to do this the process is similar to starting up another instance of Snort to read the new rules, and then swapping it with the existing instance; this means that this method uses quite a bit more memory. If you run out of memory, the system will kill a process (this will change under core update 125).

If you're short on memory and you don't have reason to expect your network to be deliberately targeted, it should be OK not to check this option.

For an estimate of the extra memory, look at how much memory your Snort processes are using under 'Status > Services' in the WUI - you'll use about as much extra memory as one of the Snort processes is using. This is in addition to the memory used by the updater itself - which could be up to 140MB, depending on the selected rulesets.

4) If you have 'Apply policy changes' checked and you change the default policy it will not make any changes to the already existing rules unless, at some point in the future, the rule is changed. In this case the rule will be enabled or disabled according to the selected default policy and the policy of the rule. If you have 'Apply policy changes' unchecked the updater will not make changes to the state of existing rules, but only to new rules.

Unfortunately there's no list of which rules belong in which policy (or at least I can't find one). The Snort FAQ gives the information as to how Talos VRT assign policies, but I doubt you'll find it very helpful. The updater attempts to synthesise the policy from the data included in the rule.

The updater applies the following algorithm to work out the policy:
  1. If there's metadata in the rule giving the policy, use it.
  2. otherwise use the priority of the rule with a priority of one corresponding to 'Connectivity' and four to 'Max-Detect'.
  3. make sure the resulting policy is either 'Connectivity' or 'Balanced' if the rule is distributed uncommented and 'Security' or 'Max-Detect' if it's commented (this corresponds to the process that the rule source uses to decide whether they're going to distribute the rule commented or uncommented).

The end result is a policy that should be a good approximation to the policy as decided by the supplier.

The most important outcomes of this process is that each policy has more rules included than the next lower policy and that selecting the 'Balanced' policy will select the same rules as just applying the update without any processing.


If you select 'Balanced' as your default policy you will get the same rules enabled or disabled as you would using a manual download of the rules, with only the changes you've made from the WUI.

5) You need to check or uncheck the rule categories that you want in the Intrusion Detection WUI; this controls whether Snort reads the rules from the corresponding rulefile. If a category is not checked, none of the rules in that category will be seen by Snort, no matter whether they're enabled or disabled or whether it's done manually or automatically from the updater. Note that the updater will still update the rules in the disabled categories - it's just that the rules won't be used.

For the individual rules under the top level categories, their state shown reflects the state of the rule. Any changes made by the updater will be shown here and any changes made manually will be taken into account the next time that an update is downloaded. this means that in most cases you can leave the state of individual rules to the updater, but you can still change the state of rules manually if you want.

I hope this answers all your questions adequately. Unfortunately some of the answers are a little completed.

Tim

Hellfire
Posts: 580
Joined: November 8th, 2015, 8:54 am

Re: IDS Rule updater - with rule state persistance

Post by Hellfire » November 17th, 2018, 8:45 pm

Thanks Tim for this comprehensive answer. I will go through your posting the next days.

Highly appreciated, thanks!
Image

skyfighter
Posts: 4
Joined: November 15th, 2018, 7:12 pm

Re: IDS Rule updater - with rule state persistance

Post by skyfighter » November 21st, 2018, 6:29 am

In the log files I get the error code "Oinkmaster failed returning 65280" and the number of activated rules doesn't change all the time.
What does that mean and what can I do?

TimF
Posts: 60
Joined: June 10th, 2017, 7:27 pm

Re: IDS Rule updater - with rule state persistance

Post by TimF » November 21st, 2018, 8:08 pm

The most likely explanation is that some pf the old rulefiles in /etc/snort/rules have the wrong permissions. All the files should be owned by nobody and -rw-r--r-- . To fix:

Code: Select all

chown nobody.nobody /etc/snort/rules/*
chmod 0644 /etv/snort/rules/*
If that doesn't work, try running /usr/local/bin/ids-update.pl from the command line, and look to see what errors you get.

skyfighter
Posts: 4
Joined: November 15th, 2018, 7:12 pm

Re: IDS Rule updater - with rule state persistance

Post by skyfighter » November 22nd, 2018, 5:29 pm

TimF wrote:
November 21st, 2018, 8:08 pm
The most likely explanation is that some pf the old rulefiles in /etc/snort/rules have the wrong permissions. All the files should be owned by nobody and -rw-r--r-- . To fix:

Code: Select all

chown nobody.nobody /etc/snort/rules/*
chmod 0644 /etv/snort/rules/*
If that doesn't work, try running /usr/local/bin/ids-update.pl from the command line, and look to see what errors you get.
This fixed it for me, thanks :)

User avatar
H&M
Posts: 436
Joined: May 29th, 2014, 9:38 pm
Location: Europe

Re: IDS Rule updater - with rule state persistance

Post by H&M » November 22nd, 2018, 8:27 pm

Hi Tim,

Will it work for a fresh installed system that has never downloaded any rule?

in my case (fresh instal) I get these:

Code: Select all

 /usr/local/bin/ids-update.pl
[root@black-x86-64 ~]#  /usr/local/bin/ids-update.pl
(6) Starting Snort update check
(7) Connection and disk space checks OK
(7) Reading Oinkmaster configuration
(3) Failed to open snort config file /etc/snort/rules/emerging.conf: No such file or directory
Failed to open snort config file /etc/snort/rules/emerging.conf: No such file or directory at /usr/local/bin/ids-update.pl line 1899.
I assume I have to make a manual download at least once, right?

PS: the snort.conf file with all active rules is already there, copied from an older system

Late edit:

IDSUpdate.cgi seems to have a problem to store the settings. No matter what I setup in there, next time when I open it is contains other values:

Download limit (kbit/s) = 0
Default policy = Connectivity (although I've setup Max-Detect)

Where are stored the values - just to check them...

Some errors are also reported by Squid in error_log:

Code: Select all

cat error_log
[Sun Nov 18 00:01:01.683533 2018] [mpm_event:notice] [pid 14563:tid 139745323794944] AH00489: Apache/2.4.34 (Unix) OpenSSL/1.1.0i configured -- resuming normal operations
[Sun Nov 18 00:01:01.683799 2018] [core:notice] [pid 14563:tid 139745323794944] AH00094: Command line: '/usr/sbin/httpd'
[Thu Nov 22 23:10:27 2018] idsupdate.cgi: Use of uninitialized value in string eq at /srv/web/ipfire/cgi-bin/idsupdate.cgi line 84.
[Thu Nov 22 23:10:27 2018] idsupdate.cgi: Use of uninitialized value $settings{"EMAIL"} in string eq at /srv/web/ipfire/cgi-bin/idsupdate.cgi line 193.
[Thu Nov 22 23:10:27 2018] idsupdate.cgi: Use of uninitialized value $mailsettings{"USEMAIL"} in string eq at /srv/web/ipfire/cgi-bin/idsupdate.cgi line 234.
[Thu Nov 22 23:10:42 2018] idsupdate.cgi: Use of uninitialized value $mailsettings{"USEMAIL"} in string eq at /srv/web/ipfire/cgi-bin/idsupdate.cgi line 234.
[Thu Nov 22 23:10:54 2018] idsupdate.cgi: Use of uninitialized value $mailsettings{"USEMAIL"} in string eq at /srv/web/ipfire/cgi-bin/idsupdate.cgi line 234.
[Fri Nov 23 08:14:01 2018] idsupdate.cgi: Use of uninitialized value in string eq at /srv/web/ipfire/cgi-bin/idsupdate.cgi line 84.
[Fri Nov 23 08:14:01 2018] idsupdate.cgi: Use of uninitialized value $mailsettings{"USEMAIL"} in string eq at /srv/web/ipfire/cgi-bin/idsupdate.cgi line 234.
[Fri Nov 23 08:14:15 2018] idsupdate.cgi: Use of uninitialized value $mailsettings{"USEMAIL"} in string eq at /srv/web/ipfire/cgi-bin/idsupdate.cgi line 234.
[Fri Nov 23 10:04:25 2018] idsupdate.cgi: Use of uninitialized value in string eq at /srv/web/ipfire/cgi-bin/idsupdate.cgi line 84.
[Fri Nov 23 10:04:25 2018] idsupdate.cgi: Use of uninitialized value $mailsettings{"USEMAIL"} in string eq at /srv/web/ipfire/cgi-bin/idsupdate.cgi line 234.
[Fri Nov 23 10:04:42 2018] idsupdate.cgi: Use of uninitialized value $mailsettings{"USEMAIL"} in string eq at /srv/web/ipfire/cgi-bin/idsupdate.cgi line 234.


Thanks,
H&M

TimF
Posts: 60
Joined: June 10th, 2017, 7:27 pm

Re: IDS Rule updater - with rule state persistance

Post by TimF » November 23rd, 2018, 6:59 pm

You need to manually download your rules before the updater will work - it uses the existing rule files to work out which sets of rule to download.

I think the error:

Code: Select all

(3) Failed to open snort config file /etc/snort/rules/emerging.conf: No such file or directory
is due to the old snort.conf file being used without any rule files existing. This should disappear when you download a set of rules.

For the second set of errors, the settings are in /var/ipfire/idsupdate: settings should be owned by nobody.nobody and status by root.root (status may not exist yet). Both should be -rw-r--r--. The directory should also be owned by nobody.nobody and drwxr-xr-x.

User avatar
H&M
Posts: 436
Joined: May 29th, 2014, 9:38 pm
Location: Europe

Re: IDS Rule updater - with rule state persistance

Post by H&M » November 23rd, 2018, 7:48 pm

Thank you,

I did not know about idsupdate folder existence...

For the record, here is how it looks after install:

Code: Select all

ls -l /var/ipfire/idsupdate/
total 264
-rw-r--r-- 1 root   root    74493 Nov 23 20:34 emerging_threats_oinkmaster.conf
-rw-r--r-- 1 nobody nobody    128 Nov 23 10:04 settings
-rw-r--r-- 1 root   root      141 Nov 23 20:42 status
-rw-r--r-- 1 root   root   180300 Nov 22 23:20 talos_vrt_oinkmaster.conf
I changed it like this:

Code: Select all

ls -l /var/ipfire/idsupdate/
total 264
-rw-r--r-- 1 nobody nobody  74493 Nov 23 20:34 emerging_threats_oinkmaster.conf
-rw-r--r-- 1 nobody nobody    128 Nov 23 10:04 settings
-rw-r--r-- 1 root   root      141 Nov 23 20:42 status
-rw-r--r-- 1 nobody nobody 180300 Nov 22 23:20 talos_vrt_oinkmaster.conf
That seems to solve all errors:

Code: Select all

 /usr/local/bin/ids-update.pl
(6) Starting Snort update check
(7) Connection and disk space checks OK
(7) Reading Oinkmaster configuration
(7) Reading classification file /etc/snort/rules/classification.config
(7) Reading classification file /etc/snort/rules/TALOS_VRT_classification.config
(7) Reading classification file /etc/snort/rules/EMERGING_THREATS_classification.config
(7) Check for Talos VRT registered or subscribed update
(7) Versions: Old 35bf06ce6b161e5f8d83d95bdb8abba6, new 35bf06ce6b161e5f8d83d95bdb8abba6
(7) Check for Emerging Threats Open No-GPL update
(7) Versions: Old e12c4ea090fb3bca80319a336971f7f3, new e12c4ea090fb3bca80319a336971f7f3
(6) No updates available
(6) Checking that Snort is running correctly
.

But there was an update available for emerging threats rules available ...

Snort.conf is manually updated to contain all rules - list is obtained with

Code: Select all

ls -1 /etc/snort/rules/*.rules
Then list is manually added to short.conf.

Thank you!

H&M

Hellfire
Posts: 580
Joined: November 8th, 2015, 8:54 am

Re: IDS Rule updater - with rule state persistance

Post by Hellfire » December 10th, 2018, 7:57 pm

TimF wrote:
November 17th, 2018, 6:15 pm
1) It will update any rulesets that you've previously downloaded, not just the currently selected one. (It looks at the rulefiles in /etc/snort/rules - community.rules is the community rules, emerging-*.rules is Emerging Threats, anything else is Talos VRT). This means that any rules listed in the Intrusion Detection WUI will be kept up to date.
Hi Tim,

would it be possible to start over with IDS with the following steps?

1) Delete all current rulessets from /etc/snort/rules/*.rules
2) Re-Download rule sets e.g. "Emerginthreats.net Community Rules" and "Snort/VRT GPLv2 Cummunity Rules"
3) Do not activate any of those listed categories and rules.
4) Change the default poliy of IDS Updater to e.g. "Security".
5) Activate option "Enable live update" and "Apply policy changes".
6) Manually update the rulsets with /usr/local/bin/ids-update.pl

Will those given steps automatically enable the appropriate categories and rules according to their policy and I do not have to worry about the rules anymore?

cu,
Michael
Image

TimF
Posts: 60
Joined: June 10th, 2017, 7:27 pm

Re: IDS Rule updater - with rule state persistance

Post by TimF » December 11th, 2018, 2:54 pm

Hi,

Unfortunately that won't work, since it'll read the state of the rulesets. I have though about adding a button to force the policy, but with the change to Suricata coming up that's not going to happen soon.

The only way I can think of to force a policy is:
  1. Delete all the rulefiles fron /etc/snort/rules, leaving one in each category that you're interested in, that is community.rules (for the community rules), emerging-*.rules (for the Emerging Threats rules) and everything else (for the Talos VRT rules).
  2. Empty these rule files, either with a text editor or by echo > <filename>.rules.
  3. Change the default policy of the updater and activate option "Apply policy changes".
  4. Delete the file /var/ipfire/idsupdate/status
  5. Either run the updater from the command line with /usr/local/bin/ids-update.pl or wait for the next update to be due.
I think this will do what you want, but I've not tried it out.

Hellfire
Posts: 580
Joined: November 8th, 2015, 8:54 am

Re: IDS Rule updater - with rule state persistance

Post by Hellfire » December 13th, 2018, 6:12 pm

TimF wrote:
December 11th, 2018, 2:54 pm
I think this will do what you want, but I've not tried it out.
Thank Tim, I will give it a try once my VM is up again...
Image

Hellfire
Posts: 580
Joined: November 8th, 2015, 8:54 am

Re: IDS Rule updater - with rule state persistance

Post by Hellfire » December 13th, 2018, 6:20 pm

Hi Tim,

I really doubt that IDS Updater is doing anything on my IPFire - this does not mean the updater works wrong, that's no rant at all, however, the appropriate logs don't list any changes within the rules.

Here is the current log and my configuration:
Log:
IDS Updater Log.png
Updater config:
IDS Updater.png
IDS config:
IDS Configuration.png
So maybe I'm totally wrong regarding the purpose and functioning of IDS updater or sthg does not work properly on my side. I can see from the logs and statistics that some rule definitions have been downloaded, but I see no changes, at least not in the logs.

Because of the vast amount of rules I even cannot say for sure that any of those rules have been touched the last two days. Please don't bother, a question: if the updater decided that a rule must be activated according to the policy set and the appropriate options, I assume that this change hould be logged, right. May I further assume that any of those rules will get a (new) checkmark, too?

Thank you,
Michael
Image

Hellfire
Posts: 580
Joined: November 8th, 2015, 8:54 am

Re: IDS Rule updater - with rule state persistance

Post by Hellfire » December 13th, 2018, 6:31 pm

Next question I would like to start in a different posting:

The IDS updater logs shows up a link for some, let's say, misconfigurated flowbits/rules sets, like:
Flowbits.png
So I tried to find which rule is the source of those issues. How would you start searching?

Let's take this example below. What does #1 exactly mean? Should I disable this rule because I've not set up any depending rules?

Let's head over to #2 There is a type test and set/unset: I assume that test was done by the updater and the action I've to take is to set or unset an appropriate sub-rule to get "[default]ET.Adobe.Site.Download" working, correct?

Anyway - each of my questions leads me to the main one: how do I find those rules in question? I've tried to perform a text search in browser, but the main problem I'm facing is, that I do not know which category the rule is in.
Flowbits example.png
Flowbits example.png (10.25 KiB) Viewed 598 times
Thanks again,
Michael
Image

Hellfire
Posts: 580
Joined: November 8th, 2015, 8:54 am

Re: IDS Rule updater - with rule state persistance

Post by Hellfire » December 13th, 2018, 7:25 pm

Hellfire wrote:
December 13th, 2018, 6:12 pm
TimF wrote:
December 11th, 2018, 2:54 pm
I think this will do what you want, but I've not tried it out.
Thank Tim, I will give it a try once my VM is up again...
Unfortunately this did not work.

I preserved two files from each rule set as suggested: emerging-exploit.rules and community.rules, deleted the other *.rules and wiped the content. Afterwards, I manually ran IDS updater and let it finish. At the end, both files are still empty and no further rule sets/categories have been loaded or created.
So I headed over to the IDS WebIf page and forced an update of community rules and emergingthreats.net - to no avail either. No futher files have been loaded. I then added Talos rule set for registered users and re-tried. Now some (new) files were fetched.

Once again I deleted the status file from the updater and manually ran ids-update.pl. After its finish I checked the rules files last moditication date and time and noticed that none of them have been touched in any way. No log was created either.

OK, last try: change the default policy from Security to Max-Detect and pressed save. The list box switched back to Connectivity, hence I could not finally test if changing to a new policy would force the IDS updater to modify any *.rules files.

Hence, I think the IDS Updater did not made any changes to the rules according to the default policy which is set to Security at the moment.

Michael
Image

TimF
Posts: 60
Joined: June 10th, 2017, 7:27 pm

Re: IDS Rule updater - with rule state persistance

Post by TimF » December 13th, 2018, 8:56 pm

it will only do an update when the rules have changed. It determines this by looking at the MD5SUM (this prevents it from downloading the large rulefiles unless necessary). You can force a change by modifying the appropriate lines in /var/ipfire/idsupdate/status. I usually just add a 'z' to the end of the appropriate lines since it's not a valid hex character.

Post Reply