IDS, Intrusion Detection System. What rule provider is best and what rules are best?

General questions.
Post Reply
User avatar
raffe
Posts: 17
Joined: August 20th, 2018, 8:40 am

IDS, Intrusion Detection System. What rule provider is best and what rules are best?

Post by raffe » September 26th, 2018, 7:55 am

Hi!

I have two questions.

What Snort Rules supplier is best for IPFire (and why)?
Some examples of what I have read that make we wonder, and I don't feel I am capable of doing an educated choice because I know to little about these things.
  • If I choose one or another, will the rules under "intrusion detection system rules" change, or can I choose the same rules regardless of rule supplier? When I change rule supplier, all under "intrusion detection system rules" seems to be the same and the same check boxes I checked before are still chosen.
  • I think I also read somewhere that there was problems with rules that have been chosen were reset when updating (but not if you use https://forum.ipfire.org/viewtopic.php?f=27&t=20965 ). Is that the same for all rule suppliers
  • When I read https://forum.ipfire.org/viewtopic.php? ... e&start=30 VRT seem to have problems, so should I therefore go with Emergingthreaths? Or if they all work, is one of them better?
  • If I read https://forum.ipfire.org/viewtopic.php?f=27&t=21014 I read that Emerging Threats ruleset includes rules that just look for known suspect IP addresses. Using Snort rules to block IP addresses this way is actually inefficient. There is a more efficient way of doing it in Snort using the reputation processor, but this doesn't work in the way that Snort is set up on IPFire, which is why the Talos VRT equivalent isn't enabled. There has been some discussion about providing a method of loading IP address blacklists into the firewall, but I'm not aware of any decision being made.
    Is it therefore better with Emerging Threats IP blocking, or better without? Is it about how much CPU/RAM is uses or is it unsecure?
So, what free "Snort rules update" is best when I use IPFire? Is it "Emergingthreaths.net, "Snort/VRT GPLv2 Community Rules" or "Sourcefire VRT rules for registered users"?

What Snort Rules are best for a general home user (and why)?
I have a family and we do our everyday surfing and some gaming. Sometimes my kid have friends over, and they use Internet. I have a SFTP server, but otherwise no web servers etc.

For this setup I have chosen these rules, and it seems to be working.

Code: Select all

No = app-detect.rules
Yes = attack-responses.rules
Yes = backdoor.rules
Yes = bad-traffic.rules
Yes = blacklist.rules
Yes = botnet-cnc.rules
Yes = browser-chrome.rules
Yes = browser-firefox.rules
Yes = browser-ie.rules
Yes = browser-other.rules
Yes = browser-plugins.rules
Yes = browser-webkit.rules
No = chat.rules
Yes = community.rules
Yes = content-replace.rules
Yes = ddos.rules
Yes = dns.rules
No = emerging-activex.rules
Yes = emerging-attack_response.rules
Yes = emerging-botcc.portgrouped.rules
Yes = emerging-botcc.rules
No = emerging-chat.rules
No = emerging-ciarmy.rules
Yes = emerging-compromised.rules
Yes = emerging-current_events.rules
No = emerging-deleted.rules
No = emerging-dns.rules
Yes = emerging-dos.rules
No = emerging-drop.rules
Yes = emerging-dshield.rules
Yes = emerging-exploit.rules
No = emerging-ftp.rules
No = emerging-games.rules
No = emerging-icmp.rules
No = emerging-icmp_info.rules
No = emerging-imap.rules
No = emerging-inappropriate.rules
No = emerging-info.rules
Yes = emerging-malware.rules
No = emerging-misc.rules
Yes = emerging-mobile_malware.rules
No = emerging-netbios.rules
No = emerging-p2p.rules
No = emerging-policy.rules
No = emerging-pop3.rules
No = emerging-rbn-malvertisers.rules
No = emerging-rbn.rules
No = emerging-rpc.rules
No = emerging-scada.rules
Yes = emerging-scan.rules
No = emerging-shellcode.rules
No = emerging-smtp.rules
No = emerging-snmp.rules
No = emerging-sql.rules
No = emerging-telnet.rules
No = emerging-tftp.rules
No = emerging-tor.rules
No = emerging-trojan.rules
No = emerging-user_agents.rules
No = emerging-voip.rules
No = emerging-web_client.rules
No = emerging-web_server.rules
No = emerging-web_specific_apps.rules
Yes = emerging-worm.rules
No = experimental.rules
Yes = exploit-kit.rules
Yes = exploit.rules
Yes = file-executable.rules
Yes = file-flash.rules
Yes = file-identify.rules
Yes = file-image.rules
Yes = file-java.rules
Yes = file-multimedia.rules
Yes = file-office.rules
Yes = file-other.rules
Yes = file-pdf.rules
No = finger.rules
No = ftp.rules
No = icmp-info.rules
No = icmp.rules
No = imap.rules
Yes = indicator-compromise.rules
Yes = indicator-obfuscation.rules
Yes = indicator-scan.rules
No = indicator-shellcode.rules
Yes = malware-backdoor.rules
Yes = malware-cnc.rules
Yes = malware-other.rules
Yes = malware-tools.rules
No = misc.rules
No = mysql.rules
No = netbios.rules
No = nntp.rules
No = oracle.rules
Yes = os-linux.rules
Yes = os-mobile.rules
No = os-other.rules
No = os-solaris.rules
Yes = os-windows.rules
Yes = other-ids.rules
No = p2p.rules
No = phishing-spam.rules
No = policy-multimedia.rules
No = policy-other.rules
No = policy-social.rules
No = policy-spam.rules
No = policy.rules
No = pop2.rules
No = pop3.rules
No = protocol-dns.rules
No = protocol-finger.rules
Yes = protocol-ftp.rules
No = protocol-icmp.rules
No = protocol-imap.rules
No = protocol-nntp.rules
No = protocol-other.rules
No = protocol-pop.rules
No = protocol-rpc.rules
No = protocol-scada.rules
No = protocol-services.rules
No = protocol-snmp.rules
No = protocol-telnet.rules
Yes = protocol-tftp.rules
No = protocol-voip.rules
No = pua-adware.rules
No = pua-other.rules
No = pua-p2p.rules
No = pua-toolbars.rules
No = rservices.rules
No = server-apache.rules
No = server-iis.rules
No = server-mail.rules
No = server-mssql.rules
No = server-mysql.rules
No = server-oracle.rules
No = server-other.rules
No = server-samba.rules
No = server-webapp.rules
No = smtp.rules
No = specific-threats.rules
Yes = spyware-put.rules
Yes = virus.rules
No = web-activex.rules
No = web-attacks.rules
No = web-cgi.rules
Yes = web-client.rules
No = web-coldfusion.rules
No = web-frontpage.rules
No = web-iis.rules
No = web-misc.rules
No = web-php.rules
No = x11.rules
Do that seem reasonable? Have I missed something? How do you think when you choose the rules?
Image

TimF
Posts: 83
Joined: June 10th, 2017, 7:27 pm

Re: IDS, Intrusion Detection System. What rule provider is best and what rules are best?

Post by TimF » September 26th, 2018, 9:46 pm

Hi,

The answer to your questions is (unsurprisingly) it depends.

(Warning - long post coming up)

First some definitions for the sake of discussion:
  • Ruleset - a complete set of rules coming in a single download. Contains a number of rulefiles. These are what you select in the pulldown on the WUI.
  • Rulefile - a set of rules that address a similar problem area. Contains a number of rules. These are what you select with the checkboxes in the WUI when the page comes up initially.
  • Rule - a set of instructions for detecting a condition.
Note that rules are not necessarily independent - for example a rule to detect malware in a PDF file will rely on another rule that detects that the traffic is actually a PDF file - if the latter rule isn't enabled, the former rule won't work. The basic detection rules are generally in emerging-policy, emerging-info, and file-identify (I think) - but there are other dependencies.

The rulesets are:
  • Talos VRT Subscription. This is probably the best ruleset - apparently they expend quite a bit of effort in order to ensure that the ruleset is of high quality. Downsides - it's generally only updated twice a week and you have to pay for it. Note Talos VRT was formerly known as Sourcefire.
  • Talos VRT Registered. This a one month old version of the subscription ruleset. The quality is the same, you don't have to pay, but it is a month old.
  • Emerging threats open. This is a free ruleset. It's smaller than the Talos VRT ruleset, and is therefore less comprehensive. It's updated every weekday. These rules are in rulefiles starting with 'emerging-' on the WUI.
  • Community rules. While this is available separately, it's also included in all the other rulesets, although it's a month out of date in the Talos VRT Registered ruleset and of uncertain age in the Emerging Threats ruleset. Note that this ruleset is curated by Talos VRT. If you download this separately these rules are in the rulefile 'community.rules' on the WUI.
Note that you can have multiple rulesets installed, which is why the available rulefiles doesn't seem to change when you download a new set. This isn't a problem, but if you really want to get rid of an old ruleset do the following:
  1. Log on to the system via ssh.
  2. Delete the appropriate rule files from /etc/snort/rules:
    • Community rules - community.rules
    • Emerging threats - rule files starting emerging-
    • Talos VRT - any other rule file.
  3. Edit /etc/snort/snort.conf and delete the include lines at the end of the file corresponding to the rule files you've deleted - don't just comment the lines out.
So, what ruleset should you use? For the sake of completeness (since other people might view this post later) I'll cover more scenarios than just yours. Note that you may well be constrained by how fast your IPFire computer is and how much memory it's got.
  • Large business/organisation. You should either have your own cyber security team or a contract with a specialist company. Ask their advice - if they can't give it or can't explain it fire them and get someone better. The answer won't be 'load this ruleset on a computer running IPFire' (or if it is, fire them) - it'll be more complicated and, of course, more expensive. This is also the advice for any organisation where the consequences of compromise are serious (for example, hospitals, utilities etc), no matter what size.
  • Medium sized business/organisation. Consider if you need a cyber security team, but the minimum would be the Talos VRT Subscription with a large number of rules enabled. If you're using my automatic update script you would be aiming to use the Security-over-Connectivity policy.
  • Small business/organisation. Consider the consequences of a compromise. If they're serious, either to you or to someone else (don't forget your responsibilities under the GDPR), you should be using the Talos VRT Subscription, otherwise you may get by with either Talos VRT Registered or Emerging Threats.
  • Home use. The Emerging Threats ruleset is probably sufficient, but you could use the Talos VRT Registered set. A Policy of Balanced-between-Security-and-Connectivity is probably sufficient. If you volunteer for a charity or similar and as a consequence keep either personal or financial information on your home network, you should consider the Talos VRT Subscription, but you should be eligible for the personal use licence, which is much cheaper.
As well as this, if your computer has limited processing power or memory, it may tip the balance towards the Emerging Threats ruleset.

This is all in my opinion, and I am not a cyber security professional.

So what rules do I use? None of the above scenarios. Note that (probably due to security briefings long ago) I tend to be careful a little more careful than many people on these matters.

I run two IPFire systems: one at home, and one for a small charity. In both cases I use the Talos VRT Registered, Emerging Threats open and community rules. I primarily use the Talos VRT set, with the community ruleset to bring the community rules up to date and then I add the emerging-current-events fulefile to address newer threats than the month old Talos VRT registered rules. In consequence I have about 10000 rules enabled. I also run an IP blocklist (more on that later).

I have no problems with the Talos VRT ruleset. I think that the reason for the reports of problems is that the Emerging Threats rulesets, by default, has IP blocklist rules enabled. These generate an alert when a packet is received from one of the IP Addresses; you can get from several hundred to several thousand of these per day, however this is misleading because these packets would be blocked by the firewall anyway. If you disable these rulefiles you'll get very few rules alerting - most days I don't get any. This is a good thing - if you're getting several hundred alerts dues to IP addressing being blocked, it's unlikely you'll spot the small number of messages that tell you you've actually been compromised. The Talos VRT rulesets don't have equivalent rules enabled and so you don't get the large number of spurious messages.

Do you need IP Blocklists? If you're not exposing any services to the internet then probably not. The firewall's default rules will block any unsolicited traffic unless it happens to hit a port that's open due to an outgoing connecting, and in that case the application using the port will almost certainly reject the traffic.

If you're providing a service visible from the internet then a blocklist is probably a good idea. In this case rather than using the IDS rules I would use one of the scripts that are intended to install a blocklist in the firewall - this solves the problem of not being able to see important IDS alerts. There's at least three scripts about, but I can only find two at the moment:
The reason for not using the Emerging Threats rulefiles for doing this blocking is twofold: Firstly it's inefficient - blocking using the firewall uses less memory and less processor power, and secondly, as mentioned above, you tend to get so many alerts triggered by these rules that you can't see the alerts you actually need to worry about.

What rulefiles should you enable? That's going to depend on the use you make of the system, the amount of memory you've got and the speed of your computer in reference to the speed of the internet connection.

In general, the Emerging Threats rulefile for a threat are will have fewer rules than the equivalent Talos VRT rulefile, so if you're short of memory or processing power go for the Emerging Threats version. I also don't see any point in including both the Talos VRT and Emerging Threats rules for the same topic.

So, go through the list of rulefiles and decide which ones are relevant to your situation. Bear in mind that you may have to include some rulefiles you don't expect, like the emerging-policy, emerging-info, and file-identify ones that are relied on by rules in other rulefiles. Also, I remember reading an article an number of years ago that WINE had advanced to the point where it was capable of running some Windows malware, so even if your computers are all Linux, it may be appropriate to enable some Windows rules. The same applies to Open/Libre Office and Microsoft Office - each can open the other's files.

In your case the list of rulefiles looks reasonable, but you need to enable emerging-policy and emerging-info, and double check that you're not using both the Emerging Threats and Talos VRT versions of a rule.

Once you've set up some rules, let them run for a while and then check memory usage (Status -> Memory from the WUI). If it looks like you're running short of memory you'll have to use fewer rules. Also check the processor usage (Status -> System). Again, if you're running short of processor power you need to use fewer rules.

You also need to check for IDS alerts (Logs -> IDS Logs), preferably daily. You may well get the occasional report of malware from someone on the internet probing your system, but if you get lots of reports of malware then you'll have to track down the device that's responsible and clean it (which is another topic).

Finally, you need to keep your rules up to date. There are new threats appearing everyday, so this is vital. You can do this from the WUI by selecting the ruleset and then downloading it, or you can use one of the scripts:
If you use the last script it will remember the enable/disable state of individual rules and you can also select a policy which determines whether new or changed rules are enabled or disabled.. The drawback is that this script uses a lot of memory when it runs. Using the other script or a manual update will reset the enable/disable state of individual rules to a default (which corresponds to the policy balanced-between security-and-connectivity. This may or may not be a problem. If you find that something your doing on your network triggers rules and that leads to guardian blocking the device, you then either have to use the second script or alternatively do manual updates and resign yourself to having to change the rule state each time.

I hope this helps answer you questions. Unfortunately there's no simple answer, and you may well have to make adjustments to get everything working correctly. I had to do quite a bit of work initially, but now my systems are in a state where I just need to check the logs regularly, otherwise they run quite happily without my intervention.

silverknight
Posts: 15
Joined: June 27th, 2010, 2:01 pm

Re: IDS, Intrusion Detection System. What rule provider is best and what rules are best?

Post by silverknight » September 28th, 2018, 9:58 pm

Just had to reply to say that TimF laid out an extremely well written and accurate response. This is wiki worthy honestly.

Hellfire
Posts: 694
Joined: November 8th, 2015, 8:54 am

Re: IDS, Intrusion Detection System. What rule provider is best and what rules are best?

Post by Hellfire » September 29th, 2018, 7:44 pm

++++

This would be worth adding to the wiki as silverknight suggested more or less ;)
Image

cibgiu
Posts: 28
Joined: November 7th, 2012, 12:53 pm

Re: IDS, Intrusion Detection System. What rule provider is best and what rules are best?

Post by cibgiu » October 1st, 2018, 6:13 am

Hi TimF,

thanks for sharing your information, knowledge and time.

C.

User avatar
raffe
Posts: 17
Joined: August 20th, 2018, 8:40 am

Re: IDS, Intrusion Detection System. What rule provider is best and what rules are best?

Post by raffe » October 2nd, 2018, 8:52 am

Thanks for an extremely good answer! ;D I am still digesting the content and planing how to proceed :D

I have added the answer to the Wiki: https://wiki.ipfire.org/configuration/services/ids (scroll down to 'What rule provider is best and what rules are best?')
Image

Hellfire
Posts: 694
Joined: November 8th, 2015, 8:54 am

Re: IDS, Intrusion Detection System. What rule provider is best and what rules are best?

Post by Hellfire » October 2nd, 2018, 10:40 am

raffe wrote:
October 2nd, 2018, 8:52 am
I have added the answer to the Wiki: https://wiki.ipfire.org/configuration/services/ids (scroll down to 'What rule provider is best and what rules are best?')
PERFECT! Thanks!
Image

dnl
Posts: 375
Joined: June 28th, 2013, 11:03 am

Re: IDS, Intrusion Detection System. What rule provider is best and what rules are best?

Post by dnl » November 8th, 2018, 9:56 am

Thanks for the great post TimF. I've still got a lot to learn about using an IDS!
IPFire 2.x (Latest Update) on x86_64 Intel Bay Trail CPU, 4GiB RAM, RED + GREEN + BLUE + ORANGE

Post Reply