Snort Rules Update

General questions.
dnl
Posts: 375
Joined: June 28th, 2013, 11:03 am

Re: Snort Rules Update

Post by dnl » November 6th, 2018, 8:33 am

I'm not sure what has changed, but since the last core update to IPFire this script no longer checks and downloads rules when necessary. It is downloading rules every single run. Since I had it running in /etc/fcron.hourly, I've been downloading a 98 MiB snortrules.tar.gz file every hour, or 2352 MiB per day!

The ETVersion.txt file appears to be preventing ET rules being downloaded every day. It's only the snort rules which are a problem.

I don't have a large download quota, which is why I noticed. Can you please check that the md5 check is still working?
IPFire 2.x (Latest Update) on x86_64 Intel Bay Trail CPU, 4GiB RAM, RED + GREEN + BLUE + ORANGE

gitarman94
Posts: 7
Joined: March 17th, 2018, 1:37 pm
Location: US

Re: Snort Rules Update

Post by gitarman94 » November 7th, 2018, 11:21 am

??? So, I'm not quite sure how to fix this issue. I changed the code to correct for the issue that was discovered, however, everything is now working as expected except for the fact that the oinkmaster.pl file doesn't appear to be incrementing the version of Snort. Below is the results of running the updated script, it's discovering the version difference in Snort (29111 vs 29120), it downloads the newer correct version, and then runs it. The second command below is a manual run of the install script and it's normal looking output and supposed success. In the end, no matter what I do, the version number on Snort doesn't update to 29120 (29.12.0). Not sure if this is a bug on their side, or if it's something I missed. Going to attach the new code here and hope that someone else can do a verification that I didn't miss something.


[root@IPAddress]# perl snortupdate.pl
Use of uninitialized value $vnumer in substitution (s///) at snortupdate.pl line 146.
Use of uninitialized value $vnumer in numeric gt (>) at snortupdate.pl line 148.
currentWebSnortVersion = 29120
local_vrt_v = 29111
currentWebETVersion = 9044
currentLocalETVersion = 9044
Update found!
Downloading... https://www.snort.org/rules/snortrules- ... ?oinkcode=####
Processing rules downloaded...
Update Successfull for https://www.snort.org/rules/snortrules- ... ?oinkcode=####
-----------------------
[root@IPAddress]# /usr/local/bin/oinkmaster.pl -v -s -u file:///var/tmp/snortrules.tar.gz -C /var/ipfire/snort/oinkmaster.conf -o /etc/snort/rules
Loading /var/ipfire/snort/oinkmaster.conf
Adding file to ignore list: local.rules.
Adding file to ignore list: deleted.rules.
Adding file to ignore list: snort.conf.
Found gzip binary in /bin
Found tar binary in /bin
Copying file from /var/tmp/snortrules.tar.gz... done.
Archive successfully downloaded, unpacking... done.
Setting up rules structures... done.
Processing downloaded rules...
disabled 0, enabled 0, modified 0, total=34008
Setting up rules structures... done.
Comparing new files to the old ones... done.

[***] Results from Oinkmaster started 20181106 15:20:22 [***]

[*] Rules modifications: [*]
None.

[*] Non-rule line modifications: [*]
None.

Code: Select all

#!/usr/bin/perl
#################################
# Snort Rules Update for IPFire #
# ----- Contributions by: ----- #
# --------- Kick@ss ----------- #
# ----------- H&M ------------- #
# -------- gitarman94 --------- #
#################################

use autodie ':io';
use strict;
use warnings;
use HTTP::Request;
use LWP::UserAgent;
use LWP::Protocol::https;


my $Version = `snort -V 2>&1 | grep 'Version'`;
my ($local_vrt_v) = $Version =~ m/(\d+\.[\d\.]*)/;
$local_vrt_v =~ s/\.//g;
my $oinkcode='none';
my $Snort_Orange = 0;
my $Snort_BLUE = 0;
my $Snort_GREEN = 0;
my $Snort_Active = 0;
my $url = '';
getSnortSettings();
my $currentWebSnortVersion;
if ($oinkcode !~ m/none/)
{
    my $snortList = webGet('https://snort.org/downloads/registered/md5s.txt');
    $currentWebSnortVersion = webSnortParse($snortList);#iteration through list and sets this var to current sub-point release number from website
};
my $currentLocalETVersion = readFile("/var/tmp/ETVersion.txt"); #itterate through local file to get currently installed ET version
my $ETList = webGet('https://rules.emergingthreats.net/version.txt');
my $ETSnortVersion = webGet('http://rules.emergingthreats.net/open/');
my $currentWebETVersion = webETParse();
parseETPrimary();
versionComparer();
verifySnortIsRunning();
exit(0);

sub parseETPrimary
{
  my $tempET;
  while ($ETSnortVersion =~ m/(snort-\d+\.\d+\.\d+\.*)/g)
  {
    $tempET = $1;
    last;
  }
  $ETSnortVersion = $tempET;
}

sub getSnortSettings
{
  open my $fh, '<', "/var/ipfire/snort/settings"; #open file
  while (my $line = <$fh>) #itterate throught the file line by line
  {
      if ($line =~ m/ENABLE/ | $line =~ m/OINKCODE/) #only cares about lines that have the word 'ENABLE' in them
      {
          if ($line =~ m/ORANGE/) #looking for orange settings
          {
              my @lineItems = split('[=]',$line);
              if ($lineItems[1] =~ m/on/) #if the setting is "on"
              {
                  $Snort_Orange = 1; #default is 0 but if orange is enabled then make it so
              }
          }
          elsif ($line =~ m/BLUE/) #looking for blue settings
          {
              my @lineItems = split('[=]',$line);
              if ($lineItems[1] =~ m/on/) #if the setting is "on"
              {
                  $Snort_BLUE = 1; #default is 0 but if blue is enabled then make it so
              }
          }
          elsif ($line =~ m/GREEN/) #looking for green settings
          {
              my @lineItems = split('[=]',$line);
              if ($lineItems[1] =~ m/on/) #if the setting is "on"
              {
                  $Snort_GREEN = 1; #default is 0 but if green is enabled then make it so
              }
          }
          elsif ($line =~ m/OINKCODE/) #get oinkcode from within the settings file automatically
          {
              my @lineItems = split('[=]',$line);
              if (length($lineItems[1]) == 41) #get the length and match to the 41 character string it should be, if not proper length, then it's not a real oinkcode
              {
                  $oinkcode = $lineItems[1];
              }
              elsif ((length($lineItems[1]) gt 2 && length($lineItems[1]) lt 41) || length($lineItems[1]) gt 41)
              {
                system("logger -t SnortUpdate 'WARNING: OINKCODE appears to be of incorrect length.'");
              }
          }
          else
          {
              my @lineItems = split('[=]',$line);
              if ($lineItems[1] =~ m/on/) #if the setting is "on"
              {
                  $Snort_Active = 1; #default is 0 but if red is enabled then make it so
              }
          }
      }
  }
  close $fh;
}

sub webGet
{ #used to http query snort for current version of version 2
    my $url = shift;
    my $ua = LWP::UserAgent->new( ssl_opts => { verify_hostname => 1 } );
    my $response = $ua->get($url);

    my $test1 = $response->decoded_content;
  #  my $request = HTTP::Request->new(GET => $url);
  #  my $response = $ua->request($request);
    if ($response->is_success)
    {
        return $response->decoded_content; #returns list found on above site
    }
}

sub webETParse
{
    chomp $ETList; #remove extra line/spaces
    return $ETList;
}

sub webSnortParse
{  #Itterator to split and parse and get the snort version via the MD5 website
    my $snortList = shift;
    my $current_md5;
    my $vnum = 0;
    foreach my $line (split('\n',$snortList))
    {
      chomp $line;

      next unless ($line);
      my @file = split(/\s+/, $line );
      my @vstring = split( '[.]', $file[2], 2 );
      my $testfile = $vstring[0];
      my @vnumCompare = split( '-', $vstring[0] );
      my $vnumer = $vnumCompare[2];
      $vnumer =~ s/\.//g;

      if ($vnumer > $vnum)
      {
        $vnum = $vnumer;
      }
    }
    return $vnum;
}

sub readFile
{
    my $localFile = shift; #we're passing a variable into the subroutine, this sets it to a local variable
    my $currentLocalVersion = 0; #default value, in case we don't find the file
    if (-e $localFile)
    {
        open my $fh, '<', $localFile; #open file
        while (my $row = <$fh>) #itterate throught the file line by line (if multiple)
        {
            chomp $row; #removes empty lines and takes only good data
            $currentLocalVersion = $row; #should only be 1 line in the file so we'll take whatever it gives us
        }
        close $fh;
    }
    return $currentLocalVersion;
}


sub versionComparer
{ #now we test that above WebVersions against our local versions, if newer we'll update
      # VRT Community
          #$url=" https://www.snort.org/rules/community";
      #EmergingThreats Community
          #$url="http://rules.emergingthreats.net/open/$ETSnortVersion/emerging.rules.tar.gz";
      # VRT Subscripted & VRT Community
          #$url="https://www.snort.org/rules/snortrules-snapshot-$currentWebSnortVersion.tar.gz?oinkcode=$oinkcode\n https://www.snort.org/rules/community";
      # VRT Subscripted & EmergingThreats Community
          #$url="https://www.snort.org/rules/snortrules-snapshot-$currentWebSnortVersion.tar.gz?oinkcode=$oinkcode\n http://rules.emergingthreats.net/open/$ETSnortVersion/emerging.rules.tar.gz";
      my $performUpdate = 1;
      #We're testing for " !~ m/none/ " in the oinkcode section, if we don't have a valid code, no point in running them
      if ($oinkcode !~ m/none/ && $currentWebSnortVersion gt $local_vrt_v && $currentWebETVersion gt $currentLocalETVersion)
      { #updates both snort registered and EmergingThreats
        print("currentWebSnortVersion = $currentWebSnortVersion\nlocal_vrt_v = $local_vrt_v\ncurrentWebETVersion = $currentWebETVersion\ncurrentLocalETVersion = $currentLocalETVersion\n");
        $url = "https://www.snort.org/rules/snortrules-snapshot-$currentWebSnortVersion.tar.gz?oinkcode=$oinkcode\n http://rules.emergingthreats.net/open/$ETSnortVersion/emerging.rules.tar.gz";
      }
      elsif ($oinkcode !~ m/none/ && $currentWebSnortVersion gt $local_vrt_v && $currentWebETVersion le $currentLocalETVersion)
      { #updates only snort registered
        print("currentWebSnortVersion = $currentWebSnortVersion\nlocal_vrt_v = $local_vrt_v\ncurrentWebETVersion = $currentWebETVersion\ncurrentLocalETVersion = $currentLocalETVersion\n");
        $url = "https://www.snort.org/rules/snortrules-snapshot-$currentWebSnortVersion.tar.gz?oinkcode=$oinkcode";
      }
      elsif ($currentWebETVersion gt $currentLocalETVersion)
      { #updates only emergingthreats
          $url = "http://rules.emergingthreats.net/open/$ETSnortVersion/emerging.rules.tar.gz";
      }
      else
      { #no updates are performed as there were no newer versions available
        $performUpdate = 0;
      }
      if ($performUpdate == 1 && runUpdate() == 1)
      { #only runs if there are updates available and the updater itself ran without error
          writeFile(); #write the latest version to local files so we now know we have the latest
          #performing these system tasks from here, as we don't want to restart the snort service if we don't have to
          system("logger -t SnortUpdate 'SnortUpdate finished.'");
          system("logger -t SnortUpdate 'Changing ownership for rule files...'");
          system("chown -R nobody:nobody /etc/snort/rules");
          system("logger -t SnortUpdate 'Ownership permissions updated.'");
          system("logger -t SnortUpdate 'Restarting Snort...'");
          system("/usr/local/bin/snortctrl restart");
      }
      else
      {
        system("logger -t SnortUpdate 'No updates found.'");
        print "No updates found.\n";
      }
}

sub writeFile
{
  open my $fh2, '>', "/var/tmp/ETVersion.txt"; #open file for writing
  print $fh2 "$currentWebETVersion"; #actually write to file
  close $fh2; #close file
}

sub runUpdate
{
  my @df = `/bin/df /var`; #runs system command to get HD size and free disk space
  foreach my $line (@df) #multiple lines are returned, need to loop through them
  {
        my $errormessage = '';
        next if $line =~ m/^Filesystem/; #skips lines that have the word "Filesystem", meaning it's a header line
        my $return;

        if ($line =~ m/dev/) #only cares about lines that have the word 'dev' in them
        {
              my @temp = split(/\s+/,$line); #split the line at the spaces
              if ($temp[3]<300000) #Third value is free space, need 300MB
              {
                  $errormessage = "Not enough disk space, less then 300MB is available";
                  system("logger -t SnortUpdate 'Error encountered while update database: $errormessage'");
                  return 2;
              }
              else
              {
                  my @lns=split('\n',$url); #splitting at a new line and setting the URL to a variable
                  foreach my $l (@lns) #if more than 1 then we just loop
                  {
                      chomp $l;
                      print "Update found!\n";
                      print "Downloading... $l \n";
                      sleep(1);
                      system("logger -t SnortUpdate 'Download start for: $l'");
                      system("wget -r -o /var/tmp/log --output-document=/var/tmp/snortrules.tar.gz $l");
                      sleep(3);
                      $return = `cat /var/tmp/log 2>/dev/null`;

                      if ($return =~ "ERROR")
                      {
                          $errormessage = $return;
                          system("logger -t SnortUpdate 'Error prevented the download: $errormessage'");
                          print $errormessage;
                          return 2;
                      }
                      else
                      {
                          system("logger -t SnortUpdate 'Processing rules downloaded...'");
                          print("Processing downloaded rules...\n");
                          system("/usr/local/bin/oinkmaster.pl -v -s -u file:///var/tmp/snortrules.tar.gz -C /var/ipfire/snort/oinkmaster.conf -o /etc/snort/rules >/var/tmp/log 2>&1");
                          sleep(2);
                          system("logger -t SnortUpdate 'Update Successfull for $l'");
                          print "Update Successfull for $l\n";
                      }
                  }
              }
          }
      }
      return 1;
}

sub verifySnortIsRunning
{
    system("logger -t SnortUpdate 'Verifying Snort is running...'");

    if (($Snort_Orange == 1 && ! -e "/var/run/snort_orange0.pid") |
    ($Snort_BLUE == 1 && ! -e "/var/run/snort_blue0.pid") |
    ($Snort_GREEN == 1 && ! -e "/var/run/snort_green0.pid") |
    ($Snort_Active == 1 && (! -e "/var/run/snort_red0.pid" && ! -e "/var/run/snort_ppp0.pid")))
    { #if any of the services are enabled and are not started, go ahead and start them 'cause peeps be creepin'
        system("/usr/local/bin/snortctrl restart"); #start command... it knows what to do
        system("logger -t SnortUpdate 'Not all services running, starting now...'");
        print "Not all services running, starting now...\n";
    }
}

ummeegge
Community Developer
Community Developer
Posts: 4998
Joined: October 9th, 2010, 10:00 am

Re: Snort Rules Update

Post by ummeegge » November 7th, 2018, 2:16 pm

Hi all,
just to let you know, Suricata is currently in development process --> https://git.ipfire.org/?p=people/stevee ... t-suricata as a substitute for Snort and it seems like there is already a "Automatic rule updater" within
Image
which will be possibly released in Januar 2019 --> https://wiki.ipfire.org/devel/telco/2018-11-05 . Testing images can be found in here --> https://people.ipfire.org/~stevee/suricata/ . May feedback and or contributions might be nice for the development ?

Greetings,

UE
Image
Image

User avatar
H&M
Posts: 471
Joined: May 29th, 2014, 9:38 pm
Location: Europe

Re: Snort Rules Update

Post by H&M » November 7th, 2018, 6:55 pm

Hello,

There is an other post for the IDS Rules updates: viewtopic.php?p=120102#p120100

There is also an wiki post with a link toward above: https://wiki.ipfire.org/optimization/st ... _hardening

Hope it helps.

Late edit: I invested a lot in tunning snort (custom threshold.conf). Suricata is not a valid option unless the tunning gets ported to suricate: all specific SID/false positives tunned for my specific usage.

dnl
Posts: 375
Joined: June 28th, 2013, 11:03 am

Re: Snort Rules Update

Post by dnl » November 8th, 2018, 9:52 am

Thanks for that ummegge! I'm looking forward to it.

Although do note that "daily" updates aren't as good as "check hourly for an update" because you might get your rules 23 hours late!
H&M wrote:
November 7th, 2018, 6:55 pm
Late edit: I invested a lot in tunning snort (custom threshold.conf). Suricata is not a valid option unless the tunning gets ported to suricate: all specific SID/false positives tunned for my specific usage.
Sorry to hear that!
I feel I can tune mine better and am looking forward to using Suricata.
IPFire 2.x (Latest Update) on x86_64 Intel Bay Trail CPU, 4GiB RAM, RED + GREEN + BLUE + ORANGE

dnl
Posts: 375
Joined: June 28th, 2013, 11:03 am

Re: Snort Rules Update

Post by dnl » November 18th, 2018, 5:54 am

gitarman94 wrote:
November 7th, 2018, 11:21 am
??? So, I'm not quite sure how to fix this issue. I changed the code to correct for the issue that was discovered, however, everything is now working as expected except for the fact that the oinkmaster.pl file doesn't appear to be incrementing the version of Snort. Below is the results of running the updated script, it's discovering the version difference in Snort (29111 vs 29120), it downloads the newer correct version, and then runs it. The second command below is a manual run of the install script and it's normal looking output and supposed success. In the end, no matter what I do, the version number on Snort doesn't update to 29120 (29.12.0). Not sure if this is a bug on their side, or if it's something I missed. Going to attach the new code here and hope that someone else can do a verification that I didn't miss something.
Sorry I missed this when I replied earlier.
I'll try your update out soon.
IPFire 2.x (Latest Update) on x86_64 Intel Bay Trail CPU, 4GiB RAM, RED + GREEN + BLUE + ORANGE

dnl
Posts: 375
Joined: June 28th, 2013, 11:03 am

Re: Snort Rules Update

Post by dnl » December 2nd, 2018, 7:31 am

gitarman94 wrote:
November 7th, 2018, 11:21 am
??? So, I'm not quite sure how to fix this issue. I changed the code to correct for the issue that was discovered, however, everything is now working as expected except for the fact that the oinkmaster.pl file doesn't appear to be incrementing the version of Snort. Below is the results of running the updated script, it's discovering the version difference in Snort (29111 vs 29120), it downloads the newer correct version, and then runs it. The second command below is a manual run of the install script and it's normal looking output and supposed success. In the end, no matter what I do, the version number on Snort doesn't update to 29120 (29.12.0). Not sure if this is a bug on their side, or if it's something I missed. Going to attach the new code here and hope that someone else can do a verification that I didn't miss something.
Unfortunately, I'm still having the same problem.

Code: Select all

[root@ipfire /tmp]#  ./snortupdate.pl.new 
Use of uninitialized value $vnumer in substitution (s///) at ./snortupdate.pl.new line 146.
Use of uninitialized value $vnumer in numeric gt (>) at ./snortupdate.pl.new line 148.
currentWebSnortVersion = 29120
local_vrt_v = 29111
currentWebETVersion = 9068
currentLocalETVersion = 9068
Update found!
Downloading... https://www.snort.org/rules/snortrules-snapshot-29120.tar.gz?oinkcode=example
Processing downloaded rules...
Update Successfull for https://www.snort.org/rules/snortrules-snapshot-29120.tar.gz?oinkcode=example
Stopping Intrusion Detection System on ppp0...                                                                       [  OK  ]
Starting Intrusion Detection System on ppp0...
Spawning daemon child...
My daemon child 14617 lives...
Daemon parent exiting (0)
This repeats every time the script is executed.
IPFire 2.x (Latest Update) on x86_64 Intel Bay Trail CPU, 4GiB RAM, RED + GREEN + BLUE + ORANGE

Post Reply