ntopng for IPFire

Help on building IPFire & Feature Requests
Post Reply
ummeegge
Community Developer
Community Developer
Posts: 4676
Joined: October 9th, 2010, 10:00 am

ntopng for IPFire

Post by ummeegge » October 1st, 2017, 10:19 am

Hi all,
since i have (mostly) integrated the community edition of ntopng --> http://www.ntop.org/products/traffic-analysis/ntop/ which currently includes the following packages
- https://github.com/ntop/ntopng
- https://github.com/ntop/nDPI
- https://redis.io/
- https://github.com/json-c/json-c
- http://zeromq.org/
- https://github.com/maxmind/geoip-api-c # Has been dropped with v.3.7 and substituted with libmaxminddb.
- https://github.com/maxmind/libmaxminddb
- https://dev.mysql.com/downloads/mysql/ # Perl modules only. No functionality only dependency

into IPFire, i wanted to provide it also to the community for extensions/feedback/testings if someone is interested, an In- Uninstaller for 32 bit and 64 bit systems are available (currently no Arm) in here --> https://gitlab.com/ummeegge/ntopng-ipfi ... /README.md some explanations can be found what the installer does and what not. You can execute the installer via:

USE TESTING SYSTEMS FOR THIS
Keep also in mind that ntopng can uses a lot of RAM, do not use machines with little free RAM for this installation.

Code: Select all

cd /tmp &&
curl -O https://gitlab.com/ummeegge/ntopng-ipfire/raw/master/scripts/ntopng-installer.sh &&
chmod +x ntopng-installer.sh
./ntopng-installer.sh
All building files are located in here --> https://gitlab.com/ummeegge/ntopng-ipfire . All binaries in here --> http://people.ipfire.org/~ummeegge/ntopng/ .

Since gocart has also build packages for ntopng (the stable not the dev version) on IPFire --> https://forum.ipfire.org/viewtopic.php? ... 15#p111142 (thanks gocart;) you have now the choice ;-)

As ever, testings, feedback and further development might be nice.

Greetings,

UE

EDIT/Fixes with the help of the Ntopng team (thanks by the way to those guys ;-) for the current development version 3.*_dev:
- https://github.com/ntop/ntopng/issues/1522
- https://github.com/ntop/ntopng/issues/1520
- https://github.com/ntop/ntopng/issues/1545
- https://github.com/ntop/ntopng/issues/1544
- https://github.com/ntop/ntopng/issues/1833
- https://github.com/ntop/ntopng/issues/1935

- Uploaded version v.3.1.171121 [Community build]
- Added update function to installer.
31.01.2018
- Updated version to ntopng-3.3.180128, nDPI-2.2.180128 <-- both are DEV versions, redis-4.0.6, json-c-0.13-20171207, zeromq-4.2.3 .
An updater has been integrated so ntopng´s history, the configuration file, protos.txt and the redis.conf should be save while an update.
24.06.2018
- Update to v.3.3.180624, nDPI-v.2.2.180624 <-- Dev versions, redis-4.0.8, json-c-0.13.1, zeromq-4.2.3 and the MySQL perl modules has been added since IPFire do not provide MySQL anymore but Ntopng uses it as dependency. Since the MySQL functionalities in the community version are not that useful/comprehensive --> https://www.ntop.org/products/traffic-analysis/ntop/ i decided to keep the installation more lightweight (no MySQL functionality) but have had to install the perl modules to bring Ntopng to life.
27.06.2018
- Update to v.3.5.180626 fixed problem https://github.com/ntop/ntopng/issues/1833 .
20.08.2018
- Transfered repo to Gitlab
23.08.2018
- Update to ntopng version v.3.7.180823 [Community build] . For further informations take a look in here --> https://forum.ipfire.org/viewtopic.php? ... 63#p118363 .
29.08.2018
- Updated version v.3.7.180828 [Community build] . For further informations/fixes --> https://github.com/ntop/ntopng/issues/1935 take a look in here --> https://forum.ipfire.org/viewtopic.php? ... 05#p118454 .
Image
Image
Image

User avatar
gocart
Posts: 518
Joined: December 16th, 2013, 4:43 pm
Location: Germany

Re: ntopng for IPFire

Post by gocart » October 1st, 2017, 3:25 pm

Hi UE...

if have been test to build ntopng for IPFire. The build is not an problem. My project with ntopng fails, because it needs LuaJit for work. LuaJit is not working with an grsec kernel! No chance... :(. You can have my lfs and rootfiles if you want. The ipfire kernel kills the ntopng main process short after start.

https://www.freelists.org/post/luajit/l ... c-kernel,1

ps. rspamd have the same problem with laujit..

Greetings, gocart
Attachments
ntopng.zip
(8.05 KiB) Downloaded 70 times

ummeegge
Community Developer
Community Developer
Posts: 4676
Joined: October 9th, 2010, 10:00 am

Re: ntopng for IPFire

Post by ummeegge » October 1st, 2017, 3:48 pm

Hallo gocart,
gocart wrote:
October 1st, 2017, 3:25 pm
LuaJit is not working with an grsec kernel! No chance... :(. You can have my lfs and rootfiles if you want. The ipfire kernel kills the ntopng main process short after start.
that´s true have had the same issue here but you can disable grsec for single binaries via paxctl, this one

Code: Select all

paxctl -c /usr/bin/ntopng
paxctl -pemrxs /usr/bin/ntopng
solved that issue and ntopng works good then.

Thanks for your offer for the LFS files, have had there also some issues which needed other ways --> https://github.com/ummeegge/ntopng-ipfire but am currently at the beginning with that.

Grüssle,

UE
Image
Image
Image

User avatar
gocart
Posts: 518
Joined: December 16th, 2013, 4:43 pm
Location: Germany

Re: ntopng for IPFire

Post by gocart » October 1st, 2017, 4:39 pm

Thanks UE,

your paxctrl hint works... :) :) :)
ps. if you need the nDPI 2.1.0, i have extracted this for me from git. The 1.6 is very old. I can upload my tar.xz package if you want or you can download the 2.0 here: https://github.com/ntop/nDPI/archive/2.0.tar.gz

gocart

ummeegge
Community Developer
Community Developer
Posts: 4676
Joined: October 9th, 2010, 10:00 am

Re: ntopng for IPFire

Post by ummeegge » October 1st, 2017, 5:40 pm

gocart wrote:
October 1st, 2017, 4:39 pm
your paxctrl hint works... :) :) :)
good to hear.
gocart wrote:
October 1st, 2017, 4:39 pm
ps. if you need the nDPI 2.1.0, i have this extracted for me from git. The 1.6 is very old. I can upload my tar.xz package if you want.
Also good that you said that, i somehow missed to name the nDPI package correctly in the LFS, have used nDPI (and ntopng too) via git clone, both versions are 3 days old and so i need to call nDPI more correctly 2.1.0 , will do that with the next build.

Have wrote an GeoIP updater, made also an Google account to get the API browser key to bring the "Geo Map" to live but somehow it currently do not work. The "Countries" section works with the *.dat´s but the Map needs some more... Did you give them also a try ?

UE
Image
Image
Image

User avatar
gocart
Posts: 518
Joined: December 16th, 2013, 4:43 pm
Location: Germany

Re: ntopng for IPFire

Post by gocart » October 1st, 2017, 6:08 pm

2.1.0 is the dev branch. You can download the offical 2.0 here: https://github.com/ntop/nDPI/archive/2.0.tar.gz. For the geoip test I have not yet come. My build runs since 1 hour... ;)
Ps. you dont need:

Code: Select all

cd $(DIR_SRC)/nDPI && make install
The ntopng "cd $(DIR_APP) && make install" installs all you need... This is a strange package.
Auch Grüssle.. :)

ummeegge
Community Developer
Community Developer
Posts: 4676
Joined: October 9th, 2010, 10:00 am

Re: ntopng for IPFire

Post by ummeegge » October 1st, 2017, 7:11 pm

Yes indeed both packages reminds me a little on the disharmonic orchestra :D , have had problems with the regular version and the well known "cd ..; git clone https://github.com/ntop/nDPI.git; cd nDPI; ./autogen.sh; make; cd ../ntopng" message, whereby i think the problem was the version number on the unpacked nDPI directory ntopng also could not find an already compiled version of nDPI. That´s why i landed in the DEV section where both versions works while a build with another but seems to be also stable at runtime.
A little gloom are the version differences all that enterprise stuff is very understandable from their perspective but a little sad from the OpenSource sighting.
Nevertheless, nice feature what i have seen until now.
gocart wrote:
October 1st, 2017, 6:08 pm
Ps. you dont need:

Code: Select all

cd $(DIR_SRC)/nDPI && make install
The ntopng "cd $(DIR_APP) && make install" installs all you need...
Yes, i think so too, the ndpi libs and ndpiReader should come nevertheless with an make install only for ntopng to the disk (make currently also a clean build to check this), moreover the manual deletion of /usr/lib/pkgconfig/libndpi.pc and the whole /usr/include/libndpi-2.1.0 dir to prevent the

Code: Select all

fatal error: ../lib/third_party/include/libcache.h: No such file or directory
if you do not make a clean build after every change, do not need to make manually anymore...

nProbe is currently no solution for me (License), have tried to build also the PF_RING which looks promising (linkable to tcpdump, snort, etc.) as a pcap turbo but in there the same then before, a little an strange package.

The historical thing might be interesting too, have seen the ELK-Stack --> https://github.com/QXIP/Qbana again ;) (external solution might also be better for large histories {DATA}), but e.g. MySQL support seems to be a professional version feature.

Let´s see what the new build and experience brings to light.

Grüssle und nen schönen Abend noch.

UE
Image
Image
Image

User avatar
gocart
Posts: 518
Joined: December 16th, 2013, 4:43 pm
Location: Germany

Re: ntopng for IPFire

Post by gocart » October 2nd, 2017, 9:50 am

Hi UE,
i have 2 things for you. install.sh: cert generation for https:

Code: Select all

# create https cert 
/usr/bin/openssl genrsa -out rsa.key 4096 > /dev/null 2>&1;
/bin/cat /etc/certparams | sed "s/HOSTNAME/`hostname -f`/" | /usr/bin/openssl req -new -key rsa.key -out rsa.csr /dev/null 2>&1;
/usr/bin/openssl x509 -req -days 1825 -sha256 -in rsa.csr -signkey rsa.key -out rsa.crt /dev/null 2>&1;
cat rsa.key rsa.crt > /usr/share/ntopng/httpdocs/ssl/ntopng-cert.pem
rm -f rsa.key rsa.crt rsa.csr;
and bind WebIF to green interface install.sh:

Code: Select all

GREENIP=$(ifconfig | gawk '
    /^[a-z]/ {interface = $1}
    interface == "green0" && match($0, /^.*inet addr:([.0-9]+)/, a) {
        print a[1]
        exit
    }
')
sed  -i "s|greenip|${GREENIP}|g" /etc/ntopng/ntopng.conf;
config template

Code: Select all

# HTTP port. Set to 0 to disable http server. Default port: 3000.
--http-port=greenip:3000

# See usage of --http-port above. Default: 3001
--https-port=greenip:3001
WebIf on 127.0.0.1 is useless.
if you do not make a clean build after every change, do not need to make manually anymore...
If you not do make install nDPI, all problems are gone. I found this info in the docs...

Grüße, gocart

ummeegge
Community Developer
Community Developer
Posts: 4676
Joined: October 9th, 2010, 10:00 am

Re: ntopng for IPFire

Post by ummeegge » October 2nd, 2017, 7:04 pm

Hallo gocart,
gocart wrote:
October 2nd, 2017, 9:50 am
i have 2 things for you. install.sh:
will integrate both, have thinking about to place that stuff into a potential installer but it is more practiable in install.sh. Have made there also some detection for interfaces and subnets which will also be automatically set into ntopng.conf.
An GeoIP updater might goes to a potential installer, maybe better to decide this individually, let´s see...
gocart wrote:
October 2nd, 2017, 9:50 am
If you not do make install nDPI, all problems are gone. I found this info in the docs...
yes have checked that out an can confirm it, even the ndpiReader (logically also the header and the *.so´s) needs to be cleaned out of the ROOTFILES and aren´t available under /usr/bin/. I thought before that for example e.g. the protos.txt --> https://github.com/ntop/nDPI/blob/dev/e ... protos.txt would be correlated via the ndpiReader but all will be done via ntopng . Have tested and extend also an own protos.txt which works good until now and gives also some flexibility at the hand.

Have compiled ntopng also with "make geoip" and will give it a try with this to bring the Geo Map to live even i think ntopng only donwloads them too, unpacks it and places them into the appropriate directory if they are not findable in ntopng specifics --> https://github.com/ntop/ntopng/blob/dev ... le.in#L186 . Have integrated the GeoIP *.dats already and the "Countries" section works without problems but the Geo Map do not.

With more time more testings :) .

Grüsse,

UE
Image
Image
Image

User avatar
gocart
Posts: 518
Joined: December 16th, 2013, 4:43 pm
Location: Germany

Re: ntopng for IPFire

Post by gocart » October 3rd, 2017, 12:29 pm

HI UE,
next... the GEOIP updater (chron script):

Code: Select all

#!/bin/bash
cd /tmp/ntopng
wget -q http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
wget -q http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz
gunzip GeoIPASNum.dat.gz GeoLiteCity.dat.gz
mv -f GeoIPASNum.dat GeoLiteCity.dat /usr/share/ntopng/httpdocs/geoip/
rm -rdf /tmp/ntopng 
service ntopng restart
and an nice extra script for ipfire, a service mapper (from debian) /usr/sbin/service:

Code: Select all

#!/bin/sh
###############################################################################
#                                                                             #
# IPFire.org - A linux based firewall                                         #
# Copyright (C) 2007-2016  IPFire Team  <info@ipfire.org>                     #
#                                                                             #
# This program is free software: you can redistribute it and/or modify        #
# it under the terms of the GNU General Public License as published by        #
# the Free Software Foundation, either version 3 of the License, or           #
# (at your option) any later version.                                         #
#                                                                             #
# This program is distributed in the hope that it will be useful,             #
# but WITHOUT ANY WARRANTY; without even the implied warranty of              #
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the               #
# GNU General Public License for more details.                                #
#                                                                             #
# You should have received a copy of the GNU General Public License           #
# along with this program.  If not, see <http://www.gnu.org/licenses/>.       #
#                                                                             #
###############################################################################
# /usr/sbin/service

is_ignored_file() {
	case "$1" in
		skeleton | README | *.dpkg-dist | *.dpkg-old | rc | rcS | single | reboot | bootclean.sh)
			return 0
		;;
	esac
	return 1
}

VERSION="`basename $0` ver. ipfire"
USAGE="Usage: `basename $0` < option > | --status-all | \
[ service_name [ command | --full-restart ] ]"
SERVICE=
ACTION=
SERVICEDIR="/etc/init.d"
OPTIONS=

if [ $# -eq 0 ]; then
   echo "${USAGE}" >&2
   exit 1
fi

cd /
while [ $# -gt 0 ]; do
  case "${1}" in
    --help | -h | --h* )
       echo "${USAGE}" >&2
       exit 0
       ;;
    --version | -V )
       echo "${VERSION}" >&2
       exit 0
       ;;
    *)
       if [ -z "${SERVICE}" -a $# -eq 1 -a "${1}" = "--status-all" ]; then
          cd ${SERVICEDIR}
          for SERVICE in * ; do
            case "${SERVICE}" in
              functions | halt | killall | single| linuxconf| kudzu)
                  ;;
              *)
                if ! is_ignored_file "${SERVICE}" \
		    && [ -x "${SERVICEDIR}/${SERVICE}" ]; then
                        if ! grep -qs "\(^\|\W\)status)" "$SERVICE"; then
                          #printf " %s %-60s %s\n" "[?]" "$SERVICE:" "unknown" 1>&2
                          echo " [ ? ]  $SERVICE" 1>&2
                          continue
                        else
                          out=$(env -i LANG="$LANG" PATH="$PATH" TERM="$TERM" "$SERVICEDIR/$SERVICE" status 2>&1)
                          if [ "$?" = "0" -a -n "$out" ]; then
                            #printf " %s %-60s %s\n" "[+]" "$SERVICE:" "running"
                            echo " [ + ]  $SERVICE"
                            continue
                          else
                            #printf " %s %-60s %s\n" "[-]" "$SERVICE:" "NOT running"
                            echo " [ - ]  $SERVICE"
                            continue
                          fi
                        fi
                  #env -i LANG="$LANG" PATH="$PATH" TERM="$TERM" "$SERVICEDIR/$SERVICE" status
                fi
                ;;
            esac
          done
          exit 0
       elif [ $# -eq 2 -a "${2}" = "--full-restart" ]; then
          SERVICE="${1}"
          if [ -x "${SERVICEDIR}/${SERVICE}" ]; then
            env -i LANG="$LANG" PATH="$PATH" TERM="$TERM" "$SERVICEDIR/$SERVICE" stop
            env -i LANG="$LANG" PATH="$PATH" TERM="$TERM" "$SERVICEDIR/$SERVICE" start
            exit $?
          fi
       elif [ -z "${SERVICE}" ]; then
         SERVICE="${1}"
       elif [ -z "${ACTION}" ]; then
         ACTION="${1}"
       else
         OPTIONS="${OPTIONS} ${1}"
       fi
       shift
       ;;
   esac
done

# use the traditional sysvinit
if [ -x "${SERVICEDIR}/${SERVICE}" ]; then
   exec env -i LANG="$LANG" PATH="$PATH" TERM="$TERM" "$SERVICEDIR/$SERVICE" ${ACTION} ${OPTIONS}
else
   echo "${SERVICE}: unrecognized service" >&2
   exit 1
fi
now you can write "service xxxx restart" ;)
Grüße gocart

ummeegge
Community Developer
Community Developer
Posts: 4676
Joined: October 9th, 2010, 10:00 am

Re: ntopng for IPFire

Post by ummeegge » October 3rd, 2017, 2:15 pm

Moin gocart :) ,
gocart wrote:
October 3rd, 2017, 12:29 pm
and an nice extra script for ipfire, a service mapper (from debian) /usr/sbin/service:
nice one, cause i always get a finger convulsion after the second slash :D ... and here an auto complete of the initscripts dir after typing service.

Code: Select all

cat >> ~/.bashrc <<"EOF"

# /usr/sbin/service as /etc/init.d/ substitution get´s auto complete for /etc/init.d/ directory
complete -W "$(ls /etc/init.d/)" service

EOF
(logout and login needs to be done) :) ...
gocart wrote:
October 3rd, 2017, 12:29 pm
next... the GEOIP updater (chron script):
Have already this one --> https://github.com/ummeegge/ntopng-ipfi ... updater.sh
gocart wrote:
October 2nd, 2017, 9:50 am
i have 2 things for you. install.sh: cert generation for https:
While the cert generation OpenSSL have thrown this error

Code: Select all

...
unknown option /dev/null
req [options] <infile >outfile
i have changed it a little to --> https://github.com/ummeegge/ntopng-ipfi ... ll.sh#L150 .
gocart wrote:
October 2nd, 2017, 9:50 am
and bind WebIF to green interface install.sh
This did not work here but i have changed it to this --> https://github.com/ummeegge/ntopng-ipfi ... ll.sh#L165 .

- Free symlinks are detected and set automatically .
- Disable mprotect for ntopng --> https://github.com/ummeegge/ntopng-ipfi ... all.sh#L99 .
- Investigate subnets and interfaces automatically and set it into ntopng.conf --> https://github.com/ummeegge/ntopng-ipfi ... ll.sh#L106 .

No extra work todo install.sh do start ntopng but redis needs to be started before (possible installer work, can also make a check for this in install.sh ), if redis is not started before ntopng delivers a warning --> https://github.com/ummeegge/ntopng-ipfi ... g.init#L23 <-- am not finished with this, will make some more checks into install.sh...

Have also compiled ntopng with 'make geoip' which delivers the GeoIP dat´s but i get there an error:

Code: Select all

Stopping Traffic Analyzer and Flow Collector...
Error Traversing Database for ipnum = 3234438033 - Perhaps database is corrupt?                                                                                                      [  OK  ]
also i think the dat´s do not needs to be added into the package, the GeoIP updater can do this if someone wants it. Even it delivers here currently only the countries and the flags to the IP addresses.

Schönen Tag der deutschen Einheit dir noch O0 , vielleicht ja bis später.

Grüsse,

UE
Image
Image
Image

Hellfire
Posts: 523
Joined: November 8th, 2015, 8:54 am

Re: ntopng for IPFire

Post by Hellfire » October 3rd, 2017, 3:53 pm

Hi guys, nice one - can't wait installing this on my IPFire!

So, GO GO Go ;D
Image

User avatar
gocart
Posts: 518
Joined: December 16th, 2013, 4:43 pm
Location: Germany

Re: ntopng for IPFire

Post by gocart » October 4th, 2017, 8:55 am

Hi UE,
thanks for your work on github! :)

I have a packet filter for ntopng to exclude own web interface and multicast traffic (ntopng.conf):

Code: Select all

# packet filter config
--packet-filter="ip and not proto ipv6 and not ether host ff:ff:ff:ff:ff:ff and not net (224.0.0.0/8 or 239.0.0.0/8) and not (host greenip and port 3001)"
Error Traversing Database for ipnum = 3234438033 - Perhaps database is corrupt?
Hmm... I build this package with libsodium 1.0.12, zeromq 3.2.5 geoip 1.6.9 and luajit 2.0.5 and i don't have this error... ??? The deps are separate lfs.

I think, it's not useful to default bind packet capture on all interfaces of IPFire, green0 blue0 is okay. Red0 or tun or orange gives no extra info for outgoing/incoming traffic analysis. Ok red0/ppp if you only host server services on IPFire. More tests needed... ;)

Grüße, gocart

ummeegge
Community Developer
Community Developer
Posts: 4676
Joined: October 9th, 2010, 10:00 am

Re: ntopng for IPFire

Post by ummeegge » October 4th, 2017, 10:07 am

Hi gocart,
gocart wrote:
October 4th, 2017, 8:55 am
thanks for your work on github! :)
your welcome, have pushed a new version to Github --> https://github.com/ummeegge/ntopng-ipfi ... dfea3096ae which is currently in building state for 32 and 64 bit.
gocart wrote:
October 4th, 2017, 8:55 am
I have a packet filter for ntopng to exclude own web interface and multicast traffic (ntopng.conf):

Code: Select all

# packet filter config
--packet-filter="ip and not proto ipv6 and not ether host ff:ff:ff:ff:ff:ff and not net (224.0.0.0/8 or 239.0.0.0/8) and not (host greenip and port 3001)"
Great :) . Wanted also to take a look to the lua scriptings, let´s see what´s possible in there but this will come at the end of the DEV phase.
gocart wrote:
October 4th, 2017, 8:55 am
Error Traversing Database for ipnum = 3234438033 - Perhaps database is corrupt?
Hmm... I build this package with libsodium 1.0.12, zeromq 3.2.5 geoip 1.6.9 and luajit 2.0.5 and i don't have this error... ??? The deps are separate lfs.
OK, may there is somewhere the dogs grave...
gocart wrote:
October 4th, 2017, 8:55 am
Red0 or tun or orange gives no extra info for outgoing/incoming traffic analysis
Your right but by choosing e.g. tun, you get a clean overview of what happens in particular in the tunnels and the infos won´t mixed with all the others togehter, i am very happy with the tun interface :) selection . With red|ppp it might be a little the same... but you are right most important infos comes out of the LAN´s infos, if someone do not want´s them it is fairly easy to delete them over the ntopng.conf.
gocart wrote:
October 4th, 2017, 8:55 am
Ok red0/ppp if you only host server services on IPFire. More tests needed... ;)
Yes indeed.

Greetings,

UE

EDIT: Have i said that i really do love 'service ntopng restart' :D .
Image
Image
Image

ummeegge
Community Developer
Community Developer
Posts: 4676
Joined: October 9th, 2010, 10:00 am

Re: ntopng for IPFire

Post by ummeegge » October 4th, 2017, 1:11 pm

Have added example protos.txt --> https://github.com/ummeegge/ntopng-ipfi ... dfea3096ae .

Grüssle,

UE

EDIT: Fixed proto.txt permissions, added init start to redis install.sh, Fixed typo --> https://github.com/ummeegge/ntopng-ipfi ... dfea3096ae
Image
Image
Image

Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests