ntopng for IPFire

Help on building IPFire & Feature Requests
User avatar
gocart
Posts: 539
Joined: December 16th, 2013, 4:43 pm
Location: Germany

Re: ntopng for IPFire

Post by gocart » October 5th, 2017, 4:52 pm

Hi @all!

I have ntopng 3.4 release online. Build with libsodium 1.0.15, zeromq 4.2.2 geoip 1.6.12. All needed libs are included in the download package. This package needs only redis as an dependency. Please install it first. Packet capture and web interface is only bind to the green0 interface. This can be edit in the /etc/ntopng/ntopng.conf. Restart the service after config file changes. The geoip databases will be download while installation.

The web interface can be accessed via https://[GREENIP]:3001 or without TLS on port 3000. First must change the default password for admin. Do not use special chars for password (*,#,! etc... ). No login possible with this passwords. (possible bug or weakness on integrated web server)

Packages:
Redis 4.0.11 x86 and x64
NtopNG 3.4 x86 and x64

Please give feedback in this thread!

Ps. If a admin password reset needed, use the redis-cli command (5 is the internal redis db number i using for ntopng):

Code: Select all

redis-cli
select 5
del user.admin.password
exit
And login with admin and admin.

Greetings, gocart
Last edited by gocart on October 4th, 2018, 8:52 pm, edited 9 times in total.

ummeegge
Community Developer
Community Developer
Posts: 4736
Joined: October 9th, 2010, 10:00 am

Re: ntopng for IPFire

Post by ummeegge » October 5th, 2017, 5:40 pm

Hi gocart,
the password reset is here no problem via ntopng "Manage Users" menu. But if i uninstall all and install it again the old PWD is still known (rdb has been deleted) need to take a deeper look for this.
Also, did you have the same behavior:

Code: Select all

1125:M 05 Oct 19:14:27.998 # Warning: 32 bit instance detected but no memory limit set. Setting 3 GB maxmemory limit with 'noeviction' policy now.

1125:M 05 Oct 19:14:28.002 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1125:M 05 Oct 19:14:28.002 # Server initialized
1125:M 05 Oct 19:14:28.003 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1125:M 05 Oct 19:14:28.003 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
? If so how/or_did you manage this ?

You can define also a DB name via e.g. '--dbfilename ntopng.rdb' .

How much RAM do swallows both on you system ?

I will take more time with this package.

Greetings,

UE
Image
Image
Image

User avatar
gocart
Posts: 539
Joined: December 16th, 2013, 4:43 pm
Location: Germany

Re: ntopng for IPFire

Post by gocart » October 5th, 2017, 6:27 pm

Hi UE,

this messages comes from redis on 32bit i think. My redis package install.sh set some sysctl.conf settigs:

Code: Select all

# disable hugepages
echo never > /sys/kernel/mm/transparent_hugepage/enabled

SYSCTL="/etc/sysctl.conf";
# add sysctl config
STRING=$(grep -A 1 'net.core.somaxconn' ${SYSCTL});
if [[ "${STRING}" ]]; then
  echo "net.core.somaxconn is present";
else
  echo "Add net.core.somaxconn = 1024 to ${SYSCTL}";
  echo "" >> ${SYSCTL};
  echo "net.core.somaxconn = 1024" >> ${SYSCTL};
fi;
STRING=$(grep -A 1 'vm.overcommit_memory' ${SYSCTL});
if [[ "${STRING}" ]]; then
  echo "vm.overcommit_memory is present";
else
  echo "Add vm.overcommit_memory = 1 to ${SYSCTL}";
  echo "vm.overcommit_memory = 1" >>  ${SYSCTL};
fi;
STRING=$(grep -A 1 'vm.nr_hugepages' ${SYSCTL});
if [[ "${STRING}" ]]; then
  echo "vm.nr_hugepages is present";
else
  echo "Add vm.nr_hugepages = 0 to ${SYSCTL}";
  echo "vm.nr_hugepages = 0" >> ${SYSCTL};
fi;
A reboot is needed. I work primary on x64 since end of 2016...
the password reset is here no problem via ntopng "Manage Users" menu
not, if you can't login as admin... My systems have between 2 and 4 GB Ram (all productive are x64, only test vm's are x86).
You can define also a DB name via e.g. '--dbfilename ntopng.rdb' .
I mean the internal redis db. You can separate applications using redis into different internal redis databases. The application can see only your data.

Code: Select all

# Redis connection. <fmt> is [h[:port[:pwd]]][@db-id] | db-id is the id
--redis=127.0.0.1:6379@5
The db-id...
Greetings, gocart

ummeegge
Community Developer
Community Developer
Posts: 4736
Joined: October 9th, 2010, 10:00 am

Re: ntopng for IPFire

Post by ummeegge » October 5th, 2017, 7:20 pm

gocart wrote:
October 5th, 2017, 6:27 pm
this messages comes from redis on 32bit i think.
this is only the first message but the others

Code: Select all

           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 4.0.2 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 6771
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

6771:M 05 Oct 18:42:07.328 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
6771:M 05 Oct 18:42:07.328 # Server initialized
6771:M 05 Oct 18:42:07.328 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
6771:M 05 Oct 18:42:07.328 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.

appears also for 64 bit. Have find also those fixes but i need to check a little deeper for side affects with the sysctl modifications.
gocart wrote:
October 5th, 2017, 6:27 pm
not, if you can't login as admin...
This is strange, have never had that issue here. But a strange one here is as mentioned, the WI password survived the redis and ntopng uninstallation. Haven´t find until now the reason why.

Also after every installation i have the following entries in messages:

Code: Select all

Oct  5 17:59:50 ipfire-prime ntopng: [NetworkInterface.cpp:394] ERROR: Unknown aggregation value for interface pcap [rsp: probe_ip]
Oct  5 17:59:50 ipfire-prime ntopng: [NetworkInterface.cpp:394] ERROR: Unknown aggregation value for interface pcap [rsp: probe_ip]
Oct  5 17:59:51 ipfire-prime ntopng: [NetworkInterface.cpp:394] ERROR: Unknown aggregation value for interface pcap [rsp: probe_ip]
Oct  5 17:59:53 ipfire-prime ntopng: [NetworkInterface.cpp:394] ERROR: Unknown aggregation value for interface pcap [rsp: probe_ip]
Oct  5 17:59:56 ipfire-prime ntopng: [NetworkInterface.cpp:394] ERROR: Unknown aggregation value for interface pcap [rsp: probe_ip]
Oct  5 18:42:53 ipfire-prime ntopng: [NetworkInterface.cpp:394] ERROR: Unknown aggregation value for interface pcap [rsp: probe_ip]
Oct  5 18:42:53 ipfire-prime ntopng: [NetworkInterface.cpp:394] ERROR: Unknown aggregation value for interface pcap [rsp: probe_ip]
Oct  5 18:42:54 ipfire-prime ntopng: [NetworkInterface.cpp:394] ERROR: Unknown aggregation value for interface pcap [rsp: probe_ip]
Oct  5 18:42:56 ipfire-prime ntopng: [NetworkInterface.cpp:394] ERROR: Unknown aggregation value for interface pcap [rsp: probe_ip]
Oct  5 18:42:59 ipfire-prime ntopng: [NetworkInterface.cpp:394] ERROR: Unknown aggregation value for interface pcap [rsp: probe_ip]
but after installation no entries at all and all goes fine.

Have stressed Redis with a

Code: Select all

redis-cli --lru-test 10000000
today a little to check the RAM consumption with that results:

Code: Select all

 Private  +   Shared  =  RAM used	Program

 36.8 MiB +   1.9 MiB =  38.7 MiB	squid (2)
140.4 MiB +  25.1 MiB = 165.5 MiB	ntopng
221.3 MiB + 764.0 KiB = 222.1 MiB	redis-server
291.2 MiB +   2.4 MiB = 293.7 MiB	snort

Some besides from here. Greetings,

UE
Image
Image
Image

User avatar
gocart
Posts: 539
Joined: December 16th, 2013, 4:43 pm
Location: Germany

Re: ntopng for IPFire

Post by gocart » October 5th, 2017, 8:01 pm

@UE

i have change the password for admin with * and/or # or other special character in password and logout. After this can't login as admin. Only after del user.admin.password can i login again.

Code: Select all

ipfire-prime ntopng: [NetworkInterface.cpp:394] ERROR: Unknown aggregation value for interface pcap
I don't hve this error. "pcap" is not an interface... nDPI version Problem ???
Grüße, gocart

ummeegge
Community Developer
Community Developer
Posts: 4736
Joined: October 9th, 2010, 10:00 am

Re: ntopng for IPFire

Post by ummeegge » October 6th, 2017, 8:20 am

Hi gocart,
gocart wrote:
October 5th, 2017, 8:01 pm
i have change the password for admin with * and/or # or other special character in password and logout. After this can't login as admin.
have no problem with this here ??? , even i use "!"§$%&/()=?`*#_:;" as password and logout, i can login via admin account with the before added "!"§$%&/()=?`*#_:;" password, also change it back to something else is no problem. May this has been fixed in the new version since i use ntopng-v.3.1.171005 which differs to yours.
gocart wrote:
October 5th, 2017, 8:01 pm

Code: Select all

[NetworkInterface.cpp:394] ERROR: Unknown aggregation value for interface pcap
I don't hve this error. "pcap" is not an interface... nDPI version Problem ???
The code which generates this is here --> https://github.com/ntop/ntopng/blob/4b2 ... e.cpp#L379 located. So this problem causes this settings
Image
i do not have had the zeromq libs installed on that system. Have build them and installed them, proceeded as before (deleted everything installed it again) and the error messages has been disappeared.

Question: Did you have had another intend why you have build ZeroMQ ?

Another thing: have limited the maxmemory in redis to 150 MB which seems to be no problem at all

Code: Select all

-> redis-cli config get maxmemory
1) "maxmemory"
2) "157286400"
which deliver:

Code: Select all

 46.5 MiB +   1.3 MiB =  47.8 MiB	redis-server
 67.1 MiB + 816.0 KiB =  67.9 MiB	redis-cli
while ...
15000 Gets/sec | Hits: 4022 (26.81%) | Misses: 10978 (73.19%)
14750 Gets/sec | Hits: 3930 (26.64%) | Misses: 10820 (73.36%)
15000 Gets/sec | Hits: 4153 (27.69%) | Misses: 10847 (72.31%)
and the peak was:

Code: Select all

217.0 MiB +   1.3 MiB = 218.3 MiB	redis-server (2)
232.6 MiB + 816.0 KiB = 233.4 MiB	redis-cli
while ...
15250 Gets/sec | Hits: 9023 (59.17%) | Misses: 6227 (40.83%)
15250 Gets/sec | Hits: 8976 (58.86%) | Misses: 6274 (41.14%)
14750 Gets/sec | Hits: 8742 (59.27%) | Misses: 6008 (40.73%)
15250 Gets/sec | Hits: 9202 (60.34%) | Misses: 6048 (39.66%)
after that the RAM consumption for redis-server falls back to

Code: Select all

144.8 MiB +   1.3 MiB = 146.1 MiB	redis-server
326.7 MiB + 816.0 KiB = 327.5 MiB	redis-cli
while ...
11750 Gets/sec | Hits: 8122 (69.12%) | Misses: 3628 (30.88%)
12000 Gets/sec | Hits: 8399 (69.99%) | Misses: 3601 (30.01%)
12000 Gets/sec | Hits: 8389 (69.91%) | Misses: 3611 (30.09%)
gocart wrote:
October 5th, 2017, 6:27 pm
I mean the internal redis db. You can separate applications using redis into different internal redis databases. The application can see only your data.

Code: Select all

# Redis connection. <fmt> is [h[:port[:pwd]]][@db-id] | db-id is the id
--redis=127.0.0.1:6379@5
The db-id...
Much better will do that too.

Let´s see what happens else.

Greetings,

UE
Image
Image
Image

User avatar
gocart
Posts: 539
Joined: December 16th, 2013, 4:43 pm
Location: Germany

Re: ntopng for IPFire

Post by gocart » October 6th, 2017, 11:01 am

Hi UE,
cant find a ntopng-v.3.1.171005 on the net. I have "ntopng Community v.3.0.171005".
Did you have had another intend why you have build ZeroMQ ?
I use zeromq for amavisd (Mail Gateway).
have limited the maxmemory in redis to 150
My redis package is limited to 64MB. redis.conf:

Code: Select all

# LIMITS
maxmemory 64mb
maxmemory-policy noeviction
and have no problems...
Gruße, gocart

ummeegge
Community Developer
Community Developer
Posts: 4736
Joined: October 9th, 2010, 10:00 am

Re: ntopng for IPFire

Post by ummeegge » October 6th, 2017, 11:28 am

gocart wrote:
October 6th, 2017, 11:01 am
cant find a ntopng-v.3.1.171005 on the net. I have "ntopng Community v.3.0.171005".
the 3.1.x is part of the current 'DEV' branch

Code: Select all

-> ntopng -V
v.3.1.171005	[Community build]
GIT rev:	dev:c58efebb7a69bf1dd018b17d9b46bd8e439b0e99:20171005

gocart wrote:
October 6th, 2017, 11:01 am
Did you have had another intend why you have build ZeroMQ ?
I use zeromq for amavisd (Mail Gateway).
Interesting. ZeroMQ are also used for nProbe that´s why i´am asking ;) .
gocart wrote:
October 6th, 2017, 11:01 am
have limited the maxmemory in redis to 150
My redis package is limited to 64MB. redis.conf:

Code: Select all

# LIMITS
maxmemory 64mb
maxmemory-policy noeviction
and have no problems...
Also interesting.

Greetings,

UE
Image
Image
Image

ummeegge
Community Developer
Community Developer
Posts: 4736
Joined: October 9th, 2010, 10:00 am

Re: ntopng for IPFire

Post by ummeegge » October 7th, 2017, 8:40 am

Hi all,
have made also my 2 cents on this here. Here --> https://forum.ipfire.org/viewtopic.php? ... 65#p111061 you can find the in- uninstaller for ntopng on IPFire.

Greetings,

UE
Image
Image
Image

User avatar
gocart
Posts: 539
Joined: December 16th, 2013, 4:43 pm
Location: Germany

Re: ntopng for IPFire

Post by gocart » October 7th, 2017, 8:14 pm

Hi UE,
i updated my package to 3.1.xxx last git clone checkout. My new binary ignores the "--packet-filter" option. Please can you test my configuration with your binary. In need to filter own ntopng web server traffic. Here is my line:

Code: Select all

--packet-filter="ip and not proto ipv6 and not ether host ff:ff:ff:ff:ff:ff and not net (224.0.0.0/8 or 239.0.0.0/8) and not (host greeip and port 3001)"
This works with 3.0 perfect. The bpf syntax can you see here: https://biot.com/capstats/bpf.html
Greetings, gocart

ummeegge
Community Developer
Community Developer
Posts: 4736
Joined: October 9th, 2010, 10:00 am

Re: ntopng for IPFire

Post by ummeegge » October 8th, 2017, 7:49 am

Moin gocart,
gocart wrote:
October 7th, 2017, 8:14 pm
My new binary ignores the "--packet-filter" option. Please can you test my configuration with your binary. In need to filter own ntopng web server traffic.
this do not work here either, also no log entries or something which points out an --packet-filter problem ::) . Tried also a filter like this

Code: Select all

--packet-filter="ip and not ip multicast and not ether broadcast and not (host 192.168.{greenIPFire-IP} and port 3001)"
whereby it seems that IPv6, Multi- and Broadcasts are filtered (fast test now) but the "redwood-broker" (port 3001) are still listed...

Grüsse,

UE
Image
Image
Image

ummeegge
Community Developer
Community Developer
Posts: 4736
Joined: October 9th, 2010, 10:00 am

Re: ntopng for IPFire

Post by ummeegge » October 28th, 2017, 2:26 pm

Hi all,
@gocart have communicated a little with the ntopng people whereby the '--packet-filter' option problem in the dev version seemed to be known and as stated in here --> https://github.com/ntop/ntopng/issues/1520 also solved (here currently only partly). Made that changes too but stucked at first with a segfault --> https://github.com/ntop/ntopng/issues/1522 which is meanwhile fixed. My currenty state is, 32bit system do works with the '--packet-filter ' option (but different syntax then the above one) but 64bit do not ::) ?? and both versions gets also a segfault -->

Code: Select all

Oct 24 13:14:05 ipfire-prime kernel: ntopng[27053]: segfault at 7f387909c9d0 ip 00007f387daf16ae sp 00007ffd1cec2500 error 4 in libpthread-2.25.so[7f387dae9000+19000]
Oct 24 13:14:05 ipfire-prime kernel: grsec: From 192.168.7.2: Segmentation fault occurred at 00007f387909c9d0 in /usr/bin/ntopng[ntopng:27053] uid/euid:1005/1005 gid/egid:1005/1005, parent /sbin/init[init:1] uid/euid:0/0 gid/egid:0/0
Oct 24 13:14:05 ipfire-prime kernel: grsec: From 192.168.7.2: bruteforce prevention initiated due to crash of /usr/bin/ntopng against uid 1005, banning suid/sgid execs for 15 minutes.  Please investigate the crash report for /usr/bin/ntopng[ntopng:27053] uid/euid:1005/1005 gid/egid:1005/1005, parent /sbin/init[init:1] uid/euid:0/0 gid/egid:0/0
Oct 24 13:14:10 ipfire-prime ntopng: [PcapInterface.cpp:259] ERROR: Unable to set on tun0 filter ip and not proto ipv6 and not ether host ff:ff:ff:ff:ff:ff and not net (224.0.0.0/8 or 239.0.0.0/8) and not (host 192.16.7.1 and port 3001). Filter ignored.
Oct 24 13:14:13 ipfire-prime ntopng: [PcapInterface.cpp:259] ERROR: Unable to set on tun1 filter ip and not proto ipv6 and not ether host ff:ff:ff:ff:ff:ff and not net (224.0.0.0/8 or 239.0.0.0/8) and not (host 192.168.7.1 and port 3001). Filter ignored.
( <-- PcapInterface logging works now ;) and grsec jumps also into but ntopng do not crashes and all works as expected, possibly there needs to be done some more work on this...
May i open up another issue in Github but since i can not reproduce the segfault in chroot to debug it via GDB am currently not 100% sure why that´s happen (config ?). Let´s see.

Some news from here :) .

Grüssle,

UE
Image
Image
Image

User avatar
gocart
Posts: 539
Joined: December 16th, 2013, 4:43 pm
Location: Germany

Re: ntopng for IPFire

Post by gocart » October 29th, 2017, 10:13 am

Hi UE,

good to hear that an thanks for your work. I will update my build and will test it :) I'll get in touch.

Schöne Sonntagsgrüße,
gocart

User avatar
gocart
Posts: 539
Joined: December 16th, 2013, 4:43 pm
Location: Germany

Re: ntopng for IPFire

Post by gocart » October 29th, 2017, 11:30 am

@UE
the packet filter option now works again... 8) and on my short test, i have no segfaults. I am on commit 45c1128.
gocart.

ummeegge
Community Developer
Community Developer
Posts: 4736
Joined: October 9th, 2010, 10:00 am

Re: ntopng for IPFire

Post by ummeegge » October 30th, 2017, 6:26 am

Hi gocart,
gocart wrote:
October 29th, 2017, 11:30 am
and on my short test, i have no segfaults
have checked this a little deeper and this segfault:

Code: Select all

Oct 30 06:44:18 ipfire-prime kernel: ntopng[20972]: segfault at 7f6b12cfb9d0 ip 00007f6b177506ae sp 00007fff92075080 error 4 in libpthread-2.25.so[7f6b17748000+19000]
Oct 30 06:44:18 ipfire-prime kernel: grsec: From 192.168.7.2: Segmentation fault occurred at 00007f6b12cfb9d0 in /usr/bin/ntopng[ntopng:20972] uid/euid:1005/1005 gid/egid:1005/1005, parent /sbin/init[init:1] uid/euid:0/0 gid/egid:0/0
Oct 30 06:44:18 ipfire-prime kernel: grsec: From 192.168.7.2: bruteforce prevention initiated due to crash of /usr/bin/ntopng against uid 1005, banning suid/sgid execs for 15 minutes.  Please investigate the crash report for /usr/bin/ntopng[ntopng:20972] uid/euid:1005/1005 gid/egid:1005/1005, parent /sbin/init[init:1] uid/euid:0/0 gid/egid:0/0
happens while stopping ntopng via initscript. Entries are in /var/log/messages located can you please double check this and if there are no segfault can you please post your initscript ?

The packet-filter option works here too on both machines, have had a syntax error on the 32bit machine.

Thanks und Grüsse,

UE
Image
Image
Image

Post Reply