Pmacct - network monitoring

Help on building IPFire & Feature Requests
ummeegge
Community Developer
Community Developer
Posts: 4122
Joined: October 9th, 2010, 10:00 am

Pmacct - network monitoring

Post by ummeegge » September 8th, 2015, 7:06 am

Hi all,
i wanted to introduce a tool named Pmacct --> http://www.pmacct.net/ . As Pmaccts hompage told, this tool gives (beneath more) possibilities for
"network management tasks like billing, graphing network resources usage, live or historical traffic trends analysis, steering BGP peerings, real-time alerting and certain SLA monitoring."
Have compiled it now and so there is an pmacct IPFire package available for testing environments. The package can be found in here http://people.ipfire.org/~ummeegge/pmacct/ .

Installation:

- Navigate to

Code: Select all

cd /opt/pakfire/tmp
- Download the needed package from here

Code: Select all

http://people.ipfire.org/~ummeegge/pmacct
Compare the SHA256 sum

Code: Select all

sha256sum pmacct-*.ipfire
If correct,
unpack it with a

Code: Select all

tar xvf pmacct-1.5.1-1.ipfire
and install it with a

Code: Select all

./install.sh
You can set the console free with a [CTRL-c].

Related files in this package are:
/usr/bin/pmacct
/usr/bin/pmmyplay
/usr/sbin/nfacctd
/usr/sbin/pmacctd
/usr/sbin/sfacctd
/usr/sbin/uacctd
/etc/pmacct/<-- config, examples and database presets

This package was compiled with the following options:

Code: Select all

-> pmacctd -V
Promiscuous Mode Accounting Daemon, pmacctd 1.5.1 (20150215-01)
 --prefix=/usr --sysconfdir=/etc/pmacct --enable-sqlite3 --enable-mysql --enable-ulog

For suggestions, critics, bugs, contact me: Paolo Lucente <paolo@pmacct.net>.
There is currently no Initscript in the package but this should be no problem and may comes with a later version.

So if someone is interessted in this topic, some feedback, testing results, or different setups might be interessting.

This topic started in here --> https://forum.ipfire.org/viewtopic.php?f=50&t=14699 with questions to IP-accounting so this thread should be related to pmacctd only to step a little deeper into that matter.

Greetings,

UE

EDIT(s):
- Added new pmacctd package with initscript and symlinks, in here --> http://people.ipfire.org/~ummeegge/pmac ... nitscript/ it is located. pmacctd starts after mysql and stops before mysql
Image
Image
Image

Trikolon
Community Developer
Community Developer
Posts: 550
Joined: October 16th, 2008, 6:21 am
Location: Erlangen
Contact:

Re: Pmacct - network monitoring

Post by Trikolon » September 8th, 2015, 7:50 am

Dear all,
current situation ist, that pmacctd with uacctd is working with no big CPU/System performance impact. I noticed that multicasts on 224.x.x.x subnet is accounted, too.

Currently I try to get rid of it by adding this line to the config file:
pcap_filter: not net 224.0.0.0 mask 240.0.0.0

Ben

Edit: pcap did not work! Currently testing filters based on tcpdump syntax.. lot of fun!

ummeegge
Community Developer
Community Developer
Posts: 4122
Joined: October 9th, 2010, 10:00 am

Re: Pmacct - network monitoring

Post by ummeegge » September 8th, 2015, 3:44 pm

Hi all,
Trikolon wrote:Edit: pcap did not work! Currently testing filters based on tcpdump syntax.. lot of fun!

Really lots of fun :D i like that tool...
If you´d find a good operating mode for uacctd, it might be nice if you append the config to the thread.

Just tested a little Pmacctd with MySQL and the /etc/pmacct/sql/pmacct-create-db_v8.mysql template which works nice until now. A little howto, the Config and some Bash scripts to read the DB out will may follow :) .

The current RAM usage vor MySQL and Pmacctd are:

Code: Select all

 Private  +   Shared  =  RAM used   Program
  8.2 MiB +   6.9 MiB =  15.1 MiB   pmacctd (2)
188.0 KiB +   1.2 MiB =   1.4 MiB   mysqld_safe
 18.4 MiB +   2.3 MiB =  20.8 MiB   mysqld
 


The database works since more or less 3 days in a small testing environment 1-4 Clients :

Code: Select all

-> ls -la /srv/mysql/pmacct/
total 1736
drwx------ 2 mysql mysql    4096 Sep  5 18:51 .
drwxr-xr-x 5 mysql mysql    4096 Sep  8 12:38 ..
-rw-rw---- 1 mysql mysql    9224 Sep  5 18:51 acct_v8.frm
-rw-rw---- 1 mysql mysql 1297996 Sep  8 17:39 acct_v8.MYD
-rw-rw---- 1 mysql mysql  444416 Sep  8 17:39 acct_v8.MYI
-rw-rw---- 1 mysql mysql      65 Sep  5 18:51 db.opt


EDIT: The new package is currently in working progress and contains also an initscript and the symlinks for the runlevels.
Image
Image
Image

ummeegge
Community Developer
Community Developer
Posts: 4122
Joined: October 9th, 2010, 10:00 am

Re: Pmacct - network monitoring

Post by ummeegge » September 9th, 2015, 7:50 am

New pmacct package with initscript is now in here --> http://people.ipfire.org/~ummeegge/pmacct available. It starts after MySQL and it stop before MySQL. It is currently hold simple and findable under /etc/rc.d/init.d/pmaccd, if you have some further ideas, let it me know.
The usage for it is:

Code: Select all

-> /etc/init.d/pmacctd 
Usage: /etc/init.d/pmacctd {start|stop|restart|status}
To get pmacctds initscript function into the IPFire WUI under 'Status-->Services-->Addon-Services' the following command can be used:

Code: Select all

touch /opt/pakfire/db/installed/meta-pmacctd
after this pmacctd appears in the WUI where you can start, stop and set the start on boot hook.
Image

Greetings,

UE
Image
Image
Image

Trikolon
Community Developer
Community Developer
Posts: 550
Joined: October 16th, 2008, 6:21 am
Location: Erlangen
Contact:

Re: Pmacct - network monitoring

Post by Trikolon » September 9th, 2015, 8:02 am

Thanks UE.
Currently my setup includes green and orange network and I want to know what host is using all my LTE volume :)

Basically it is working and I have a munin plugin to visualize the results but I have some filtering problems. The point is, that I just want to see the traffic to and from external but not the internal traffic between the local systems.. this seems to be little tricky.

Here are a short part of my config I am testing right now:

Code: Select all

daemonize: true
pidfile: /var/run/uacctd.pid
!#syslog: daemon
 
uacctd_group : 1
plugins: memory[host_in], memory[host_out]
 
aggregate[host_in]: dst_host
aggregate[host_out]: src_host
 
aggregate_filter[host_in]: dst net 192.168.0.0/24 and not src net 192.168.2.0/24, dst net 192.168.2.0/24 and not src net 192.1
68.0.0/24, not src net 224.0.0.0/24
 
aggregate_filter[host_out]: src net 192.168.0.0/24 and not dst net 192.168.2.0/24, src net 192.168.2.0/24 and not dst net 192.
168.0.0/24, not src net 224.0.0.0/24
 
!networks_file: /etc/pmacct/nfacctd.networks
 
imt_path[host_in]: /tmp/pmacct_host_in.pipe
imt_path[host_out]: /tmp/pmacct_host_out.pipe
 
!
!pcap_filter: not net 224.0.0.0 mask 240.0.0.0
!pcap_filter: not host 224.0.0.251
 
pmacctd_flow_buffer_buckets: 65536
pmacctd_flow_buffer_size: 128Mb
plugin_pipe_size: 10240000
plugin_buffer_size: 10240


"!" is a comment.

Any ideas on this?

Best regards
Ben

ummeegge
Community Developer
Community Developer
Posts: 4122
Joined: October 9th, 2010, 10:00 am

Re: Pmacct - network monitoring

Post by ummeegge » September 9th, 2015, 9:13 am

Hi Ben,
what happens if you grab the complete traffic from all interfaces

Code: Select all

interface: any

and include/exclude the different subnets over the aggregation filter ? So it should be possible to see the external IP´s but also the internal ones from green and orange and the BYTES field usage could points out which one uses your LTE volume.

As a first idea.

Greetings,

UE
Image
Image
Image

Trikolon
Community Developer
Community Developer
Posts: 550
Joined: October 16th, 2008, 6:21 am
Location: Erlangen
Contact:

Re: Pmacct - network monitoring

Post by Trikolon » September 9th, 2015, 9:55 am

It is even worse. I need to create own aggreaget and filter rules for each subnet (green and orange in my case) and exclude each from each other. But in LAN there is more traffic than just from local subnets. There are also multicasts and broadcasts from several systems what I am trying to filter at the moment...

Edit:
Current config:

Code: Select all

daemonize: true
pidfile: /var/run/uacctd.pid
uacctd_group : 1

plugins: memory[hostgreen_in], memory[hostgreen_out], memory[hostorange_in], memory[hostorange_out]
networks_file: /etc/pmacct/nfacctd.networks
 
!green
aggregate[hostgreen_in]: dst_host
aggregate[hostgreen_out]: src_host
 
aggregate_filter[hostgreen_in]: ((dst net 192.168.0.0/24) and (not src net 192.168.0.0/24)) and ((not src net 224) or (not src net 239))
aggregate_filter[hostgreen_out]: (src net 192.168.0.0/24) and (not dst net 192.168.0.0/24) and ((not dst net 224) or (not dst net 239))
 
imt_path[hostgreen_in]: /tmp/pmacct_hostgreen_in.pipe
imt_path[hostgreen_out]: /tmp/pmacct_hostgreen_out.pipe
 
!orange
aggregate[hostorange_in]: dst_host
aggregate[hostorange_out]: src_host
 
aggregate_filter[hostorange_in]: ((dst net 192.168.2.0/24) and (not src net 192.168.2.0/24)) and ((not src net 224) or (not src net 239))
aggregate_filter[hostorange_out]: ((src net 192.168.2.0/24) and (not dst net 192.168.2.0/24)) and ((not dst net 224) or (not dst net 239))
 
imt_path[hostorange_in]: /tmp/pmacct_hostorange_in.pipe
imt_path[hostorange_out]: /tmp/pmacct_hostorange_out.pipe
 
pmacctd_flow_buffer_buckets: 65536
pmacctd_flow_buffer_size: 128Mb
plugin_pipe_size: 10240000
plugin_buffer_size: 10240

ummeegge
Community Developer
Community Developer
Posts: 4122
Joined: October 9th, 2010, 10:00 am

MySQL with Pmaccd

Post by ummeegge » September 9th, 2015, 2:03 pm

I wanted to introduce a short howto make MySQL and Pmacctd working togehter.

Prerequisite:
1) MySQL needs to be installed which can be done via Pakfire

Code: Select all

pakfire install mysql
2) Pmaccd needs to be installed on that system too, see the first post for how you can do this.

If both are installed, we need to setup a DATABASE and a TABLE for Pmacctd.
Pmacctd delivers therefor templates which you can find under /etc/pmacct/sql. I used now 'pmacct-create-db_v8.mysql' which looks like this:

Code: Select all

drop database if exists pmacct;
create database pmacct;

use pmacct;

drop table if exists acct_v8; 
create table acct_v8 (
	agent_id INT(4) UNSIGNED NOT NULL,
	class_id CHAR(16) NOT NULL,
	mac_src CHAR(17) NOT NULL,
	mac_dst CHAR(17) NOT NULL,
	vlan INT(2) UNSIGNED NOT NULL,
	as_src INT(4) UNSIGNED NOT NULL,
	as_dst INT(4) UNSIGNED NOT NULL,
	ip_src CHAR(15) NOT NULL,
	ip_dst CHAR(15) NOT NULL,
	port_src INT(2) UNSIGNED NOT NULL,
	port_dst INT(2) UNSIGNED NOT NULL,
	tcp_flags INT(4) UNSIGNED NOT NULL,
	ip_proto CHAR(6) NOT NULL, 
	tos INT(4) UNSIGNED NOT NULL, 
        packets INT UNSIGNED NOT NULL,
	bytes BIGINT UNSIGNED NOT NULL,
	flows INT UNSIGNED NOT NULL,
	stamp_inserted DATETIME NOT NULL,
	stamp_updated DATETIME,
	PRIMARY KEY (agent_id, class_id, mac_src, mac_dst, vlan, as_src, as_dst, ip_src, ip_dst, port_src, port_dst, ip_proto, tos, stamp_inserted)
);
So you can see the 'Primary Keys' whereby this informations will be grabbed by pmacctd if possible. To feed this data into MySQL the following command was used:

Code: Select all

mysql < /etc/pmacct/sql/pmacct-create-db_v8.mysql
and the privileges needs to be granted. Start MySQL commandline with a simple

Code: Select all

mysql
and set the privileges:

Code: Select all

GRANT ALL PRIVILEGES on pmacct.* TO 'pmacct'@'localhost' IDENTIFIED BY 'pmacct';
thats all. No you can check the new DATABASE and the TEMPLATE with a:

Code: Select all

mysql> show databases;
which looks like this:
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| pmacct |
| test |
+--------------------+
4 rows in set (0.00 sec)
to switch into pmacct database you can use:

Code: Select all

mysql> use pmacct;
A

Code: Select all

mysql> show tables;
delivers the following if anything went right

Code: Select all

+------------------+
| Tables_in_pmacct |
+------------------+
| acct_v8          | 
+------------------+
1 row in set (0.00 sec)
to get an overview of the table:

Code: Select all

mysql> describe acct_v8;
which looks then like this:

Code: Select all

+----------------+---------------------+------+-----+---------+-------+
| Field          | Type                | Null | Key | Default | Extra |
+----------------+---------------------+------+-----+---------+-------+
| agent_id       | int(4) unsigned     | NO   | PRI | NULL    |       | 
| class_id       | char(16)            | NO   | PRI | NULL    |       | 
| mac_src        | char(17)            | NO   | PRI | NULL    |       | 
| mac_dst        | char(17)            | NO   | PRI | NULL    |       | 
| vlan           | int(2) unsigned     | NO   | PRI | NULL    |       | 
| as_src         | int(4) unsigned     | NO   | PRI | NULL    |       | 
| as_dst         | int(4) unsigned     | NO   | PRI | NULL    |       | 
| ip_src         | char(15)            | NO   | PRI | NULL    |       | 
| ip_dst         | char(15)            | NO   | PRI | NULL    |       | 
| port_src       | int(2) unsigned     | NO   | PRI | NULL    |       | 
| port_dst       | int(2) unsigned     | NO   | PRI | NULL    |       | 
| tcp_flags      | int(4) unsigned     | NO   |     | NULL    |       | 
| ip_proto       | char(6)             | NO   | PRI | NULL    |       | 
| tos            | int(4) unsigned     | NO   | PRI | NULL    |       | 
| packets        | int(10) unsigned    | NO   |     | NULL    |       | 
| bytes          | bigint(20) unsigned | NO   |     | NULL    |       | 
| flows          | int(10) unsigned    | NO   |     | NULL    |       | 
| stamp_inserted | datetime            | NO   | PRI | NULL    |       | 
| stamp_updated  | datetime            | YES  |     | NULL    |       | 
+----------------+---------------------+------+-----+---------+-------+
19 rows in set (0.01 sec)
Lets start Pmacctd:
- Since an initscript version --> http://people.ipfire.org/~ummeegge/pmacct is available, i would suggest to use this one.
- The configuration file for pmacctd (findable under /etc/pmacct/pmacctd.conf.example <-- please rename it to pmacctd.conf) looks like this one:

Code: Select all

!
! pmacctd configuration example
!
! Did you know CONFIG-KEYS contains the detailed list of all configuration keys
! supported by 'nfacctd' and 'pmacctd' ?
!
! debug: true
!

interface: any
!daemonize: false
daemonize: true
pidfile: /var/run/pmacctd.pid
aggregate: src_mac,dst_mac,src_host,dst_host,proto,src_port,dst_port
!aggregate: src_host,dst_host
! aggregate: src_net,dst_net
plugins: mysql
sql_user: pmacct
sql_passwd: pmacct
sql_db: pmacct
sql_table: acct_v8
sql_table_version: 8
sql_refresh_time: 90
! sql_optimize_clauses: true
sql_history: 10m
sql_history_roundoff: mh
! sql_preprocess: qnum=1000, minp=5
!
! networks_file: ./networks.example
! ports_file: ./ports.example
! sampling_rate: 10
! sql_trigger_time: 1h
!
'!' is used to outcomment directives.

Now you can start Pmacctd via console/SSH with a

Code: Select all

pmacctd -d -f /etc/pmacct/pmacctd.conf
or even better with the initscript and an

Code: Select all

/etc/init.d/pmaccd start
which delivers the following output:

Code: Select all

-> /etc/init.d/pmacctd start
Starting the pmacct daemon... 
DEBUG ( /etc/pmacct/pmacctd.conf ): plugin name/type: 'default'/'core'.
DEBUG ( /etc/pmacct/pmacctd.conf ): plugin name/type: 'default'/'mysql'.
DEBUG ( /etc/pmacct/pmacctd.conf ): interface:any
DEBUG ( /etc/pmacct/pmacctd.conf ): daemonize:true
DEBUG ( /etc/pmacct/pmacctd.conf ): pidfile:/var/run/pmacctd.pid
DEBUG ( /etc/pmacct/pmacctd.conf ): aggregate:src_mac,dst_mac,src_host,dst_host,proto,src_port,dst_port
DEBUG ( /etc/pmacct/pmacctd.conf ): sql_user:pmacct
DEBUG ( /etc/pmacct/pmacctd.conf ): sql_passwd:pmacct
DEBUG ( /etc/pmacct/pmacctd.conf ): sql_db:pmacct
DEBUG ( /etc/pmacct/pmacctd.conf ): sql_table:acct_v8
DEBUG ( /etc/pmacct/pmacctd.conf ): sql_table_version:8
DEBUG ( /etc/pmacct/pmacctd.conf ): sql_refresh_time:90
DEBUG ( /etc/pmacct/pmacctd.conf ): sql_history:10m
DEBUG ( /etc/pmacct/pmacctd.conf ): sql_history_roundoff:mh
DEBUG ( /etc/pmacct/pmacctd.conf ): debug:true
WARN ( default/core ): debug is enabled; forking in background. Console logging will get lost.       [  OK  ]
ps to check it double:

Code: Select all

-> ps aux | grep pmacct
root     21227  0.0  0.3  14284  7684 ?        Ss   15:36   0:00 pmacctd: Core Process [default]                 
root     21228  0.0  0.4  19224  8720 ?        S    15:36   0:00 pmacctd: MySQL Plugin [default] 
After a little while the first entries can be overviewed. To check the Table, go into MySQL again, use the acct_v8 table and select the desired entries, in this example i use all entries:

Code: Select all

mysql
mysql> use pmacct;
Database changed
mysql> select * from acct_v8;
a shortend output looks like this:

Code: Select all

+----------+----------+-------------------+-------------------+------+--------+--------+-----------------+-----------------+----------+----------+-----------+----------+-----+---------+------------+-------+---------------------+---------------------+
| agent_id | class_id | mac_src           | mac_dst           | vlan | as_src | as_dst | ip_src          | ip_dst          | port_src | port_dst | tcp_flags | ip_proto | tos | packets | bytes      | flows | stamp_inserted      | stamp_updated       |
+----------+----------+-------------------+-------------------+------+--------+--------+-----------------+-----------------+----------+----------+-----------+----------+-----+---------+------------+-------+---------------------+---------------------+
|        0 | unknown  | 0:0:0:0:0:0       | 0:0:0:0:0:0       |    0 |      0 |      0 | 192.168.7.2    | 192.168.7.18   |    51259 |      222 |         0 | tcp      |   0 |     192 |      13000 |     0 | 2015-09-05 18:50:00 | 2015-09-05 18:57:01 | 
|        0 | unknown  | 0:0:0:0:0:0       | 0:0:0:0:0:0       |    0 |      0 |      0 | 192.168.7.18   | 192.168.7.2    |      222 |    51259 |         0 | tcp      |   0 |     140 |      25228 |     0 | 2015-09-05 18:50:00 | 2015-09-05 18:57:01 | 
|        0 | unknown  | 0:0:0:0:0:0       | 0:0:0:0:0:0       |    0 |      0 |      0 | 192.168.20.2   | 192.168.20.1   |        0 |        0 |         0 | icmp     |   0 |      13 |       1352 |     0 | 2015-09-05 18:50:00 | 2015-09-05 19:00:01 | 
|        0 | unknown  | 0:0:0:0:0:0       | 0:0:0:0:0:0       |    0 |      0 |      0 | 192.168.20.1   | 192.168.20.2   |        0 |        0 |         0 | icmp     |   0 |      13 |       1352 |     0 | 2015-09-05 18:50:00 | 2015-09-05 19:00:01 | 
|        0 | unknown  | 0:0:0:0:0:0       | 0:0:0:0:0:0       |    0 |      0 |      0 | 192.168.7.2    | 192.168.7.18   |    51261 |      800 |         0 | tcp      |   0 |      57 |       8366 |     0 | 2015-09-05 18:50:00 | 2015-09-05 18:57:01 | 
|        0 | unknown  | 0:0:0:0:0:0       | 0:0:0:0:0:0       |    0 |      0 |      0 | 192.168.7.18   | 192.168.7.2    |      800 |    51261 |         0 | tcp      |   0 |      53 |      37817 |     0 | 2015-09-05 18:50:00 | 2015-09-05 18:57:01 | 
|        0 | unknown  | 0:0:0:0:0:0       | 0:0:0:0:0:0       |    0 |      0 |      0 | 127.0.0.1       | 127.0.0.1       |    51945 |       53 |         0 | udp      |   0 |      25 |       1609 |     0 | 2015-09-05 18:50:00 | 2015-09-05 19:00:01 | 
|        0 | unknown  | 0:0:0:0:0:0       | 0:0:0:0:0:0       |    0 |      0 |      0 | 192.168.20.2   | 8.8.8.8         |    47537 |       53 |         0 | udp      |   0 |       1 |         73 |     0 | 2015-09-05 18:50:00 | 2015-09-05 18:55:31 | 
|        0 | unknown  | 0:0:0:0:0:0       | 0:0:0:0:0:0       |    0 |      0 |      0 | 192.168.20.2   | 8.8.4.4         |    47537 |       53 |         0 | udp      |   0 |       1 |         73 |     0 | 2015-09-05 18:50:00 | 2015-09-05 18:55:31 | 
|        0 | unknown  | 0:0:0:0:0:0       | 0:0:0:0:0:0       |    0 |      0 |      0 | 8.8.8.8         | 192.168.20.2   |       53 |    47537 |         0 | udp      |   0 |       1 |        705 |     0 | 2015-09-05 18:50:00 | 2015-09-05 18:55:31 | 
|        0 | unknown  | 0:0:0:0:0:0       | 0:0:0:0:0:0       |    0 |      0 |      0 | 8.8.4.4         | 192.168.20.2   |       53 |    47537 |         0 | udp      |   0 |       1 |        705 |     0 | 2015-09-05 18:50:00 | 2015-09-05 18:55:31 | 
|        0 | unknown  | 0:0:0:0:0:0       | 0:0:0:0:0:0       |    0 |      0 |      0 | 127.0.0.1       | 127.0.0.1       |       53 |    51945 |         0 | udp      |   0 |      25 |       4408 |     0 | 2015-09-05 18:50:00 | 2015-09-05 19:00:01 | 
|        0 | unknown  | 0:0:0:0:0:0       | 0:0:0:0:0:0       |    0 |      0 |      0 | 192.168.20.2   | 178.63.73.246   |    39283 |      443 |         0 | tcp      |   0 |      38 |       7235 |     0 | 2015-09-05 18:50:00 | 2015-09-05 18:57:01 | 
|        0 | unknown  | 0:0:0:0:0:0       | 0:0:0:0:0:0       |    0 |      0 |      0 | 178.63.73.246   | 192.168.20.2   |      443 |    39283 |         0 | tcp      |   0 |      34 |      36790 |     0 | 2015-09-05 18:50:00 | 2015-09-05 18:57:01 | 
|        0 | unknown  | 0:0:0:0:0:0       | 0:0:0:0:0:0       |    0 |      0 |      0 | 192.168.7.2    | 192.168.7.18   |    51262 |      800 |         0 | tcp      |   0 |      16 |       2642 |     0 | 2015-09-05 18:50:00 | 2015-09-05 18:55:31 |
not good to display it in the forum :o , but it is structured...

Fast overview over the shell:

Code: Select all

echo "use pmacct; SELECT ip_src,ip_dst,port_dst,bytes,stamp_updated  FROM acct_v8" | mysql
whereby the output was reduced to source_IP, destination_IP, destination_port,bytes, time_stamp and update_stamp.

Fast and first idea howto get the data of one specific IP (example with '192.168.7.2') a little better formated:

Code: Select all

echo "use pmacct; SELECT mac_src,ip_src,ip_dst,port_dst,bytes,stamp_updated  FROM acct_v8" | \
mysql | \
awk 'BEGIN { printf "%-20s %-18s %-18s %-12s %-15s %-11s %-15s\n" , \
"SRC_MAC", "SRC_IP", "DEST_IP", "DST_PORT", "BYTES", "STAMP", "UPDATED" \
}
/192.168.7.2/ { printf "%-20s %-18s %-18s %-12s %-15s %-11s %-15s\n", $1,$2,$3,$4,$5,$6,$7 }'
interessted in how much bytes was used by this IP ?:

Code: Select all

echo "use pmacct; SELECT ip_src,ip_dst,bytes FROM acct_v8" | \
mysql | \
awk '/192.168.7.2/ {sum+=$3} END{print sum}' | \
awk '{ sum=$1 ; hum[1024**3]="GB";hum[1024**2]="MB";hum[1024]="KB"; for (x=1024**3; x>=1024; x/=1024){ if (sum>=x) { printf "%.2f %s\n",sum/x,hum[x];break } }}'
there are a lot more possible but this should be enough for the first :) .

Greetings,

UE

EDIT: Since i´m nobbing around with both MySQL and Pmacctd ideas/corrections/further_improvments might be nice 8) .

@Ben
To sort Multi- Broadcast or even localhost out, the following command might be working ? :

Code: Select all

echo "use pmacct; SELECT mac_src,ip_src,ip_dst,port_dst,bytes,stamp_updated  FROM acct_v8" | mysql | awk '
BEGIN { printf "%-20s %-18s %-18s %-12s %-15s %-11s %-15s\n" , \
"SRC_MAC", "SRC_IP", "DEST_IP", "DST_PORT", "BYTES", "STAMP", "UPDATED" } \
{ printf "%-20s %-18s %-18s %-12s %-15s %-11s %-15s\n", $1,$2,$3,$4,$5,$6,$7 }' | \
sed -e '/\.*.\.*.\.*.255/d' -e '/224.0.0.251/d'  -e '/127.0.0.1/d'
Image
Image
Image

Trikolon
Community Developer
Community Developer
Posts: 550
Joined: October 16th, 2008, 6:21 am
Location: Erlangen
Contact:

Re: Pmacct - network monitoring

Post by Trikolon » September 10th, 2015, 12:53 pm

Dear all,
I was thinking about using the iptables with ULOG and pmacctd possibility because I have some problems to just account the traffic to and from external and not the internal one. As in the Traffic Accounting topic mentioned there is a way to mark such traffic. But:

Code: Select all

1066 CONFIG_IP_NF_TARGET_SYNPROXY=m
1067 # CONFIG_IP_NF_TARGET_ULOG is not set
1068 CONFIG_NF_NAT_IPV4=m


Is it possible to activate ULOG for the next Core update?

Regards
Ben

@UE:
Looks very interessting! I need more time to get hands on it..

Edit: After reading your last post more in detail, I am thinking about doing the whole filtering in bash.. may be the results are more clear there and debugging is much easier.

ummeegge
Community Developer
Community Developer
Posts: 4122
Joined: October 9th, 2010, 10:00 am

Re: Pmacct - network monitoring

Post by ummeegge » September 11th, 2015, 5:28 am

Good morning Ben,
Trikolon wrote:@UE:
Looks very interessting! I need more time to get hands on it..

take the time you need, i think it might be useful to sort important things out in that manner but also to become a better overview it might be a good idea to test all that stuff so more pro and cons can help to find the best individual way.

To read out the PIPES with the memory plugin is handy and uncomplicated too but if a reboot, Pmacct crash or a FW crash happens all collected data goes to the nirvana. A DB is more save in that manner i think, even if MySQL or Pmacct crashes, it might be possible to CHECK or REPAIR a TABLE or you loos only a few data not the whole bunch.

I give a little Bash script now a shoot which may should solve the following problems ? :
Trikolon wrote:But in LAN there is more traffic than just from local subnets. There are also multicasts and broadcasts from several systems what I am trying to filter at the moment...

and may this too ?
Trikolon wrote:Currently my setup includes green and orange network and I want to know what host is using all my LTE volume :)

Basically it is working and I have a munin plugin to visualize the results but I have some filtering problems. The point is, that I just want to see the traffic to and from external but not the internal traffic between the local systems.. this seems to be little tricky.


The script looks now like this:

Code: Select all

#!/bin/bash -

#
# Investigate total usage of transfer data from all LAN IPs
#
# ummeegge $date 10.09.2015
############################################################
# DB = MySQL with acct_v8 template
#

## Vars
# LAN ip address room
LANIPS="192.168";
# Time search values
YEAR=$(date +"%Y\-");
MONTH=$(date +"\-%m\-");
DAY=$(date +"\-%m\-%d");

## Search for LAN IPs, without Multi- and Broadcast traffic value
# Read out MySQL with needed data
IPS=$(echo "use pmacct; SELECT ip_src,ip_dst,bytes,stamp_updated  FROM acct_v8" | \
mysql | \
# Delete not needed traffic amount -> Multi- and Broadcast
sed -e '/\.*.\.*.\.*.255/d' -e '/224.0.0.\.*/d' | \
# Print source IPs
awk '{ print $2 }' | \
# Sort them and filter repeated lines
sort -t. -k 3,3n -k 4,4n | uniq | \
# Search for LAN addresses
grep "${LANIPS}");

## Investigate total usage of transfer data in GB for in and out for all LAN IPs and to a special date 
# gigabytes calculated in old value and for today
clear;
echo -e "         \e[0;31mIPs          -       GBs\e[0m";
echo "--------------------------------------------";
for i in ${IPS}; do
    echo "use pmacct; SELECT ip_src,ip_dst,bytes,stamp_updated FROM acct_v8" | \
    mysql | \
    grep "${DAY}" | \
    awk '/'"$i"'/ { total=total + $3/1024/1024/1024 } END { print "IP '"$i"' used - "total " Gigabyte" }'
    echo "--------------------------------------------";
done
echo;

# End script


Trikolon wrote:Edit: After reading your last post more in detail, I am thinking about doing the whole filtering in bash.. may be the results are more clear there and debugging is much easier.

I think so. There are a lot of 3rd party software for Pmacct data evaluation out there mostly to display the traffic under different circumstances, but if you want only some specific informations or even an active response (transfer limits, etc.) there is the need to process values further and with Bash a few lines should be enough to reach a lot of goals.

May you can find in there some useful hints.

Greetings,

UE
Image
Image
Image

Trikolon
Community Developer
Community Developer
Posts: 550
Joined: October 16th, 2008, 6:21 am
Location: Erlangen
Contact:

Re: Pmacct - network monitoring

Post by Trikolon » September 11th, 2015, 7:20 am

Good morning,
your script looks pretty nice! I will test it and I totally agree with your sql vs. memory ideas. I will change my setup to mysql asap.. but this we i am totally busy.

Here is my script I am testing since yesterday. It looks good, but the traffic of vnstat is a little bit more than accounted with pmacctd. One possible option is that I am not accounting traffic from the red0 device and the LTE Modem device. I need to analyze this more in detail.

pmacctd.conf:

Code: Select all

daemonize: true
pidfile: /var/run/uacctd.pid
 
interface: any
uacctd_group : 1
plugins: memory[all]
networks_file: /etc/pmacct/nfacctd.networks
 
pmacctd_flow_buffer_buckets: 65536
pmacctd_flow_buffer_size: 128Mb
plugin_pipe_size: 10240000
plugin_buffer_size: 10240
 
aggregate[all]: src_mac,dst_mac,src_host,dst_host,proto,src_port,dst_port
imt_path[all]: /tmp/pmacct_all.pipe


trafficaccounting.sh

Code: Select all

#!/bin/bash
 
TRAFFICFILE='/tmp/traffic.log'
/usr/bin/pmacct -p /tmp/pmacct_all.pipe -s > $TRAFFICFILE
 
TRAFFICFILE_TMP_UP='/tmp/traffic_tmp_up.log'
TRAFFICFILE_TMP_DOWN='/tmp/traffic_tmp_down.log'
 
# cleaning up
echo > $TRAFFICFILE_TMP_UP
echo > $TRAFFICFILE_TMP_DOWN
 
while read line; do
        if [ -z "${line}" ]; then
                continue
        elif [[ "${line}" == *"_"* ]]; then
                continue
        elif [[ "${line}" == *"255"* ]]; then
                continue
                elif [[ "${line}" == *"127."* ]]; then
                continue
                elif [[ "${line}" == *"192.168.3."* ]]; then #RED Subnet
                continue
                elif [[ "${line}" == *"239."* ]]; then
                continue
                elif [[ "${line}" == *"224."* ]]; then
                continue
                elif [[ "${line}" == *"232."* ]]; then
                continue
                elif [[ "${line}" == *"233."* ]]; then
                continue
                elif [[ "${line}" == *"234."* ]]; then
                continue
        else
            # Exclude local traffic between hosts
                                if [[ "$(echo ${line} | awk '{print $3}')" == *"192.168"* ]] && [[ "$(echo ${line} | awk '{print $4}')" == *"192.168"* ]]; then
                                                continue
                                else
                                                # upload / out
                                                if [[ "$(echo ${line} | awk '{print $3}')" == *"192.168"* ]]; then
                                                                echo ${line} >> $TRAFFICFILE_TMP_UP
                                                # download / in
                                                elif [[ "$(echo ${line} | awk '{print $4}')" == *"192.168"* ]]; then
                                                        echo ${line} >> $TRAFFICFILE_TMP_DOWN
                                                else
                                                                echo "Nothing accounted"
                                                fi
                                fi
                fi
done < $TRAFFICFILE
 
# UP
awk '{arr[$3]+=$9;} END {for (i in arr) print i, arr[i]}' $TRAFFICFILE_TMP_UP > /tmp/trafficOut.log
# DOWN
awk '{arr[$4]+=$9;} END {for (i in arr) print i, arr[i]}' $TRAFFICFILE_TMP_DOWN > /tmp/trafficIn.log


The script is pretty much quick and dirty but for testing its ok.

Best regards
Ben

Trikolon
Community Developer
Community Developer
Posts: 550
Joined: October 16th, 2008, 6:21 am
Location: Erlangen
Contact:

Re: Pmacct - network monitoring

Post by Trikolon » September 17th, 2015, 8:56 am

Dear all,
I did some more testing and basically it is working, but I still got a small accounting error of ~50-100MB per day. I changed my setup to log to mysql and after that doing some ugly bash scripting (extremely ineffective) but is is doing its job.

Now I want to test the netfilter ulog module but for this I needed to change some things (recompiling the kernel, adding some modules, etc..). Everything I did is here:
http://git.ipfire.org/?p=people/trikolo ... gd_pmacctd

If someone is interessted in this, please check whether it is working or not.

Best regrads
Ben

Edit:
Here is the script for calculating the traffic per IP and excluding internal traffic:

Code: Select all

#!/bin/bash
TRAFFICFILE='/tmp/traffic.log'
echo "use pmacct; SELECT ip_src,ip_dst,port_dst,bytes,stamp_updated  FROM acct_v8" | mysql | grep $(date +"%Y-%m-%d") | \
grep -v 255 | grep -v 127. | grep -v 239 | grep -v 224 | grep -v 232 | grep -v 233 | grep -v 234 | grep -v 192.168.3. | \
grep -v 192.168.0.10 | grep -v 192.168.2.10 | egrep -v '(192.168.*){2}' > $TRAFFICFILE

#Upload
cat $TRAFFICFILE | awk '{print $1 " " $4}' | grep 192.168. | awk '{arr[$1]+=$2;} END {for (i in arr) print i, arr[i]}' > /tmp/trafficOut.log
#Download
cat $TRAFFICFILE | awk '{print $2 " " $4}' | grep 192.168. | awk '{arr[$1]+=$2;} END {for (i in arr) print i, arr[i]}' > /tmp/trafficIn.log


BTW: 192.168.0.10 and 192.168.2.10 are the IPs of the Ipfire interfaces (=gateway).

ummeegge
Community Developer
Community Developer
Posts: 4122
Joined: October 9th, 2010, 10:00 am

Re: Pmacct - network monitoring

Post by ummeegge » September 18th, 2015, 4:58 pm

Hi Ben,
thanks for your ideas :) . I give currently the Ulogd scenario also a try (still in building process) but i have extend also the Kernel config for 'kernel.config.armv5tel-ipfire-kirkwood' and 'kernel.config.armv5tel-ipfire-multi' , which i can´t in fact test at this moment but may this changes, will see.

Have checked also the Print Plugin, will write here also a little howto in the next few days.

It could also be nice if we think about what scenarios might be useful in the filter section (also with e.g. IPTables, sendEmail integration) so we could may go for further Bash scriptings ? An e.g. whiptail backend to store those filters and give may an easier configuration tool by the hand may also another step ?! Will see.

Greetings,

UE
Image
Image
Image

Trikolon
Community Developer
Community Developer
Posts: 550
Joined: October 16th, 2008, 6:21 am
Location: Erlangen
Contact:

Re: Pmacct - network monitoring

Post by Trikolon » October 11th, 2015, 6:39 pm

Hi,
any updates on this?

Ben

ummeegge
Community Developer
Community Developer
Posts: 4122
Joined: October 9th, 2010, 10:00 am

Re: Pmacct - network monitoring

Post by ummeegge » October 12th, 2015, 12:31 pm

Hi Ben,
from my side currently not, some time ago i made some more tests with the print plugin but left that behind.
Pmacct makes a smooth job in the above described configuration. And from your side ? Something new ? What do you think about the already written --> https://forum.ipfire.org/viewtopic.php? ... 849#p90370 ?

Greetings,

UE
Image
Image
Image

Post Reply

Who is online

Users browsing this forum: No registered users and 0 guests