IP Accounting

Help on building IPFire & Feature Requests
joseadias
Posts: 2
Joined: April 19th, 2015, 6:50 pm

IP Accounting

Post by joseadias » August 16th, 2015, 10:43 pm

All,

I've been looking for an "IP accounting" type of solution for my firewall. The old IPCop 1.4.21 had a addon that would count IP traffic in and out of the firewall. This is mighty helpful when you have teenagers and a quota (what do you mean 320 GB per month is not enough ?!?)

IPFire is a great, flexible, firewall distribution, but I'd like to add this if possible.

I've added some iptables rules to essentially count the traffic but I have questions about the implementation.

For now, is this a good idea, or am I barking the wrong tree?

let me know what you all think.

ummeegge
Community Developer
Community Developer
Posts: 5001
Joined: October 9th, 2010, 10:00 am

Re: IP Accounting

Post by ummeegge » August 18th, 2015, 3:01 pm

Hi oseadias,
IP accounting via IPTables is surley a widely used way e.g. --> http://www.tldp.org/LDP/nag2/x-087-2-ac ... fwadm.html. I try currently to build pmacct --> http://www.pmacct.net/ which might also be interessting in that manner. iptraf-ng --> http://wiki.ipfire.org/en/addons/iptraf-ng/start delivers also possibilities to do that job but you will need in all cases to adapt it to your needs via scripts to set transfer limit restrictions.

UE
Image
Image

joseadias
Posts: 2
Joined: April 19th, 2015, 6:50 pm

Re: IP Accounting

Post by joseadias » August 18th, 2015, 11:39 pm

Hello ummeegge,

Your project sounds very interesting, but it's not yet available on IPFire, and to be honest it's much bigger than what I intent it to be.

I'm looking for a high level view at the home page, like this.

Network IP address Status
LAN 192.168.42.253/24 Proxy on
Wireless 192.168.242.241/24 Proxy on

traffic in: 14G

traffic out: 2057M



Currently I'm still at the design stage so not much code has gone into scripts. To find out general volume I ssh into the firewall and do this:
[root@harold ipaccounting]# ./ipaccounting.sh status | grep 192.168.242.9
7392K 10G all -- * * 0.0.0.0/0 192.168.242.9
3701K 799M all -- * * 192.168.242.9 0.0.0.0/0

and I can see that my daughter has been very busy on her phone: 10GB in and 800MB out in about 7 days.

I'm looking for a general consensus: besides myself, is this needed/wanted?

vkykam
Posts: 9
Joined: April 17th, 2015, 1:16 am

Re: IP Accounting

Post by vkykam » August 19th, 2015, 2:05 am

joseadias wrote:Hello ummeegge,

I'm looking for a general consensus: besides myself, is this needed/wanted?


I'd love to see something like this, so that we can keep tabs on our users to see who's abusing our VSAT connection. Currently we're using transparent proxy, so there's no built-in ways with IPFire to count the https traffic.

ummeegge
Community Developer
Community Developer
Posts: 5001
Joined: October 9th, 2010, 10:00 am

Re: IP Accounting

Post by ummeegge » August 21st, 2015, 7:36 am

Hi all,
have compiled now pmacct for IPFire. In here --> http://people.ipfire.org/~ummeegge/pmacct/ you can find the IPFire package.
Installation:
To install it, copy it to /opt/pakfire/tmp unpack it with an

Code: Select all

tar xvf pmacct-1.5.1-1.ipfire

and install it with an

Code: Select all

./install.sh

to set the console free you can use [CTRL-c].

Haven´t tried that much with pmacct but it is very powerful but can also be used in a simple way. There is also a print plugin available where you can write to e.g. a plain text file or in a csv format. Here --> http://wiki.pmacct.net/ is the pmacct wiki home.

For a simple, first impression you can use e.g.

Code: Select all

pmacctd -c src_host,dst_host,proto,dst_port -P memory -i any

in one terminal and in the second this one

Code: Select all

pmacct -p /tmp/collect.pipe -s

to display the pipe output, tmux --> http://wiki.ipfire.org/en/addons/tmux/start can also be used to use both commands in one window (at one glance). The first command are used in memory only without databases and could looks like this example.

Code: Select all

-> pmacct -p /tmp/collect.pipe -s
SRC_IP           DST_IP           DST_PORT  PROTOCOL    PACKETS             BYTES
192.168.7.8     192.168.7.2       50880     tcp         14                  804
192.168.20.2    192.168.20.1      0         icmp        2                   208
192.168.7.2     192.168.7.8       444       tcp         57                  12293
192.168.7.8     192.168.7.2       50979     tcp         21                  3713
192.168.7.2     192.168.7.8       222       tcp         44                  6113
192.168.20.1    192.168.20.2      0         icmp        2                   208
192.168.7.8     192.168.7.2       50957     tcp         38                  11248

For a total of: 7 entries

The usage of configuration files (under /etc/pmacct) is also possible. Initscripts aren´t currently integrated into the package but should be no problem if there are further interrests in that manner.

Nevertheless, i have compiled pmacct with mysql and sqlite3 support.
May you are interessted to give it some tries.

Greetings,

UE

EDIT: Related files in this package are:
/usr/bin/pmacct
/usr/bin/pmmyplay
/usr/sbin/nfacctd
/usr/sbin/pmacctd
/usr/sbin/sfacctd
/usr/sbin/uacctd
/etc/pmacct/<-- config, examples and database presets
Image
Image

smicha
Posts: 4
Joined: June 6th, 2015, 10:14 am

Re: IP Accounting

Post by smicha » August 23rd, 2015, 4:40 pm

Hi,

ummeegge wrote:have compiled now pmacct for IPFire. In here --> http://people.ipfire.org/~ummeegge/pmacct/ you can find the IPFire package.


thank you for the packet "pmacct".
I can install it without any problems.

Now, I try to use pmacct in combination with the iptables "ULog" interface which is available in kernel module "ipt_ULOG.ko".
See: http://blog.raptor2101.de/2012/03/07/li ... ccaunting/

The problem is now, there is no kernel module "ipt_ULOG.ko" in ipfire.
How I can get the kernel module "ipt_ULOG.ko"?
Does someone have compiled it?

Many thanks
Michael

ummeegge
Community Developer
Community Developer
Posts: 5001
Joined: October 9th, 2010, 10:00 am

Re: IP Accounting

Post by ummeegge » August 24th, 2015, 6:21 am

Hi Michael,
your welcome.
smicha wrote:Now, I try to use pmacct in combination with the iptables "ULog" interface which is available in kernel module "ipt_ULOG.ko".
See: http://blog.raptor2101.de/2012/03/07/li ... ccaunting/

The problem is now, there is no kernel module "ipt_ULOG.ko" in ipfire.
How I can get the kernel module "ipt_ULOG.ko"? Does someone have compiled it?

There seems to be a outcommented version of ipt_ULOG.ko for Armv5tel but only for the RPI (RaspberryPi) --> https://github.com/ipfire/ipfire-2.x/se ... pt_ULOG.ko .
After a little research i have found out that this kernel module is deprecated and have been replaced with nfnetlink_log. Reference can be found under Linus Git repo --> https://github.com/torvalds/linux/blob/ ... link_log.c .
So the IPFire sources got this one --> https://github.com/ipfire/ipfire-2.x/se ... &type=Code but it is also outcommented and thus also not findable in the actuall IPFire ISO´s.
I´am not sure if this module is a full-fledged supplementation for your purposes, but may if i have more time for that i can give it a shoot to compile it.
So i would suggest to try some other ways (there are a lots of) to use pmacct.

Greetings,

UE
Image
Image

ummeegge
Community Developer
Community Developer
Posts: 5001
Joined: October 9th, 2010, 10:00 am

Re: IP Accounting

Post by ummeegge » August 25th, 2015, 9:38 am

Have compiled now also the nfnetlink Kernel modules. In here --> http://people.ipfire.org/~ummeegge/pmacct/ the nfnetlink_modules.tar.gz can be found.
@Michael
You can give it a try if you can use them also in that --> http://blog.raptor2101.de/2012/03/07/li ... ccaunting/ topic. The package includes all nfnetlink modules, probably more than you need.
Testing results might be interessting...


EDIT: I think the modules onyl are useless --> http://www.netfilter.org/projects/ulogd/ !

Greetings,

UE
Image
Image

Trikolon
Community Developer
Community Developer
Posts: 552
Joined: October 16th, 2008, 6:21 am
Location: Erlangen
Contact:

Re: IP Accounting

Post by Trikolon » September 4th, 2015, 8:58 am

Hi,
uacctd is not working. I guess pmacctd is not compiled with --enable-ulog. Could you please recompile it with this option? Thanks!

Ben

ummeegge
Community Developer
Community Developer
Posts: 5001
Joined: October 9th, 2010, 10:00 am

Re: IP Accounting

Post by ummeegge » September 5th, 2015, 8:51 am

Hi Ben,
have uploaded the new compiled pmacct --> http://people.ipfire.org/~ummeegge/pmacct/ . This version includes now --enable-ulog, have added also --enable-threads .

Code: Select all

-> pmacct -V
pmacct, pmacct client 1.5.1 (20150215-01)
 --prefix=/usr --sysconfdir=/etc/pmacct --enable-sqlite3 --enable-mysql --enable-threads --enable-ulog

For suggestions, critics, bugs, contact me: Paolo Lucente <paolo@pmacct.net>.


Have added also beneath an pmacctd.conf.example a networks.lst.example and ports.lst.example into /etc/pmacct which are taken from /etc/pmacct/examples.
All files should be renamed/activated/modified_to_the_individual_needs, if there are ideas/extensions for an initscript/configuration_ideas a discussion could be interessting ?!

Might be great if you share also your setup with us. If there are other things to work out in this topic, we could think about to make an own topic for this ?!

Let´s see.

Greetings,

UE
Image
Image

ummeegge
Community Developer
Community Developer
Posts: 5001
Joined: October 9th, 2010, 10:00 am

Re: IP Accounting

Post by ummeegge » September 6th, 2015, 2:05 pm

Hi all,
smicha wrote:Now, I try to use pmacct in combination with the iptables "ULog" interface which is available in kernel module "ipt_ULOG.ko".
See: http://blog.raptor2101.de/2012/03/07/li ... ccaunting/

I`d gave http://blog.raptor2101.de/2012/03/07/li ... ccaunting/ a short try without the '-j ULOG' rules (without IPTables) and the created pipes delivers also an appropriate output.

Let´s give it a fast shoot:
The configuration file /etc/pmacct/pmaccd.conf looks now pretty similar to this one from the above mentioned howto:

Code: Select all

!
! pmacctd configuration example
!
! Did you know CONFIG-KEYS contains the detailed list of all configuration keys
! supported by 'nfacctd' and 'pmacctd' ?
!
daemonize: true
pidfile: /var/run/uacctd.pid
! syslog: daemon
 
uacctd_group : 1
plugins: memory[host_in], memory[host_out]
 
aggregate[host_in]: dst_host
aggregate[host_out]: src_host
 
aggregate_filter[host_in]: dst net 192.168.7.0/24
aggregate_filter[host_out]: src net 192.168.7.0/24
! Check only one Host
!aggregate_filter[host_in]: dst net 192.168.7.200
!aggregate_filter[host_out]: src net 192.168.7.200

 
imt_path[host_in]: /tmp/pmacct_host_in.pipe
imt_path[host_out]: /tmp/pmacct_host_out.pipe
! ...


the /24 subnetmask grabs the green0 interface in my network.
If you start now the configuration with a

Code: Select all

pmacctd -d -f /etc/pmacct/pmaccd.conf

it looks like this:

Code: Select all

-> pmacctd -d -f /etc/pmacct/pmacctd.conf 
DEBUG ( /etc/pmacct/pmacctd.conf ): plugin name/type: 'default'/'core'.
DEBUG ( /etc/pmacct/pmacctd.conf ): plugin name/type: 'host_in'/'memory'.
DEBUG ( /etc/pmacct/pmacctd.conf ): plugin name/type: 'host_out'/'memory'.
DEBUG ( /etc/pmacct/pmacctd.conf ): daemonize:true
DEBUG ( /etc/pmacct/pmacctd.conf ): pidfile:/var/run/uacctd.pid
DEBUG ( /etc/pmacct/pmacctd.conf ): syslog:daemon
DEBUG ( /etc/pmacct/pmacctd.conf ): uacctd_group:1
DEBUG ( /etc/pmacct/pmacctd.conf ): aggregate[host_in]:dst_host
DEBUG ( /etc/pmacct/pmacctd.conf ): aggregate[host_out]:src_host
DEBUG ( /etc/pmacct/pmacctd.conf ): aggregate_filter[host_in]:dst net 192.168.7.0/24
DEBUG ( /etc/pmacct/pmacctd.conf ): aggregate_filter[host_out]:src net 192.168.7.0/24
DEBUG ( /etc/pmacct/pmacctd.conf ): imt_path[host_in]:/tmp/pmacct_host_in.pipe
DEBUG ( /etc/pmacct/pmacctd.conf ): imt_path[host_out]:/tmp/pmacct_host_out.pipe
DEBUG ( /etc/pmacct/pmacctd.conf ): debug:true
WARN ( default/core ): debug is enabled; forking in background. Console logging will get lost.

an

Code: Select all

ps aux | grep uacct

delivers

Code: Select all

root      9165  0.0  0.6  26592 12368 ?        Ss   15:47   0:00 uacctd: Core Process [default]        
root      9166  0.0  0.3  18608  7704 ?        S    15:47   0:00 uacctd: IMT Plugin [host_in]         
root      9167  0.0  0.3  18608  7700 ?        S    15:47   0:00 uacctd: IMT Plugin [host_out]

and in /tmp the pipes are created

Code: Select all

-> ls -la /tmp
total 2
srwxrwxrwx  1 root root    0 Sep  6 15:47 pmacct_host_in.pipe
srwxrwxrwx  1 root root    0 Sep  6 15:47 pmacct_host_out.pipe

they gives also output which can be checked with an

Code: Select all

pmacct -s -p /tmp/pmacct_host_in.pipe

and an

Code: Select all

pmacct -s -p /tmp/pmacct_host_out.pipe

a possible output for the out pipe could looks like this:

Code: Select all

SRC_IP           PACKETS               BYTES
192.168.7.2      287                   15600
192.168.7.120    271                   40664

For a total of: 2 entries

and the in pipe like this:

Code: Select all

DST_IP           PACKETS               BYTES
192.168.7.2      271                   40664
192.168.7.255    6                     468
192.168.7.120    281                   15132

For a total of: 3 entries

.
@Michael
I´am not sure what else informations in the above mentioned howto should be delivered ?

Greetings,

UE
Image
Image

Trikolon
Community Developer
Community Developer
Posts: 552
Joined: October 16th, 2008, 6:21 am
Location: Erlangen
Contact:

Re: IP Accounting

Post by Trikolon » September 7th, 2015, 8:46 am

Hi all,
I tried the mentioned setup and something is not working as expected. According to vnstat I have ~180MB for today.
Here is my pmacctd.conf:

Code: Select all

daemonize: true
pidfile: /var/run/uacctd.pid
#syslog: daemon
 
uacctd_group : 1
plugins: memory[host_in], memory[host_out]
 
aggregate[host_in]: dst_host
aggregate[host_out]: src_host
 
aggregate_filter[host_in]: dst net 192.168.0.0/24, dst net 192.168.2.0/24 ## Green and Orange
aggregate_filter[host_out]: src net 192.168.0.0/24, src net 192.168.2.0/24
 
imt_path[host_in]: /tmp/pmacct_host_in.pipe
imt_path[host_out]: /tmp/pmacct_host_out.pipe


Output:

Code: Select all

[root@ipfire ~]# pmacct -s -p /tmp/pmacct_host_out.pipe
SRC_IP           PACKETS               BYTES
192.168.0.11     6                     456
192.168.0.1      5933                  3099919
192.168.2.1      12342                 730541
192.168.0.193    1146                  160214
192.168.0.253    17186                 1509136
192.168.0.7      1946                  483251
192.168.0.10     1028                  170143
192.168.0.4      5137                  1678591
192.168.0.176    43200                 1831317
192.168.0.9      3178                  931789
192.168.0.189    1                     48
 
For a total of: 11 entries
[root@ipfire ~]# pmacct -s -p /tmp/pmacct_host_in.pipe
DST_IP           PACKETS               BYTES
192.168.0.11     2                     152
192.168.0.193    822                   465014
192.168.0.253    29002                 48761564
192.168.0.9      2519                  152281
192.168.2.1      11305                 5000597
192.168.0.10     1110                  195896
192.168.0.176    57045                 85028424
192.168.0.7      2343                  133430
192.168.0.4      4906                  2607033
192.168.0.189    1                     48
192.168.0.1      5747                  421809
192.168.0.255    86                    9856
 
For a total of: 12 entries


Any ideas on that?

Best regards
Ben

smicha
Posts: 4
Joined: June 6th, 2015, 10:14 am

Re: IP Accounting

Post by smicha » September 7th, 2015, 2:50 pm

Hi ,

ummeegge wrote:@Michael
I´am not sure what else informations in the above mentioned howto should be delivered ?


The result is what I will have :-)
I will try it on my setup.

As far as I see, the "pmacctd " now capture the traffic on the interface (please correct me if I'am wrong) and I have to see if this "extra" task have an impact of the performance of the system?
The way via iptables looks much more effective, but if system running fine, than it absolute perfect!

Thank you!!!
Michael

ummeegge
Community Developer
Community Developer
Posts: 5001
Joined: October 9th, 2010, 10:00 am

Re: IP Accounting

Post by ummeegge » September 7th, 2015, 3:43 pm

Hi all,
Trikolon wrote:Any ideas on that?

currently not, i tried not that much with pmacctd more reading than experiments but your values shows a big gap. Will check that may also in next days.

smicha wrote:As far as I see, the "pmacctd " now capture the traffic on the interface (please correct me if I'am wrong) and I have to see if this "extra" task have an impact of the performance of the system?

Yes it does. I played yesterday a little bit around with pmacctd on my ALIX board with activated Proxy and URL-Filter (only a few MB´s RAM left) without problems, so i have at this time no hard values how much performance in what needs pmacctd related to IPTables more or less, this theme shurley depends also on the kind of configuration, the 'memory' plugin for example needs other kinds/values of resources than the database plugin.

Are you guys interessted to bring pmacctd into an own development topic cause this infos are very specific and may a little missplaced under this one ?

Greetings,

UE
Image
Image

Trikolon
Community Developer
Community Developer
Posts: 552
Joined: October 16th, 2008, 6:21 am
Location: Erlangen
Contact:

Re: IP Accounting

Post by Trikolon » September 7th, 2015, 8:09 pm

An own topic sounds reasonable.

So I tested it today during the day and basically it shows valid data. E.g.

vnstat: 536MB / 263MB

pmacctd: 551MB / 388MB

I started pmacctd this morning after using ca. 30-50MB which are not accounted. It is very interesting that pmacctd accounts more traffic than vnstat.

Best regards
Ben

Post Reply