Pondering to upgrade HW, now using a Banana Pi R1

General questions.
Saiyato
Posts: 36
Joined: October 9th, 2018, 6:55 pm

Pondering to upgrade HW, now using a Banana Pi R1

Post by Saiyato » October 31st, 2018, 9:05 pm

Good evening all,

TL;DR: see benchmarks

Where the VPN is expected to offer 256 bit encryption, either CBC or GCM and the type used is OpenVPN (=single threaded, I know).

BPI R1
Encryption throughput: ~81Mbps | ~77MiB/s
OpenVPN throughput: ~27Mbps
Speedtest (without VPN): ~55Mbps
Speedtest (with VPN): ~12-20Mbps
Power consumption: 2-4W = 17-35kWh/year = ~4-8 Euro/year (22 cents per kWh)

APU2B4/APU2C4
Encryption throughput:~998Mbps | ~952MiB/s
OpenVPN throughput: ~56Mbps
Speedtest (without VPN): ~90Mbps | ~86MiB/s
Speedtest (with VPN): ~50Mbps | ~48MiB/s
Power consumption: 6-12W = 53-106kWh/year = ~12-23 Euro/year (22 cents per kWh)

Qotom-Q355G4 (i5-5250U)
Encryption throughput: >300Mbps | >286MiB/s
OpenVPN throughput: >300Mbps | >286MiB/s
Speedtest (without VPN): gigabit?
Speedtest (with VPN): ~300Mbps (according to source)
Power consumption: 15-18W = 131-158kWh/year = ~29-38 Euro/year (22 cents per kWh)
But support seems to be very limited, which means updates are few (if any), see last source: no meltdown/spectre updates seem to exist.
Images and LAN port layout: https://imgur.com/a/J70ai

Sources for the Qotom:
https://forum.netgate.com/topic/129326/ ... ireguard/7
https://forum.netgate.com/topic/113483/ ... push-1gb/3
https://www.reddit.com/r/HomeNetworking ... _open_vpn/
https://forum.kuketz-blog.de/viewtopic. ... 2&start=30

/TL;DR

I've been looking for an 'introduce yourself' thread, but couldn't find the section. So here it goes! Hi all, I'm Rachid, a tinkerer from the Netherlands, I have some Linux experience and I'm learning more and more every day. I work at a software company, just not as a developer, so I can read code in some languages. This makes it easier to test stuff and understand what's going on.

I've been using IPFire for some time now (in my home) and decided to upgrade to the newest version. I had to reinstall the SD-card, which was a bit tougher than expected, mostly because I didn't fully understand what was happening.

Should you be interested, I have some additions to the manual, which can be found here

The first steps are quite straight forward:
Download the latest armv5tel IPFire image, unpack the image to a (Micro)SD card.
This is quite easy, you can write the image to the SD card any way you like, some examples are:
  • OSX: Etcher.io
  • Windows: Win32 Disk Imager
  • Linux: Etcher.io
Or xzcat:

Code: Select all

xzcat ipfire-2.21.2gb-ext4.armv5tel-full-core122.img.xz > /dev/{SDCARD}
Where {SDCARD} should be replaced with the device you are seeing as the SD card (dependent on your reader)

After writing the image to the SD card you need to perform two more steps to make it work:
  • Update the uEnv.txt file
  • Write the bootloader to the first sectors of the SD card
The first wasn't too tough, you can either put it in a machine with a gui and edit the file with your favorite editor, since it's a FAT partition. Or you can mount it on the CLI and edit the file there.

The second part gave me some trouble, as I didn't quite fathom what I was doing, eventually, after some help from Arne I understood I was writing the bootloader. At this point you must mount partition 3, in my case sdb3 and find the bin file(directory: usr/share/u-boot/banana_pi).
Then you will copy this file to the root 'device', in my case /dev/sdb (but /dev/mmcblk and /dev/sdc are possible as well; note that you wont be selecting a partition to copy to).

Effectively this translated to the following command for me:

Code: Select all

dd if=u-boot-sunxi-with-spl.bin of=/dev/sdb bs=1K seek=8
This command will copy the contents of bin to the first sectors;
  • BS = block size, the chunks to use when copying
  • seek = the start block, 8 means it will skip the first seven
I don't have the output anymore, but it writes around 500Kb, you can find more info here.

No we can boot the device properly, connect a serial cable to the header and you will see the first boot.

Fritzbox issue
The first problem I then ran into is that IPFire conflicted with my Fritzbox (7390), I was getting weird 'blocked' messages when visiting sites and port 8182 was shown every time. Apparently IPFire using fixed IP on red0 and the Fritzbox running DHCP was conflicting. I read something about the proxy being problematic, but no setting I tried in IPFire solved it, I had to disable static IP in IPFire and just setup the IP in the Fritzbox (infinite lease).

After the last hurdle I restored the backups I made in version 2.19 in my fresh 2.21 instance. I installed the packages I used before and everything was running great.

A few weeks later, I decided I wanted to use a VPN service for my internet traffic. I figured it would make the most sense to do this in IPFire, that way all my devices would automatically go over that VPN. Some research learned that this was easily done with OpenVPN.

OpenVPN tunnel issue
However, I already setup an OpenVPN server instance on the IPFire box, so the tutorials (telling me to use tun0) needed to attention. Ifconfig told me that tun0 was already in use by the server, I recognized the IP-range to be the VPN range. No problem, I'll just use tun1 instead, let's hope it surfaces when I connect... fingers crossed. Luckily it did! So I change the iptables route to:

Code: Select all

iptables -t nat -A POSTROUTING -s xxx.yyy.0.0/16 -o tun1 -j MASQUERADE
Everything in the xxx.yyy-range would be going over the VPN tunnel to the internet. I connected to the VPN service and after a short drop, the internet would go through the tunnel instead. Awesome!

Speed tests
Before the VPN I would get around 40-60 mbps up/down (depending on timing and test)
speedof.me result
speedtest.net result

Just one down side... the VPN is costly in terms of CPU for the BPI R1, when performing speed tests I see 80-100% CPU usage. Yikes! Hence the topic title... I think the hardware is out of its bounds now... The encryption is AES-256 CBC (auth = SHA-512); now I don't know too much about VPN encryption, but from what I've been told, this is sufficient for home use.

The speeds would drop to 12-20 mbps, which is below par in my opinion, I'm willing to sacrifice _some_ speed, but since my line is 100/100 and throughput of the BPI only was around 60-ish... dropping to 20 is a bridge too far.

Verdict
The BPI was a nice solution for simple home use and tinkering, for as long as it took, but I think, with my current requirements I just need more power/throughput. However I still want to use devices that won't bankrupt me in terms of power consumption. So I've been digging and found some options:

1. I'm preaching to choir now, but the IPFire DUO or Eco (Plus) box are supported pieces of hardware (which is a bonus), but also a tad bit pricey. Maybe a bit too much for simple home use.
2. The Qotom product range looks a lot like the DUO/Eco, the have the QOTOM-Q355G4 (Core i5 and AES-NI support), they go for about 300 Euros
3. The Firewall Micro Appliances also go for about 300 Euros, they are running on J1900 procs
4. PC-engines also supplied micro PC's, I've seen them being mentioned on the forum, the seem to be running T40E procs

In terms of power consumption the follow applies (according to the big web)

1. ~6-13 watts
2. ~10-15 watts
3. ~10-18 watts
4. ~6-12 watts (depending on model)

At this point I'm leaning towards the Qotom product, since the i5 part appeals to me, but I can't really argument it. Mostly because I have no benchmarks to show for. Plus the price does seem to be reasonable. Can anyone maybe shed some light on this predicament in terms of performance/price/power consumption?

Many thanks in advance for the insights! :)

Cheers, Rachid
Last edited by Saiyato on November 30th, 2018, 12:56 pm, edited 5 times in total.
Image
Image

cfusco
Posts: 118
Joined: March 23rd, 2015, 4:19 pm

Re: Pondering to upgrade HW, now using a Banana Pi R1

Post by cfusco » November 1st, 2018, 2:24 pm

this is my machine:

Code: Select all

https://www.pcengines.ch/apu2d4.htm
More details about the hardware:

Code: Select all

https://fireinfo.ipfire.org/profile/4722a6dfa05ebffc7a865016b5f8c3a13c89e825
lm sensor output:

Code: Select all

lm_sensors output

 ath10k_hwmon-pci-0100
 Adapter: PCI adapter
 temp1:        +45.0 C

 k10temp-pci-00c3
 Adapter: PCI adapter
 temp1:        +57.2 C  (high = +70.0 C)
                        (crit = +105.0 C, hyst = +104.0 C)

 fam15h_power-pci-00c4
 Adapter: PCI adapter
 power1:        9.59 W  (interval =   0.01 s, crit =   6.00 W)
I use openvpn from the office to my home. This is the CPU load during a time machine backup of my macbook to the nas I have at home:
system.cgi.png
system.cgi3.png
system.cgi2.png
Roadwarrior stats during the same time intervall:
netovpnrw.cgi.png
Overall, I am very happy with the machine. It's cheap, low power and extremely reliable. I had a problem with mSata ssd which got disconnected by the kernel. Probably it's related to the trim operation. It has happened 3 times so far and it has bee 6 months since the last time. Let me know if there is anything else you would like to know.
Image

Saiyato
Posts: 36
Joined: October 9th, 2018, 6:55 pm

Re: Pondering to upgrade HW, now using a Banana Pi R1

Post by Saiyato » November 1st, 2018, 4:36 pm

Thanks for the info, this helps a lot indeed.

Quick question, the VPN you have running is from a macbook to your IPFire instance, so IPFire is the server. Am I right?

Because this seems to be working good for the BPI as well, but if I connect to a VPN from IPFire, i.e. become the client, I see a huge increase in terms of CPU. Because then the IPFire instance would need to encrypt/decrypt all data on red0; which appears to be very costly.

But that's exactly what this study is meant to show, how do other hardware platforms perform on constant VPN's. :)

Also, I can read that the BPI just has a simple switch, meaning during boot all terminals are bridged. Is that the same for the APU, or does that appliance strictly separates the terminals?

Your input is appreciated :)
Image
Image

cfusco
Posts: 118
Joined: March 23rd, 2015, 4:19 pm

Re: Pondering to upgrade HW, now using a Banana Pi R1

Post by cfusco » November 1st, 2018, 6:47 pm

Saiyato wrote:
November 1st, 2018, 4:36 pm
Thanks for the info, this helps a lot indeed.

Quick question, the VPN you have running is from a macbook to your IPFire instance, so IPFire is the server. Am I right?
Yes, my APU2 running IPFIre is the VPN server, and my iPhone, iPad and macbook are the clients.
Because this seems to be working good for the BPI as well, but if I connect to a VPN from IPFire, i.e. become the client, I see a huge increase in terms of CPU. Because then the IPFire instance would need to encrypt/decrypt all data on red0; which appears to be very costly.
I see, no I never tryied to use my IPFIre as client. I should try that some time.
Also, I can read that the BPI just has a simple switch, meaning during boot all terminals are bridged. Is that the same for the APU, or does that appliance strictly separates the terminals?

Your input is appreciated :)
Do you mean the ethernet ports? They are not bridged, they are separate and independent cards (Intel i210AT), and they work pretty well. The gigabit is pretty much ok. https://wiki.ipfire.org/hardware/pcengines/apu2b4
Image

Saiyato
Posts: 36
Joined: October 9th, 2018, 6:55 pm

Re: Pondering to upgrade HW, now using a Banana Pi R1

Post by Saiyato » November 1st, 2018, 9:22 pm

Awesome info! I was leaning towards the Qotom, but the PCEngines option seems very good indeed! :)

I just want to see some benchmarks when using the appliance as a VPN client with high encryption settings. Just curious whether I want too much for my home network, or if I can actually effectively tackle this. >:D

I'm now having trouble getting clamav to work in core 124, maybe that's because the ARM platform isn't the main platform. Oh well, enough for today, thanks for the info so far, it really helps.
Image
Image

Saiyato
Posts: 36
Joined: October 9th, 2018, 6:55 pm

Re: Pondering to upgrade HW, now using a Banana Pi R1

Post by Saiyato » November 6th, 2018, 12:49 pm

Just a quick update, I've noticed that the throughput for the BPI is limited, probably due to insufficient CPU power (or, code is not optimised for ARM arch).

The top processes vary, but I'm seeing a high CPU load when stressing the network, which would indicate the BPI is running on its toes.

Top CPU heavy processes when downloading large files:
ksoftirqd -> ~90-96%
snort -> ~15-20%
guardian -> ~5-10%
kworker/1:x -> ~5-10%

Code: Select all

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
    7 root      20   0     0    0    0 R   95  0.0  43:36.88 ksoftirqd/0
 2732 root      20   0  273m 248m 4528 R   11 24.9  30:09.23 snort
31753 root      20   0     0    0    0 I   10  0.0   0:38.47 kworker/1:0
 3425 root      20   0 59856  19m 2312 S    8  2.0 140:11.73 guardian
32518 root      20   0     0    0    0 I    1  0.0   0:00.60 kworker/0:1
 2091 root      20   0  6528 2816 2448 S    1  0.3  20:07.23 hostapd
Top CPU heavy processes when performing a speedtest (speedof.me/speedtest.net)
snort -> ~80-90%
ksoftirqd -> ~70-80%
guardian -> ~15-20%
kworker/1:x -> ~5-10%

Code: Select all

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2732 root      20   0  273m 248m 4528 R   91 24.9  29:56.01 snort
    7 root      20   0     0    0    0 R   74  0.0  42:44.68 ksoftirqd/0
 3425 root      20   0 59856  19m 2312 S   17  2.0 140:00.26 guardian
31753 root      20   0     0    0    0 I    7  0.0   0:33.80 kworker/1:0
 2091 root      20   0  6528 2816 2448 S    4  0.3  20:05.24 hostapd
This would mean that snort is active when performing speed tests, which would impact the shown speed. Subsequently this means that ksoftirqd would be the bottleneck when downloading large files.

A simple google comes up with this thread about ksoftirqd. Possible causes for high ksoftirqd loads are:

- faulty disk(s)
- insufficient disk space

Since this is probably not the case, as it _only_ happens under heavy load, it must be the third option mentioned (I think):

- multicast isues

Now multicast issues sounds a bit vague, I agree, but it does make sense. The signal from the FritzBox (henceforth known as FB) does include the IPTV VLAN (4), so VLAN 4 (IPTV) and 6 (internet) are connected to red0. I was unable to easily get IPTV working on green0, so I know there are signals coming through (I get a few seconds of video and audio). This does seem to support the thesis that the multicast signal is not beneficial for my network speeds and it supports the theory that softirq is busy and thus ksoftirqd.

I don't know why this behaviour seems to be instigated by downloading large files, nor do I know how to test this with large amounts of small files. But I presume this will only cause snort to be the bottleneck.

Anyone willing to provide their €.02?

Update: added the values from the top command, those seem to be more real-time than the graphs.
Also added the CPU graphs from the test period.
Attachments
load_graph.png
cpu_usage_bpi.png
Image
Image

Saiyato
Posts: 36
Joined: October 9th, 2018, 6:55 pm

Re: Pondering to upgrade HW, now using a Banana Pi R1

Post by Saiyato » November 8th, 2018, 4:11 pm

I've been investigating how to measure VPN throughput and maybe even find some benchmarks for other hardware.

OpenSSL speed
The first option I found was the openSSL speed command, so I decided to run this on the BPI, my laptop (on a virtual machine -> Kali) and see what I could find for the PCEngines APU2B4.

1. The elapsed parameter measures in real time instead of CPU time.

2. The evp parameter doesn't control AES-NI, but calls optimised AES code and CPU paths; see this thread.

3. The third parameter(s) are the ciphers used.

BPI R1 (Banana Pi R1 a.k.a. Lamobo R1)

Code: Select all

[xxx@ipfire openvpn]# openssl speed -elapsed -evp aes-128-cbc aes-256-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256 cbc       9291.82k     9964.35k    10233.60k    10300.07k    10264.58k    10190.85k
aes-128-cbc      10859.26k    12883.07k    13601.11k    13733.55k    13347.50k    13620.57k
Laptop (XPS 13 9360 i7-8550u)

Code: Select all

[xxx@kali openvpn]# openssl speed -elapsed -evp aes-128-cbc aes-256-cbc
type             16 bytes      64 bytes      256 bytes     1024 bytes     8192 bytes    16384 bytes
aes-256 cbc      91668.01k    100368.49k      99107.75k     222482.43k     258138.11k     259205.80k
aes-128-cbc     773949.03k   1322253.97k    1400459.78k    1426772.31k    1401446.40k    1352886.95k
Found values for the APU2B4

Code: Select all

[root@ipfire ~]# openssl speed -elapsed -evp aes-256-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc      61005.48k    94578.77k   115948.37k   122893.31k   124788.74k
I'm not sure how this affects the VPN speed, but I'm presuming, the more higher the number, the better (=more throughput). This seems to prove my theory that the BPI is (by far) not strong enough for my 100/100 line and acceptable speeds over VPN (OVPN AES-256-CBC).

The results are (at best for AES-256-CBC):

Update 21-11-2018 based on the comment of Arne
BPI = 10190.85 = ~9,95 Mbps ~81.53Mbps | ~77.75MiB/s
Laptop = 1352886.95 = ~253.13 Mbps ~10823.1Mbps | 10321.71MiB/s
APU2B4 = 124788.74 = ~121.86 Mbps ~998.31Mbps | 952.06MiB/s

Testing over the openVPN application
The previous test only test en- and decryption, but does not take the application or HMAC into account. The below commands do exactly that, it generates a temporary key and asks VPN to process a lot of packets.

Code: Select all

openvpn --genkey --secret /tmp/secret
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc
BPI R1 (Banana Pi R1 a.k.a. Lamobo R1)

Code: Select all

[xxx@ipfire openvpn]# openvpn --genkey --secret /tmp/secret
[xxx@ipfire openvpn]# time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc
Thu Nov  8 16:23:49 2018 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode


real    1m56.231s
user    1m52.892s
sys     0m0.437s

[xxx@ipfire openvpn]# time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-128-cbc
Thu Nov  8 16:26:07 2018 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode

real    1m45.174s
user    1m42.776s
sys     0m0.852s
Laptop (XPS 13 9360 i7-8550u)

Code: Select all

[xxx@kali openvpn]# openvpn --genkey --secret /tmp/secret
[xxx@kali openvpn]# time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc
Thu Nov  8 16:23:49 2018 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode


real    0m1.101s
user    0m1.084s
sys     0m0.000s

[xxx@kali openvpn]# time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-128-cbc
Thu Nov  8 16:26:07 2018 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode

real    0m1.078s
user    0m1.054s
sys     0m0.004s
Interpreting the results

( 3200 / execution_time_seconds ) = Projected Maximum OpenVPN Performance in Mbps

The magic number of 3200 comes from summing 1 to 20000, multiply by 2 for encrypt and decrypt and by 8 bits/byte and divide by 1,000,000 for a result of Mbps

source: https://www.snbforums.com/threads/openv ... vpn.33416/

So the results are:

BPI = 3200/ 116.231 = ~27.53 Mbps
Laptop = 3200/ 1.101 = ~2906.45 Mbps

This would mean the last method estimates a throughput of around 30 Mbps, while the first test (encryption) would imply 10-ish (at best).

I don't have the numbers for the last test for the APU2B4, but I'd like them ::) if someone is able to provide the numbers, then I'd be a happy camper. This would help me in choosing the correct hardware. My goal is to use the full 100/100 line over VPN (I know the VPN provider is able to provide the necessary speeds); even better would be to be able to handle 500/500 (for the future).

I did find a thread for the APU2, claiming that using the following settings improved their speeds from ~40 Mbps to ~70-90 Mbps.

Code: Select all

fast-io
sndbuf 524288
rcvbuf 524288
I.e. upping the buffer from 256KiB to 512KiB.
This is good news for me, as 70-ish would be very acceptable, they however also seem to advise to use AES-128-CBC instead of AES-256-CBC.

If anyone has any thoughts on this, or maybe correct me when I'm drawing the wrong conclusions. Please feel free; I'm here to learn and maybe help other people decide on hardware as well. :)

Cheers!
Last edited by Saiyato on November 21st, 2018, 3:09 pm, edited 2 times in total.
Image
Image

cfusco
Posts: 118
Joined: March 23rd, 2015, 4:19 pm

Re: Pondering to upgrade HW, now using a Banana Pi R1

Post by cfusco » November 8th, 2018, 7:39 pm

My settings: AES-256-CBC, TLS-Authentification-Key 2048 bit, certificates 4096 bit, no optimization. These are my results. Much slower for the first test compared to what you posted. Do you have any idea why? Also my machine is an APU2d4 so it should be as fast as the one posted in that link. Keep in mind that I did those tests while connected from my office to my home with OpenVPN. I will try again to do it from my lan (in case this is the problem) and I will post later the results. I will try the optimization you suggested (great google-fu by the way), but I will need a couple of days. When I will do that I will post the results. Please let me know if you need some more tests.

Code: Select all

[root@ipfire]# # openssl speed -elapsed -evp aes-128-cbc aes-256-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256 cbc for 3s on 16 size blocks: 1927054 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 64 size blocks: 501227 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 256 size blocks: 127711 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 1024 size blocks: 87762 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 8192 size blocks: 11034 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 16384 size blocks: 5522 aes-256 cbc's in 3.00s
Doing aes-128-cbc for 3s on 16 size blocks: 6257682 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 64 size blocks: 3918896 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 1753547 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 549102 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 73599 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 16384 size blocks: 36850 aes-128-cbc's in 3.00s
OpenSSL 1.1.0i  14 Aug 2018
built on: reproducible build, date unspecified
options:bn(64,64) md2(char) rc4(8x,int) des(int) aes(partial) blowfish(ptr) 
compiler: gcc -DZLIB -DZLIB_SHARED -DDSO_DLFCN -DHAVE_DLFCN_H -DNDEBUG -DOPENSSL_THREADS -DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DPURIFY -DOPENSSLDIR="\"/etc/ssl\"" -DENGINESDIR="\"/usr/lib64/engines-1.1\""  -O2 -pipe -Wall -fexceptions -fPIC -m64 -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -Wa,--noexecstack
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256 cbc      10277.62k    10692.84k    10898.01k    29956.10k    30130.18k    30157.48k
aes-128-cbc      33374.30k    83603.11k   149636.01k   187426.82k   200974.34k   201250.13k

Code: Select all

[root@ipfire]# openvpn --genkey --secret /tmp/secret
[root@ipfire]# time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc
Thu Nov  8 20:26:18 2018 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode

real	0m55.956s
user	0m54.783s
sys	0m0.782s
Image

Saiyato
Posts: 36
Joined: October 9th, 2018, 6:55 pm

Re: Pondering to upgrade HW, now using a Banana Pi R1

Post by Saiyato » November 8th, 2018, 8:35 pm

Awesome, these tests are greatly appreciated! :)

I don't know why your speeds are slower than the ones from the IPFire benchmark. Those tests might be performed with different openVPN versions, but I don't know if that influences speed that much. The fact that you were connected can impact its performance to en- and decrypt yes, let's hope that is the reason ;)

Your speed was around 30 Mbps and the benchmark by IPFire was around 120 Mbps, which is quite a big difference. The openVPN client speed is pretty acceptable though, ~56 seconds equals around 57 Mbps. Also that number too might be better when not connected over a VPN.
Image
Image

cfusco
Posts: 118
Joined: March 23rd, 2015, 4:19 pm

Re: Pondering to upgrade HW, now using a Banana Pi R1

Post by cfusco » November 11th, 2018, 7:52 pm

Saiyato wrote:
November 8th, 2018, 8:35 pm
Awesome, these tests are greatly appreciated! :)

I don't know why your speeds are slower than the ones from the IPFire benchmark. Those tests might be performed with different openVPN versions, but I don't know if that influences speed that much. The fact that you were connected can impact its performance to en- and decrypt yes, let's hope that is the reason ;)

Your speed was around 30 Mbps and the benchmark by IPFire was around 120 Mbps, which is quite a big difference. The openVPN client speed is pretty acceptable though, ~56 seconds equals around 57 Mbps. Also that number too might be better when not connected over a VPN.
I repeated the tests from inside the lan, connecting to the IPFire machine trough the serial port, with the OpenVPN server off, and I got exactly the same results.

I guess, this is how it works. I hope it helps you anyway. Let me know if there is anything else I can do.

Code: Select all

type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256 cbc      10375.45k    10851.31k    11000.49k    29686.10k    30001.83k    30010.03k
aes-128-cbc      33875.95k    83323.75k   150267.14k   186448.21k   195376.47k   199376.90k

Code: Select all

real    0m56.555s
user    0m56.074s
sys     0m0.123s
Image

User avatar
Arne.F
Core Developer
Core Developer
Posts: 8086
Joined: May 7th, 2006, 8:57 am
Location: BS <-> NDH
Contact:

Re: Pondering to upgrade HW, now using a Banana Pi R1

Post by Arne.F » November 12th, 2018, 3:48 pm

BPI = 10190.85 = ~9,95 Mbps
This is wrong. The output of openssl speed is usually in kBytes not it kbit. So
10190.85 = ~79,6 Mbps
But this is only the CPU time for the AES 256 encryption. OpenVPN do many more things (Split packets if the payload are too large, do integrity check usw...)
Your speed was around 30 Mbps and the benchmark by IPFire was around 120 Mbps, which is quite a big difference.
Here is something wrong. The APU2 should get 120MByte/s with AES-NI and 30 MByte/s without so it looks that AES-NI was not used...
Check the -evp parameter:
./openssl speed -evp AES256
Arne

Support the project on the donation!

Image

Image

Image
PS: I will not answer support questions via email and ignore IPFire related messages on my non IPFire.org mail addresses.

Saiyato
Posts: 36
Joined: October 9th, 2018, 6:55 pm

Re: Pondering to upgrade HW, now using a Banana Pi R1

Post by Saiyato » November 13th, 2018, 12:21 pm

Arne.F wrote:
November 12th, 2018, 3:48 pm
BPI = 10190.85 = ~9,95 Mbps
This is wrong. The output of openssl speed is usually in kBytes not it kbit. So
10190.85 = ~79,6 Mbps
But this is only the CPU time for the AES 256 encryption. OpenVPN do many more things (Split packets if the payload are too large, do integrity check usw...)
Thanks for the clarification, I did misinterpret the values, so the Mbps value is calculated by multiplying by 8 (bits to bytes) and dividing by 1000 (kilo to mega). I believe the division by 1024 is reserved for binary values (i.e. kibibytes and mebibytes; according to the ISO 80000-13). Here to learn :) I will update the values in the previous posts.

Is there a better way to determine VPN throughput for a machine, as you are right, a VPN packet transfer comprises of more than just encryption. Should the second test, which only gives you a number of seconds needed to transfer packets over the application, still not utilizing the network layer I suppose, but I can't be sure.
Arne.F wrote:
November 12th, 2018, 3:48 pm
Your speed was around 30 Mbps and the benchmark by IPFire was around 120 Mbps, which is quite a big difference.
Here is something wrong. The APU2 should get 120MByte/s with AES-NI and 30 MByte/s without so it looks that AES-NI was not used...
Check the -evp parameter:
./openssl speed -evp AES256
I've read that the AES-NI instruction set was introduced to improve encryption speeds on modern processors; but do you actually have to configure anything to be using it in your VPN connection? Or is it auto-selected?
Image
Image

Saiyato
Posts: 36
Joined: October 9th, 2018, 6:55 pm

Re: Pondering to upgrade HW, now using a Banana Pi R1

Post by Saiyato » November 13th, 2018, 12:23 pm

Also on a different note, I did notice improvement in speed after disabling QoS.

I know that this would impact performance, as packets need to be analysed, so I need to defend the real effect against the claim that it affects my network possitively when turned off. Because, if it didn't add anything, it wouldn't be there. ;)
Image
Image

cfusco
Posts: 118
Joined: March 23rd, 2015, 4:19 pm

Re: Pondering to upgrade HW, now using a Banana Pi R1

Post by cfusco » November 13th, 2018, 2:07 pm

Arne.F wrote:
November 12th, 2018, 3:48 pm
Here is something wrong. The APU2 should get 120MByte/s with AES-NI and 30 MByte/s without so it looks that AES-NI was not used...
Check the -evp parameter:
./openssl speed -evp AES256
Indeed:

Code: Select all

[root@ipfire cfusco]# openssl speed -evp AES256 -elapsed
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-cbc for 3s on 16 size blocks: 6035698 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 64 size blocks: 3440620 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 1388799 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 408754 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 53767 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 16384 size blocks: 26975 aes-256-cbc's in 3.00s
OpenSSL 1.1.0i  14 Aug 2018
built on: reproducible build, date unspecified
options:bn(64,64) md2(char) rc4(8x,int) des(int) aes(partial) blowfish(ptr) 
compiler: gcc -DZLIB -DZLIB_SHARED -DDSO_DLFCN -DHAVE_DLFCN_H -DNDEBUG -DOPENSSL_THREADS -DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DPURIFY -DOPENSSLDIR="\"/etc/ssl\"" -DENGINESDIR="\"/usr/lib64/engines-1.1\""  -O2 -pipe -Wall -fexceptions -fPIC -m64 -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -Wa,--noexecstack
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-cbc      32190.39k    73399.89k   118510.85k   139521.37k   146819.75k   147319.47k
Image

Saiyato
Posts: 36
Joined: October 9th, 2018, 6:55 pm

Re: Pondering to upgrade HW, now using a Banana Pi R1

Post by Saiyato » November 21st, 2018, 3:34 pm

Ok, so I updated the values for the openssl speedtest, this would only apply to encryption. Thanks Arne for pointing this out, these values are actually promising.

What remains to be validated is an actual speedtest of the throughput of an APU2 (of a client behind the APU2). That way I can make an educated decision, which I'm almost ready to make based on the results so far.

Unfortunately I don't know how to create tables - if it's even possible - in phpBB, so here's the sum up:

Where the VPN is expected to offer 256 bit encryption, either CBC or GCM and the type used is OpenVPN (=single threaded, I know).

BPI R1
Encryption throughput: ~81Mbps | ~77MiB/s
OpenVPN throughput: ~27Mbps
Speedtest (without VPN): ~55Mbps
Speedtest (with VPN): ~12-20Mbps
Power consumption: 2-4W = 17-35kWh/year = ~4-8 Euro/year (22 cents per kWh)

APU2B4/APU2C4
Encryption throughput:~998Mbps | ~952MiB/s
OpenVPN throughput: ~56Mbps
Speedtest (without VPN): ~90Mbps | ~86MiB/s
Speedtest (with VPN): ~50Mbps | ~48MiB/s
Power consumption: 6-12W = 53-106kWh/year = ~12-23 Euro/year (22 cents per kWh)

Qotom-Q355G4 (i5-5250U)
Encryption throughput: >300Mbps | >286MiB/s
OpenVPN throughput: >300Mbps | >286MiB/s
Speedtest (without VPN): gigabit?
Speedtest (with VPN): ~300Mbps (according to source)
Power consumption: 15-18W = 131-158kWh/year = ~29-38 Euro/year (22 cents per kWh)
But support seems to be very limited, which means updates are few (if any), see last source: no meltdown/spectre updates seem to exist.
Images and LAN port layout: https://imgur.com/a/J70ai

Sources for the Qotom:
https://forum.netgate.com/topic/129326/ ... ireguard/7
https://forum.netgate.com/topic/113483/ ... push-1gb/3
https://www.reddit.com/r/HomeNetworking ... _open_vpn/
https://forum.kuketz-blog.de/viewtopic. ... 2&start=30

Note: these numbers are compiled from various sources

@cfusco: I don't know what your connection is (mine is 100/100), but if yours is comparable, can you perform a speedtest, just to check the throughput? Assuming you have some addons enabled (proxy/clamav/IDS). :)

Also, I might be better off using the box as an IPSEC client, the throughput for IPSEC should be way better than OVPN.
Last edited by Saiyato on November 30th, 2018, 12:56 pm, edited 3 times in total.
Image
Image

Post Reply