Benchmarking on the Helios4

— 3390 words — 17 min

Table of Contents
  1. Running OMV with EXT4 on LVM on RAID 1
    1. SMB Performance
    2. SSHFS Performance
    3. Test on the Helios4 locally
    4. Test with Bonnie++
  2. EXT4 on RAID 1
  3. EXT4 on LUKS on RAID 1
    1. Unaligned LUKS Sector Size and EXT4 Block Size
    2. Aligned LUKS Sector Size and EXT4 Block Size
    3. Encryption Offloaded to the CESA Unit
  4. Table of Results with Bonnie++
  5. Out of Scope

If you don’t know about the Helios4, you can read about it here.

First of all, during benchmarking a lot of things can go wrong (I really recommend watching these 5 mintues of a lightning talk about benchmarking). Hence, take the results with a grain of salt. While I’m not doing any fancy calculations here, I am still not an expert. During the tests the Helios was always idle, or, at least, I did not do anything else. For instance, it could have happened that a cronjob was run during a benchmark, but I don’t think that this should have a high impact. Further, I was running the tests (with bonnie) multiple times.

The Helios4 has support for four drives but I use it currently with 2x4TB drives.

At the end of this post, you’ll find a table summarizing the results.

Running OMV with EXT4 on LVM on RAID 1🔗

For the first few tests, I actually installed OpenMediaVault (Helios4 Wiki) as I never used it before and wanted to try it out. I literally followed the instructions on the Official Kobol wiki which meant

I installed and activated SMB as well and after I was able to connect to it finally (permission things…), I used dd to push some data over the network to get a feeling for the speed.

It’s important to note that I did not change any configration or tried to make any optimizations. This was run with the default settings.

SMB Performance🔗

As you can see in the command, I wanted to copy 10 GB originally, but aborted at 3.1 GB, which took 160s at a rate of about 20 MB/s. Using the same command to read the written data again resulted in about the same speed.

w=20MB/s, r=20MB/s

╭─max@host /run/user/1000/gvfs/smb-share:server=helios4,share=data/test
╰─$ dd if=/dev/zero of=test.img bs=1M count=10000 status=progress
3283091456 bytes (3,3 GB, 3,1 GiB) copied, 160 s, 20,5 MB/s^C
3132+0 records in
3132+0 records out
3284140032 bytes (3,3 GB, 3,1 GiB) copied, 160,051 s, 20,5 MB/s

╭─max@host /run/user/1000/gvfs/smb-share:server=helios4,share=data/test
╰─$ dd if=test.img of=/dev/null bs=1M status=progress
3275751424 bytes (3,3 GB, 3,1 GiB) copied, 166 s, 19,7 MB/s
3132+0 records in
3132+0 records out
3284140032 bytes (3,3 GB, 3,1 GiB) copied, 166,45 s, 19,7 MB/s

Copying a 15 GB video file from my computer onto the Helios4 took almost exactly 13 minutes, which is pretty much the rate shown by dd: 15.000 MB / (13*60s) = 19.2 MB/s.

I was a little confused about this speed. The Helios4 has a gigabit ethernet port, my computer and all other involved components as well. Theoretically, this means a throughput of about 125 MB/s. I did not expect to reach this speed in practice, but only 20 MB/s seemed to be a bit low?

SSHFS Performance🔗

Next, I tried SSHFS. You don’t need anything special, the only requirement is the ssh daemon and ssh is running on all my servers, anyway.

Again, I did not change any configurations or made any optimizations. This was run with the default settings when you mount a folder with SSHFS.

The usage of sshfs is sshfs [user@]host:[dir] mountpoint [options]. The directoy on the Helios4 is mounted to ~/test locally, which you can achieve with

sshfs helios4:/srv/dev-disk-by-uuid-dfb36876-37ed-4860-9a86-ecf608a3d986/data/test ~/test/

I used the same dd command but with count=1000 so that 1 GB is transferred because I did not want to wait for 10 GB.

w=29MB/s, r=31MB/s

╭─max@host ~/test
╰─$ dd if=/dev/zero of=test.img bs=1M count=1000 status=progress
1044381696 bytes (1,0 GB, 996 MiB) copied, 35 s, 29,6 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1,0 GB, 1000 MiB) copied, 35,5097 s, 29,5 MB/s
╭─max@host ~/test
╰─$ dd if=test.img of=/dev/null bs=1M status=progress
1023410176 bytes (1,0 GB, 976 MiB) copied, 33 s, 31,0 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1,0 GB, 1000 MiB) copied, 33,8286 s, 31,0 MB/s

I transmitted the same 15 GB video file as above as well, which took about 8 minutes: 15.000 MB / (8*60s) = 31 MB/s. That’s definitely better than 19.2 MB/s.

So roughly, SSHFS seems to be about 10 MB/s faster than SMB. Interesting. This actually spiked my interest to actually do some benchmarking on the Helios4 itself. Up until now, I only wanted to get a feel for transmission speeds, which I could expect.

Test on the Helios4 locally🔗

I connected to the Helios4 with ssh and run basically the same dd test as before, this time with a 10 GB file (count=10000). This shows the writing and reading speeds to the disk if no network is involved.

w=158MB/s, r=212Mb/s

╭─max@helios4 /srv/dev-disk-by-uuid-dfb36876-37ed-4860-9a86-ecf608a3d986/data/test
╰─$ dd if=/dev/zero of=test.img bs=1M count=10000 status=progress
5071962112 bytes (5.1 GB, 4.7 GiB) copied, 32 s, 158 MB/s^C
4944+0 records in
4944+0 records out
5184159744 bytes (5.2 GB, 4.8 GiB) copied, 32.7079 s, 158 MB/s

╭─max@helios4 /srv/dev-disk-by-uuid-dfb36876-37ed-4860-9a86-ecf608a3d986/data/test
╰─$ dd if=test.img of=/dev/null bs=1M status=progress
5084545024 bytes (5.1 GB, 4.7 GiB) copied, 24 s, 212 MB/s
4944+0 records in
4944+0 records out
5184159744 bytes (5.2 GB, 4.8 GiB) copied, 24.5122 s, 211 MB/s

Great! Writing happens with about 150 MB/s and reading at about 210 MB/s. This shows that the disk I/O is definitely not the limiting factor. Now, let’s do a test with a proper tool: bonnie++

Test with Bonnie++🔗

With bonnie++ (website) you can test the performance of your filesystem and hard drives. The manpage tells you everything you need to know, but let’s look at the used options quickly:

I really like the result:

w=146MB/s, rw=78MB/s, r=177MB/s

/usr/sbin/bonnie++ -d /srv/dev-disk-by-uuid-dfb36876-37ed-4860-9a86-ecf608a3d986/data/test/perform  -c 1 -s 4024 -n 1 -f -b
Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
helios4       4024M            146m  48 78.1m  34            177m  45 147.1   5
Latency                         185ms     383ms               104ms     435ms
Version  1.98       ------Sequential Create------ --------Random Create--------
helios4             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  1 1672561147   0 +++++ +++ -1046580777   0 -1081091648   0 +++++ +++ -898267616   0
Latency               177ms     276us   90292us   59943us      24us     103ms

EXT4 on RAID 1🔗

Next, I removed OpenMediaVault and built my own setup. First, I created an EXT4 filesytem directly on the RAID 1 to compare the result to the previous test. The difference is that no LVM is used anymore. Both tests were run with no encryption in use.

The RAID 1 can be accessed with the device /dev/md0.

The following command creates the EXT4 filesystem. By default, a block size of 4096 is used. This is not relevant now, but it will be later when a LUKS container is used for encryption.

sudo mkfs.ext4 /dev/md0

Mount and run the test.

sudo mount /dev/mapper/cryptroot /mnt/md0
/usr/sbin/bonnie++ -d /mnt/md0/ -c 1 -s 4024 -n 1 -f -b

Result: w=160MB/s, rw=96MB/s, r=160MB/s

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
helios4       4024M            160m  54 96.6m  39            160m  37 145.0   4
Latency                         170ms     287ms               194ms     471ms
Version  1.98       ------Sequential Create------ --------Random Create--------
helios4             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  1 -569212263   0 +++++ +++ 1545298178   0 1958692144   0 +++++ +++ 1912692157   0
Latency               199ms     264us     116ms     119ms      40us   91260us

Second run: w=164MB/s, rw=95MB/s, r=167MB/s

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
helios4       4024M            164m  53 95.9m  38            167m  39 146.9   4
Latency                         176ms     517ms             95391us     400ms
Version  1.98       ------Sequential Create------ --------Random Create--------
helios4             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  1 -222359118   0 +++++ +++ 1549587127   0 -1213966720   0 +++++ +++ -2058643209   0
Latency               194ms     268us     100ms     105ms      17us   89368us

Interesting! While reading seems to be a bit slower, the writing and rw value are slightly higher than on the setup with OVM.

EXT4 on LUKS on RAID 1🔗

Instead of creating the EXT4 filesystem directly on the RAID 1 device, we will use a LUKS container to provide encryption and create an EXT4 filesystem on top of it.

It’s important to note that these results cannot be compared to the previous speed measurements because no encryption was used there. As you will see, the additional computing power for the encryption takes its performance toll.

First, let’s briefly look at the two 4 TB drives. Both, /dev/sda and /dev/sdb report a physical sector size of 4096 bytes and a logical size of 512 bytes. Hence, it should be ensured that the LUKS container and the filesystem use a block (sector) size of 4096 to use the drive efficiently.

helios4:~:% sudo hdparm -I /dev/sda | grep 'Sector size'
Logical  Sector size:                   512 bytes
Physical Sector size:                  4096 bytes
helios4:~:% sudo hdparm -I /dev/sdb | grep 'Sector size'
Logical  Sector size:                   512 bytes
Physical Sector size:                  4096 bytes

Unaligned LUKS Sector Size and EXT4 Block Size🔗

The LUKS container was created with default options:

sudo cryptsetup luksFormat /dev/md0

And by default, LUKS uses a sector size of 512 bytes. Information about a LUKS container can be read with the cryptsetup command as follows.

helios4:md0:% sudo cryptsetup status /dev/mapper/cryptroot
/dev/mapper/cryptroot is active and is in use.
  type:    LUKS2
  cipher:  aes-xts-plain64
  keysize: 512 bits
  key location: keyring
  device:  /dev/md0
  sector size:  512
  offset:  32768 sectors
  size:    7813740160 sectors
  mode:    read/write

The LUKS container can be opened with sudo cryptsetup open /dev/md0 cryptroot.

Thereafter, the EXT4 filesystem was created with default options, too:

sudo mkfs.ext4 /dev/mapper/cryptroot

The default block size is 4096 as can be seen with command dumpe2fs:

helios4:md0:% sudo dumpe2fs /dev/mapper/cryptroot | grep 'Block size'
dumpe2fs 1.44.5 (15-Dec-2018)
Block size:               4096

The sector size of the LUKS container and the block size of the filesystem are not identical. This is not recommended and will likely result in a performance loss. But I wanted to know how big the difference between aligned and unaligned sector/blocksize actually is.

A friendly reminder about the speed with an unencrypted EXT4 filesystem on the RAID 1:

w=164MB/s, rw=95MB/s, r=167MB/s

Bonnie++ is run with /usr/sbin/bonnie++ -d /mnt/md0/ -c 1 -s 4024 -n 1 -f -b.

Result: w64MB/s, rw=34MB/s, r=53MB/s

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
helios4       4024M           64.1m  20 34.1m  13           53.8m  12 150.5   5
Latency                        1146ms    1184ms             73276us     361ms
Version  1.98       ------Sequential Create------ --------Random Create--------
helios4             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  1 -1499258492   0 +++++ +++ 1469852849   0 162904477   0 +++++ +++ -1983072567   0
Latency               193ms     281us     171ms     167ms      20us     210ms

Second run: w=64MB/s, rw=30MB/s, r=53MB/s

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
helios4       4024M           64.2m  20 30.6m  12           53.1m  12 147.4   4
Latency                        1044ms     922ms               155ms     312ms
Version  1.98       ------Sequential Create------ --------Random Create--------
helios4             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  1 693104966   0 +++++ +++ -1357348910   0 1029287914   0 +++++ +++ 1121994707   0
Latency               230ms     270us     108ms     162ms      39us     142ms

I did a few more runs but they all yielded the same results. As you can see, the speed drops significantly.

Aligned LUKS Sector Size and EXT4 Block Size🔗

The following setup uses the correct sector size of 4096 for the LUKS container, which slightly improves the benchmark results.

The commands to create the setup:

sudo cryptsetup luksFormat --sector-size 4096 /dev/md0
sudo cryptsetup open /dev/md0 cryptroot

Now, the sector size is 4096:

helios4:~:% sudo cryptsetup status /dev/mapper/cryptroot
/dev/mapper/cryptroot is active.
  type:    LUKS2
  cipher:  aes-xts-plain64
  keysize: 512 bits
  key location: keyring
  device:  /dev/md0
  sector size:  4096
  offset:  32768 sectors
  size:    7813740160 sectors
  mode:    read/write

Create the EXT4 filesystem.

sudo mkfs.ext4 /dev/mapper/cryptroot
sudo dumpe2fs /dev/mapper/cryptroot | grep 'Block size'

The block size of EXT4 is again 4096 (as the last time).

sudo mount /dev/mapper/cryptroot /mnt/md0
/usr/sbin/bonnie++ -d /mnt/md0/ -c 1 -s 4024 -n 1 -f -b

Running the test with bonnie++ returns the following result, which is indeed a few megabytes better than previously. In my opinion, the encryption is worth it and I’ll happily take the performance trade-off.

w=73MB/s, rw=37MB/s, r=59MB/s

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
helios4       4024M           73.3m  23 37.4m  14           59.4m  14 143.5   4
Latency                         728ms     748ms             97800us     387ms
Version  1.98       ------Sequential Create------ --------Random Create--------
helios4             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  1 1961043105   0 +++++ +++ -1124638686   0 -110386393   0 +++++ +++ 1338972754   0
Latency               244ms     288us     148ms     210ms      25us     186ms

Second go: w=73MB/s, rw=32MB/s, r=59MB/s

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
helios4       4024M           73.4m  23 32.2m  13           59.5m  14 152.3   3
Latency                         694ms     618ms             78881us     397ms
Version  1.98       ------Sequential Create------ --------Random Create--------
helios4             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  1 289626538   0 +++++ +++ -1096656038   0 1782974649   0 +++++ +++ 665618027   0
Latency               173ms     260us     110ms     170ms      24us     102ms

Encryption Offloaded to the CESA Unit🔗

Then I remembered that the Helios4 has a CESA unit (Cryptographic Engines and Security Accelerator unit), which can be used to offload encryption. I thought that using the CESA unit will improve the benchmark results further, but the results were quite disappointing.

While creating the LUKS container, the cipher aec-cbc-essiv:sha256 must be specified to use the unit. This can be achieved with the following command. We keep the correct sector size, too.

sudo cryptsetup -c aes-cbc-essiv:sha256 luksFormat --sector-size 4096 /dev/md0

Open the container.

sudo cryptsetup open /dev/md0 cryptroot

Verify cipher and sector size.

helios4:~:% sudo cryptsetup status /dev/mapper/cryptroot
/dev/mapper/cryptroot is active.
  type:    LUKS2
  cipher:  aes-cbc-essiv:sha256
  keysize: 256 bits
  key location: keyring
  device:  /dev/md0
  sector size:  4096
  offset:  32768 sectors
  size:    7813740160 sectors
  mode:    read/write

Create filesystem.

sudo mkfs.ext4 /dev/mapper/cryptroot

Check block size.

helios4:~:% sudo dumpe2fs /dev/mapper/cryptroot | grep 'Block size'
dumpe2fs 1.44.5 (15-Dec-2018)
Block size:               4096

Let’s mount the partition and start the test!

sudo mount /dev/mapper/cryptroot /mnt/md0
/usr/sbin/bonnie++ -d /mnt/md0/ -c 1 -s 4024 -n 1 -f -b

w=48MB/s, rw=29MB/s, r=60MB/s

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
helios4       4024M           48.8m  16 29.9m  12           60.7m  14 145.8   5
Latency                        1150ms    1852ms             58338us     347ms
Version  1.98       ------Sequential Create------ --------Random Create--------
helios4             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  1 -2068442525   0 +++++ +++ 1658309631   0 -747844300   0 +++++ +++ 614890684   0
Latency               171ms     262us     155ms     173ms      27us     139ms

second run: w=48MB/s, rw=27MB/s, r=59MB/s

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
helios4       4024M           48.6m  16 27.5m  11           59.4m  13 147.9   5
Latency                        1193ms    1148ms             93477us     356ms
Version  1.98       ------Sequential Create------ --------Random Create--------
helios4             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  1 -932939759   0 +++++ +++ 827095864   0 1905523812   0 +++++ +++ -1495813491   0
Latency               181ms     270us     214ms     238ms      21us     140ms

Well, the result is not very convincing. With the encryption offloaded to the CESA unit, the results are worse than before.

Table of Results with Bonnie++🔗

All setups use a RAID 1 and the EXT4 filesystem with a block size of 4096.

The exception is the EXT4 filesystem created by OMV. I’m not sure which block size was used as I created it with the OVM web GUI and I didn’t check it in the terminal. However, I assume that the default block size of 4096 was used.

Nonetheless, if another block size than 4096 was used, this might explain the worse performance of the setup of OMV with EXT4 on LVM on RAID 1 versus the plain EXT4 on the RAID 1; but perhaps this is just the indirection layer of LVM?

SetupLUKS sector sizeencryptionwrite MB/sread-write MB/sread MB/s
OMV - EXT4 on LVMno luksnone14678177
EXT4no luksnone16096160
EXT4 on LUKS512aes-xts-plain64643453
EXT4 on LUKS4096aes-xts-plain64733759
EXT4 on LUKS (CESA)4096aes-cbc-essiv:sha256482960

Regarding the CESA unit, there exists a HTTPS Benchmark in the official wiki.

Out of Scope🔗

There may be many aspects, which affect the measurement itself as well as how you evaluate the results. For instance, I did not look at the energy usage or CPU usage while the HTTPS benchmark takes the CPU usage into account.

Personally, I settled on the RAID 1 with an EXT4 filesystem on a LUKS container (with the correct sector size, of course). This provides encryption, which is a must-have, and reasonable speed. Furthermore, no other software but ssh is required, which is already running anyway.


Articles from blogs I follow around the net

Status update, May 2024

Hi! Sadly, I need to start this status update with bad news: SourceHut has decided to terminate my contract. At this time, I’m still in the process of figuring out what I’ll do next. I’ve marked some SourceHut-specific projects as unmaintained, such as sr.ht-…

via emersion May 21, 2024

How and why to make a /now page on your site

Background I used to wonder what my friend Benny Lewis was doing. He has a website and social media accounts, but neither gave an overview of what he’s doing now. Then I realized some people might wonder the same about me. So in 2015, I made a /now page on my…

via Derek Sivers blog May 18, 2024

Email DNS Records Cheatsheet

A quick summary of the SMTP related DNS records together with brief examples.

via Signs of Triviality April 12, 2024

Generated by openring