jgillman's Liquid Web Update Unofficial tips, tricks, and happenings from a Liquid Web sales engineer

25Jan/130

Rough Gluster Benchmark

So, I've decided to benchmark GlusterFS.

The volume configuration was a distributed-replicated one in a 2x2 configuration of 1GB Storm Instances. Each instance was connected to a 150GB HPBS volume formatted XFS.

I then created 2 more 1GB Storm instances for the "clients". From there, I concurrently ran iozone with the same parameters as my HPBS benchmark.

Instances all used the latest version of Debian Squeeze. Compile target was the same as well as the HPBS test.

I'll just toss the final numbers down. Again - both of these nodes were running iozone concurrently.

IOPS Test

Client 1

Children see throughput for 32 initial writers = 4446.52 ops/sec
Children see throughput for 32 readers = 11907.26 ops/sec
Children see throughput for 32 random readers = 1142.15 ops/sec
Children see throughput for 32 random writers = 373.25 ops/sec

Client 2

Children see throughput for 32 initial writers = 4507.93 ops/sec
Children see throughput for 32 readers = 11926.79 ops/sec
Children see throughput for 32 random readers = 1212.09 ops/sec
Children see throughput for 32 random writers = 407.77 ops/sec

Comparison

SSD Disk Test

Throughput Test

Client 1

Children see throughput for 32 initial writers = 19018.32 KB/sec
Children see throughput for 32 readers = 45684.51 KB/sec
Children see throughput for 32 random readers = 4636.80 KB/sec
Children see throughput for 32 random writers = 1550.62 KB/sec

Client 2

Children see throughput for 32 initial writers = 19118.81 KB/sec
Children see throughput for 32 readers = 45607.64 KB/sec
Children see throughput for 32 random readers = 4797.18 KB/sec
Children see throughput for 32 random writers = 1613.63 KB/sec

Comparison

SSD Throughput Test

26Nov/120

My Finalized HPBS Benchmark

Well, the numbers are in. During the weekend, the iozone IOPS test completed for HPBS. In addition, I also ran a throughput test using iozone as well. Instead of updating the previous article, I figure I would just write a fresh one.

As a recap, here is the test environment that I used:

AMD FX-6100 Proc (3314 Mhz) Storm Server
3831MB RAM
Debian 6.0.3 x64 upgraded to the latest stable packages as of 21 November 12 (brings it up to Debian 6.0.6)
150GB HPBS volume formatted ext4
iozone 3_414 compiled with linux-AMD64 as the target

For the sake of simplicity, I passed the exact same parameters as the Storm SSD tests:

One thing I would like to point out is that the Amazon numbers are NOT for their Elastic Block Storage product (EBS), but rather for whatever methodology they use for their instance storage. For Liquid Web, our Storm Servers utilize local storage. One customer I have talked to has indicated that he believes the EBS numbers will fare even lower than the numbers we have when we tested against Amazon in the tables above.

So on to the numbers! Our first specimen here is the finalized output of the first test that I had run: I/O

Iozone: Performance Test of File I/O
Version $Revision: 3.414 $
Compiled for 64 bit mode.
Build: linux-AMD64

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
Vangel Bojaxhi, Ben England.

Run began: Wed Nov 21 10:52:22 2012

OPS Mode. Output is in operations per second.
Include fsync in write timing
No retest option selected
Record Size 4 KB
File size set to 4194304 KB
Command line used: /root/iozone -l 32 -O -i 0 -i 1 -i 2 -e -+n -r 4K -s 4G
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Min process = 32
Max process = 32
Throughput test with 32 processes
Each process writes a 4194304 Kbyte file in 4 Kbyte records

Children see throughput for 32 initial writers = 10377.15 ops/sec
Parent sees throughput for 32 initial writers = 9485.68 ops/sec
Min throughput per process = 305.25 ops/sec
Max throughput per process = 348.35 ops/sec
Avg throughput per process = 324.29 ops/sec
Min xfer = 916573.00 ops

Children see throughput for 32 readers = 12519.84 ops/sec
Parent sees throughput for 32 readers = 12519.41 ops/sec
Min throughput per process = 363.21 ops/sec
Max throughput per process = 412.03 ops/sec
Avg throughput per process = 391.24 ops/sec
Min xfer = 924357.00 ops

Children see throughput for 32 random readers = 349.18 ops/sec
Parent sees throughput for 32 random readers = 349.18 ops/sec
Min throughput per process = 10.22 ops/sec
Max throughput per process = 11.50 ops/sec
Avg throughput per process = 10.91 ops/sec
Min xfer = 931745.00 ops

Children see throughput for 32 random writers = 1026.03 ops/sec
Parent sees throughput for 32 random writers = 892.02 ops/sec
Min throughput per process = 28.77 ops/sec
Max throughput per process = 34.68 ops/sec
Avg throughput per process = 32.06 ops/sec
Min xfer = 875945.00 ops

iozone test complete.

I'm honestly not sure of the differences between "Children see throughput" and "Parent Sees throughput" (and Google wasn't helping), so for the case of this article, I'll go with the average of both of these numbers. Going with this, we're looking at the following:

4k Sequential Write: 9,931.42 ops/sec
4k Sequential Read: 12,519.63 ops/sec
4k Random Read: 349.18 ops/sec
4k Random Write: 959.03 ops/sec <-- !!

As you'll see, all tests showed higher performance than the Amazon instance previously tested (which was not done by myself). The real interesting thing though is how the random write numbers just blow away not only the Amazon m1.large instance, but the Storm 8GB instance as well. I believe this is related to how HPBS is setup, of which I'm not 100% technically versed at this time. Still, quite interesting given that conventional wisdom would indicate reads would be faster - especially when they aren't sequential.

Let's move on to the throughput test:

Iozone: Performance Test of File I/O
Version $Revision: 3.414 $
Compiled for 64 bit mode.
Build: linux-AMD64

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
Vangel Bojaxhi, Ben England.

Run began: Fri Nov 23 20:25:07 2012

Include fsync in write timing
No retest option selected
Record Size 4 KB
File size set to 2097152 KB
Command line used: /root/iozone -l 32 -i 0 -i 1 -i 2 -e -+n -r 4K -s 2G
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Min process = 32
Max process = 32
Throughput test with 32 processes
Each process writes a 2097152 Kbyte file in 4 Kbyte records

Children see throughput for 32 initial writers = 38776.21 KB/sec
Parent sees throughput for 32 initial writers = 35388.11 KB/sec
Min throughput per process = 1149.08 KB/sec
Max throughput per process = 1295.61 KB/sec
Avg throughput per process = 1211.76 KB/sec
Min xfer = 1854184.00 KB

Children see throughput for 32 readers = 41301.22 KB/sec
Parent sees throughput for 32 readers = 41298.75 KB/sec
Min throughput per process = 807.03 KB/sec
Max throughput per process = 1489.52 KB/sec
Avg throughput per process = 1290.66 KB/sec
Min xfer = 1136272.00 KB

Children see throughput for 32 random readers = 2552.88 KB/sec
Parent sees throughput for 32 random readers = 2552.88 KB/sec
Min throughput per process = 74.07 KB/sec
Max throughput per process = 82.81 KB/sec
Avg throughput per process = 79.78 KB/sec
Min xfer = 1875968.00 KB

Children see throughput for 32 random writers = 4564.77 KB/sec
Parent sees throughput for 32 random writers = 3715.49 KB/sec
Min throughput per process = 122.68 KB/sec
Max throughput per process = 165.65 KB/sec
Avg throughput per process = 142.65 KB/sec
Min xfer = 1549772.00 KB

iozone test complete.

Using the methodology as presented above:
4k Sequential Write: 37.08 MB/sec
4k Sequential Read: 41.30 MB/sec
4k Random Read: 2.55 MB/sec
4k Random Write: 4.14 MB/sec

Again, so you don't have to scroll up, here are the results from the SSD benchmark previously performed:

Again, HPBS seems to really show its muscle compared to Amazon and even local Storm storage when it comes to random writes.

So, what does all this mean? Well, basically this is what I'm seeing with a Storm Baremetal Server doing nothing but running iozone. Actual performance of a product system will most likely vary. To what extent, I am not sure.

21Nov/121

Initial HPBS Performance Numbers

As you may be aware, Liquid Web has released its block storage product, High Performance Block Storage.

As such, I've gone ahead and run some simple benchmarks of my own, which I present to you. Your mileage may vary, but this is what I was seeing.

Server Information:
AMD FX-6100 Proc (3314 Mhz)
3831MB RAM
Debian 6.0.3 x64 upgraded to the latest stable packages
HPBS volume formatted ext4
iozone 3_414 compiled with linux-AMD64 as the target

Command run: ~/iozone -l 32 -O -i 0 -i 1 -i 2 -e -+n -r 4K -s 4G > ~/hpbs-iops.txt
*Note: The iozone executable was located in /root, the actual test was still being run on HPBS

Here is the output. First test result is for 4k Sequential Write, Second is for Sequential Read.

Iozone: Performance Test of File I/O
Version $Revision: 3.414 $
Compiled for 64 bit mode.
Build: linux-AMD64

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
Vangel Bojaxhi, Ben England.

Run began: Wed Nov 21 10:52:22 2012

OPS Mode. Output is in operations per second.
Include fsync in write timing
No retest option selected
Record Size 4 KB
File size set to 4194304 KB
Command line used: /root/iozone -l 32 -O -i 0 -i 1 -i 2 -e -+n -r 4K -s 4G
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Min process = 32
Max process = 32
Throughput test with 32 processes
Each process writes a 4194304 Kbyte file in 4 Kbyte records

Children see throughput for 32 initial writers = 10377.15 ops/sec
Parent sees throughput for 32 initial writers = 9485.68 ops/sec
Min throughput per process = 305.25 ops/sec
Max throughput per process = 348.35 ops/sec
Avg throughput per process = 324.29 ops/sec
Min xfer = 916573.00 ops

Children see throughput for 32 readers = 12519.84 ops/sec
Parent sees throughput for 32 readers = 12519.41 ops/sec
Min throughput per process = 363.21 ops/sec
Max throughput per process = 412.03 ops/sec
Avg throughput per process = 391.24 ops/sec
Min xfer = 924357.00 ops

As of this writing, the random read and write tests are still being run. I will update this article once these tests are complete.

With that said, what I'm seeing so far is pretty decent compared to AWS storage (not EBS mind you).
I'm basing this off of the chart supplied by Liquid Web when we benched our Storm SSD servers to AWS:

As you can note, I ran the same command that was used to bench Storm SSD. You'll notice that AWS has been beat so far (pending the updated random read/write tests). Conferring with a customer who has had experience with EBS, he has indicated that HPBS numbers will look even better in comparison to EBS.

Stay tuned!

   
FireStats icon Powered by FireStats
Optimization WordPress Plugins & Solutions by W3 EDGE