Public Cloud Providers: 2015 Snapshot

This snapshot provides an examination of the performance on Amazon EC2, DigitalOcean, LiquidWeb Storm VPS, Rackspace Cloud Servers.

Background

In July ’15 we were asked by LiquidWeb to evaluate their Storm VPS product against similar offerings from Amazon Web Services (AWS), DigitalOcean and Rackspace. The purpose of the evaluation was to validate their claims that performance of the Storm VPS product is greater than its main competitors. To compare the various products, Geekbench 3 and fio benchmark tests were run to gauge the CPU, memory and disk performance.

The Servers

The selection of virtual machines (VMs) included compute-optimized, memory-optimized and storage-optimized variants, as well as general purpose type VMs. The VMs ranged from 1vCPU to 12vCPU depending on the optimization type that was being compared. All four VM types were evaluated on CPU and memory performance, while only the general purpose and storage-optimized types were tested for disk performance. Three sets of VMs were examined for every VM type, with each set tested for 24-hours.

*The AWS m4 general purpose VMs do not come with local storage and were instead tested using the General Purpose SSD – Elastic Block Storage (EBS) using 150GB and 600GB volumes. Depending on the volume size, the EBS volumes can achieve a max IOPS of 10,000.

Provider	Region	Server	vCPU	RAM	Disk (gb)
Amazon	US East	m3.medium	1	3.75	1×4
Amazon	US East	m4.large	2	8	150
Amazon	US East	m3.large	2	7.5	1×32
Amazon	US East	m4.xlarge	4	16	600
Amazon	US East	m3.xlarge	4	15	2×40
Amazon	US East	c4.large	2	3.75	n/a
Amazon	US East	c3.large	2	3.75	2×16
Amazon	US East	c4.xlarge	4	7.5	n/a
Amazon	US East	c3.xlarge	4	7.5	2×40
Amazon	US East	r3.large	2	15.25	1×32
Amazon	US East	i2.xlarge	4	30.5	1×800
Digital Ocean	New York City	1GB	1	1	30
Digital Ocean	New York City	2GB	2	2	40
Digital Ocean	New York City	4GB	2	4	60
Digital Ocean	New York City	8GB	4	8	80
Digital Ocean	New York City	16GB	8	16	160
Digital Ocean	New York City	32GB	12	32	320
LiquidWeb	Zone C	1GB	n/a	1	50
LiquidWeb	Zone C	2GB	n/a	2	100
LiquidWeb	Zone C	4GB	2	4	150
LiquidWeb	Zone C	8GB	4	8	300
LiquidWeb	Zone C	16GB	4	16	600
LiquidWeb	Zone C	32GB	6	32	400
Rackspace	DFW	General1-1	1	1	20
Rackspace	DFW	General1-2	2	2	40
Rackspace	DFW	General1-4	4	4	80
Rackspace	DFW	Compute1-4	2	3.75	n/a
Rackspace	DFW	Compute1-8	4	7.5	n/a
Rackspace	DFW	Memory1-15	2	15	n/a
Rackspace	DFW	I/O1-15	4	15	150
Rackspace	DFW	I/O1-30	8	30	300

Findings of the Public Cloud Providers 2015 Comparison

The charts below show the results of the combined CPU and memory Geekbench test and the fio disk test. The results are displayed to show the min, 5th percentile, median, 95th percentile and max.

Overall, LiquidWeb was able to stand by their position of offering high performance, both for processing and storage performance. CPU and memory performance was relatively consistent for the small sized VMs, increasing in variability as the number of vCPUs rose. Storage performance doesn’t appear to have been tied to the server size.

CPU & Memory Performance

The Geekbench 3 multi-core score takes into account integer and floating point tasks for the CPU and the bandwidth of the memory. The score is heavily influenced by the number of vCPUs and less on the amount of memory included in the instance.

As the chart below shows, LiquidWeb takes up a lot of the top spots. LiquidWeb’s 6vCPU machine offers the highest performance, albeit with a high level of variability, but still beats out DigitalOcean’s 12vCPU 32GB VM.

Rackspace and Digital Ocean were also able to carve out strong performance amongst the higher vCPU machines.

Amazon’s newer c4 and m4 classes show the improvement from the previous c3 and m3 generations. However the performance is still lower than similarly sized VMs on the other providers.

Disk Performance – Sequential Operations

The fio test measures the IOPS of the storage using a 4-kb block size, with 5x 200mb files and direct I/O, for 50/50 read/write. Local disk provided with the VM was tested were available.

AWS’s m3 local storage achieved the highest median and max values for the sequential r/w operations.

Aside from the two AWS machines, LiquidWeb followed with the second highest disk performance, ranging from 20k to 35k IOPS.

Rackspace’s storage-optimized I/O type VMs show higher disk performance than the general purpose VMs, as they should.

On both DigitalOcean and LiquidWeb, the smallest VMs (1GB) measured the highest sequential r/w IOPS for each of the providers. Perhaps it’s the providers’ way of giving users a taste of the good stuff as they get started on a smaller ‘dose’ before moving onto the bigger and more powerful VMs. That or their client base mostly centers on the smaller servers and they have to keep their applications’ IO requirements satisfied.

Disk Performance – Random Operations

The random r/w operations using fio typically highlight the performance improvement of SSD over traditional spinning disk. However, for this evaluation all the providers offer SSD by default.

As with the sequential r/w results, AWS’s m3 instances offer the highest median and maximum IOPS for the storage test.

Thereafter, LiquidWeb takes the top spots after AWS, offering relatively similar median IOPS levels for its VMs.

Digital Ocean follows AWS and LiquidWeb, but displays a significant amount of variability. While Rackspace does not perform as highly, the disk performance is tightly clustered, showing consistent performance patterns.

Public Cloud Providers: 2015 Snapshot Conclusion

LiquidWeb can indeed say that performance is one of their key attributes for the Storm VPS product line. They have strong processing power and storage performance. AWS doesn’t have the top processing performance, but strong storage depending on which instance type is used. DigitalOcean has decent processing and storage performance, but it can be unstable. Rackspace’s performance may be a bit lower, but it is consistent. As this data was from July ’15 and only three sets of servers, more recent and more extensive testing would be recommended if you’re considering a move. However, the results above provide a first step in finding the right cloud provider(s) for your specific use case.

If you would like assistance in interpreting the data from our Public Cloud Providers: 2015 Snapshot, or help in choosing the right cloud provider for your business, then contact our team today.