Cloud IaaS Value: Rackspace VS. Amazon in Price and Performance

Introduction

Disclaimer: We were not hired by any provider to write this blog post, and we do not endorse any cloud providers in this article. For more information on these providers and others, or if you have any questions, please contact us.

Rackspace, which integrated OpenStack with its cloud offering, recently lowered its pricing in the introduction of its Open Cloud. But has that change really affected the price-to-performance value for its customers? We analyze IaaS cloud industry pricing on a bi-monthly basis, and we record performance data 4x a day, 365 days a year with our CloudSpecs Performance system. From our observations, we have not seen a fluctuation in Rackspace performance, though pricing has dropped. With this new price change, Rackspace delivers a higher value for its clients in search of good performance. But how much price:performance value does the new Rackspace deliver, especially when compared to an industry giant like Amazon?

From a pure performance perspective, we’ve seen Amazon and Rackspace closely matched, though Amazon’s performance is spikey and unpredictable. But when taking pricing into account to calculate the value to the client, Amazon’s aggressive pricing has historically worked to their advantage.

Findings

Apart from the much smoother, predictable performance of Rackspace, its new pricing, though lower, does not amount to enough to compete against Amazon’s overall performance value. This does not, however, take into account the possible lack of predictability in performance, nor other, service-side factors, like support, which Rackspace is renowned for in the web hosting industry.

Rackspace

Strengths

Steady and predictable performance
Better average performance scores over 30 day period
Superior Disk and Internal Network Speed Performance

Weaknesses

Higher price
Lower or closely matched performance on CPU

Amazon

Strengths

Low Price
Better Price-to-Performance CloudSpecs values

Weaknesses

Spikey performance
Lower performance on Disk and Internal Network Speed

Details

To enlarge an image, please click on it below. For CloudSpecs score, the higher the number, the better value of price-to-performance. Lower CloudSpecs score(s) are pegged to the highest score, which is 100.

CPU Performance Comparison using 7-Zip Compression:
Amazon Average 30-day Score: 6,952 MIPS
Rackspace Average 30-day Score: 5,786 MIPS

Amazon CloudSpecs Score: 100
Rackspace CloudSpecs Score: 59

A test of 7-zip, an open-source utility for manipulating archives, and its integrated benchmark feature.

Disk I/O Performance Comparison using DBench:
Amazon Average 30-day Score: 75 MiB/S
Rackspace Average 30-day Score: 130 MiB/S

Amazon CloudSpecs Score: 81
Rackspace CloudSpecs Score: 100

DBENCH is a tool to generate I/O workloads to either a filesystem or to a networked CIFS or NFS server. It can even talk to an iSCSI target. DBENCH can be used to stress a filesystem or a server to see which workload it becomes saturated and can also be used for preditcion analysis to determine “How many concurrent clients/applications performing this workload can my server handle before response starts to lag?”

RAM Performance Comparison using STREAM:
Amazon Average 30-day Score: 16,736
Rackspace Average 30-day Score: 6,616

Amazon CloudSpecs Score: 100
Rackspace CloudSpecs Score: 28

The STREAM Benchmark is the de facto industry standard benchmark for the measurement of computer memory (RAM) bandwidth.

Internal Network Performance Comparison using Iperf:
Amazon Average 30-day Score: 552 Mibit/S
Rackspace Average 30-day Score: 396 Mibit/S

Amazon CloudSpecs Score: 100
Rackspace CloudSpecs Score: 51

Iperf was developed by NLANR/DAST as a modern alternative for measuring maximum TCP and UDP bandwidth performance. Iperf allows the tuning of various parameters and UDP characteristics. Iperf reports bandwidth, delay jitter, datagram loss.

Methodology

As mentioned earlier, we monitor performance 4 times a day, 365 times a year with our CloudSpecs system, a software suite of open-source, industry-standard server performance tests. From those findings, we average performance in a period of time (for purposes of this post, it is 30 days). Using that average, we plug in pricing to figure out a value score based on how much the server cost us and how much performance we’re actually getting out of it. The best value IaaS cloud provider is given a score of 100, and every other IaaS provider has a score pegged to it, so the other IaaS providers’ scores are in relation to the best IaaS provider’s value.

Price:Performance CloudSpecs Score calculation methodology:

provider_value(P) = [Provider test score over a period of time] / [Provider price]
best_provider_value = max(provider_values)
Provider’s CloudSpecs Score = 100 * provider_value(P) / best_provider_value

Pricing/Server Cost:

Amazon’s XLarge Instance: $0.64 per hour

Rackspace’s 15GB Server: $0.90 per hourServer configuration is a bit tricky because both Amazon and Rackspace only provide tiered structures, where you must select a pre-configured package (called “instances” on Amazon and “servers” on Rackspace) that contains a set amount of CPU, RAM, and disk space.

Server Configurations: