The Study
1&1 commissioned a benchmarking analysis study to help evaluate the performance and price performance of its new Cloud Servers offering. The offering’s VM performance was compared against the relative performance of both mass-market cloud providers and smaller, competitive cloud providers. The VM components measured were the virtual cores, memory, block storage, and internal network on 1&1 Cloud Servers, Amazon Web Services (AWS), Aruba Cloud, Microsoft Azure and CloudSigma. Specifically, the study examines raw performance and price-performance value of several VMs of varying sizes. Findings from the study show that the 1&1 Cloud Server provides competitive levels of raw performance and price-indexed performance. 1&1 displayed strong price-performance value throughout the course of the study.
Performance & Price-Performance Key Findings:
- 1&1 exhibited highest performance on vCPU and storage performance; 1&1 and Aruba Cloud offered highest memory performance.
- 1&1 exhibited up to 3x higher price-performance value across the vCPU, memory and greater on the storage tests.
- 1&1 and AWS VMs exhibited the most stable performance over the test period.
- AWS’s EBS storage exhibited the most stable performance on 4 vCPU and 8 vCPU machines over the 72-hour test period.
- CloudSigma offered the best performance and value for internal network.
For this study, Cloud Spectator evaluated performance by benchmarking the VMs using Geekbench 3, FIO and Iperf. The respective benchmarks measured vCPU & Memory, Storage and Internal Network. Testing occurred during the month of May, with tests running continuously over 72-hours. Pricing was examined in conjunction with the performance tests.
The Method
Cloud Spectator tested virtual machines from 1&1, Amazon, Aruba Cloud, Azure, and CloudSigma for three 24-hour intervals, for a total of 72-hours of testing on each provider. Testing occurred on randomly selected days during the month of May 2015. Iterations of the test suite were run continuously per 24-hour test cycle. Each iteration captured performance data on virtual processors, memory bandwidth, block storage IOPS, and internal network throughput. Tests within the suite are categorized in the table below.
Test | Resources Measured | Description |
Geekbench 3 | Processor & Memory | Geekbench 3 is a licensable product designed by Primate Labs. It is a collection of tests designed to simulate real-world scenarios, providing an indication of processor and memory performance. Tests results are broken out between processor (integer & floating point) and memory scores. |
Fio | Block Storage | Fio is an open source tool designed to stress IO. Cloud Spectator configured Fio to run sequential read, sequential write, random read, and random write tests to gather performance data on block storage IOPS. |
Iperf | Network Throughput | Iperf is an open source tool used to measure network throughput between client and server. By default, Iperf connects between the machines and measures throughput performance using a TCP protocol. Cloud Spectator used the default TCP protocol and transferred data bi-directionally. |
Table 1: The tests listed in this table were used to collect quantitative data on performance across the selected VMs (see Table 2) on all the providers examined in the study.
To standardize VM sizes between the providers, comparable virtual machine sizes were chosen based on equivalent amounts of virtual cores reported by the provider. Block storage was matched by comparing a 100GB, 400GB or 800GB volume attached to each VM. A separate, equivalently sized virtual machine was provisioned within the same region/availability zone to act as a server for internal network testing. All virtual machines ran using Ubuntu Linux 14.04. All VMs were provisioned in Europe data centers. Table 2 provides details of VMs used for each provider.
2 vCPU Machines | Shorthand | vCPUs | Memory | Block Storage |
1&1 Cloud Server 2-4 | 1&1 2c4m | 2 | 4GB | 100GB |
1&1 Cloud Server 2-8 | 1&1 2c8m | 2 | 8GB | 100GB |
AWS c4.large | AWS c4.large | 2 | 3.75GB | 100GB |
AWS m3.large | AWS m3.large | 2 | 7.5GB | 100GB |
Aruba Cloud 2-4 | Aruba 2-4 | 2 | 4GB | 100GB |
Aruba Cloud 2-8 | Aruba 2-8 | 2 | 8GB | 100GB |
Azure A2 Standard | Azure A2 | 2 | 3.5GB | 100GB |
Azure D2 | Azure D2 | 2 | 7GB | 100GB |
CloudSigma 2-4 | CS 2-4 | 2 | 4GB | 100GB |
CloudSigma 2-8 | CS 2-8 | 2 | 8GB | 100GB |
4 vCPU Machines | Shorthand | vCPUs | Memory | Block Storage |
1&1 Cloud Server 4-8 | 1&1 4c8m | 4 | 8GB | 400GB |
1&1 Cloud Server 4-15 | 1&1 4c15m | 4 | 15GB | 400GB |
AWS c4.xlarge | AWS c4.xlarge | 4 | 7.5GB | 400GB |
AWS m3.xlarge | AWS m3.xlarge | 4 | 15GB | 400GB |
Aruba Cloud 4-8 | Aruba 4-8 | 4 | 8GB | 400GB |
Aruba Cloud 4-15 | Aruba 4-15 | 4 | 15GB | 400GB |
Azure A3 Standard | Azure A3 | 4 | 7GB | 400GB |
Azure D3 | Azure D3 | 4 | 14GB | 400GB |
CloudSigma 4-8 | CS 4-8 | 4 | 8GB | 400GB |
CloudSigma 4-15 | CS 4-15 | 4 | 15GB | 400GB |
8 vCPU Machines | Shorthand | vCPUs | Memory | Block Storage |
1&1 Cloud Server 8-15 | 1&1 8c15m | 8 | 15GB | 800GB |
1&1 Cloud Server 8-30 | 1&1 8c30m | 8 | 30GB | 800GB |
AWS c4.2xlarge | AWS c4.2xlarge | 8 | 15GB | 800GB |
AWS m3.2xlarge | AWS m3.2xlarge | 8 | 30GB | 800GB |
Aruba Cloud 8-15 | Aruba 8-15 | 8 | 15GB | 800GB |
Aruba Cloud 8-30 | Aruba 8-30 | 8 | 30GB | 800GB |
Azure A4 Standard | Azure A4 | 8 | 14GB | 800GB |
Azure D4 | Azure D4 | 8 | 28GB | 800GB |
CloudSigma 8-15 | CS 8-15 | 8 | 15GB | 800GB |
CloudSigma 8-30 | CS 8-30 | 8 | 30GB | 800GB |
Table 2: illustrates the VMs that were used in the comparison standardized by vCPU cores.
*Note: Amazon Block Storage testing examined EBS General Purpose (SSD) volumes; Azure Block Storage testing examined the Page Blob Storage volumes; CloudSigma Block Storage testing examined the SSD Storage volumes
Block Storage vs Local Storage
This study examined disk performance for block storage. Advantages in block storage include persistence, greater degrees of resiliency and general dependability. Data stored on block devices rest in a redundant SAN array, which replicates data across volumes.
DigitalOcean is excluded from the current study due to its local storage, which is not examined in the scope of this study. All other providers offer block storage, which provides an added level of resiliency.
Price-Performance: The CloudSpecs Score
The CloudSpecs Score, which is an indexed score for price-performance value, was calculated by taking the monthly price of each server size and performance results of the components and using a formula to find the level of performance per unit of price. The resulting number was then pegged to the highest result for that server class, which gets a score of 100; thus, value numbers are relative.
For example, if Provider A scores 100 and Provider B scores 50, then Provider A shows 2x more price-performance value than Provider B. Please see the formula below to calculate the CloudSpecs Score.
- provider_value = {provider performance score} / {provider cost}
- best_provider_value = max{provider_values}
- Provider’s CloudSpecs Score = 100*provider_value / best_provider_value
Provider | Data Center | 2vCPU/4GB RAM | 2vCPU/8GB RAM | 4vCPU/8GB RAM | 4vCPU/15GB RAM | 8vCPU/15GB RAM | 8vCPU/30GB RAM |
1&1 Cloud Server | EU | € 29.99 | € 79.20 | € 93.60 | € 144.00 | € 172.80 | € 280.80 |
Amazon EC2 | EU (Ireland) | € 95.04 | € 108.72 | € 208.08 | € 236.16 | € 416.88 | € 473.76 |
Aruba Cloud | Germany | € 64.80 | € 79.20 | € 158.40 | € 183.60 | € 313.20 | € 367.20 |
Azure (Microsoft) | West Europe | € 68.10 | € 104.60 | € 143.65 | € 216.59 | € 287.24 | € 433.11 |
Table 3: The table above shows the monthly price of the VMs used on the five providers. Rackspace pricing incorporates the required service fee at the Managed Infrastructure level. For providers with hourly pricing, usage of 720 hours per month was assumed in the monthly pricing above. Price of Block Storage is included with the VM costs.
The Preliminary Results
The study was broken out into VMs of 2vCPUs, 4vCPUs and 8vCPUs. A representative selection of tests are shown below to highlight the trends and general findings of the study.
The chart to the left shows the aggregated CPU & Memory performance of the 2vCPU VMs. The minimum, 5th percentile, median, 95th percentile and maximum values of the test scores are displayed.
As indicated in the chart, 1&1’s VMs offered higher performance than the VMs of the other providers examined. 1&1 and AWS showed the most consistent performance throughout the complete 72 hours of testing.
The chart shown below displays a subsection of the results shown in the chart above. Different VMs completed a different amount of test iterations in the given 72-hour time limit due to VM performance. Therefore, the first 440 iterations which were completed by all provider VMs, are displayed below (250 for 8vCPU VMs). As observed in the chart above and below, both 1&1’s VMs score consistently around 4700. CloudSigma displayed a large fluctuation for the 2-8 VM in the chart above, with the variability displayed by the dip in performance on the chart below.
The combined CPU & Memory performance of the 4vCPU and 8vCPU VMs also showed 1&1 scoring higher on average than the four other providers examined.
The block storage performance evaluation displayed similar patterns across the different sized VMs and disks, as well as the four sub-tests of the FIO benchmark (random read, random write, sequential read, and sequential write). 1&1 recorded the highest IOPS, followed by Amazon (on the 400GB and 800GB volumes; Azure scored higher on the 100GB volume), with the remaining three providers rounding out the bottom.
The results of the Random Read tests are shown below. 1&1, which leverages SolidFire’s SSD technology, displays significantly higher performance than the other four providers, but highly variable results between 2000 and 4500 IOPS. While disk performance is very stable on AWS for the 4 and 8vCPU VMs, performance drops after a limited period of time on the 2 vCPU VMs, due to the time limitation of AWS’s EBS burst.
The internal network performance for the majority of the providers examined were stable and offered similar levels of throughput. CloudSigma offered relatively untethered network throughput at significantly higher levels, albeit with a great degree of variability. The internal network throughput of the 8vCPU VMs is shown below.
The price-performance of 1&1 came out on top across the vCPU, memory and storage tests of all the VMs. CloudSigma offered higher price-performance value for the internal network test on the 4vCPU VM, but 1&1 offers higher value on the 2vCPU and 8vCPU VMs. The value offered by 1&1 is due to the combination of relatively high performance scores and low price points.
As indicated by the 4vCPU/800GB charts below, 1&1 offered up to 3x the value of the other providers for CPU & memory price-performance, and even greater value for storage. The median test scores and monthly VM prices were used in the charts shown below.
General Consideration
1&1 offers greater performance on vCPU, memory and storage tests. However, due to the relatively new nature of the product, performance degradation may occur as more users join and resource contention increases.
1&1 sustains high IOPS on its SolidFire-backed SAN storage, regardless of VM size. To attain higher performance on providers such as Amazon, users must provision larger volume sizes and/or pay for additional IOPS.
The Full Report
The complete results of the benchmarking study will be released in June 2015. The full report will feature a complete methodology and analysis on all providers examined in this study.
You can examine 1&1’s offering here: http://www.1and1.co.uk/dynamic-cloud-server
Stay tuned to the Cloud Spectator Blog page or sign up for our newsletter to receive notification when the full report is out.