Cloud Vendor Evaluation: 1&1

The Study

1&1 commissioned a benchmarking analysis study to help evaluate the performance and price performance of its new Cloud Servers offering. The offering’s VM performance was compared against the relative performance of both mass-market cloud providers and smaller, competitive cloud providers. The VM components measured were the virtual cores, memory, block storage, and internal network on 1&1 Cloud Servers, Amazon Web Services (AWS), Aruba Cloud, Microsoft Azure and CloudSigma. Specifically, the study examines raw performance and price-performance value of several VMs of varying sizes. Findings from the study show that the 1&1 Cloud Server provides competitive levels of raw performance and price-indexed performance. 1&1 displayed strong price-performance value throughout the course of the study.

Performance & Price-Performance Key Findings:

For this study, Cloud Spectator evaluated performance by benchmarking the VMs using Geekbench 3, FIO and Iperf. The respective benchmarks measured vCPU & Memory, Storage and Internal Network. Testing occurred during the month of May, with tests running continuously over 72-hours. Pricing was examined in conjunction with the performance tests.

The Method

Cloud Spectator tested virtual machines from 1&1, Amazon, Aruba Cloud, Azure, and CloudSigma for three 24-hour intervals, for a total of 72-hours of testing on each provider. Testing occurred on randomly selected days during the month of May 2015. Iterations of the test suite were run continuously per 24-hour test cycle. Each iteration captured performance data on virtual processors, memory bandwidth, block storage IOPS, and internal network throughput. Tests within the suite are categorized in the table below.

Table 1: The tests listed in this table were used to collect quantitative data on performance across the selected VMs (see Table 2) on all the providers examined in the study.

To standardize VM sizes between the providers, comparable virtual machine sizes were chosen based on equivalent amounts of virtual cores reported by the provider. Block storage was matched by comparing a 100GB, 400GB or 800GB volume attached to each VM. A separate, equivalently sized virtual machine was provisioned within the same region/availability zone to act as a server for internal network testing. All virtual machines ran using Ubuntu Linux 14.04. All VMs were provisioned in Europe data centers. Table 2 provides details of VMs used for each provider.

Table 2: illustrates the VMs that were used in the comparison standardized by vCPU cores.
*Note: Amazon Block Storage testing examined EBS General Purpose (SSD) volumes; Azure Block Storage testing examined the Page Blob Storage volumes; CloudSigma Block Storage testing examined the SSD Storage volumes

Block Storage vs Local Storage

This study examined disk performance for block storage. Advantages in block storage include persistence, greater degrees of resiliency and general dependability. Data stored on block devices rest in a redundant SAN array, which replicates data across volumes.

DigitalOcean is excluded from the current study due to its local storage, which is not examined in the scope of this study. All other providers offer block storage, which provides an added level of resiliency.

Price-Performance: The CloudSpecs Score

The CloudSpecs Score, which is an indexed score for price-performance value, was calculated by taking the monthly price of each server size and performance results of the components and using a formula to find the level of performance per unit of price. The resulting number was then pegged to the highest result for that server class, which gets a score of 100; thus, value numbers are relative.

For example, if Provider A scores 100 and Provider B scores 50, then Provider A shows 2x more price-performance value than Provider B. Please see the formula below to calculate the CloudSpecs Score.

1. provider_value = {provider performance score} / {provider cost}
2. best_provider_value = max{provider_values}
3. Provider’s CloudSpecs Score = 100*provider_value / best_provider_value

Table 3: The table above shows the monthly price of the VMs used on the five providers. Rackspace pricing incorporates the required service fee at the Managed Infrastructure level. For providers with hourly pricing, usage of 720 hours per month was assumed in the monthly pricing above. Price of Block Storage is included with the VM costs.

The Preliminary Results

As indicated in the chart, 1&1’s VMs offered higher performance than the VMs of the other providers examined. 1&1 and AWS showed the most consistent performance throughout the complete 72 hours of testing.

The chart shown below displays a subsection of the results shown in the chart above. Different VMs completed a different amount of test iterations in the given 72-hour time limit due to VM performance. Therefore, the first 440 iterations which were completed by all provider VMs, are displayed below (250 for 8vCPU VMs). As observed in the chart above and below, both 1&1’s VMs score consistently around 4700. CloudSigma displayed a large fluctuation for the 2-8 VM in the chart above, with the variability displayed by the dip in performance on the chart below.

The combined CPU & Memory performance of the 4vCPU and 8vCPU VMs also showed 1&1 scoring higher on average than the four other providers examined.

The block storage performance evaluation displayed similar patterns across the different sized VMs and disks, as well as the four sub-tests of the FIO benchmark (random read, random write, sequential read, and sequential write). 1&1 recorded the highest IOPS, followed by Amazon (on the 400GB and 800GB volumes; Azure scored higher on the 100GB volume), with the remaining three providers rounding out the bottom.

The results of the Random Read tests are shown below. 1&1, which leverages SolidFire’s SSD technology, displays significantly higher performance than the other four providers, but highly variable results between 2000 and 4500 IOPS. While disk performance is very stable on AWS for the 4 and 8vCPU VMs, performance drops after a limited period of time on the 2 vCPU VMs, due to the time limitation of AWS’s EBS burst.

The internal network performance for the majority of the providers examined were stable and offered similar levels of throughput. CloudSigma offered relatively untethered network throughput at significantly higher levels, albeit with a great degree of variability. The internal network throughput of the 8vCPU VMs is shown below.

The price-performance of 1&1 came out on top across the vCPU, memory and storage tests of all the VMs. CloudSigma offered higher price-performance value for the internal network test on the 4vCPU VM, but 1&1 offers higher value on the 2vCPU and 8vCPU VMs. The value offered by 1&1 is due to the combination of relatively high performance scores and low price points.

As indicated by the 4vCPU/800GB charts below, 1&1 offered up to 3x the value of the other providers for CPU & memory price-performance, and even greater value for storage. The median test scores and monthly VM prices were used in the charts shown below.

General Consideration

1&1 offers greater performance on vCPU, memory and storage tests. However, due to the relatively new nature of the product, performance degradation may occur as more users join and resource contention increases.

1&1 sustains high IOPS on its SolidFire-backed SAN storage, regardless of VM size. To attain higher performance on providers such as Amazon, users must provision larger volume sizes and/or pay for additional IOPS.

The Full Report

The complete results of the benchmarking study will be released in June 2015. The full report will feature a complete methodology and analysis on all providers examined in this study.

You can examine 1&1’s offering here: http://www.1and1.co.uk/dynamic-cloud-server

Stay tuned to the Cloud Spectator Blog page or sign up for our newsletter to receive notification when the full report is out.