The Study
NxtGen commissioned a benchmarking analysis report to examine the relative performance of major VM components including virtual cores, memory, block storage, and internal network for NxtGen Enterprise Cloud Services (ECS), Amazon Web Services (AWS), and Rackspace Cloud Servers. Specifically, the study examines raw performance and price-indexed performance for two common workloads: web servers and databases. Findings from the study show that NxtGen ECS provides competitive levels of raw performance and price-indexed performance. NxtGen displayed particularly strong price-performance value, providing the most value across all five use cases examined in the study.
Performance & Price-Performance Key Findings: |
- For the majority of tests, NxtGen exhibited the highest storage and internal network performance; Rackspace exhibited the highest vCPU and memory performance.
- NxtGen exhibited the highest price-performance value across all tests and workload scenarios.
For this study, Cloud Spectator evaluated performance by benchmarking the VMs using Geekbench 3, FIO and Iperf. The respective benchmarks measured vCPU & Memory, Storage and Internal Network. Testing occurred over the course of a 5-day period. Pricing was examined in conjunction with the performance tests.
The Method
Cloud Spectator tested virtual machines from Amazon, NxtGen, and Rackspace for a period of five days between March 17, 2015 and March 21, 2015. One-iteration of the test suite was run per day. Each iteration captured performance data on virtual processors, memory bandwidth, block storage IOPS, and internal network throughput. Block storage was tested instead of local, ephemeral storage because of its persistence, resiliency and dependability. Tests within the suite are categorized in the table below.
Test | Resources Measured | Description |
Geekbench 3 | Processor & Memory | Geekbench 3 is a licensable product designed by Primate Labs. It is a collection of tests designed to simulate real-world scenarios, providing an indication of processor and memory performance. Tests results are broken out between processor (integer & floating point) and memory scores. |
Fio | Block Storage | Fio is an open source tool designed to stress IO. Cloud Spectator configured Fio to run sequential read, sequential write, random read, and random write tests to gather performance data on block storage IOPS. |
Iperf | Network Throughput | Iperf is an open source tool used to measure network throughput between client and server. By default, Iperf connects between the machines and measures throughput performance using a TCP protocol. Cloud Spectator used the default TCP protocol and transferred data bi-directionally. |
Table 1: The tests listed in this table were used to collect quantitative data on performance across the selected VMs (see Table 2) on all the providers examined in the study.
In an effort to standardize VM sizes between the providers, comparable virtual machine sizes were chosen based on equivalent amounts of virtual cores reported by the provider. Block storage was matched by comparing a 100GB, 500GB or 1000GB volume attached to each VM. A separate, equivalently sized virtual machine was provisioned within the same region/availability zone to act as a server for internal network testing. All virtual machines ran using Ubuntu Linux 14.04. The c4 instance type was selected for the Amazon 2vCPU VM comparison despite offering more memory than the comparable VMs on the other providers. At the time of the study, the c4 instances were the closest match to the other two providers’ VM sizes. The amount of memory of the c4 instances did not significantly affect the memory bandwidth performance. Amazon and Rackspace machines were provisioned in data centers as close to NxtGen’s Bangalore data center as possible. Table 2 provides details of VMs used for each provider.
2 vCPU Machines | vCPUs | Memory | Block Storage |
Amazon c4.large | 2 | 3.75GB | 100GB |
NxtGen ECS | 2 | 2GB | 100GB |
Rackspace General1-2 | 2 | 2GB | 100GB |
4 vCPU Machines | vCPUs | Memory | Block Storage |
Amazon c4.large | 4 | 15GB | 500GB |
NxtGen ECS | 4 | 15GB | 500GB |
Rackspace General1-2 | 4 | 15GB | 500GB |
8 vCPU Machines | vCPUs | Memory | Block Storage |
Amazon c4.large | 8 | 30GB | 1000GB |
NxtGen ECS | 8 | 30GB | 1000GB |
Rackspace General1-2 | 8 | 30GB | 1000GB |
Table 2: illustrates the VMs that were used in the comparison standardized by vCPU cores.
*Note: Amazon Block Storage testing examined EBS General Purpose (SSD) volumes; Rackspace Block Storage testing examined the Cloud Block Storage SSD volumes; NxtGen Block Storage testing examined the SSD Storage volumes
The CloudSpecs Score, which is an indexed score for price-performance value, was calculated by taking the monthly price of each server size and performance results of the components and using a formula to find the level of performance per unit of price. The resulting number was then pegged to the highest result for that server class, which gets a score of 100; thus, value numbers are relative.
For example, if Provider A scores 100 and Provider B scores 50, then Provider A shows 2x more price-performance value than Provider B. Please see the formula below to calculate the CloudSpecs Score.
- provider_value = {provider performance score} / {provider cost}
- best_provider_value = max{provider_values}
- Provider’s CloudSpecs Score = 100*provider_value / best_provider_value
Provider | Data Center | 2vCPU | 4 vCPU | 8vPU | 100GB HDD | 500GB HDD | 1000GB HDD |
Amazon EC2 | Singapore | $110.96 | $286.16 | $572.32 | $12.00 | $60.00 | $120.00 |
NextGen ECS | Bangalore | $43.75 | $146.05 | $268.30 | $14.48 | $72.38 | $144.77 |
Rackspace Cloud Server | Hong Kong | $65.70 | $492.75 | $985.50 | $54.97 | $274.85 | $549.69 |
Table 3: The table above shows the price of the VMs used on the three providers. Rackspace pricing incorporates the required service fee at the Managed Infrastructure level. For providers with hourly pricing, usage of 730 hours per month was assumed in the monthly pricing above.
A CloudSpecs score was produced for each of the following components on all the providers: vCPU, memory, sequential/random read/write IOPS, and internal network (see Table 4). The separate scores were aggregated based on the relevant components examined in each category to produce a VM CloudSpecs Score for each virtual machine size. The VM CloudSpecs scores were then averaged to produce the Category’s CloudSpecs Score. For example, on the Static Web Server:
- VM CloudSpecs Score = ({vcpu_cloudspecs_score} + {read_iops_cloudspecs_score})/2
- Static Web Server CloudSpecs Score = ({2vcpu_VM_CloudSpecs_Score} + {4vcpu_VM_CloudSpecs_Score})/2
The Use-Cases
Web Servers
NxtGen’s web server performance was examined in three categories: (1) static web server, which includes small websites; (2) read/write web server, which covers a majority of websites that users can download and upload content from/to; and (3) high-traffic web server, which encompasses large web architectures that sustain high amounts of traffic and transactions.
Databases
Apart from the different database applications available, such as MySQL, PostgreSQL, Microsoft SQL, and Oracle, the applicable virtual machine sizes for database environments and numerous use cases can make benchmarking a particular database a very complex process. Cloud Spectator’s approach in this study examines the server-side factors involved that affect database performance, namely the processor, memory, disk, and—in the case of large, distributed databases—internal network.