How To Compare Cloud Providers, Services, and Prices
Back in 2014 we tested Microsoft’s Azure A-series public cloud offering against the then newly released D-series. From this data, we created a report for GigaOM entitled ‘Generational Performance Comparison: Microsoft Azure’s A-Series and D-Series’.
The D-series Virtual Machines (VMs) are focused on higher performance with more memory. With processors touted as 60% faster than those within their predecessors, the A-series, the D-series VMs also sport local solid-state drives (SSD) for better storage performance.
But now the even newer Dv2-series VMs boast a 35% improvement on their predecessor in processor performance. The Dv2 VMs have the same memory and storage configurations, yet are priced about 5% lower than the older generation D-series VMs.
The purpose of this blog post is to test just how much better the performance of the new Dv2-series is compared to the original D-series VMs in CPU, memory and storage. The performance and price-performance values are used to evaluate the raw power of each VM class, as well as the economic value of performance to price.
The table below lists the VMs we examined in our Azure Dv2 vs Ds comparison:
| Type | vCPU | RAM (GB) | Storage (GB) | D-series $$ | Dv2-series $$ |
| D1 | 1 | 3.5 | 50 | $0.077 | $0.073 |
| D2 | 2 | 7 | 100 | $0.154 | $0.146 |
| D3 | 4 | 14 | 200 | $0.308 | $0.293 |
| D4 | 8 | 28 | 400 | $0.616 | $0.585 |
| D11 | 2 | 14 | 100 | $0.195 | $0.185 |
| D12 | 4 | 28 | 200 | $0.390 | $0.371 |
| D13 | 8 | 56 | 400 | $0.780 | $0.741 |
| D14 | 16 | 112 | 800 | $1.542 | $1.482 |
* The D5v2 wasn’t tested because the D-series doesn’t have a D5 VM type available.
The Methodology
Performance testing was run on 3 separate Linux VMs (running Ubuntu 14.04) on each type and series for 6 hours per VM. The continuous testing yielded several data points for examination. The minimum, 5th percentile, median, 95th percentile and maximum values were derived from the combined results of each VM type/generation.
Testing with Geekbench3 and Fio
The Geekbench 3 test suite is comprised of 27 individual tasks/workloads: 13 integer workloads, 10 floating point workloads, and 4 memory-bandwidth tasks. For more information on Geekbench 3 and to see its individual workloads, please see http://www.primatelabs.com/geekbench/
Fio is an open source I/O generator that spawns a number of threads and processes to conduct a particular type of I/O action specified. For the purpose of this study, Fio was used to measure disk IOPS by tracking direct I/O to the VM’s network storage. 5 x 200MB files were created for sequential operations testing, and 5 x 200MB files were created for random operations testing. All operations were 50% read and 50% write. Each test iteration used a 4KB block size and lasted 60 seconds.
Calculating the Price-Performance Value
Cloud Spectator’s price-performance calculation, the CloudSpecs Score™, provides information on how much performance is realized for each unit of cost. The CloudSpecs Score™ is an indexed, comparable score ranging from 0-100, indicative of value based on a combination of cost and performance. The value is scaled. For example, a Cloud Service Provider (CSP) with a score of 100 gives 4x the value of a CSP with a score of 25. The CloudSpecs Scores™ in this report can only be compared with equivalent configurations. In this example, for instance, a 1vCPU instance on the D-Series compared to a 1vCPU virtual machine on the Dv2-Series.
The Calculation of CloudSpecs Score™
- Provider_value = [Provider Performance Score] / [Provider Cost]
- Best_provider_value = max{provider_values}
- CSP’s CloudSpecs Score = 100*provider_value / best_provider_value
The Comparison
The following section shows the side-by-side comparison of the performance tests and price-performance values for the CPU, memory and storage of the D-series and Dv2-series VMs.
Azure Dv2 vs Ds: CPU Comparison
Since Geekbench doesn’t offer a broken-out CPU Score, we took the average of the median Integer and Floating Point scores to get the Azure Dv2 vs Ds performance scores shown below.
As you can see above, the Dv2 VMs do in fact offer higher performance compared to the older D-series VMs. Our testing showed an average performance increase of 33% from D-series to Dv2-series VMs, which is very close to Azure’s 35% average. However, factoring price into the mix, for price-performance value, the Dv2-series is on average a 39% greater value compared to its predecessor.
Azure Dv2 vs Ds: Memory Comparison
The memory bandwidth tests are taken from Geekbench’s memory score, which runs a series of Add, Copy, Scale and Triad workloads and aggregates the individual results.
While the overall stability of the performance levels fluctuated for some VMs more than others, the average difference between the D-series and Dv2-series was only 2-3%. Looking at the next dimension of price-performance value, the balance shifted back and forth between the generations depending on the VM size for a relative net neutral.
Azure Dv2 vs Ds: Storage Comparison
To test the performance of the disks, we ran Fio for sequential and random read/write operations (50% read, 50% write). The performance scores listed for the storage performance is the input/output per second (iops).
Once again, as the memory and storage configurations are the same for both generations of D-series VMs, performance is very similar between the two.
Storage performance is nearly identical between the generations for all VM types with the exception of the D14. Variability was much higher for both generations on the D14 type VM, although the newer Dv2 VM displayed 17% higher performance. Price-performance was slightly higher for the newer Dv2 VMs largely due to the lower price of the new offering.
The Conclusion
The verdict is that the Dv2 offers better CPU performance as promised, and similar memory and storage performance. Due to the lower cost, the Dv2-series is a better value than its predecessor, the D-series.
Related posts
