Geographic Cloud Performance Comparison

Back in May, we were asked about differences in cloud offerings between different geographic regions served by the same providers. Given that many of the larger providers advertise the same environment (similar hardware, configuration, etc.) from region to region, one would assume that performance between products be similar.

We knew that pricing would vary across regions, with the US usually being less expensive compared to Europe and Asia. This would impact the price-performance values of VMs across different geographies.

To answer the performance questions, and compare value from region to region, we decided to run some benchmarks in a geographic cloud performance comparison.

Earlier in the year, we released a couple reports where we examined the performance and price-performance value of some of the larger cloud IaaS providers in the EU and US (2016 Top 10 European IaaS Cloud Price Performance Report and 2016 Top 10 North American IaaS Cloud Price Performance Report) We did not, however, conduct a cloud performance comparison of vendors that made it onto both reports to see if there was consistency in the performance and value they offered in both regions. Also, the 2016 Top 10 Asia-Pacific IaaS Cloud Price Performance Report hadn’t been released yet, which left a major region out of the comparison.

To satiate our curiosity, we ran a few VMs across a large provider (AWS) and a known but less dominant provider (Dimension Data) in the cloud IaaS space. We used the same tests as in the reports mentioned above to measure the CPU, memory, storage and network performance on a couple VMs for two hours each. The VMs were hosted across the US, Europe, and Asia with details shown below.

*Storage results & pricing are for EBS General Purpose SSD

**Storage results & pricing are for High Performance*Storage results & pricing are for EBS General Purpose SSD

It’s important to note that this cloud performance comparison was done with a small sample size in mind, just for curiosity’s sake. If you are interested in an actual study of the major differences of performance and price-performance across different geographies, please reach out to us at [email protected] or visit our site and fill out the contact form.

The Results

CPU Performance & Price-Performance

The consistency of CPU performance across different regions was relatively high for the majority of VMs. Consistency, in this scenario, mainly depended on the variability of performance in general for each VM, which, as shown below, was significantly higher on Dimension Data compared to Amazon. Nonetheless, when examining the median CPU scores across the US, EU and Asia, both Amazon and Dimension Data varied by 2% or less for the 4vCPU and 8vCPU VM sizes. On the 2vCPU VM, Dimension Data varied by 9%, however, that appeared to largely be due to the variability in general of their 2vCPU VMs.

Factoring price into the comparison, the variation in regional pricing clearly stood out given the standardization of CPU performance. Amazon, which had the lowest variability in performance between regions, was largely differentiated in value due to higher costs in Europe, then Asia. Dimension Data had the same price structure for the Europe and Asia regions selected, resulting in similar value, differentiated only by performance. Interestingly enough, on Dimension Data there was a relatively consistent value despite the performance variation and cost differences.

Storage Performance & Price Performance

For the storage performance, we’ll focus on the Random Read/Write results (Sequential had similar results). This is really where the performance throttling policies of the individual providers stood out. Amazon’s EBS offering, which is pegged to the amount of storage you have provisioned, displayed consistency across the three regions with less than 1% variation. Dimension Data, on the other hand, did not appear to throttle storage performance, as the IOPS values differ up and down across the same regions. Variability on Dimension Data’s median storage values across regions was as high as 350%.

Given Amazon’s flat performance across regions, the only variation in price-performance value was caused by the pricing differentials, which altered the values across the regions by an average of 34%. Dimension Data’s variation in value was largely caused by the performance differences highlighted above, with value variability as hugh as 280% on the 4vCPU – 500GB VMs.

Internal Network Performance and Price Performance

Amazon’s performance throttling was seen yet again on the internal network performance of its VMs. This time the performance appeared to correlate with the size of the VM, with larger VMs (based on CPU count) offering more internal network throughput. Compared across regions, Amazon again offered less than 1% variation in performance. Dimension Data did not appear to throttle their performance, although the network throughput appeared to be clustered around similar, albeit somewhat wide ranges of performance. Variation across regions reaches 55% on Dimension Data.

Similar to what was observed on Amazon’s storage price-performance value, the only variation for Amazon’s internal network value was based on price, which averaged 34%. Dimension Data displayed some performance variation along with its price differences, which resulted in value variation of up to 66% on its 2vCPU VMs.

Conclusions from the Geographic Cloud Performance Comparison

Based on the light sampling in this cloud performance comparison, the general takeaway was that performance policies on providers largely impacted the performance variation between the different geographies they serve. If performance had been throttled to the same levels across all the geographies served, there would have been consistency between all the regions. While performance throttling allows providers to offer a consistent level of expected performance to their customers, it does limit the potential for a higher level of performance when resources aren’t in heavy contention. On the flip side, when resources are being heavily contended between users on a single server, performance throttling allows users at least a certain level of consistent usability. Also, from a planning perspective, knowing what performance level can consistently be expected from VMs in whichever region is needed allows for better forecasting of costs. All else equal on performance, price differences across regions play a significant part in the value differentials across data centers, with some geographies costing a third more than others, as observed above.