Benchmark floating point computations with Python
Early this year, the Cloud Spectator research team conducted an exhaustive industry study to determine 2017’s best IaaS service provider in North America based on price-performance value of public cloud offerings. The report was released in late February.We set out to rank the best IaaS service providers and arm the enterprise buyer with an unbiased view of the marketplace. The 2017 Cloud IaaS Providers Comparison analyzes the performance of four different VM sizes for each provider, and reports on the performance and stability results of CPU, Block Storage, and Memory.
The results were eye-opening, uncovering a 7x difference between the first and tenth ranked IaaS service providers analyzed in the report. For cloud consumers, this discrepancy highlights the critical need to conduct performance testing of applications prior to making a move to the cloud, or risk overspending just to maintain sufficient application performance on infrastructure that should have cost less.
It is important to note that since conducting our Top 10 Cloud Vendor Benchmark 2016, OVH, a French cloud computing company, not included in last year’s report, has made a very strong showing in the North American market— second only to 1&1.
Other Cloud Service Providers (CSPs) such as HP Cloud and Verizon Cloud have since left the market or reduced their services. Verizon shut down two Public Cloud Service, informing users to move their workloads to its Virtual Private Cloud (VPC).
The following IaaS providers were included in the 2017 Top 10 Cloud IaaS Providers Comparison North America Report:
|
|
The graph below ranks the best IaaS service providers based on their overall CloudSpecs Score™. Please refer to the complete 2017 Cloud IaaS Providers Comparison for full details on testing methodology and ranking criteria.
Key Findings of the Best Cloud Providers 2017 Study
- Looking solely at performance, virtual machines (VMs) differed by almost 2x across CSPs. For block storage IOPS, that difference spread by 15x across CSPs.
- Rackspace demonstrated the highest median VM performance.
- Microsoft Azure demonstrated the best performance stability.
- Rackspace Cloud demonstrated the highest median block storage performance.
- The small VM category exhibits a difference of 1.7x between the highest and lowest performing CSP VMs. The Extra Large VM category exhibits a difference of 1.9x between the highest and lowest performing CSP VMs.
- Microsoft Azure showed the lowest performance variability over the testing period. Amazon AWS demonstrated controlled performance throttling on disk IOPS. The impact of throttling on AWS EBS disks are determined by the size of the disk.
- When looking at price-performance value, more than a 7x difference emerged among the compared IaaS providers.
- 1&1 achieved the highest CloudSpecs Score™ measuring value as a component of price and performance. This is due to good VM performance and the most competitive packaged pricing found in the study.
- While certain providers such as Rackspace may have achieved above-average performance rankings, their value scores were lower due to significantly higher costs.
- For price-performance value, 1&1 and OVH consistently ranked as the two best providers across all VM sizes from small to extra-large.
- For the price-performance value of VMs alone, 1&1 maintained the highest CloudSpecs Score™ , with OVH following closely in second.
- Regarding block storage value, 1&1 received the highest CloudSpecs Score™
Performance of the Best Cloud Providers
A lack of transparency in the public cloud IaaS marketplace for performance often leads to misinformation or false assumptions.
Users or potential buyers may be inclined to view cloud computing as a commodity, differentiated mostly by price and services. The reality of performance in cloud computing is that it impacts the user differently from one CSP to the next, involving everything from physical hardware (e.g., Intel or AMD, SSD or spinning disc) to the cost of the virtualized resources.
By identifying environments based on performance rather than resource count, users can maximize value in the cloud.
The study tested four VM sizes with attached block storage for each CSP. For each VM size and attached block storage, Cloud Spectator provisioned and tested three VMs simultaneously for a total of 12 running VMs (nine for the CSPs that didn’t offer the extra-large size). Tests were conducted over a period of 24 hours to account for fluctuations throughout the day.
Get the complete 2017 Cloud IaaS Providers Comparison here.
Related posts
