How to Compare, AWS, Azure, Google, and IBM

Every day, we’re faced with choices— from what to eat or watch on Netflix to the type of shampoo we buy, we’re faced with choices each and every day. And getting started on the Cloud is no different. Is it AWS, Azure, Google Cloud, or IBM Cloud?

For any business looking to move their operations to the Cloud, they must first start by performing a Cloud service provider comparison.

Comparing Cloud providers is becoming an increasingly challenging task. These days, the Cloud is dominated by four powerhouse providers — Amazon Web Services (AWS), Azure, Google Cloud, and IBM Cloud. While each provider packs their own amazing strengths, picking a leader really depends on the specific needs of a specific enterprise.

In this blog, we give a breakdown of three variables you need to understand and evaluate in order to accurately compare AWS, Azure, Google, and IBM as they relate to your business needs. These variables are: Cloud performance, price, and price-performance.

NOTE: This three-part blog series is available in this eBook, Comparing Cloud Providers: A Key Metric for Seeing Through the Clouds. Or read Part One at: How to Compare Cloud Providers, Services, and Prices and Part Three at: Seeing Through The Clouds: The Power of Cloud Transparency.

The Top Three Variables Needed to Compare AWS, Azure, Google, and IBM

Cloud Performance

Cloud performance is an essential attribute of all Cloud services, whether they are IaaS compute services (virtual machines), storage services, network services, or the myriad of services that are now available from global Cloud providers.

Performance data is critical as it tells the actual story of how a given virtual machine (VM) configuration performs under various operating conditions and exceeds the limited information typically provided in a provider’s spec sheet.

Determining Cloud Performance

Cloud performance can only be assessed by monitoring the performance of specific Cloud services when running workloads. One way to gather this performance data is by performing cloud service provider comparison benchmarking: see the Cloud Transparency Platform.

For virtual machines, the performance attributes of interest will be the computation performance, which measures the speed and capacity of the virtual CPUs, or vCPUs. Benchmarking tools, such as Geekbench 4, provide many sample workloads that can simulate real-world scenarios.

Another key attribute to test is the storage performance of a VM configuration, whether it includes local storage or attached storage. A useful tool for measuring storage performance is FIO, or Flexible Input/Output Tester, which helps determine how storage performs under various sequential read and write operations, as well as random read and write operations using various block sizes.

There are many other synthetic benchmarking tools available on the market for these general testing approaches, as well as many industry-specific benchmarking tools for financial services, life sciences, and others. The key is to obtain repeatable testing results that can be used to compare AWS, Azure, Google, IBM, and others on a basis of VM performance.

Cloud Pricing

Cloud pricing models are often difficult to compare on an apples-to-apples basis across providers. Furthermore, price alone is not sufficient to understand what provider and what Cloud services will best suit a particular workload or, in the aggregate, a given Cloud strategy. Pricing should be monitored on a standardized and consistent basis to capture on-going changes.

Collecting Cloud Pricing Information

Pricing data must be collected and mapped to specific Cloud services in order to facilitate greater comparative value of this data. Depending on the provider, collecting Cloud pricing data can be relatively easy or more challenging. Many providers offer a pricing API (application programming interface) that allows an application to automatically process the API to collect Cloud pricing data. Others offer a spreadsheet tool for all of their pricing data, while others only offer such data via their website, making price data collection more difficult. In some cases, you may have to actually speak with a sales representative.

Once pricing data has been collected, of course, it must be maintained. There should be an ongoing price updating process that ensures accurate and up-to-date pricing information is collected. Pricing data is dynamic, and thus it must be continually updated and maintained. Using an ongoing pricing data collection process will enable continuous monitoring of prices, detection of price changes, as well spotting new discounting strategies that might be deployed.

Price-Performance Analysis: A Universal Cloud Comparison Metric

Pricing information, when combined with performance analytics, provides a universal metric for comparing Cloud services from multiple providers: Price-Performance.

Furthermore, price-performance is an easy metric to calculate. Once the performance criteria have been identified and measured, along with valid and current pricing data for the respective Cloud services, price-performance can be determined.

Price-performance is calculated by comparing Cloud service performance values to the monthly price for that Cloud service. For example, storage performance can be evaluated using tools such as FIO, and specific tests used within the FIO utility, such as random reads and random write performance using 4K block sizes.

If a Cloud storage service achieves 10,000 input-output operations per second (IOPS), and its monthly price is $95, the resulting price-performance value is: 105 IOPS/dollar.

This simple price-performance (or value) calculation provides a universal approach for comparing performance and price to understand what Cloud providers and services are best suited to your specific workloads.

To highlight the importance of comparing Cloud services on an apples-to-apples basis, we can use the following scenarios. In the chart below, five different virtual machines from multiple providers are compared based on the monthly price for a given VM.

Figure 1: Comparing Cloud VMs by Price Only

From this data, VM 5 is the most expensive, while VM 2 has the lowest monthly cost for this particular VM configuration. It appears that VM 2 is the best value on a pure price basis.

Next, we can evaluate the same Cloud services on the basis of their CPU performance. Again, performance data must be generated via execution of industry-accepted benchmarking tools and methodologies to ensure apples-to-apples comparisons. The Cloud performance for these virtual machines is provided in Figure 2 below.

Figure 2: Comparing Provider VMs by Performance Only

From the performance data illustrated in Figure 2, we can see that VM 4 has the highest performance using the GeekBench4 multi-core score, which evaluates computational performance. VM 5 is next, followed by VM 1, 3 and 2 respectively.

Based on the computational performance scores, VM 4 offers the best performance. Is the performance data sufficient to make a purchasing decision on Cloud services? To best answer that question, we need to examine one more comparison: price-performance.

In Figure 3 below, the price and performance data are combined into a price-performance evaluation. In this comparison, CPU performance data for a given VM is divided by the monthly price to arrive at a performance for a given dollar spent on a particular Cloud service, in this case, VMs. The data is presented below:

Figure 3: Price-Performance Comparison of Virtual Machines

As Figure 3 clearly illustrates, VM 1 offers more compute value for the dollar spent, followed by VM 2 and then VM 4, followed by VMs 3 and 5 respectively. Recall that VM 2 had the lowest price, but yet fell to the second best in price-performance value. While VM 4 was the top performer by CPU performance, VM 4 was middle of the pack in price-performance value. Conversely, VM 1 was middle of the pack in performance but represents the best overall value in CPU performance per dollar spent.

Price, performance, and price-performance (value) provide three different approaches for comparing Cloud services. However, the price-performance metric provides a simple universal means by which Cloud services can be compared and purchased and thus becomes part of the Cloud Transparency foundation.

Coming Up Next…

Stay tuned for part 3 of our Cloud transparency series where we’ll discuss how price-performance analytics can help enable objective comparisons of Cloud providers and services.