Public IaaS Performance MUST be Monitored Over Time

Abstract

Public IaaS cloud is a multi-tenant environment where users of that cloud must share resources. The leads to fluctuations in performance, especially as more users provision cloud environments on the same physical server. One simple performance measurement is not enough to predict performance over time.

Introduction

After our recent posts on 30-day results of internal network performance of Amazon VS. Zunicore and price-performance of Amazon VS. Rackspace, People have asked me why Cloud Spectator’s CloudSpecs Performance Test System doesn’t just provision a server, run all our tests on it, record the results, and move on. Thinking back, I should’ve made this point first. It’s simple: the majority of the public IaaS cloud market has not developed a level of steady performance that can overcome factors like multi-tenancy and over-provisioned servers. And by majority, I mean the gorilla taking up 70% of the market.

So what if it’s multi-tenant?

Let me put in into perspective for you by using an all-American addiction, coffee.

Keeping in mind legacy servers (whether in your own data center, colocation, or web-access dedicated servers), imagine a coffee maker. You have it in your own, and every morning at 6AM, you wake up, and your awesome coffee maker’s timer tells it to start brewing. By the time you’re done brushing your teeth and getting dressed, you go into the kitchen and that fresh cup of coffee is waiting for you. Now, you only need to time the coffee maker once to know that it will always take 40 seconds to prepare that coffee for you. You live happily ever after.

Now, let’s take a public IaaS scenario: you discover the coffee shop. Your coffee maker broke down, and you’re starting to get into espresso anyway, but don’t want to put down $300 for an espresso maker. Every morning, you go through your same tradition of cleaning and dressing yourself, and then driving off, but making a pit-stop at Starbucks this time. But here’s the question: how much time do you need to set aside for getting your coffee at the coffee shop? That’s impossible to predict because you just don’t know how many people are already piled up in that coffee shop, waiting for that same morning pick-me-up. Sometimes it’s 5 minutes if you get a good spot in line, other times it’s 10-15 minutes, but if you’re really lucky, you might just get it in 2. Point is, there’s no guarantee that you’ll reach your final destination at the exact same time each time.

Public IaaS clouds are coffee shops, and users are the customers who make use of the coffee to become a vital part of their daily function.