Benchmarking Amazon Web Services (AWS) Vs Rackspace

The following is an article about load testing by performanceXpert.

"Comparing bandwidth on a AWS Micro instance and a Rackspace 512MB server

Cloud computing is an emerging trend which enables the rapid deployment of new web based applications. While cloud computing providers do not commit to a performance level or end-user experience, they encourage their users to test and benchmark before deploying.

When you host a server at your own facility, you know according to your network interface and the bandwidth provided by your ISP what to expect in-terms of bandwidth allocation. If you find yourself asking what your bandwidth could be via a cloud provider, then this article is for you.

The goal of this article is to understand if there are any bandwidth limitations associated with the cloud providers, AWS and Rackspace, and if there are such limitations that can lead to bottlenecks, what are they.

Choosing the cloud resource

To start our test, I chose a Micro type resource from AWS and a 512MB server from Rackspace. I used an out-of-the-box Ubuntu installation and installed Apache2 on both resources. I did not change the default configuration. This way, we have a “vanilla” Ubuntu and a "vanilla" Apache installation on similar sized cloud resources. To benchmark the two providers, I chose a very basic load scenario. The load scenario simulates users executing a simple business process which contains a short user sequence. In our scenario a user opens a browser, downloads a large file (4MB), waits for 10 seconds then loads a small HTML file (25KB) and then stops. The load scenario includes a gradual user ramp up from 0 to up to 100 concurrent users. This gradual ramp up should tell us, if there is a bottleneck and where.

The results were quite obvious.

AWS results
*** The full results generated by PerformanceXpert can be found here: http://performancexpert.com/report/8989. ***

It appears that the AWS resource could sustain a load of up to about 50 concurrent users before it got affected by the load.

What was the reason the load had this affect on the server? Actually the answer was quite obvious.

The bottleneck created by the bandwidth is clearly visible. It is not surprising that once the bandwidth limit was reached the response time started to climb up. In this case, it appears that the bandwidth provided by AWS was 550MB per minute which is 73Mbps.

Although bandwidth is the obvious suspect to create the bottleneck, it is important to verify that it is actually the cause. To confirm that the found bottleneck was actually due to a bandwidth limitation we ran the same test only with a smaller file. After running the two tests we will compare the maximum bandwidth and hit ratio to determine which is responsible for the bottleneck.

The last test generated 270 hits per minute.

For the next test we used 3MB instead of 4MB file.

*** The full results generated by PerformanceXpert can be found here: http://performancexpert.com/report/8994. ***

The bandwidth bottleneck was still eminent. In this case again, it appears that the bandwidth provided by AWS was 550MB per minute.

Only this time when checking the hit ratio, we see a different figure.

Looking at the hit ratio graph from the last test, we can see that the hit ratio reached a capacity of 350 hits per minute as apposed to the 270 hits per minute figure that was reached in the previous test. As the maximum bandwidth was the same for the two tests and the hit ratio varied, it is clear the the cause for the bottleneck is in fact bandwidth limitation.

Rackspace Results
*** The full results generated by PerformanceXpert can be found here: http://performancexpert.com/report/8986. ***

The resource hosted at Rackspace provided a lower performance level in terms of bandwidth compared to the one hosted at AWS . It could only sustain about 20 concurrent users before the load presented its affects.

The reason, again, was quite obvious.

In this case, it appears that the bandwidth provided by Rackspace was 160MB per minute which is 21.3Mbps. The last test generated 80 hits per minute.

To verify that the cause for the bottleneck is bandwidth limitation, again, we ran the same test only with a smaller file.

*** The full results generated by PerformanceXpert can be found here: http://performancexpert.com/report/8992. ***

After running the same test with a smaller file, 3MB instead of 4MB, we saw that the bandwidth bottleneck was still eminent. It appears that the bandwidth provided by Rackspace was 160MB per minute.

When checking the hit ratio, we see that maximum hit ratio is just over 100 hits per minute as apposed to 80 hits per minute in the previous test.

Here again, the last test confirms that the bottleneck results from a bandwidth limitation.

Conclusions

It is hard to compare “apples” to “oranges” when you try to benchmark two cloud providers. The resources are not similar. For the external novice user, the Micro instance of AWS has the greatest resemblance to the 512MB server of Rackspace.

Applying the same load scenario in order to test the bandwidth limitation generated the following results:

AWS provided bandwidth on a type Micro instance was: 73Mbps
Rackspace provided bandwidth on a type 512MB server was: 21.3Mbps
*** These figures are not absolute and can vary according to your test environment. ***

The goal of the test above was to illustrate a way to find a bottleneck created by bandwidth and not necessarily state the bandwidth limitations provided by each provider. Numerous parameters can affect the results of the test. Therefore real bandwidth figures can easily be different than the ones mentioned in this test. To know real live data, we suggest approaching the cloud providers themselves or testing on your own."

A good blog by Alon Girmonsky.
http://performancexpert.com/blog/benchmarking-amazon-web-services-aws-vs...