Stress Testing Repository: Websites & Computers

Alon Girmonsky @ Performance Xpert has written a good article about how to identify performance limits.

"Adaptive Load Scenario - How to Identify Performance Limits
Posted Sat, 05/28/2011 - 12:02 by Alon Girmonsky
Every good test plan should include testing your application under different load scenarios such as: gradual load, stress, endurance, etc.

However before applying any load scenario it is a good practice to follow two very important steps:

Define your performance expectations
Identify your application's performance limits

XBOsoft of Software Quality Improvement had this to say about performance testing.

" Testing software performance used to be an afterthought, but today, it is a critical activity in software development projects especially for web-based software applications. As features build up from iteration to iteration and databases continue to grow, access times can expand geometrically.

John Crout had some things to say on this subject. He is with Quality Assurance/Load tester. "Being connected to a system won't be helpful unless they are all doing the most CPU/memory/IO intensive work available to users, at the same time. If its possible for the system to initiate something load-intensive at the same time, then this also has to be considered. What would I do?

"

When building a web site. it is always a good idea to load test it before putting it out to the public. Here is what mgalvin has to say on load testing:

"Building a web site is not terribly hard, especially with all the great tools available these days. However, building a high profile, high traffic, high performance web site is not quite so easy. It generally requires a team of experienced developers and system administrators each doing there part to ensure every part of the complete system is running optimally at all times.

I read a discussion thread today and a guy named AlkalineSolo asked a very broad question about what it takes to be a good load tester. That's a good question, but there are many books and tons of articles on the web that cover load testing, performance testing, and stress testing. I will share 3 of the best answers to AlalineSolo's question with you:

James Pulley gave a good answer. He said it takes longer than a few sentences.

http://activedirectorytools.net/exchange-load-generator

Exchange Load Generator is a simulation tool to measure the impact of MAPI, OWA, IMAP, POP and SMTP on Exchange Servers. You should use this tool in a test environment and not in your exchange production environment.

Exchange Load Generator allows you to test how a server running exchange responds to email loads. To simulate an email load your run this tool on client computers.

Running the tool on client computers will send multiple requests to the exchange server causing a mail load.

I went to an Association of Software Testing conference once. It was the typical hotel food and breakout sessions that you would expect. Some of the people there were brilliant. Some of them were highly ego-centric. A few of them were obviously considered to be "rock stars" in the world of software testing, and I wasn't prepared for the dynamics of discussions that ensued.

Scott Barber made an announcement that the Performance Special Interest Group would be meeting during the conference - at an off time in the evening. Sounded good to me, so I showed up. It was very disappointing. The first part of the meeting was spent listening to Barber pontificating on how he was so much better than the "tool jockeys" (his words) that know how to use LoadRunner because he understands the system architecture and all the aspects of performance engineering. The arrogant nature of his approach was hard to swallow, but I enjoyed hearing his perspective on the industry and where it was heading.

His view on performance testing is interesting, so I would like to share some of his recent post that he advertised on the front page of the AST newsletter. Seems like an abuse of power since he is the President or Executive Director or last man standing - not sure of his real title now. My understanding is the AST is barely surviving as an organization; thus, having any new content is probably a good thing for the group. The post isn't on the AST site though. Barber put it on his own site and then published a link as the main content in the newsletter. Oh well, that seems consistent.

The following is an excerpt from his post about defining context driven testing. It's only his opinion, not that of the AST. I'll leave you to judge if it has any real value to us stress & performance testers.

In their article entitled Best Practices for Speeding Up Your Web Site, the Yahoo Developer Network lists 7 categories containing 35 ways to make your site ready for stress testing.

Since about 50% of your site's daily visitors have an empty cache, your success in meeting their needs hinges on making your page streamlined and quick to load. Any of these performance improvements will pay off exponentially in stress testing when you are running 1,000+ concurrent users.

Visual Studio Load Test enables you to stress test your application throughout its development lifecycle with realistically modeled simulated load. Scenario and goal-based load testing ensures that your solution copes with real-world demands and behaves in a predictable manner.

LoadStorm is a web stress testing tool based on cloud infrastructure to make it low cost. You can build scenarios for hitting pages and submitting forms like login and e-commerce. Free for up to 25 users and scales to 100,000 concurrent users. Real-time graphs show response times, RPS, and throughput. $40 for 1,000 users.

eValid LoadTest is an Internet Explorer tool for web server stress testing. It performs functions needed for page timing and tuning, transaction monitoring, and realistic load generation.

SiteBlaster is a web site load and stress tool built for USGovXML. USGovXML is an index to publicly available web services and XML data sources provided by the US Government. When testing is complete, a report is available that can be viewed or printed. SiteBlaster simulates Internet Explorer web browsing functionality and is best used to test those sites that use URL query strings to pass data to its web pages.

LoadManager is a tool for specific industries: banks, telecom, and industrial companies. Multi-platform using Eclipse and Java. Consists of one controller and several agent modules that can be spread over several machines.

Loadea is a Windows XP-based load testing tool. Control and capture modules use C# for building test scenario scripts, schedule stress test execution, and XML for test data. The analysis module provides reporting capabilities.

LoadRunner is an enterprise performance testing tool with arguably the largest user base and most expensive cost. It can test most types of technology, and it is very feature-rich. However, it was written in the 1980's and carries with it some usability baggage from that era. It's the grandaddy of all stress tools.

IBM Rational Performance Tester is an enterprise class performance testing tool that has been around for nearly 25 years. Used by large companies with extensive testing needs for not only homegrown applications, but enterprise apps like SAP or Seibel because the tool has special extensions for those.

SilkPerformer is an enterprise load testing tool. It is an integrated performance testing solution that makes it possible to create powerful, realistic load tests for thousands of users running business scenarios across a broad range of enterprise application environments including the latest web 2.0 technologies. Some say it is nearly as expensive as LoadRunner.

JMeter - The king of desktop open source stress testing tools. JMeter is written in Java by the Apache Software Foundation. It can be used to simulate many users hitting a server, network or object to test its strength or to analyze overall performance under different load types.

OpenSTA - 'Open System Testing Architecture' is a free, open source web load/stress testing application, licensed under the Gnu GPL. Utilizes a distributed software architecture based on CORBA. OpenSTA binaries available for Windows.

Curl-Loader - (also known as "omes-nik" and "davilka") is an open-source tool written in C-language, simulating application load and application behavior of thousands and tens of thousand HTTP/HTTPS and FTP/FTPS clients, each with its own source IP-address. In contrast to other tools curl-loader is using real C-written client protocol stacks, namely, HTTP and FTP stacks of libcurl and TLS/SSL of openssl, and simulates user behavior with support for login and authentication flavors. The goal of the project is to deliver a powerful and flexible open-source testing solution as a real alternative to Spirent Avalanche and IXIA IxLoad.

The Grinder - a Java load testing framework that makes it easy to run a distributed test using many load injector machines. It is freely available under a BSD-style open-source license. It has a generic approach to load testing anything that has a Java API. This includes common cases such as HTTP web servers, SOAP and REST web services, and application servers (CORBA, RMI, JMS, EJBs), as well as custom protocols. Test scripts are written in Jython. A graphical console allows multiple load injectors to be monitored and controlled, and provides centralised script editing and distribution.

ApacheBench - (sometimes simply called "ab") a Perl API for Apache benchmarking and regression testing. Intended as the foundation for a complete benchmarking and regression testing suite for transaction-based mod_perl sites. For stress-testing server while verifying correct HTTP responses. Based on the Apache 1.3.12 ab code. Available via CPAN as .tar.gz file.

SearchSoftwareQuality's Stress Testing for Performance - Part 1

SearchSoftwareQuality's Stress Testing for Performance - Part 2

SearchSoftwareQuality's Stress Testing for Performance - Part 3

PerformanceWiki - Application Tuning Tips

AlertSite - Online Performance is Business Performance

InfoSys - Problems with Performance Testing in a Shared Environment

Web Performance Today - Performance and the Cloud

Lots of good blogs about stress testing are listed here. Check them out by clicking on the name at the beginning of each entry.

Web Performance Today is a company blog for Strangeloop. Most posts are written by Joshua Bixby, the company president. Despite the "strange" name, Strangeloop's blog is one of the best in our industry because they post about site acceleration solutions and causes of stress testing failures.

Dynatrace is a company blog that focuses on aspects of web application performance. Plenty of good information and posts are very frequent. This vendor offers an application performance management software package. That's why they have abundant info on application tuning, system performance engineering, monitoring, and web architecture.

LoadStorm is a company blog about stress testing from a web geek's perspective. Typically has good commentary, and they tend to put engineering discussions into common language. A dry sense of humor is blended in.

One question that is often asked is how hosters can benefit from the Windows Azure Platform. While the platform can be used to deploy many types of web apps we expect many partners including hosters to develop on top of the Windows Azure platform infrastructure. In this episode of Azure Lessons Learned I chat with Paul Lappas, VP Engineering at GoGrid and Mehul Shah and Madhavrao Pachupate from Blue Star Infotech. GoGrid has been working on a hybrid solution that builds on the GoGrid infrastructure to assist in development and load testing of Windows Azure applications.