Stress Testing Repository: Websites & Computers

I read an article today by Esther Shein that shared some good information about websites handling peak capacity for retailers during the holiday season. She warns us not to recreate "the cautionary tale of Target" whereby unprecedented number of online shoppers inundate a site causing it to crash. In Target's case, the site crashed twice.

According to a spokesperson for Target, "We are working around the clock to ensure that our site is operating efficiently and delivering an exceptional guest experience that's reflective of Target's brand.'

These holiday retail website failures are nothing new. In 2005, it happened to Best Buy and Circuit City when they experienced very large spikes in traffic.

Esther offers these tips for good performance of your websites during the holiday rush:

"Test early to make sure there's enough capacity and that loads are balanced correctly.

"Make sure traffic predictions are vetted by enough internal stakeholders so you're not guessing what your peak might be.

"Check everything from application servers to your network firewall, all the way down to the speed of your Internet connection -- and check more than twice.

The following article was found in the Boston Globe written by Michael B Farrell. It is about online retailers being prepared for the Black Friday rush. This is where good stress testing, load testing, and performance testing is important.

"If only it had an “Easy’’ button.

Instead, Staples.com, the second-largest online seller, behind Amazon, had to get ready for the holiday rush by boosting staff, adding training for its customer service representatives on the telephone and on Twitter, and creating a special Internet center with holiday deals on everything from laptops to ballpoint pens.

The Framingham retail giant, which averages 300,800 online visitors a day, is expecting three times its normal Web traffic this weekend.

With Internet shopping expected to reach a record of nearly $60 billion this holiday season, a cluster of Boston-area companies, from Akamai Technologies Inc. to Sapient Corp., are working behind the scenes to make sure retail websites can handle the sudden surge of traffic. One of the biggest challenges will be managing sites over multiple platforms. Mobile shopping is expected to climb this year, with at least 30 percent of adults using their smartphones to comparison shop, according to Forrester Research of Cambridge.

The following is an article about performance testing written by Charles Rich. He is telling all business websites to get ready for Black Friday/Cyber Monday.

"Black Friday is just the beginning of that time of year when most businesses finally start to turn a genuine profit. The business your IT department supports is depending on you to keep the sales process flowing smoothly and swiftly. What practices can you put in place today to ensure your mission-critical applications stay up and running throughout the busy holiday shopping season?

1. Use application performance monitoring (APM) to understand the performance of your applications.

Each business depends on its own unique set of applications. This makes using an APM solution to understand application performance and prevent user impact very important.

APM allows you to load test in pre-production to see what will happen when multiple business transactions are being executed using different applications and then continue to use the same tool in production. Of course, load testing can deliver varying results, so it’s also important to use APM to evaluate whether your applications have any hidden interdependency conflicts in real-world execution.

The following is another good article written by Betsy O Zikakis on performance testing.

"It's wise to look at it as an additional chance to bring in new business on top of it being one of the best types of advertisement for your brand and whatever products or services you deal in when you're running a website for your business. Not only is it crucial that your page contain everything your visitors need to know, and how to attain it, but it should also be functioning to the best of its ability, allowing for the ultimate customer experience, without problems, while people are on your webpage.

Not only costing you money to rectify, but also losing you profits at the same time, there are a lot of things that might, and are likely to go wrong, and the last thing you want to do is be caught unaware, making it even harder to come back from what may turn into an expensive issue at some point. Letting you monitor the amount of traffic you are able to maintain as well as a range of other analytics that should be of a huge help to you as your company expands, performance testing is an essential tool for both your regular website, and also your mobile applications, letting you adapt your business plan to cater to your findings.

Another good article by Elliott on performance testing.

"The TelQA Testing Series
This article is part of a series of documents which cover the basics of performance testing. See www.telqa.com for other articles in this series and additional downloadable testing resources.

Overview
This article provides you with an introduction to determining the aims and objectives of your performance tests.

During the course of this article we will look at the performance testing aims for:

System testing – performance testing for a complete system/application
Development phase testing – performance tests that support application development
System Testing
To begin with, let’s first consider performance tests which target a complete system. We can split the performance testing aims into four broad categories:

Quantifying the overall performance level of a system
Comparing the performance of a system against known performance thresholds (e.g. Service Level Agreements)
Assisting with performance tuning or the removal of performance bottlenecks
Assessing system scalability

Elliot, from TelQA, has written some good blogs about performance testing.

"The TelQA Testing Series
This article is part of a series which cover the basics of performance testing. See www.telqa.com for other articles in this series and additional downloadable testing resources.

Overview
This article presents a general testing strategy that will help you create and deploy successful performance tests. Although testing methods must always be flexible enough to adapt to individual projects, having a general structure can be very helpful for both experienced testers and those just getting started.

The Performance Testing Process
A typical performance test consists of four main stages:

1. Test Planning
Determine what you want to achieve from the test
Define the type of test(s) you are going to perform
Select a suitable test environment
Specify inputs (user types and number of users) and expected outputs (performance data collection, page timers, transaction timers etc.)
2. Test Development
Capture realistic user interaction for each user type and save as editable scripts.
Model captured scripts to create reusable virtual users (VUs)
Select appropriate performance monitors for target application
Create test using the defined proportion of VU types. The test should also include realistic load application, delays etc.
3. Test Deployment

Elliot, from TelQA, has written some good blogs about performance testing.

"The TelQA Testing Series
This article is part of a series which cover the basics of performance testing. See www.telqa.com for other articles in this series and additional downloadable testing resources.

Overview
This article presents a general testing strategy that will help you create and deploy successful performance tests. Although testing methods must always be flexible enough to adapt to individual projects, having a general structure can be very helpful for both experienced testers and those just getting started.

The Performance Testing Process
A typical performance test consists of four main stages:

1. Test Planning
Determine what you want to achieve from the test
Define the type of test(s) you are going to perform
Select a suitable test environment
Specify inputs (user types and number of users) and expected outputs (performance data collection, page timers, transaction timers etc.)
2. Test Development
Capture realistic user interaction for each user type and save as editable scripts.
Model captured scripts to create reusable virtual users (VUs)
Select appropriate performance monitors for target application
Create test using the defined proportion of VU types. The test should also include realistic load application, delays etc.
3. Test Deployment

The following is a well written article by Nathesh, TMCnet contributor, on load testing.

"Compuware (News - Alert) Corporation, a technology performance company, has unveiled its Compuware Performance Index, which offers a real-world view into the performance of top U.S. retailers' Web and mobile sites and highlights user satisfaction with their online experience in the upcoming holiday season.

Cyber Monday (News - Alert) 2011 is on target to be the single highest online shopping day in history and retailers are expected to equip themselves in order to meet the challenges associated with surging demand, particularly maintaining superior user experience and application response time.

The company explained that last year alone online shoppers spent $1 billion on Cyber Monday in what was then the largest online purchase day in history. This is a time when milliseconds of delay can make the difference between a customer completing a transaction or abandoning the site to shop elsewhere. In short, Web and mobile performance are critical business issues.

I have another good article on load testing written by Betsy Zukakis.

"Corporate networks are typically busy enterprises composed of lots of data and software, which can be complex to manage and maintain on a regular basis.

The volume of traffic that an average network must bear ebbs and flows depending upon the hour of the day and the needs of those who depend on the system. In most cases, the workday hours are when the most demands are being placed on the system because that’s when everyone in the company is on duty and needs to get things done.

But if your firm is an international one, there’s a good chance that some group somewhere on the planet is going to be working at full dudgeon 24 hours out of the day. It’s not only helpful, but mandatory that you test your network to determine how much pressure it can bear before the system begins to break down.

A well-established approach of checking your network for possible weaknesses that could cause system overloads and server crashes is called a load test. Network managers can see how well the corporate enterprise holds up when they generate false workloads on the network to test how particular parts react.

Joelhunbillz has written a good blog on load testing. It goes as follows:

"Poorly performing websites are connection between negligence regarding the proper upkeep of an internet site. The effects these malfunctioning websites may cause negative impacts regarding website Return of Investment (ROI) and sales production of the advertised product. Lack of appropriate tests online leads to high odds of site errors and poor loading performance. This will likely pose grave effects on the web based business like loss of potential costumer sales and partners, reflect an ineffective corporate image, reduced possible profits and oversized hardware.

The following question was asked of Micro Focus Knowledge Base.

"Problem:
How can I loadtest an application if I am unable to record at the protocol level?
Resolution:
There are a number of approaches that one can consider when a protocol approach is not feasible. This may be the case when the protocol used by the SUT is not directly supported or where client-side logic, proprietary encryption, complex session handling, or custom standards for data exchange between client and server are used.

The approaches include:
Browser Driven Load Testing
Since SilkPerformer 2011 you can use real Web browsers (Internet Explorer) to generate load. In this way, you can leverage the AJAX logic built into Web browsers to precisely simulate complex AJAX behavior during testing. This powerful testing approach provides results that reflect real-world end user browsing experience, including rendering time and protocol-level statistics. more information on BDLT can be found in the BDLT tutorial at START | PROGRAMS | SILK | SILKPERFORMER |DOCUMENTATION | TUTORIALS | BROWSER DRIVEN LOAD TESTING

GUI Level Testing

The following is an article on load testing, written by Anil Sharma, TMCnet contributor.

"Shunra (News - Alert), a provider of solutions for application performance engineering, and Keynote Systems, a solutions provider for Internet and mobile cloud monitoring and load testing, announced that they have combined the capabilities of Shunra's vCat networking emulation solution with Keynote's KITE (Keynote Internet Testing Environment) Web application performance tool to offer a joint solution.

The new solution can be used to offer meaningful and actionable results in pre-production Web application testing environments and sort out potential performance failures that result from application design flaws and network infrastructure bottlenecks, prior to production deployment.

The companies explained that with Keynote KITE, vCat delivers a comprehensive performance test environment that empowers one to determine precisely how their Web application will perform for end users across WiFi (News - Alert), broadband and mobile connections. This combined solution delivers the data users need to proactively validate application performance, identify and remedy performance issues, and ensure a quality user experience.

The following is a good article on load testing, written by Betsy Zukakis.

"For your business to succeed today, you must keep up with technology. That means being part of the cell phone app world. The revolution in cell phone applications is continuously growing. To assure your start-up of success, you need to take part in this technology and create and develop mobile apps of your own. In a tough marketplace, your mobile application will have to grab attention and be well designed. It will be vital that your app consistently works well for users. To ensure this, you need to locate a professional app testing service.

Application testing is the best place to put your money if your business is on a budget. Your app will not be successful if it does not work well. Feel confident and certain about the app testing firm you hire. Ask many questions before you make a selection. Your first issue should be a quick turnaround. You will want full reports so that you know what areas your application company is testing. These are simple questions that will get your service completed on time with a full range of testing.

I found a good blog on performance testing posted by Ashish.

"Performance testing is required in every field. Without doing some validation for performance testing, quality and success cannot be said to be achieved. Similarly in the field of computer science and engineering, performance testing in software applications is of great importance. Performance testing is done to find out the execution speed and time of the program, and to ensure its effectiveness. Software performance testing basically involves some quantitative tests that can be performed (in a computer lab for example), number of millions of instructions per second (MIPS) and measurement of response time. It also involves some tests for qualitative aspects such as scalability, interoperability and reliability.
Stress testing is carried out simultaneously with performance testing. So finally we can define software performance testing as a testing in software engineering that is done to find out the measure of some qualitative or quantitative aspect under a specific workload. Sometimes, it is also used to relate other quantitative and qualitative aspects such as usage of resources, scalability and reliability. Software performance testing is a concept of performance engineering which is very essential to build good software.

I found this to be very informative on load testing.

"Poorly performing websites are connection between negligence depending on proper maintenance of a web site. The effect of the malfunctioning websites could cause negative impacts when it comes to website Return of Investment (ROI) and purchases creation of the advertised product. Not enough appropriate tests online brings about high probability of site errors and poor loading performance. This may pose grave effects on your own online business such as loss of potential costumer sales and partners, reflect an ineffective corporate image, reduced possible profits and oversized hardware.

I found this blog to be interesting on the subject of load testing.

"Poorly performing websites are results of negligence based on the proper repair off a site. The consequence these malfunctioning websites could potentially cause negative impacts regarding website Return of Investment (ROI) and sales output of the advertised product. Absence of appropriate tests on-line leads to high chances of site errors and poor loading performance. This could pose grave effects with your web business including loss in potential costumer sales and partners, reflect an ineffective corporate image, reduced possible profits and oversized hardware.

I found a very interesting blog, written by Scott Price, on throttling web performance.

"Have you ever wondered why your web access speed is sometimes blazing fast, while at other times you are waiting for response at a slow-as-molasses pace? Yeah, me too. I've been obsessed with web performance and page speed since 1996, so I'm sensitive to slowdowns.

Would you be surprised to find out that telecom/cable providers are intentionally and deliberately slowing down web access? Yeah, me too. Empirical data from test results show that at least one provider is chocking users access 85% of the time! U.S. providers are slowing you down twenty-three percent (23%) of the time. Globally, it's even worse - 32% of the tests show provider slowdown.

There is software that proves it. Internet access operators don't want to talk about it openly, but when you dig into their standard service contracts, it's confirmed. They "throttle" your speed whenever they want based on their needs...not yours.

NY Times Sheds Light on Throttling Slowdown Sources

The following is an article by techopedia.com, giving its definition of load testing.

"Definition - What does Load Testing mean?
Load testing is a software testing technique used to examine the behavior of a system when subject to both normal and extreme expected load conditions. Load testing is generally performed under controlled laboratory conditions in order to distinguish between two different systems.
Load testing is designed to test the non-functional requirements of a software application.

Load testing is sometimes referred to as longevity or endurance testing.
Techopedia explains Load Testing
In the software development process, the term load testing is often used interchangeably with other forms of testing such as performance testing, volume testing and reliability testing. In simple words, load testing can be considered the simplest form of performance testing. In load testing, a system or a component is subject to varying load conditions, which are sometimes well beyond the normal limits, in order to determine the behavior of the system at peak load. This process is referred to as stress testing.

Load testing technique may be used in the following scenarios:
Testing an e-commerce website's shopping cart capacity
Testing hard disk drive capability to read and write as per its specifications
Testing an email server to handle email traffic

HP has a blog on Marketwire, Inc. about performance testing. The blog is as follows:

"HP today announced that organizations worldwide are creating a competitive advantage by accelerating the reliable, secure delivery of applications and services with the HP Application Lifecycle Management (ALM) 11 software platform.

HP ALM 11 allows businesses and governments to automate the workstreams associated with application modernization -- including requirements management, project planning and tracking, development management, quality assurance, defect resolution, performance validation, and release management.

HP ALM 11 is a component of the HP IT Performance Suite, an enterprise performance platform that enables IT management to improve performance with operational intelligence. The suite delivers a comprehensive view across all IT assets, automates IT management and adjusts IT performance to meet specific business goals.

The following is from a blog by THN News about load testing.

"Without a doubt, public cloud has been in demand as more and more companies look to innovate in the continuously popular area. On Thursday, another provider added its own cloud solution.

Colocation and IaaS operator Phoenix Nap announced its Secured Cloud product. It’s an offering that combines some pretty high support names including networking supported by Cisco, server virtualization backed by VMware, data storage through NetApp, and load balancing thanks to f5.

The company says it’s currently accepting sign-ups for the solution and the cloud appears to be compatible with operating systems such as Microsoft Windows Server, CentOS, and Red Hat.

Commenting in a press release by the company, Pheonix NAP president Ian McClarty stated, “We are very excited to release Secured Cloud to market.”

“This public cloud offering we believe is unlike many other solutions in that it combines true enterprise-grade technologies on the back-end with a user interface that is simple enough for non-technical users and robust enough for tech-savvy individuals,” he went on to say.

Meanwhile, McClarty also discussed the platform’s user interface, stressing specifics such as “ease-of-use” and the cloud’s “pay-as-you-go pricing model.”