The following is an article from stackoverflow.com about user loadtesting.
"I'm working on a user load testing application for web servers and I'm trying to implement a feature for automatically ramping up the maximum number of "users" that a server can handle. I want to spawn test users until some threshold for the average response time and/or http request failure ratio is met, and then I want to kill/spawn users until a stable state just below the thresholds is found.
Essentially, I want to find the maximum stable number of concurrent users that still meets the requirements, as fast as possible.
I can of course figure out an algorithm for this myself but I'm thinking that there might be existing ramp up/ramp down algorithms that I could use. If anyone has knowledge on this I would love if you could point me in the right direction!
link|improve this question
This depends a lot on what's going on and if the system begins to decay gradually or if there is a discrete drop in performance (e.g. "healthy" -> "dead").
The following is from SmartBear Software about loadUI for loadtesting.
"loadUI. The art of Load Testing.
They say that beauty is in the eye of the beholder. We'd say it's in loadUI.
loadUI is a free and open source tool that deliver a visual, drag-and-drop Load Testing experience – in real-time. loadUI lets you create, configure, and update your tests while running them, making your testing interactive and flexible. Add to that the perfect fit with soapUI, and you’ve got yourself a package full of testing nitroglycerin.
Distributed Testing in Real-Time.
Before loadUI, load testing was static and counter intuitive. You had to first create the tests and then run them to the end before you could update them. With loadUI you can create, configure, and update your tests in real-time while the tests are running.
Elegant and Intuitive Design
loadUI is the first software to deliver a visual, drag-and-drop load testing experience. It is also the first software to fully eliminate manual testing and scripting. Everything – from test design to distribution to analytics – is designed with simplicity and ease-of-use in mind. The first time you start loadUI, you’ll know how to use it by instinct.
Scalable, Real-World Testing
A press release from marketwire, announces new version of regulatory reporting solution by Lombard Risk Management.
"Lombard Risk Announces New Version of Regulatory Reporting Solution
- Version 5 of STB-Reporter: Web-Based With Stress Testing and MIS
LONDON, ENGLAND, Sep 06, 2011 (MARKETWIRE via COMTEX) -- Lombard Risk Management plc UK:LRM -3.27% ("Lombard Risk"), a leading provider of integrated collateral management and liquidity, regulatory and MIS reporting solutions for the financial services industry, announces a new version of its regulatory reporting solution - using web technology, with significantly enhanced and many new features - as a key component of its integrated solution.
The increase in demand for regulatory compliance post financial crises is a major challenge for firms around the world. Lombard Risk's regulatory reporting solution meets those requirements, automating the process end-to-end from data collection to electronic output. The Lombard Risk regulatory reporting solution is used by more than 250 firms around the world; over 40% of the UK-based financial institutions use it to submit FSA and Bank of England returns; and last year Lombard Risk saw 30+ firms use its regulatory reporting, with LISA for liquidity scenario analysis, to meet the FSA's tough new liquidity regulations.
The following is a blog from Joe Colantonio about "performance testing basics-four steps to performance Nirvana"
We all know that performance testing can be frustrating. So many things can go wrong, and many of them are hard to find. No worries. Take a deep breath. Breathe in and Breathe out. Become one with your favorite performance test tool and follow these four steps to experience performance nirvana.
Run & Measure it
Fire off your tests and make sure you're getting repeatable results. NEVER go by the results of just one test. Going by the results of one test is like watching just one quarter of a football game and thinking you know who won the game. Bad idea. You want to run at least three sets of the same exact test without changes before comparing results.
After running the same exact test many times, take a look at the results and ensure that each run had approximately the same results. If they are not roughly the same, stop and find the cause. This is key, because if your results have more than a 5% difference between them and you make a tuning change, you wouldn't be able to determine whether change helped or not – since your baseline is unpredictable to start with. For my first round of testing, I usually start with brief, 15 minute, one & three user tests.
The following article is a press release from marketwire about performance testing by AMAX.
"AMAX ( www.amax.com ), the leading provider of high performance computing (HPC) and comprehensive virtualization solutions, announced at its booth # 553 during VMworld 2011, the joint collaboration with VMware to extensively test and validate a virtualized Hadoop Distributed File System (HDFS). VMworld, one of the largest virtualization-focused events of the year, is taking place at the Venetian Hotel in Las Vegas, Nevada.
High performance virtualized file systems are vital to organizations looking to achieve scalability, increase performance, and continually accelerate innovation. With rapid adoption of virtualization technology and the massive increase in popularity of Hadoop HDFS, organizations are truly able to lower costs and increase file system reliability of their virtualized IT infrastructure. Design expertise and implementation continues to be a dilemma for those looking for a performance optimized reliable way to store large files and run work across multiple nodes.
The following is a good article about load testing and stress testing from webperformance.
With more than 45,000 members, New York Road Runners (NYRR) is one of the world's leading running organizations. NYRR runs a year-round calendar of more than fifty races, including the famed ING New York City Marathon. Web Performance has performed NYRR's load-testing services for the last three race seasons, each time ensuring the websites and servers designed to handle New York City Marathon traffic are in the same peak condition as the runners themselves.
To help site visitors follow the runners' progress during a race, Web Performance set up the NYRR application to interface with Google Maps. This up-to-the-minute functionality, combined with an enormous amount of traffic over a very short duration, means the New York City Marathon site falls into one of the worst-case scenarios for server load and website performance.
The following article is from The BrowserMob Blog about load testing.
"Stay in the driver’s seat and in control! When launching a new website or application there are many things to consider to get ready to go “live” – and one of the most important aspects is load testing.
Simply put, load testing helps you determine how many customers the website or application will support – before you potentially find out the hard way (i.e. when users actually come to your site and you have to scramble to make last minute capacity improvements).
Following our blog series on Load Testing Best Practices, Webmetrics will be hosting a Twitter Chat for you to ask any and all questions about load testing.
More details are below:
Who: Joel Weierman, sr. manager professional services, for Webmetrics
What: Twitter Chat hosted by @Webmetrics, Joel Weierman, to facilitate any questions around load testing best practices, including planning, configuration, scripting, execution and analysis.
Where: Twitter – hashtag: #WorryLessLoadTest
When: Wednesday, September 14th from 4:00 pm – 5:00 pm ET
How: It’s simple – just ask your questions by including the #WorryLessLoadTest hashtag through Twitter. You will see the questions tweeted out from @StephanieMktg, and Joel Weierman will promptly respond from @Webmetrics
Twitter Hashtag: #WorryLessLoadTest
The following is a press release from realwire.com. The article is about CloudSigma Ag and loadtesting.
"CloudSigma AG, a leading provider of powerful cloud servers is pleased to welcome Neotys, an innovative provider of load testing harnessing the elasticity of public clouds for its customers through its NeoLoad cloud testing solution.
Patrick Baillie, CEO of CloudSigma commented 'I'm delighted to see Neotys really taking advantage of the true scaling capabilities of our cloud in a dramatic way. We also designed our on-demand billing to meet bursty compute needs so that Neotys has exceptional purchasing efficiency when deploying load tests within our cloud.
100 servers in under 3 minutes
A critical performance characteristic for NeoLoad is provisioning time. Their customers wish to minimise lag between order time and load testing. CloudSigma worked closely with Neotys to assist them with an optimal implementation that can deliver on this key metric for them with their NeoLoad product.
For example, NeoLoad can now spin up 100s of cloud servers on the CloudSigma cloud for customers in under 3 minutes.
The following is an article by Matt Perdeck from ASP.net performance.
Load testing web sites with StresStimulus, part 1 - Getting started
Load testing web sites with StresStimulus, part 2 - Advanced features
It is a good idea to load test your web site before taking it live, to make sure it won't break or perform poorly under load. Unfortunately, load testing tools are either very expensive or hard to use, or both (my book ASP.NET Site Performance Secrets discusses two load testing tools, WCAT and Visual Studio 2010 Ultimate). A new load testing tool, StresStimulus, is claiming to change all this.
StresStimulus is a fairly simple load tester that integrates directly into Fiddler, a free tool that lets you inspect HTTP messages going to and from your browser. As you will see in this article, that makes it easy to record load testing sessions and to inspect message generated by StresStimulus and their responses from your site. Additionally, it correctly handles hidden fields such as ViewState, which makes it attractive for load testing classic ASP.NET web sites.
This article first lists the features of StresStimulus, and then shows how to get started with your first load test. Part 2 goes into more advanced topics such as parameterization.
Load testing basics
The following is posted by Katherine Meyer on website V3im.com. It is an article about load testing. "Does your load time measure up"
"So you have a super cool website with great content, a simple and well organized structure, killer graphics and search engine optimized content?
While all this is great, one of the most important things (aside from making sure your SEO is spot on) is your site load time. We cannot say it enough – check to make sure site isn’t going to under perform because it takes forever to load.
I know, I know. We wrote about this a few weeks ago but we can’t stress enough how important site load time is to a good user experience.
I saw a stat that actually inspired this post: 47% of consumers expect a website to load in two seconds or less. Even more compelling, if it takes longer than three seconds for your site to load you could be looking at a 40% consumer abandonment rate. Which is potentially a lot of business leaving your site for a faster, but maybe not better competitor. So let’s make a pact, please promise to add a website load test to your regular maintenance routine for your site and we promise to stop talking about this…for now.
This article was written by Christopher Merrill, from Web Performance, about scenarios for load testing.
"This article discusses how to build a portfolio of scenarios for load testing automation. You have rarely only a single scenario to test, so you have to decide which scenarios to include in your test plans. Criticality, frequency, difficulty and verifiability are the four key factors that should be considered to determine the scenarios to automate.
Author: Christopher Merrill, Web Performance
Originally published in September 2010 on Web Performance Blog
What to test?
Occasionally, we encounter customers who have only a single scenario to test. For example, one client developed an application in which the only scenario of interest involved a user registering him/herself with the system and scheduling an appointment. When this happens, the tester may devote their efforts to the accurate simulation, testing and analysis of this single scenario. You are unlikely to be so lucky.
In any moderately complex system, there are dozens or even hundreds of scenarios that are candidates for load testing. You are not likely to have the time or resources to test them all. As a result, you must make some tough decisions about which scenarios will be included in the test plans.
There are a number of key factors that will raise or lower the importance of each scenario:
The following is an article, written by Kevin Shalvey, for Investors Business Daily.
"Keynote Systems (KEYN) has gained from the sale of its top competitor, as well as its mobile push.
Compuware (CPWR) bought Keynote's archrival Gomez in late 2009 for $295 million, and Keynote investors were keen to track what the new owner would do with it.
But instead of ramping up Gomez, Compuware seemed to step back, says Curtis Shauger, an analyst with Caris & Co.
"The behavior of Gomez after the acquisition has been a de-escalation of price competition," Shauger said. "That's allowed (Keynote) to more fully benefit from the more fundamental growth of the Internet, where they had been losing (market) share to Gomez the last several years."
Gomez's services have been expanded since the acquisition, responds Mark Hillman, vice president of strategy for Compuware. He says the company is seeking new customers for its "load testing" services, which predict how websites will handle future traffic.
Keynote, too, tracks how fast websites load and perform — both on conventional computers and on mobile devices — for corporate and other enterprise clients. This business, what it calls its Internet business, accounts for more than half of its total sales.
But Keynote began its mobile push about five years ago, and now that business accounts for almost half of its revenue.
A press release, from Business Wire, says " OpSource launches validated Cloud Solutions program and certifies fourteen cloud applications."
"OpSource expands partner program to bring enterprise-class solutions to market quickly. OpSource, Inc., the leader in enterprise cloud and managed hosting, today announced its Validated Cloud Solutions Program, naming an initial set of 14 partners that are leveraging the OpSource platform to deliver enterprise-class cloud-based services. With this certification, enterprise customers are assured that an OpSource-validated solution has been tested on the OpSource Cloud. Validated partners gain access to a secure cloud platform and expand their market reach.
The initial Validated Cloud Solutions partners participating in the program include:
-- ANX -- secure cloud access
-- aPersona -- anonymous 2nd factor login confirmation
-- AppDynamics -- application performance management
-- AppFirst -- application problem resolution
-- Aryaka -- WAN optimization and application acceleration
-- Compuware -- web application load and performance testing
-- CTERA Networks -- cloud storage appliance
-- Evolven -- cloud configuration management
-- Librato -- virtual server resource monitoring and management
-- NRG Global -- application load testing
-- nSolutions -- visibility and control for cloud compliance
The following is an article from EMC. The website for EMC is www.EMC.com.
"EMC Corporation EMC +2.73% , the number one choice of customers for information infrastructure in VMware virtual and cloud environments according to a recent Wikibon survey, today announced that it has shattered the performance limits it previously set for storage throughput and bandwidth in a VMware vSphere® 5 environment. With performance and scalability parameters now stretched out even further, EMC is accelerating customers' journey to the cloud by removing potential performance-oriented barriers for organizations as they virtualize their mission critical applications.
The following is another good article from BrowserMob about load testing.
"Several holiday seasons ago, Shopatron experienced a number of challenges in
dealing with their own success. In the words of Dave Cumberland, VP of engineering. “There were too many times when we just held our breath, hoping we
could handle loads 2 to 10 times larger than expected,” he said. “We needed to
up our game.”
Building Confidence through Load Testing
To gear up for the 2010 holiday rush, Cumberland chose to load test with
BrowserMob, a Neustar Service. He focused on how new website features, for
example changes to checkout, withstood the kind of increased volume November and December bring.
“We target 99.999% uptime,” said Cumberland, “along with average page loads
of 2.5 seconds. BrowserMob allowed us to test each website feature under really high loads. We started by using a model of expected user behavior—consumers, manufacturers, and retailers—and testing features individually for
each type of traffic. We see higher traffic at certain times of the day, so that was
a factor too. We tested lots and lots of scenarios and finally, to be really sure, ran
a ‘perfect storm’ scenario with traffic spikes from all our users.”
The following is a good article from browsermob.com., written by Joel Weierman, about load testing best practices.
"We just wrapped up our five-part Load Testing Best Practices series. We hope you find these steps simple and convenient as you embark on your next load test.
When planning a load test, make certain you take all variables into account. For instance, before you load test a website or application, you should ask yourself the following: How much load should I test with (stress test vs. load test)? How many tests should I run? Should I test on a real or virtual browser? Once you have answered these, you will be ready to start and configure your test.
Configuring your load test is particularly important. Load tests vary depending on where your traffic comes from and how many users your website sees daily. By working through this during the configuration phase, you will determine whether you need to run a high volume or low volume test, and you can move into scripting.
Websites and potential scenarios have become increasingly complex, making load test scripting more complex as well. Therefore, it’s important to make your comprehensive script organized, robust and reusable. Once scripting is complete, you can execute your load test.
neustar Insights, in Webmetrics Performance, has a good article written by Erin Bush about load testing. It is as follows:
"Ahh load testing. We all know we need to do it, but how many of us–especially those of us with relatively small sites–actually do it? Testing allows you to determine how many customers your website or application will support–before you find out the hard way (when users hit your site, crash it, and you have to scramble to make last minute capacity improvements).
I have to admit that once during my tenure as a web product manager, I didn’t test a redesigned site before it went live. At all. (Ack, did I just say that out loud?) I’m sure you can image that the result was not pretty. I had 48 hours of up and down time while I incrementally upgraded my hosting service to allow for the traffic load. True story.
I’m no longer in charge of such things, but I can tell you, I’d never make that mistake again.
Even high-profile sites can crash under traffic pressure. Just yesterday, SxSW launched their much-anticipated and annually recurring Panel Picker. This is the platform on which they ingeniously allow the interested crowd to source (at least partially) their panel choices for the conference. The site went up at 4pm EST and an hour later, it was down.
SV-Load Test Service has a good article on load testing. The link to their website is www.scivisum.co .uk/web-load-testing-service.
"SciVisum Load Testing Service gives you access to expert testers with specialist knowledge to spot problems quickly and provide insights into the root causes of problems. SciVisum’s Load Testing Service offers guaranteed expansive and complex testing sessions often unobtainable from self-servicing due to departmental time constraints.
Let us do the hard work; Our test engineers deliver Load Testing remotely, we write the scripts and perform the tests – no hardware, software or monitoring agents are necessary and no engineering time is lost - capacity planning without the capacity pain.
The service includes an individual, clear analytical report, meaningful to all departments of the business team and includes a full debrief of the whole website testing.
Load Testing Data Sheet PDF
Self Service Load Testing Portal
Benefits to Technical and Commercial Teams
Which Service Is Right For You?
Which Service? Summary
SV-Load Test Portal
Utilize SciVisum’s high quality hardware and resources
Complete control over your website load testing requirements
Ability to re-run tests ad-hock after making periodic adjustments the website code
On-demand and out of hours testing
Access to information in real-time
An article by Simon Brown, on codingthearchitechure.com website, is about load testing for developers.
"September sees me picking up the travel baton again and one of the things I'm doing is a half-day session at the Skills Matter Progressive .NET Tutorials in London on the 6th.
The session is called Load Testing for Developers and it's exactly what it says on the tin ... an introduction to load testing for developers. You can be as progressive as you like with the .NET platform, but performance and scalability problems can still rear their ugly heads regardless of the technology you're using.
Have you ever built a software system and your users have complained that it’s too slow? I have; debugging live performance and scalability issues with business sponsors watching over your shoulder isn’t fun! Load testing is an often forgotten and seemingly difficult task that many people shy away from but a basic level of load testing is often enough to give you confidence that you've satisfied expectations regarding performance and scalability. This tutorial will look at how to load test your website and you’ll learn:
What load testing is all about.
How to implement a load testing script using the free and open source Apache JMeter tool.
How to run a load test and monitor the environment (the load testing client and your website server environment).
The following article written by Gregory Milby, on the Attacker.com website, is about the Apache Benchmark load testing tool.
"Apache has a great load testing tool that appears to rival the best out there. It can test against concurrent connections, number of loads per connection, and give understandable results – actually useful! Anyone who develops wants to know how much stress (how many visitors) their application can withstand and how it will perform under a real world load.
My OS is Ubuntu (Natty atm), and apt makes it easy to install by just typing:
gmilby@mini64:~/Dropbox$ sudo apt-get install apache2-utils
1. Number of simulated connections – this simulates the number of visitors to your webapp, and if there are bugs and code errors, it will result to a very poor performance in actual deployment which can slowed down if there are a lot of users in your website.