Using open source tools for performance testing (Google video)

http://www.myloadtest.com/open-source-tools-performance-testing/

She uses JMeter for testing web-based applications through the GUI, uses The Grinder for API-based testing, and does not use OpenSTA because it only works on Windows.

Other points during the presentation:

You should use the same monitoring for Load testing that you use for Production monitoring (so you don’t have to account for the differences in load that a different monitoring system will put on the system).
If you are running Unix-based systems, don’t sustain CPU above 80%.
Google tracks a summary of every performance test in a central database. The database also contains information on every piece of software that is installed on the machines in the test environment.
If I am unfamiliar with the system, I don’t trust it. One of the things that I have realised is that
A) the system will fail in the place where they tell me that nothing could go wrong.
B) developers are totally delusional about their own software, and frequently they will just forget about things that they’ve done two weeks ago.
I run every test 5 times. I want to see that I have some sort of statistical consistency.
Performance testing should not be used as a tool to find memory leaks; but it can.
Performance testing without monitoring? Don’t bother. Why waste your time?
If you are going to do any performance testing, make sure that database sizes are somewhat realistic. They don’t have to be exactly the same, but they have to be the same order of magnitude otherwise the results you are getting are completely off.
Execute a stress test. Find how your system is failing. Find where it is failing. Do find out how the system handles overload. There are no good defence mechanisms against people out there, and you can’t predict sudden popularity (eg/ Google Earth).
Start a test after a decent warm-up period. Don’t start 100000 users all at once.
Quite often people don’t know about everything that is running on a complex system. Maybe there is a low priority process that is running with high priority. This can usually be fixed by niceing the process down. Quite often there are debug things that are still running also.
Monitor the machines that are collecting the monitoring data and the load generators (not just the system under test).
Performance Testing and QA is about risk analysis. If I believe it is high risk, I want to take a look at it.
When I am doing performance testing, the first thing I try to do is eliminate the network. I want to simplify my problem. I am interested in the machines, and my hope is that the network provided will handle everything I need. Once everything is profiled and understood, we will do some tests that include the network. If you can, put everything on the same subnet and same switch. It will make you a much happier performance tester in the first pass. Debugging networking problems is (not) fun.
(When talking about testing on smaller sized systems than Production). You can’t test on a 386. Extrapolation will kill you. You will run out of some resource that you never expected, and you can’t predict this ahead of time. For final validation, you really want to get some time on the Production hardware before it goes live. If the system is not being used for Production, it should not be that hard to get hold of it for a week or a weekend.
Find more open-source performance tools at opensourcetesting.org
There is another summary of her talk available on Robert Baillie’s blog.

You might also want to have a look at Becoming a Software Testing Expert; a 1 hour presentation delivered by software testing expert, and author of Lessons Learned in Software Testing, James Bach on June 13, 2006. His presentation is available for download from his website.

       
This entry was posted on Thursday, November 2nd, 2006 at 3:19 pm and is filed under General. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.