Category |
Examples |
Test Requirements |
Test Procedure |
Performance | · Traffic capacity · Transaction / data throughput · UI response time · Call setup time · Memory Utilization |
· Instrumentation of the code to help in measurements (time) · Special tools to load the SUT and measure time, throughput. |
· Identify the dependencies – configuration, external and system state · Identify the values the dependencies can take and decide on the different combinations on which testing will be done. · Execute tests and record measurements · Co-relate parameters and dependencies by plotting charts |
Usability | · Installation · Routine Procedures · Documentation |
· UI Subjects matching user profiles | · Identify critical procedures, sections of UI etc.. for which usability is of concern · Identify the subjects · Test with subjects – observe/videotape · Analyze |
Reliability | Multiple rounds of system testing | · Collect the failure and execution time data from multiple rounds of testing. · Use a suitable software reliability model (e.g.: Goel and Okumoto,1979) to predict reliability | |
Security | Firewalls, Networks | Can not prove system is secure. Establish to sufficient degree of confidence that it is secure enough. | Use of tiger teams (professional hackers!!!) |
Configuration | Hardware and software configurations Redundancy means more configurations | · Identify the combinations · Select important (more used or more susceptible) ones · Do a selected subset of test cases · Degradation and restoration testing (transition between configurations) | |
Volume | · No. of subscribers · Predefined load profile over time |
Automation required | |
Stress | BHCA | Tool to generate the stress condition | Real loads can be used for Volume and stress testing – they are the best and worst kind of load to use for testing. |
Recovery | Automatic restart Switchover | Simulate the condition to initiate recovery.Confirm resumption of processing without transaction compromise. |
Table 2: Categories of non-functional testing
图一根据MIEL描述了软件开发/维护的生命周期。在需求阶段需要确定非功能性需求。接着,对这些需求进行分析,那些可以测试的以及从客户的角度看来非常重要的需求将写入需求说明书,而其他的需求则作为设计目标加入设计中。
图一还指出了软件测试生命周期的不同阶段。请看表一中不同阶段的输入、输出和活动。测试用例开发和测试软件开发是并行活动,可以同时进行。
非功能性需求是相当难确定的,而且像可用性和可维护性这样的需求趋于变得很主观,因此很不容易测试。在这种情况下,确定那些能限定系统这些方面性能的客观属性会很有帮助。在另外一些情况下,比如抗压性和性能方面,在测试中如何进行模拟成为一个挑战。这就要求在软件测试生命周期的每一个阶段都要十分仔细,以确保为测试那些已确定的非功能性属性做好准备。