suddenly, the feeling of missing friends in Shanghai is getting stronger and stronger inside of my heart.

How are you?My friends!

今天在当搬运工，搬实验室机器。

累坏了

http://testertested.blogspot.com/2007/02/there-is-no-four-and-six-in-testing.html

这篇文章的作者所表达的意思就是指测试人员做测试的时候不要被边界值方法的固定思路所影响，而是要在脑袋里有一个概念-“测试无界限”。

在文章末尾还给了一个JAMES BACH和MIKE KELLY做的一个边界值的教学录像，我看了，不错，对扩展测试人员的思路有好处。

推荐强度：+++++++++++++++++++++....

-----------

3月2号修改了一个字，把“测试无边界”改成“测试无界限”

感觉 这样子更好

gong xi fa cai!

li shi dou lai!

plz send back your comments to me.

thank you

AlexanderIII

Thursday, April 14, 2005

More on performance vs. load testing

再谈谈性能测试vs负载测试

I recently got some comments/questions related to my previous blog entry on performance vs. load vs. stress testing. Many people are still confused as to exactly what the difference is between performance and load testing. I've been thinking more about it and I'd like to propose the following question as a litmus test to distinguish between these two types of testing: are you actively profiling your application code and/or monitoring the server(s) running your application? If the answer is yes, then you're engaged in performance testing. If the answer is no, then what you're doing is load testing.
最近我得到了一些关于先前发表在博客上的贴子“performance vs. load vs. stress testing”的意见和疑问。我发现仍然还有很多人对于这两个概念有所混淆。近来，我又仔细想了一下它们的区别，然后得到了用以下这个“试金石”的区分方法：

你是否积极*的去剖析应用程序的代码和（或者）运行着程序的服务器（群的状况）？如果答案是“YES”，那你就是在进行性能测试。假若答案是“NO”，那就可以说明你所做的就是负载测试了。

Another way to look at it is to see whether you're doing more of a white-box type testing as opposed to black-box testing. In the white-box approach, testers, developers, system administrators and DBAs work together in order to instrument the application code and the database queries (via specialized profilers for example), and the hardware/operating system of the server(s) running the application and the database (via monitoring tools such as vmstat, iostat, top or Windows PerfMon). All these activities belong to performance testing.
另外一种可以分辨的方法就是看测试人员做的是白盒测试多还是黑盒测试多。在走白盒的路子的时候，测试人员，开发人员，系统管理员和DBA都要通力合作才可以使用工具来解决问题*，比如说通过使用特定的探测器*来识别应用代码及数据库查询，通过vmstat,iostat,top或者Windows PerfMon工具来监测运行了应用程序和数据库的服务器上的硬件及操作系统（资源数据*），所有的这些都属于性能测试。

The black box approach is to run client load tools against the application in order to measure its responsiveness. Such tools range from lightweight, command-line driven tools such as httperf, openload, siege, Apache Flood, to more heavy duty tools such as OpenSTA, The Grinder, JMeter. This type of testing doesn't look at the internal behavīor of the application, nor does it monitor the hardware/OS resources on the server(s) hosting the application. If this sounds like the type of testing you're doing, then I call it load testing.
黑盒的路子就是在客户端运行负载工具来看应用程序的反应如何。这类型的工具范围从功能少的命令行运行的工具，httperf,openload,siege,Apache Flood，到功能比较强大的工具OpenSTA,The Grinder,JMeter. 这种类型的测试不看应用程序的内部的运作，也不监视运行着被测试程序的服务器硬件及操作系统的资源情况。假如你所做的看上去和这种情况相似的话，那么我觉得它就可定义为负载测试。

In practice though the 2 terms are often used interchangeably, and I am as guilty as anyone else of doing this, since I called one of my recent blog entries "HTTP performance testing with httperf, autobench and openload" instead of calling it more precisely "HTTP load testing". I didn't have access to the application code or the servers hosting the applications I tested, so I wasn't really doing performance testing, only load testing.
然而，在实际中这两个术语又常常被互相的代替使用，就连我自己也有点心虚，因为我和其它人一样，在之前写的一篇文章"HTTP performance testing with httperf, autobench and openload"标题命名得不太准确，其实用"HTTP load testing"的话，反而会更准确点，因为我在里面又没有接触到代码，也没有接触到运行着被测程序的服务器。所以说，其实我并没有做性能测试，做的其实就是负载测试。

I think part of the confusion is that no matter how you look at these two types of testing, they have one common element: the load testing part. Even when you're profiling the application and monitoring the servers (hence doing performance testing), you still need to run load tools against the application, so from that perspective you're doing load testing.
我想混淆两者的部分原因是因为不管你对它们的看法是怎么样的，它们都有一个共同之处，就是负载测试的部分。即使你的确是正在分析*应用程序[代码51testing]和监视服务器（在做性能测试），但你仍然还是要使用负载工具给应用程序加压，所以从这一观点来讲，是可以讲你正在做负载测试。

As far as I'm concerned, these definitions don't have much value in and of themselves. What matters most is to have a well-established procedure for tuning the application and the servers so that you can meet your users' or your business customers' requirements. This procedure will use elements of all the types of testing mentioned here and in my previous entry: load, performance and stress testing.
到目前为止，我所提及到的这些定义其实都是没有什么太大的价值。实际上最应该关注的是为了可以达到你的用户或者你的商业客户的需求，而去调整应用程序及服务器的一个好流程。整个流程会涉及到的元素*包括了这篇文章及先前我写的那篇“load performance and stress testing.”里所讲过的所有测试类型的内容。

Here's one example of such a procedure. Let's say you're developing a Web application with a database back-end that needs to support 100 concurrent users, with a response time of less than 3 seconds. How would you go about testing your application in order to make sure these requirements are met?
我举个例子来讲讲这个流程。比如说你正在开发一个有数据库后台的WEB应用程序，要求它可以支持100个并发用户，反应时间要小于3秒。在这个情况下，你会怎么样测试这个程序，让它可以达到需求？

1. Start with 1 Web/Application server connected to 1 Database server. If you can, put both servers behind a firewall, and if you're thinking about doing load balancing down the road, put the Web server behind the load balancer. This way you'll have one each of different devices that you'll use in a real production environment.
1. 将一台Web或者一台Application服务器和一台数据库服务器连在一起。如果可以的话，可以把两个服务器放在一个防火墙后面。假如你想以后要应用负载平衡的技术的话，你可以考虑在Web服务器前放个负载平衡器。这种方法就可以让你在一种更贴近实际环境[的情况下进行测试了*51testing]。

2. Run a load test against the Web server, starting with 10 concurrent users, each user sending a total of 1000 requests to the server. Step up the number of users in increments of 10, until you reach 100 users.
2. 对Web 服务器开始进行负载测试，先上10个并发用户，每个用户向服务器发1000个请求，然后再逐步10个10个的增加并发用户，直到100个并发用户。

3. While you're blasting the Web server, profile your application and your database to see if there are any hot spots in your code/SQL queries/stored procedures that you need to optimize. I realize I'm glossing over important details here, but this step is obviously highly dependent on your particular application.
3.你可以在轰炸WEB服务器的时候[意指对WEB服务器加不同负载的整个过程]，分析应用程序/SQL查询/存储过程*中是否存在需要优化改善的地方。我意识到在这一点有点过于强调了（*），这一步明显的取决于具体的项目（*）。

Also monitor both servers (Web/App and Database) via command line utilities mentioned before (top, vmstat, iostat, netstat, Windows PerfMon). These utilities will let you know what's going on with the servers in terms of hardware resources. Also monitor the firewall and the load balancer (many times you can do this via SNMP) -- but these devices are not likely to be a bottleneck at this level, since they usualy can deal with thousands of connections before they hit a limit, assuming they're hardware-based and not software-based.
同时还可以通过前面提到过的命令行工具（top,vmstat,iostat,netstat,Windows PerfMon）来监视web/app和database的服务器。这些工具可以让你了解服务器硬件资源的状况。同时也可以监视防火墙及负载平衡器[可以使用SNMP来做这种事]，但由于这种设备具有很高的处理连接的能力，它们在这一层上并不会有瓶颈的问题出现，因此我们可以假设它们不是基于软件，而是基于硬件的。

This is one of the most important steps in the whole procedure. It's not easy to make sense of the output of these monitoring tools, you need somebody who has a lot of experience in system/network architecture and administration. On Sun/Solaris platforms, there is a tool called the SE Performance Toolkit that tries to alleviate this task via built-in heuristics that kick in when certain thresholds are reached and tell you exactly what resource is being taxed.
现在这一步就是整个流程中最重要的一环。要清楚的理解这些监控工具输出数据并不是一个容易的事情，这需要一些很有系统/网络 结构和管理经验的人的帮助。[让我挥挥小手，]举个例子来讲（译者注：让我想起51峰哥上课意气风发的挥挥小手，我就自己打上前面的[]里的内容了，呵呵！峰哥不要生气！）在SUN/SOLARIS平台上，就有一个叫SE Performance Toolkit的工具可以减轻这种痛苦。它可以通过内建的试探法*[启发式法？]当系统到极限的时候开始工作并告诉你具体承受着负载的资源情况*。

4. Let's say your Web server's reply rate starts to level off around 50 users. Now you have a repeatable condition that you know causes problems. All the profiling and monitoring you've done in step 3, should have already given you a good idea about hot spots in your applicationm about SQL queries that are not optimized properly, about resource status at the hardware/OS level.
4. 唔。。。假设现在你的WEB服务器的回应速度已经平稳在50个用户的情况下*，你重现了一个会导致问题的环境条件*。通过上面的第三步的分析与监控，应该已经让你清楚的了解到你的应用程序内的问题点，没有被充分优化的SQL查询语句，以及硬件及操作系统的资源状况。
At this point, the developers need to take back the profiling measurements and tune the code and the database queries. The system administrators can also increase server performance simply by throwing more hardware at the servers -- especially more RAM at the Web/App server in my experience, the more so if it's Java-based.
到了这里，开发人员就需要取消*掉分析测量而开始调试代码及数据库的查询语句了。系统的管理员则同时可以通过简单的增加更多的硬件资源来提高服务器的性能，从我的经验上来看，特别要给WEB/APP服务器增加RAM，假如它是基于JAVA的则应该更多*。
5. Let's say the application/database code, as well as the hardware/OS environment have been tuned to the best of everybody's abilities. You re-run the load test from step 2 and now you're at 75 concurrent users before performance starts to degrade.
5. 好了，现在application/database代码，以及硬件/操作系统的环境都已经被调整得各个器件都发挥出它最大的功能了。然后你就应该从上面的第二步重新开始进行增加负载的测试，然后你就发现系统在性能开始下降前就，已经可以上到75个并发用户了。
At this point, there's not much you can do with the existing setup. It's time to think about scaling the system horizontally, by adding other Web servers in a load-balanced Web server farm, or adding other database servers. Or maybe do content caching, for example with Apache mod_cache. Or maybe adding an external caching server such as Squid.
在这一时刻，基于现在的这种优化过的硬件设备，你也没什么可以再做的了。也许是时候通过在负载平衡WEB服务器群组内增加其它的WEB服务器，或者增加其它的数据库服务器，也许还可以使用内容高速缓存技术（如使用Apache mod_cache），或者增加一个外部的高速缓存服务器（如Squid）等方法，来扩展一下你的系统了。
One very important product of this whole procedure is that you now have a baseline number for your application for this given "staging" hardware environment. You can use the staging setup for nightly peformance testing runs that will tell you whether changes in your application/database code caused an increase or a decrease in performance.
这整个的流程引出了一个非常重要的产物，就是一组在特定硬件条件下，针对你的应用程序基线化了的标准数值*。而你则可以通过每个阶段的设备来跑每晚的性能测试，而这个又可以让你得知你所改动的应用程序/数据库代码是否会引起性能上的消长*。

6. Repeat above steps in a "real" production environment before you actually launch your application.
6. 你可以在正式应用程序发布前，重复以上的步骤。
All this discussion assumed you want to get performance/benchmarking numbers for your application. If you want to actually discover bugs and to see if your application fails and recovers gracefully, you need to do stress testing. Blast your Web server with double the number of users for example. Unplug network cables randomly (or shut down/restart switch ports via SNMP). Take out a disk from a RAID array. That kind of thing.
这里所讨论的话题都是以你想得到你的应用程序的性能/基准的数值为假设，如果你是想要真正去发现bugs而且去看一下你的应用程序是不是可以在失效后，正常的恢复*，那么你所要做的是压力测试。比如说给你的WEB服务器加上两倍的用户数量，随机的拔掉网线（或者通过SNMP关掉/重启switch ports），从RAID阵列里拿个磁盘出来，等等。
The conclusion? At the end of the day, it doesn't really matter what you call your testing, as long as you help your team deliver what it promised in terms of application functionality and performance. Performance testing in particular is more art than science, and many times the only way to make progress in optimizing and tuning the application and its environment is by trial-and-error and perseverance. Having lots of excellent open source tools also helps a lot.

结尾了？其实到了最后，你将你的测试命名为什么都无关紧要，只要你可以帮助你们团队交付出曾经承诺过的功能及性能就行了。[说实在的，]性能测试是一种艺术活，而不是一种科学技术活。很多时候，能够使应用程序及它的运行环境的优化及调试，可以有所进展，只能靠持续不断的反复试验。同时一些优秀的开源工具也可以帮助很多。

posted by Grig Gheorghiu at 3:20 PM

;)