What is Exploratory Testing?

And How it Differs from scrīpted Testing

aY,q{/R_0by James Bach51Testing软件测试网ux6O)sh L

This first appeared onwww.StickyMinds.comas a column feature

Exploratory software testing is a powerful and fun approach to testing. In some situations, it can be orders of magnitude more productive than scrīpted testing. I haven't found a tester yet who didn't, at least unconsciously, perform exploratory testing at one time or another. Yet few of us study this approach, and it doesn't get much respect in our field. It's high time we stop the denial, and publicly recognize the exploratory approach for what it is: scientific thinking in real-time. Friends, that's a good thing.51Testing软件测试网!d%U3iPI6aF

Concurrent Test Design and Execution

5i&H'{ n S8nkaH9B0The plainest definition of exploratory testing is test design and test execution at the same time. This is the opposite of scrīpted testing (predefined test procedures, whether manual or automated). Exploratory tests, unlike scrīpted tests, are not defined in advance and carried out precisely according to plan. This may sound like a straightforward distinction, but in practice it's murky. That's because "defined" is a spectrum. Even an otherwise elaborately defined test procedure will leave many interesting details (such as how quickly to type on the keyboard, or what kinds of behavīor to recognize as a failure) to the discretion of the tester. Likewise, even a free-form exploratory test session will involve tacit constraints or mandates about what parts of the product to test, or what strategies to use. A good exploratory tester will write down test ideas and use them in later test cycles. Such notes sometimes look a lot like test scrīpts, even if they aren't. Exploratory testing is sometimes confused with "ad hoc" testing. Ad hoc testing normally refers to a process of improvised, impromptu bug searching. By definition, anyone can do ad hoc testing. The term "exploratory testing"--coined by Cem Kaner, in Testing Computer Software-- refers to a sophisticated, thoughtful approach to ad hoc testing. In the last decade, James Whittaker (at Florida Tech), Cem Kaner and I have worked to identify the skills and techniques of excellent exploratory testing. For one example of a fully defined and articulated process of exploratory testing, see the General Functionality and Stability Test Procedure for Microsoft's Windows 2000 Compatibility Certification program. This document is publicly available on Microsoft's web site, or on my site athttp://www.satisfice.com/tools/procedure.pdf.51Testing软件测试网(Xq ]3g@1O

Balancing Exploratory Testing With scrīpted Testing

To the extent that the next test we do is influenced by the result of the last test we did, we are doing exploratory testing. We become more exploratory when we can't tell what tests should be run, in advance of the test cycle, or when we haven't yet had the opportunity to create those tests. If we are running scrīpted tests, and new information comes to light that suggests a better test strategy, we may switch to an exploratory mode (as in the case of discovering a new failure that requires investigation). Conversely, we take a more scrīpted approach when there is little uncertainty about how we want to test, new tests are relatively unimportant, the need for efficiency and reliability in executing those tests is worth the effort of scrīpting, and when we are prepared to pay the cost of documenting and maintaining tests.

{M+b~8@he results of exploratory testing aren't necessarily radically different than those of scrīpted testing, and the two approaches to testing are fully compatible. Companies such as Nortel and Microsoft commonly use both approaches on the same project. Still there are many important differences between the two approaches.

Why Do Exploratory Testing?

p(a^@!d_3P
^z L)|h0Recurring themes in the management of an effective exploratory test cycle are tester, test strategy, test reporting and test mission. The scrīpted approach to testing attempts to mechanize the test process by taking test ideas out of a test designer's head and putting them on paper. There's a lot of value in that way of testing. But exploratory testers take the view that writing down test scrīpts and following them tends to disrupt the intellectual processes that make testers able to find important problems quickly. The more we can make testing intellectually rich and fluid, the more likely we will hit upon the right tests at the right time. That's where the power of exploratory testing comes in: the richness of this process is only limited by the breadth and depth of our imagination and our emerging insights into the nature of the product under test. In the rapid testing classes at Satisfice, Inc., we have equipment that watches testers invent tests in real-time. When the instructor makes a new suggestion for what to test, or provides new information to the testers about the product, we observe and measure how a roomful of exploratory testers reacts to that information. Free from the encumbrance of pre-documentation, they immediately incorporate new ideas into their tests.

scrīpting has its place. I can imagine testing situations where efficiency and repeatability are so important that we should scrīpt or automate them. For example, in the case where a test platform is only intermittently available, such as a client-server project where there are only a few configured servers available and they must be shared by testing and development. The logistics of such a situation may dictate that we scrīpt tests carefully in advance to get the most out of every second of limited test execution time. Exploratory testing is especially useful in complex testing situations, when little is known about the product, or as part of preparing a set of scrīpted tests. The basic rule is this: exploratory testing is called for any time the next test you should perform is not obvious, or when you want to go beyond the obvious. In my experience, that's most of the time.51Testing软件测试网Q6[r,rhEhp

其实，每个测试员都在不知不觉地在用到探索性测试方法，但是很少有人学习和重视这种方法。现在是时候认识一下探索性测试方法了：科学的实时的思考。51Testing软件测试网8M]L\h4A

ConcurrentTestDesign and Execution51Testing软件测试网
D-~ J
a1^(D a.E

同时设计测试和执行测试

       对探索性测试的最直白的定义是：同时设计测试和执行测试。这与剧本化的测试方法相反（预先定义好测试步骤）。探索性测试不像剧本化的测试，不会预先定义，不会严格按照计划开展。然而，即使是精确定义的测试步骤也会有很多有趣的细节遗留给测试员（例如：在键盘上敲击的速度、怎样的行为才认为是错误）；即使是方式非常自由的探索性测试也会对测试产品的哪些部分作出规定，或规定采用什么测试策略。51Testing软件测试网,j:p\,T
h[+E)Z8H

       好的探索性测试者会把测试的想法写下来，并应用在后来的测试循环中。这些记录下来的东西看起来有点像测试脚本。51Testing软件测试网Tz-_CR mqW$_K

 51Testing软件测试网N0iH SHb

       探索性测试有时候会与即兴测试(ad hoc testing)混淆。即兴测试通常是指临时准备的、即席的bug搜索的测试过程。从定义可以看出，谁都可以做即兴测试。由Cem Kaner提出的探索性测试，相比即兴测试是一种精致的、有思想的过程。

Balancing Exploratory Testing With scrīpted Testing

平衡探索性测试与剧本化测试51Testing软件测试网9h&}/k`c
{RUz

       如果做到了下一项测试被我们所做的上一测试的结果所影响，那么我们就是在做探索性测试。当我们在测试循环之前不知道应该运行什么测试时，或者我们还没机会创建测试，我们应该更多地探索。51Testing软件测试网v
I.[8[ ?/a

 51Testing软件测试网3d!N2P(r5UIK'qy[4W

       如果我们正在执行剧本化的测试的时候，新的信息提示我们可以有更好的测试策略，我们应该转成探索模式（例如发现了新的错误需要进行调查）。

 51Testing软件测试网 gzy9V,X3A)~

       相反地，当我们非常清楚我们要做什么测试以及怎样做时，我们应该采取剧本化的测试方法。新的测试相对没那么重要，执行测试的效率和可靠性的需要使得测试值得剧本化，值得我们把它们文档化并维护。51Testing软件测试网AicXX"vm4z

       探索性的测试结果与剧本化的测试并没有根本性的区别，两种测试方式是完全兼容的。像Nortel和微软这样的公司通常在项目中两种方法都使用。51Testing软件测试网K5XbIl:lb

 51Testing软件测试网-q5Vi7aj YK

Why Do Exploratory Testing?51Testing软件测试网B*v^(Wm-AG"s.l8m

为什么要做探索性测试?

       在有效的探索性测试循环管理中的重复主题是：测试、测试策略、测试报告和测试任务。测试剧本化的方式企图将测试过程机械化，从测试设计者的脑袋中把测试的思想抽取出来并放到纸面上。这种测试方法的好处很多。51Testing软件测试网J(TH:Uga+eZs e$A"h

       但是探索性测试者持这样的观点：把测试剧本化地写下来并按照它们来测试会破坏快速寻找重要问题这一智力的过程。这一智力的过程越丰富、越流畅，我们越有机会在正确的时间执行正确的测试。这就是探索性测试的威力所在：测试过程的丰富性只是受限于我们的思维的广度和深度，还有我们对测试产品的洞察能力。

       剧本化的测试是有它存在的意义的，我可以想象测试的效率和可重复性是那么的重要，所以我们应该剧本化或自动化测试。在测试环境间歇有效的情况下，例如C/S结构的项目，只有几个配置的服务器有效并且要在测试和开发之间共享。这种情形下我们应该把测试小心仔细地提前剧本化，以便能充分利用有限的测试执行时间。

 51Testing软件测试网!Y;}`vzZ,[2x
t

       探索性测试在复杂的测试情况下会特别有用，对产品了解甚少的情况下会特别有用，或者作为准备剧本化测试的一部分测试。基本规则是：应该在你准备进行的下一次测试内容并不明确的情况下进行探索性测试，或者你希望把这些不明确的因素明确了

 51Testing软件测试网F)or i(Mj4p