Pairwise Testing

也許我們可以節省時間和精力，有效地發現bug通過測試變量和值組合的技術。Maybe we can save time and effort and find bugs efficiently by a technique for testing variables and values in combination.

“通過檢查這種形式在相應的框中訂購一個或多個雜誌”，“選擇一項從A列和一個從B欄”;顏色和選擇，當我們購買一輛汽車，當我們從一組選擇的事情，我們創造組合與我們的選擇。"Order one or more magazines by checking the appropriate boxes on this form"; "Choose one from Column A and one from Column B"; colours and options when we're purchasing an automobile; when we choose things from a set, we create combinations with our choices.在正式的數學，組合的研究被稱為“ 組合數學 ”組合，形式上，“從一組可區分項目的多個不同項目的選擇時，選擇的順序被忽略”[Penguin1998]。In formal mathematics, the study of combinations is called " combinatorics "; combinations are, formally, "selections of a number of different items from a set of distinguishable items when the order of selection is ignored" [Penguin1998].

在計算機系統中，缺陷通常包括一個單一的情況下，任何其他條件獨立於系統中。In computer systems, defects usually involve a single condition, independent of any other condition in the system.如果有一個設備或一個變量或一個設置有問題，該問題通常是在該設備或可變或單獨設置。If there's a problem with a device or a variable or a setting, the problem is usually in that device or variable or setting alone.作為一個例子，在一個字處理程序中，一個給定的字體可能被不恰當地呈現，不論打印機，打印機連接，或Windows版本。As an example, in a word processing program, a given font may be rendered improperly, irrespective of the printer, printer connection, or version of Windows.在某些情況下，但是，有兩個或更多的壞舞蹈演員在晚會上，通常萬事大吉，但如果遇到對方，而在一定的狀態，他們在彼此的腳絆倒。In some cases, however, there are two or more bad dancers at the party; normally everything is all right, but if one encounters the other while in a certain state, they trip over each other's feet.在測試中，我們要確保我們不會錯過基於兩個或多個條件，變量或配置之間的衝突問題，所以我們經常測試的組合，以最有效地發現缺陷。In testing, we want to be sure that we don't miss problems based on conflicts between two or more conditions, variables, or configurations, so we often test in combinations in order to find defects most efficiently.除一人外的所有打印機可能使一個給定的字體完美，該打印機可能使所有字體完美，除了那一個。All printers except one might render a given font perfectly, and that printer might render all fonts perfectly except for that one.我們怎樣才能最大限度地發現在有限的時間這樣的問題，我們必須測試的機會？How can we maximize the chance of finding such a problem in the limited time we have to test?

當我們考慮這樣一個任務，我們幾乎立即開始碰到的“完整的測試”的問題。When we consider such a task, we begin almost immediately to run into the problem of "complete testing".假設一個缺陷，取決於系統中的每個變量是在一個特定的狀態。Suppose that a defect depends upon every variable in the system to be in one specific state.只要我們可以通過檢查系統中每個狀態，結合所有的其他方面在每一個他們可能狀態的各個方面嘗試所有可能的組合，我們將因此測試每一個州和國家的每個組合中制度。If only we could try all of the possible combinations by checking each aspect of the system in each of its states, combined with all of the other aspects in each of their possible states, we would thereby test every state and every combination of states in the system.有了這種性質的一個有效的測試，我們會保證找到依賴於系統的狀態的任何缺陷。With an effective test of this nature, we'd be guaranteed to find any defect that depended on the state of the system.

發現任何基於狀態的缺陷作出保證是非常可取的。That guarantee of finding any state-based defect is very desirable.可惜的是，它是完全不可行的，即使是一個小程序，用一個相對簡單的變量和每個變量相對較少可能的狀態，結合可能的有效狀態的總數是難以控制的大。Alas, it's completely infeasible; even for a small program, with a relatively simple set of variables and relatively few possible states per variable, the total number of possible valid states in combination is intractably large.要在這個數字到達，我們認為每個變量，並計算有效狀態數吧。To arrive at that number, we consider each variable, and count the number of valid states for it.然後，我們乘所有有效狀態的數目為每個變量在一​​起。We then multiply the numbers of all those valid states for each variable together.

假設有五個變量，各由一個字母表示。Suppose that there are five variables, each represented by a letter of the alphabet.並假設每個變量可以包含一個值從一到五。And suppose that each variable can contain a value from one to five.讓我們設置變量A，B，C和D都等於1。Let's set variables A, B, C, and D all equal to 1.如果這些值是固定的，變量E的值範圍從一個到五個這樣五種組合存在。If those values are fixed, variable E can have a value from one to five—so five combinations there.我們將跟踪組合的總數在我們最左邊的列，我們的第一個五年的組合將被編號從1到5。We'll keep track of the total number of combinations in our leftmost column; our first five combinations will be numbered from 1 to 5.

讓我們設置變量D兩個而不是一個，並保持A，B和C上。Let's set variable D to two instead of one, and keep A, B, and C at one.五個組合，編號為6-10。Five more combinations, numbered 6-10.

然後，我們將設置D到3，然後到4，然後到5，從1每段時間設定E到值5。Then we'll set D to 3, then to 4, and then to 5, setting E to values from 1 to 5 each time.我們必須要經過25組合之前，我們用盡所有的組合為D和E的，只有這樣才能改變C至1〜2。We'll have to go through 25 combinations before we exhaust all of the combinations for D and E, and only then can we change C to from 1 to 2.右邊的列將翻轉比較快，但我們向左移動，列A，B，和C會較少發生大的變化，就像汽車里程表。The columns on the right will roll over relatively quickly, but as we move leftwards, columns A, B, and C will change much less often, like the odometer on a car.

事實上，如果有只有五個程序中的變量，每個變量有五個可能的狀態，程序會荒謬地簡單，但我們不得不試3125組合 - 五種組合的第一個變量，乘五第二，次五為第三，乘五第四，次五為第五5×5×5×5×5。In fact, if there are only five variables in the program, each with five possible states, the program would be absurdly simple, yet we'd have to test 3125 combinations—five combinations for the first variable, times five for the second, times five for the third, times five for the fourth, times five for the fifth—5 x 5 x 5 x 5 x 5.如果我們可以測試每分鐘一組合，齊全的檢測將採取的連續七小時的每天的測試，假設我們沒有死無聊的前七個天半。If we could test one combination per minute, complete testing would take seven and a half days of continuous, seven-hour-a-day tests—assuming that we didn't die of boredom first.我們有了一種方式，以減少測試的次數到的東西，我們可以處理。There's got to be a way to reduce the number of tests into something that we can handle.

要開始思考如何減少測試次數，讓我們假設一個簡單的程序，一個更簡單的程序比上面的那個，在此我們只有兩個複選框。To start figuring out how to cut down the number of tests, let's assume a simple program—an even simpler program than the one above, in which we have only two checkboxes.一個複選框可以在兩種狀態 - 選中或取消選中之一。A checkbox can be in one of two states—checked or unchecked.試想一下，我們的缺陷取決於一個複選框被清除（即未選中）和被設置複選框B（選中）。Imagine that our defect depends upon checkbox A being cleared (that is, unchecked) and checkbox B being set (checked).如果我們嘗試所有的相互結合可能的設置，我們總能找到缺點。If we try all of the possible settings in combination with one another, we'll find the defect.我們將建立一個表，有一排為每個變量。We'll set up a table, with a row for each variable.在每個單元中，我們會把變量的可能的狀態設置或清除之一。Within each cell, we'll put one of the variable's possible states—set or cleared.在最左邊的列，我們把一個索引號，這樣我們可以通過數字引用的每個組合。In the leftmost column, we'll put an index number so that we can refer to each combination by number.

組合1，3和4的工作不錯，但組合2展品的問題。Combinations 1, 3, and 4 work all right, but Combination 2 exhibits the problem.我們就需要四個測試，以確保我們已經涵蓋了所有的組合，其中A和B可以找到。We'd require four tests to make sure that we had covered all of the combinations in which A and B could be found.兩個變量和兩個設置每個;四種組合。Two variables, and two settings for each; four combinations.

現在考慮的東西只有一點點更複雜的：有三組單選按鈕就可以了，相當於在北美旅行需要一個對話的黑箱測試。Now consider something only a little more complex: a black-box test of a dialog with three sets of radio buttons on it, representing travel needs in North America.第一組單選按鈕指定一個目標，按國家;第二注意到過道或靠窗座位的選擇，第三個能提供教練，業務或經濟類的選擇。The first set of radio buttons specifies a destination, by country; the second notes the choice of an aisle or window seat; and the third affords the choice of coach, business, or economy class.當用戶使她的選擇，並按下“確定”按鈕關閉對話框，應用程序將在特定國家的執行給定選項的每一種組合的不同的路徑。When the user makes her selections and presses "OK" to close the dialog, the application will take on a different path of execution for each combination of given options in given states.

像往常一樣，最簡單的一種問題檢測是一個由單個變量在一​​個單一的狀態觸發。As usual, the simplest kind of problem to detect is one that is triggered by a single variable in a single state.這就是所謂的單模式故障 [Phadke1997]。This is called a single-mode fault [Phadke1997].例如，考慮一個包含一個缺陷，導致其行為怪異，當“目標”選項設置為“加拿大”的應用程序。For example, consider an application that contains a defect causes it to behave bizarrely when the "destination" option is set to "Canada".當然，要找到這樣的缺陷的方法是確保我們至少運行一個測試用的“目的地”設置為“加拿大”。Naturally, the way to find such a defect would be to ensure that we run at least one test with "destination" set to "Canada".

現在假設有一個缺陷，取決於兩個條件，例如，目標字段被設置為“墨”和服務類被設置為“業務”的同時。Assume now that there is a defect that depends upon two conditions—for example, the destination field being set to "Mexico" and the service class being set to "Business" at the same time.這就是所謂的雙模式故障 。This is known as a double-mode fault .它應該是清楚的，如果我們運行至少一個測試用的“目的地”設置為“墨”和“類”字段設置為“商業”，我們會遇到的缺陷。It should be clear if we run at least one test with "destination" set to "Mexico" and the "Class" field to "Business", we'll encounter the defect.

隨著兩個對象和兩個選項各，我們可以編譯一個相對簡單的表。With two objects and a two options for each, we could compile a relatively straightforward table.然而，與任何額外的複雜性，所需的測試工作量非常迅速的增長。However, with any additional complexity, the required testing effort grows very quickly.對於兩個單選按鈕（每3個可能狀態），一個方格中，有3×3×2，或18個可能的值進行測試。For two radio buttons (each with three possible states) and one checkbox, there are 3 x 3 x 2, or 18 possible values to test.這裡有一個表，該表顯示了每個可能的組合：Here's a table that shows each possible combination:

我們發明的測試響應的風險。We invent tests in response to risks.有一個風險，即開發商寫一行代碼與單模故障，所以我們也希望確保我們在它的每一個國家，如果我們可以測試每一個變量。There's a risk that the developer has written his code with a single-mode fault, so we do want to be sure that we test every variable in each of its states if we can.目前，開發商已經寫在雙模式故障一個減少風險，所以理想情況下，我們會希望在它的每一個國家的每一個其他變量產生每個變量的至少一個測試，在每州。There is a lessened risk that the developer has written in a double-mode fault, so ideally we'll want to produce at least one test of each variable in each of its states with every other variable in each of its states.如果我們的錯誤的理論是，一個開發者做了一個雙模式故障錯誤（例如，如果有一個國家，墨西哥，和一個服務類，業務之間的衝突），我們需要至少運行一個測試，其中包括“墨”和“商務”的結合。If our theory of error is that a developer has made a double-mode fault mistake (say, if there is a conflict between one country, Mexico, and one service class, Business), we need to run at least one test that includes the combination of "Mexico" and "Business".在另一方面，如果我們運行了一些墨西哥/商務當量測試，我們可以重複勞動和浪費時間。On the other hand, if we run a number of equivalent tests on Mexico/Business, we could be duplicating effort and wasting time.

最困難的這類問題找到了黑箱測試是其中幾個變量都參與其中，而且每個人都必須在一個特定的，獨特的狀態來觸發的問題。The most difficult kind of problem to find by black-box testing is one in which several variables are involved, and each must be in a specific, unique state to trigger the problem.例如，當全國字段設置為“美國”，座位選擇“過道”，而服務類，以“第一”，那麼只有這樣，這個bug被觸發。For example, when the country field is set to "USA", the seat choice to "aisle", and the service class to "First", then and only then will the bug be triggered.這就是所謂的三模故障時 ，或者更一般地作為多模式故障 ，它描述了具有三個或多個參數相關聯的故障。This is known as a triple-mode fault , or more generally as a multi-mode fault , which describes faults associated with three or more parameters.對於測試，發現此類問題取決於以下兩種方法之一：測試變量的所有組合，並在方案;或一些其他技術結合...For testers, finding this kind of problem depends on one of two approaches: testing all combinations of variables and states in the program; or some other technique combined with...好了，運氣。well, luck.第一個選項是斷然不可能的事情，但最重要的程序。The first option is flatly impossible for anything but the most trivial program.關鍵的第二個選項是優化測試，以便將作為幸運越好。The key to the second option is optimize the tests so to be as lucky as possible.

1997年，研究人員在Telcordia科技（前身為貝爾通信研究，或Bellcore實驗室）由悉達多達拉爾等發表了一篇論文。In 1997, researchers atTelcordia Technologies(formerly Bell Communications Research, or Bellcore) published a paper by Siddharta Dalal et.人“的組合設計方法，以自動測試生成”[Telcordia公司1997]。al., "The Combinatorial Design Approach to Automatic Test Generation" [Telcordia 1997].Telcodia的研究表明，“大多數現場的故障是由兩種不正確的單值或值對的相互作用引起的。”Telcodia's studies suggest that "most field faults were caused by either incorrect single values or by an interaction of pairs of values."如果這是普遍正確的，我們應該把我們的測試在單模式和雙模式故障的風險。If that's generally correct, we ought to focus our testing on the risk of single-mode and double-mode faults.Telcordia的紙張表明，我們可以通過選擇測試，使得1）每個變量的每個狀態進行測試，以及2）在它的每一個狀態中的每個變量是在一對與每個狀態的所有其他變量測試得到優良的覆蓋範圍。The Telcordia paper suggests that we can get excellent coverage by choosing tests such that 1) each state of each variable is tested, and 2) each variable in each of its states is tested in a pair with every other variable in each of its states.這就是所謂的成對測試或全雙測試 。This is called pairwise testing or all-pairs testing .

專注於單，雙模式故障大大降低的，我們必須執行組合的數量。Focusing on single- and double-mode faults reduces dramatically the number of combinations that we have to run.如果所有在一個給定的組合在對存在於其他的組合，我們就可以丟棄組合。If all of the the pairs in a given combination exist in other combinations, we can drop that combination.例如，考慮14組合從上面表4：墨西哥/商務配對也出現在組合5，業務/窗口對出現的組合13，而墨西哥/窗口的組合出現在組合11。For example, consider combination 14 from Table 4 above: the Mexico/Business pair also appears in combination 5, the Business/Window pair appears in combination 13, and the Mexico/Window combination appears in combination 11.我們可以下降14組合不留任何對未經測試。We can drop combination 14 without leaving any of its pairs untested.

這裡有一個表，根據上面的表4，該測試所有值對的：Here's a table, based on Table 4 above, that tests all of pairs of values:

此表可以確保加拿大與每個教練，商務艙和頭等艙的至少測試一次，並與過道狀態的複選框，然後在窗口的狀態。This table makes sure that Canada is tested at least once with each of Coach, Business Class, and First Class, and with the checkbox in the Aisle state and the Window state.同樣，每一個選項與所有其他選項進行測試。Similarly, every option is tested with every other option.我們捕捉到雙模式故障組合中3號外露;我們不需要測試組合12，因為它的雙均受組合15（其中測試目標=美國和座椅=窗口），組合11（=級教練和座椅=窗口）和組合3（其中測試目標=美國和Class =教練，並公開他們之間的衝突）。We capture the double-mode fault exposed in combination number 3; we need not test combination 12, since its pairs are covered by combination 15 (which tests Destination=USA and Seat=Window), combination 11 (Class=Coach and Seat=Window) and combination 3 (which tests Destination=USA and Class=Coach, and exposes the conflict between them).使用全對理論為基礎進行選擇的測試，在這種情況下，我們削減的測試要求數量的一半，從18到九。Using the all-pairs theory as the basis for selecting tests, in this case we cut in half the number of test requirements, from 18 to nine.

現在，我們是如何設法簸出重複的？Now, how did we manage to winnow out the duplicates?在這種情況下，所有可能的組合的表18的條目。In this case, the table of all possible combinations is 18 entries.