产品开发中的测试和测试人员

On testers and testing

Over the years, I’ve come to hold some strong. opinions on testing, testers and the entire business of quality assurance. Inspired by thisposton Facebook’s testing, I wanted to write this down so I can point people to it. Some of this is controversial. In fact, even mentioning some of this in conversation has caused people to flip the bozo bit on me.

1. Most product teams don’t need a separate testing role. And even if they do, the ratio of full time dev:full time test should be in the order of >20:1. For evidence, look no further than some of the most successful engineering teams of all time. Whether it be the Facebook of today or the original NT team from 30 years ago, some great software products have come from teams with no or little testers.
2. Developers need to test their own code. Period. The actual mechanism is unimportant. It could be unit tests, full fledged automated tests or button mashing manually or some combination thereof. If your developers can’t/won’t or somehow think it’s ‘beneath them’, you need better developers.
3. Here’s a politically incorrect thing to say. Several large engineering organizations start hiring testers from the pool of folks who couldn’t cut it as developers. I’ve been in/heard of a scary number of dev interviews where somebody has said “(S)he is not good enough to be a dev. Maybe a test role?”. This leads to the widely held but rarely spoken of perception that someone in test is not as smart as someone in dev. And due to the widespread nature of #3, the few insanely smart people who are actually genuinely passionate about quality and testing get a raw deal. I know this since I worked with a few of them.
4. Tracking code coverage is dangerous. Since it is so easily measurable, it often becomes a proxy for the actual goal - building quality software. If your dev team is spending their cycles writing silly tests to hit every rarely used code path - just because it shows up in some status report - you have a problem. Code coverage is one of many signals that needs to be used but since it is so seductively numeric in nature, it drowns out many others. It falls victim toGoodhart’s law.
5. I’m yet to see well-written test code in organizations that have testers separate from developers. Sadly, this is accepted as a fact of life when it doesn’t need to be.
6. Just like metrics like code coverage are overused, other quality signals are underused. Signals like - how many support emails has this generated? Actuallyusingthe product all the time yourself and detecting issues. Analyzing logs from production and customer installations. All the other tactics mentioned in the Facebook post at the top.
7. A common tactic by engineering leaders is to try and reduce the size of their testing orgs by moving to automated testing. This is a huge mistake. If you have a user-facing product, you absolutely need manual eyes on it. Sit someone down from the Windows organization at Microsoft for coffee and they’ll blame the focus on automated testing for a lot of the issues in Windows Vista. The mistake here is assuming that you need a full-time test person to use the product.

这些年来，我对测试工作、测试人员，以及整个软件质量管理体系形成了一些明确的观点。受一篇关于Facebook的测试的帖子的启发，我想把这些写下来，用以拿给人看。有些观点是有争议的。事实上，即使在交谈中稍微表现出这样的看法，都会招致人们的鄙视。

　　1.大多数的开发团队并不需要一个独立的测试角色。即使有一个，他的所有的开发时间比上所有的测试时间应该>20:1。证据吗?光看看一些从古至今最成功的软件开发团队就知道了。不论是当今的Facebook，还是30年前最初的NT团队，很多伟大的产品都是出自没有或很少测试人员的团队。

　　2.开发人员应该测试自己的代码。没什么可说的。背后的道理并不重要。这包括单元测试，全覆盖的自动化测试或手工测试或组合测试。如果你的开发人员不能/不愿意或认为这“不归我管”，那你需要更好的程序员。

　　3.我有一些政治上不正确的话要说。一些大型的软件开发公司会从一些不能胜任开发工作的程序员中选择测试人员。我经历/听说过很多在面试开发人员过程中有人说“他/她做开发不行。也许可以做测试?”这导致了一个被广泛默认，但很少公开宣称的认识：做测试的不如做开发的聪明。正是由于这普遍的偏见，少数一些对质量和测试工作极具热情和天赋的人受到了不公平的待遇。我知道他们，因为我曾经和一些这样的人一起工作过。

　　4.追求代码测试覆盖率是危险的。因为它很容易测量，它经常会变成真正目标的替代品——开发有质量的软件。如果你的开发团队把功夫都用在写愚蠢的测试，来覆盖每一个罕有能用到的代码路径——因为这些数据会出现在一些报告里——你有问题了。测试覆盖率只是我们用的的很多指标中的一个，你需要使用它，但并不是因为它以自然的数字呈现，它就能压倒其它指标。它成了古德哈特定律的牺牲品。

　　5.我也曾看到有些公司里有独立的测试人员，他们写出非常好的测试代码。不幸的是，这被人们当成了一种常识，即使是在根本不需要这样的时候。

　　6.正像测试覆盖率被过度使用，有些质量指标是被忽略轻视了。比如：过程中产生了多少技术支持邮件?自己是否在任何时候都在真正的使用自己的产品，检测里面的问题?分析从生产环境和客户安装那里产生的日志文件。所有的这些策略都在上面的Facebook的帖子里有提到过。

　　7.技术领导的一个常见的减少测试队伍的体积的办法是做自动化测试。这是个巨大的错误。如果你有一个用户直接接触的产品，你绝对需要用人眼去测试它。如果你和微软Windows团队的人坐下来一起和咖啡，你会听到他们抱怨过度依赖自动化测试导致了Windows Vista大量的bug。这个错误说明的问题就是：你需要一个全职的测试人员来使用你的产品。

　　本文的作者Sriram Krishnan是一名程序员，曾在Yahoo和微软工作过，开发过很多软件，曾被纽约时报报道，写过一本书，本文是他的一篇博客。

　　原文链接：On testers and testing