What do the test team do...

Now, I don’t mean to offend anybody. But it occurred to me recently that not everybody knows exactly what it is that testers actually do when they’re landed with a new project or task. This is perfectly understandable considering I would struggle to describe other project roles in any amount of detail.

The two most asked questions of a software tester are:

“Do you just push every button and make sure it works?”

No.. well… Yes… But that’s not all we do…

“Why don’t we just let developers test stuff?”

The same reason you get someone else to proof-read your work. You don’t see your own mistakes. Our developers are awesome at MAKING things, but they only have to think of ONE way to make it work*. There are hundreds of ways something can fail. So we have to be awesome at BREAKING things!

*Yes, I know this isn’t true, but, hypothetically, in order to make my point.

Also, users do some pretty spectacular and unpredictable things. We have to predict the unpredictable.

So here’s your brief: You have 10 minutes to put together a test plan for a light switch. There is no spec so feel free to make a note of any questions or assumptions you have while doing this.

OK. Your time is up-let’s see how you did.

You probably had some of the following questions around the requirements, or at least made some assumptions before you got stuck in:

• What are the requirements for the light switch?
• Are they sensible?
• Which are the most important?
• Is there a detailed spec and/or wireframe?
• Is anything odd, missing, confusing, contradictory or superfluous?

Once we have asked and answered all of these questions, then we can actually start testing.

• Does it do what it is supposed to do?
• Is it fit for its main purpose?

Everybody should have the obvious answer to this. The light switch should switch a light bulb on and off, but this isn’t the only requirement.

• Does the spec fit the requirements? Is it a light switch for my mate Barry who has no arms? In which case a foot-pedal is probably a better solution.
• Should it do anything else?
• Is there anything it should specificallynotdo?
• What is it made from? Will it fall apart? Will it electrocute the first person to use it?

What about Unit testing? Before you even go anywhere near a wall, you might want to test the switch in isolation with a ‘stub’ providing mock connections. In this case a multi-meter that shows when the switch is on and off.

Then you can start integration testing. Does it still work when it is ‘wired in’ to the rest of the system? If not, why not? Has the bulb broken? Is there a break somewhere else in the circuit? Is someone somewhere else flipping another switch at exactly the same time? This is not really up to the tester to answer. But it’s helpful if you can provide extra information.

So, you flip the switch and the light goes on great, the switch works; job done. Everything passed with flying colours! No, not quite; you also have to make sure that things don’tnotwork. Without checking the wiring, how do we know it’s not turning a light OFF in another room? If it were a database then you couldSelect * from light bulbs where state = ‘ON’and see if you get the numbers you were expecting. But it’s not always possible to do this so occasionally you have to get an electrician in!

When things don’t work

Raising a bug.

• What EXACTLY did you do? Maybe you walked over a nylon carpet and caused a short circuit? Document your steps up to the point something failed.
• What environment are you using? Has the power been turned off to that room today?
• What possible factors are effecting your testing? Have you got wet hands? Does this light switch only work between 3-5pm?
• Is it easy for the person doing the fixing to identify the problem area? Take a photo of the light switch. Draw an arrow on the wall. Label it as broken.
• It’s helpful if you can include as much information as possible so the person responsible for fixing the problem knows exactly what went wrong.

When it does fail, does it fail ‘safely’ or will it electrocute the person using it? Are your light switches load balanced? Do you have a fail-over light switch installed?

Other things to test

Performance testing:

• What is the estimated usage?
• How many users and how often?
• Does it work under real life applications?

Load testing – Does it still respond under a high load?

• What happens if 5 people try and use it at once?
• Or one person flips it on and off lots of times or really fast?
• What if there’s a power surge and the voltage doubles?

Stress testing:

• Increase everything you did for load testing until it breaks!

Automation and Regression Testing:

• If someone installs another light switch in the room in 6 months time, we need to make sure that this one still works.
• Can we program a robot to test this light switch so we don’t have to do it again?
  (Obviously we don’t use robots, but we can use software that mimics a person using the system).

Design and UX!

Testers are end users with privileges. If something is difficult or annoying for a tester to do 50 times a day then it’s likely that your users will struggle too.

• Is it easy/straightforward to use?
• Does it look nice?
• Does it look the same on all platforms? What happens if I try and install it on an old wall that doesn’t support light switches?
• What if your wall is made from glass? Well I’m sorry, you can’t install a light switch in windows! (haha!)

Is it accessible?

• If it’s a black switch on a black wall, how can you see it in the dark?
• If it’s too high up the wall, would a smaller user be able to reach it?
• What if a person was blind/deaf/disabled would they still be able to use it?
• If blind how would they know that they had used it successfully – should there be a comfort message?
• Although a blind person may not need a light switch, we must also consider other switches like heaters, kettles or pedestrian crossings.

What are your problem solving skills like?

It’s not always straightforward. Sometimes you have to find a way round limitations or obstacles. Sometimes you’re given a system with no documentation and told to test it. How do you figure out what it does? A particularly stubborn tester would give one of two answers:

“As there are no requirements to meet then everything has passed!”
or
“As there are no requirements then it should do nothing so everything has failed!”

But awesome testers who like the people they work with will probably have a poke around and annoy you with questions for an hour while they cobble together a test plan.

Have a look at these classic problem solving riddles to get your brain in gear:

The fox, goose and bag of beans
Three Light Switches<– I managed to find one that was light switch related!

At the end of the day – it gets dark!

(Yes, light switch pun intended!)

Is the customer happy? Will the user be happy? Have we captured the requirements accurately and provided a suitable solution?
Yes! Hopefully

Can we assure him that a Technophobia brand light switch won’t short circuit and burn his house down?
Well,. no, but we can tell him all of the things that we have checked for.

What we can do is cover as many scenarios as time, money and our devious little brains will allow.

• We can show that our tests match their requirements and that we have completed all of them.
• If not (for whatever reason) then at least we have identified the most important ones and satisfied them.
• Time and budget permitting, we have also checked for any nasty side effects.
• We have made sure it looks pretty and is accessible to all.

How do you know when to stop testing? There are several factors that determine this. Here are some of the most common and a couple of my favourites:

• Time’s up. We must install this light switch NOW regardless of any remaining bugs.
• Money has run out. We can’t afford to pay you to test for a minute longer.
• The Piñata Principle – you keep hitting it until nothing else comes out.
• Flogging a dead horse – there are still problems but there’s no point carrying on while the project is in this state.

So, you see just how much we can drag out something as simple as a light switch. It’s no surprise that when you start applying these principals to software things get a lot more complicated.

(“Test This Light Switch” Inspired by an interview question I borrowed fromEric Jacobson’s Software Testing Blogand expanded on.)