I just read a post by Tony Bruce on making excuses for why a bug was not found until later testing cycles/UAT/Production. Good post, great read, as always. You can find it at dancedwiththetester.blogspot.co.uk
In the last part of the post Tony writes:
I am not a gate keeper, I am an information provider, I have limited time to provide what information I can. I will do my best, the people around me will do their best and sometimes we will not be able to investigate everything.
This reminded me of a matter that has been bugging me a bit for a while; I see a lot of people stating that “Testing is…” and then some definition or list of tasks as if this is the One and Only Truth.
Oh yeah? Is it really? According to whom? Who decides?
First, just to be clear, I agree with the idea that testing is, or at least should be, about uncovering information and assisting the rest of the project in making quality related decisions. Testers are people too, with brains, and having them mindlessly following scripted checks that hopefully will not fail, and thus not uncover any new information, is downright cruel (yeah yeah yeah, there are always exceptions).
Michael Bolton collected a few different, but similar, definitions of testing in his comment on Alan Page’s blog a while ago:
James Bach says that testing is “Questioning a product in order to evaluate it.”
Jerry Weinberg says that testing is “gathering information with the intention of informing a decision.”
Cem Kaner says that testing is (deep breath, now) “A technical, empirical investigation of a product, done on behalf of stakeholders, with the intention of revealing quality-related information of the kind that they seek.”
I [Michael Bolton] say that “testing is the investigation of systems composed of people, computer programs, and related products and services.” What’s more, I agree with the three above, since, broadly speaking, we all mean more or less the same thing.
I have not personally sat down developed my own tailored definition. At the moment I am content with saying that these definitions presents a world of testing where I want to work. It makes sense to me that people would want this job to be done on a project, and it is an opportunity for the person doing the job to learn all sorts of wonderful things.
However…
While I have a fair idea about what I want testing to be, or what I mean when I talk about testing, I find it quite clear that not everybody shares that idea.
Regardless of where in the continuum between fully scripted or pure exploration the applied test approach is , people seem to have very different ideas about what the purpose of testing is or should be. Some people is adamant about testing being performing some precisely described operation, and then comparing the output, or at least the part of the output they thought about when describing the test, to an expected result. In some cases it seems that the most important objective for the “testers” is to produce a specific amount of paperwork, regardless of what it actually contains, so that everything is according to The Process (regardless of whether that descriptive document actually fits the project and the context at all). In this world of boiler plate text and ill-fitting templates, ticking of the check boxes is what counts as getting things done, and how its is done is of less interest. Some times there is the added desire to Verify the Requirements, because we all know how that “Assures Quality”. I have seen variations all over the spectrum. From what I just described, all the way to testers and developers pairing up, focusing on uncovering the mysteries of the product together and prioritizing shipping great stuff over obsessing over “expected results”.
My point is, before this get way too long… people have different ideas and expectations.
You might not agree with them, and you might very well try to explain your view, hoping to persuade somebody, or maybe to be enlightened yourself, but in the end… is it not whoever pays for your services who decide what testing IS… in that given context?
+1 Testing 
” is it not whoever pays for your services who decide what testing IS… in that given context?”
Are you just saying that for the concept of testing, which has a wide array of (sometimes conflicting) definitions, that a specific example is more salient and pragmatic? If so, then I agree.
Here’s why I say “Testing is/isn’t” (or more often a version of “this is/isn’t what I do”):
If I don’t make it clear to the people I work with what it is that I do (and what I don’t do) then I can expect them to not understand my service to them. I can expect them to treat me like a gate keeper, or a sign-off process at the end of development. I can expect developers to dislike me for pointing out mistakes and “breaking” the software rather than acting as a system investigator for their benefit. I can expect to own quality myself despite never actually coding any of the software.
If we treat dictionary definitions as orbs of information by which we identify the things around us we are in trouble. Dictionaries exist to document the meaning of words as we use them, that’s why they keep reprinting the same words with different/additional definitions. Testing means whatever we need it to mean in the long term.
So if I say, with some conviction, “Testing is/isn’t…” then I’m trying to change opinion, raise conciousness about an idea and make amendments to the language (domain and wider) to change the way people think about testing for (I think) the better. It’s how language evolves, and it’s beautifully scientific. I won’t do it as effectively if I say “testing is usually…” or “In my opinion…”. That’s about it, really.
But of course we know there is NO “Testing is…” statement that always holds true for EVERY context. The ones you listed are heuristic definitions, and we judge their elegance by their lack of notable counterexamples.
So if I notice someone saying “testing is…” I usually gauge whether or not I’ll respond by how silly my counterexamples are.. and that, I suppose, is a definition of where to draw the line at pedantry.
Thanks a lot for the comment.
From what you write it seems to me you are operating in a market where there is plenty of testing jobs available and that you are free to select what to do and how you do it. Thats great.
I don’t think this is the case for all testers though. In many cases people find themselves in one of the (relatively) few positions available in their area, and they have to make the best out of that. In those cases it might not be so easy to just tell the boss “what I do, and what I don’t”.
Yeah, I know, “if you’re boss tells you to do testing in a different way than you prefer, you should just quit”. I’ve seen and heard that argument too. Might be possible for some, not for others.
Anyways, thanks for taking the time.
Even in a position where there are few other positions available I’d say it’s possible to tell your boss how they might make better use of you. I would always do what I am told in the end as employment isn’t a democracy. I don’t know of many suitable positions available in my area, certainly, as I live in quite a rural part of the country.
I’m not condoning the idea that anyone should “just quit”, I certainly didn’t mean to convey that idea at all (and I don’t believe that I said that). I think it’s entirely foolish not to open people’s minds to what I have the capacity to do (compared to what I’m told to do) no matter where I work.
Essentially what I was trying to say is that if you own your reputation as a tester you can give them your opinion on your potential and hope that they ask you to do something sensible. If you choose to do things you’re told to do without offering alternatives and advice based on your knowledge and experience then you can expect to be used as a scapegoat for project problems and disliked by your peers as a work checker that points out all their mistakes.
Everyone defines their own meaning of “testing”. I’m just trying to (tactfully) change some minds.
Great post that is a good reminder of different perspectives regarding the definition of testing. When we are trying to form our own understanding about what testings is, we need to be aware of how different that is in the end from many others views.
You mentioned in the end:
“You might not agree with them, and you might very well try to explain your view, hoping to persuade somebody, or maybe to be enlightened yourself, but in the end… is it not whoever pays for your services who decide what testing IS… in that given context?”
This is one of those challenging matters I also think about as a tester. Like, how should we behave if we work in a project and we are said that you will not test anything unless it is done against requirements (I’ve heard a project manager declaring this as their approach in one testing event)? I know many would say that we leave and say this is not for us. Or is there other ways to handle the situation if we for some reason are stuck in a project like that for a while? Like, unless there is any threat that our actions cause danger, should we /accidentally/ test a thing or another here and there or for example harder push the need for additional testing?
Overall, I find it challenging to draw the boundaries on what is our job as a tester. Depending on our context and skills, we can often affect on many things in software development. In the end it’s up to what we should and what we shouldn’t.
-Aleksis
Hi Aleksis, thanks for the comment 🙂
I guess the “easy” answer is to make sure you are so damn good at what you want to do that any potential employer would be begging you to come do it for them.
Easy peasy indeed.
My impression is that many organisations prioritise the short term (or short sighted?) goal of producing the “correct” documents according to some process description over the need for good content in those documents. When talking about it, everybody agrees about the need for good testing and good quality, but once work is to be done people come dragging in the good old templates thats been used the last twenty years and nobody is even remotely interested in ensuring the testers have the skills needed to design good tests. As long as work progress as defined in the process, everything is just peachy.
in such cases one can of course choose to quit, or try to start discussions… and get the ball rolling. But that might be difficult if you are one of very few in the organisation who has read a testing related blog, book, or magazine the last twenty years.
I think it was Jurgen Appelo (http://noop.nl) who wrote something like: The only reason why there are bad companies, is that people don’t quit.
I guess there is quite a bit of truth in that but it is not for everybody to just up and go create their own business if there is none nearby that suits their needs.
…and if the available jobs are scarce, and you need a steady income… I guess you might not be the one having the final say in defining “Testing”.
Sorry, just to pick you up on your last paragraph: You don’t need to have the final say. You should always do what you’re told by your boss. But not having the final say does not mean that you do not have a say. If someone asks me, for example, to write a set of manual regression scripts I will say something like the following:
“I can do that for you but it will probably take a long time. It is likely they will become out of date or unused or we will inherit a large maintenance cost, and I don’t think we’ll see a return on that investment in terms of visibility of quality.”
I can then back that up with reason, evidence, alternatives and argument. If at the end of that they still want a set of manual regression scripts then I’ll write them. But when we’re a few months down the line and we’re spending half our time updating the scripts and half our time not finding bugs then nobody can accuse me of being the problem because I voiced my concerns about the entire enterprise. If I hadn’t said anything then I would have been silently complicit in the whole sordid affair.