I was recently part of a panel discussion around the topic “What is quality?” and an interesting question came up. Is it always a choice between speed of delivery and the quality of that delivery? The thinking being that if you focus on one you will sacrifice the other. Therefore, you have to choose one or find some way to balance the two which typically translates to being mediocre at both. That’s when it occurred to me that this is actually a version of working harder in teams and what we should be doing in working smarter.
Working Harder for Speed or Quality
Working harder in the context means doing the work in the same way you have always done it but trying to do more of it. This typically leads to people taking one of two approaches. One is by working longer hours and the other is taking shortcuts by skipping steps not providing immediate value, then using the spare time for doing other work.
At first both these strategies work quite well, you get the speed improvement and quality remains about the same. The problem is that all the effort is put into doing the work and nothing into improving the capability to do the work. If sustained for long periods those initial speed improvements will begin to slow and quality will diminish as the system gets harder to work with. Essentially you degrade your capability to do that work which further slows you down and brings quality down with it.
This is probably why most people, including myself, have always thought of speed and quality as trade-offs against each other. Every time we’ve tried to go faster, the quality of the system has gone down. On the other hand when we work more systematically and usually slower, however, things remain more stable.
But is there another way?
Working Smarter for Speed and Quality
In this case working smarter is not focusing on speed or quality but on your capability to do the work. Instead of just trying to go faster or improving the quality of the work, attention is focused on how that work is done. Specifically if any improvement can be made that could lead to improved speed, quality, or both.
This is where things can be a little counterintuitive. At first, while the team experiments with their capability, their speed is likely to drop as they figure out a new way of working. There is an even higher chance of quality being affected as things can be missed, or unintended bugs are introduced. But, as long as the team can collaborate effectively and learn from their failures the speed and the quality is very likely to improve in the long run.
The Trap of Working Harder
The problem with working harder is that it’s addictive once you start. When you work harder you tend to see immediate results so it feels productive. Not only that but others notice it too and are likely to praise you for going the extra mile. That initial speed boost can make it feel like you’re getting more done in less time, especially if you’re taking a shortcut here and there. This can trap you into thinking that working harder is the best way to work. The long-term damage of not maintaining your way of working is that it can end up being the only way to get things done.
The Virtuous Cycle of Working Smarter
Once teams start improving their capability to do the work it tends to free them up as they are working more efficiently. This additional capacity is re-invested into spending more time on improving their capability further which can create a virtuous cycle of improvement.
Let’s call it Continuous Improvement
Personally I’m not a fan of saying you shouldn’t work harder but work smarter instead. Not only does it come across patronising but it leads to more ambiguity and misunderstanding. I do appreciate that we need to have some form of common language and having a defined terminology helps minimise the ambiguity and risk of misunderstandings.
If we must call “working smarter” something then I’d opt for continuous improvement. While there is still a chance of it being misinterpreted, at least it has two of the keywords that working smarter is trying to achieve.
Conclusion
Working harder and working smarter are two ways you can approach and think about how you are working in your teams. What we need to recognise is that working harder only allows you to choose between speed of delivery or the quality of that delivery. While working smarter, over time, can enable teams to deliver a quality product at speed.
A good place to get started is having a shared understanding of what working harder looks like for you and your teams. How does it help and hinder your ability to deliver at speed or a quality release? From that identify areas that you can measure so you can objectively show that things are either getting faster, but quality is dropping or slower but quality is stable/increasing. Some good measures of speed are throughput of delivery such as leads times and release frequency. Quality can be tricker as it depends on what quality means to your team but a good proxy can be stability of releases based on change failure rates and mean time to recovery.
From there you can start experimenting with different ways of working that could incrementally improve your ability to do the work. All the while measuring to see if it is moving you towards delivering a quality product but at a pace that is faster than what came before. Which will enable you to move towards working smarter and begin continuously improving your ways of working
Further reading
I’ve only briefly described these two ways of working in this post. A much more thorough and detailed explanation is given in a 2001 California Business Review article “Nobody ever gets credit for fixing problems that never happen: Creating and Sustaining Process Improvement” by Nelson Repenning and John Sterman for which I must thank Joep Schuurkes who shared it with me on twitter.
Special thanks to Sarah Irving for proofreading and providing numerous suggestions to help make this post better than I could have alone.
Exploratory testing is about testing in an unpredictable context and therefore detecting unpredictable failures in our software. Automated testing is about testing in a predictable context and therefore detecting predictable failures. The mistake we make with automation is we try to apply it to the wrong context. You can’t use testing methods developed for predictable context in an unpredictable environment.
While there is nothing physically stopping you neither practice is particularly efficient if used in the wrong context. Exploratory testing in a predictable environment would just confirm what you already knew only slower and less consistent when repeating the testing . While automated testing in an unpredictable environment would lead to false negatives.
It’s also not a one size fits all solution either as we work in both contexts. Predictable when initially developing the software and unpredictable once running in the live environment.
The only way you can replace exploratory testing with automation is to make the test environment predictable. But that would then mean you are trying to detect predictable issues. This then negates the outcome you were looking for which is trying to detect unpredictable or complex failures.
Testing in unpredictable contexts
The best way to detect unpredictable failures is to use methodologies that can operate in an unpredictable environment.
One of the best known methods is exploratory testing (sometime called manual testing) but there are other technique too. Such as monitoring of the live environment. Which is good for issues we can predict in an unpredictable environment. Observability using logs, graphs and other telemetry to see how the system is behaving in the live environment. This is helpful for issues we can’t predict and need to debug in the live environment. Phased rollout of features using techniques such as feature toggles, blue/green deployments, canary releasing etc. Useful for limiting the impact of unintended issues in a unpredictable environment. Basically anything that allows you to slowly enable a feature for subsets of users.
Using monitoring and observability in conjunction with phased rollouts can greatly improve your ability to understand and limit how new code behaves in unpredictable environments.
Testing in predictable contexts
This is not to say automated testing is invaluable as it can help detect smaller predictable issues. Which if left unchecked could develop into larger unknown failures that only occur with the right mix of other smaller issues. Some issues maybe within our control (software we develop) and some outside of our control (other people’s software). For software in our control (a predictable environment) automated testing is almost a prefect match. For software outside of our control (an unpredictable environment) contract testing, exploratory testing, monitoring and observability and phased roll outs of software is preferable.
Control and isolation
Next time you’re looking at testing techniques think about how much control (and therefore isolation) you have over your test environment. The greater the level of control then the more automation you should consider, but the less control you have then the more you should consider exploratory testing coupled with monitoring, observability and phased rollouts.
Testing techniques
The following diagram will help you see how different testing techniques stack up against each other. This is by no means an exhaustive list and is only comparing them on a speed of feedback, value of feedback and testing environment bases. So the next time you get into a discussion about testing you could use these characteristics as a good way to frame that discussion.
Testing techniques plotted on a speed, value and environment axis
Are there testing techniques that should be plotted on the chart?
Do you agree with the axis? Is there another more important characteristics of testing that should be captured?
If you ask testers what does QA stand for most are likely to say its Quality Assurance and is typically described as providing confidence that some quality criteria will be fulfilled. Some people in the testing community believe that it’s actually Quality Awareness*. The thinking goes how can tester assure the quality of something if they never built it in the first place? All they can do is make your team aware of the quality. I agree with both explanations and believe that they are different sides of the same quality coin.
Read on to see what this means for testers and how it’s a useful to consider both in teams.
*I’ve also heard Quality Advocate and my favourite Question Askers, both fit this model of Quality Awareness.
…that someone could be their Products Owners (PO), the organisation they work for, the team they work with and their end users and all these groups of people could have very different views on what value means to them and even contradicting in some cases.
https://www.jitgo.uk/lenses-of-quality/
Again from the same post on what value is:
Each of these groups of people view quality with a different lens therefore see the same system differently to one another. We as testers should help our teams to see quality through these different lenses by helping them identify these groups and what their measures of quality are.
https://www.jitgo.uk/lenses-of-quality/
Simply put value will depend on the viewpoint of the person. Identify whats viewpoint that person sees the product/system through then you’re halfway there to working out what’s valuable to that person. If you take this a step further and see what incentives drive that persons viewpoint then you might identify whats valuable to them too. But thats not as easy as it sounds.
For any given team there are multiple members and stakeholders therefore they are likely to be a number of unique and overlapping quality attributes too. Identifying the key ones for each cohort of people is a valuable exercise for any team. It might help explain why some people are never happy no matter what you deliver.
a statement that something will certainly be true or will certainly happen, particularly when there has been doubt about it
One way to think about assurance is that it’s a promise that an outcome will happen so as to give others confidence. In this case it’s quality and as mentioned earlier quality means value to someone. Therefore QA or Quality Assurance means providing confidence in the quality of the product to stakeholders. This it is about certainty that an outcome will happen not a guarantee. You are saying that best efforts will be made and this is an assurances in making that happen.
knowing something; knowing that something exists and is important
interest in and concern about a particular situation or area of interest
This would lead to QA or Quality Awareness to be a person who is interested in quality (value), understands its importance and has knowledge about the quality of a product or system. To take this a step further someone who works in Quality Awareness understands that quality is value to someone, knows who those people are, what viewpoints they hold and possibly what incentives drive those views. They are then able to apply this knowledge to the system and subsequently increases their teams awareness of overall system quality. Essentially Quality Awareness is about building awareness of what quality is in a team, how it is affected and who it matters too.
Quality Awareness sits at the intersection of the team that produces the system, the domain in which the system operates in and users of the system.
Two sides of the same coin
Both are focused on quality but one is about improving the teams understanding of what quality means for their stakeholders. While the other is focused on maintaining (and hopefully improving) the quality of the product.
In this scenario Quality Awareness can improve Quality Assurance by giving it the metrics with which quality is being assessed on. In this model Quality Assurance can actually fulfil it’s job of providing confidence that the quality of the product is being upheld. Why, because it is taking into account who the stakeholders are and what is valuable to them. Then converting that value into a measurable metric which the engineering team can either:
assess themselves against to make sure they are doing what they said they would do or
provide the stakeholders the metrics to improve their confidence that not only is the engineering team doing their job but maintaining and perhaps even improving it.
The thing to keep in mind is not all quality values can be converted into an easy to measure metric. You can use proxy measures to give you an idea but some measures are always inherently subjective. On top of the this the systems we build are interdependent on other systems which are out of our control and can affect the quality of our systems. Therefore techniques such as exploratory testing can be very beneficial as it can help build a fuller awareness of what quality means for your product.
Back to Quality Assurance?
Does this mean we should go back to using QA again and naming ourselves the QA Team? No, we’ve come a long way in some areas of our industry and going back to QA teams might bring back all those old problems. Such as test team silos, testers being the gatekeepers and why didn’t we catch that bug?
How is this helpful?
Where this can be useful is giving teams another lens through which to look at their testing approach and think is this heading towards building assurance of quality or is this about raising awareness of quality. By separating out into these two camps you can see the value it’s actually going to bring and if its worth the investment. It might also help clear up who should be doing what and when.
Using this model of awareness and assurance could be helpful for testers trying to figuring out what it is that they want to do with their careers. Do you want to learn more about building team confidence in quality through test automation (Quality Assurance) or building a quality culture within teams (Quality Awareness)?
Below are my personal notes from Amy Edmundson excellent article Strategies for learning from failure. It’s a long read but I highly recommend it over my notes as it goes into a lot more detail then I have covered.
Summary
Not all failures are the same and categorisation of failures can make a big difference in enabling learning from them.
Why should testers care?
Considering we deal with software failure all the time we have a tendency to forget the human cost of failures. Especially in terms of how that failure occurred (the team), how that failure affects the users and the outcome for the business. This article is a great introduction in how we can learn from failure first and then how we could enable our teams and business to learn from them by reframing errors as different types of failure.
[Organisations] that catch, correct, and learn from failure before others do will succeed
Amy Edmundson
Amy classifies failure into three types of categories
Preventable
Complex
Intelligent
But we have a tendency to view all failures as one type. In software testing we group them into different levels of risk but generally all failures are error. Which means something isn’t right and should be avoided. We’ve started to try and learn from them but the need for interdisciplinary teams to do so is a cost that is often too high to pay so doesn’t happen very often. I think if we focused our efforts to investigate complex failures we can use the learnings to start minimising preventable issues and stop some of the them happening altogether.
How should we respond to failure?
Some people believe that respond constructively to failures could give rise to an anything-goes attitude. They think that If people aren’t blamed for failures, then how else will they try as hard as possible to do their best work? But this has a tendency to try and avoid failure and in some cases cover it up.
What we actually need is culture that makes it safe to admit and report on failure (so we can learn from them) which coexist with high standards for performance (to make use of that learning to get better).
The blame game
If people see failure as something to be avoid you end up in the blame game. Which has a spectrum of reasons for failure from blameworthy to praiseworthy:
🤔Notice how things that are blameworthy are about individuals but praiseworthy are all about the things.
I wonder how many time people don’t blame others but themselves for the failure and hence keeping quiet or downplaying issues when they occur?
To embrace failure we need to classify it better then the catch all term that failure encourages. Amy Edmundson suggest these three categories: preventable, complex and intelligent failures.
Preventable
These are usually found in routine tasks that are well defined and the outcomes are well understood
Preventable failures tend to occur when we deviate from this routine
In software engineering certain routine task can and should be automated. Such as build processes and specific types of checks
If they do need to be performed manually then tasks lists and check lists are well suited to these types of tasks
Note: exploratory testing falls under intelligent failures
Failures which result from these types of tasks can usually be mitigate through better understanding of the work we do, how we do it but most importantly why
When we spot these types of failures (deviation from the routine) we should immediately address them
This is in part about stopping errors from being passed down the process and building quality in
Complex failures
Many systems we work in are complex and too big for any one person and in most cases even groups of people to fully understand
This means complex systems can be unpredictable and ambiguous and fail in ways we could not have anticipated
The way in which complex failures occur can in some cases be traced to things all happening in just the right way
But assuming failures will never occur can be counter productive and we should build into the process to handle what happens when things go wrong
When complex failures do occur we should recognise them as such and investigated them in a praiseworthy way to understand all the components that led to the failure and identify if any of the smaller issues that resulted in the failure can be made preventable
For example
Most accidents in hospitals result from a series of small failures that went unnoticed and unfortunately lined up in just the wrong way.
Intelligent failures
Named by the Duke University professor of management Sim Sitkin as intelligent failures
These are the failures that occur during experimentation
They help you understand what works and what doesn’t
And importantly quickly
These are situations where the answers are not knowable in advanced
The only way you can find out is to actually do it
Exploratory testing is all about raising awareness of intelligent failures
As Amy Edmondson calls them they are failures at the frontier
Situations that haven’t happened before
Or maybe won’t happen again
For software engineering this is a lot of the work that we are doing
Hence agile software development so we can adapt to the changing environment
To do things in a way that helps you learn from your work
We should be producing lots of intelligent failures that help us learn about the system we’re building , the people that use it and the domain in which it used
Exploratory testing is all about exploring a system and seeing in what ways it can fail to better understand how it works
Small experiments over Big Bang experiments
At the frontier, the right kind of experimentation produces good failures quickly. Managers who practice it can avoid the unintelligent failure of conducting experiments at a larger scale than necessary.
Trail and failure?
“Trial and error” is a common term for the kind of experimentation needed in these settings, but it is a misnomer, because “error” implies that there was a “right” outcome in the first place.
Tolerance of failure
We need to be able to accept complex and intelligent failures and understand that doing so does not mean mediocrity. Tolerance is actually something that we need in order to be able to learn from these types of failures. The problem with failure is that there is almost always an emotional element to it and so needs leadership to enable the learning that needs to happen.
How do you learn from failure?
Leaders should insist that their organizations develop a clear understanding of what happened—not of “who did it”—when things go wrong.
This requires consistently:
reporting failures, small and large;
systematically analysing them; and
proactively searching for opportunities to experiment.
Anyone working on experimental work needs to clearly know that the faster we fail the faster we will succeed but most people don’t understand this subtle but important concept.
The quicker things fail the quicker you can pivot or try another idea that can succeed
But the longer that failure takes the longer you are executing on an idea that will not help your objective
What is the opportunity cost of working on one thing and not the other?
Some people may approach experimental work as if it’s well defined and understood such as production line style of work where you need to produce the same thing over and over.
For example, statistical processes control, which uses data analysis to assess unwarranted variances, is not good for catching and correcting random invisible glitches such as software bugs.
In a typical software team this would be predefined test cases or automated checks
There are three main ways to learn from failure: detection, analysis, and experimentation.
Detection
We need to detect and make issues visible earlier on in our processes before they become bigger issues later on
Don’t shoot the messenger
Unfortunately a lot of people are reluctant to raise issues early on in the process for all manor of reasons. The biggest culprit being people unwilling to take interpersonal risks in raising issues.
One of the best ways to combat this is for management to lead by example and not only encourage the raising of issues earlier on in the process no matter how small but also applauding the people that do and having a system in place to make something happen about it.
Another issue is a human tendency to not admit failure due to the stigma attached to it “it failed therefore I’ve failed”. Therefore people keep going hoping that things will get better when they should have admitted failure or worse they haven’t realised they’ve failed due to inadequate measures or goal when starting out.
Changing the stigma around failure is one way to improve the situation such as failure parties to encourage the reporting of failures and help people look at the situation in another way.
Example of how other organisations detect errors
Through speaking up supported by management from Amy Edmundson:
In researching errors and other failures in hospitals, I discovered substantial differences across patient-care units in nurses’ willingness to speak up about them. It turned out that the behavior of midlevel managers—how they responded to failures and whether they encouraged open discussion of them, welcomed questions, and displayed humility and curiosity—was the cause. I have seen the same pattern in a wide range of organizations.
Building quality in
The idea of the andon cord from the Toyota production system is doing just this; noticing small deviations in process and correcting them there and then to constantly improve the system.
For software engineering this is all about building quality into the process instead of inspecting it at the end. Inspecting at the end is almost too late to make difference due to the increased cost in time and cognitive load to make the change. This usually ends in discussion such as /users are never going to notice X/, /no one is ever going to do Y/ or /let’s see if it’s going to become a problem first/.
Analysis
Once failures have been detected it is important to not just look at the symptoms of the problem and move on but to dig into the root cause of the issues.
Unfortunately we tend to not want to do this as it can be painful to admit that something went wrong especially if we are the cause of it and can negatively affect our self esteem and confidence. There is also an element of interpersonal risk associated with admitting failure that can add towards people not wanting to spend too long looking at issues too deeply. “What if people think I’m incompetent?”
Culture is another aspect that needs to be in place for inquire into failure to occur. Digging into failures needs:
inquiry and openness, patience, and a tolerance for causal ambiguity
But a lot of organisational cultures are geared towards actions and results not reflection as needed for learning from failure.
We are also highly susceptible to fundamental attributes error. This is where we downplay our responsibility and blame external factors when we fail and do the opposite when others do.
Amy research back in 2010 showed that failure analysis is often limited and ineffective – sadly I think this is still the case for a lot of organisations.
Analysing complex failures is difficult as they tend to occur across teams and departments and due to the reason listed above most people only focus on the symptoms rather then getting at the underlying causes of the failures. Therefore it’s best to use multidisciplinary teams to carry out the investigation with the support of management that you are looking at what happened not what someone did or didn’t do.
From the NASA Colembine disaster
A team of leading physicists, engineers, aviation experts, naval leaders, and even astronauts devoted months to an analysis of the Columbia disaster.
They conclusively established not only the first-order cause: (symptom)
a piece of foam had hit the shuttle’s leading edge during launch—but also
second-order causes: (underlying reason)
A rigid hierarchy and schedule-obsessed culture at NASA made it especially difficult for engineers to speak up about anything but the most rock-solid concerns.
Experimentation
A critical activity for effective learning is strategically producing failures—in the right places, at the right times—through systematic experimentation.
For scientists
* 70% of experiments will fail
* They recognise that failure is not optional but a part of the process
* And that Failure holds valuable information that they need to extract and learn from /before the competition/ 🤔
In contrast when product companies design new products they plan for success. So they setup the product for optimal conditions that work instead of representative ones that they can actually learn from. Therefore the pilot only produced information about what does work not what doesn’t.
From Amy Edmundson:
A small and extremely successful suburban pilot had lulled Telco executives into a misguided confidence.
The problem was that the pilot did not resemble real service conditions: It was staffed with unusually personable, expert service reps and took place in a community of educated, tech-savvy customers.
But DSL was a brand-new technology and, unlike traditional telephony, had to interface with customers’ highly variable home computers and technical skills.
This added complexity and unpredictability to the service-delivery challenge in ways that Telco had not fully appreciated before the launch.
A more useful pilot at Telco would have tested the technology with limited support, unsophisticated customers, and old computers.
It would have been designed to discover everything that could go wrong—instead of proving that under the best of conditions everything would go right.
Of course, the managers in charge would have to have understood that they were going to be rewarded not for success but, rather, for producing intelligent failures as quickly as possible.
What incentives are you setting up for your employees? The things you reward are the things you will get.
What makes exceptional organisations?
exceptional organisations are those that go beyond detecting and analysing failures and try to generate intelligent ones for the express purpose of learning and innovating.
Can you think of any organisation that purposely inject failures into their system to see how they behave? Hint they named the tool after monkeys 🐒 and in the process created a whole new discipline: Chaos engineering. These experiments don’t have to be that big either:
[you] don’t have to do dramatic experiments with large budgets. Often a small pilot, a dry run of a new technique, or a simulation will suffice.
recognise the inevitability of failure in today’s complex work organizations. Those that catch, correct, and learn from failure before others do will succeed
Back in March 2018 I visited The Design Museum in London and came across the above installation.
What you can see is technology design classics all the way from the first transistor radios on one side to the very first digital clocks on the other. With everything else in-between.
If you stand back far enough you begin to see that they are not just randomly placed on the wall but in a particular order. As each piece of technology progresses in its evolution you begin to notice that it starts acquiring functionality from the technology around it. Not only that but they start to shrink in size at the same time. Eventually you realise that all of that technology has been assumed into one device: The mobile phone which is placed right in the centre of the wall.
With the older technology its size and its complexity was on show for all to see. The mobile phone however is different. It actually looks quite simple on the outside with only a screen and a few buttons. But once you turn it on you begin to realise that this is something quite different to what has come before. It can not only provide all of the functionality from the technology that came before it but much more through the use of the internet. This isn’t just limited to mobile phones but pretty much all technology that comes after. From TV’s, speakers and wrist watches everything is slowly being interconnected via the internet.
The interesting thing about a lot of this new technology is that it is actually been developed and controlled by only a handful of companies. Who on average have more resources than a lot of other more traditional companies combined. On top of that they have oriented themselves around the users unlike any other company before always working to provide them with best experience they can come up with. It’s almost like they know every users is a click a way from moving onto the next thing but something keeps those users coming back. It sometimes look hopeless competing against them, so what do we do?
Software is eating the world
Marc Andreessen back in 2011 wrote that “Software is eating the world” which actually gives us some hope. Software allows us to compete again and perhaps tempt those users away. Remember just as the competition we are only a click a way too. But what is going to get those users to click something new?
We need to be able to try different ideas and get them in-front of our users to start seeing what works and what doesn’t based on real data and not just what people think is working.
Leadership to build Collaboration and Purpose
However to be able to start doing that we need to start working better together as software teams. Simply having the best developers is not going to cut it. Research from Google’s Aristotle project showed that this wasn’t the case but 5 other team dynamics where better predicators of well functioning teams. These being psychological safety, dependability, structure & clarity and meaning & impact.
Side note: Psychological safety is all about leadership and interpersonal risk taking and not just saying this is a safe space. Read The Fearless Organisation to learn more.
Once we can collaborate more effectively we can build psychological safety, dependability and structure into the team. From there we can start working on the teams purpose. What is the teams reason for being, what are they trying to accomplish, how will this help the organisation? Purpose is all about providing the team clarity, meaning and impact. But simply asking people to collaborate and giving them a purpose isn’t going to build the team dynamics set out earlier. It’s going to need leadership to build the type of collaboration we need that has those characteristics. Leaders will need to be more hands on demonstrating interpersonal risk, building dependability between team members and setting up what the initial structure to the team is.
What is quality?
For arguments sake let say you’ve been able to get someway to doing that. Now what? Do the user of your systems just magically start appearing? Team collaboration is only one part, now you need to start iterating on the system. You could just get the team to build whatever they think is a good idea and get them to do it as fast as they can. The risk is releasing half-baked systems that end up causing you or worse, your users more problems then before. The thing is users tend to want a quality product, but quality is subjective and so means different things depending on your view point. From the lenses of quality :
For your Organisation quality could be whatever helps them reach their targets for that quarter or year.
For your Product owner their measure of a quality product could be a system or feature released on time.
For your Team it could be a system that they can build, deploy, maintain easily.
For your Users, well it could be something as simple as it just works. – Lenses of quality
Building Quality in via Testability
If quality means different things to different people how can you build quality into a product? By building in testability instead. What testability does is start to make your system objective. Meaning that instead of people saying the system feels easier to work with or they think it works correctly you use tests to back up those feelings. Those tests have to be built into the system during development. It is not something that can be added on very easily after the fact and especially by people who haven’t built the system in the first place. Testability is not about testing the system end-to-end but piece-by-piece. Each piece being a specific type of behaviour the system provides and tested in isolation from the other pieces. The scope and definition of the behaviours should be decided on by the team collaboratively. Unit testing can help with testing like this but everyone has a different opinion on what a unit is and therefore have very different approaches to testing a unit:
4\ Everyone seems to have a different opinion on what makes a unit but also what makes a a good and bad unit tests
Which is why I have a problem with calling them unit tests and outlined how you could define them by calling them code tests first and then building a team understanding of what they are.
This type of testing is what I think gets us towards what W Edward Deeming (1900-1993) known in his time as the leading management thinker on quality when he said we should
“Cease dependence on mass inspection. Build quality into the product from the start” – W Edward Deeming
So do we just need to work better together and building in testability to solve all our quality issues?
Software ate the world, so all the worlds problems get expressed in software
It’s been 9 years since Marc Andreessen wrote Software is eating the world. Ben Evans (a business analyst who worked for Andreessen) recently said in his presentation Standing on the shoulders of giants
“Software ate the world, so all the worlds problems get expressed in software” – Ben Evans
You can build in all the quality measures you want but that doesn’t address any of the problems we’ve intentionally encoded into the system. You are going to need someone who understands how the team works (and how the problems are encoded into the system), knows how the system is deployed into the real world (and the domains in which it is used) and who those users are (and what they expect of it). That someone already exists within teams but most teams have simply been using them as a safety net to check their work and to channel my inner Deeming “Carry out mass inspections of our systems”. We’ve called them Testers but maybe it’s time we start to think of them as something else?
Software levels the playing field again and allows us to innovate in ways that no other tool before it has ever allowed. However to do so we need to work collaboratively as teams to build testability into our software systems and testers to raise awareness of what quality is for our products. From this foundation we can begin to compete again and really start offering ours users that temptation to click something new.
Which not realising at the time but fitted in nicely with Gerald Weinberg’s description of what is quality:
Quality is value to someone
For most teams that someone could be their Products Owners (PO), the organisation they work for, the team they work with and their end users. All these groups of people could have very different views on what value means to them and even contradicting views in some cases.
For your organisation quality could be whatever helps them reach their targets for that quarter or year.
For your Product Managers (PM) or Owners their measure of a quality product could be a system or feature released on time.
For your team it could be a system that they can build, deploy, maintain and add to easily.
For your end users, well it could be something as simple sounding as it just works.
Also it’s not that the testers or developers don’t care about shipping early (what the PM wants), it’s more that they might care about maintainability or it does what we said it would do more than shipping early.
All of this could be just the tip of the iceberg and there could be many other people and views on what quality means to them.
As testers we need to help development teams understand that quality is measured by people in different ways.
Lenses of Quality
One of the ways I’ve started to help teams understand this is via the idea of lenses of quality.
Each of these groups of people view quality with a different lens therefore see the same system differently to one another. We as testers should help our teams to see quality through these different lenses by helping them identify these groups and what their measures of quality are.
This would help teams to start thinking about who their stakeholders are and how they are likely to perceive the systems that they build.
Would it possible to line up all the different lenses and be able to focus on one common quality metric?
If so would this be more like a microscope pulling into focus the hidden details or more like a telescope and allow you to see far into the distance?
