To build peoples confidence with automation you first need to understand why you’re doing it.
Why do we automate things?
If you look at automation in general the reason to do it is because we have some sort of repetitive manual task that we want to be able to do automatically. By doing so you would remove any inconsistency that could occur from doing it manually. This would also make the output of the process reliable and repeatable as and when you need it. In short automation can make processes consistent, reliable and repeatable.
This usually leads to other benefits too such as the ability to scale up the automated process in terms of frequency and speed all while reducing costs in some scenarios. Essentially you can take advantage of economies of scale.
The benefits of consistency, reliability, repeatability and scalability is that this helps the people associated with that process to have confidence in the output of that automaton. They can either see the process happening again and again or can inspect the output to validate their confidence in the process. You could even take it step further and automate the inspection too.
But what about test automation?
The above works well for say automating a physical process such as making a glass bottle. You can either see how the bottle is made or inspect the end product. But when it comes to test automation you can’t “see” the test occurring (or any software process for that matter) and the only output is likely to be a result: pass or fail.
The only way to gain confidence in the automation is to either inspect the code (process) or your confidence is based more on the person doing the automation. You trust that they wouldn’t fake it or maliciously do anything wrong.
If that confidence is lacking then the only way to feel confident that the system being tested works is to test the system again. Therefore any of the benefits gained from automation have been lost as you are now duplicating the effort. The biggest loss being the economies of scale.
This is by far one of the biggest reasons why tester in teams have very little confidence in the automation. They don’t know what is covers, how it works or even if it’s being done to a high standard. If your job is to understand and raise risks within the team then this almost leaves them with no choice but to test it again.
Building confidence
If you’re a developer or automation specialist then you have two options in improving peoples confidence in the output of the automation. Help them “see” the process or build their trust with you. Both of which will go some way to improving their confidence with the process.
Better yet, help them understand the principles behind the automation which if done with humility and compassion is naturally going to lead to those people trusting you as well. By helping them understand the principles behind the automation you enable them to work out for themselves what is and isn’t being automated but also to what standard it is happening to. This then lets them see where the gaps are in the process which they can raise as risks or work to plug them up.
I’ve written about building a team understanding of unit testing which details how you can document your principles in a way that is accessible. You can use this method to document any team principle not just unit testing.
Whenever you talk about unit testing with teams they never tell you what it means to them. They go straight to of course we do and show you 100s of passing tests. Interesting thing is by calling it unit testing everyone thinks they are talking about the same thing. But when you start digging into how they understand it you begin to see that everyone talks about it and understands it differently.
What do the unit tests test?
A selection of responses to the question what do the unit tests test
A unit means different things to different people but we never stop and ask what does a unit and unit testing mean to you? Why? Well that could be risky as you’re potentially questioning someone’s ability. Which probably says more about psychological safety in your team but thats a topics for another day.
A Unit means different things to different people
So what should you call it then? Well maybe as a stop gap just call it what it is a test that checks code; Code test. Now I know what you’re thinking “thats way too generic!” Which is kind of the point because when you do that the first thing people ask is “What’s a code test?” Now you can start the discussion without anyone feeling that you’re questioning their ability.
What are Code tests?
How do you build a team understanding of what it is?
One of the best and easiest ways is to get the team together and pose them three questions:
Three questions to ask teams about unit testing
What does a unit mean to you in unit testing?
What characteristics make a good unit test?
What characteristics make a bad unit test?
Hand out sticky notes or use whatever online tool your team prefer. (Miro is a pretty good online collaborative white board). Then ask each question one at a time. If you can do it in person then doing this in a big room with lots of wall space is best as it allows for people to talk to one-another during the idea generating stage. Allowing them to talk is advantageous as people will build on top of each others ideas. But this may not be practical for distributed teams.
Building a Team Understanding of Code Tests
Once everyone has had a chance to contribute, group and theme the responses. Then as a team look through them and see if there are any contradictions or if anyone strongly disagrees with the groups. If there is then this is a perfect time to build the teams understanding of what code testing is.
Building a Team Understanding of Code Tests
If you’re looking for some inspiration then watching as a group Ian Cooper: TDD, where did it all go wrong and J.B. Rainsberger: Integrated Tests Are A Scam are a good places to start. Both these talks are quite old now so more up-to-date versions maybe available.
You may find that you need to run the sticky note exercise again to build consensus but essentially you want the groups agreement on what a unit is and what are good and bad characteristic of a test. This will give you a high level understanding of what a code test is.
What do you do once you have group agreement?
So You’ve got a high level understanding but you need to turn that group understanding into something more solid. Something that gives them
Alignment with each others understanding
Autonomy with how they actually implement code level tests
But also Accountability so not only is it their responsibility to do but to do it well
autonomy, alignment and accountability
You could just say “look at the code for examples” but as we’ve seen from before this isn’t always the best way as the intent behind the code may not be clear to everyone that reads it.
Ideally it would be something that is lightweight, but not too light that it’s too open to interpretation e.g. sticky notes. But not too heavy either that no one ever reads e.g. 10,000 word essay hidden in a Wiki.
Lightweight documentation
We need to document it in a way that is quick and easy to read and therefore remember.
The best way to demonstrate this is through an example. Now this example isn’t describing code testing (you need to have that discussion with the team first) however it has all the elements we are looking for.
Example principle
The title is short and to the point which makes it easier to remember but also acts like super short summary of the principle itself.
The first paragraph describes what it is about. The language used is really easy to understand too. It takes no effort to read and comprehend. This allows the reader to spend more time understanding the content rather than trying to decipher the words used.
The second and third paragraphs detail good and bad behaviours respectively. Finally they have a list of links that show where they have demonstrated this behaviour.
The great thing about this structure is that each part builds on top of the previous part. The title gets built on by the description. The good and bad behaviours builds on top of the description and the links give concrete examples of those behaviours so the reader can see them in action or even gives them the opportunity to add their own.
Back to the sticky notes
What To Do With The Sticky Notes
You might have worked this out already by those sticky notes will map onto this simple title/description/good/bad framework quite easily. The what does a unit mean to you would be used to write the description of what unit testing is. The key points from the good and bad characteristics would make up the good and bad behaviours descriptions. Finally all those unit tests you have should be used to demonstrate where those good and bad behaviours have been shown in your code base. You’re on your own for coming up with a snappy title.
Autonomy, Alignment and Accountability
You’ve got your lightweight documentation but how does this relate to creating team autonomy, building alignment between developers and making them accountable for their actions?
Building Alignment through a common language
The description is all about what a unit is and gives a common language for the team to use when talking about code testing. This helps to build aliment between team members.
Creating Autonomy through why not how
The good behaviours say nothing about how to write good code tests just what makes a good test within this team. Hence the focus on characteristics during the sticky notes session. The good behaviours coupled with the bad act as guard rails in what we do want and less of what we don’t. This works to keep the developer autonomy as they still have to workout how to actually do it. If they are unsure they have links to where the team have actually implemented tests that demonstrate this behaviour or they can always speak to the other developers.
Autonomy & Alignment enables Accountability
By documenting the principle using easy to comprehend language to build a common team vocabulary and describing behaviours instead of instructions to create autonomy you increase the responsibility within the development team that they are accountable for enabling the principle. Not only that it makes it that much easier for people to find more information and lowers the barrier to approaching the subject in the first place.
One of the great things about documenting things is that you can point at that thing and say you don’t agree which is a lot easier then pointing at a person and saying the same thing.
How Does This Map Onto Alignment Autonomy Accountability?
In Summary
By following this model you can begin to create a team understanding of what unit testing means to them and create a unified language so that they can talk about. It also lowers the barrier to understanding the approach for others which really helps to improve the overall team confidence in what unit testing does and doesn’t give them.
Documenting your teams understanding of unit testing using this lightweight model means that when people eventually leave that knowledge doesn’t leave with them or slowly erode from the teams memory. Another benefit is as new members join the team they can use this to build up their understanding of how the team approach unit testing.
There is a risk that the information does become outdated but you could use the new joiners as motivation for the team to re-visit old principles and see if they are still valid or need updating. You never know by including the new joiner in this process they may add something that you hadn’t considered before and gives them an opportunity to start positively contributing to the team. At a minimum it kicks starts the conversation again and allows the team to visit old assumptions and behaviours.
You could also use this model to document other principles that the team would like to work by all while maintaining their individual autonomy, alignment with one another and emphasising accountability that it’s up to them to make it happen.
Now you can see if it still makes sense calling them code test, unit tests or something else all together.
Below are my personal notes from Amy Edmundson excellent article Strategies for learning from failure. It’s a long read but I highly recommend it over my notes as it goes into a lot more detail then I have covered.
Summary
Not all failures are the same and categorisation of failures can make a big difference in enabling learning from them.
Why should testers care?
Considering we deal with software failure all the time we have a tendency to forget the human cost of failures. Especially in terms of how that failure occurred (the team), how that failure affects the users and the outcome for the business. This article is a great introduction in how we can learn from failure first and then how we could enable our teams and business to learn from them by reframing errors as different types of failure.
[Organisations] that catch, correct, and learn from failure before others do will succeed
Amy Edmundson
Amy classifies failure into three types of categories
Preventable
Complex
Intelligent
But we have a tendency to view all failures as one type. In software testing we group them into different levels of risk but generally all failures are error. Which means something isn’t right and should be avoided. We’ve started to try and learn from them but the need for interdisciplinary teams to do so is a cost that is often too high to pay so doesn’t happen very often. I think if we focused our efforts to investigate complex failures we can use the learnings to start minimising preventable issues and stop some of the them happening altogether.
How should we respond to failure?
Some people believe that respond constructively to failures could give rise to an anything-goes attitude. They think that If people aren’t blamed for failures, then how else will they try as hard as possible to do their best work? But this has a tendency to try and avoid failure and in some cases cover it up.
What we actually need is culture that makes it safe to admit and report on failure (so we can learn from them) which coexist with high standards for performance (to make use of that learning to get better).
The blame game
If people see failure as something to be avoid you end up in the blame game. Which has a spectrum of reasons for failure from blameworthy to praiseworthy:
🤔Notice how things that are blameworthy are about individuals but praiseworthy are all about the things.
I wonder how many time people don’t blame others but themselves for the failure and hence keeping quiet or downplaying issues when they occur?
To embrace failure we need to classify it better then the catch all term that failure encourages. Amy Edmundson suggest these three categories: preventable, complex and intelligent failures.
Preventable
These are usually found in routine tasks that are well defined and the outcomes are well understood
Preventable failures tend to occur when we deviate from this routine
In software engineering certain routine task can and should be automated. Such as build processes and specific types of checks
If they do need to be performed manually then tasks lists and check lists are well suited to these types of tasks
Note: exploratory testing falls under intelligent failures
Failures which result from these types of tasks can usually be mitigate through better understanding of the work we do, how we do it but most importantly why
When we spot these types of failures (deviation from the routine) we should immediately address them
This is in part about stopping errors from being passed down the process and building quality in
Complex failures
Many systems we work in are complex and too big for any one person and in most cases even groups of people to fully understand
This means complex systems can be unpredictable and ambiguous and fail in ways we could not have anticipated
The way in which complex failures occur can in some cases be traced to things all happening in just the right way
But assuming failures will never occur can be counter productive and we should build into the process to handle what happens when things go wrong
When complex failures do occur we should recognise them as such and investigated them in a praiseworthy way to understand all the components that led to the failure and identify if any of the smaller issues that resulted in the failure can be made preventable
For example
Most accidents in hospitals result from a series of small failures that went unnoticed and unfortunately lined up in just the wrong way.
Intelligent failures
Named by the Duke University professor of management Sim Sitkin as intelligent failures
These are the failures that occur during experimentation
They help you understand what works and what doesn’t
And importantly quickly
These are situations where the answers are not knowable in advanced
The only way you can find out is to actually do it
Exploratory testing is all about raising awareness of intelligent failures
As Amy Edmondson calls them they are failures at the frontier
Situations that haven’t happened before
Or maybe won’t happen again
For software engineering this is a lot of the work that we are doing
Hence agile software development so we can adapt to the changing environment
To do things in a way that helps you learn from your work
We should be producing lots of intelligent failures that help us learn about the system we’re building , the people that use it and the domain in which it used
Exploratory testing is all about exploring a system and seeing in what ways it can fail to better understand how it works
Small experiments over Big Bang experiments
At the frontier, the right kind of experimentation produces good failures quickly. Managers who practice it can avoid the unintelligent failure of conducting experiments at a larger scale than necessary.
Trail and failure?
“Trial and error” is a common term for the kind of experimentation needed in these settings, but it is a misnomer, because “error” implies that there was a “right” outcome in the first place.
Tolerance of failure
We need to be able to accept complex and intelligent failures and understand that doing so does not mean mediocrity. Tolerance is actually something that we need in order to be able to learn from these types of failures. The problem with failure is that there is almost always an emotional element to it and so needs leadership to enable the learning that needs to happen.
How do you learn from failure?
Leaders should insist that their organizations develop a clear understanding of what happened—not of “who did it”—when things go wrong.
This requires consistently:
reporting failures, small and large;
systematically analysing them; and
proactively searching for opportunities to experiment.
Anyone working on experimental work needs to clearly know that the faster we fail the faster we will succeed but most people don’t understand this subtle but important concept.
The quicker things fail the quicker you can pivot or try another idea that can succeed
But the longer that failure takes the longer you are executing on an idea that will not help your objective
What is the opportunity cost of working on one thing and not the other?
Some people may approach experimental work as if it’s well defined and understood such as production line style of work where you need to produce the same thing over and over.
For example, statistical processes control, which uses data analysis to assess unwarranted variances, is not good for catching and correcting random invisible glitches such as software bugs.
In a typical software team this would be predefined test cases or automated checks
There are three main ways to learn from failure: detection, analysis, and experimentation.
Detection
We need to detect and make issues visible earlier on in our processes before they become bigger issues later on
Don’t shoot the messenger
Unfortunately a lot of people are reluctant to raise issues early on in the process for all manor of reasons. The biggest culprit being people unwilling to take interpersonal risks in raising issues.
One of the best ways to combat this is for management to lead by example and not only encourage the raising of issues earlier on in the process no matter how small but also applauding the people that do and having a system in place to make something happen about it.
Another issue is a human tendency to not admit failure due to the stigma attached to it “it failed therefore I’ve failed”. Therefore people keep going hoping that things will get better when they should have admitted failure or worse they haven’t realised they’ve failed due to inadequate measures or goal when starting out.
Changing the stigma around failure is one way to improve the situation such as failure parties to encourage the reporting of failures and help people look at the situation in another way.
Example of how other organisations detect errors
Through speaking up supported by management from Amy Edmundson:
In researching errors and other failures in hospitals, I discovered substantial differences across patient-care units in nurses’ willingness to speak up about them. It turned out that the behavior of midlevel managers—how they responded to failures and whether they encouraged open discussion of them, welcomed questions, and displayed humility and curiosity—was the cause. I have seen the same pattern in a wide range of organizations.
Building quality in
The idea of the andon cord from the Toyota production system is doing just this; noticing small deviations in process and correcting them there and then to constantly improve the system.
For software engineering this is all about building quality into the process instead of inspecting it at the end. Inspecting at the end is almost too late to make difference due to the increased cost in time and cognitive load to make the change. This usually ends in discussion such as /users are never going to notice X/, /no one is ever going to do Y/ or /let’s see if it’s going to become a problem first/.
Analysis
Once failures have been detected it is important to not just look at the symptoms of the problem and move on but to dig into the root cause of the issues.
Unfortunately we tend to not want to do this as it can be painful to admit that something went wrong especially if we are the cause of it and can negatively affect our self esteem and confidence. There is also an element of interpersonal risk associated with admitting failure that can add towards people not wanting to spend too long looking at issues too deeply. “What if people think I’m incompetent?”
Culture is another aspect that needs to be in place for inquire into failure to occur. Digging into failures needs:
inquiry and openness, patience, and a tolerance for causal ambiguity
But a lot of organisational cultures are geared towards actions and results not reflection as needed for learning from failure.
We are also highly susceptible to fundamental attributes error. This is where we downplay our responsibility and blame external factors when we fail and do the opposite when others do.
Amy research back in 2010 showed that failure analysis is often limited and ineffective – sadly I think this is still the case for a lot of organisations.
Analysing complex failures is difficult as they tend to occur across teams and departments and due to the reason listed above most people only focus on the symptoms rather then getting at the underlying causes of the failures. Therefore it’s best to use multidisciplinary teams to carry out the investigation with the support of management that you are looking at what happened not what someone did or didn’t do.
From the NASA Colembine disaster
A team of leading physicists, engineers, aviation experts, naval leaders, and even astronauts devoted months to an analysis of the Columbia disaster.
They conclusively established not only the first-order cause: (symptom)
a piece of foam had hit the shuttle’s leading edge during launch—but also
second-order causes: (underlying reason)
A rigid hierarchy and schedule-obsessed culture at NASA made it especially difficult for engineers to speak up about anything but the most rock-solid concerns.
Experimentation
A critical activity for effective learning is strategically producing failures—in the right places, at the right times—through systematic experimentation.
For scientists
* 70% of experiments will fail
* They recognise that failure is not optional but a part of the process
* And that Failure holds valuable information that they need to extract and learn from /before the competition/ 🤔
In contrast when product companies design new products they plan for success. So they setup the product for optimal conditions that work instead of representative ones that they can actually learn from. Therefore the pilot only produced information about what does work not what doesn’t.
From Amy Edmundson:
A small and extremely successful suburban pilot had lulled Telco executives into a misguided confidence.
The problem was that the pilot did not resemble real service conditions: It was staffed with unusually personable, expert service reps and took place in a community of educated, tech-savvy customers.
But DSL was a brand-new technology and, unlike traditional telephony, had to interface with customers’ highly variable home computers and technical skills.
This added complexity and unpredictability to the service-delivery challenge in ways that Telco had not fully appreciated before the launch.
A more useful pilot at Telco would have tested the technology with limited support, unsophisticated customers, and old computers.
It would have been designed to discover everything that could go wrong—instead of proving that under the best of conditions everything would go right.
Of course, the managers in charge would have to have understood that they were going to be rewarded not for success but, rather, for producing intelligent failures as quickly as possible.
What incentives are you setting up for your employees? The things you reward are the things you will get.
What makes exceptional organisations?
exceptional organisations are those that go beyond detecting and analysing failures and try to generate intelligent ones for the express purpose of learning and innovating.
Can you think of any organisation that purposely inject failures into their system to see how they behave? Hint they named the tool after monkeys 🐒 and in the process created a whole new discipline: Chaos engineering. These experiments don’t have to be that big either:
[you] don’t have to do dramatic experiments with large budgets. Often a small pilot, a dry run of a new technique, or a simulation will suffice.
recognise the inevitability of failure in today’s complex work organizations. Those that catch, correct, and learn from failure before others do will succeed
Now you are your own boss all those things in work that keep you honest are no longer there. If you want to sit with Facebook/WhatsApp open all day you can and no one would know. But this is also chance for disaster. If you don’t keep yourself in check then your work /life balance blurs into one big blob and either you’ll feel like all you do is work or nothing at all! One leads to burn out the other to demotivation (and on the extreme side losing your job).
You can also find yourself sitting in front of your screen for even longer periods than when in work. Those pesky in-person meetings, now on zoom, so and so asking your questions, now in slack, heading out for lunch with the work colleagues, just walk to the kitchen and back to your desk, random chats when heading to and from meetings/kitchen/loo all gone, need to chat to someone just fire up a video call. All of a sudden all those “distractions” in work have stopped and now your sitting hunched over your desk looking like Emma.
What do I do?
So what do I do? Well I use a time tracking app just to log my start and end times. This not only keep me in check that I’m working but also stops me over working now that no one else is managing my time for me.
I use Hours tracker for iOS which has a free and a one off £10 paid tier. Which makes it really easy for me to clock in and out. There are also additional features that allow you to automate that process based on location and time, but I just stick with start, stop and total times. There will be alternatives for android too just have a look around the app stores for one that stands out for you.
To make sure I’m getting up and moving regularly throughout the day I’ve started to make use of my smartwatch by using a 50 and 5 minute count down timers. Currently I’ve setup a shortcut as a complication so I can start and stop it easily by tapping on the centre timer link.
I also use my desktop calendar app with regular items coloured coded for easy scanning and todo list extensively as it makes it easier for me to see what I have to do and by when. Making my calendar open to work colleagues too means they can see when I’m free if they want to book things in with me which saves a lot of to and fro when organising meetings.
Why routines matter?
The great thing about doing the above to manage my time is that it builds routine into my daily work. Routines eventually become habits, which eventually become automatic and you spend even less time thinking about them and more time for things you do care about.
The key to success: Willpower
The other potential advantage of offloading the work to these apps is that there is some evidence that doing this can help you be more productive as it can help limit decision fatigue and conserve willpower. This is when you have too many decisions to make which can lower your willpower at which point your more likely to take the course of least resistance. This can be whatever the default options are which have a tendency not to be the best ones for you.
For instance while you’re shopping you tend to be making lots of decisions (what colour, what size, what about this compared to that etc) so by the time you reach the tills your cognitive ability is pretty fatigued. This is generally when your offered upgrades, special offers and impulse buys. To which a lot of us simply say yes and out you go with a few extra things you didn’t even know you wanted.
Therefore doing things that help boost your willpower can be really beneficial such as exercise, other activities that help you stay positive and other routine building habits. Also knowing when your willpower is likely to be low is useful as you can switch up your default options towards things that are better for you e.g keeping fruit/veg snacks refillable water bottle nearby instead of sweets and sugary drinks
What makes willpower so important? Research suggests that it’s statistically the biggest contributor to academic success over intelligence , greater success overall and wealth. Willpower is your ability to do things even though there maybe no immediate or positive feedback but could result in a future gain.
Got any good tips for building routine and habits then let me know in the comments.
Back in March 2018 I visited The Design Museum in London and came across the above installation.
What you can see is technology design classics all the way from the first transistor radios on one side to the very first digital clocks on the other. With everything else in-between.
If you stand back far enough you begin to see that they are not just randomly placed on the wall but in a particular order. As each piece of technology progresses in its evolution you begin to notice that it starts acquiring functionality from the technology around it. Not only that but they start to shrink in size at the same time. Eventually you realise that all of that technology has been assumed into one device: The mobile phone which is placed right in the centre of the wall.
With the older technology its size and its complexity was on show for all to see. The mobile phone however is different. It actually looks quite simple on the outside with only a screen and a few buttons. But once you turn it on you begin to realise that this is something quite different to what has come before. It can not only provide all of the functionality from the technology that came before it but much more through the use of the internet. This isn’t just limited to mobile phones but pretty much all technology that comes after. From TV’s, speakers and wrist watches everything is slowly being interconnected via the internet.
The interesting thing about a lot of this new technology is that it is actually been developed and controlled by only a handful of companies. Who on average have more resources than a lot of other more traditional companies combined. On top of that they have oriented themselves around the users unlike any other company before always working to provide them with best experience they can come up with. It’s almost like they know every users is a click a way from moving onto the next thing but something keeps those users coming back. It sometimes look hopeless competing against them, so what do we do?
Software is eating the world
Marc Andreessen back in 2011 wrote that “Software is eating the world” which actually gives us some hope. Software allows us to compete again and perhaps tempt those users away. Remember just as the competition we are only a click a way too. But what is going to get those users to click something new?
We need to be able to try different ideas and get them in-front of our users to start seeing what works and what doesn’t based on real data and not just what people think is working.
Leadership to build Collaboration and Purpose
However to be able to start doing that we need to start working better together as software teams. Simply having the best developers is not going to cut it. Research from Google’s Aristotle project showed that this wasn’t the case but 5 other team dynamics where better predicators of well functioning teams. These being psychological safety, dependability, structure & clarity and meaning & impact.
Side note: Psychological safety is all about leadership and interpersonal risk taking and not just saying this is a safe space. Read The Fearless Organisation to learn more.
Once we can collaborate more effectively we can build psychological safety, dependability and structure into the team. From there we can start working on the teams purpose. What is the teams reason for being, what are they trying to accomplish, how will this help the organisation? Purpose is all about providing the team clarity, meaning and impact. But simply asking people to collaborate and giving them a purpose isn’t going to build the team dynamics set out earlier. It’s going to need leadership to build the type of collaboration we need that has those characteristics. Leaders will need to be more hands on demonstrating interpersonal risk, building dependability between team members and setting up what the initial structure to the team is.
What is quality?
For arguments sake let say you’ve been able to get someway to doing that. Now what? Do the user of your systems just magically start appearing? Team collaboration is only one part, now you need to start iterating on the system. You could just get the team to build whatever they think is a good idea and get them to do it as fast as they can. The risk is releasing half-baked systems that end up causing you or worse, your users more problems then before. The thing is users tend to want a quality product, but quality is subjective and so means different things depending on your view point. From the lenses of quality :
For your Organisation quality could be whatever helps them reach their targets for that quarter or year.
For your Product owner their measure of a quality product could be a system or feature released on time.
For your Team it could be a system that they can build, deploy, maintain easily.
For your Users, well it could be something as simple as it just works. – Lenses of quality
Building Quality in via Testability
If quality means different things to different people how can you build quality into a product? By building in testability instead. What testability does is start to make your system objective. Meaning that instead of people saying the system feels easier to work with or they think it works correctly you use tests to back up those feelings. Those tests have to be built into the system during development. It is not something that can be added on very easily after the fact and especially by people who haven’t built the system in the first place. Testability is not about testing the system end-to-end but piece-by-piece. Each piece being a specific type of behaviour the system provides and tested in isolation from the other pieces. The scope and definition of the behaviours should be decided on by the team collaboratively. Unit testing can help with testing like this but everyone has a different opinion on what a unit is and therefore have very different approaches to testing a unit:
4\ Everyone seems to have a different opinion on what makes a unit but also what makes a a good and bad unit tests
Which is why I have a problem with calling them unit tests and outlined how you could define them by calling them code tests first and then building a team understanding of what they are.
This type of testing is what I think gets us towards what W Edward Deeming (1900-1993) known in his time as the leading management thinker on quality when he said we should
“Cease dependence on mass inspection. Build quality into the product from the start” – W Edward Deeming
So do we just need to work better together and building in testability to solve all our quality issues?
Software ate the world, so all the worlds problems get expressed in software
It’s been 9 years since Marc Andreessen wrote Software is eating the world. Ben Evans (a business analyst who worked for Andreessen) recently said in his presentation Standing on the shoulders of giants
“Software ate the world, so all the worlds problems get expressed in software” – Ben Evans
You can build in all the quality measures you want but that doesn’t address any of the problems we’ve intentionally encoded into the system. You are going to need someone who understands how the team works (and how the problems are encoded into the system), knows how the system is deployed into the real world (and the domains in which it is used) and who those users are (and what they expect of it). That someone already exists within teams but most teams have simply been using them as a safety net to check their work and to channel my inner Deeming “Carry out mass inspections of our systems”. We’ve called them Testers but maybe it’s time we start to think of them as something else?
Software levels the playing field again and allows us to innovate in ways that no other tool before it has ever allowed. However to do so we need to work collaboratively as teams to build testability into our software systems and testers to raise awareness of what quality is for our products. From this foundation we can begin to compete again and really start offering ours users that temptation to click something new.
We as testers are always looking at ways to improve our testing processes and one way to do that is to move away from the In Test column. You can read more about the why in my post “In Test” column but the how I left pretty vague. I’d like to outline one way in which you could but just like removing the in test column it may seem counterintuitive; by adding even more columns.
How does adding more columns help? Well let me walk you though the process and all will become clear…
Breaking down the In Test column
“What did we actually test?”
One of the first things you’re going to need to do is break down what you do in the “In Test” column. Now this might not be as easy as it sounds but one of the best ways to do this is try to answer the question “What did we test the last time a ticket went through the in test column?” You want to think about all the different activities you carried out for the last few tickets. One way to do this is get some sticky notes and on each one write down the specific type of testing you did e.g. retested defect No. 5487 or pair tested new feature X etc. Then group them up under headings that make sense such as: regression testing, accessibility testing, performance testing, smoke testing, feature testing, automated UI testing , etc. This is best done as a group with people who are familiar with the testing process as they will keep you honest with what you do and don’t do during testing.
These sub-heading are going to become your new columns that come under the banner of Testing.
Now for each sub-heading you want to create entry and exit criteria for when a ticket can be placed into that column. So for instance what entry criteria does a ticket need to begin Accessibility testing e.g. there needs to a UI element to the ticket. What criteria would that ticket need to satisfy so it can leave (exit) the column? E.g. feature run against accessibility guidelines, any issues not fixed have been communicated to accessibility team/product owner etc
So you’ve got all these extra columns now what?
Well, you could just leave it at that and simply making the work more visible the team has a better understanding of what work we as testers actually do. Or you can start to use it as a tool on where to focus your Test improvement process.
Test improvement process
By making all the different testing activities visible you get other benefits too:
Makes testing work explicit instead of being hidden under the banner of “Testing”
Entry/Exit criteria helps the team understand why that testing needs to happen
And when it’s complete
Length of time a ticket spends in the columns start to make bottlenecks (constraints) visible
Bottlenecks
By recording how long a ticket spends in each of the columns you can start to see which of the activities is taking the longest.
This will identify the most valuable candidate to start your improvement process. Anything before or after the bottleneck is not going to make as greater impact because the biggest issue isn’t being addressed.
Once you’ve identified your bottleneck you can as a team look at the entry/exit criteria as a starting point for improvements. You could look to see if the risks this type of testing is mitigating against can be address in some other way. For example earlier on in the process or rather than manually in some automated fashion but remember the automation fallacy.
Eventually you will find that the bottleneck “moves” to another part of the testing process which is your next candidate to carry out your improvement process. If you keep going you will start to see that tickets spend less and less time in the identified columns. In some cases you will see that the columns isn’t even need anymore and can be removed all together.
This is known as the 5 focusing steps from the Theory of Constraints. You can find more on Wikipedia and elsewhere but these are:
Identify the system’s constraint(s)
We do this by measuring how long each tickets spends in the columns and work out which testing task is taking the longest.
Decide how to exploit the system’s constraint(s)
By looking at the entry and exit criteria and seeing what could be improved as a team.
Subordinate everything else to the above decision(s)
Once your improvement process has been identified assign it as a task for someone to carry out or if possible as a team work on it together (swarm).
Alleviate the system’s constraint(s)
Carry out the mitigation methods identified above, update the entry/exit criteria for the column and keep measuring how long the ticket stays in the column going forward.
If in the previous steps a constraint has been broken, go back to step 1, but do not allow inertia to cause a system’s constraint
If the mitigation method(s) alleviate the bottleneck start the process over BUT don’t stop, keep going till you’ve address all the different types of testing.
Eventually you will be left with some testing that cannot be mitigated but the entry and exit criteria will indicate exactly what testing needs to happen (column heading), why you need to do it (entry criteria) and when its done (exit criteria).
You can then simply make this a part of something that happens for a ticket and you should be left with just the “In Progress” column.
Remember this strategy doesn’t just work for improving testing but all activities that a team carries out. You just need a way to identify the activities and make them visible.
Consumer Is someone (a dev team for instance) that makes use of a third party component or a combination of components (a system). They consume the service provided by the component/system.
Producer Are the people (a dev team) who build the component or system and make it available to others to use.
Test double To keep the tests fast you will be using a Test double of the producer in the majority of your tests. More specifically a stub that is very simple and responds how you tell it to.
Remember don’t mock what your don’t own.
Avoid using mocks for contract tests otherwise you’ll be creating another job for yourself if you attempt to mock the behaviour of your producers. Always think of the producer as a blackbox so don’t make assumption on how the internals of the producer work. That is not your responsibility. A stub should be simple and easy to see how it works and will generally just respond with a simple response.
What is a Contract test?
Contract tests are automated code level tests written from the viewpoint of the consumer. They check that the producer exists, responds to a given request and responds in the format expected by the consumer. A simple rule could be
For every unique call you make to the producer write a contract test
If output from a producer is going to cause a unique behaviour change in you (the consumer) then write a contract test e.g. an error condition would fall into this category
They wouldn’t go further then this and begin to check that the response contains all the correct data or the behaviour of the producer. That’s the job of the producer not the consumer. The producer will be a black box to the consumer, simply input and output. Whatever transformations that happen to the data on the inside of the producer are unknown.
If you do test that a response contains the correct data then I would only test very specific types of data. Specifically ones that if they where not returned would cause problems for your system. For which your system should handle gracefully in response.
How will Contract testing help?
Focused If the test follows the guidance above they will focus on just the boundary at which the consumer and producer interact. Therefore if they fail you know exactly where the problem is but also what the issue is as they cover only a small area of interaction. This will allow you to quickly identify if the problem is with your integration of the producer or some other part of your code.
Fast These test will be written at the code level usually with a native unit testing framework of the language you are working with. The vast majority of the tests will also execute against a stub to keep them fast. If you ran them against the actual producer then they could run slower. Also due to each test being so focused on the interaction boundary they will run well under a second allowing the whole suite of tests to execute in a matter of seconds.
Reliable Circle of control / Circle of influence Do to the simplicity of the tests the number of false positives is very low and will only fail if something had changed within the test, your interaction with the test double or the test double itself. Everything is now within your circle of control therefore any brittleness can be remedied quickly and easily.
Automated documentation You now have tests that document your usage of the dependency that are also executable so will stay up-to-date with every change you or your dependency makes.
Running the tests
These test can now be easily kept as part of the main suite of tests within the code base and run through the development pipeline as usual. Any change to the code base would result in the whole suite of contract tests running and letting the dev team know if there was any issues.
Occasionally you would also want to run the contract tests against the real dependency separately from the main build pipeline just to let you know if the contract had changed and that your test double is still a true stand-in for the real thing.
New version of the dependency
Now when a new version of the dependency is released you can run the contract tests against it and check to see if there are any breaking changes. If no issues are detected then maybe some light exploratory testing of changes detailed in the release notes.
If running the contract tests does detect an issue then it should be quick and easy to pinpoint where the issue is (you or them) and what the necessary mitigation steps should be (fix in your code or reject the release). All this while keeping your build pipeline running and your code base shippable.
If an issue is detected in the live environment then it’s going to be easy to know what changed and how to fix it. Which could be either fixing forward or backing out the change.
Confidence for the Consumer team
Contract tests allow the consuming team to move to a new version of a producer much quicker and with greater confidence than before. If something in the release notes looks risky still then your test team can carry out focused exploratory regression testing and if possible putting the update behind a feature flag for a controlled release to your end users.
What is Consumer-driven Contract Testing?
The thing with contract tests is that they are very much in the consumer domain. If the producer is making regular releases which result in the contract tests failing often then in one hand at least you know before taking the actual update but in the other you still can’t take it without work arounds or additional new releases from the producer. This may lead you to thinking about a new supplier. Why even bother with all the pain with writing contract tests when you knew this already?
What if you could help your producer see that each update is going to cause you issues before they even made a release? What if they told you prior to making the release that they need to introduce a breaking change or better yet that the current API will be deprecated after a certain date/version allowing you to move to the new API in your own time? What if you could work with your producers collaboratively that way they get what they want (easy and quick uptake of new versions) and you get what you need (new bug fixes/features, improved confidence of each update working as intended, less time testing)? This is where Consumer-driven Contract testing can help and really starts to show the benefits of Contract testing.
Benefits of Consumer-driven Contract Testing
As mentioned earlier the Contract Test sit in your circle of control. That is everything in this domain is in your direct control. The producer however is out of your control but can be in your circle of influence.
Note the level to which you will have influence over your dependency will depend on your overall relationship. If they are within the same organisation then things maybe easier, outside of your org but a supplier that you have a financial contract with then probably require some contract negotiation so not impossible but still some effort. No financial contract and just something you use through an open source license then contract testing is all you will likely have as a relationship.
One of the ways to start moving your producers into your circle of influence is to start a dialogue with them around your contract tests. These tests will show the producer exactly how you integrate their service and the types of response you expect from them. Also due to the simplicity of the tests and test doubles it should be easy for them to understand without your intervention (another good reason to keep them focused and simple).
Showing them the tests is a good place to start (it’s just code that’s what we are all working with none of that touchy, feely stuff about relationships) but a better way to progress the relationship, sorry, chat would be see if they could run the contract tests as a part of their development pipeline. Perhaps every time they plan to make a release or better yet on every commit (another reason to keep the tests fast and reliable).
This way they not only see how you use them, but they get an early warning if any changes in their code is likely to cause their consumers any problems. They can then see if they really need to make that change or see how they can mitigate the impact to their consumers. If they need to do it they can start a dialog with their consumers and start to migrate them onto a new API. This all helps to improve the relationship between consumer and producers, facilitated with some simple tests. Who knew testing could build stronger relationships between development teams?
Who owns the Contract tests?
Just in case it’s not clear the responsibility to write the contract tests in the first place is always with the consumers. It’s only them that know how they plan to use and integrate the producers. The producers can always offer best practice and how they intend consumer to use their services but it’s up to the consumers to decide if they plan to use the service the way it was intended.
Contract tests only become Consumer-driven once they are executed by the producers. Until then they are just Contract tests and even then just in name. If they test anything more then what was outlined earlier they become something else entirely.
New problems to solve
Figuring out how to share the tests, run them, making the results visible and letting consumers and producers know about breaking changes is a whole host of other issues that need to be resolved. The web testing frameworks have already made some progress in this area but I don’t know of any tools that facilitate this between internal teams other then having access to each other’s build infrastructure and source code repos.
Don’t use contract tests to do functional testing
Contract tests need to be quick and simple to understand and therefore only test at the boundary. If you go further than this they will become more complicated and harder for other teams to understand.
It’s not the producers team to understand how you use their service but giving them some insight into how you integrate it could be beneficial to both teams. There is nothing stopping you from writing more integrated tests but don’t expect your producer to run these. This is your responsibility and the feedback from this would be more beneficial to you than them. Besides you don’t want them thinking you’re trying to fob your testing onto them.
If you do more testing further than what was described above don’t call them contract tests otherwise you’ll cause more confusion. Be specific and call them what they are.
I’ve been thinking a lot about contract testing lately and trying to explain why it’s a good idea. I thought I’d start by getting my initial arguments for it down and go from there.
Got an opinion then let me know.
Note: This is a first draft (published 24/10/19) and I’ll (hopefully) revisit it again soon but in the meanwhile here is me thinking out in the open…
Update 18/11/19:Added more details on what contract testing actually is.
The “Contract Testing Chat”: The Reality, The Problem, The Possible Solution?
Aim: To encourage dev teams to use contract testing to manage their integration of dependencies
The Reality
Within any of the systems we produce there are components from external teams as this allows us to focus on what is important to us and let the other teams take care of thing that are not our core competency.
In an ideal situation we would probably make everything ourselves so we have complete control but that would require significant amounts of Time, Money and Skills.
Time – To train your existing staff or recruit the people that have the skills and then allow them to actually build the component/systems.
Money – To hire the people and all the necessary resources they need to do the job.
Skills – That the person needs to be able to do the job
Some organisations can throw money at the situation and recruit the best in the industry and do everything in-house. Think of large organisations with deep pockets and large global brands.
Others (like us) don’t have this luxury and have to rely on external teams and component makers to make up for the parts we choose not to focus on. This leaves us with a dilemma.
The Problem
Do we just simply trust that these external components (dependencies) will work as we hope and the teams maintaining them will let us know when things change? What most teams do is integrate the component and put it through a couple of rounds of exploratory testing just to be sure things still work as we intended. If an issue is found then it’s a matter of understating what the problem is and where the problem could actually be and who’s responsibility it is to fix it.
This strategy works quite well once the initial issues have been ironed out. Eventually though a new version of the component is released and you need to decide if you test everything again or trust the release notes and just do focused regression testing.
Possible solutions
Focused Regression Testing If you just do focused regression testing and an issue is found in the live environment then trust in the dependence maintainer and possibly the development team integrating the component is diminished. The general response to this is to do a full regression of the integrated component every time.
Full Regression Testing Full Regression Testing usually takes more time, money and skills so teams only integrate newer versions of the dependencies if they really have too. Generally when it contains something that they need e.g. a new feature or bug fix that affects the team directly.
But because of the large gaps between integration of the previous and latest dependence there are likely to be even more changes then the team anticipated so not only does a full regression now have to happen but there are likely to be more issues found leading to even longer lead times in integrating the component. The blame game normally starts about now, see below.
Automated end-to-end testing Some teams try to address this problem with automated end-to-end UI testing. Why? Well that’s what the Testers are doing during regression testing right? Just checking the functionality of the system and finding all the issues. So if we can automate this then we can not only find these issues faster but repeatedly and freeing up the Testers to do other things. It almost looks like you address the time, money and skill question is one initial up-front cost of building out the automation see UI Automation, what is it good for?
Unfortunately these test only find what you program them to find and not only that the more components the end-to-end test run through the greater the chance of failure from false positives. If it does find an issue then you need to work out where the problem actually is: the test or the code. If it’s the code then another developer needs to investigate where that issue is and you’re heading to the blame game backlog issue.
The Blame game
The blame game is when the dependency maintainer blames the integrating team for not taking updates often enough and attempting to integrate the component in a way that they didn’t intend. On top of that any issues now found by the integrating team needs to go into the dependence maintainers backlog to be prioritised as they have other competing work to be getting on with. It’s not like they are the only team integrating their component. Meanwhile the integration team is blaming the maintainers for sneaking in features and bug fixes that they never asked for and holding up their development process.
Last resort solution?
Find another supplier Once The Blame game starts then this usually leads the integrating team down one of two paths. Find another more responsive supplier. Perhaps paying an external team to the company might solve their dependency problems. This is throwing money at the problem (see money, time and skill from earlier) or they…
Build it in-house This is all about taking back control and making it the teams responsibility to build the component. No more having to worry about another teams backlog or building things that have no relevance to your team. This is the skills part of the money, time and skills from earlier.
Both of the above solutions is a break down of the relationship between maintainers and integrators or more commonly your dev team and those PITA’s over in <insert location/team/department name here> 😆
Is there anything else we can try that could help with all the issues above and prevent the relationship breakdown? We ended up down this path as we needed to address the time, money and skills costs we couldn’t afford as a team, but all the options above results in one of the core costs having to be paid.
A possible solution?
Contract testing and Consumer-Driven Contract testing Contract testing helps address the time cost by allowing an external team maintain a dependency. This also addresses the money question as the responsibility to fund that team essentially becomes someone else problem along with the skills issue. All the dev team needs to do is integrate the dependency. So how is Contract testing going to actually help?
Whenever I see people talking about automated testing I always wonder what type of testing they actually mean? Eventually someone will mention the framework they are using and all too often it’s a UI based automation tool that allows tests to be written end-to-end (A-E2E-UI). They are usually very good at articulating what they think these tests will give them: fast automated tests that they no longer need to run manually, amongst other reasons.
But what they fail to look at is the types of behaviours these A-E2E-UI tests encourage and discourage within teams.
They have a tendency to encourage
Writing more integrated testing with the full stack rather then isolated tests
Isolated behaviour tests (e.g. unit, integration, contract tests etc) run faster and help pinpoint where issues could be
A-E2E-UI test will just indicate that a specific user journey is not working. While useful from an end user prospective someone still needs to investigate why. This can lead to just re-running it to see if it’s an intermittent error. Which is only made worse by tests giving false negatives which full stack tests are more likely to because of having more moving parts
Testing becomes someone else responsibility
This is more apparent when the A-E2E-UI test are done by somebody else in the team and not the pair developing the code
Notice ‘pair’ if you’re not a one-person development army then why are you working alone?
Pairs tend to produce better code of higher quality with instant feedback from a real person
It might be slower at first but it’s worth it to go faster later
This is really important for established businesses with paying customers
A research paper called The Costs and Benefits of Pair Programming backs this up but it’s nearly 20 years old now so if you know of anything more recent let me know in the comments.
Pushing testing towards the end of the development life cycle
The only way A-E2E-UI tests work is through a fully integrated system therefore testing gets pushed later into the development cycle
You could use Test doubles for parts but then that is not an end-to-end test.
Slower feedback loops for development teams
Due to testing being pushed to the later stages of development developers go longer without feedback into how their work is progressing
This problem is increased further when the A-E2E-UI tools are not familiar to the developers who subsequently wait for the development pipeline to run their tests instead of doing it locally
Duplication of testing
As the A-E2E-UI test suits get bigger and bigger it becomes hard and harder to see what is and isn’t covered by automation
This leads to teams starting to test things at other levels (code and most likely exploratory testing ) which all add to the development time
These are just some of the behaviours I’ve observed A-E2E-UI tests encourage, but they also discourage other behaviours which maybe desirable.
They can discourage development teams from
Building testability into the design of the systems
Why would you if you know you can “easily” tests something end-to-end with an automation tool?
Maintainability of the code base
By limiting the opportunities to build a more testable design you decrease the maintainability of the code though tests
If you need to make a change it’s harder to see what the change in the code affects
By having more fine grained tests you can pinpoint where issues exist
A-E2E-UI tests just indicate that a journey has broken and how it could affect the end users
Not where the problem was actually introduced
Building quality at the source
You are deferring testing towards the end of the development pipeline when everything has been integrated. Instead of when you are actively developing the code.
Are you really going to go back and add in the tests especially if you know an end-to-end test is going to cover it?
The responsibility to test your work
With the “safety net” of the A-E2E-UI tests you send the message that it’s ok if something slips though development
If it affects anything the A-E2E-UI tests will catch it
What we should be encouraging is that it’s the developers responsibility to build AND test their work
They should be confidant that once they have finished that piece of code it can be shipped
The A-E2E-UI tests should acts as another layer to build on your teams confidence that nothing catastrophic will impact the end users. Think of them as a canary in the coal mine. If it stops chirping then something is really wrong…
More granular feedback loops
By having A-E2E-UI tests you’re less likely to write unit and integration tests which give you fast feedback on how that part of the code behaves
Remember code level tests should be testing behaviour not implementation details
If A-E2E-UI tests cause undesirable behaviours in teams should we stop writing them? While they are valuable at demonstrating end users journeys we shouldn’t be putting so much of our confidence that our system works as intended into them. They should be another layer which helps build the teams confidence that the system hangs together.
If we put the vast majority of our effort and confidence into these automated end-to-end tests than we risk losing one of the teams greatest abilities: building testability into the design of our systems. But just like the automated UI tests building in testability takes conscious effort. This will take time, patients and experience for the whole team to understand and benefit from.
A limit on the amount of work that can be pulled into the team and most likely columns on your board that you use to manage your teams work.
Why?
The idea being that instead of work being pushed into the team as a when it comes up the team pulls the work into their work stream if they have capacity to actually work on it. This prevents people before you in the delivery pipeline from starting new work/tickets before they have handed their current ticket off.
I’ll use a very crude board below to demonstrate how this would work and the issues it tries to solve.
In this case below there are 4 devs to 2 testers. Which mean that the dev team can do more work than the test team can pick up.
As you can see in this example the test column has 5 items but only capacity to work on 2 items at a time with 3 waiting to be worked on next. Mean while the dev team has already started working on 4 new items. This is a push based system where as work is completed it is pushed onto the next person/team/column.
The test team in this example can just pick up the next item in their list of things to do and devs can keep busy working on the next ticket.
So what is wrong with this? Well one thing is that part completed work is sitting in a queue (the 3 tickets waiting in the test column). If there is a problem with the tickets that the Testers are working on then it would need to go back into dev, but they are busy working on new tickets so what now?
Jump to Option 1 if you think they should stop working on their ticket to deal with the issue.
Jump to Option 2 if you think they should finish their current ticket and then pickup the issue.
Jump to Option 3 to see what applying a WIP limit would do.
Option 1: Dev stop working on current ticket and picks up the issue
If the developers jump onto the problem ticket then they have part completed work now sitting in the queue. There is also a cognitive load issue in that the dev now needs to context switch to understand the problem ticket again and depending on the issue it could take some time to fix. In the mean time what is the tester to do? They could pick up one of the tickets in the queue or wait until the dev comes back with a fix. As we don’t know how long this is going to take the testers decided to pick up a new ticket instead of waiting around.
Move to Everyone is busy again…
Option 2: Dev completes existing ticket first and then goes back to the new issue
If the developer opts to complete the ticket they are currently working on then the ticket with the issue is now waiting in a queue but which does it go into? Back into Ready, Dev or stays in the Test column? In some ways you could think of this ticket with the issue as rework as it’s going backwards through the process not forward as you would expect. Also what is the Tester to do now that they are blocked from working on the ticket? Do they pick up a new ticket or wait?
Move to everyone is busy again…
Everyone is busy again and all is good…
Well not exactly we now have part completed work just waiting around providing value to no one. On top of that if a live issue or something else happens (unplanned work) and the dev or tester was away from work (e.g. training, illness or holiday) then you could end up in a situation where you now have 3 potential tickets partly completed. Who would then pick up this work? One of the other devs or testers who may have no context and potentially spend even more time trying to get familiar with the work.
This is when you would likely to see your lead times starting to increase due to in progress work now waiting in queues. Want to know what lead time is and why is this important to understand? Then let me know in the comments and I’ll follow up.
What if the fix requires some significant changes to the system which could impact the other work that is currently in progress. Would the other tester have to stop as the fix could invalidate their testing? What about the other devs who now need to pull in the changes requiring them to now understand the fix and how it would impact their work. This one issue (yes on the extreme side of thinking) could affect up to 9 tickets, all 4 tickets in Dev and 5 tickets in Test. Also do tickets in Done mean released or ready for release? If it’s ready for release then the this fix could also affect them. This leads to a lot of value stuck in your system of delivering software as apposed to getting that value out and into the hands of your end users.
Is there another way?
The two options above are very much a pushed based system where work is pushed onto the next stage as it is completed whether there is capacity in the next stage to do the task or not.
So how would a WIP limit actually help this situation? Are you not just artificially stopping productive people from doing what they are paid to do? In essence yes that is actually what you are doing. Stop people moving onto more work before the existing work is actually done. So how is this a good thing?
Let see what applying a WIP limit to the above situations would do.
Option 3: Implement a WIP limit which encourages a pull based system of work not push.
In the scenario above new work can only be pulled into Dev or Test if someone is actually free to do the work which means that until there is free capacity downstream they can not pull in new work and must sit with the ticket until it can be handed off. Now all the issues with Option 1 and 2 can not occur as you can’t pick up new work until there is free space.
In this case the bottleneck is still test so there is a likelihood that a developer is likely to be waiting around more often than a tester. So what is the developer to do? They can simply wait for the tester to become free or maybe look to see if the ticket can be tested in such a way that when the tester does become free they only need to have a conversation about what testing the developer has already done to mitigate any issues. The tester can then decide if they wish to do more exploratory testing on the ticket as they no longer need to check the ticket for basic functionality or simply move it straight to Done.
As the tester is in short supply and being one of the best feedback mechanism in your development process (apart from real end users of course) you are only going to be passing them the tickets that is most worthy of their time not any old ticket that could have easily been verified by some automated test at the code or perhaps UI level. This starts to encourage your team to not use your test team as a safety net of checking developers work but more informing on the quality of it. But don’t forget quality means different things to different people.
If the tester finds an issue with a ticket and it needs to go back then there should be capacity within development to pick it back up straight away with no issues around dropping existing tickets. Not only that the tester doesn’t have a queue of tickets waiting for them which adds the pressure of them trying to work faster and possibly miss things. Instead they stay with the ticket and possibly assist the developer with understanding the issue and providing them with real time feedback on their fix.
This could even lead to a scenario where the tester and developer are paired together from the very beginning of the ticket and possibly even removing the In Test Column…
By leaving capacity within the dev team then you are leaving slack in the system. This slack or free engineer could be used to help with technical debt, live issues or even working with the testers to make their lives easier and make the work they do spread even further.
This also has the added benefit of helping the team to “Focus on Finishing”. If you start something then you see it through to the end without it waiting around in queues for people to become free to work on it. It also starts getting that whole team mentality towards getting things done and improving the system as a whole instead of individuals working on their own patch of the system.
WIP limits get a lot of criticism from some people in that they are simply slowing things down by placing limits on productive parts of the engineering team and that they should be left to keep being productive. But what they do show is that there are bottlenecks in any system and by keeping some slack in the team you can start addressing them and start making your development teams more productive by focusing on finishing and improving everyday work.
A word of warning when implementing WIP limits
When first using wip limits in software teams be warned that they can cause a lot of disruption. All of a sudden parts of the team that where able to keep busy are now waiting for other parts of the team to be freed up. If this isn’t managed well then you could end up with a lot of unhappy team members and especially if there wasn’t any perceived problems in the previous way of working. This can be even more problematic if you are now asking people to do work that they hadn’t needed to do before like developers getting more hands on with testing.
Another side affect of wip limits is that they cause people within the team to have to communicate more to find out where work is coming from next or how they can help each other. Previously they would have just picked off whatever was waiting in their next column. This can cause a lot of the uncomfort for some as they don’t know what they need to do next and don’t have the perceived safety of the next/waiting for columns.
One of the biggest issues I’ve seen cause these types of issues are individuals within a teams focused on working on tickets as apposed to working collaboratively towards a goal.
We need to keep in mind that the tickets are just how the overall goal has been broken up into distinct deliverable pieces of testable value. The aim being does the new information we have just learnt head us in the direction we want to go in?
This is one place I think tester can help add some real value to teams. Testers can help the teams understand if these iterations are actually helping to achieve the goal but also how the actions of the team affect the overall system within which they work.
Maybe a better way to frame the goal is a hypothesis that the team is going to test as a goal sounds like you already know what the outcome will be.
What do you think of WIP limits do they aid or hinder team productivity?
Have you transcended WIP limits all together and found other ways to stop queues forming?
Let me know in the comments.
Update 18/6/19: Interesting post on modelling WIP limits to see how different limits affect the throughput of work Why limiting work-in-progress works
