How to learn from failure

Reading time 13 minutes

Below are my personal notes from Amy Edmundson excellent article Strategies for learning from failure. It’s a long read but I highly recommend it over my notes as it goes into a lot more detail then I have covered.

Summary

Not all failures are the same and categorisation of failures can make a big difference in enabling learning from them.

Why should testers care?

Considering we deal with software failure all the time we have a tendency to forget the human cost of failures. Especially in terms of how that failure occurred (the team), how that failure affects the users and the outcome for the business. This article is a great introduction in how we can learn from failure first and then how we could enable our teams and business to learn from them by reframing errors as different types of failure.

[Organisations] that catch, correct, and learn from failure before others do will succeed
Amy Edmundson

Amy classifies failure into three types of categories

Preventable
Complex
Intelligent

But we have a tendency to view all failures as one type. In software testing we group them into different levels of risk but generally all failures are error. Which means something isn’t right and should be avoided. We’ve started to try and learn from them but the need for interdisciplinary teams to do so is a cost that is often too high to pay so doesn’t happen very often. I think if we focused our efforts to investigate complex failures we can use the learnings to start minimising preventable issues and stop some of the them happening altogether.

How should we respond to failure?

Some people believe that respond constructively to failures could give rise to an anything-goes attitude. They think that If people aren’t blamed for failures, then how else will they try as hard as possible to do their best work? But this has a tendency to try and avoid failure and in some cases cover it up.

What we actually need is culture that makes it safe to admit and report on failure (so we can learn from them) which coexist with high standards for performance (to make use of that learning to get better).

The blame game

If people see failure as something to be avoid you end up in the blame game. Which has a spectrum of reasons for failure from blameworthy to praiseworthy:

🤔Notice how things that are blameworthy are about individuals but praiseworthy are all about the things.

I wonder how many time people don’t blame others but themselves for the failure and hence keeping quiet or downplaying issues when they occur?

To embrace failure we need to classify it better then the catch all term that failure encourages. Amy Edmundson suggest these three categories: preventable, complex and intelligent failures.

Preventable

These are usually found in routine tasks that are well defined and the outcomes are well understood
Preventable failures tend to occur when we deviate from this routine
In software engineering certain routine task can and should be automated. Such as build processes and specific types of checks
If they do need to be performed manually then tasks lists and check lists are well suited to these types of tasks
- Note: exploratory testing falls under intelligent failures
Failures which result from these types of tasks can usually be mitigate through better understanding of the work we do, how we do it but most importantly why
When we spot these types of failures (deviation from the routine) we should immediately address them
This is in part about stopping errors from being passed down the process and building quality in

Complex failures

Many systems we work in are complex and too big for any one person and in most cases even groups of people to fully understand
This means complex systems can be unpredictable and ambiguous and fail in ways we could not have anticipated
The way in which complex failures occur can in some cases be traced to things all happening in just the right way
But assuming failures will never occur can be counter productive and we should build into the process to handle what happens when things go wrong
When complex failures do occur we should recognise them as such and investigated them in a praiseworthy way to understand all the components that led to the failure and identify if any of the smaller issues that resulted in the failure can be made preventable
- For example
- Most accidents in hospitals result from a series of small failures that went unnoticed and unfortunately lined up in just the wrong way.

Intelligent failures

Named by the Duke University professor of management Sim Sitkin as intelligent failures
These are the failures that occur during experimentation
They help you understand what works and what doesn’t
- And importantly quickly
These are situations where the answers are not knowable in advanced
The only way you can find out is to actually do it
Exploratory testing is all about raising awareness of intelligent failures
As Amy Edmondson calls them they are failures at the frontier
- Situations that haven’t happened before
- Or maybe won’t happen again
For software engineering this is a lot of the work that we are doing
- Hence agile software development so we can adapt to the changing environment
- To do things in a way that helps you learn from your work
- We should be producing lots of intelligent failures that help us learn about the system we’re building , the people that use it and the domain in which it used
- Exploratory testing is all about exploring a system and seeing in what ways it can fail to better understand how it works

Small experiments over Big Bang experiments

At the frontier, the right kind of experimentation produces good failures quickly. Managers who practice it can avoid the unintelligent failure of conducting experiments at a larger scale than necessary.

Trail and failure?

“Trial and error” is a common term for the kind of experimentation needed in these settings, but it is a misnomer, because “error” implies that there was a “right” outcome in the first place.

Tolerance of failure

We need to be able to accept complex and intelligent failures and understand that doing so does not mean mediocrity. Tolerance is actually something that we need in order to be able to learn from these types of failures. The problem with failure is that there is almost always an emotional element to it and so needs leadership to enable the learning that needs to happen.

How do you learn from failure?

Leaders should insist that their organizations develop a clear understanding of what happened—not of “who did it”—when things go wrong.

This requires consistently:

reporting failures, small and large;
systematically analysing them; and
proactively searching for opportunities to experiment.

Anyone working on experimental work needs to clearly know that the faster we fail the faster we will succeed but most people don’t understand this subtle but important concept.

The quicker things fail the quicker you can pivot or try another idea that can succeed
But the longer that failure takes the longer you are executing on an idea that will not help your objective
What is the opportunity cost of working on one thing and not the other?

Some people may approach experimental work as if it’s well defined and understood such as production line style of work where you need to produce the same thing over and over.

For example, statistical processes control, which uses data analysis to assess unwarranted variances, is not good for catching and correcting random invisible glitches such as software bugs.

In a typical software team this would be predefined test cases or automated checks

There are three main ways to learn from failure: detection, analysis, and experimentation.

Detection

We need to detect and make issues visible earlier on in our processes before they become bigger issues later on

Don’t shoot the messenger

Unfortunately a lot of people are reluctant to raise issues early on in the process for all manor of reasons. The biggest culprit being people unwilling to take interpersonal risks in raising issues.

One of the best ways to combat this is for management to lead by example and not only encourage the raising of issues earlier on in the process no matter how small but also applauding the people that do and having a system in place to make something happen about it.

Another issue is a human tendency to not admit failure due to the stigma attached to it “it failed therefore I’ve failed”. Therefore people keep going hoping that things will get better when they should have admitted failure or worse they haven’t realised they’ve failed due to inadequate measures or goal when starting out.

Changing the stigma around failure is one way to improve the situation such as failure parties to encourage the reporting of failures and help people look at the situation in another way.

Example of how other organisations detect errors

Through speaking up supported by management from Amy Edmundson:

In researching errors and other failures in hospitals, I discovered substantial differences across patient-care units in nurses’ willingness to speak up about them. It turned out that the behavior of midlevel managers—how they responded to failures and whether they encouraged open discussion of them, welcomed questions, and displayed humility and curiosity—was the cause. I have seen the same pattern in a wide range of organizations.

Building quality in

The idea of the andon cord from the Toyota production system is doing just this; noticing small deviations in process and correcting them there and then to constantly improve the system.

For software engineering this is all about building quality into the process instead of inspecting it at the end. Inspecting at the end is almost too late to make difference due to the increased cost in time and cognitive load to make the change. This usually ends in discussion such as /users are never going to notice X/, /no one is ever going to do Y/ or /let’s see if it’s going to become a problem first/.

Analysis

Once failures have been detected it is important to not just look at the symptoms of the problem and move on but to dig into the root cause of the issues.

Unfortunately we tend to not want to do this as it can be painful to admit that something went wrong especially if we are the cause of it and can negatively affect our self esteem and confidence. There is also an element of interpersonal risk associated with admitting failure that can add towards people not wanting to spend too long looking at issues too deeply. “What if people think I’m incompetent?”

Culture is another aspect that needs to be in place for inquire into failure to occur. Digging into failures needs:

inquiry and openness, patience, and a tolerance for causal ambiguity

But a lot of organisational cultures are geared towards actions and results not reflection as needed for learning from failure.

We are also highly susceptible to fundamental attributes error. This is where we downplay our responsibility and blame external factors when we fail and do the opposite when others do.

Amy research back in 2010 showed that failure analysis is often limited and ineffective – sadly I think this is still the case for a lot of organisations.

Analysing complex failures is difficult as they tend to occur across teams and departments and due to the reason listed above most people only focus on the symptoms rather then getting at the underlying causes of the failures. Therefore it’s best to use multidisciplinary teams to carry out the investigation with the support of management that you are looking at what happened not what someone did or didn’t do.

From the NASA Colembine disaster

A team of leading physicists, engineers, aviation experts, naval leaders, and even astronauts devoted months to an analysis of the Columbia disaster.
They conclusively established not only the first-order cause: (symptom)
- a piece of foam had hit the shuttle’s leading edge during launch—but also
second-order causes: (underlying reason)
- A rigid hierarchy and schedule-obsessed culture at NASA made it especially difficult for engineers to speak up about anything but the most rock-solid concerns.

Experimentation

A critical activity for effective learning is strategically producing failures—in the right places, at the right times—through systematic experimentation.

For scientists
* 70% of experiments will fail
* They recognise that failure is not optional but a part of the process
* And that Failure holds valuable information that they need to extract and learn from /before the competition/ 🤔

In contrast when product companies design new products they plan for success. So they setup the product for optimal conditions that work instead of representative ones that they can actually learn from. Therefore the pilot only produced information about what does work not what doesn’t.

From Amy Edmundson:

A small and extremely successful suburban pilot had lulled Telco executives into a misguided confidence.
The problem was that the pilot did not resemble real service conditions: It was staffed with unusually personable, expert service reps and took place in a community of educated, tech-savvy customers.
But DSL was a brand-new technology and, unlike traditional telephony, had to interface with customers’ highly variable home computers and technical skills.
This added complexity and unpredictability to the service-delivery challenge in ways that Telco had not fully appreciated before the launch.
A more useful pilot at Telco would have tested the technology with limited support, unsophisticated customers, and old computers.
It would have been designed to discover everything that could go wrong—instead of proving that under the best of conditions everything would go right.
Of course, the managers in charge would have to have understood that they were going to be rewarded not for success but, rather, for producing intelligent failures as quickly as possible.
What incentives are you setting up for your employees? The things you reward are the things you will get.

What makes exceptional organisations?

exceptional organisations are those that go beyond detecting and analysing failures and try to generate intelligent ones for the express purpose of learning and innovating.

Can you think of any organisation that purposely inject failures into their system to see how they behave? Hint they named the tool after monkeys 🐒 and in the process created a whole new discipline: Chaos engineering. These experiments don’t have to be that big either:

[you] don’t have to do dramatic experiments with large budgets. Often a small pilot, a dry run of a new technique, or a simulation will suffice.

recognise the inevitability of failure in today’s complex work organizations. Those that catch, correct, and learn from failure before others do will succeed
Amy Edmundson

Building Quality in via Testability

7 minute read

Back in March 2018 I visited The Design Museum in London and came across the above installation.

What you can see is technology design classics all the way from the first transistor radios on one side to the very first digital clocks on the other. With everything else in-between.

If you stand back far enough you begin to see that they are not just randomly placed on the wall but in a particular order. As each piece of technology progresses in its evolution you begin to notice that it starts acquiring functionality from the technology around it. Not only that but they start to shrink in size at the same time. Eventually you realise that all of that technology has been assumed into one device: The mobile phone which is placed right in the centre of the wall.

With the older technology its size and its complexity was on show for all to see. The mobile phone however is different. It actually looks quite simple on the outside with only a screen and a few buttons. But once you turn it on you begin to realise that this is something quite different to what has come before. It can not only provide all of the functionality from the technology that came before it but much more through the use of the internet. This isn’t just limited to mobile phones but pretty much all technology that comes after. From TV’s, speakers and wrist watches everything is slowly being interconnected via the internet.

The interesting thing about a lot of this new technology is that it is actually been developed and controlled by only a handful of companies. Who on average have more resources than a lot of other more traditional companies combined. On top of that they have oriented themselves around the users unlike any other company before always working to provide them with best experience they can come up with. It’s almost like they know every users is a click a way from moving onto the next thing but something keeps those users coming back. It sometimes look hopeless competing against them, so what do we do?

Software is eating the world

Marc Andreessen back in 2011 wrote that “Software is eating the world” which actually gives us some hope. Software allows us to compete again and perhaps tempt those users away. Remember just as the competition we are only a click a way too. But what is going to get those users to click something new?

We need to be able to try different ideas and get them in-front of our users to start seeing what works and what doesn’t based on real data and not just what people think is working.

Leadership to build Collaboration and Purpose

However to be able to start doing that we need to start working better together as software teams. Simply having the best developers is not going to cut it. Research from Google’s Aristotle project showed that this wasn’t the case but 5 other team dynamics where better predicators of well functioning teams. These being psychological safety, dependability, structure & clarity and meaning & impact.

Side note: Psychological safety is all about leadership and interpersonal risk taking and not just saying this is a safe space. Read The Fearless Organisation to learn more.

Once we can collaborate more effectively we can build psychological safety, dependability and structure into the team. From there we can start working on the teams purpose. What is the teams reason for being, what are they trying to accomplish, how will this help the organisation? Purpose is all about providing the team clarity, meaning and impact. But simply asking people to collaborate and giving them a purpose isn’t going to build the team dynamics set out earlier. It’s going to need leadership to build the type of collaboration we need that has those characteristics. Leaders will need to be more hands on demonstrating interpersonal risk, building dependability between team members and setting up what the initial structure to the team is.

What is quality?

For arguments sake let say you’ve been able to get someway to doing that. Now what? Do the user of your systems just magically start appearing? Team collaboration is only one part, now you need to start iterating on the system. You could just get the team to build whatever they think is a good idea and get them to do it as fast as they can. The risk is releasing half-baked systems that end up causing you or worse, your users more problems then before. The thing is users tend to want a quality product, but quality is subjective and so means different things depending on your view point. From the lenses of quality :

For your Organisation quality could be whatever helps them reach their targets for that quarter or year.

For your Product owner their measure of a quality product could be a system or feature released on time.

For your Team it could be a system that they can build, deploy, maintain easily.

For your Users, well it could be something as simple as it just works. – Lenses of quality

Building Quality in via Testability

If quality means different things to different people how can you build quality into a product? By building in testability instead. What testability does is start to make your system objective. Meaning that instead of people saying the system feels easier to work with or they think it works correctly you use tests to back up those feelings. Those tests have to be built into the system during development. It is not something that can be added on very easily after the fact and especially by people who haven’t built the system in the first place. Testability is not about testing the system end-to-end but piece-by-piece. Each piece being a specific type of behaviour the system provides and tested in isolation from the other pieces. The scope and definition of the behaviours should be decided on by the team collaboratively. Unit testing can help with testing like this but everyone has a different opinion on what a unit is and therefore have very different approaches to testing a unit:

What do the unit test test? — 4\ Everyone seems to have a different opinion on what makes a unit but also what makes a a good and bad unit tests

Which is why I have a problem with calling them unit tests and outlined how you could define them by calling them code tests first and then building a team understanding of what they are.

This type of testing is what I think gets us towards what W Edward Deeming (1900-1993) known in his time as the leading management thinker on quality when he said we should

“Cease dependence on mass inspection. Build quality into the product from the start” – W Edward Deeming

So do we just need to work better together and building in testability to solve all our quality issues?

Software ate the world, so all the worlds problems get expressed in software

It’s been 9 years since Marc Andreessen wrote Software is eating the world. Ben Evans (a business analyst who worked for Andreessen) recently said in his presentation Standing on the shoulders of giants

“Software ate the world, so all the worlds problems get expressed in software” – Ben Evans

You can build in all the quality measures you want but that doesn’t address any of the problems we’ve intentionally encoded into the system. You are going to need someone who understands how the team works (and how the problems are encoded into the system), knows how the system is deployed into the real world (and the domains in which it is used) and who those users are (and what they expect of it). That someone already exists within teams but most teams have simply been using them as a safety net to check their work and to channel my inner Deeming “Carry out mass inspections of our systems”. We’ve called them Testers but maybe it’s time we start to think of them as something else?

Software levels the playing field again and allows us to innovate in ways that no other tool before it has ever allowed. However to do so we need to work collaboratively as teams to build testability into our software systems and testers to raise awareness of what quality is for our products. From this foundation we can begin to compete again and really start offering ours users that temptation to click something new.

How to move away from the In Test column

We as testers are always looking at ways to improve our testing processes and one way to do that is to move away from the In Test column. You can read more about the why in my post “In Test” column but the how I left pretty vague. I’d like to outline one way in which you could but just like removing the in test column it may seem counterintuitive; by adding even more columns.

How does adding more columns help? Well let me walk you though the process and all will become clear…

Breaking down the In Test column

“What did we actually test?”

One of the first things you’re going to need to do is break down what you do in the “In Test” column. Now this might not be as easy as it sounds but one of the best ways to do this is try to answer the question “What did we test the last time a ticket went through the in test column?” You want to think about all the different activities you carried out for the last few tickets. One way to do this is get some sticky notes and on each one write down the specific type of testing you did e.g. retested defect No. 5487 or pair tested new feature X etc. Then group them up under headings that make sense such as: regression testing, accessibility testing, performance testing, smoke testing, feature testing, automated UI testing , etc. This is best done as a group with people who are familiar with the testing process as they will keep you honest with what you do and don’t do during testing.

These sub-heading are going to become your new columns that come under the banner of Testing.

Now for each sub-heading you want to create entry and exit criteria for when a ticket can be placed into that column. So for instance what entry criteria does a ticket need to begin Accessibility testing e.g. there needs to a UI element to the ticket. What criteria would that ticket need to satisfy so it can leave (exit) the column? E.g. feature run against accessibility guidelines, any issues not fixed have been communicated to accessibility team/product owner etc

So you’ve got all these extra columns now what?

Well, you could just leave it at that and simply making the work more visible the team has a better understanding of what work we as testers actually do. Or you can start to use it as a tool on where to focus your Test improvement process.

Test improvement process

By making all the different testing activities visible you get other benefits too:

Makes testing work explicit instead of being hidden under the banner of “Testing”
Entry/Exit criteria helps the team understand why that testing needs to happen
- And when it’s complete
Length of time a ticket spends in the columns start to make bottlenecks (constraints) visible

Bottlenecks

By recording how long a ticket spends in each of the columns you can start to see which of the activities is taking the longest.
This will identify the most valuable candidate to start your improvement process. Anything before or after the bottleneck is not going to make as greater impact because the biggest issue isn’t being addressed.

Once you’ve identified your bottleneck you can as a team look at the entry/exit criteria as a starting point for improvements. You could look to see if the risks this type of testing is mitigating against can be address in some other way. For example earlier on in the process or rather than manually in some automated fashion but remember the automation fallacy.

Eventually you will find that the bottleneck “moves” to another part of the testing process which is your next candidate to carry out your improvement process. If you keep going you will start to see that tickets spend less and less time in the identified columns. In some cases you will see that the columns isn’t even need anymore and can be removed all together.

This is known as the 5 focusing steps from the Theory of Constraints. You can find more on Wikipedia and elsewhere but these are:

Identify the system’s constraint(s)
We do this by measuring how long each tickets spends in the columns and work out which testing task is taking the longest.
Decide how to exploit the system’s constraint(s)
By looking at the entry and exit criteria and seeing what could be improved as a team.
Subordinate everything else to the above decision(s)
Once your improvement process has been identified assign it as a task for someone to carry out or if possible as a team work on it together (swarm).
Alleviate the system’s constraint(s)
Carry out the mitigation methods identified above, update the entry/exit criteria for the column and keep measuring how long the ticket stays in the column going forward.
If in the previous steps a constraint has been broken, go back to step 1, but do not allow inertia to cause a system’s constraint
If the mitigation method(s) alleviate the bottleneck start the process over BUT don’t stop, keep going till you’ve address all the different types of testing.

Eventually you will be left with some testing that cannot be mitigated but the entry and exit criteria will indicate exactly what testing needs to happen (column heading), why you need to do it (entry criteria) and when its done (exit criteria).

You can then simply make this a part of something that happens for a ticket and you should be left with just the “In Progress” column.

Remember this strategy doesn’t just work for improving testing but all activities that a team carries out. You just need a way to identify the activities and make them visible.

WIP limits: choose your own adventure edition

What are they?

A limit on the amount of work that can be pulled into the team and most likely columns on your board that you use to manage your teams work.

Why?

The idea being that instead of work being pushed into the team as a when it comes up the team pulls the work into their work stream if they have capacity to actually work on it. This prevents people before you in the delivery pipeline from starting new work/tickets before they have handed their current ticket off.

I’ll use a very crude board below to demonstrate how this would work and the issues it tries to solve.

In this case below there are 4 devs to 2 testers. Which mean that the dev team can do more work than the test team can pick up.

As you can see in this example the test column has 5 items but only capacity to work on 2 items at a time with 3 waiting to be worked on next. Mean while the dev team has already started working on 4 new items. This is a push based system where as work is completed it is pushed onto the next person/team/column.
The test team in this example can just pick up the next item in their list of things to do and devs can keep busy working on the next ticket.

So what is wrong with this? Well one thing is that part completed work is sitting in a queue (the 3 tickets waiting in the test column). If there is a problem with the tickets that the Testers are working on then it would need to go back into dev, but they are busy working on new tickets so what now?

Jump to Option 1 if you think they should stop working on their ticket to deal with the issue.
Jump to Option 2 if you think they should finish their current ticket and then pickup the issue.
Jump to Option 3 to see what applying a WIP limit would do.

Option 1: Dev stop working on current ticket and picks up the issue

If the developers jump onto the problem ticket then they have part completed work now sitting in the queue. There is also a cognitive load issue in that the dev now needs to context switch to understand the problem ticket again and depending on the issue it could take some time to fix. In the mean time what is the tester to do? They could pick up one of the tickets in the queue or wait until the dev comes back with a fix. As we don’t know how long this is going to take the testers decided to pick up a new ticket instead of waiting around.

Move to Everyone is busy again…

Option 2: Dev completes existing ticket first and then goes back to the new issue

If the developer opts to complete the ticket they are currently working on then the ticket with the issue is now waiting in a queue but which does it go into? Back into Ready, Dev or stays in the Test column? In some ways you could think of this ticket with the issue as rework as it’s going backwards through the process not forward as you would expect. Also what is the Tester to do now that they are blocked from working on the ticket? Do they pick up a new ticket or wait?

Move to everyone is busy again…

Everyone is busy again and all is good…

Well not exactly we now have part completed work just waiting around providing value to no one. On top of that if a live issue or something else happens (unplanned work) and the dev or tester was away from work (e.g. training, illness or holiday) then you could end up in a situation where you now have 3 potential tickets partly completed. Who would then pick up this work? One of the other devs or testers who may have no context and potentially spend even more time trying to get familiar with the work.

This is when you would likely to see your lead times starting to increase due to in progress work now waiting in queues. Want to know what lead time is and why is this important to understand? Then let me know in the comments and I’ll follow up.

What if the fix requires some significant changes to the system which could impact the other work that is currently in progress. Would the other tester have to stop as the fix could invalidate their testing? What about the other devs who now need to pull in the changes requiring them to now understand the fix and how it would impact their work. This one issue (yes on the extreme side of thinking) could affect up to 9 tickets, all 4 tickets in Dev and 5 tickets in Test. Also do tickets in Done mean released or ready for release? If it’s ready for release then the this fix could also affect them. This leads to a lot of value stuck in your system of delivering software as apposed to getting that value out and into the hands of your end users.

Is there another way?

The two options above are very much a pushed based system where work is pushed onto the next stage as it is completed whether there is capacity in the next stage to do the task or not.

So how would a WIP limit actually help this situation? Are you not just artificially stopping productive people from doing what they are paid to do? In essence yes that is actually what you are doing. Stop people moving onto more work before the existing work is actually done. So how is this a good thing?

Let see what applying a WIP limit to the above situations would do.

Option 3: Implement a WIP limit which encourages a pull based system of work not push.

In the scenario above new work can only be pulled into Dev or Test if someone is actually free to do the work which means that until there is free capacity downstream they can not pull in new work and must sit with the ticket until it can be handed off. Now all the issues with Option 1 and 2 can not occur as you can’t pick up new work until there is free space.

In this case the bottleneck is still test so there is a likelihood that a developer is likely to be waiting around more often than a tester. So what is the developer to do? They can simply wait for the tester to become free or maybe look to see if the ticket can be tested in such a way that when the tester does become free they only need to have a conversation about what testing the developer has already done to mitigate any issues. The tester can then decide if they wish to do more exploratory testing on the ticket as they no longer need to check the ticket for basic functionality or simply move it straight to Done.

As the tester is in short supply and being one of the best feedback mechanism in your development process (apart from real end users of course) you are only going to be passing them the tickets that is most worthy of their time not any old ticket that could have easily been verified by some automated test at the code or perhaps UI level. This starts to encourage your team to not use your test team as a safety net of checking developers work but more informing on the quality of it. But don’t forget quality means different things to different people.

If the tester finds an issue with a ticket and it needs to go back then there should be capacity within development to pick it back up straight away with no issues around dropping existing tickets. Not only that the tester doesn’t have a queue of tickets waiting for them which adds the pressure of them trying to work faster and possibly miss things. Instead they stay with the ticket and possibly assist the developer with understanding the issue and providing them with real time feedback on their fix.
This could even lead to a scenario where the tester and developer are paired together from the very beginning of the ticket and possibly even removing the In Test Column…

By leaving capacity within the dev team then you are leaving slack in the system. This slack or free engineer could be used to help with technical debt, live issues or even working with the testers to make their lives easier and make the work they do spread even further.

This also has the added benefit of helping the team to “Focus on Finishing”. If you start something then you see it through to the end without it waiting around in queues for people to become free to work on it. It also starts getting that whole team mentality towards getting things done and improving the system as a whole instead of individuals working on their own patch of the system.

WIP limits get a lot of criticism from some people in that they are simply slowing things down by placing limits on productive parts of the engineering team and that they should be left to keep being productive. But what they do show is that there are bottlenecks in any system and by keeping some slack in the team you can start addressing them and start making your development teams more productive by focusing on finishing and improving everyday work.

A word of warning when implementing WIP limits

When first using wip limits in software teams be warned that they can cause a lot of disruption. All of a sudden parts of the team that where able to keep busy are now waiting for other parts of the team to be freed up. If this isn’t managed well then you could end up with a lot of unhappy team members and especially if there wasn’t any perceived problems in the previous way of working. This can be even more problematic if you are now asking people to do work that they hadn’t needed to do before like developers getting more hands on with testing.

Another side affect of wip limits is that they cause people within the team to have to communicate more to find out where work is coming from next or how they can help each other. Previously they would have just picked off whatever was waiting in their next column. This can cause a lot of the uncomfort for some as they don’t know what they need to do next and don’t have the perceived safety of the next/waiting for columns.

One of the biggest issues I’ve seen cause these types of issues are individuals within a teams focused on working on tickets as apposed to working collaboratively towards a goal.

We need to keep in mind that the tickets are just how the overall goal has been broken up into distinct deliverable pieces of testable value. The aim being does the new information we have just learnt head us in the direction we want to go in?

This is one place I think tester can help add some real value to teams. Testers can help the teams understand if these iterations are actually helping to achieve the goal but also how the actions of the team affect the overall system within which they work.

Maybe a better way to frame the goal is a hypothesis that the team is going to test as a goal sounds like you already know what the outcome will be.

What do you think of WIP limits do they aid or hinder team productivity?

Have you transcended WIP limits all together and found other ways to stop queues forming?

Let me know in the comments.

Update 18/6/19: Interesting post on modelling WIP limits to see how different limits affect the throughput of work Why limiting work-in-progress works

What is quality?

tl:dr: Lenses of quality is a way to think about what quality means to software development teams.

A while back someone on twitter asked what does quality mean to you. To which I responded:

https://twitter.com/JitGo/status/1010031582395224064

Which not realising at the time but fitted in nicely with Gerald Weinberg’s description of what is quality:

Quality is value to someone

For most teams that someone could be their Products Owners (PO), the organisation they work for, the team they work with and their end users. All these groups of people could have very different views on what value means to them and even contradicting views in some cases.

For your organisation quality could be whatever helps them reach their targets for that quarter or year.

For your Product Managers (PM) or Owners their measure of a quality product could be a system or feature released on time.

For your team it could be a system that they can build, deploy, maintain and add to easily.

For your end users, well it could be something as simple sounding as it just works.

Also it’s not that the testers or developers don’t care about shipping early (what the PM wants), it’s more that they might care about maintainability or it does what we said it would do more than shipping early.

All of this could be just the tip of the iceberg and there could be many other people and views on what quality means to them.

As testers we need to help development teams understand that quality is measured by people in different ways.

Lenses of Quality

One of the ways I’ve started to help teams understand this is via the idea of lenses of quality.

Each of these groups of people view quality with a different lens therefore see the same system differently to one another. We as testers should help our teams to see quality through these different lenses by helping them identify these groups and what their measures of quality are.

This would help teams to start thinking about who their stakeholders are and how they are likely to perceive the systems that they build.

Would it possible to line up all the different lenses and be able to focus on one common quality metric?

If so would this be more like a microscope pulling into focus the hidden details or more like a telescope and allow you to see far into the distance?

Got an opinion then say so in the comments.

Junaid Valimulla: Are we nearly there yet?

Guest post from Junaid Valimulla currently working as a Senior Test Engineer who is continually learning and honing his craft. I first met Junaid back in my Sony Ericssion days as a Test Analyst and was always impressed with his ability to pick things up quickly and to start helping teams deliver on their goals but most of all his great sense of humour.

I hope you enjoy this post as much as I did as I think his metaphor so aptly describes what a lot of us as testers have felt.

Are we nearly there yet?

Every parent can attest to hearing these words at least once in their life. It is often said by a frustrated child who has has been jailed in the back of the car for what seems like a lifetime to them whilst on a journey to their grandparent’s house.

Now if we look at this from a tester perspective, it become obvious that it is not too dissimilar to the question uttered by many a tester all too often:

“Are we ready to test yet?”

Except in this case the tester is the frustrated child, the developer is the parent driving the car and the grandparent’s house is the ready to test column!

This is the age-old problem, which must change if we are to work in a truly agile environment. Testing has to be seen less as the thing that happens at the end and more of the thing that happens throughout! Testers need to be collaborated with, right from the kick off. The old cumbersome approach encourages the ‘throw over the wall’ mentality, which often leads to a build up of tickets and testing tasks and reaffirms the notion that testing is a bottleneck.

This change can be achieved but requires a number of things to happen. First and foremost their needs to be a mind set change from all parties. This is not just the testers, but developers, designers etc. All those that are involved with the creation of the product need to buy in to the collective responsibility mantra. This however is not something a tester can change and possibly influence alone.

Another thing that can be done and is something I believe the tester can drive, is make changes to the scrum board. Every team I have worked in has had a ‘Ready for test’ or ‘Testing’ column. A column, which ALWAYS comes after the ‘Development’ column. This has to change. This column, this simple yet powerful column undoes all that we are trying achieve. This column maintains the old tired way of thinking.

So, for testing to really be recognised on the board it must be removed off it? Really?? This may sound contradictory but it is the only way that testing will be seen as an on going task performed throughout rather than the bottleneck ‘bit at the end’! No longer will tickets be in ‘Development’ for developers and ‘Testing’ for testers. They will be ‘In Progress’ for all to work together and collaborate. This will encourage conversation between testers and developers, encourage faster feedback and ultimately require less testing at the end! Which is what we all want right?

To conclude what columns would I have then? Simple:

Ready for Sprint/Ready to pick up
In Progress
Done

More about Junaid:

Senior Test Engineer (BBC Sport App) with over 13 years experience working in the testing discipline. Testing software for a number of companies ranging from Mobile OS’s and Apps, Websites, TV STB’s, Wireless Routers and everything in-between!

You can find him on Twitter

A DevOps one-liner?

tl;dir You can try but you risk people either misinterpreting or making assumptions so you end up having to explain what you mean anyway.

I was recently asked if I could explain what DevOps was to someone new to the software industry and in attempting to do so I came up with a one-liner

A principled approach to ship working software reliably, repeatedly and consistently to enable fast feedback that the system is providing value to the end users.

But the more I thought about it the more I realised how loaded this fairly simple one line actually was and needed to break down each part to make sure that I didn’t mislead the person inquiring about what it was.

Principled approach

The idea behind this is to mitigate the cargo cult that has a tendency to happen with agile initiatives. This is where teams blindly follow the ‘rules’ without ever understanding the why or given the time to understand. By making your DevOps initiative a principled approach you not only force the team to state what their principles are but in trying to articulate them they have to actually understand them.

By making them principles you can avoid dictating to teams how they should work and what tools they should use. This should be a team decision with justification stemming back to the principles. This then gives the team a mechanism for assessing what they actually need rather than what the sales team of the tool thinks they need. Also because they are principles they can at a future date be easily revised once the team has learnt what is and isn’t working for them.

So what should your principles be? A good place to start would be the DevOps Handbook or go back to Agile Manifesto and adapt them to your context.

Ship working software

This must be to the live environment not staging, development or some sort of testing environment, but where real users actually interact with the system.

It doesn’t have to be every single users but at least a subset. I’d also go to say that beta groups don’t count here as they tend to be self selecting and may not be as representative of your real users.

The shipping part must also be done by the team building the system not some sort of operations team. They must be able to do this themselves otherwise they’ll never feel the pressure of their actions affecting end users. There is nothing like knowing your work has direct consequences on users to make sure you’ve done everything to make sure your changes actually work.

Of course it must functionally work which leads onto

Reliably, repeatedly and consistently

Pushing to the live environment must be reliable meaning that it works every time and doesn’t need specialist knowledge to be able to deploy. Ideally everyone in the team should be able to push from Developers and Testers to BAs and the Product Owner. At a bare minimum everyone of the Developers and Testers can push to live.

It must be repeatable so that you can deploy either once a week to once a minute (that might be a little on the extreme side but being able to allows you to push out fixes a lot quicker). It also helps de-risk releases and turns them into non-events, it’s just something you do like pushing to source control.

For pushing to live to be successful at being reliable and repeatable it must also be consistent. It’s the same process for everyone, there is only one (official) mechanism (preferably one button push) and the outcome is always the same meaning no nasty surprises like taking down the whole system. If it does break then backing out the change is as simple as one button or because you can push to live so easily a fix can be turned around quickly.

Fast feedback

To ship working software reliably, repeatably, and consistently you are going to need some good feedback systems.

Building in testability into your product from the start provides early feedback that functionally the feature works. Continuous integration lets you know that it integrates with the rest of the system early and if you need to course correct. Having your Testers involved from the very beginning of the the feature helps them to inform the whole team how the feature is likely works for the end users.

Coming up with results for which the feature will be measured against at the idea stages means the telemetry needed is built in from the start and helps the team to understand if it is heading in the direction the organisation wants. Not only that it helps the team to see if any subsequent alterations or other features from other teams have any effects on your work (negative or positive).

Providing value to the end users

This is possibly the most important aspect of any system being built. If it’s not meeting some sort of end user need that they find valuable then it doesn’t matter how well you do the other parts of your system, it isn’t going to last long and nor are you as a team or organisation. Ideally almost everything you’re doing is in direct (a new feature) or indirect (CI pipeline to deploy faster) benefit for the users.

If you can justify the effort that either improves the system for the end users or makes it easier for you to deploy faster or understand what is and isn’t working then it’s worth doing.

One of the best ways to start focusing on the end users is to know who they are.

This sounds simple to focus on the end user and if you ask any team if they do I’m certain almost all will say yes. But if you look at the work the teams is outputting it’s sometimes very difficult to see how it benefits the end users or if they were even considered again after the initial discussion on the new feature/update. A lot of teams haven’t even had contact with their end users or seen them actually user their systems.

Focusing on the end users isn’t simple thinking about them every now and again but actually finding out who they are, why they use the systems and how they actually use it. Remember a lot of people say one thing but actually do another so make sure you have multiple ways to understand how your users interact with your systems.

From there you can start seeing how you’re providing value to them even if they can’t articulate it to you. This way you’ll always stay a head of the competition and if any new entrants are likely to start stealing users from you.

Summary

You can come up with a one liner but you risk simplifying something too much that people think it’s easier than it is or they make assumptions which leads to a whole host of other problems. Such as people thinking they are talking about the same thing but both have a different understanding of it e.g. a quick ( and very unscientific) Twitter poll to gauge people’s thinking of DevOps varies quite a bit

So what's your thinking of #DevOps? Is it just...
— Jit Gosai (@JitGo) June 11, 2017

It’s one of the reasons why I say we should just stop calling it DevOps to get people to explain what they mean rather than everyone assume they are all talking about the same thing.

In test column

Update 4/1/20: I’ve also written about How to move away from the In Test Column

A colleague recently asked me why I think the in test column is a bad idea. So I had a quick search around and couldn’t find anything specifically on the topic I decided to put some of my ideas down on why we shouldn’t have one and instead to opt for a more generic In Progress column.

What’s a board?

Most delivery teams have boards of some sort to make the work they are doing more visible and show progress towards some goal. The two most common ones I see are physical boards with sticky notes and avatars representing individuals in the team

The other is a digital one usually Jira or Trello

The usual configuration of these boards are Backlog, In Dev, In Test, Done with tickets moving from left to right as the work progresses though each column. This helps the team visualise where work is currently up to and making it less abstract.

This all seems sensible as you can monitor the number of tickets in each column and create whatever stats you need to report on but from my experience these types of boards that break down in progress work into Dev and Test roles encourage certain types of anti-patterns.

Inner team silos

Boards like this have a tendency to encourages invisible silos within teams. The backlog column becomes Product Owners (PO)/Business Analyst (BA)/Project Managers (PM) responsibility and overall ownership

The Dev and Test column’s belonging to their respective disciplines and things only getting into done when Test say they are done.

Therefore settings up the disciplines into their silo’s and only doing work that is in their column and handing work off to the next column. Rather than jumping into different columns when they can help or pairing on tasks.

Pushing testing towards the end of the delivery pipeline

With the In Test column coming towards the end of the board and normally the one just before Done it starts pushing testing towards the end of the delivery pipeline rather than continuously throughout development. It also starts putting Test in the position of releasing the software rather than it being a team decision.

It almost sets up Test to be the last line of defence before a release.

More work in progress

Another consequence is that Devs starts picking up new work from the backlog while the Test team are verifying the previous ticket. If Test find an issue the ticket has to either go back into the Backlog or straight back into Dev.

Now the team has to decide do we let the Devs switch context, fix the issue so Test can carry on. Or finish their existing ticket, make Test wait, let them context switch to pick up something new from the Test column?

Context switching for either side isn’t a good thing as there is always the usually spin up time to get familiar with the ticket. Which can lead to people taking short cuts to get the work done quicker.

Testing becomes the bottleneck

The only constraint on the Devs doing more work (tickets) is what can be pulled in from the backlog and the amount they can do. Given the nature of our work (building software solutions) teams always have more Developers then Testers. So teams have a tendency to use the Dev teams to 100% capacity churning out more and more tickets.

As the Devs begin to get through more work, tickets begin to pile up in the Test column waiting for Testers to become free. This naturally leads to Test becoming the perceived bottleneck in the system as this is where the tickets are being held up.

How do you fix the bottleneck?

3 options come to mind and 2 that are usually discussed are

1) Add more testers

This is what generally happens if the teams/company can hire them or borrow test resource from somewhere else but for most this isn’t really sustainable or even an option.

2) Automate more of the Test teams work

This usually leads to more automated UI tests which is surely all that the Testers are doing… right?

See UI Automation, what is it go for? (AKA the Automation fallacy) on why automating at this level isn’t going to give you want you want.

The third which is rarely discussed is

3) Remove the In Test (and Dev) columns

This may sound strange as it looks like all you’re simply doing is hiding the work so let me explain.

You switch to having an In progress area instead. This is pretty much any work the team is currently working on and by this I mean actually doing not something they were doing and are now waiting for some information. This should be almost anything: meetings, training, coding, spike work, testing, absolutely anything any of your teams members could be doing that day that takes longer than eg 1 hour or whatever makes sense to your team to have on the board.

Now instead of just adding a Dev (or Devs if they pair) to a ticket also include a Tester as soon a the ticket gets pulled in. As the Devs are getting up to speed with the ticket the Testers should be too forming a 2-3 way pair/group (I’ll refer to this as pair for the duration of the post). This pair will stay with the ticket all the way to Done. No handoffs, no queuing up work in other columns. It is their responsibility to see the work to Done, preferably to production with Testers providing there and then feedback to the Devs on how the work is proceeding and helping them to bolster any automated testing eg unit or integration testing.

If you find that the Devs are taking longer on a ticket then it’s probably a sign that the ticket is too big and needs to be broken up into smaller, independently deliverable parts.

By having just a In Progress column and pairing Dev & Test from the start will foster a more collaborative approach to getting the work actually done in a team. It also stops more work being pulled in while other tickets are still in progress, reduces the amount of context switching for all and starts addressing the issue of Testing becoming the bottleneck.

Remember Testing isn’t something that happens once the Devs have finished it’s something that should be happening continuously.

Update 4/1/20: Looking for details on how to do this then checkout my post on How to move away from the In Test Column

How to break the rules?

Dan North gave a really interesting talk at last years GoTo Conference called How to break the rules.

He takes Eliyahu Goldratt 4 steps of how technology is adopted which he presented in a series of lectures he gave on why people didn’t take on his ideas of theory of constraints (TOC) from his book The Goal.

Eliyahu stated that organisation need to go through 4 steps before a new technology can be successfully adopted:

What is the power of the technology
- What does it do for you?
What limitation does the technology diminish?
- What will it make better
What rules enable us to manage that limitation presently?
- And how much are we wedded to those rules
What new rules will we need?
- For the technology to succeed

Dan than takes these 4 rules and applies them to real companies that either succeeded or failed to take advantage of new technologies. The interesting part is for the companies that failed what rules (part 3) they needed to break to make use of the new technology.

I’ve made my rough notes on the talk available below but I highly recommend watching it.

My notes from the talk:

Talks about The Goal (book) and Beyond the Goal – lectures by Goldratt

Series of lectures about 20 years after The Goal was released
He tries to attempt to explain why people didn’t apply his Theory of constraints successfully if at all.

First two lectures are: How to adopt Technology?

What is technology: The application of knowledge

For it to be adopted then answer these questions:

What is the power of the technology
- What does it do for you?
What limitation does the tech diminish?
- What will it make better
What rules enable us to manage that limitation ?
- How much are we wedded to those rules
What new rules will we need?
- For the technology to succeed

He then goes on to apply these questions to real companies.

MRP – first application of computers in a business situation
- Calculated cost of materials for manufacture DuPoint
- Which allowed them to ship faster then others
- But when the competition tried it they couldn’t compete as to take advantage of the system you need to change the rest of your business
Goes on to apply to ERP and
Cloud computing (orgs moving to it)
But also Continuose delivery

Interesting: Dell becomes as big as it did because it could work in smaller batches then the competition . All the others companies offered you a fixed machine. Dell changed this by going online only and allowing you to customise your PC and could ship faster than the competition.

Swedish saying: When talking to farmer use farmers words

Commenting on how we collaborate across divisions
If you want to collaborate successfully you need to either be speaking the same language or use words the other understands

A lot of organisations are failing with Agile/Kanban/Scrum etc because they still have all the exiting rules in place (see point 3 earlier).

To be able to adopt the new technology you need to move to the new rules (point 4) otherwise you’re still doing the same thing.

Used the example of Amazon not doing Cost accounting and moving to throughput accounting and flow of value

Cost accounting is what each department cost
Through put accounting is what is being produced by the department and what value does that provide

The problem with step 3 is that it eventually becomes the culture of the organisation and trying to unpick it then is really hard.

So How to break the rules?

Understand the power of technology
Recognise the limitation the technology will diminish
identify the existing rules we use to manage the limitation
Identify and implement the new rules

Summary

Another great talk from Dan and goes to show a lot of our problems in software development have been issues in other industries.

Not only that they have been solved but we tend to overlook them as they don’t look like our industry. What has manufacturing got to do with writing software? In all don’t take my word for it go watch the talk and start having a look at Eliyahu Goldratt body work. He was onto something…