The courage to supercharge your testability

Testability is all about building quality-in. It’s about identifying known issues before they become a problem while coding. Pairing testers into this process can supercharge the testability feedback loop. It can allow you to pick up known and unknown issues.

But pairing devs and testers together needs courage. Courage so that both disciplines can take interpersonal risks and share hard things such as what they don’t know, don’t understand or mistakes they’ve made. This will need both groups to listen, understand and ask questions to help each other through the process. Both groups will need to show curiosity, humility and empathy for one another. You will not only feel uncomfortable during the process but it will take time too. The temptation to go back to inspecting for quality – dev and test handing work off to each other – will be hard to resist.

Pairing for testability is not just pair programming but working together to understand what the behaviour of the code being written should and shouldn’t do.

Devs and testers should work together to leverage the skills that each have, not get hung up about the skills they lack. If your pair is more exploratory focused identify ways that allow you to make the best use of those skills. If they are more technically inclined then focus there.

Remember the key is to build quality-in not inspect for quality. So what can you do now that helps your team move in that direction?

Test Automation: Don’t report the bugs it catches

Reading time: 3 minutes

Don’t report the bugs your test automation catches. Report the reduction in uncertainty that the system works.

When you report the bugs you send the signal that test automation is there to catch bugs. But that’s not what it’s for. Test automation is there to tell you if your system is still behaving as you intended it to.

What are automated tests for?

Each automated test should be some isolated aspect of the behaviour of the system. Collectively these tests tell you that when you make a change to the system it still behaves as you want it to. What automated tests do is reduce your uncertainty that the system still behaves as you expect it to.

Framing test automation as reducing uncertainty

Framing test automation as reducing uncertainty help emphasize that there are always things we don’t know. Whereas if you frame it as increased certainty it can give the impression that we know more than we do.

Framing testing as increasing certainty
Framing testing as reducing uncertainty

What happens when a test passes or fails

When an automated test passes it’s sending a signal that this specific behaviour still exists. Therefore reducing some of your uncertainty that whatever changes you made have not affected this specific behaviour.

When a test fails it signals that this expected behaviour didn’t occur, but that’s it. What it doesn’t tell you is if it is a bug or if it was due to the change to the system. Someone still needs to investigate the failure to tell you that.

So what we should report is to what extent our uncertainty has been reduced by these tests. But how do we do that?

How to frame test automation as reducing uncertainty

Well a good place to start is to help people understand what behaviour is covered by the tests. For instance, you could categorise the behaviour of your system into 3 buckets such as primary, secondary and tertiary.

Primary could be things that are core to your product’s existence. For example for a streaming service, this could be video playback, playback controls and sign up etc. Tests in this bucket must pass before a release can be made.

Secondary could be behaviour that supports the primary behaviours but if they didn’t exist would be annoying at most but still allows the core features to function. For example, searching for new content or advanced playback controls (think variable playback speeds). Tests in this bucket can fail but they should not render the application unusable. Issues discovered here can be fixed with a patch release.

Tertiary behaviours could be experiments, new features that haven’t yet been proven out or other less frequently used features that are not considered core. Tests in this bucket can also fail and don’t have to be fixed with patch releases.

But be careful of accessibility behaviours falling into Secondary and Tertiary buckets. They might not be your biggest users but those features are critical for others to be able to use your systems.

Defining these categories is a team exercise with all the main stakeholders as it is key that they have a joint understanding of what the categories mean and what behaviours can fall into them.

Then when you report that your primary and secondary tests are passing you signal that the core and supporting features are behaving as expected. This reduces the team’s uncertainty that the system behaves as we expect. You can then decide what you want to do next.

Three things of 2021

Every week I spend some time reflecting on what I learned or found interesting and this is a summary of my year. After doing this for nearly 3 years one of the biggest ways it’s helped me with is seeing the thread through my work which reminds me of this quote:

You can’t connect the dots looking forward; you can only connect them looking backwards. So you have to trust that the dots will somehow connect in your future

Steve Jobs’ 2005 Stanford Commencement Address

Where is that thread leading me? On a strategy that could help with improving team collaboration and heading towards a more generative culture.

Remote workshops

Remote workshops are constrained in ways that I hadn’t appreciated before the lock down. Such as by tools, participants work environments and people just getting tried in ways that just doesn’t happen in real life. I’m going to be following up with a blog post on 13 things I’ve learned from running remote workshops so keep an eye out if you want to know more.

Uncertainty

Your ability to identify and work through uncertainty, I believe, will be a big predictor in how successful you will be in the long run but also how satisfied you will be with life. The more I’ve learned about uncertainty and how it affects our behaviour the more I’ve changed the way I look at uncertain situations and approach them. What I’ve found is my attitude towards uncertainty has changed in a way that has made me much more comfortable to be uncomfortable with it.

How? By identifying what about the situation makes me uncomfortable. For example a situation has multiple directions each one with unknown outcomes. Then looking at how it makes me feel uncomfortable. For example a feeling in my stomach, a tremor in my hands, a tightness in my chest, a dry throat etc

This is known as interoception or the ability to sense your internal bodily state and this Guardian article does a good job of explaining it. Only then proceeding to work through the situation and deciding which direction to go in. To be honest this is much easier said than done but with practice can become habit and almost become a default way to approach unknown situations.

My experiences of this has been that by paying attention to how the situation makes me feel internally (interception) I’m able to make much more rational decisions and feel more in-control of myself even if I don’t have control of the situation.

This I believe is what helped me get over my fear of public speaking. It’s not that I got over the fear of getting up on stage but I was able to show my brain that there was nothing to fear in the first place. Over time (and this is important) my brain learns that fear isn’t the right response and tones down my bodies automatic reaction to the situation. Which in turn make me feel much more able to handle the uncertainty of it.

This I think is what can help people move out of their comfort zones and get them more comfortable with being uncomfortable.

Psychological safety

The idea of psychological safety has been on my radar for a few years. Starting with reading the The Phoenix Project in 2016 , The DevOps Handbook book in 2017. Which led me to State of DevOps Report 2018 and hearing about Google’s Project Aristotle the same year which both mentioned psychological safety for me for the first time. But I didn’t look into it until I read Amy Edmundson’s Fearless Organisation in 2019 via reading Kim Scott’s Radical Candor: How to get what you want by saying what you mean which referenced Amy’s work.

Then all through 2020 and 2021 all I could see was how so many people are holding themselves back in their teams by not saying what’s on their minds due to the uncertainty of what would happen. But I still didn’t act on psychological safety as I believed it was confirmation bias leading me to think that it was the key to getting people to speak up.

It wasn’t until late 2021 and I did an internal talk on Psychological safety: What the heck is it and why should you care? that I began to realise that this wasn’t confirmation bias. That we have a problem with speaking up in teams but we never tried to tackle what’s preventing them from speaking in the first place.

It was only after this talk that I felt much more certain that what Google had discovered back in 2012 that psychological safety is foundational to highly effective teams. Why? As this is what enables people to speak up and share what they do and don’t know. Speaking up is key for effective inter-team collaboration and enabling them to work through problems and head towards continuous improvement.

Which teams will need if we ever want them to be able to autonomously use the 4 key metrics to improve their throughput and stability of their products.

Connecting the dots 

It is now that I feel I can now look back through all the different things I’ve done and learned over the years. And see how it is all connecting together into a strategy that could be helpful in increasing psychological safety at the team level.

I’ve worked at a product level in teams to see how listening and asking questions is key for being able to work through problems. I’ve immersed myself at the process level trying to understand and apply agile and DevOps principles to improve those products. I’ve collaborated with as many different disciplines to try and understand what their problems are at applying those principles to deliver those products.

But as Steve Job said you can’t see how things will connect in the future. I could never have predicted how all the little things I’ve done over the years would line up in the future.

You have to just trust that they will. This is why living with and working directly through uncertainty is going to be the biggest predictor of your success and happiness.

If you can get comfortable being uncomfortable, work through uncertainty and trust that things will workout you might just get what you want… or at least closer to where you want to be.

Interested to see my other past dots then check out my 3 things of 2020 and 2019.

Speed Vs Quality: Can you have both?

5 minutes reading time

I was recently part of a panel discussion around the topic “What is quality?” and an interesting question came up. Is it always a choice between speed of delivery and the quality of that delivery? The thinking being that if you focus on one you will sacrifice the other.  Therefore, you have to choose one or find some way to balance the two which typically translates to being mediocre at both.  That’s when it occurred to me that this is actually a version of working harder in teams and what we should be doing in working smarter.

Working Harder for Speed or Quality

Working harder in the context means doing the work in the same way you have always done it but trying to do more of it. This typically leads to people taking one of two approaches. One is by working longer hours and the other is taking shortcuts by skipping steps not providing immediate value, then using the spare time for doing other work. 

At first both these strategies work quite well, you get the speed improvement and quality remains about the same. The problem is that all the effort is put into doing the work and nothing into improving the capability to do the work. If sustained for long periods those initial speed improvements will begin to slow and quality will diminish as the system gets harder to work with. Essentially you degrade your capability to do that work which further slows you down and brings quality down with it. 

This is probably why most people, including myself, have always thought of speed and quality as trade-offs against each other. Every time we’ve tried to go faster, the quality of the system has gone down. On the other hand when we work more systematically and usually slower, however, things remain more stable.

But is there another way? 

Working Smarter for Speed and Quality

In this case working smarter is not focusing on speed or quality but on your capability to do the work. Instead of just trying to go faster or improving the quality of the work, attention is focused on how that work is done. Specifically if any improvement can be made that could lead to improved speed, quality, or both. 

This is where things can be a little counterintuitive. At first, while the team experiments with their capability, their speed is likely to drop as they figure out a new way of working. There is an even higher chance of quality being affected as things can be missed, or unintended bugs are introduced.  But, as long as the team can collaborate effectively and learn from their failures the speed and the quality is very likely to improve in the long run. 

The Trap of Working Harder

The problem with working harder is that it’s addictive once you start. When you work harder you tend to see immediate results so it feels productive. Not only that but others notice it too and are likely to praise you for going the extra mile. That initial speed boost can make it feel like you’re getting more done in less time, especially if you’re taking a shortcut here and there. This can trap you into thinking that working harder is the best way to work. The long-term damage of not maintaining your way of working is that it can end up being the only way to get things done.

The Virtuous Cycle of Working Smarter

Once teams start improving their capability to do the work it tends to free them up as they are working more efficiently. This additional capacity is re-invested into spending more time on improving their capability further which can create a virtuous cycle of improvement.

Let’s call it Continuous Improvement

Personally I’m not a fan of saying you shouldn’t work harder but work smarter instead. Not only does it come across patronising but it leads to more ambiguity and misunderstanding. I do appreciate that we need to have some form of common language and having a defined terminology helps minimise the ambiguity and risk of misunderstandings. 

If we must call “working smarter” something then I’d opt for continuous improvement. While there is still a chance of it being misinterpreted, at least it has two of the keywords that working smarter is trying to achieve. 

Conclusion

Working harder and working smarter are two ways you can approach and think about how you are working in your teams. What we need to recognise is that working harder only allows you to choose between speed of delivery or the quality of that delivery. While working smarter, over time, can enable teams to deliver a quality product at speed. 

A good place to get started is having a shared understanding of what working harder looks like for you and your teams. How does it help and hinder your ability to deliver at speed or a quality release? From that identify areas that you can measure so you can objectively show that things are either getting faster, but quality is dropping or slower but quality is stable/increasing. Some good measures of speed are throughput of delivery such as leads times and release frequency. Quality can be tricker as it depends on what quality means to your team but a good proxy can be stability of releases based on change failure rates and mean time to recovery.

From there you can start experimenting with different ways of working that could incrementally improve your ability to do the work. All the while measuring to see if it is moving you towards delivering a quality product but at a pace that is faster than what came before. Which will enable you to move towards working smarter and begin continuously improving your ways of working

Further reading

I’ve only briefly described these two ways of working in this post. A much more thorough and detailed explanation is given in a 2001 California Business Review article “Nobody ever gets credit for fixing problems that never happen: Creating and Sustaining Process Improvement” by Nelson Repenning and John Sterman for which I must thank Joep Schuurkes who shared it with me on twitter.

Special thanks to Sarah Irving for proofreading and providing numerous suggestions to help make this post better than I could have alone.