Category: DevOps

Three things of 2021

Every week I spend some time reflecting on what I learned or found interesting and this is a summary of my year. After doing this for nearly 3 years one of the biggest ways it’s helped me with is seeing the thread through my work which reminds me of this quote:

You can’t connect the dots looking forward; you can only connect them looking backwards. So you have to trust that the dots will somehow connect in your future…

Steve Jobs’ 2005 Stanford Commencement Address

Where is that thread leading me? On a strategy that could help with improving team collaboration and heading towards a more generative culture.

Remote workshops

Remote workshops are constrained in ways that I hadn’t appreciated before the lock down. Such as by tools, participants work environments and people just getting tried in ways that just doesn’t happen in real life. I’m going to be following up with a blog post on 13 things I’ve learned from running remote workshops so keep an eye out if you want to know more.

Uncertainty

Your ability to identify and work through uncertainty, I believe, will be a big predictor in how successful you will be in the long run but also how satisfied you will be with life. The more I’ve learned about uncertainty and how it affects our behaviour the more I’ve changed the way I look at uncertain situations and approach them. What I’ve found is my attitude towards uncertainty has changed in a way that has made me much more comfortable to be uncomfortable with it.

How? By identifying what about the situation makes me uncomfortable. For example a situation has multiple directions each one with unknown outcomes. Then looking at how it makes me feel uncomfortable. For example a feeling in my stomach, a tremor in my hands, a tightness in my chest, a dry throat etc

This is known as interoception or the ability to sense your internal bodily state and this Guardian article does a good job of explaining it. Only then proceeding to work through the situation and deciding which direction to go in. To be honest this is much easier said than done but with practice can become habit and almost become a default way to approach unknown situations.

My experiences of this has been that by paying attention to how the situation makes me feel internally (interception) I’m able to make much more rational decisions and feel more in-control of myself even if I don’t have control of the situation.

This I believe is what helped me get over my fear of public speaking. It’s not that I got over the fear of getting up on stage but I was able to show my brain that there was nothing to fear in the first place. Over time (and this is important) my brain learns that fear isn’t the right response and tones down my bodies automatic reaction to the situation. Which in turn make me feel much more able to handle the uncertainty of it.

This I think is what can help people move out of their comfort zones and get them more comfortable with being uncomfortable.

Psychological safety

The idea of psychological safety has been on my radar for a few years. Starting with reading the The Phoenix Project in 2016 , The DevOps Handbook book in 2017. Which led me to State of DevOps Report 2018 and hearing about Google’s Project Aristotle the same year which both mentioned psychological safety for me for the first time. But I didn’t look into it until I read Amy Edmundson’s Fearless Organisation in 2019 via reading Kim Scott’s Radical Candor: How to get what you want by saying what you mean which referenced Amy’s work.

Then all through 2020 and 2021 all I could see was how so many people are holding themselves back in their teams by not saying what’s on their minds due to the uncertainty of what would happen. But I still didn’t act on psychological safety as I believed it was confirmation bias leading me to think that it was the key to getting people to speak up.

It wasn’t until late 2021 and I did an internal talk on Psychological safety: What the heck is it and why should you care? that I began to realise that this wasn’t confirmation bias. That we have a problem with speaking up in teams but we never tried to tackle what’s preventing them from speaking in the first place.

It was only after this talk that I felt much more certain that what Google had discovered back in 2012 that psychological safety is foundational to highly effective teams. Why? As this is what enables people to speak up and share what they do and don’t know. Speaking up is key for effective inter-team collaboration and enabling them to work through problems and head towards continuous improvement.

Which teams will need if we ever want them to be able to autonomously use the 4 key metrics to improve their throughput and stability of their products.

Connecting the dots 

It is now that I feel I can now look back through all the different things I’ve done and learned over the years. And see how it is all connecting together into a strategy that could be helpful in increasing psychological safety at the team level.

I’ve worked at a product level in teams to see how listening and asking questions is key for being able to work through problems. I’ve immersed myself at the process level trying to understand and apply agile and DevOps principles to improve those products. I’ve collaborated with as many different disciplines to try and understand what their problems are at applying those principles to deliver those products.

But as Steve Job said you can’t see how things will connect in the future. I could never have predicted how all the little things I’ve done over the years would line up in the future.

You have to just trust that they will. This is why living with and working directly through uncertainty is going to be the biggest predictor of your success and happiness.

If you can get comfortable being uncomfortable, work through uncertainty and trust that things will workout you might just get what you want… or at least closer to where you want to be.

Interested to see my other past dots then check out my 3 things of 2020 and 2019.

Reducing Uncertainty in Software Delivery

I recently attended a half day online event that InfoQ held on Reducing Uncertainty in Software Delivery. The thing that made this half day event different was the underlying focus on testing but without a single tester present in the talks or panel discussions. The majority of speakers where developers and there was even a few Engineering managers, Product people and a CEO or two. It also appeared to me that none of them have come from a traditional testing background. However they all made points that a good tester would and then some. The advantage they appear to have over testers is that they were able to incorporate their knowledge of their discipline to give a much broader view than just focusing on the testing itself. 

A key theme that I’m seeing from these talks is that they are spending a lot of effort on learning from failure. Either by analysing ones that have happened in production or actively encouraging teams to cause failures. It was only the more advanced organisations that were taking this approach but the others were not far behind. Why? To make their systems even more resilient. Their approach appears to be using Site Reliability Engineers (SRE) to work along side their engineering teams to help them do the work but also enable the teams to extract the learnings from it too. This isn’t simply having chaos testing to cause failures or postmortems for production failure analysis but to also help teams with the people side of working with and handling failure productively.

The talks that caught my interests were Building in reliability (SRE at Gremlin), User Simulation for Rapid Outage Mitigation (SRE at Uber), and a panel discussion on Testing in production (with 2 CEOs, Product person and an Engineering Manager). 

Now this is a small sample, the speakers are very experienced and working or have worked at some of the best known web based organisations (Google, Uber etc) and US focused too. But I’m seeing a lot of things that testers could advocate for being pursued and implemented by Site Reliability Engineers (SRE). For example: 

  • testing in production,
  • building in observability,
  • pushing testing earlier in the process,
  • encouraging developers to test their own work 

The advantage SREs have is they already have the technical ability and are now starting to build out the socio-technological skills that they were lacking previously. These organisations have another advantage in that they are heavily focused on learning from their failures. So when they do get things wrong they work hard to make sure they extract as much value from that failure as possible. On top of that some of these organisations are actively causing failures within their systems to further limit catastrophic failures that could occur.  Some of these organisations have never had tester and from the looks of things never will. If you’re pursuing a true continuous improvement strategy testers could look like a bottleneck in the process slowing down information flow. How can testers enable the flow of information and what can they add that makes this information even more valuable?  

I’ve pulled my summaries of the talks I found interesting below 

Talk: User Simulation for Rapid Outage Mitigation

Uber uses an alternative approach to end-to-end testing due to their system being so big that no one person can ever fully understand it. Instead they use composable tests that each team will create that allows that team to tests their part of the system but mix in other parts pre and post steps built by their dependent teams. These are then run in a simulation environment that allows them to see how the system will perform when that change is deployed. To incentives team to build the tests they use a mixture of pain (woke-up at 3AM due to production failure) and mitigation support team (hold their hands at 3AM) to encourage them to build the tests. For example if you had these test you wouldn’t be awake at 3AM trying to mitigate the issue. They also don’t try and solve the issues at 3AM but mitigate them so others can also learn about outages that affect their system.

Talk: Building in reliability

Interesting talk focusing in on availability of systems within organisations. The speaker walked through how you could go from 99% availability to 99.99% and how it is a learning journey. Used a simple analogy going from crawling, walking and running to get your availability towards what makes sense for your organisation. Essentially can you do it manually, can you script it and can you automate it? I find this slide as a great way to help others understand what the outcomes are at each stage going from 99% to 99.99%.

Panel: Measuring Value Realisation Through Testing in Production

I usually only see these types of conversation from tester focused panels but none of this panel where testers. Tester focused panels typically focus on testers testing in production but this was very much focused on learning from real users in production. Interesting thing from my prospective was they made all the points that I would expect a reasonably experienced tester to bring. In some cases due to their roles being out of testing they focused on the costs and benefits that were outside of simply testing in production e.g. down side of A/B testing or product management mindset shifts that need to happen to embrace learning from users rather then whatever the road map they have decided says.

In some ways testers testing in production almost act like the middlemen of the learning that happens during testing. Could it be that in some cases testers are getting in the way for teams to learn effectively from testing in production?

A DevOps one-liner?

tl;dir You can try but you risk people either misinterpreting or making assumptions so you end up having to explain what you mean anyway.

I was recently asked if I could explain what DevOps was to someone new to the software industry and in attempting to do so I came up with a one-liner

 

A principled approach to ship working software reliably, repeatedly and consistently to enable fast feedback that the system is providing value to the end users.

 

But the more I thought about it the more I realised how loaded this fairly simple one line actually was and needed to break down each part to make sure that I didn’t mislead the person inquiring about what it was.

Principled approach

The idea behind this is to mitigate the cargo cult that has a tendency to happen with agile initiatives.  This is where teams blindly follow the ‘rules’ without ever understanding the why or given the time to understand.  By making your DevOps initiative a principled approach you not only force the team to state what their principles are but in trying to articulate them they have to actually understand them.

By making them principles you can avoid dictating to teams how they should work and what tools they should use. This should be a team decision with justification stemming back to the principles. This then gives the team a mechanism for assessing what they actually need rather than what the sales team of the tool thinks they need. Also because they are principles they can at a future date be easily revised once the team has learnt what is and isn’t working for them.

So what should your principles be? A good place to start would be the DevOps Handbook or go back to Agile Manifesto and adapt them to your context.

Ship working software

This must be to the live environment not staging, development or some sort of testing environment, but where real users actually interact with the system.

It doesn’t have to be every single users but at least a subset. I’d also go to say that beta groups don’t count here as they tend to be self selecting and may not be as representative of your real users.

The shipping part must also be done by the team building the system not some sort of operations team. They must be able to do this themselves otherwise they’ll never feel the pressure of their actions affecting end users. There is nothing like knowing your work has direct consequences on users to make sure you’ve done everything to make sure your changes actually work.

Of course it must functionally work which leads onto

Reliably, repeatedly and consistently

Pushing to the live environment must be reliable meaning that it works every time and doesn’t need specialist knowledge to be able to deploy. Ideally everyone in the team should be able to push from Developers and Testers to BAs and the Product Owner. At a bare minimum everyone of the Developers and Testers can push to live.

It must be repeatable so that you can deploy either once a week to once a minute (that might be a little on the extreme side but being able to allows you to push out fixes a lot quicker). It also helps de-risk releases and turns them into non-events, it’s just something you do like pushing to source control.

For pushing to live to be successful at being reliable and repeatable it must also be consistent. It’s the same process for everyone, there is only one (official) mechanism (preferably one button push) and the outcome is always the same meaning no nasty surprises like taking down the whole system. If it does break then backing out the change is as simple as one button or because you can push to live so easily a fix can be turned around quickly.

Fast feedback

To ship working software reliably, repeatably, and consistently you are going to need some good feedback systems.

Building in testability into your product from the start provides early feedback that functionally the feature works. Continuous integration lets you know that it integrates with the rest of the system early and if you need to course correct. Having your Testers involved from the very beginning of the the feature helps them to inform the whole team how the feature is likely works for the end users.

Coming up with results for which the feature will be measured against at the idea stages means the telemetry needed is built in from the start and helps the team to understand if it is heading in the direction the organisation wants. Not only that it helps the team to see if any subsequent alterations or other features from other teams have any effects on your work (negative or positive).

Providing value to the end users

This is possibly the most important aspect of any system being built. If it’s not meeting some sort of end user need that they find valuable then it doesn’t matter how well you do the other parts of your system, it isn’t going to last long and nor are you as a team or organisation. Ideally almost everything you’re doing is in direct (a new feature) or indirect (CI pipeline to deploy faster) benefit for the users.

If you can justify the effort that either improves the system for the end users or makes it easier for you to deploy faster or understand what is and isn’t working then it’s worth doing.

One of the best ways to start focusing on the end users is to know who they are.

This sounds simple to focus on the end user and if you ask any team if they do I’m certain almost all will say yes. But if you look at the work the teams is outputting it’s sometimes very difficult to see how it benefits the end users or if they were even considered again after the initial discussion on the new feature/update. A lot of teams haven’t even had contact with their end users or seen them actually user their systems.

Focusing on the end users isn’t simple thinking about them every now and again but actually finding out who they are, why they use the systems and how they actually use it. Remember a lot of people say one thing but actually do another so make sure you have multiple ways to understand how your users interact with your systems.

From there you can start seeing how you’re providing value to them even if they can’t articulate it to you. This way you’ll always stay a head of the competition and if any new entrants are likely to start stealing users from you.

Summary

You can come up with a one liner but you risk simplifying something too much that people think it’s easier than it is or they make assumptions which leads to a whole host of other problems. Such as people thinking they are talking about the same thing but both have a different understanding of it e.g. a quick ( and very unscientific) Twitter poll to gauge people’s thinking of DevOps varies quite a bit

It’s one of the reasons why I say we should just stop calling it DevOps to get people to explain what they mean rather than everyone assume they are all talking about the same thing.