I’ve been thinking a lot recently about what the future of the testers role could look like. Especially in teams that not only fully embrace CI/CD or DevOps but actually get some way towards implementing the ideas behind the approaches.
I’ve broken up my thinking into two posts with the first about whats could happen and the follow up on what they could do next. So what could happen to testers?
For a lot of software teams the testers role is to assess the quality of the work being produced by the developers. This is usually done by manually testing the software using techniques such as exploratory testing to find any issues that may impact the end users. Any issues found will be raised with the developer who did the work (as defects or an informal chat) to fix if the team deems necessary.
Some teams found this process to be one of their biggest bottlenecks in releasing software so attempted to automate more of this type of testing using UI tests with varying levels of success. Others meanwhile started to look at the testability of what they where producing and started to build quality in.
In both of those scenarios the testers role looks obsolete. With the first supposedly replacing the work they did through automation and the second by removing the role as issues are mitigated before they have end user impact. In both cases what they alway assume is that a testers role is just that, to test.
If the perceived value of testers is just to test the changes made by the development team then the future of the testers role looks bleak but could it be more than this?
Some testers are starting to repurpose the old QA acronym to mean Quality Awareness. What they are doing is shifting the perceived value of their work from purely a testing activity, which was to assure the quality of the system to that of raising awareness of the quality of the system.
This may, at first, look as if they are still doing the same work under a different name, but on closer inspection it is a vastly different role.
If you ask testers what does QA stand for most are likely to say its Quality Assurance and is typically described as providing confidence that some quality criteria will be fulfilled. Some people in the testing community believe that it’s actually Quality Awareness*. The thinking goes how can tester assure the quality of something if they never built it in the first place? All they can do is make your team aware of the quality. I agree with both explanations and believe that they are different sides of the same quality coin.
Read on to see what this means for testers and how it’s a useful to consider both in teams.
*I’ve also heard Quality Advocate and my favourite Question Askers, both fit this model of Quality Awareness.
…that someone could be their Products Owners (PO), the organisation they work for, the team they work with and their end users and all these groups of people could have very different views on what value means to them and even contradicting in some cases.
https://www.jitgo.uk/lenses-of-quality/
Again from the same post on what value is:
Each of these groups of people view quality with a different lens therefore see the same system differently to one another. We as testers should help our teams to see quality through these different lenses by helping them identify these groups and what their measures of quality are.
https://www.jitgo.uk/lenses-of-quality/
Simply put value will depend on the viewpoint of the person. Identify whats viewpoint that person sees the product/system through then you’re halfway there to working out what’s valuable to that person. If you take this a step further and see what incentives drive that persons viewpoint then you might identify whats valuable to them too. But thats not as easy as it sounds.
For any given team there are multiple members and stakeholders therefore they are likely to be a number of unique and overlapping quality attributes too. Identifying the key ones for each cohort of people is a valuable exercise for any team. It might help explain why some people are never happy no matter what you deliver.
a statement that something will certainly be true or will certainly happen, particularly when there has been doubt about it
One way to think about assurance is that it’s a promise that an outcome will happen so as to give others confidence. In this case it’s quality and as mentioned earlier quality means value to someone. Therefore QA or Quality Assurance means providing confidence in the quality of the product to stakeholders. This it is about certainty that an outcome will happen not a guarantee. You are saying that best efforts will be made and this is an assurances in making that happen.
knowing something; knowing that something exists and is important
interest in and concern about a particular situation or area of interest
This would lead to QA or Quality Awareness to be a person who is interested in quality (value), understands its importance and has knowledge about the quality of a product or system. To take this a step further someone who works in Quality Awareness understands that quality is value to someone, knows who those people are, what viewpoints they hold and possibly what incentives drive those views. They are then able to apply this knowledge to the system and subsequently increases their teams awareness of overall system quality. Essentially Quality Awareness is about building awareness of what quality is in a team, how it is affected and who it matters too.
Quality Awareness sits at the intersection of the team that produces the system, the domain in which the system operates in and users of the system.
Two sides of the same coin
Both are focused on quality but one is about improving the teams understanding of what quality means for their stakeholders. While the other is focused on maintaining (and hopefully improving) the quality of the product.
In this scenario Quality Awareness can improve Quality Assurance by giving it the metrics with which quality is being assessed on. In this model Quality Assurance can actually fulfil it’s job of providing confidence that the quality of the product is being upheld. Why, because it is taking into account who the stakeholders are and what is valuable to them. Then converting that value into a measurable metric which the engineering team can either:
assess themselves against to make sure they are doing what they said they would do or
provide the stakeholders the metrics to improve their confidence that not only is the engineering team doing their job but maintaining and perhaps even improving it.
The thing to keep in mind is not all quality values can be converted into an easy to measure metric. You can use proxy measures to give you an idea but some measures are always inherently subjective. On top of the this the systems we build are interdependent on other systems which are out of our control and can affect the quality of our systems. Therefore techniques such as exploratory testing can be very beneficial as it can help build a fuller awareness of what quality means for your product.
Back to Quality Assurance?
Does this mean we should go back to using QA again and naming ourselves the QA Team? No, we’ve come a long way in some areas of our industry and going back to QA teams might bring back all those old problems. Such as test team silos, testers being the gatekeepers and why didn’t we catch that bug?
How is this helpful?
Where this can be useful is giving teams another lens through which to look at their testing approach and think is this heading towards building assurance of quality or is this about raising awareness of quality. By separating out into these two camps you can see the value it’s actually going to bring and if its worth the investment. It might also help clear up who should be doing what and when.
Using this model of awareness and assurance could be helpful for testers trying to figuring out what it is that they want to do with their careers. Do you want to learn more about building team confidence in quality through test automation (Quality Assurance) or building a quality culture within teams (Quality Awareness)?
Whenever I see people talking about automated testing I always wonder what type of testing they actually mean? Eventually someone will mention the framework they are using and all too often it’s a UI based automation tool that allows tests to be written end-to-end (A-E2E-UI). They are usually very good at articulating what they think these tests will give them: fast automated tests that they no longer need to run manually, amongst other reasons.
But what they fail to look at is the types of behaviours these A-E2E-UI tests encourage and discourage within teams.
They have a tendency to encourage
Writing more integrated testing with the full stack rather then isolated tests
Isolated behaviour tests (e.g. unit, integration, contract tests etc) run faster and help pinpoint where issues could be
A-E2E-UI test will just indicate that a specific user journey is not working. While useful from an end user prospective someone still needs to investigate why. This can lead to just re-running it to see if it’s an intermittent error. Which is only made worse by tests giving false negatives which full stack tests are more likely to because of having more moving parts
Testing becomes someone else responsibility
This is more apparent when the A-E2E-UI test are done by somebody else in the team and not the pair developing the code
Notice ‘pair’ if you’re not a one-person development army then why are you working alone?
Pairs tend to produce better code of higher quality with instant feedback from a real person
It might be slower at first but it’s worth it to go faster later
This is really important for established businesses with paying customers
A research paper called The Costs and Benefits of Pair Programming backs this up but it’s nearly 20 years old now so if you know of anything more recent let me know in the comments.
Pushing testing towards the end of the development life cycle
The only way A-E2E-UI tests work is through a fully integrated system therefore testing gets pushed later into the development cycle
You could use Test doubles for parts but then that is not an end-to-end test.
Slower feedback loops for development teams
Due to testing being pushed to the later stages of development developers go longer without feedback into how their work is progressing
This problem is increased further when the A-E2E-UI tools are not familiar to the developers who subsequently wait for the development pipeline to run their tests instead of doing it locally
Duplication of testing
As the A-E2E-UI test suits get bigger and bigger it becomes hard and harder to see what is and isn’t covered by automation
This leads to teams starting to test things at other levels (code and most likely exploratory testing ) which all add to the development time
These are just some of the behaviours I’ve observed A-E2E-UI tests encourage, but they also discourage other behaviours which maybe desirable.
They can discourage development teams from
Building testability into the design of the systems
Why would you if you know you can “easily” tests something end-to-end with an automation tool?
Maintainability of the code base
By limiting the opportunities to build a more testable design you decrease the maintainability of the code though tests
If you need to make a change it’s harder to see what the change in the code affects
By having more fine grained tests you can pinpoint where issues exist
A-E2E-UI tests just indicate that a journey has broken and how it could affect the end users
Not where the problem was actually introduced
Building quality at the source
You are deferring testing towards the end of the development pipeline when everything has been integrated. Instead of when you are actively developing the code.
Are you really going to go back and add in the tests especially if you know an end-to-end test is going to cover it?
The responsibility to test your work
With the “safety net” of the A-E2E-UI tests you send the message that it’s ok if something slips though development
If it affects anything the A-E2E-UI tests will catch it
What we should be encouraging is that it’s the developers responsibility to build AND test their work
They should be confidant that once they have finished that piece of code it can be shipped
The A-E2E-UI tests should acts as another layer to build on your teams confidence that nothing catastrophic will impact the end users. Think of them as a canary in the coal mine. If it stops chirping then something is really wrong…
More granular feedback loops
By having A-E2E-UI tests you’re less likely to write unit and integration tests which give you fast feedback on how that part of the code behaves
Remember code level tests should be testing behaviour not implementation details
If A-E2E-UI tests cause undesirable behaviours in teams should we stop writing them? While they are valuable at demonstrating end users journeys we shouldn’t be putting so much of our confidence that our system works as intended into them. They should be another layer which helps build the teams confidence that the system hangs together.
If we put the vast majority of our effort and confidence into these automated end-to-end tests than we risk losing one of the teams greatest abilities: building testability into the design of our systems. But just like the automated UI tests building in testability takes conscious effort. This will take time, patients and experience for the whole team to understand and benefit from.
Which not realising at the time but fitted in nicely with Gerald Weinberg’s description of what is quality:
Quality is value to someone
For most teams that someone could be their Products Owners (PO), the organisation they work for, the team they work with and their end users. All these groups of people could have very different views on what value means to them and even contradicting views in some cases.
For your organisation quality could be whatever helps them reach their targets for that quarter or year.
For your Product Managers (PM) or Owners their measure of a quality product could be a system or feature released on time.
For your team it could be a system that they can build, deploy, maintain and add to easily.
For your end users, well it could be something as simple sounding as it just works.
Also it’s not that the testers or developers don’t care about shipping early (what the PM wants), it’s more that they might care about maintainability or it does what we said it would do more than shipping early.
All of this could be just the tip of the iceberg and there could be many other people and views on what quality means to them.
As testers we need to help development teams understand that quality is measured by people in different ways.
Lenses of Quality
One of the ways I’ve started to help teams understand this is via the idea of lenses of quality.
Each of these groups of people view quality with a different lens therefore see the same system differently to one another. We as testers should help our teams to see quality through these different lenses by helping them identify these groups and what their measures of quality are.
This would help teams to start thinking about who their stakeholders are and how they are likely to perceive the systems that they build.
Would it possible to line up all the different lenses and be able to focus on one common quality metric?
If so would this be more like a microscope pulling into focus the hidden details or more like a telescope and allow you to see far into the distance?
