Category: Testing

How to move away from the In Test column

We as testers are always looking at ways to improve our testing processes and one way to do that is to move away from the In Test column. You can read more about the why in my post “In Test” column but the how I left pretty vague. I’d like to outline one way in which you could but just like removing the in test column it may seem counterintuitive; by adding even more columns.

How does adding more columns help? Well let me walk you though the process and all will become clear…

Breaking down the In Test column

“What did we actually test?”

One of the first things you’re going to need to do is break down what you do in the “In Test” column. Now this might not be as easy as it sounds but one of the best ways to do this is try to answer the question “What did we test the last time a ticket went through the in test column?” You want to think about all the different activities you carried out for the last few tickets. One way to do this is get some sticky notes and on each one write down the specific type of testing you did e.g. retested defect No. 5487 or pair tested new feature X etc. Then group them up under headings that make sense such as: regression testing, accessibility testing, performance testing, smoke testing, feature testing, automated UI testing , etc. This is best done as a group with people who are familiar with the testing process as they will keep you honest with what you do and don’t do during testing.

These sub-heading are going to become your new columns that come under the banner of Testing.

Now for each sub-heading you want to create entry and exit criteria for when a ticket can be placed into that column. So for instance what entry criteria does a ticket need to begin Accessibility testing e.g. there needs to a UI element to the ticket. What criteria would that ticket need to satisfy so it can leave (exit) the column? E.g. feature run against accessibility guidelines, any issues not fixed have been communicated to accessibility team/product owner etc

So you’ve got all these extra columns now what?

Well, you could just leave it at that and simply making the work more visible the team has a better understanding of what work we as testers actually do. Or you can start to use it as a tool on where to focus your Test improvement process.

Test improvement process

By making all the different testing activities visible you get other benefits too:

  • Makes testing work explicit instead of being hidden under the banner of “Testing”
  • Entry/Exit criteria helps the team understand why that testing needs to happen
    • And when it’s complete
  • Length of time a ticket spends in the columns start to make bottlenecks (constraints) visible

Bottlenecks

By recording how long a ticket spends in each of the columns you can start to see which of the activities is taking the longest.
This will identify the most valuable candidate to start your improvement process. Anything before or after the bottleneck is not going to make as greater impact because the biggest issue isn’t being addressed.

Once you’ve identified your bottleneck you can as a team look at the entry/exit criteria as a starting point for improvements. You could look to see if the risks this type of testing is mitigating against can be address in some other way. For example earlier on in the process or rather than manually in some automated fashion but remember the automation fallacy.

Eventually you will find that the bottleneck “moves” to another part of the testing process which is your next candidate to carry out your improvement process. If you keep going you will start to see that tickets spend less and less time in the identified columns. In some cases you will see that the columns isn’t even need anymore and can be removed all together.

This is known as the 5 focusing steps from the Theory of Constraints. You can find more on Wikipedia and elsewhere but these are:

  1. Identify the system’s constraint(s)
    We do this by measuring how long each tickets spends in the columns and work out which testing task is taking the longest.
  2. Decide how to exploit the system’s constraint(s)
    By looking at the entry and exit criteria and seeing what could be improved as a team.
  3. Subordinate everything else to the above decision(s)
    Once your improvement process has been identified assign it as a task for someone to carry out or if possible as a team work on it together (swarm).
  4. Alleviate the system’s constraint(s)
    Carry out the mitigation methods identified above, update the entry/exit criteria for the column and keep measuring how long the ticket stays in the column going forward.
  5. If in the previous steps a constraint has been broken, go back to step 1, but do not allow inertia to cause a system’s constraint
    If the mitigation method(s) alleviate the bottleneck start the process over BUT don’t stop, keep going till you’ve address all the different types of testing.

Eventually you will be left with some testing that cannot be mitigated but the entry and exit criteria will indicate exactly what testing needs to happen (column heading), why you need to do it (entry criteria) and when its done (exit criteria).

You can then simply make this a part of something that happens for a ticket and you should be left with just the “In Progress” column.

Remember this strategy doesn’t just work for improving testing but all activities that a team carries out. You just need a way to identify the activities and make them visible.

Lets do it
Photo by Karim Dangou

What is Contract Testing?

And Consumer-driven contract testing

This is a follow on from Contract testing: Why do it

First some quick definitions:

Consumer
Is someone (a dev team for instance) that makes use of a third party component or a combination of components (a system). They consume the service provided by the component/system.

Producer
Are the people (a dev team) who build the component or system and make it available to others to use.

Test double
To keep the tests fast you will be using a Test double of the producer in the majority of your tests. More specifically a stub that is very simple and responds how you tell it to.

Remember don’t mock what your don’t own.

Avoid using mocks for contract tests otherwise you’ll be creating another job for yourself if you attempt to mock the behaviour of your producers. Always think of the producer as a blackbox so don’t make assumption on how the internals of the producer work. That is not your responsibility. A stub should be simple and easy to see how it works and will generally just respond with a simple response.

What is a Contract test?

Contract tests are automated code level tests written from the viewpoint of the consumer. They check that the producer exists, responds to a given request and responds in the format expected by the consumer. A simple rule could be

  • For every unique call you make to the producer write a contract test
  • If output from a producer is going to cause a unique behaviour change in you (the consumer) then write a contract test e.g. an error condition would fall into this category

They wouldn’t go further then this and begin to check that the response contains all the correct data or the behaviour of the producer. That’s the job of the producer not the consumer. The producer will be a black box to the consumer, simply input and output. Whatever transformations that happen to the data on the inside of the producer are unknown.

If you do test that a response contains the correct data then I would only test very specific types of data. Specifically ones that if they where not returned would cause problems for your system. For which your system should handle gracefully in response.

How will Contract testing help?

Focused
If the test follows the guidance above they will focus on just the boundary at which the consumer and producer interact. Therefore if they fail you know exactly where the problem is but also what the issue is as they cover only a small area of interaction. This will allow you to quickly identify if the problem is with your integration of the producer or some other part of your code.

Fast
These test will be written at the code level usually with a native unit testing framework of the language you are working with. The vast majority of the tests will also execute against a stub to keep them fast. If you ran them against the actual producer then they could run slower. Also due to each test being so focused on the interaction boundary they will run well under a second allowing the whole suite of tests to execute in a matter of seconds.

Reliable
Circle of control / Circle of influence
Do to the simplicity of the tests the number of false positives is very low and will only fail if something had changed within the test, your interaction with the test double or the test double itself. Everything is now within your circle of control therefore any brittleness can be remedied quickly and easily.

Automated documentation
You now have tests that document your usage of the dependency that are also executable so will stay up-to-date with every change you or your dependency makes.

Running the tests

These test can now be easily kept as part of the main suite of tests within the code base and run through the development pipeline as usual. Any change to the code base would result in the whole suite of contract tests running and letting the dev team know if there was any issues.

Occasionally you would also want to run the contract tests against the real dependency separately from the main build pipeline just to let you know if the contract had changed and that your test double is still a true stand-in for the real thing.

New version of the dependency

Now when a new version of the dependency is released you can run the contract tests against it and check to see if there are any breaking changes. If no issues are detected then maybe some light exploratory testing of changes detailed in the release notes.

If running the contract tests does detect an issue then it should be quick and easy to pinpoint where the issue is (you or them) and what the necessary mitigation steps should be (fix in your code or reject the release). All this while keeping your build pipeline running and your code base shippable.

If an issue is detected in the live environment then it’s going to be easy to know what changed and how to fix it. Which could be either fixing forward or backing out the change.

Confidence for the Consumer team

Contract tests allow the consuming team to move to a new version of a producer much quicker and with greater confidence than before. If something in the release notes looks risky still then your test team can carry out focused exploratory regression testing and if possible putting the update behind a feature flag for a controlled release to your end users.

What is Consumer-driven Contract Testing?

The thing with contract tests is that they are very much in the consumer domain. If the producer is making regular releases which result in the contract tests failing often then in one hand at least you know before taking the actual update but in the other you still can’t take it without work arounds or additional new releases from the producer. This may lead you to thinking about a new supplier. Why even bother with all the pain with writing contract tests when you knew this already?

What if you could help your producer see that each update is going to cause you issues before they even made a release? What if they told you prior to making the release that they need to introduce a breaking change or better yet that the current API will be deprecated after a certain date/version allowing you to move to the new API in your own time? What if you could work with your producers collaboratively that way they get what they want (easy and quick uptake of new versions) and you get what you need (new bug fixes/features, improved confidence of each update working as intended, less time testing)? This is where Consumer-driven Contract testing can help and really starts to show the benefits of Contract testing.

Benefits of Consumer-driven Contract Testing

As mentioned earlier the Contract Test sit in your circle of control. That is everything in this domain is in your direct control. The producer however is out of your control but can be in your circle of influence.

  • Note the level to which you will have influence over your dependency will depend on your overall relationship. If they are within the same organisation then things maybe easier, outside of your org but a supplier that you have a financial contract with then probably require some contract negotiation so not impossible but still some effort. No financial contract and just something you use through an open source license then contract testing is all you will likely have as a relationship.

One of the ways to start moving your producers into your circle of influence is to start a dialogue with them around your contract tests. These tests will show the producer exactly how you integrate their service and the types of response you expect from them. Also due to the simplicity of the tests and test doubles it should be easy for them to understand without your intervention (another good reason to keep them focused and simple).

Showing them the tests is a good place to start (it’s just code that’s what we are all working with none of that touchy, feely stuff about relationships) but a better way to progress the relationship, sorry, chat would be see if they could run the contract tests as a part of their development pipeline. Perhaps every time they plan to make a release or better yet on every commit (another reason to keep the tests fast and reliable).

This way they not only see how you use them, but they get an early warning if any changes in their code is likely to cause their consumers any problems. They can then see if they really need to make that change or see how they can mitigate the impact to their consumers. If they need to do it they can start a dialog with their consumers and start to migrate them onto a new API. This all helps to improve the relationship between consumer and producers, facilitated with some simple tests. Who knew testing could build stronger relationships between development teams?

Who owns the Contract tests?

Just in case it’s not clear the responsibility to write the contract tests in the first place is always with the consumers. It’s only them that know how they plan to use and integrate the producers. The producers can always offer best practice and how they intend consumer to use their services but it’s up to the consumers to decide if they plan to use the service the way it was intended.

Contract tests only become Consumer-driven once they are executed by the producers. Until then they are just Contract tests and even then just in name. If they test anything more then what was outlined earlier they become something else entirely.

New problems to solve

Figuring out how to share the tests, run them, making the results visible and letting consumers and producers know about breaking changes is a whole host of other issues that need to be resolved. The web testing frameworks have already made some progress in this area but I don’t know of any tools that facilitate this between internal teams other then having access to each other’s build infrastructure and source code repos.

Don’t use contract tests to do functional testing

Contract tests need to be quick and simple to understand and therefore only test at the boundary. If you go further than this they will become more complicated and harder for other teams to understand.

It’s not the producers team to understand how you use their service but giving them some insight into how you integrate it could be beneficial to both teams. There is nothing stopping you from writing more integrated tests but don’t expect your producer to run these. This is your responsibility and the feedback from this would be more beneficial to you than them. Besides you don’t want them thinking you’re trying to fob your testing onto them.

If you do more testing further than what was described above don’t call them contract tests otherwise you’ll cause more confusion. Be specific and call them what they are.

Contract Testing, Why do it?

I’ve been thinking a lot about contract testing lately and trying to explain why it’s a good idea.  I thought I’d start by getting my initial arguments for it down and go from there.


Got an opinion then let me know.

Note: This is a first draft (published 24/10/19) and I’ll (hopefully) revisit it again soon but in the meanwhile here is me thinking out in the open… 

Update 18/11/19: Added more details on what contract testing actually is.

The “Contract Testing Chat”: The Reality, The Problem, The Possible Solution?

Aim: To encourage dev teams to use contract testing to manage their integration of dependencies

The Reality 

Within any of the systems we produce there are components from external teams as this allows us to focus on what is important to us and let the other teams take care of thing that are not our core competency. 

In an ideal situation we would probably make everything ourselves so we have complete control but that would require significant amounts of Time, Money and Skills.

Time – To train your existing staff or recruit the people that have the skills and then allow them to actually build the component/systems.

Money – To hire the people and all the necessary resources they need to do the job.

Skills – That the person needs to be able to do the job

Some organisations can throw money at the situation and recruit the best in the industry and do everything in-house. Think of large organisations with deep pockets and large global brands.

Others (like us) don’t have this luxury and have to rely on external teams and component makers to make up for the parts we choose not to focus on. This leaves us with a dilemma. 


The Problem 

Do we just simply trust that these external components (dependencies) will work as we hope and the teams maintaining them will let us know when things change? What most teams do is integrate the component and put it through a couple of rounds of exploratory testing just to be sure things still work as we intended. If an issue is found then it’s a matter of understating what the problem is and where the problem could actually be and who’s responsibility it is to fix it.

This strategy works quite well once the initial issues have been ironed out. 
Eventually though a new version of the component is released and you need to decide if you test everything again or trust the release notes and just do focused regression testing.

Possible solutions


Focused Regression Testing If you just do focused regression testing and an issue is found in the live environment then trust in the dependence maintainer and possibly the development team integrating the component is diminished. The general response to this is to do a full regression of the integrated component every time. 

Full Regression Testing Full Regression Testing usually takes more time, money and skills so teams only integrate newer versions of the dependencies if they really have too. Generally when it contains something that they need e.g. a new feature or bug fix that affects the team directly. 

But because of the large gaps between integration of the previous and latest dependence there are likely to be even more changes then the team anticipated so not only does a full regression now have to happen but there are likely to be more issues found leading to even longer lead times in integrating the component. The blame game normally starts about now, see below. 

Automated end-to-end testing Some teams try to address this problem with automated end-to-end UI testing. Why? Well that’s what the Testers are doing during regression testing right? Just checking the functionality of the system and finding all the issues. So if we can automate this then we can not only find these issues faster but repeatedly and freeing up the Testers to do other things. It almost looks like you address the time, money and skill question is one initial up-front cost of building out the automation see UI Automation, what is it good for? 

Unfortunately these test only find what you program them to find and not only that the more components the end-to-end test run through the greater the chance of failure from false positives. If it does find an issue then you need to work out where the problem actually is: the test or the code. If it’s the code then another developer needs to investigate where that issue is and you’re heading to the blame game backlog issue. 

The Blame game 

The blame game is when the dependency maintainer blames the integrating team for not taking updates often enough and attempting to integrate the component in a way that they didn’t intend. On top of that any issues now found by the integrating team needs to go into the dependence maintainers backlog to be prioritised as they have other competing work to be getting on with. It’s not like they are the only team integrating their component. Meanwhile the integration team is blaming the maintainers for sneaking in features and bug fixes that they never asked for and holding up their development process. 

Last resort solution? 

Find another supplier Once The Blame game starts then this usually leads the integrating team down one of two paths. Find another more responsive supplier. Perhaps paying an external team to the company might solve their dependency problems. This is throwing money at the problem (see money, time and skill from earlier) or they…

Build it in-house This is all about taking back control and making it the teams responsibility to build the component. No more having to worry about another teams backlog or building things that have no relevance to your team. This is the skills part of the money, time and skills from earlier.

Both of the above solutions is a break down of the relationship between maintainers and integrators or more commonly your dev team and those PITA’s over in <insert location/team/department name here> 😆

Is there anything else we can try that could help with all the issues above and prevent the relationship breakdown? We ended up down this path as we needed to address the time, money and skills costs we couldn’t afford as a team, but all the options above results in one of the core costs having to be paid.

A possible solution? 

Contract testing and Consumer-Driven Contract testing 
Contract testing helps address the time cost by allowing an external team maintain a dependency. This also addresses the money question as the responsibility to fund that team essentially becomes someone else problem along with the skills issue. All the dev team needs to do is integrate the dependency. So how is Contract testing going to actually help? 

See What is Contract testing for more details.

The unintended consequences of automated UI tests

Whenever I see people talking about automated testing I always wonder what type of testing they actually mean? Eventually someone will mention the framework they are using and all too often it’s a UI based automation tool that allows tests to be written end-to-end (A-E2E-UI). 
They are usually very good at articulating what they think these tests will give them: fast automated tests that they no longer need to run manually, amongst other reasons.

But what they fail to look at is the types of behaviours these A-E2E-UI tests encourage and discourage within teams. 

They have a tendency to encourage  

  • Writing more integrated testing with the full stack rather then isolated tests 
    • Isolated behaviour tests (e.g. unit, integration, contract tests etc) run faster and help pinpoint where issues could be
    • A-E2E-UI test will just indicate that a specific user journey is not working. While useful from an end user prospective someone still needs to investigate why. This can lead to just re-running it to see if it’s an intermittent error. Which is only made worse by tests giving false negatives which full stack tests are more likely to because of having more moving parts 
  • Testing becomes someone else responsibility 
    • This is more apparent when the A-E2E-UI test are done by somebody else in the team and not the pair developing the code 
    • Notice ‘pair’ if you’re not a one-person development army then why are you working alone? 
      • Pairs tend to produce better code of higher quality with instant feedback from a real person 
      • It might be slower at first but it’s worth it to go faster later 
      • This is really important for established businesses with paying customers 
      • A research paper called The Costs and Benefits of Pair Programming backs this up but it’s nearly 20 years old now so if you know of anything more recent let me know in the comments.
  • Pushing testing towards the end of the development life cycle 
    • The only way A-E2E-UI tests work is through a fully integrated system therefore testing gets pushed later into the development cycle 
    • You could use Test doubles for parts but then that is not an end-to-end test.
  • Slower feedback loops for development teams 
    • Due to testing being pushed to the later stages of development developers go longer without feedback into how their work is progressing 
    • This problem is increased further when the A-E2E-UI tools are not familiar to the developers who subsequently wait for the development pipeline to run their tests instead of doing it locally
  • Duplication of testing 
    • As the A-E2E-UI test suits get bigger and bigger it becomes hard and harder to see what is and isn’t covered by automation 
    • This leads to teams starting to test things at other levels (code and most likely exploratory testing ) which all add to the development time 

These are just some of the behaviours I’ve observed A-E2E-UI tests encourage, but they also discourage other behaviours which maybe desirable. 

They can discourage development teams from

  • Building testability into the design of the systems 
    • Why would you if you know you can “easily” tests something end-to-end with an automation tool? 
  • Maintainability of the code base
    • By limiting the opportunities to build a more testable design you decrease the maintainability of the code though tests 
    • If you need to make a change it’s harder to see what the change in the code affects
    • By having more fine grained tests you can pinpoint where issues exist
    • A-E2E-UI tests just indicate that a journey has broken and how it could affect the end users
    • Not where the problem was actually introduced  
  • Building quality at the source 
    • You are deferring testing towards the end of the development pipeline when everything has been integrated.  Instead of when you are actively developing the code.
    • Are you really going to go back and add in the tests especially if you know an end-to-end test is going to cover it?
  • The responsibility to test your work 
    • With the “safety net” of the A-E2E-UI tests you send the message that it’s ok if something slips though development 
    • If it affects anything the A-E2E-UI tests will catch it
    • What we should be encouraging is that it’s the developers responsibility to build AND test their work
    • They should be confidant that once they have finished that piece of code it can be shipped 
    • The A-E2E-UI tests should acts as another layer to build on your teams confidence that nothing catastrophic will impact the end users. Think of them as a canary in the coal mine. If it stops chirping then something is really wrong…   
  • More granular feedback loops
    • By having A-E2E-UI tests you’re less likely to write unit and integration tests which give you fast feedback on how that part of the code behaves 
    • Remember code level tests should be testing behaviour not implementation details 

If A-E2E-UI tests cause undesirable behaviours in teams should we stop writing them? While they are valuable at demonstrating end users journeys we shouldn’t be putting so much of our confidence that our system works as intended into them. They should be another layer which helps build the teams confidence that the system hangs together. 

If we put the vast majority of our effort and confidence into these automated end-to-end tests than we risk losing one of the teams greatest abilities: building testability into the design of our systems. But just like the automated UI tests building in testability takes conscious effort. This will take time, patients and experience for the whole team to understand and benefit from.

What is quality? 

tl:dr: Lenses of quality is a way to think about what quality means to software development teams.
A while back someone on twitter asked what does quality mean to you. To which I responded:
https://twitter.com/JitGo/status/1010031582395224064

Which not realising at the time but fitted in nicely with Gerald Weinberg’s description of what is quality:
   Quality is value to someone

For most teams that someone could be their Products Owners (PO), the organisation they work for, the team they work with and their end users. All these groups of people could have very different views on what value means to them and even contradicting views in some cases.

For your organisation quality could be whatever helps them reach their targets for that quarter or year.

For your Product Managers (PM) or Owners their measure of a quality product could be a system or feature released on time.

For your team it could be a system that they can build, deploy, maintain and add to easily.

For your end users, well it could be something as simple sounding as it just works.

Also it’s not that the testers or developers don’t care about shipping early (what the PM wants), it’s more that they might care about maintainability or it does what we said it would do more than shipping early.

All of this could be just the tip of the iceberg and there could be many other people and views on what quality means to them.

As testers we need to help development teams understand that quality is measured by people in different ways.

Lenses of Quality

One of the ways I’ve started to help teams understand this is via the idea of lenses of quality.

Each of these groups of people view quality with a different lens therefore see the same system differently to one another. We as testers should help our teams to see quality through these different lenses by helping them identify these groups and what their measures of quality are.

This would help teams to start thinking about who their stakeholders are and how they are likely to perceive the systems that they build.

Would it possible to line up all the different lenses and be able to focus on one common quality metric?

If so would this be more like a microscope pulling into focus the hidden details or more like a telescope and allow you to see far into the distance?

Got an opinion then say so in the comments.

Junaid Valimulla: Are we nearly there yet?

Guest post from Junaid Valimulla currently working as a Senior Test Engineer who is continually learning and honing his craft. I first met Junaid back in my Sony Ericssion days as a Test Analyst and was always impressed with his ability to pick things up quickly and to start helping teams deliver on their goals but most of all his great sense of humour.

I hope you enjoy this post as much as I did as I think his metaphor so aptly describes what a lot of us as testers have felt.

Are we nearly there yet?

Every parent can attest to hearing these words at least once in their life. It is often said by a frustrated child who has has been jailed in the back of the car for what seems like a lifetime to them whilst on a journey to their grandparent’s house.

Now if we look at this from a tester perspective, it become obvious that it is not too dissimilar to the question uttered by many a tester all too often:

“Are we ready to test yet?”

Except in this case the tester is the frustrated child, the developer is the parent driving the car and the grandparent’s house is the ready to test column!

This is the age-old problem, which must change if we are to work in a truly agile environment. Testing has to be seen less as the thing that happens at the end and more of the thing that happens throughout! Testers need to be collaborated with, right from the kick off. The old cumbersome approach encourages the ‘throw over the wall’ mentality, which often leads to a build up of tickets and testing tasks and reaffirms the notion that testing is a bottleneck.

This change can be achieved but requires a number of things to happen. First and foremost their needs to be a mind set change from all parties. This is not just the testers, but developers, designers etc. All those that are involved with the creation of the product need to buy in to the collective responsibility mantra. This however is not something a tester can change and possibly influence alone.

Another thing that can be done and is something I believe the tester can drive, is make changes to the scrum board. Every team I have worked in has had a ‘Ready for test’ or ‘Testing’ column. A column, which ALWAYS comes after the ‘Development’ column. This has to change. This column, this simple yet powerful column undoes all that we are trying achieve. This column maintains the old tired way of thinking.

So, for testing to really be recognised on the board it must be removed off it? Really?? This may sound contradictory but it is the only way that testing will be seen as an on going task performed throughout rather than the bottleneck ‘bit at the end’! No longer will tickets be in ‘Development’ for developers and ‘Testing’ for testers. They will be ‘In Progress’ for all to work together and collaborate. This will encourage conversation between testers and developers, encourage faster feedback and ultimately require less testing at the end! Which is what we all want right?

To conclude what columns would I have then? Simple:

  • Ready for Sprint/Ready to pick up
  • In Progress
  • Done

 

More about Junaid:

Senior Test Engineer (BBC Sport App) with over 13 years experience working in the testing discipline. Testing software for a number of companies ranging from Mobile OS’s and Apps, Websites, TV STB’s, Wireless Routers and everything in-between!

You can find him on Twitter

In test column

Update 4/1/20: I’ve also written about How to move away from the In Test Column

A colleague recently asked me why I think the in test column is a bad idea. So I had a quick search around and couldn’t find anything specifically on the topic I decided to put some of my ideas down on why we shouldn’t have one and instead to opt for a more generic In Progress column.

What’s a board?

Most delivery teams have boards of some sort to make the work they are doing more visible and show progress towards some goal. The two most common ones I see are physical boards with sticky notes and avatars representing individuals in the team

The other is a digital one usually Jira or Trello

The usual configuration of these boards are Backlog, In Dev, In Test, Done with tickets moving from left to right as the work progresses though each column. This helps the team visualise where work is currently up to and making it less abstract.

This all seems sensible as you can monitor the number of tickets in each column and create whatever stats you need to report on but from my experience these types of boards that break down in progress work into Dev and Test roles encourage certain types of anti-patterns.

Inner team silos

Boards like this have a tendency to encourages invisible silos within teams. The backlog column becomes Product Owners (PO)/Business Analyst (BA)/Project Managers (PM) responsibility and overall ownership

The Dev and Test column’s belonging to their respective disciplines and things only getting into done when Test say they are done.

Therefore settings up the disciplines into their silo’s and only doing work that is in their column and handing work off to the next column. Rather than jumping into different columns when they can help or pairing on tasks.

Pushing testing towards the end of the delivery pipeline

With the In Test column coming towards the end of the board and normally the one just before Done it starts pushing testing towards the end of the delivery pipeline rather than continuously throughout development. It also starts putting Test in the position of releasing the software rather than it being a team decision.

It almost sets up Test to be the last line of defence before a release.

More work in progress

Another consequence is that Devs starts picking up new work from the backlog while the Test team are verifying the previous ticket. If Test find an issue the ticket has to either go back into the Backlog or straight back into Dev.

Now the team has to decide do we let the Devs switch context, fix the issue so Test can carry on. Or finish their existing ticket, make Test wait, let them context switch to pick up something new from the Test column?

Context switching for either side isn’t a good thing as there is always the usually spin up time to get familiar with the ticket. Which can lead to people taking short cuts to get the work done quicker.

Testing becomes the bottleneck

The only constraint on the Devs doing more work (tickets) is what can be pulled in from the backlog and the amount they can do. Given the nature of our work (building software solutions) teams always have more Developers then Testers. So teams have a tendency to use the Dev teams to 100% capacity churning out more and more tickets.

As the Devs begin to get through more work, tickets begin to pile up in the Test column waiting for Testers to become free. This naturally leads to Test becoming the perceived bottleneck in the system as this is where the tickets are being held up.

How do you fix the bottleneck?

3 options come to mind and 2 that are usually discussed are

1) Add more testers

This is what generally happens if the teams/company can hire them or borrow test resource from somewhere else but for most this isn’t really sustainable or even an option.

2) Automate more of the Test teams work

This usually leads to more automated UI tests which is surely all that the Testers are doing… right?

See UI Automation, what is it go for? (AKA the Automation fallacy) on why automating at this level isn’t going to give you want you want.

The third which is rarely discussed is

3) Remove the In Test (and Dev) columns

This may sound strange as it looks like all you’re simply doing is hiding the work so let me explain.

You switch to having an In progress area instead. This is pretty much any work the team is currently working on and by this I mean actually doing not something they were doing and are now waiting for some information. This should be almost anything: meetings, training, coding, spike work, testing, absolutely anything any of your teams members could be doing that day that takes longer than eg 1 hour or whatever makes sense to your team to have on the board.

Now instead of just adding a Dev (or Devs if they pair) to a ticket also include a Tester as soon a the ticket gets pulled in. As the Devs are getting up to speed with the ticket the Testers should be too forming a 2-3 way pair/group (I’ll refer to this as pair for the duration of the post). This pair will stay with the ticket all the way to Done. No handoffs, no queuing up work in other columns. It is their responsibility to see the work to Done, preferably to production with Testers providing there and then feedback to the Devs on how the work is proceeding and helping them to bolster any automated testing eg unit or integration testing.

If you find that the Devs are taking longer on a ticket then it’s probably a sign that the ticket is too big and needs to be broken up into smaller, independently deliverable parts.

By having just a In Progress column and pairing Dev & Test from the start will foster a more collaborative approach to getting the work actually done in a team. It also stops more work being pulled in while other tickets are still in progress, reduces the amount of context switching for all and starts addressing the issue of Testing becoming the bottleneck.

Remember Testing isn’t something that happens once the Devs have finished it’s something that should be happening continuously.

Update 4/1/20: Looking for details on how to do this then checkout my post on How to move away from the In Test Column

UI Automation, what is it good for? 

TL;DR: What automation at the UI level does and doesn’t give you.
UPDATE: I originally wrote this back in March 2015, lost it in my drafts and found it again recently so thought I get it out there. Don’t agree then let me know in the comments.

Automation fallacy

Every time I speak with different teams and organisations a theme constantly comes up, UI automation and how it’s going to solve all their problems. The thinking goes that if we can automate more of our tests – read test scripts –  then the Testers no longer have to check that item anymore. This then frees them up to do more interesting things like exploratory testing or that the Tester can be done away with altogether.

There is also a notion that automating all the regression checks will drop the regression test cycle from days to hours. This then supposable allows the team to move faster and release  quicker then before.

What everyone seems to miss is that Automated checks are generally built to check one thing and will tell you if that thing is still there or behaving as the script has been programmed to tell you. If anything else happens that wasn’t programmed into the check then it fails or stops dead, relaying on someone having to then go look and see what went wrong.

A Tester on the other hand can look for workarounds, workout what may have caused the issue or go find other issues based on the information they’ve just learned.

So should we give up on UI automation and face that we’ve got to do rounds and rounds of regression testing and hire more Testers? Well no, what we need to do is ask ourselves

Why are we automating?

It’s looks like a simple question and most people (including myself in the past) would be able to give you a list of answers but what we forget to question is by automating this check what does it tell me when it passes or fails? If it passes does that now mean I no longer have to check that feature or scenario again? If it fails what does that tell me? That I have to check that scenario manually?

When a check fails what do we expect the team to do? Stop everything and investigate the issue? Carry on as normal and hope someone else will check it? Ignore the issue altogether? Who is responsible for checking the issue? Developers, Testers, dedicated automation engineers?

There are a lot of reasons that people give as to why they want to automate their testing such as

  • Reduce test/regression testing
    • The reason for regression testing is to see if the changes you’ve made to your code base haven’t broken anything existing.
    • Unless you have automated all your UI Checks/regression suits then automation is not going to help you as much as you think it will
  • Spot issues/bugs faster
    • Automation doesn’t find new bugs it only tells you that the check you’ve scripted has broken in someway. You need to tell the script that if action A doesn’t produce result B then fail with an error message. What normally happens is the check fails in away your didn’t anticipate.  Don’t forget if you knew before hand how something would break you would probably have put in a fix. Thats why they are called defects something behaving the way you didn’t want/anticipate
  • Free up Testers
    • Potentially but that is if they trust the automation
  • Consistently check a feature the same way
    • This is one thing a automation check is very good at
  • Something that is laborious or difficult to setup and check
    • Another good candidate for automation. We use it to do policy testing of our apps as it’s time consuming and prone to error when trying to manually test
  • We’re doing Behaviour Driven Development (BDD)
    • BDD is not about automating but more about collaborating to understand and create features. The automation is just one small part of it and even then it’s not about testing the UI but the business logic which could be tested at the Unit level
    • If you ever hear a development team saying ’The BDD tests are failing’ then its good indicator that they are probably using BDD incorrectly
  • To release faster
    • Again because you need to do less testing, see Reduce test/regression testing
  • It’s a part of continues integration/delivery so we have to
    • No thought into what you are automating other then it’s what people say you have to do
  • Test manager or some other higher up tells you to
    • Someone thinks that just telling a development team to automate their testing will help them, see above
  • People within the development team or key stockholders don’t trust the developers work
    • The test team are being used as a safety net to check the developers work which tends to have a self-fulfilling prophecy for the developers who start using the test team as that

What does an Automated Check actually do?

Let’s start with an example from an mobile app but could very easily be any platform of your choice:

A simple automated scenario could be when the home page is loaded and I’ve selected an option then I expect to see items X, Y and Z.

Things this scenario will need to do are:

     Start the application
     Wait for it to load up
     Select a menu option
     Wait for the new screen to load
     Then check that the expected items are on screen.

animated gif of example automated check
Click to view animated gif of example automated check

So you can run this check over and over and know that as long as the sequence doesn’t change and the items you are checking for are there then the test will pass. What it isn’t going to tell you though is

  • Formatting issues with any of the screens loading up
  • It is starting to take longer for pages to load in
  • The ordering of the menu options has changed
  • There are new menu options
  • The items you are checking can be seen by the automation framework but nothing is actually visible on screen
  • There are new items on screen that the check is not looking for

All of the above could also be scripted into the check but would likely take quite a bit of effort and you can’t always predict how an app will behave and therefore not be able to script for it.

This is where a real Tester has the advantage. You don’t need to tell a Tester to look for these things they will do this without being prompted and not only that generally a lot faster then an automated check. They can also tell you if it doesn’t feel right or perform in away that would be acceptable for end users which can be very hard to quantify and therefore automate. They can also use the information that they’ve just learned and apply it to what else they can discover. An automated test isn’t going to be able to do this, not with the tools we are using at the moment

Where a Tester can’t match an automated check (or will find very hard to) is checking the same thing in the same way consistently and quickly. As long as they are no physical moving parts an automated check can normally carry out the scenario above in seconds only being delayed by waiting for things to install or load.

So should we stop automating our checks?

Before any team starts to think about automating their testing via the UI they should first, as a team, ask themselves:

Why are we automating?

It sounds like a simple question but as I explained earlier people tend to have differing views on what the automation is actually going to do for them. By talking about why they want to automate they are more likely to come up with solutions that will actually address the problems.

One of the main benefits that I’ve seen from automation especially at the UI is faster feedback that the app:

  • Can actually be installed on a real device/displayed in a browser
  • The app can start without crashing
  • The app can reach any endpoints that it relies on
  • The apps core feature, the one thing it is designed to do actually works for your users e.g.
    • BBC iPlayer: Can video actually be played
    • Google maps: can provide directions to a destination
    • Amazon: allows you to buy products
    • Facebook: The feed shows you what your friends and family are doing

To do this manually, every time a build is made, could take some time but not only that is very tedious and from my experience just doesn’t happen. What tends to happen in this scenario is that developers will wait and see what comes back when Testers finally do test the app. This could be some time from when the change was made and the Testers finally being able to test it.

The longer this feedback loop is the hard it is to fix due to the overhead in understanding what went wrong and what changed to cause the issue. This is exacerbated when working with legacy code especially when not written by the developer making that change.

By automating just the core journey the development team know very quickly that whatever was last committed hasn’t caused a catastrophic failure and that the apps core feature is still functioning. If there is a failure then you can back out the change (or ideally fix it) and get back to a working state. This helps the whole team know that the app works and improves the team overall confidence that if I install this app it’s actually going to be worth their time. There is nothing more frustrating especially in mobile development to get a build, find the device you want to test on and install it only to find it can’t carry out it’s main job for the user or worse crashes on start.

When things fail so easily and obviously it does nothing to instil confidence in the development team more so when your key stakeholders find the issues. This also allows you to start using your Testers for what they a really good at testing and not just checking your developers work.

Core Journey

We use the concept of PUMA  to decide what our core journeys are and ultimately what we should and shouldn’t automate. A generally rule of thumb is if it’s not a core journey then can it be covered by a unit/integration test not invoking the UI. If it still can’t then why would automating it help? Who would do it? How often does it need to run and how quickly do we need feedback that it’s broken? Could we monitor the app stats to check if is still working rather then automating it? If it does break how bad would your users be affected/perception be? Could it be controlled by a feature toggle that allows it to be switched off in the live environment?

So the next time someone asks why don’t you just automate your testing ask them “Why are we automating?” You might realise that the problem they perceive can easily be addressed by one simple automated check rather then 100’s of automated UI checks.

The Do’s and Don’ts of Mobile UI Automation

(…from my experience)

Original posted on Medium

My name is Jitesh ‘Jit’ Gosai and I’m a Senior Developer in Test (DiT) at the BBC working in Mobile Platforms, Digital.

I originally wrote this post well over a year ago (2014!) and recently came across it again so thought I might as well get it out there. Most of the points still stand so should be useful for anyone getting into automation or are already on the road in doing so. Got any other tips then let me know in the comments or Twitter (@JitGo).

During our journey of automating our mobile user interface (UI) testing for iPlayer, we’ve learnt many things (good and bad) that I would like to share with you to help you with your work.
Below, I have compiled some do’s and don’ts to help you and hopefully save you from making the same mistakes as we did.

Android: adb (android device bridge) is your friend

Learn all the little things you can do with adb from simply listing connected devices, to grabbing screenshots. Adb will allow you to do quite a lot with a connected device so make sure you are comfortable with it. My suggestion is to begin with the google developer docs and then start searching around for things you would like to do; for example waiting for a connected device to have started, restarting devices, or sending simple commands to control the UI (don’t try to automate your test this way, you’ll be looking at a lot of pain).

Do learn to control your devices remotely

Learning to control your devices remotely will save you a lot of effort both in time and flow, especially if you have devices connected to your build server which is not physically nearby (in our case, locked in a server room)
Being able to remotely view the device screen and restart the device is very useful and can save you countless journeys when your device ends up in a state you can’t identify. It is also great for being able to return your device to the home screen and therefore into a known state before resuming testing.

Do ditch the android SDK emulator

We tried to use this but found it too slow and unreliable, randomly crashing or disconnecting itself from adb. Intel HAXM was faster but proved to be too flaky and would also crash intermittently. Maybe it was something to do with our setup but we just went with connecting a device to run the jobs and proved to be a lot more reliable. Genymotion is another option which is free for small teams so could work for you.

Do be patient!

When first starting out don’t be tempted to keep playing with your test/build environment for little improvements. Let it settle and run. Know exactly what needs fixing and do just that, not things that are nice to have but essential. Then once you have stability then start to slowly add the things that may help improve speed but with an eye on reliability.

Do be available for pairing

Pair with devs and build tests together. This gets devs familiar with how to write them, encourages them to write tests and stops you being the bottleneck when they break. Also show the UI tests to your Testers to get them familiar with what can and can’t be automated. I’ve found that by walking Testers through an automated UI test vastly improves their knowledge of how they work and things that it can miss.

Do test one thing and one thing only

Don’t be tempted to cover lots of things in one test as it can be harder to tell what has broken when they fail and harder to debug.

Don’t use (or use very sparingly) canned steps

When using step definitions for automating tests it is very tempting to keep reusing steps you’ve already written which at first works well. It can even allow non-technical members of the team to automate test but can result in very verbose scenarios – which are harder to read and more difficult to amend at a later date. We always create methods for interactions with the app e.g. go to home or go to channels in the case for iPlayer. We then use these in our step definitions. This way if the path to get to the to home or channels pages changes then you just update the method and all tests get the change, no need to update all your tests. It also allows you to write tests faster in the long run.

Do use the Page object pattern or similar

This will vastly improve the readability, maintainability and reusability of your tests and teach anyone who will be working with the code how to use it effectively.

Do push your test framework (calabash, appium etc) as far down in the stack as possible

Don’t litter your test code with your frameworks commands but instead delegate that to a module that you access via an interface.
This way if a command changes in a new version you only need to update in one place and if you decide to switch frameworks you can potentially a lot easier.

Don’t fall in love (with your tools!)

They are just that, tools, to help you in your task. If you find that it is causing you more problems and no matter how much searching you do no one can help then dump it and switch to something else (if there is).

Don’t automate everything under the sun

Automate the areas that are going to give the most valuable feedback i.e. that something has broken. We’ve found that UI test are most useful when the dev’s are actively working in those areas so focus your UI tests there.

Do have the need for speed

Keep your tests as fast as possible. Running iOS test through the simulator is faster than devices so we tend to stick with running them there. But for android we use devices as it’s more stable. So you need to weigh up what you need – stability always trumps speed. With that said keep your tests as fast as possible. Don’t use sleeps in your tests (or avoid them at all costs). Use waits that will check repeatedly for something to be on/off screen before proceeding or raising an error.

Do stub it out!

Stub out the data your app uses.We use Charles Proxy with some ruby scripts to automate the launching, closing and loading in of config files. The main reason we used this approach was that devs and tester were familiar with it so the learning curve was easier and was a quick solution that we came up with until a more appropriate solution could be developed/re-purposed such as REST-assured or WireMock.

Do use Metrics

Stats, stats and more stats. Use data to back up your ideas and collect as much as you can
Stats such as test execution time, pass/fail rates, number of tests, no of runs and charts these. We us dashing as it’s quick to setup and offers a very nice visual way to display data.

Dashboards displayed by the app development team. From the Top left app statistics via Graphana and app usage. Bottom left dashing board showing test status, code metrics using custom bubble graphs. Far right screen showing current build status

 

dashboard showing stats on TV stands

Dashboards displayed by the app development team. From the Top left app statistics via Graphana and app usage. Bottom left dashing board showing test status, code metrics using custom bubble graphs. Far right screen showing current build status

I’ve found that the Riskshaw widget to be very useful. Some teams have also been experimenting with AtlasBoard which looks promising.

Do learn your tools

If you are using Ruby (as we are) then learn to use a REPL such as Pry. This will save you countless amount of time when debugging or even creating tests. Watch this video by Conrad Irwin for a great introduction to Pry and REPL driven development.

Do have multiple tests around any given area of functionality

This way if a test fails you know it could be flaky but if all the tests in that area fail you know straight away something is wrong.

Do read these posts!

This excellent post by Gojko Adzic for anyone attemptig to automate testing at the UI level.
If you are on the journey of UI automation or plan to start then I also highly recommend reading this post by Dan North, The Siren Song of Automated UI Testing.To give a balance view on automated UI testing and that it’s no silver bullet to removing manual testing altogether or other (arguably better) forms of automated unit/component testing.

Got any tips on automated testing? Then let us know in the comments and you never know it may make it in the list above.

 

 

 

 

How to keep up to date with the testing industry?

I recently spoke at UKStar 2018 and the team there asked me a few questions on how I keep up to date in Testing and the industry as a whole which I’ve reproduced here.

What is your favourite testing book/blog? Why is this your favourite?

Two books come to mind. The first is Thinking Fast and Slow by Daniel Kahneman. This book is about how we make decisions and the fact that we are not really all that good at doing it, let alone realising what we are basing our choices on.

Secondly is the Freakonomics set of books by Steven Levitt and Stephen Dubner. Freakonomics is a great way to see how you can go about answering questions that seem impossible at first; but by using some more unconventional methods, you can answer almost anything.

What I enjoy about these books is that neither of them are about testing or software development but the questions that they both aim to answer are so relevant to the work we do.

How do you keep up to date with the software testing industry?

Whenever I’m asked this question it reminds me of what Martin Fowler always says to engineers when they ask him how do they become a better developer? His response is always  “understand the business that you work in”. So rather than focusing on testing I take a broader view and look at our industry as a whole.

I tend to follow a number of bloggers whose insight into our industry I have found invaluable.

Ben Thompson’s Stratechery blog has some excellent indepth analysis on the software industry as a whole and how some of the larger organisations (Google, Amazon, Facebook, Apple etc) operate.

Benedict Evans weekly newsletter and Charles Arthur’s daily Overspill site are also great for finding technology focused articles from around the web that you probably would have missed, or never even come across, always with some illuminating commentary.

Twitter is a great source of information too. I tend to build up my feed by following people both inside and outside of the tech industry to get a broad range of ideas and influence.

What is the biggest misconception about testing that you’ve heard?

That automating all your testing will speed up the delivery of your products. A lot of teams still see testing as that bottleneck at the end of development stopping the release. So if you automate all your testing you will be able to ship faster.  

What they almost always fail to see is that their releases are either too big so the test team can’t isolate and feedback on the risky areas or they still see testers as simply running check lists against the product. Hence thinking that they can be replaced with automation. Tester’s always do so much more than this but their work has had a history of being boiled down to “Have all the tests passed”.