How to move away from the In Test column

We as testers are always looking at ways to improve our testing processes and one way to do that is to move away from the In Test column. You can read more about the why in my post “In Test” column but the how I left pretty vague. I’d like to outline one way in which you could but just like removing the in test column it may seem counterintuitive; by adding even more columns.

How does adding more columns help? Well let me walk you though the process and all will become clear…

Breaking down the In Test column

“What did we actually test?”

One of the first things you’re going to need to do is break down what you do in the “In Test” column. Now this might not be as easy as it sounds but one of the best ways to do this is try to answer the question “What did we test the last time a ticket went through the in test column?” You want to think about all the different activities you carried out for the last few tickets. One way to do this is get some sticky notes and on each one write down the specific type of testing you did e.g. retested defect No. 5487 or pair tested new feature X etc. Then group them up under headings that make sense such as: regression testing, accessibility testing, performance testing, smoke testing, feature testing, automated UI testing , etc. This is best done as a group with people who are familiar with the testing process as they will keep you honest with what you do and don’t do during testing.

These sub-heading are going to become your new columns that come under the banner of Testing.

Now for each sub-heading you want to create entry and exit criteria for when a ticket can be placed into that column. So for instance what entry criteria does a ticket need to begin Accessibility testing e.g. there needs to a UI element to the ticket. What criteria would that ticket need to satisfy so it can leave (exit) the column? E.g. feature run against accessibility guidelines, any issues not fixed have been communicated to accessibility team/product owner etc

So you’ve got all these extra columns now what?

Well, you could just leave it at that and simply making the work more visible the team has a better understanding of what work we as testers actually do. Or you can start to use it as a tool on where to focus your Test improvement process.

Test improvement process

By making all the different testing activities visible you get other benefits too:

Makes testing work explicit instead of being hidden under the banner of “Testing”
Entry/Exit criteria helps the team understand why that testing needs to happen
- And when it’s complete
Length of time a ticket spends in the columns start to make bottlenecks (constraints) visible

Bottlenecks

By recording how long a ticket spends in each of the columns you can start to see which of the activities is taking the longest.
This will identify the most valuable candidate to start your improvement process. Anything before or after the bottleneck is not going to make as greater impact because the biggest issue isn’t being addressed.

Once you’ve identified your bottleneck you can as a team look at the entry/exit criteria as a starting point for improvements. You could look to see if the risks this type of testing is mitigating against can be address in some other way. For example earlier on in the process or rather than manually in some automated fashion but remember the automation fallacy.

Eventually you will find that the bottleneck “moves” to another part of the testing process which is your next candidate to carry out your improvement process. If you keep going you will start to see that tickets spend less and less time in the identified columns. In some cases you will see that the columns isn’t even need anymore and can be removed all together.

This is known as the 5 focusing steps from the Theory of Constraints. You can find more on Wikipedia and elsewhere but these are:

Identify the system’s constraint(s)
We do this by measuring how long each tickets spends in the columns and work out which testing task is taking the longest.
Decide how to exploit the system’s constraint(s)
By looking at the entry and exit criteria and seeing what could be improved as a team.
Subordinate everything else to the above decision(s)
Once your improvement process has been identified assign it as a task for someone to carry out or if possible as a team work on it together (swarm).
Alleviate the system’s constraint(s)
Carry out the mitigation methods identified above, update the entry/exit criteria for the column and keep measuring how long the ticket stays in the column going forward.
If in the previous steps a constraint has been broken, go back to step 1, but do not allow inertia to cause a system’s constraint
If the mitigation method(s) alleviate the bottleneck start the process over BUT don’t stop, keep going till you’ve address all the different types of testing.

Eventually you will be left with some testing that cannot be mitigated but the entry and exit criteria will indicate exactly what testing needs to happen (column heading), why you need to do it (entry criteria) and when its done (exit criteria).

You can then simply make this a part of something that happens for a ticket and you should be left with just the “In Progress” column.

Remember this strategy doesn’t just work for improving testing but all activities that a team carries out. You just need a way to identify the activities and make them visible.

Junaid Valimulla: Are we nearly there yet?

Guest post from Junaid Valimulla currently working as a Senior Test Engineer who is continually learning and honing his craft. I first met Junaid back in my Sony Ericssion days as a Test Analyst and was always impressed with his ability to pick things up quickly and to start helping teams deliver on their goals but most of all his great sense of humour.

I hope you enjoy this post as much as I did as I think his metaphor so aptly describes what a lot of us as testers have felt.

Are we nearly there yet?

Every parent can attest to hearing these words at least once in their life. It is often said by a frustrated child who has has been jailed in the back of the car for what seems like a lifetime to them whilst on a journey to their grandparent’s house.

Now if we look at this from a tester perspective, it become obvious that it is not too dissimilar to the question uttered by many a tester all too often:

“Are we ready to test yet?”

Except in this case the tester is the frustrated child, the developer is the parent driving the car and the grandparent’s house is the ready to test column!

This is the age-old problem, which must change if we are to work in a truly agile environment. Testing has to be seen less as the thing that happens at the end and more of the thing that happens throughout! Testers need to be collaborated with, right from the kick off. The old cumbersome approach encourages the ‘throw over the wall’ mentality, which often leads to a build up of tickets and testing tasks and reaffirms the notion that testing is a bottleneck.

This change can be achieved but requires a number of things to happen. First and foremost their needs to be a mind set change from all parties. This is not just the testers, but developers, designers etc. All those that are involved with the creation of the product need to buy in to the collective responsibility mantra. This however is not something a tester can change and possibly influence alone.

Another thing that can be done and is something I believe the tester can drive, is make changes to the scrum board. Every team I have worked in has had a ‘Ready for test’ or ‘Testing’ column. A column, which ALWAYS comes after the ‘Development’ column. This has to change. This column, this simple yet powerful column undoes all that we are trying achieve. This column maintains the old tired way of thinking.

So, for testing to really be recognised on the board it must be removed off it? Really?? This may sound contradictory but it is the only way that testing will be seen as an on going task performed throughout rather than the bottleneck ‘bit at the end’! No longer will tickets be in ‘Development’ for developers and ‘Testing’ for testers. They will be ‘In Progress’ for all to work together and collaborate. This will encourage conversation between testers and developers, encourage faster feedback and ultimately require less testing at the end! Which is what we all want right?

To conclude what columns would I have then? Simple:

Ready for Sprint/Ready to pick up
In Progress
Done

More about Junaid:

Senior Test Engineer (BBC Sport App) with over 13 years experience working in the testing discipline. Testing software for a number of companies ranging from Mobile OS’s and Apps, Websites, TV STB’s, Wireless Routers and everything in-between!

You can find him on Twitter

The Do’s and Don’ts of Mobile UI Automation

(…from my experience)

Original posted on Medium

My name is Jitesh ‘Jit’ Gosai and I’m a Senior Developer in Test (DiT) at the BBC working in Mobile Platforms, Digital.

I originally wrote this post well over a year ago (2014!) and recently came across it again so thought I might as well get it out there. Most of the points still stand so should be useful for anyone getting into automation or are already on the road in doing so. Got any other tips then let me know in the comments or Twitter (@JitGo).

During our journey of automating our mobile user interface (UI) testing for iPlayer, we’ve learnt many things (good and bad) that I would like to share with you to help you with your work.
Below, I have compiled some do’s and don’ts to help you and hopefully save you from making the same mistakes as we did.

Android: adb (android device bridge) is your friend

Learn all the little things you can do with adb from simply listing connected devices, to grabbing screenshots. Adb will allow you to do quite a lot with a connected device so make sure you are comfortable with it. My suggestion is to begin with the google developer docs and then start searching around for things you would like to do; for example waiting for a connected device to have started, restarting devices, or sending simple commands to control the UI (don’t try to automate your test this way, you’ll be looking at a lot of pain).

Do learn to control your devices remotely

Learning to control your devices remotely will save you a lot of effort both in time and flow, especially if you have devices connected to your build server which is not physically nearby (in our case, locked in a server room)
Being able to remotely view the device screen and restart the device is very useful and can save you countless journeys when your device ends up in a state you can’t identify. It is also great for being able to return your device to the home screen and therefore into a known state before resuming testing.

Do ditch the android SDK emulator

We tried to use this but found it too slow and unreliable, randomly crashing or disconnecting itself from adb. Intel HAXM was faster but proved to be too flaky and would also crash intermittently. Maybe it was something to do with our setup but we just went with connecting a device to run the jobs and proved to be a lot more reliable. Genymotion is another option which is free for small teams so could work for you.

Do be patient!

When first starting out don’t be tempted to keep playing with your test/build environment for little improvements. Let it settle and run. Know exactly what needs fixing and do just that, not things that are nice to have but essential. Then once you have stability then start to slowly add the things that may help improve speed but with an eye on reliability.

Do be available for pairing

Pair with devs and build tests together. This gets devs familiar with how to write them, encourages them to write tests and stops you being the bottleneck when they break. Also show the UI tests to your Testers to get them familiar with what can and can’t be automated. I’ve found that by walking Testers through an automated UI test vastly improves their knowledge of how they work and things that it can miss.

Do test one thing and one thing only

Don’t be tempted to cover lots of things in one test as it can be harder to tell what has broken when they fail and harder to debug.

Don’t use (or use very sparingly) canned steps

When using step definitions for automating tests it is very tempting to keep reusing steps you’ve already written which at first works well. It can even allow non-technical members of the team to automate test but can result in very verbose scenarios – which are harder to read and more difficult to amend at a later date. We always create methods for interactions with the app e.g. go to home or go to channels in the case for iPlayer. We then use these in our step definitions. This way if the path to get to the to home or channels pages changes then you just update the method and all tests get the change, no need to update all your tests. It also allows you to write tests faster in the long run.

Do use the Page object pattern or similar

This will vastly improve the readability, maintainability and reusability of your tests and teach anyone who will be working with the code how to use it effectively.

Do push your test framework (calabash, appium etc) as far down in the stack as possible

Don’t litter your test code with your frameworks commands but instead delegate that to a module that you access via an interface.
This way if a command changes in a new version you only need to update in one place and if you decide to switch frameworks you can potentially a lot easier.

Don’t fall in love (with your tools!)

They are just that, tools, to help you in your task. If you find that it is causing you more problems and no matter how much searching you do no one can help then dump it and switch to something else (if there is).

Don’t automate everything under the sun

Automate the areas that are going to give the most valuable feedback i.e. that something has broken. We’ve found that UI test are most useful when the dev’s are actively working in those areas so focus your UI tests there.

Do have the need for speed

Keep your tests as fast as possible. Running iOS test through the simulator is faster than devices so we tend to stick with running them there. But for android we use devices as it’s more stable. So you need to weigh up what you need – stability always trumps speed. With that said keep your tests as fast as possible. Don’t use sleeps in your tests (or avoid them at all costs). Use waits that will check repeatedly for something to be on/off screen before proceeding or raising an error.

Do stub it out!

Stub out the data your app uses.We use Charles Proxy with some ruby scripts to automate the launching, closing and loading in of config files. The main reason we used this approach was that devs and tester were familiar with it so the learning curve was easier and was a quick solution that we came up with until a more appropriate solution could be developed/re-purposed such as REST-assured or WireMock.

Do use Metrics

Stats, stats and more stats. Use data to back up your ideas and collect as much as you can
Stats such as test execution time, pass/fail rates, number of tests, no of runs and charts these. We us dashing as it’s quick to setup and offers a very nice visual way to display data.

Dashboards displayed by the app development team. From the Top left app statistics via Graphana and app usage. Bottom left dashing board showing test status, code metrics using custom bubble graphs. Far right screen showing current build status

I’ve found that the Riskshaw widget to be very useful. Some teams have also been experimenting with AtlasBoard which looks promising.

Do learn your tools

If you are using Ruby (as we are) then learn to use a REPL such as Pry. This will save you countless amount of time when debugging or even creating tests. Watch this video by Conrad Irwin for a great introduction to Pry and REPL driven development.

Do have multiple tests around any given area of functionality

This way if a test fails you know it could be flaky but if all the tests in that area fail you know straight away something is wrong.

Do read these posts!

This excellent post by Gojko Adzic for anyone attemptig to automate testing at the UI level.
If you are on the journey of UI automation or plan to start then I also highly recommend reading this post by Dan North, The Siren Song of Automated UI Testing.To give a balance view on automated UI testing and that it’s no silver bullet to removing manual testing altogether or other (arguably better) forms of automated unit/component testing.

Got any tips on automated testing? Then let us know in the comments and you never know it may make it in the list above.

How to keep up to date with the testing industry?

I recently spoke at UKStar 2018 and the team there asked me a few questions on how I keep up to date in Testing and the industry as a whole which I’ve reproduced here.

What is your favourite testing book/blog? Why is this your favourite?

Two books come to mind. The first is Thinking Fast and Slow by Daniel Kahneman. This book is about how we make decisions and the fact that we are not really all that good at doing it, let alone realising what we are basing our choices on.

Secondly is the Freakonomics set of books by Steven Levitt and Stephen Dubner. Freakonomics is a great way to see how you can go about answering questions that seem impossible at first; but by using some more unconventional methods, you can answer almost anything.

What I enjoy about these books is that neither of them are about testing or software development but the questions that they both aim to answer are so relevant to the work we do.

How do you keep up to date with the software testing industry?

Whenever I’m asked this question it reminds me of what Martin Fowler always says to engineers when they ask him how do they become a better developer? His response is always “understand the business that you work in”. So rather than focusing on testing I take a broader view and look at our industry as a whole.

I tend to follow a number of bloggers whose insight into our industry I have found invaluable.

Ben Thompson’s Stratechery blog has some excellent indepth analysis on the software industry as a whole and how some of the larger organisations (Google, Amazon, Facebook, Apple etc) operate.

Benedict Evans weekly newsletter and Charles Arthur’s daily Overspill site are also great for finding technology focused articles from around the web that you probably would have missed, or never even come across, always with some illuminating commentary.

Twitter is a great source of information too. I tend to build up my feed by following people both inside and outside of the tech industry to get a broad range of ideas and influence.

What is the biggest misconception about testing that you’ve heard?

That automating all your testing will speed up the delivery of your products. A lot of teams still see testing as that bottleneck at the end of development stopping the release. So if you automate all your testing you will be able to ship faster.

What they almost always fail to see is that their releases are either too big so the test team can’t isolate and feedback on the risky areas or they still see testers as simply running check lists against the product. Hence thinking that they can be replaced with automation. Tester’s always do so much more than this but their work has had a history of being boiled down to “Have all the tests passed”.