(…from my experience)

Original posted on Medium

My name is Jitesh ‘Jit’ Gosai and I’m a Senior Developer in Test (DiT) at the BBC working in Mobile Platforms, Digital.

I originally wrote this post well over a year ago (2014!) and recently came across it again so thought I might as well get it out there. Most of the points still stand so should be useful for anyone getting into automation or are already on the road in doing so. Got any other tips then let me know in the comments or Twitter (@JitGo).

During our journey of automating our mobile user interface (UI) testing for iPlayer, we’ve learnt many things (good and bad) that I would like to share with you to help you with your work.
Below, I have compiled some do’s and don’ts to help you and hopefully save you from making the same mistakes as we did.

Android: adb (android device bridge) is your friend

Learn all the little things you can do with adb from simply listing connected devices, to grabbing screenshots. Adb will allow you to do quite a lot with a connected device so make sure you are comfortable with it. My suggestion is to begin with the google developer docs and then start searching around for things you would like to do; for example waiting for a connected device to have started, restarting devices, or sending simple commands to control the UI (don’t try to automate your test this way, you’ll be looking at a lot of pain).

Do learn to control your devices remotely

Learning to control your devices remotely will save you a lot of effort both in time and flow, especially if you have devices connected to your build server which is not physically nearby (in our case, locked in a server room)
Being able to remotely view the device screen and restart the device is very useful and can save you countless journeys when your device ends up in a state you can’t identify. It is also great for being able to return your device to the home screen and therefore into a known state before resuming testing.

Do ditch the android SDK emulator

We tried to use this but found it too slow and unreliable, randomly crashing or disconnecting itself from adb. Intel HAXM was faster but proved to be too flaky and would also crash intermittently. Maybe it was something to do with our setup but we just went with connecting a device to run the jobs and proved to be a lot more reliable. Genymotion is another option which is free for small teams so could work for you.

Do be patient!

When first starting out don’t be tempted to keep playing with your test/build environment for little improvements. Let it settle and run. Know exactly what needs fixing and do just that, not things that are nice to have but essential. Then once you have stability then start to slowly add the things that may help improve speed but with an eye on reliability.

Do be available for pairing

Pair with devs and build tests together. This gets devs familiar with how to write them, encourages them to write tests and stops you being the bottleneck when they break. Also show the UI tests to your Testers to get them familiar with what can and can’t be automated. I’ve found that by walking Testers through an automated UI test vastly improves their knowledge of how they work and things that it can miss.

Do test one thing and one thing only

Don’t be tempted to cover lots of things in one test as it can be harder to tell what has broken when they fail and harder to debug.

Don’t use (or use very sparingly) canned steps

When using step definitions for automating tests it is very tempting to keep reusing steps you’ve already written which at first works well. It can even allow non-technical members of the team to automate test but can result in very verbose scenarios – which are harder to read and more difficult to amend at a later date. We always create methods for interactions with the app e.g. go to home or go to channels in the case for iPlayer. We then use these in our step definitions. This way if the path to get to the to home or channels pages changes then you just update the method and all tests get the change, no need to update all your tests. It also allows you to write tests faster in the long run.

Do use the Page object pattern or similar

This will vastly improve the readability, maintainability and reusability of your tests and teach anyone who will be working with the code how to use it effectively.

Do push your test framework (calabash, appium etc) as far down in the stack as possible

Don’t litter your test code with your frameworks commands but instead delegate that to a module that you access via an interface.
This way if a command changes in a new version you only need to update in one place and if you decide to switch frameworks you can potentially a lot easier.

Don’t fall in love (with your tools!)

They are just that, tools, to help you in your task. If you find that it is causing you more problems and no matter how much searching you do no one can help then dump it and switch to something else (if there is).

Don’t automate everything under the sun

Automate the areas that are going to give the most valuable feedback i.e. that something has broken. We’ve found that UI test are most useful when the dev’s are actively working in those areas so focus your UI tests there.

Do have the need for speed

Keep your tests as fast as possible. Running iOS test through the simulator is faster than devices so we tend to stick with running them there. But for android we use devices as it’s more stable. So you need to weigh up what you need – stability always trumps speed. With that said keep your tests as fast as possible. Don’t use sleeps in your tests (or avoid them at all costs). Use waits that will check repeatedly for something to be on/off screen before proceeding or raising an error.

Do stub it out!

Stub out the data your app uses.We use Charles Proxy with some ruby scripts to automate the launching, closing and loading in of config files. The main reason we used this approach was that devs and tester were familiar with it so the learning curve was easier and was a quick solution that we came up with until a more appropriate solution could be developed/re-purposed such as REST-assured or WireMock.

Do use Metrics

Stats, stats and more stats. Use data to back up your ideas and collect as much as you can
Stats such as test execution time, pass/fail rates, number of tests, no of runs and charts these. We us dashing as it’s quick to setup and offers a very nice visual way to display data.

Dashboards displayed by the app development team. From the Top left app statistics via Graphana and app usage. Bottom left dashing board showing test status, code metrics using custom bubble graphs. Far right screen showing current build status

Dashboards displayed by the app development team. From the Top left app statistics via Graphana and app usage. Bottom left dashing board showing test status, code metrics using custom bubble graphs. Far right screen showing current build status

I’ve found that the Riskshaw widget to be very useful. Some teams have also been experimenting with AtlasBoard which looks promising.

Do learn your tools

If you are using Ruby (as we are) then learn to use a REPL such as Pry. This will save you countless amount of time when debugging or even creating tests. Watch this video by Conrad Irwin for a great introduction to Pry and REPL driven development.

Do have multiple tests around any given area of functionality

This way if a test fails you know it could be flaky but if all the tests in that area fail you know straight away something is wrong.

Do read these posts!

This excellent post by Gojko Adzic for anyone attemptig to automate testing at the UI level.
If you are on the journey of UI automation or plan to start then I also highly recommend reading this post by Dan North, The Siren Song of Automated UI Testing.To give a balance view on automated UI testing and that it’s no silver bullet to removing manual testing altogether or other (arguably better) forms of automated unit/component testing.

Got any tips on automated testing? Then let us know in the comments and you never know it may make it in the list above.