Category: Testing

Originally posted on the BBC website 30 June 2014

This is the second part of a three post series exploring how the BBC iPlayer Mobile testing team has integrated automated user interface (UI) testing into their development practice.

This post will deal with automation.

By creating collaborative feature files through the “3 Amigos” sessions and setting up a robust system for creating and disseminating them, the natural next step is to begin automating them to increase productivity and quality.

To make the tests as easy as possible to write we implemented the page object pattern so that the developers were clear about how to write more maintainable and less flaky tests. This also meant that test were written more consistently and allowed for more code reuse.

In addition to the page object pattern, we created helper modules that contained all the commands that they would need to drive the app, so it was easier for developers to quickly look up what commands are available, and demonstrated how to use the inbuilt debug tools to query the app to find the screen elements.

Although we explored many different options, we decided to use Calabash and Ruby as the predominant tools to automate our tests as they worked cleanly with Cucumber (which is our test runner) and because Calabash had support for both iOS and Android. To help everyone get to grips with the new systems, internal workshops are held to step developers through real life examples, aiding them to organise the feature folders, creating page objects and types of Calabash commands available to drive the app. By providing step by step guidance, everyone is able to get a strong understanding of the process and where they come into it.

Initially creating the automated UI tests is a slow process as you are required to create a fair amount of support code (including page objects, working out how to access elements on screen and working around timing issues with the app) but once these foundational aspects are set in place, automating tests gets faster and faster.

If a developer ever gets into difficulty Developers in Test are available to pair up to help iron out any problems.

There are many advantages to developers writing the automation tests. Ownership creates a sense of responsibility and a smoother process for delivering and testing the products. It also drives the developers to look at the results and take advantage of the benefits of faster feedback.

With developers using the feature files to write their tests, it ensures that the product is as intended, rather than based on an assumption, which speeds up the development process. The benefits of this is that everyone takes mutual responsibility for automation and prevents testing being pushed to manual when a DiT is absent or unavailable, which keeps the process moving more succinctly and effectively.

Another benefit of using Calabash is that it uses the accessibility labels to access on screen elements. If the developers build the tests they have to enable the labels therefore helping to make the app more accessible. For more information on accessibility practices see Senior Accessibility Specialist Henny Swan’s blog posts. 

You may be wondering what the DiT’s are doing if the developers are creating all the automation code?

DiT’s remain embedded in the team and available for pairing to help automate tests that are not straightforward. They help build up tools to aid automation e.g. worker methods to carry out complex interactions or how a feature could be automated if not immediately obvious. They help keep Continuous Integration (CI) jobs running and investigate brittle tests. DiT’s also tend to be the experts with the automation frameworks so advise if a feature is worth automating or it’s better to test it manually.

Once the feature file has been automated the tests are pushed into the main build pipeline. They will be run approximately 4 times a day and a subset on each check in of code. We have our build jobs status displayed on large screens (one of the advantages of working near the TV platforms team is that they have a lot of reference TV’s that we can use when not being tested on) so if anything fails the whole team know straightway.

In the final post of this series I’ll tell you we handle legacy and new features and what the future holds for our team.

Originally posted on the BBC website 06 August 2014

Originally posted on the BBC website 30 June 2014

In this three part series of blog posts I will be exploring how the BBC iPlayer Mobile team has integrated automated user interface (UI) integration testing into their development practice.

I’m a Senior Developer in Test (DiT) working in Mobile Platforms, BBC Future Media. I work with the BBC iPlayer Mobile team to help them automate their testing, investigate new tools and advising how best to use them in their everyday work, sharing this with other teams across the BBC. In the 16 months that I have been with the BBC I have seen a great deal of change in development practice, which I will be sharing in this series of posts.

I was initially brought onto the team to identify how to automate a greater number of tests in order to increase the speed of release without risking the quality of the end product.

When I first joined the team it was apparent that the developers had all individually started to automate some of the tests, however it became clear that there was no continuity to the test scripts, with each developer using their own styles. Inevitably, when a script broke, if it wasn’t investigated by the developer who wrote the test, it would take a long time to identify the problem and to repair the issue. Because of this, it would usually result in adding a simple patch to keep it running or by disabling it. Because of the issues around automation, the team began to lose confidence in the testing method and reverted back to manual.

The lack of systems within the process was problematic in itself with some features having a lot of automation testing carried out and others receiving little or none, with no-one taking responsibility for ensuring that the testing was happening. This meant that each test was insular with only the designated developer having access to the results.

From the outset, it was decided to take things slowly and begin with the area that would give the most value with the least amount of effort. The team understood that feature files are a great way to describe how the systems should work and that a collaborative approach was needed for successful implementation. It was here that we decided to use the idea of the ‘3 Amigos’ to write the features.

3 Amigos

To set up the ‘3 Amigos’ we needed to recruit a developer from each platform (iOS and Android), a tester, a product owner/business analyst and a DiT. Now this is obviously more than three “amigos” however we needed to have a representative from each area of the process and the DiT to lead the sessions until everyone felt comfortable with the process and able to run them independently.

The advantage of having a DiT, or anyone experienced in writing feature files, is to act as chair and mentor. They are able to guide the team to write concise sceanrios and ensure conversation stays on track. They also help to make sure that everyone in the meeting contributes and is comfortable with what the features where specifying.

Ordinarily, the process would start with the user story, created earlier by the Business Analyst (BA) working with the Product Owner. This will help to identify each scenario to cover the feature and only entering into the given/when/then steps if it wasn’t immediately clear how a scenario would play or if there was confusion amongst the team. Once the sessions are over the DiT or BA will flesh out the remaining given/when/then, attaching it to the user story in Jira.

Because BBC iPlayer is available on iOS and Android we only ever had one feature file that both products would use. This would make sure that we kept feature parity and aided us to start delivering features on both platforms at the same time.

‘3 Amigos’ helped everyone involved develop a strong understanding of a feature and how it may need to be altered to work on each platform. This also helped to foster a more collaborative approach to creating feature files and to develop a better understanding of what the Product Owners wanted without the solution being prescriptive on the team, letting them decide how it should work.

Anyone not involved with the 3 Amigos session could read the feature file or speak with any of the developers or testers present to get a heads up. We try to make sure that different developers and testers attend the ‘3 Amigos’ to make sure everyone can run a session without a particular person becoming a bottle neck.

Once the developer has picked up the ticket to develop, they will submit the feature file into our source control system removing any reference to the feature apart from the user story and any acceptance criteria leaving only a link to the location of the feature file for future access. This ensured there was only ever one version of the truth and if any changes were required then there would be an audit trail to identify who made the alteration.

In my next post I will expand on how we use the feature files to automate our testing.

Originally posted on the BBC website 30 June 2014