What is quality?

tl:dr: Lenses of quality is a way to think about what quality means to software development teams.

A while back someone on twitter asked what does quality mean to you. To which I responded:

https://twitter.com/JitGo/status/1010031582395224064

Which not realising at the time but fitted in nicely with Gerald Weinberg’s description of what is quality:

Quality is value to someone

For most teams that someone could be their Products Owners (PO), the organisation they work for, the team they work with and their end users. All these groups of people could have very different views on what value means to them and even contradicting views in some cases.

For your organisation quality could be whatever helps them reach their targets for that quarter or year.

For your Product Managers (PM) or Owners their measure of a quality product could be a system or feature released on time.

For your team it could be a system that they can build, deploy, maintain and add to easily.

For your end users, well it could be something as simple sounding as it just works.

Also it’s not that the testers or developers don’t care about shipping early (what the PM wants), it’s more that they might care about maintainability or it does what we said it would do more than shipping early.

All of this could be just the tip of the iceberg and there could be many other people and views on what quality means to them.

As testers we need to help development teams understand that quality is measured by people in different ways.

Lenses of Quality

One of the ways I’ve started to help teams understand this is via the idea of lenses of quality.

Each of these groups of people view quality with a different lens therefore see the same system differently to one another. We as testers should help our teams to see quality through these different lenses by helping them identify these groups and what their measures of quality are.

This would help teams to start thinking about who their stakeholders are and how they are likely to perceive the systems that they build.

Would it possible to line up all the different lenses and be able to focus on one common quality metric?

If so would this be more like a microscope pulling into focus the hidden details or more like a telescope and allow you to see far into the distance?

Got an opinion then say so in the comments.

Junaid Valimulla: Are we nearly there yet?

Guest post from Junaid Valimulla currently working as a Senior Test Engineer who is continually learning and honing his craft. I first met Junaid back in my Sony Ericssion days as a Test Analyst and was always impressed with his ability to pick things up quickly and to start helping teams deliver on their goals but most of all his great sense of humour.

I hope you enjoy this post as much as I did as I think his metaphor so aptly describes what a lot of us as testers have felt.

Are we nearly there yet?

Every parent can attest to hearing these words at least once in their life. It is often said by a frustrated child who has has been jailed in the back of the car for what seems like a lifetime to them whilst on a journey to their grandparent’s house.

Now if we look at this from a tester perspective, it become obvious that it is not too dissimilar to the question uttered by many a tester all too often:

“Are we ready to test yet?”

Except in this case the tester is the frustrated child, the developer is the parent driving the car and the grandparent’s house is the ready to test column!

This is the age-old problem, which must change if we are to work in a truly agile environment. Testing has to be seen less as the thing that happens at the end and more of the thing that happens throughout! Testers need to be collaborated with, right from the kick off. The old cumbersome approach encourages the ‘throw over the wall’ mentality, which often leads to a build up of tickets and testing tasks and reaffirms the notion that testing is a bottleneck.

This change can be achieved but requires a number of things to happen. First and foremost their needs to be a mind set change from all parties. This is not just the testers, but developers, designers etc. All those that are involved with the creation of the product need to buy in to the collective responsibility mantra. This however is not something a tester can change and possibly influence alone.

Another thing that can be done and is something I believe the tester can drive, is make changes to the scrum board. Every team I have worked in has had a ‘Ready for test’ or ‘Testing’ column. A column, which ALWAYS comes after the ‘Development’ column. This has to change. This column, this simple yet powerful column undoes all that we are trying achieve. This column maintains the old tired way of thinking.

So, for testing to really be recognised on the board it must be removed off it? Really?? This may sound contradictory but it is the only way that testing will be seen as an on going task performed throughout rather than the bottleneck ‘bit at the end’! No longer will tickets be in ‘Development’ for developers and ‘Testing’ for testers. They will be ‘In Progress’ for all to work together and collaborate. This will encourage conversation between testers and developers, encourage faster feedback and ultimately require less testing at the end! Which is what we all want right?

To conclude what columns would I have then? Simple:

Ready for Sprint/Ready to pick up
In Progress
Done

More about Junaid:

Senior Test Engineer (BBC Sport App) with over 13 years experience working in the testing discipline. Testing software for a number of companies ranging from Mobile OS’s and Apps, Websites, TV STB’s, Wireless Routers and everything in-between!

You can find him on Twitter

A DevOps one-liner?

tl;dir You can try but you risk people either misinterpreting or making assumptions so you end up having to explain what you mean anyway.

I was recently asked if I could explain what DevOps was to someone new to the software industry and in attempting to do so I came up with a one-liner

A principled approach to ship working software reliably, repeatedly and consistently to enable fast feedback that the system is providing value to the end users.

But the more I thought about it the more I realised how loaded this fairly simple one line actually was and needed to break down each part to make sure that I didn’t mislead the person inquiring about what it was.

Principled approach

The idea behind this is to mitigate the cargo cult that has a tendency to happen with agile initiatives. This is where teams blindly follow the ‘rules’ without ever understanding the why or given the time to understand. By making your DevOps initiative a principled approach you not only force the team to state what their principles are but in trying to articulate them they have to actually understand them.

By making them principles you can avoid dictating to teams how they should work and what tools they should use. This should be a team decision with justification stemming back to the principles. This then gives the team a mechanism for assessing what they actually need rather than what the sales team of the tool thinks they need. Also because they are principles they can at a future date be easily revised once the team has learnt what is and isn’t working for them.

So what should your principles be? A good place to start would be the DevOps Handbook or go back to Agile Manifesto and adapt them to your context.

Ship working software

This must be to the live environment not staging, development or some sort of testing environment, but where real users actually interact with the system.

It doesn’t have to be every single users but at least a subset. I’d also go to say that beta groups don’t count here as they tend to be self selecting and may not be as representative of your real users.

The shipping part must also be done by the team building the system not some sort of operations team. They must be able to do this themselves otherwise they’ll never feel the pressure of their actions affecting end users. There is nothing like knowing your work has direct consequences on users to make sure you’ve done everything to make sure your changes actually work.

Of course it must functionally work which leads onto

Reliably, repeatedly and consistently

Pushing to the live environment must be reliable meaning that it works every time and doesn’t need specialist knowledge to be able to deploy. Ideally everyone in the team should be able to push from Developers and Testers to BAs and the Product Owner. At a bare minimum everyone of the Developers and Testers can push to live.

It must be repeatable so that you can deploy either once a week to once a minute (that might be a little on the extreme side but being able to allows you to push out fixes a lot quicker). It also helps de-risk releases and turns them into non-events, it’s just something you do like pushing to source control.

For pushing to live to be successful at being reliable and repeatable it must also be consistent. It’s the same process for everyone, there is only one (official) mechanism (preferably one button push) and the outcome is always the same meaning no nasty surprises like taking down the whole system. If it does break then backing out the change is as simple as one button or because you can push to live so easily a fix can be turned around quickly.

Fast feedback

To ship working software reliably, repeatably, and consistently you are going to need some good feedback systems.

Building in testability into your product from the start provides early feedback that functionally the feature works. Continuous integration lets you know that it integrates with the rest of the system early and if you need to course correct. Having your Testers involved from the very beginning of the the feature helps them to inform the whole team how the feature is likely works for the end users.

Coming up with results for which the feature will be measured against at the idea stages means the telemetry needed is built in from the start and helps the team to understand if it is heading in the direction the organisation wants. Not only that it helps the team to see if any subsequent alterations or other features from other teams have any effects on your work (negative or positive).

Providing value to the end users

This is possibly the most important aspect of any system being built. If it’s not meeting some sort of end user need that they find valuable then it doesn’t matter how well you do the other parts of your system, it isn’t going to last long and nor are you as a team or organisation. Ideally almost everything you’re doing is in direct (a new feature) or indirect (CI pipeline to deploy faster) benefit for the users.

If you can justify the effort that either improves the system for the end users or makes it easier for you to deploy faster or understand what is and isn’t working then it’s worth doing.

One of the best ways to start focusing on the end users is to know who they are.

This sounds simple to focus on the end user and if you ask any team if they do I’m certain almost all will say yes. But if you look at the work the teams is outputting it’s sometimes very difficult to see how it benefits the end users or if they were even considered again after the initial discussion on the new feature/update. A lot of teams haven’t even had contact with their end users or seen them actually user their systems.

Focusing on the end users isn’t simple thinking about them every now and again but actually finding out who they are, why they use the systems and how they actually use it. Remember a lot of people say one thing but actually do another so make sure you have multiple ways to understand how your users interact with your systems.

From there you can start seeing how you’re providing value to them even if they can’t articulate it to you. This way you’ll always stay a head of the competition and if any new entrants are likely to start stealing users from you.

Summary

You can come up with a one liner but you risk simplifying something too much that people think it’s easier than it is or they make assumptions which leads to a whole host of other problems. Such as people thinking they are talking about the same thing but both have a different understanding of it e.g. a quick ( and very unscientific) Twitter poll to gauge people’s thinking of DevOps varies quite a bit

So what's your thinking of #DevOps? Is it just...
— Jit Gosai (@JitGo) June 11, 2017

It’s one of the reasons why I say we should just stop calling it DevOps to get people to explain what they mean rather than everyone assume they are all talking about the same thing.

In test column

Update 4/1/20: I’ve also written about How to move away from the In Test Column

A colleague recently asked me why I think the in test column is a bad idea. So I had a quick search around and couldn’t find anything specifically on the topic I decided to put some of my ideas down on why we shouldn’t have one and instead to opt for a more generic In Progress column.

What’s a board?

Most delivery teams have boards of some sort to make the work they are doing more visible and show progress towards some goal. The two most common ones I see are physical boards with sticky notes and avatars representing individuals in the team

The other is a digital one usually Jira or Trello

The usual configuration of these boards are Backlog, In Dev, In Test, Done with tickets moving from left to right as the work progresses though each column. This helps the team visualise where work is currently up to and making it less abstract.

This all seems sensible as you can monitor the number of tickets in each column and create whatever stats you need to report on but from my experience these types of boards that break down in progress work into Dev and Test roles encourage certain types of anti-patterns.

Inner team silos

Boards like this have a tendency to encourages invisible silos within teams. The backlog column becomes Product Owners (PO)/Business Analyst (BA)/Project Managers (PM) responsibility and overall ownership

The Dev and Test column’s belonging to their respective disciplines and things only getting into done when Test say they are done.

Therefore settings up the disciplines into their silo’s and only doing work that is in their column and handing work off to the next column. Rather than jumping into different columns when they can help or pairing on tasks.

Pushing testing towards the end of the delivery pipeline

With the In Test column coming towards the end of the board and normally the one just before Done it starts pushing testing towards the end of the delivery pipeline rather than continuously throughout development. It also starts putting Test in the position of releasing the software rather than it being a team decision.

It almost sets up Test to be the last line of defence before a release.

More work in progress

Another consequence is that Devs starts picking up new work from the backlog while the Test team are verifying the previous ticket. If Test find an issue the ticket has to either go back into the Backlog or straight back into Dev.

Now the team has to decide do we let the Devs switch context, fix the issue so Test can carry on. Or finish their existing ticket, make Test wait, let them context switch to pick up something new from the Test column?

Context switching for either side isn’t a good thing as there is always the usually spin up time to get familiar with the ticket. Which can lead to people taking short cuts to get the work done quicker.

Testing becomes the bottleneck

The only constraint on the Devs doing more work (tickets) is what can be pulled in from the backlog and the amount they can do. Given the nature of our work (building software solutions) teams always have more Developers then Testers. So teams have a tendency to use the Dev teams to 100% capacity churning out more and more tickets.

As the Devs begin to get through more work, tickets begin to pile up in the Test column waiting for Testers to become free. This naturally leads to Test becoming the perceived bottleneck in the system as this is where the tickets are being held up.

How do you fix the bottleneck?

3 options come to mind and 2 that are usually discussed are

1) Add more testers

This is what generally happens if the teams/company can hire them or borrow test resource from somewhere else but for most this isn’t really sustainable or even an option.

2) Automate more of the Test teams work

This usually leads to more automated UI tests which is surely all that the Testers are doing… right?

See UI Automation, what is it go for? (AKA the Automation fallacy) on why automating at this level isn’t going to give you want you want.

The third which is rarely discussed is

3) Remove the In Test (and Dev) columns

This may sound strange as it looks like all you’re simply doing is hiding the work so let me explain.

You switch to having an In progress area instead. This is pretty much any work the team is currently working on and by this I mean actually doing not something they were doing and are now waiting for some information. This should be almost anything: meetings, training, coding, spike work, testing, absolutely anything any of your teams members could be doing that day that takes longer than eg 1 hour or whatever makes sense to your team to have on the board.

Now instead of just adding a Dev (or Devs if they pair) to a ticket also include a Tester as soon a the ticket gets pulled in. As the Devs are getting up to speed with the ticket the Testers should be too forming a 2-3 way pair/group (I’ll refer to this as pair for the duration of the post). This pair will stay with the ticket all the way to Done. No handoffs, no queuing up work in other columns. It is their responsibility to see the work to Done, preferably to production with Testers providing there and then feedback to the Devs on how the work is proceeding and helping them to bolster any automated testing eg unit or integration testing.

If you find that the Devs are taking longer on a ticket then it’s probably a sign that the ticket is too big and needs to be broken up into smaller, independently deliverable parts.

By having just a In Progress column and pairing Dev & Test from the start will foster a more collaborative approach to getting the work actually done in a team. It also stops more work being pulled in while other tickets are still in progress, reduces the amount of context switching for all and starts addressing the issue of Testing becoming the bottleneck.

Remember Testing isn’t something that happens once the Devs have finished it’s something that should be happening continuously.

Update 4/1/20: Looking for details on how to do this then checkout my post on How to move away from the In Test Column

How to break the rules?

Dan North gave a really interesting talk at last years GoTo Conference called How to break the rules.

He takes Eliyahu Goldratt 4 steps of how technology is adopted which he presented in a series of lectures he gave on why people didn’t take on his ideas of theory of constraints (TOC) from his book The Goal.

Eliyahu stated that organisation need to go through 4 steps before a new technology can be successfully adopted:

What is the power of the technology
- What does it do for you?
What limitation does the technology diminish?
- What will it make better
What rules enable us to manage that limitation presently?
- And how much are we wedded to those rules
What new rules will we need?
- For the technology to succeed

Dan than takes these 4 rules and applies them to real companies that either succeeded or failed to take advantage of new technologies. The interesting part is for the companies that failed what rules (part 3) they needed to break to make use of the new technology.

I’ve made my rough notes on the talk available below but I highly recommend watching it.

My notes from the talk:

Talks about The Goal (book) and Beyond the Goal – lectures by Goldratt

Series of lectures about 20 years after The Goal was released
He tries to attempt to explain why people didn’t apply his Theory of constraints successfully if at all.

First two lectures are: How to adopt Technology?

What is technology: The application of knowledge

For it to be adopted then answer these questions:

What is the power of the technology
- What does it do for you?
What limitation does the tech diminish?
- What will it make better
What rules enable us to manage that limitation ?
- How much are we wedded to those rules
What new rules will we need?
- For the technology to succeed

He then goes on to apply these questions to real companies.

MRP – first application of computers in a business situation
- Calculated cost of materials for manufacture DuPoint
- Which allowed them to ship faster then others
- But when the competition tried it they couldn’t compete as to take advantage of the system you need to change the rest of your business
Goes on to apply to ERP and
Cloud computing (orgs moving to it)
But also Continuose delivery

Interesting: Dell becomes as big as it did because it could work in smaller batches then the competition . All the others companies offered you a fixed machine. Dell changed this by going online only and allowing you to customise your PC and could ship faster than the competition.

Swedish saying: When talking to farmer use farmers words

Commenting on how we collaborate across divisions
If you want to collaborate successfully you need to either be speaking the same language or use words the other understands

A lot of organisations are failing with Agile/Kanban/Scrum etc because they still have all the exiting rules in place (see point 3 earlier).

To be able to adopt the new technology you need to move to the new rules (point 4) otherwise you’re still doing the same thing.

Used the example of Amazon not doing Cost accounting and moving to throughput accounting and flow of value

Cost accounting is what each department cost
Through put accounting is what is being produced by the department and what value does that provide

The problem with step 3 is that it eventually becomes the culture of the organisation and trying to unpick it then is really hard.

So How to break the rules?

Understand the power of technology
Recognise the limitation the technology will diminish
identify the existing rules we use to manage the limitation
Identify and implement the new rules

Summary

Another great talk from Dan and goes to show a lot of our problems in software development have been issues in other industries.

Not only that they have been solved but we tend to overlook them as they don’t look like our industry. What has manufacturing got to do with writing software? In all don’t take my word for it go watch the talk and start having a look at Eliyahu Goldratt body work. He was onto something…

UI Automation, what is it good for?

TL;DR: What automation at the UI level does and doesn’t give you.

UPDATE: I originally wrote this back in March 2015, lost it in my drafts and found it again recently so thought I get it out there. Don’t agree then let me know in the comments.

Automation fallacy

Every time I speak with different teams and organisations a theme constantly comes up, UI automation and how it’s going to solve all their problems. The thinking goes that if we can automate more of our tests – read test scripts – then the Testers no longer have to check that item anymore. This then frees them up to do more interesting things like exploratory testing or that the Tester can be done away with altogether.

There is also a notion that automating all the regression checks will drop the regression test cycle from days to hours. This then supposable allows the team to move faster and release quicker then before.

What everyone seems to miss is that Automated checks are generally built to check one thing and will tell you if that thing is still there or behaving as the script has been programmed to tell you. If anything else happens that wasn’t programmed into the check then it fails or stops dead, relaying on someone having to then go look and see what went wrong.

A Tester on the other hand can look for workarounds, workout what may have caused the issue or go find other issues based on the information they’ve just learned.

So should we give up on UI automation and face that we’ve got to do rounds and rounds of regression testing and hire more Testers? Well no, what we need to do is ask ourselves

Why are we automating?

It’s looks like a simple question and most people (including myself in the past) would be able to give you a list of answers but what we forget to question is by automating this check what does it tell me when it passes or fails? If it passes does that now mean I no longer have to check that feature or scenario again? If it fails what does that tell me? That I have to check that scenario manually?

When a check fails what do we expect the team to do? Stop everything and investigate the issue? Carry on as normal and hope someone else will check it? Ignore the issue altogether? Who is responsible for checking the issue? Developers, Testers, dedicated automation engineers?

There are a lot of reasons that people give as to why they want to automate their testing such as

Reduce test/regression testing
- The reason for regression testing is to see if the changes you’ve made to your code base haven’t broken anything existing.
- Unless you have automated all your UI Checks/regression suits then automation is not going to help you as much as you think it will
Spot issues/bugs faster
- Automation doesn’t find new bugs it only tells you that the check you’ve scripted has broken in someway. You need to tell the script that if action A doesn’t produce result B then fail with an error message. What normally happens is the check fails in away your didn’t anticipate. Don’t forget if you knew before hand how something would break you would probably have put in a fix. Thats why they are called defects something behaving the way you didn’t want/anticipate
Free up Testers
- Potentially but that is if they trust the automation
Consistently check a feature the same way
- This is one thing a automation check is very good at
Something that is laborious or difficult to setup and check
- Another good candidate for automation. We use it to do policy testing of our apps as it’s time consuming and prone to error when trying to manually test
We’re doing Behaviour Driven Development (BDD)
- BDD is not about automating but more about collaborating to understand and create features. The automation is just one small part of it and even then it’s not about testing the UI but the business logic which could be tested at the Unit level
- If you ever hear a development team saying ’The BDD tests are failing’ then its good indicator that they are probably using BDD incorrectly
To release faster
- Again because you need to do less testing, see Reduce test/regression testing
It’s a part of continues integration/delivery so we have to
- No thought into what you are automating other then it’s what people say you have to do
Test manager or some other higher up tells you to
- Someone thinks that just telling a development team to automate their testing will help them, see above
People within the development team or key stockholders don’t trust the developers work
- The test team are being used as a safety net to check the developers work which tends to have a self-fulfilling prophecy for the developers who start using the test team as that

What does an Automated Check actually do?

Let’s start with an example from an mobile app but could very easily be any platform of your choice:

A simple automated scenario could be when the home page is loaded and I’ve selected an option then I expect to see items X, Y and Z.

Things this scenario will need to do are:

     Start the application
     Wait for it to load up
     Select a menu option
     Wait for the new screen to load
     Then check that the expected items are on screen.

Click to view animated gif of example automated check

So you can run this check over and over and know that as long as the sequence doesn’t change and the items you are checking for are there then the test will pass. What it isn’t going to tell you though is

Formatting issues with any of the screens loading up
It is starting to take longer for pages to load in
The ordering of the menu options has changed
There are new menu options
The items you are checking can be seen by the automation framework but nothing is actually visible on screen
There are new items on screen that the check is not looking for

All of the above could also be scripted into the check but would likely take quite a bit of effort and you can’t always predict how an app will behave and therefore not be able to script for it.

This is where a real Tester has the advantage. You don’t need to tell a Tester to look for these things they will do this without being prompted and not only that generally a lot faster then an automated check. They can also tell you if it doesn’t feel right or perform in away that would be acceptable for end users which can be very hard to quantify and therefore automate. They can also use the information that they’ve just learned and apply it to what else they can discover. An automated test isn’t going to be able to do this, not with the tools we are using at the moment

Where a Tester can’t match an automated check (or will find very hard to) is checking the same thing in the same way consistently and quickly. As long as they are no physical moving parts an automated check can normally carry out the scenario above in seconds only being delayed by waiting for things to install or load.

So should we stop automating our checks?

Before any team starts to think about automating their testing via the UI they should first, as a team, ask themselves:

Why are we automating?

It sounds like a simple question but as I explained earlier people tend to have differing views on what the automation is actually going to do for them. By talking about why they want to automate they are more likely to come up with solutions that will actually address the problems.

One of the main benefits that I’ve seen from automation especially at the UI is faster feedback that the app:

Can actually be installed on a real device/displayed in a browser
The app can start without crashing
The app can reach any endpoints that it relies on
The apps core feature, the one thing it is designed to do actually works for your users e.g.
- BBC iPlayer: Can video actually be played
- Google maps: can provide directions to a destination
- Amazon: allows you to buy products
- Facebook: The feed shows you what your friends and family are doing

To do this manually, every time a build is made, could take some time but not only that is very tedious and from my experience just doesn’t happen. What tends to happen in this scenario is that developers will wait and see what comes back when Testers finally do test the app. This could be some time from when the change was made and the Testers finally being able to test it.

The longer this feedback loop is the hard it is to fix due to the overhead in understanding what went wrong and what changed to cause the issue. This is exacerbated when working with legacy code especially when not written by the developer making that change.

By automating just the core journey the development team know very quickly that whatever was last committed hasn’t caused a catastrophic failure and that the apps core feature is still functioning. If there is a failure then you can back out the change (or ideally fix it) and get back to a working state. This helps the whole team know that the app works and improves the team overall confidence that if I install this app it’s actually going to be worth their time. There is nothing more frustrating especially in mobile development to get a build, find the device you want to test on and install it only to find it can’t carry out it’s main job for the user or worse crashes on start.

When things fail so easily and obviously it does nothing to instil confidence in the development team more so when your key stakeholders find the issues. This also allows you to start using your Testers for what they a really good at testing and not just checking your developers work.

Core Journey

We use the concept of PUMA to decide what our core journeys are and ultimately what we should and shouldn’t automate. A generally rule of thumb is if it’s not a core journey then can it be covered by a unit/integration test not invoking the UI. If it still can’t then why would automating it help? Who would do it? How often does it need to run and how quickly do we need feedback that it’s broken? Could we monitor the app stats to check if is still working rather then automating it? If it does break how bad would your users be affected/perception be? Could it be controlled by a feature toggle that allows it to be switched off in the live environment?

So the next time someone asks why don’t you just automate your testing ask them “Why are we automating?” You might realise that the problem they perceive can easily be addressed by one simple automated check rather then 100’s of automated UI checks.

The Do’s and Don’ts of Mobile UI Automation

(…from my experience)

Original posted on Medium

My name is Jitesh ‘Jit’ Gosai and I’m a Senior Developer in Test (DiT) at the BBC working in Mobile Platforms, Digital.

I originally wrote this post well over a year ago (2014!) and recently came across it again so thought I might as well get it out there. Most of the points still stand so should be useful for anyone getting into automation or are already on the road in doing so. Got any other tips then let me know in the comments or Twitter (@JitGo).

During our journey of automating our mobile user interface (UI) testing for iPlayer, we’ve learnt many things (good and bad) that I would like to share with you to help you with your work.
Below, I have compiled some do’s and don’ts to help you and hopefully save you from making the same mistakes as we did.

Android: adb (android device bridge) is your friend

Learn all the little things you can do with adb from simply listing connected devices, to grabbing screenshots. Adb will allow you to do quite a lot with a connected device so make sure you are comfortable with it. My suggestion is to begin with the google developer docs and then start searching around for things you would like to do; for example waiting for a connected device to have started, restarting devices, or sending simple commands to control the UI (don’t try to automate your test this way, you’ll be looking at a lot of pain).

Do learn to control your devices remotely

Learning to control your devices remotely will save you a lot of effort both in time and flow, especially if you have devices connected to your build server which is not physically nearby (in our case, locked in a server room)
Being able to remotely view the device screen and restart the device is very useful and can save you countless journeys when your device ends up in a state you can’t identify. It is also great for being able to return your device to the home screen and therefore into a known state before resuming testing.

Do ditch the android SDK emulator

We tried to use this but found it too slow and unreliable, randomly crashing or disconnecting itself from adb. Intel HAXM was faster but proved to be too flaky and would also crash intermittently. Maybe it was something to do with our setup but we just went with connecting a device to run the jobs and proved to be a lot more reliable. Genymotion is another option which is free for small teams so could work for you.

Do be patient!

When first starting out don’t be tempted to keep playing with your test/build environment for little improvements. Let it settle and run. Know exactly what needs fixing and do just that, not things that are nice to have but essential. Then once you have stability then start to slowly add the things that may help improve speed but with an eye on reliability.

Do be available for pairing

Pair with devs and build tests together. This gets devs familiar with how to write them, encourages them to write tests and stops you being the bottleneck when they break. Also show the UI tests to your Testers to get them familiar with what can and can’t be automated. I’ve found that by walking Testers through an automated UI test vastly improves their knowledge of how they work and things that it can miss.

Do test one thing and one thing only

Don’t be tempted to cover lots of things in one test as it can be harder to tell what has broken when they fail and harder to debug.

Don’t use (or use very sparingly) canned steps

When using step definitions for automating tests it is very tempting to keep reusing steps you’ve already written which at first works well. It can even allow non-technical members of the team to automate test but can result in very verbose scenarios – which are harder to read and more difficult to amend at a later date. We always create methods for interactions with the app e.g. go to home or go to channels in the case for iPlayer. We then use these in our step definitions. This way if the path to get to the to home or channels pages changes then you just update the method and all tests get the change, no need to update all your tests. It also allows you to write tests faster in the long run.

Do use the Page object pattern or similar

This will vastly improve the readability, maintainability and reusability of your tests and teach anyone who will be working with the code how to use it effectively.

Do push your test framework (calabash, appium etc) as far down in the stack as possible

Don’t litter your test code with your frameworks commands but instead delegate that to a module that you access via an interface.
This way if a command changes in a new version you only need to update in one place and if you decide to switch frameworks you can potentially a lot easier.

Don’t fall in love (with your tools!)

They are just that, tools, to help you in your task. If you find that it is causing you more problems and no matter how much searching you do no one can help then dump it and switch to something else (if there is).

Don’t automate everything under the sun

Automate the areas that are going to give the most valuable feedback i.e. that something has broken. We’ve found that UI test are most useful when the dev’s are actively working in those areas so focus your UI tests there.

Do have the need for speed

Keep your tests as fast as possible. Running iOS test through the simulator is faster than devices so we tend to stick with running them there. But for android we use devices as it’s more stable. So you need to weigh up what you need – stability always trumps speed. With that said keep your tests as fast as possible. Don’t use sleeps in your tests (or avoid them at all costs). Use waits that will check repeatedly for something to be on/off screen before proceeding or raising an error.

Do stub it out!

Stub out the data your app uses.We use Charles Proxy with some ruby scripts to automate the launching, closing and loading in of config files. The main reason we used this approach was that devs and tester were familiar with it so the learning curve was easier and was a quick solution that we came up with until a more appropriate solution could be developed/re-purposed such as REST-assured or WireMock.

Do use Metrics

Stats, stats and more stats. Use data to back up your ideas and collect as much as you can
Stats such as test execution time, pass/fail rates, number of tests, no of runs and charts these. We us dashing as it’s quick to setup and offers a very nice visual way to display data.

Dashboards displayed by the app development team. From the Top left app statistics via Graphana and app usage. Bottom left dashing board showing test status, code metrics using custom bubble graphs. Far right screen showing current build status

I’ve found that the Riskshaw widget to be very useful. Some teams have also been experimenting with AtlasBoard which looks promising.

Do learn your tools

If you are using Ruby (as we are) then learn to use a REPL such as Pry. This will save you countless amount of time when debugging or even creating tests. Watch this video by Conrad Irwin for a great introduction to Pry and REPL driven development.

Do have multiple tests around any given area of functionality

This way if a test fails you know it could be flaky but if all the tests in that area fail you know straight away something is wrong.

Do read these posts!

This excellent post by Gojko Adzic for anyone attemptig to automate testing at the UI level.
If you are on the journey of UI automation or plan to start then I also highly recommend reading this post by Dan North, The Siren Song of Automated UI Testing.To give a balance view on automated UI testing and that it’s no silver bullet to removing manual testing altogether or other (arguably better) forms of automated unit/component testing.

Got any tips on automated testing? Then let us know in the comments and you never know it may make it in the list above.

How to keep up to date with the testing industry?

I recently spoke at UKStar 2018 and the team there asked me a few questions on how I keep up to date in Testing and the industry as a whole which I’ve reproduced here.

What is your favourite testing book/blog? Why is this your favourite?

Two books come to mind. The first is Thinking Fast and Slow by Daniel Kahneman. This book is about how we make decisions and the fact that we are not really all that good at doing it, let alone realising what we are basing our choices on.

Secondly is the Freakonomics set of books by Steven Levitt and Stephen Dubner. Freakonomics is a great way to see how you can go about answering questions that seem impossible at first; but by using some more unconventional methods, you can answer almost anything.

What I enjoy about these books is that neither of them are about testing or software development but the questions that they both aim to answer are so relevant to the work we do.

How do you keep up to date with the software testing industry?

Whenever I’m asked this question it reminds me of what Martin Fowler always says to engineers when they ask him how do they become a better developer? His response is always “understand the business that you work in”. So rather than focusing on testing I take a broader view and look at our industry as a whole.

I tend to follow a number of bloggers whose insight into our industry I have found invaluable.

Ben Thompson’s Stratechery blog has some excellent indepth analysis on the software industry as a whole and how some of the larger organisations (Google, Amazon, Facebook, Apple etc) operate.

Benedict Evans weekly newsletter and Charles Arthur’s daily Overspill site are also great for finding technology focused articles from around the web that you probably would have missed, or never even come across, always with some illuminating commentary.

Twitter is a great source of information too. I tend to build up my feed by following people both inside and outside of the tech industry to get a broad range of ideas and influence.

What is the biggest misconception about testing that you’ve heard?

That automating all your testing will speed up the delivery of your products. A lot of teams still see testing as that bottleneck at the end of development stopping the release. So if you automate all your testing you will be able to ship faster.

What they almost always fail to see is that their releases are either too big so the test team can’t isolate and feedback on the risky areas or they still see testers as simply running check lists against the product. Hence thinking that they can be replaced with automation. Tester’s always do so much more than this but their work has had a history of being boiled down to “Have all the tests passed”.

Automating BBC iPlayer mobile testing part three: legacy vs new features

Originally posted on the BBC website 30 June 2014

This is the third and final part my series of posts about how the BBC iPlayer Mobile team automate their user interface testing. You can find the first part here as well as the second part here.

Legacy features Vs. New features

Native BBC iPlayer applications on mobile have been around since 2010. As with any tool, we inevitably had a smaller range of automation options available to us in the early days. This means we have a large suite of manual regression tests that are executed prior to deploying new application versions.

It was a decision agreed by everyone that going back and trying to retro fit automated tests to legacy features would be very time consuming and the team wouldn’t get the benefits of having automated tests for the features they were currently building. This meant that we would then only build automation tests for features that are actively being developed.

To help address the backlog of manual tests the DiT’s on the team, when free, will pair up with Test Engineers and see what legacy areas of the app would benefit from automated tests. Using this approach we are slowly building up an automated regression suite to be run each night on our latest development build.

The Future

At present the automation tests are only executed on a small handful of iOS and Android devices plugged directly into our Continuous Integration (CI) server while we get our build processes stable and reporting less false positives. The long term ambition is to run these automation tests on as many real, physical devices as we can. We’re working closely with our Test Tools team who are currently developing a device testing platform for Mobile, Tablet and even smart TVs that will help us scale our testing efforts (look out for a blog post on this soon!)

With the team well underway with creating feature files collaboratively and automating features we hope that this will enable us to release new versions of the app quicker and with more confidence around stability and quality.

Since we started automating more of our testing it has raised some questions on its effectiveness on the development process and whether it really is the right way forward. What do you think? How would you move forward?

50-70 tests are quite easy to manage but what happens when you start to get to 100, 200 or even 500 tests?

UI tests tend to be more flaky than other types of automated testing due to testing the system as a whole rather than in focused units. This leads to greater number of points of failure so how do you limit the number of places the tests could fail?

Our tests currently use a cross platform framework (Calabash) but would a more platform specific tool be better i.e. iOS instruments and Androids Espresso? The DiT team intend to spend some time evaluating the alternative solutions.

UI automation tests only tell you that a specific thing is still working e.g. you can still add an item to your download queue.

However, it will not tell you that the item added is incorrect (perhaps the episode of Eastenders you selected and added is actually Top Gear when you play it back). This is an example that an automated test would never be able to verify and shows where manual testing effort continues to be required and add value.

Automated tests don’t prevent bugs they just tell you that a bug exists (admittedly sooner than typical manual tests). We consider them a small part of the bigger picture of software development which should include other best practices of unit testing, test driven development, pair programming and excellent manual testing techniques.

What would you do? How would you move forward with the automation testing? Comment below and we can explore them in future blog posts.

We’ve learned a lot and still have a lot more to learn but I will be posting again with lessons learned on best practices on how to automate testing for mobile.

If there are other topics that interest you about testing then do let me know in comments!

Originally posted on the BBC website 15 August 2014

Automating testing for BBC iPlayer mobile part two: automation

Originally posted on the BBC website 30 June 2014

This is the second part of a three post series exploring how the BBC iPlayer Mobile testing team has integrated automated user interface (UI) testing into their development practice.

This post will deal with automation.

By creating collaborative feature files through the “3 Amigos” sessions and setting up a robust system for creating and disseminating them, the natural next step is to begin automating them to increase productivity and quality.

To make the tests as easy as possible to write we implemented the page object pattern so that the developers were clear about how to write more maintainable and less flaky tests. This also meant that test were written more consistently and allowed for more code reuse.

In addition to the page object pattern, we created helper modules that contained all the commands that they would need to drive the app, so it was easier for developers to quickly look up what commands are available, and demonstrated how to use the inbuilt debug tools to query the app to find the screen elements.

Although we explored many different options, we decided to use Calabash and Ruby as the predominant tools to automate our tests as they worked cleanly with Cucumber (which is our test runner) and because Calabash had support for both iOS and Android. To help everyone get to grips with the new systems, internal workshops are held to step developers through real life examples, aiding them to organise the feature folders, creating page objects and types of Calabash commands available to drive the app. By providing step by step guidance, everyone is able to get a strong understanding of the process and where they come into it.

Initially creating the automated UI tests is a slow process as you are required to create a fair amount of support code (including page objects, working out how to access elements on screen and working around timing issues with the app) but once these foundational aspects are set in place, automating tests gets faster and faster.

If a developer ever gets into difficulty Developers in Test are available to pair up to help iron out any problems.

There are many advantages to developers writing the automation tests. Ownership creates a sense of responsibility and a smoother process for delivering and testing the products. It also drives the developers to look at the results and take advantage of the benefits of faster feedback.

With developers using the feature files to write their tests, it ensures that the product is as intended, rather than based on an assumption, which speeds up the development process. The benefits of this is that everyone takes mutual responsibility for automation and prevents testing being pushed to manual when a DiT is absent or unavailable, which keeps the process moving more succinctly and effectively.

Another benefit of using Calabash is that it uses the accessibility labels to access on screen elements. If the developers build the tests they have to enable the labels therefore helping to make the app more accessible. For more information on accessibility practices see Senior Accessibility Specialist Henny Swan’s blog posts.

You may be wondering what the DiT’s are doing if the developers are creating all the automation code?

DiT’s remain embedded in the team and available for pairing to help automate tests that are not straightforward. They help build up tools to aid automation e.g. worker methods to carry out complex interactions or how a feature could be automated if not immediately obvious. They help keep Continuous Integration (CI) jobs running and investigate brittle tests. DiT’s also tend to be the experts with the automation frameworks so advise if a feature is worth automating or it’s better to test it manually.

Once the feature file has been automated the tests are pushed into the main build pipeline. They will be run approximately 4 times a day and a subset on each check in of code. We have our build jobs status displayed on large screens (one of the advantages of working near the TV platforms team is that they have a lot of reference TV’s that we can use when not being tested on) so if anything fails the whole team know straightway.

In the final post of this series I’ll tell you we handle legacy and new features and what the future holds for our team.

Originally posted on the BBC website 06 August 2014