The unintended consequences of automated UI tests

Whenever I see people talking about automated testing I always wonder what type of testing they actually mean? Eventually someone will mention the framework they are using and all too often it’s a UI based automation tool that allows tests to be written end-to-end (A-E2E-UI). 
They are usually very good at articulating what they think these tests will give them: fast automated tests that they no longer need to run manually, amongst other reasons.

But what they fail to look at is the types of behaviours these A-E2E-UI tests encourage and discourage within teams. 

They have a tendency to encourage  

• Writing more integrated testing with the full stack rather then isolated tests 
  • Isolated behaviour tests (e.g. unit, integration, contract tests etc) run faster and help pinpoint where issues could be
  • A-E2E-UI test will just indicate that a specific user journey is not working. While useful from an end user prospective someone still needs to investigate why. This can lead to just re-running it to see if it’s an intermittent error. Which is only made worse by tests giving false negatives which full stack tests are more likely to because of having more moving parts 
• Testing becomes someone else responsibility 
  • This is more apparent when the A-E2E-UI test are done by somebody else in the team and not the pair developing the code 
  • Notice ‘pair’ if you’re not a one-person development army then why are you working alone? 
    • Pairs tend to produce better code of higher quality with instant feedback from a real person 
    • It might be slower at first but it’s worth it to go faster later 
    • This is really important for established businesses with paying customers 
    • A research paper called The Costs and Benefits of Pair Programming backs this up but it’s nearly 20 years old now so if you know of anything more recent let me know in the comments.
• Pushing testing towards the end of the development life cycle 
  • The only way A-E2E-UI tests work is through a fully integrated system therefore testing gets pushed later into the development cycle 
  • You could use Test doubles for parts but then that is not an end-to-end test.
• Slower feedback loops for development teams 
  • Due to testing being pushed to the later stages of development developers go longer without feedback into how their work is progressing 
  • This problem is increased further when the A-E2E-UI tools are not familiar to the developers who subsequently wait for the development pipeline to run their tests instead of doing it locally
• Duplication of testing 
  • As the A-E2E-UI test suits get bigger and bigger it becomes hard and harder to see what is and isn’t covered by automation 
  • This leads to teams starting to test things at other levels (code and most likely exploratory testing ) which all add to the development time 

These are just some of the behaviours I’ve observed A-E2E-UI tests encourage, but they also discourage other behaviours which maybe desirable. 

They can discourage development teams from

• Building testability into the design of the systems 
  • Why would you if you know you can “easily” tests something end-to-end with an automation tool? 
• Maintainability of the code base
  • By limiting the opportunities to build a more testable design you decrease the maintainability of the code though tests 
  • If you need to make a change it’s harder to see what the change in the code affects
  • By having more fine grained tests you can pinpoint where issues exist
  • A-E2E-UI tests just indicate that a journey has broken and how it could affect the end users
  • Not where the problem was actually introduced  
• Building quality at the source 
  • You are deferring testing towards the end of the development pipeline when everything has been integrated.  Instead of when you are actively developing the code.
  • Are you really going to go back and add in the tests especially if you know an end-to-end test is going to cover it?
• The responsibility to test your work 
  • With the “safety net” of the A-E2E-UI tests you send the message that it’s ok if something slips though development 
  • If it affects anything the A-E2E-UI tests will catch it
  • What we should be encouraging is that it’s the developers responsibility to build AND test their work
  • They should be confidant that once they have finished that piece of code it can be shipped 
  • The A-E2E-UI tests should acts as another layer to build on your teams confidence that nothing catastrophic will impact the end users. Think of them as a canary in the coal mine. If it stops chirping then something is really wrong…   
• More granular feedback loops
  • By having A-E2E-UI tests you’re less likely to write unit and integration tests which give you fast feedback on how that part of the code behaves 
  • Remember code level tests should be testing behaviour not implementation details 

If A-E2E-UI tests cause undesirable behaviours in teams should we stop writing them? While they are valuable at demonstrating end users journeys we shouldn’t be putting so much of our confidence that our system works as intended into them. They should be another layer which helps build the teams confidence that the system hangs together. 

If we put the vast majority of our effort and confidence into these automated end-to-end tests than we risk losing one of the teams greatest abilities: building testability into the design of our systems. But just like the automated UI tests building in testability takes conscious effort. This will take time, patients and experience for the whole team to understand and benefit from.