I’ve been thinking a lot about contract testing lately and trying to explain why it’s a good idea.  I thought I’d start by getting my initial arguments for it down and go from there.

Got an opinion then let me know.

Note: This is a first draft (published 24/10/19) and I’ll (hopefully) revisit it again soon but in the meanwhile here is me thinking out in the open… 

Update 18/11/19: Added more details on what contract testing actually is.

The “Contract Testing Chat”: The Reality, The Problem, The Possible Solution?

Aim: To encourage dev teams to use contract testing to manage their integration of dependencies

The Reality 

Within any of the systems we produce there are components from external teams as this allows us to focus on what is important to us and let the other teams take care of thing that are not our core competency. 

In an ideal situation we would probably make everything ourselves so we have complete control but that would require significant amounts of Time, Money and Skills.

Time – To train your existing staff or recruit the people that have the skills and then allow them to actually build the component/systems.

Money – To hire the people and all the necessary resources they need to do the job.

Skills – That the person needs to be able to do the job

Some organisations can throw money at the situation and recruit the best in the industry and do everything in-house. Think of large organisations with deep pockets and large global brands.

Others (like us) don’t have this luxury and have to rely on external teams and component makers to make up for the parts we choose not to focus on. This leaves us with a dilemma. 

The Problem 

Do we just simply trust that these external components (dependencies) will work as we hope and the teams maintaining them will let us know when things change? What most teams do is integrate the component and put it through a couple of rounds of exploratory testing just to be sure things still work as we intended. If an issue is found then it’s a matter of understating what the problem is and where the problem could actually be and who’s responsibility it is to fix it.

This strategy works quite well once the initial issues have been ironed out. 
Eventually though a new version of the component is released and you need to decide if you test everything again or trust the release notes and just do focused regression testing.

Possible solutions

Focused Regression Testing If you just do focused regression testing and an issue is found in the live environment then trust in the dependence maintainer and possibly the development team integrating the component is diminished. The general response to this is to do a full regression of the integrated component every time. 

Full Regression Testing Full Regression Testing usually takes more time, money and skills so teams only integrate newer versions of the dependencies if they really have too. Generally when it contains something that they need e.g. a new feature or bug fix that affects the team directly. 

But because of the large gaps between integration of the previous and latest dependence there are likely to be even more changes then the team anticipated so not only does a full regression now have to happen but there are likely to be more issues found leading to even longer lead times in integrating the component. The blame game normally starts about now, see below. 

Automated end-to-end testing Some teams try to address this problem with automated end-to-end UI testing. Why? Well that’s what the Testers are doing during regression testing right? Just checking the functionality of the system and finding all the issues. So if we can automate this then we can not only find these issues faster but repeatedly and freeing up the Testers to do other things. It almost looks like you address the time, money and skill question is one initial up-front cost of building out the automation see UI Automation, what is it good for? 

Unfortunately these test only find what you program them to find and not only that the more components the end-to-end test run through the greater the chance of failure from false positives. If it does find an issue then you need to work out where the problem actually is: the test or the code. If it’s the code then another developer needs to investigate where that issue is and you’re heading to the blame game backlog issue. 

The Blame game 

The blame game is when the dependency maintainer blames the integrating team for not taking updates often enough and attempting to integrate the component in a way that they didn’t intend. On top of that any issues now found by the integrating team needs to go into the dependence maintainers backlog to be prioritised as they have other competing work to be getting on with. It’s not like they are the only team integrating their component. Meanwhile the integration team is blaming the maintainers for sneaking in features and bug fixes that they never asked for and holding up their development process. 

Last resort solution? 

Find another supplier Once The Blame game starts then this usually leads the integrating team down one of two paths. Find another more responsive supplier. Perhaps paying an external team to the company might solve their dependency problems. This is throwing money at the problem (see money, time and skill from earlier) or they…

Build it in-house This is all about taking back control and making it the teams responsibility to build the component. No more having to worry about another teams backlog or building things that have no relevance to your team. This is the skills part of the money, time and skills from earlier.

Both of the above solutions is a break down of the relationship between maintainers and integrators or more commonly your dev team and those PITA’s over in <insert location/team/department name here> 😆

Is there anything else we can try that could help with all the issues above and prevent the relationship breakdown? We ended up down this path as we needed to address the time, money and skills costs we couldn’t afford as a team, but all the options above results in one of the core costs having to be paid.

A possible solution? 

Contract testing and Consumer-Driven Contract testing 
Contract testing helps address the time cost by allowing an external team maintain a dependency. This also addresses the money question as the responsibility to fund that team essentially becomes someone else problem along with the skills issue. All the dev team needs to do is integrate the dependency. So how is Contract testing going to actually help? 

See What is Contract testing for more details.