Testing in production

We test a lot at LMAX Exchange and we do lots of different types of testing. Testing is completely integrated throughout our software development process and while much of our testing is automated we also have a team of dedicated test professionals who do a fantastic job of manual exploratory testing. All this testing involves the usual array of integration, test, compatibility, performance and staging environments – none of which are exactly cheap. So it’s gratifying that all the investment and energy we put into testing actually pays off and as a result we have an exceptionally low level of software defects in production. Some people would probably be happy to leave it at that.

Over the last few months we have started testing in our production environments as well. The motivations for this are fairly compelling. No matter how similar to production you make your staging environment it will still never actually be the same thing. No matter how careful you are in configuring your software and infrastructure components there is always the potential for problems due to external factors or human error – even when all of your processes are completely automated. In the end whenever you release new software or make a change to an environment the only way you will ever know for sure that it works is to try it.