LMAX Exchange
We test a lot at LMAX Exchange and we do lots of different types of testing. Testing is completely integrated throughout our software development process and while much of our testing is automated we also have a team of dedicated test professionals who do a fantastic job of manual exploratory testing. All this testing involves the usual array of integration, test, compatibility, performance and staging environments none of which are exactly cheap. So it’s gratifying that all the investment and energy we put into testing actually pays off and as a result we have an exceptionally low level of software defects in production. Some people would probably be happy to leave it at that.
Over the last few months we have started testing in our production environments as well. The motivations for this are fairly compelling. No matter how similar to production you make your staging environment it will still never actually be the same thing. No matter how careful you are in configuring your software and infrastructure components there is always the potential for problems due to external factors or human error – even when all of your processes are completely automated. In the end whenever you release new software or make a change to an environment the only way you will ever know for sure that it works is to try it.
Smoke testing – particularly as part of a release – isn’t exactly new or unusual and in principle what we’re doing is similar but there are a few refinements. For one thing we don’t want any evidence of our testing mixed up with real business information. So we’ve created a separate ‘virtual venue’ that shares the same physical environment but which is completely isolated in data terms from the real exchange. Both exchanges use all of the same software and hardware components in exactly the same ways, together with all of the same physical and logical wiring. This allows us to do some fairly comprehensive testing and have a high degree of confidence that if one venue is working correctly then in all probability they both are.
Not surprisingly the initiative to do this started with software releases. We normally put out a new release every two weeks and this goes pretty smoothly. Being able to run real tests on production environments now allows us to flag up issues a lot earlier and thus have more time to address them.
As soon as the ability to test in live was available we started using it for infrastructure changes as well. As a result we now have a lot more confidence that everything is still working whatever we change. The additional testing has virtually no impact on latency or capacity so there’s nothing to stop us from using it throughout the week to augment our business level monitoring.
Overall we’ve had a very positive experience with testing in production. Clearly it is not a substitute for all the other great testing that we still do but we’re convinced it adds a lot of value.