LMAX Exchange
LMAX Exchange is a sophisticated piece of technology and it comprises a reasonable number of moving parts that have been assembled with a lot of love and care. As is the case with most non-trivial IT solutions, much of how the exchange is put together and also how it behaves is controlled by a large number of external configuration parameters. These properties define everything that conceivably could change from the URLs for LMAX Exchange venues down to the password for a given server and there are lots of them.
Configuration can also be used to address operational or release problems. It provides a way to quickly change or disable services or features that are playing up. Such configuration changes are often intended to be temporary but if they are made in the field they must be reflected back into the repository which is a manual process and a potential source of problems.
Managing all this configuration effectively is really important both for reliable service delivery and also for general development and operational efficiency. This is not as easy as it sounds.
Configuration complexity grows rapidly. We have a large number of configuration properties to start with and this is compounded by necessary differences between our many development, test and production environments. Clearly we strive to keep our environments as similar as possible but it is simply not feasible for them all to be completely identical. For one thing the cost would be astronomical but also any changes we needed to make would have to be implemented everywhere at the same time, which if you think about it just isn’t practical. Configuration complexity is a fact of life that we and many other people just have to deal with.
We also have some constraints on how we can address the problem. Compliance with UK financial services regulation requires us to maintain a separation between our development and IT operations teams. This separation means that the development team must have no access to our production environments (including any parts of the production configuration that would enable them to do so) and that IT operations must not be able to modify the application. This means that different parts of our configuration have to be separate and also managed by different teams. It also means that Devops in any real sense is not an option.
The standard approach for managing configuration is to adopt a hierarchical scheme where properties are defined at a number of different levels from global down to granular. In this case properties only need to be defined at a given level if they should be different from the relevant default at a higher level. This can drastically reduce the proliferation of property definitions but in reality it doesn’t necessarily make management easier. Comprehending, modifying and debugging the effective configuration needs to take account of more information, that is more dispersed, much of which is irrelevant and some of which is misleading. Given that it can introduce new problems and also make them harder to fix it’s a reasonable question whether this approach without any additional support is actually much of an improvement over dumb repetition.
The real problems with the hierarchical approach are partly technical and partly process. These boil down to providing transparency and traceability, finding and preventing errors, and ensuring that process is followed. We’ve addressed them with some facilities we’ve built into the exchange and through some additional automated testing.
Simply knowing what the effective configuration is in a given context is a good start. It is also very useful to know where a particular piece of configuration came from. Nothing too hard about that. Clearly we need to ensure that any secure configuration does not become visible to the wrong people in the process which is fair enough.
Automated configuration tests help us to identify a lot of potential problems. For a start we ensure that all configuration properties have a defined default and any properties that override a default aren’t allowed to have the same value. In some cases it’s optional whether a default is overridden but for certain properties it’s mandatory so we check for those too. Other tests check things like making sure that secure properties aren’t defined in the wrong place and that we always use logical network addresses (so that we can test the production configuration before release). The list goes on and we add new configuration tests all the time.
Finally we check that the configuration we have in a given environment such as production matches that expected from our repository. This picks up cases where the configuration has had to be changed by IT operations for some reason and ensures that we factor that back in where necessary. This deals with what was historically the main process headache.
This is all simple stuff but definitely worth doing. Configuration is a big problem for many organisations but it doesn’t have to be.