LMAX Exchange
My team was constantly explaining to partners/clients how the combination of our message rate + their distance from LMAX Exchange + their 20 levels of depth request + their bandwidth = can create a terribly congested pipe for their session with LMAX Exchange which in turn would result in a very poor experience for them.
Unfortunately, we had to spend many hours trolling through our very detailed FIX logs as well as requesting TCP logs from our IS colleagues in order to get the data we needed in a format that clearly explains and documents this experience so that we can be more precise with our explanations. The FIX logs allow us to review the timestamps from the time LMAX Exchange creates the FIX market data message to the timestamp of when it is put on the wire.
This gives us a clear idea of whether there is a significant delay in getting the message out of LMAX Exchange to the client. The TCP logs then give us the details of the client’s TCP window size and we can use this value to identify that when we see a TCP Window Size returned to us in a rapidly diminishing value then this clearly points to the client being unable to process our messages quickly enough. These messages end up backing up on our side waiting to be sent in the next TCP packet where the window size was !=0
Obviously, if a client is far away from LMAX Exchange then it will take longer to send and receive messages. Since most clients connect to LMAX Exchange via the Internet, if they are far away from LMAX Exchange then the chances of having messages dropped is significantly higher. If a client is,
- geographically distant from LD4 (LMAX datacentre)
- requesting 20 levels of depth
- subscribed to ALL LMAX instruments
then it is highly improbable that all those messages will actually make it to the client. On top of this, his business logic will be unable to process the huge number of messages that he does receive so the messages will start to get backed up on the LMAX Exchange side waiting to be sent.
When the client starts to get backed up like this, he sends a TCP ACK that contains his TCP window size. The number contained in this field is an indicator of how much “room” his system has to process more messages. If this value is returned as zero, then his system has no “room” left to process anymore messages and LMAX Exchange has to hold onto the messages until TCP window size !=0. This cycle leads to a brutal pattern of playing catch-up with the latest message which only exacerbates the problem. Clients having this issue will have a very poor experience.
For every second you are double your capacity, it will take 4 times (seconds) as long to recover.
Recently, our Systems Projects team took it upon themselves to empathise with me and relive me of some of this very manual process of trolling through logs. They have built a front end that calculates the FIX logs timestamp differential and extracts the corresponding TCP log for window size.
This new shiny GUI now provides my team with a 6hr snapshot of all our Market Data FIX connections. This can be further drilled down into individual users showing details like their Avg MBits per second, Max Mbits per second, Avg Outbound Latency, Max Outbound Latency, Max TCP Packets per second, as well as many more details.
Now that we have this new and shiny toy, we are much better positioned to help our clients by making sure that we optimize the message rate they receive from us. By reducing their subscription level to something more tolerable, like top of book only, and by helping them with message rate optimisation, we can significantly reduce the chances of dropped or missed messages while still delivering the most current update to them.