LMAX Exchange Getting Up To 50% Improvement in Latency From Azul’s Zing JVM

InfoQ, May 2013 – Developers at the LMAX Exchange, an execution venue for trading FX established in the City of London in October 2010, have begun testing Azul’s Zing JVM as a way of improving their already impressive response times and throughput rates.

LMAX Exchange has become well known, even outside of financial services, at least in part by dint of being willing to talk publicly about technology choices. The firm open-sourced a key component of its software stack, the Disruptor framework, in March 2011 and has spoken at QCon and other conferences about its work.

At the end of last year, LMAX Exchange completed the latest re-build of its production data centre, moving from Hewlett-Packard servers with Intel’s HexaCore Dunnington processor to Dell 520 and 720 with dual socket, 8 core, Intel Sandy Bridge CPUs. The Dells have 64 and 128 GB of memory respectively. “The servers are commodity, with not much specific tuning,” Michael Barker, Head of Software at LMAX Exchange, told InfoQ.

For our performance critical journal based storage we use 15k SAS raid with battery backed write cache. The tests we did showed that a raid array was as fast/faster than SSDs for our workload, which is testament to the quality of the journalling code. We use flash in some other areas.

At the same time as updating the servers the firm switched from CentOS 5 to CentOS 6 as the operating system. “One of the reasons to stay with CentOS (as opposed to a distribution like Fedora Core) is that it is a supported platform for Azul Zing,” Barker told us. Currently though, LMAX Exchange is using Oracle’s HotSpot JVM in production.

The servers are running in an Equinix data centre west of London, which is where a lot of other financial service firms have their servers and applications, so the latency hop between the systems is comparatively small. The firm also has a DR data centre in London, and a third site at a hosting centre in north London which holds the test and development servers along with a Jenkins based Continuous Integration system using the OpenStack private cloud. “We also make extensive use of Cloud services from Amazon and Rackspace,” Baker told us, “for those systems that don’t hold customer data and are not latency sensitive.”

The firm handles an average load of 1,000-2,000 orders per second with peaks of over 5,000. On rare occasions it has gone to over 20,000 orders/second. “We test using our peak load as the sustained baseline, and regularly run tests to find the point at which the system breaks (so called ‘Red Line’ tests).”

LMAX Exchange publish data on execution performance; order latency is 1.5ms on average, whilst the trade latency is less than 3ms end to end, including real time pre-trade risk control.

Whilst these figures are impressive, the variations, caused primarily by stop-the-world pauses in the CMS collector that is part of HotSpot, are becoming a significant problem. LMAX Exchange tried upgrading to the CMS version in JDK 7, but encountered around a 20% increase in the length of GC pauses for the same work load. The reasons for this weren’t entirely clear, but Barker suggested it was probably down to a need to re-tune the collector. That Zing’s collector (C4) typically requires little or no tuning was a major selling point for LMAX Exchange.

I think that we really needed to do retuning of our GC setting and investigating whether JDK 7 specific options like -XX:+UseCondCardMark and -XX:+UseNUMA should be applied. One of the other big reasons to go with Azul is the reduced need to tune the collector. The general recommendation is that you should re-tune from scratch on each new version of the JDK, which sounds fine in theory, but can be impractical. Collector tuning in Oracle JDK is essentially walking through a large search space for a result that meets your needs. Experience, knowledge and guess-work can crop significant chunks off that search space, but even then an extensive tuning exercise can take weeks. For example, our full end-to-end performance test takes 1 hour (10 minutes build & deploy, 10 minutes warm-up, 40 minutes testing), so I could reasonably run 8 runs a day. If you consider the number of different collectors (CMS, Parallel, Serial,…) and all of their associated options (new and old sizes, survivor spaces, survivor ratios,…) how many runs do I need to do to get effective coverage of that search space: 20, 30, more? With Zing the defaults work significantly better than a finely tuned Oracle JDK. We still have some investigation over whether we can get a bit more out of the Zing VM through tuning (e.g. fewer collector threads as our allocation rate is relatively low). However, tuning Zing is just that, i.e. looking to eke out the very best from the system; compared to the Oracle JDK where tuning from the defaults can be the difference between usable and unusable. The effort involving in tuning does come with an opportunity cost. I would much rather have the developers that would typically be involved with GC tuning (they are probably the ones that have the best working knowledge of software performance) be focusing on improving the performance of other areas of the system.

Another option would be to look at two newer collectors – IBM’s Balanced GC and Oracle’s G1. Though the two were developed independently they appear remarkably similar, and aim to provide consistent low pauses over time. They use incremental compaction, a technique which assumes that some regions of memory are more popular than others, allowing the collector to compact a single region at a time and only scan the regions pointing into it when remapping all potential references. This hypothesis may be true for a majority of applications, but it isn’t universally true. Moreover, whilst both collectors have a mostly concurrent marker, and a mostly incremental compaction for the old generation, they also have a fall-back to a stop-the-world collector. Finally, in common with every other commercial collector apart from Azul’s C4, they use a non-concurrent, stop-the-world collector for the young generation. By contrast, Azul’s C4 uses the same fully concurrent, compacting algorithm across both the young and old generation with no fall-back. A previous InfoQ article explored the algorithm in some detail. Given this, LMAX Exchange didn’t try either G1 or Balanced GC. Barker told us, “Both still suffer from the same fundamental problem that CMS does, in that whilst they can delay potential pauses they don’t eradicate them completely.” Therefore

…of the GC implementations available on the market, the only other options that look [feasible] are IBM’s Metronome collector and JRockit’s Real Time collector. Metronome is quite interesting and I would have liked to spend some more time looking into it, however I ran into a couple of issues and I wasn’t able to test it within the time window I had. In IBM’s defence – we were pushing quite hard into some corners of the JDK that aren’t used by most applications and my report got in front the right people and was fixed quickly. With JRockit Real Time, unfortunately Oracle are not continuing on with its development and it won’t be moving beyond version JDK 1.6, which felt like a dead-end as a technology choice. Azul Zing was the one that worked and it showed positive results, that and the team over at Azul were really keen to work closely with us to improve the product for our use case, which sold me on the product.

The results LMAX Exchange are seeing are remarkable: a 10-20% improvement in the mean latency, increasing to around a 50% improvement at the 99th percentile. Moreover

At the max/99.99th percentile with HotSpot the number would jump all over the place so it is hard to produce a relative comparison, except to say that the Zing values are much more stable. A bad run with HotSpot could easily be an order of magnitude worse.

In terms of throughput

Zing gives us the ability to lift what we call our “red-line” – the throughput value at which the latency starts to drop off a cliff. This effect often manifests as a second order effect of GC pauses. If we get a stall that is sufficiently long, we will start to drop packets. The process of packet redelivery can create back-pressure throughout the rest of the system sending client latencies through the roof. Having a more efficient collector with very short predictable pauses should allow us to increase our “red-line”.

Barker points out that there are plenty of other problems which can contribute to variance in performance – bad code, OS jitter, lock contention, and lack of bandwidth between LMAX Exchange and its customers being some examples. “However, GC was the at the top of the list and as with most performance tuning exercises you start with the biggest offender first.”

Azul has seen good take-up amongst financial services companies for Zing. Indeed Azul CTO Gil Tene told InfoQ that whilst LMAX Exchange is well known for their technological excellence and leading edge performance, it isn’t the most aggressive low latency customer Azul has in the industry – “We have several that are even larger and more demanding in latencies (e.g. the latency bounds in Equities tend to be more demanding than they are in FX).

“Part of the reason Zing is so attractive to these companies is that it remains the only collector that eliminates stop-the-world pauses from the young generation as well as the old generation. Whilst young generation pauses are shorter, where an application is particularly performance sensitive they still matter. As a result, Tene told us, “All we have to do is point to newgen pauses in other JVMs and say: ‘those too will be gone’.”

…Furthermore, the fact that it can handle multi-GB-per-sec allocations without worsening latencies or pauses, makes it very appealing for developers who have been trying hard not to allocate things because “it hurts”. With Zing, you can use plenty of memory to go fast, instead of trying to avoid using it so that things won’t hurt and jitter.For production use Zing is priced on an annual subscription/server. Unsurprisingly the vendor is reluctant to reveal pricing information, though it is in line with a supported Oracle or IBM JVM.Zing is closed source, but is made available for free to open-source developers, so you can try it out whilst developing code.

Download as a PDF –

Media Contacts

For further information please contact:

LMAX Exchange (London):

Barbara Pozdorovkina T: +44 20 3192 2510 E: [email protected]

About LMAX Exchange

LMAX Exchange Group is a dynamic, visionary and award-winning financial technology company. Recognised as one of the UK’s fastest growing technology firms, LMAX Exchange is leading the transformation of the global FX industry to transparent, fair, precise and consistent execution. Operating one global marketplace for trading FX, metals, indices and commodities, LMAX Exchange delivers open access, transparency and a level playing field to all market participants.

LMAX Exchange Group offers all clients the ability to trade on LMAX Exchange central limit order book, driven by streaming no ‘last look’ limit-order liquidity from top tier banks and non-bank financial institutions. Servicing funds, banks, brokerages, asset managers and proprietary trading firms, LMAX Exchange offers an anonymous, regulated, rules-based trading environment, order execution in strict price/time priority, and access to real-time streaming market and trade data, enabling all market participants to control execution quality and total trading costs.

Offering a comprehensive range of instruments and ultra-low latency execution, LMAX Exchange operates a global FX exchange infrastructure with matching engines in London, New York and Tokyo.

LMAX Exchange - a unique vision for global FX.

2023

40 Top Innovators in Financial Markets – TabbFORUMmore

LMAX Group CEO, David Mercer has been named, for the second year running in the TabbFORUM 40 Top Innovators in Financial Markets. This second annual list recognises the achievements and influence of 40 outstanding disruptors and difference-makers having the greatest impact at the intersection of finance and technology. close

Best Execution Venue – Finance Magnates Awardsmore

LMAX Group has been awarded Best Execution Venue for 2023 by Finance Magnates at its London Summit Awards ceremony. The awards spotlight and garner attention for leading participants that have excelled across key segments, and are voted by industry peers.close

Best Cryptocurrency Exchange – Finance Magnates Awardsmore

LMAX Digital has been awarded Best Cryptocurrency Exchange for 2023 by Finance Magnates at its London Summit Awards ceremony. The awards spotlight and garner attention for leading participants that have excelled across key segments, and are voted by industry peers.close

Best Crypto Trading Platform – Financial News Awardsmore

For the second year running, LMAX Digital has been awarded “Best Crypto Trading Platform” by leading industry publication, Financial News in its Excellence in Trading & Tech Awards. The awards are voted on by a panel of independent judges and senior editorial staff, and celebrate the companies and individuals that stand out for the strength of their businesses in Continental Europe and the UK.close

Best Exchange – Hedgeweek European Digital Assets Awardsmore

LMAX Digital has been awarded Best Exchange at the Hedgeweek European Digital Assets Awards. The awards provide a unique insight into the most highly-regarded and successful fund managers and service providers in the digital assets space. The awards are voted on by participants in the financial services industry.close

2022 - 2021

40 Top Innovators in Financial Markets – TabbFORUMmore

LMAX Group CEO, David Mercer has been named in the TabbFORUM 40 Top Innovators in Financial Markets, recognised as one of the technology-driven innovators, disruptors and difference-makers having the greatest impact in the sector. close

Best Exchange (Innovation) – Hedgeweek European Digital Assets Awardsmore

LMAX Digital has been awarded the Hedgeweek European Digital Assets Awards “Best Exchange (Innovation)”. The awards are granted to the top firms in their categories and announced at their annual awards ceremony.close

Best Crypto Trading Platform – Financial News Awardsmore

LMAX Digital has been awarded the 2022 Financial News Awards “Best Crypto Trading Platform”. The awards are presented to the best-performing providers in their category.close

Best Execution Venue – Finance Magnates Awardsmore

LMAX Group has been awarded the Finance Magnates Award “Best Execution Venue” for 2022. The awards are presented to the best-performing firms in their categories and announced at the annual summit in London.close

Best Execution Venue – Global Forex Awards B2B more

LMAX Group has been awarded the 2022 Global Forex Awards B2B “Best Execution Venue”. The awards are presented to the best-performing providers in their category as voted on by business customers, traders and investors globally.close

Best Execution Venue – Finance Magnates Awardsmore

LMAX Global has been awarded the Finance Magnates Award “Best Execution Venue” for 2021. The awards are granted to the top firms in their categories and announced at the annual London summit.close

2020 - 2019

Best Cryptocurrency Exchange – Finance Magnates Awardsmore

LMAX Digital has been awarded the Finance Magnates Awards “Best Cryptocurrency Exchange”. The awards are granted to the top firms in their categories and announced at the annual London summit.close

Best Institutional Digital Asset Platform – Profit & Loss Readers' Choice Awardsmore

LMAX Digital has been awarded The Profit & Loss Readers' Choice Awards “Institutional Digital Asset Platform”. These awards are voted on by the readers and recognise excellence within the foreign exchange industry throughout the previous year. close

Crossover Award for Best Crypto/Fiat Initiative – Profit & Loss Readers' Choice Awardsmore

LMAX Group has been awarded The Profit & Loss Readers' Choice Awards “Crossover Award for Best Crypto/Fiat Initiative”. These awards are voted on by the readers and recognise excellence within the foreign exchange industry throughout the previous year. close

Best Anonymous Matching Platform – Profit & Loss Reader' Choice Awardsmore

LMAX Group has been awarded The Profit & Loss Readers' Choice Awards “Best Anonymous Matching Platform”. These awards are voted on by the readers and recognise excellence within the foreign exchange industry throughout the previous year.close

Best Execution Venue – Finance Magnates Awardsmore

LMAX Group has been awarded the Forex Magnates Awards “Best Execution Venue”. The awards are granted to the top firms in their categories and announced at the annual London summit.close

2018 - 2017

Best Crypto Solution Provider – Finance Magnates Awardsmore

LMAX Exchange has been awarded the Finance Magnates Awards “Best Institutional Crypto Solution Provider”. The awards are granted to the top firms in their categories and announced at the annual London summit.close

Best Margin Sector Platform – Profit & Loss Readers' Choice Awardsmore

LMAX Group has been awarded The Profit & Loss Readers' Choice Awards “Best Margin Sector Platform” for five consecutive years. These awards are voted on by the readers and recognise excellence within the foreign exchange industry throughout the previous year.close

2016 - 2015

Best Trading & Execution Technology – HFM US Technology Awardsmore

The HFM US Technology awards recognise hedge fund technology providers that have demonstrated exceptional customer service and innovative product development in the past 12 months. close

Best FX Trading Venue (ECN/MTF) – WSL Awardsmore

LMAX Group has been awarded The Fund Technology and WSL Awards “Best FX Trading Venue ­ ECN/MTF”. The award honours excellence among providers to the institutional trading industry. close

Deloitte UK Technology Fast 50 – Deloitte more

Deloitte UK Technology Fast 50 is a ranking of the 50 fastest-growing technology companies in the UK, based on percentage growth of fiscal year revenues over the last four years.close

Deloitte EMEA Technology Fast 500 – Deloittemore

The Deloitte EMEA Technology Fast 500 pre-eminent technology awards program is a ranking of Europe, the Middle East, and Africa's fastest-growing technology companies based on percentage revenue growth over three years.close

Fastest Growing Tech Company in the UK – The Sunday Times Tech Track 100more

LMAX Group has been recognised amongst the UK's fastest growing technology firms for three consecutive years. Tech Track 100, compiled by Fast Track annually and published by the Sunday Times each September, lists Britain's 100 fastest-growing private tech companies based on sales growth over the last three years.close

Recognised as a Tech City Future Fifty company – Tech City UK Future Fiftymore

LMAX Group is part of the Future Fifty programme, focused on accelerating the success of high-growth tech businesses.close

2014 - 2013

Best Infrastructure/Technology Initiative – WSL Institutional Trading Awardsmore

LMAX Group has been awarded The Wall Street Letter Institutional Trading Awards “Best Infrastructure/Technology Initiative” in 2014. WSL awards honour excellence among providers to the institutional trading industry.close

Best Overall Testing Project – The European Software Testing Awardsmore

The European Software Testing Awards (TESTA) recognise excellence and innovation in the software testing and quality assurance across a variety of industries.close

2012 - 2011

Best Trading System – Financial Sector Technology Awardsmore

The FST Awards recognise excellence and innovation in the field of information technology within the UK and EMEA financial services sector.close

Innovative Programming Framework – Oracle Duke's Choice Awardsmore

Oracle's Duke's Choice Awards celebrate extreme innovation in the world of Java technology and are granted to the most innovative projects using the Java platform.close

LMAX Exchange Group is the holding company of LMAX Limited and LMAX Broker Limited.

LMAX Exchange is a trading name of LMAX Limited, which operates a multilateral trading facility, authorised and regulated by the Financial Conduct Authority (firm reference number 509778) and is a company registered in England and Wales (number 6505809).

LMAX Global is a trading name of LMAX Broker Limited which is authorised and regulated by the Financial Conduct Authority (firm reference number 783200) and is a company registered in England and Wales (number 10819525).