Detergent Diary of a Derivative Crypto -Ride #2

Hello everyone, Alex is in touch again and we continue to deal with the internal structure of the crypto -rhin. Today let’s talk about higher matter – namely, about the architecture of trading. There are very few publications on this topic, sometimes there are materials about the principles of “large” exchanges (traditional NASDAQ or Chicago Exchange CME).

Rarely, but it happens that the details of the inner kitchen are given to the public – probably from the latter, it was an idea of ​​the architecture of Lmax Disruptor in 2012. You can also note the rare example-the book of Michael Lewis Flash-Boys (the film adaptation was delayed, although it was originally prepared in 2016), in which a group of high-frequency traders notices non-market methods in the behavior of trading systems and logically comes to the conclusion that if you want to, so that you want it to be so that you want to something worked as it should-do it yourself. So the IEX exchange arose (Investors Exchange), founded in 2012 and positioning itself as a fair exchange with equal conditions (including technical).

But there is much less information about the inside of the crypto -rope. Therefore, if we cannot study in detail how others work, we will create a system for our capabilities and wishes.

They say that the main thing on the exchange is the speed of work and reliability. For a simple person, a trader, this is certainly so. On the technical side, such an philistine assessment turns into dozens and dozens of internal parameters that need to be monitored and minimized (or vice versa, strive to squeeze out the maximum). But everyone converges in one thing – the fast work of trading (or the engine, as they can call it) – this is important.

But the engine for the user, and even for some employees, is a black box where the data fly and transactions come out. Architecturally – it is almost true. My colleagues on the previous project even wrote an interesting mathematical work dedicated to modeling the exchange engine as a final machine (Deterministic state machine).

We adhere to a similar approach. And so-the engine or Trading Engine-this is a group of services (micro-services) launched on a selected server and serving all trading activity according to one tool. It is these moments that are important for us-the engine is not a monolith, not a single code that does everything at once (albeit faster), but is divided into several co-dependent services that does its work.

Experienced developers can say-and why, because one could create one application, launch in different streams (Multi-Threaded Service) and would work faster. I agree in one thing – yes, such a system will work faster (although the degree of this “faster” is far from so obvious). And we just conducted many experiments on this and previous week with the implementation of multi -flow processing. Unfortunately, this path is not the best – it complicates the exchange code and its understanding, greatly complicates future changes and, in general, is generally less reliable.

We went a different way-we divided the functionality of the engine into independent modules-services, united them together using a fast data transfer bus, while leaving the care of their data and their safety behind each module. We called such a dedicated super-server TradingBox.

Think Inside the Box

Inside TradingBox, all modules work through a central broker that stores data (those modules that they make available to others) and sends messages. Messages are a unit of information exchange between modules. The central part of us is a specialized In-Memory base Redis (slightly twisted in the settings for maximum performance) and two types of communication are available to us-the queue of data storage and clean PUB/SUB (when the messages are sent by subscription, but without guarantees).

Immediately I note that yes-this is a little atypical use of Redis, we combine it as a tire of data exchange, and in fact, as a super-bustle and easy replacement of Kafka/Rabbitmq queues and as a light data storage. From the use of highly specialized messages (zeromq/nanomessage), we now deliberately refused so as not to complicate the work, if if such a load is reached, that the architecture will “choke”, the replacement can be implemented without a lot of rewriting. However, Zeromq, at least in previous versions, had its own problems, and the author of the promising Nanomessage, in general, seemed to score to the project (or rather, switched to a new version – NNG).

Module Tradinggateway – This is the main connecting link of the engine with the outside world. He is the only one who knows about the existence of another world around – since it is connected simultaneously to the inner tire of messages and to the external cluster. The exchange itself interacts with the trading module through this single data channel into which the message flow or Orderflow.

Strictly speaking, ORDERFLOW is not a clean stream of orders, there are different classes of messages, but the bottom line is that when accepting each message, the trading system changes its internal state (not even so, condition, one or more modules). Yes, the main driver of the changes are orders or trading applications. But in our case, the derivative exchange, the main ones are not even an application – but the events of the index change (MarkPrice) and messages about the abolition of the order.

The index is probably immediately clear – all the internal calculations are tied to it, each of its “tick”, a change, means not only a change in virtual profits and losses (Unrealized PNL), but also constant reassessments of liquidation levels. But about the abolition of orders is not so easy to understand. In fact, this is also due to the price of marking. The futures trade directly depends on the index, which sets the price corridor and the trader are forced to adapt to it, because you will not have time to react – and your warrant has already taken away a faster and aggressive competitor, the position is open, not even a second has passed, but has already appeared unprofitable … When the oscillation is oscillating … when the fluctuation occurs Prices, immediately the robots of the algotracers are moving in motion-depending on the capabilities of the exchange, they either adjust the previously set applications or remove and make new orders at new prices. Therefore, the ordino is optimized at the speed of not only adding a new application, but also to the rapid cancellation of a large number of orders.

Module Orderbook It just keeps the same book of applications (glass), which the traders in the terminal see. Although inside the service, algorithmically, it is represented in a completely different form than shown on the screen. For the simplicity of all other exchange systems, the Orderbook service periodically (once per second or faster) creates a “picture” (Snapshot) and transcodes it so that it is convenient to show a person. There is no longer so detailed information, only price levels and scope of applications.

Matcher – or the mechanism of comparison of trading orders, an algorithm that determines which application will be reduced with which, what the exact final price of purchase or sale. It does not work as simple as it may seem, but simplified – if the buyer’s price is equal or better than the price that the seller requested – he brings such orders and says – Deal Done! For maximum speed, this module is integrated into ORDERBOOK, and not put into a separate service (although we still think about this solution).

PositionManager – special service managing positions (open contracts). Let me remind you that in the world of derivatives we trading the Perpetual Futures Contract, that is, contracts between two traders, which after opening has no statute of limitations (or expiration). One of the parties puts on falling (Short), the second expects growth (long). For each individual trader – his position is just the number, the number of its contracts and at what price. But the exchange engine cannot work like that, it keeps full and accurate accounting of each position. For example, your order for the purchase of 100 contracts at a price of $ 21340 has been executed. For you, as a trader – you have one position for 100 contracts. And on the side of the exchange-it can be 2 different positions (40 and 60 contracts) with different counterparties, and 100 different positions under the 1st contract! Therefore, the requirements for the speed of work with PositionManager are even higher than that of the trading system itself! In addition, it is necessary to consider UPNL at every change in price for all, all positions separately, to check the liquidation for each. Therefore, even with unhurried auction and several hundred orders, there can be many thousands of positions at the same time.

Liquidator – This is a terrible dream and the main enemy of every derivative trader. And in fact – this is the most unlucky algotreer in the world! It is he who takes on himself and controls all those positions that have exhausted the margin limit and which must be sold (or, in the worst case, close). The liquidator takes on all positions, he has a privilege to aggregate positions in one, collecting one large from small. He also tries to put them to the market to get rid of it as soon as possible. But if everything went out of plan, I have to add the missing amounts from my own funds.

ExecutionenGine – An ordinary simple hard worker. When Matcher decides that the order 1 must be reduced with a suitable oncoming order with the ID, say, 42, he immediately loses interest in what is happening further – he would have managed with his work, taking into account the compressed deadlines (and the time allotted for metaching is no more than dozen millisecond). Therefore, the office clerk – Executionengine comes into the work. He once again checks the possibility of concluding a transaction, forms notifications of a trading account service, calculates the parameters of a new position – he no longer has such a rush for all this, although it is necessary to do as quickly as possible.

Fundingmanager – is responsible for the calculation of payments at the Funding Rate (Funding Rate). Recall – this is the interest rate that the parties to each other pay for each other, depending on whether it is more 0 or less. A bet for balancing positions on the market of the futures perpetual and, in fact, additional stimulation of traders keep the market trend. On our exchange, we plan to take into account thending every minute, which means that we need to quickly calculate for each individual position (and, I recall, there can be a lot, tens of thousands) How specific the money must be transferred to the balance sheet. Despite the fact that mathematics there is quite simple, but there are nuances. On the one hand, in the formula, relatively large numbers-notion amount or conditional position size (excluding leverage), is expressed in thousands, and sometimes tens to hundreds of thousands. On the other hand, the interest rate at a minute interval is very insignificant, hundredths and even thousandths of a percentage. On the third party – individual positions can be under a hundred thousand. Which means that even a very small value for each individual position can turn into a pleasant and significant number of dollars in someone else’s account after multiple addition. So the requirements for the mathematical calculations of the fanding payments are high. And yes, no calculated payment cannot be lost for any position – these payments are accumulated and constantly summed up. At this moment you received a dollar, and already in the next you are fully made in favor of the counterparty from these funds.

And so that life does not seem easy – we should not forget that there are checks of correctness and security. For example, the sum of all fanding payments of one side should exactly coincide with the payment of the other side and, as a result, each calculated period should be reduced to 0. The number of all positions of the shorts should identically coincide with the positions of Long – both in the number of contracts and in the amount of positions. As well as the amount of a blocked margin. And unrealized pnl – even that one second should be strictly 0. Yes, yes, trading perpetual futures is always a game https://gagarin.news/ with a zero sum!

I’ll finish this, thanks for your attention.

P.S. And following the tradition begun in the previous issue, several complex (!) but interesting materials that we read this week.