The previous articles have talked about some of the internal designs of the black box, including the core software structure, data structure, directory structure, etc. From this section, we will go deeper and decipher more design and implementation details inside the black box.

The first step in decrypting the black box is to know what the internal data processing process is. When we want to design a new system, the same is true. The first step is to sort out the business process and data flow. For the matching engine, it is necessary to understand: From input to output, what processing processes have passed through the middle.

As mentioned in the previous article, this matching engine defines three types of inputs: Open matching, process orders, and close matching. Let's take a look at the process behind these three inputs separately.

Opening a matching is to open the matching engine of a certain transaction target (trading pair). The transaction target that has not been matched cannot process the order, and the transaction target that has been matched cannot be opened again. Otherwise, there will be two at the same time. It is unreasonable for the engine to process orders for the same transaction target. Orders for the same transaction target can only be processed serially by one engine.

Why can't it be done in parallel? If orders for the same transaction object can be processed in parallel by multiple engines, at least a few problems will arise:

1.**Which is the transaction price? **Theoretically, there can only be one transaction price at each moment. After parallelization, multiple transaction prices will be generated, and the transaction price will be difficult to determine. 2.**How to maintain a unified entrusted ledger? **Theoretically, each transaction target has an entrustment ledger that saves all orders. After parallelization, how to maintain this unified ledger among multiple engines? If the database is maintained uniformly, it will undoubtedly reduce the matching performance; if it is divided into multiple sub-ledgers, it is difficult to guarantee the principle of price priority and time priority.

Both of the above problems are not easy to solve, so all orders can only be sequenced first and then thrown into the engine for serial processing.

When it comes to sequencing, a sequencing queue is naturally required, so when matching is enabled, the order sequencing queue corresponding to the transaction target needs to be initialized. After initializing the sequencing queue, you can actually start the engine corresponding to the transaction target. In a Go program, the engine of each transaction target runs as an independent goroutine; in other languages, such as Java, it runs as an independent thread.

After the engine is started, the transaction order book needs to be initialized to save the order. Then wait for the ordering queue to have orders taken out one by one for processing.

Also, consider another scenario, what happens when the matchmaking program restarts? Does it need to be restored after restarting the transaction target that has been matched? If necessary, how to restore it? The simplest solution is of course to use a cache, use Redis to cache the transaction targets that have been matched, and then load and reopen these transaction targets from Redis when restarting.

Therefore, there are actually two scenarios for triggering matchmaking. One is triggered by the active call of the interface, and the other is automatically loaded and started from the Redis cache after the program restarts.

Finally, the result of enabling matching is returned synchronously, so it has no asynchronous output.

To sum up, the internal process of opening matchmaking is roughly as follows:

Once matchmaking is enabled, you can receive input for processing orders. When the matching program receives a request to process an order, the first step needs to do some checks, including whether each parameter is valid, whether the order is repeated or exists, whether the engine corresponding to the transaction target has been opened, etc. After passing the check, the entire order can be cached in Redis, and then added to the ordering queue corresponding to the transaction target, waiting for the engine corresponding to the transaction target to consume it for matching processing. The process is as follows:

When the order is successfully added to the sequencing queue, the interface can synchronously return a successful response result. Subsequent processing results are output through asynchronous MQ. After the engine of the transaction target receives the order, it will produce different output results according to different situations.

We know that there are two types of action for processing an order: Place an order and Cancel an order. The business logic of order cancellation is very simple. It is to query whether the order exists in the transaction entrustment ledger. If it exists, delete the order from the entrusted ledger, and then output the order cancellation result of successful order cancellation; if it does not exist, output the order cancellation failure. Cancellation result. The business logic of placing an order is more complicated, and it needs to be processed differently according to different order types. The matchmaker version at the time of writing supports 6 different types, including two limit price types and four market price types. The following will explain the results of placing orders of different order types under different conditions.

•limit: ordinary limit price. When there is an order in the order book that can match the order, one or more transaction records may be generated, and each transaction record will generate an asynchronous output; when there is no matching order in the order book, it will be The order (full quantity or remaining quantity) is added to the order book, at which point no output is produced. •limit-ioc: IOC limit price - immediate transaction remaining cancellation. When there is an order in the order book that can match the order, one or more transaction records may be generated, and each transaction record will generate an asynchronous output; when there is no matching order in the order book, it will be The order (all or the remaining quantity) is processed for cancellation, and an output of successful cancellation will be generated at this time. •market: Default market price - immediate transaction remaining cancellation. Like the IOC price limit, when there is an order in the order queue in the opposite direction of the order (also called the counterparty) in the order book, one or more transaction records may be generated, and each transaction record will generate an asynchronous output; When the counterparty does not have an order in the order book, the order (all or the remaining quantity) will be cancelled, and an order cancellation success will be output. The difference from the IOC limit order is that the IOC limit order is specified by the user with the order price, while the market price does not need to specify the order price, and will be directly filled with the counterparty's head order until the order has been fully filled or the counterparty Until there are no more orders. •market-top5: market price - the best five levels of immediate transaction remaining cancellation. The market can be filled with orders of all price levels of the counterparty, but market-top5 can only be filled with orders within five price levels of the counterparty at most, and orders beyond the five levels will not be filled. The remaining unsettled orders will be cancelled and an output of successful cancellation will be generated. •market-top10: market price - the best ten levels of instant transaction remaining cancellation. At most, it will only trade with orders within ten price levels of the counterparty. •market-opponent: market price - the best price of the counterparty. If the counterparty does not have an order, the order will be cancelled directly and an output of successful cancellation will be generated; if the counterparty has an order, it will only be traded one level at most. The price of the counterparty's first tranche is converted into a limit order and added to the order book, and no output will be generated at this time.

In addition, every request to process an order—whether it’s placing an order or canceling an order—will also be cached in Redis, and the cache will be updated when changes are made. In this way, the order can be resumed after the program restarts.

When a transaction target is ready to be delisted, or the transaction is cancelled, or the transaction is suspended, the engine needs to be shut down. Before shutting down the engine, it is best for upstream services to stop calling the interface for processing orders, otherwise some unexpected errors may occur, although the program has already done fault-tolerant processing.

When closing the engine, there are also some simple judgments, such as judging whether the engine of the transaction target has been opened, and the engine that is not opened naturally cannot be closed.

When shutting down the engine, if there are still unprocessed orders in the sequencing queue, it should wait for these orders to be processed before actually shutting down the engine.

Finally, clear the cache as well, and clear all orders for the subject of the transaction from the cache.

The result of shutting down the engine is also returned synchronously, and there is no asynchronous output.

This section explains the core business process inside the matching black box, including the internal logic of the three inputs of opening matching, processing orders, and closing matching. After understanding these processes, we will start to talk about the code implementation in the next article.

It is customary to leave a few thinking questions: if there are concurrent requests for placing orders while the matching is closed, is it easy to cause problems? If so, where will it be generated? what is the problem? How can it be solved?