Multi-dimensional Strong-enumeration of JSON Messages with Rust

Often among streaming API calls we encounter structures like the one shown below. There may be many message variants, defining scopes across multiple dimensions such as aspect of the system (topic) and purpose of the changes (event):

// Snapshot of all markets
{
  "topic": "market",
  "event": "snapshot",
  "markets": [
    {
      "ticker": "BTC/USD",
      "price": 1000000,
      "volume": 1000
    },
    {
      "ticker": "ETH/USD",
      "price": 10000,
      "volume": 10000
    }
  ]
}

// Snapshot of user portfolio
{
  "topic": "portfolio",
  "event": "snapshot",
  "open_orders": [
    {
      "id": 0,
      "side": "sell",
      "ticker": "BTC/USD",
      "limit_price": 1100000
    }
  ]
}

// Individual updates to markets
{
  "topic": "market",
  "event": "update",
  "markets": [
    {
      "ticker": "BTC/USD",
      "price": 1000001,
      "volume": 1001
    }
  ]
}

// Individual updates to user portfolio
{
  "topic": "portfolio",
  "event": "update",
  "open_orders": [
    {
      "id": 1,
      "side": "buy",
      "ticker": "ETH/USD",
      "limit_price": 9000
    }
  ]
}

In message processor we want better readability and maintenance. Strong-typing allow us to catch potential bugs early at the build/compile stage, and codes written that way are usually more self-descriptive. The question is, how do we model the messages above to achieve such goals?

As the message processor, I want each type of the messages to have a clear and easy-to-follow code path. Something like this would be nice at the first glance:

while let Some(message) = message_stream.next() {
    match message {
        Ok(MarketSnapshot(data)) => // Handle market snapshot
        Ok(MarketUpdate(data)) => // Handle market update
        Ok(PortfolioSnapshot(data)) => // Handle portfolio snapshot
        Ok(PortfolioUpdate(data)) => // Handle portfolio update
        Err(reason) => // Handle error
        _ => {} // Ignore other messages (if any)
    }
}


Using pattern matching allow us to write concise codes and avoid layers of messy if…else syntax (especially when the message has complex structures). Moreover, pattern matching is exhaustive-enforced by the compiler so it eliminates unattended edge cases.

Going along with the initial idea, we define an enum with four variants, each representing a message:

struct MarketData {
  // ...
}

struct PortfolioData {
  // ...
}

enum Message {
    MarketSnapshot(MarketData),
    MarketUpdate(MarketData),
    PortfolioSnapshot(PortfolioData),
    PortfolioUpdate(PortfolioData),
}

The problem is that it wouldn’t map easily to the JSON schema above, because the message variants are determined by two fields: topic and event, which are parts of the JSON messages and we need extra logic for translation.

We can use Serde, the popular framework for data serialization on Rust, to do such translation by “tagging” enum representation internally, so that:

#[derive(Serialize, Deserialize)]
#[serde(tag = "topic", rename_all = "camelCase")]
enum Message {
    Market(MarketData),
    ...
}

would serialize to (and vise versa):

{
  "topic": "market", // i.e. camelCase of the enum variant "Market" thanks to the `rename_all` attribute
  ... // market data
}

But that’s not all! We have two fields to tag: topic and event, and both of them are at the root level of the JSON message. Unfortunately, Serde does not support multiple tags on one enum, so we have to do it by nesting data structures:

#[derive(Serialize, Deserialize)]
#[serde(tag = "topic", rename_all = "camelCase")]
enum MessageTopic {
    Market(MessageEvent<MarketData>),
    Portfolio(MessageEvent<PortfolioData>),
}

#[derive(Serialize, Deserialize)]
#[serde(tag = "event", rename_all = "camelCase")]
enum MessageEvent<T> {
    Snapshot(T),
    Update(T)
}

By having an enum inside another enum, we are able to tag each of them separately and still translate both at the root level of the JSON message. It would serialize to:

{
  "topic": "market",
  "event": "snapshot",
  ... // market data
}

With that, we can now write the processor like this:

while let Some(message) = message_stream.next() {
    match message {
        Ok(MessageTopic::Market(MessageEvent::Snapshot(market_data))) => // Handle market snapshot
        Ok(MessageTopic::Market(MessageEvent::Update(market_data))) => // Handle market update
        Ok(MessageTopic::Portfolio(MessageEvent::Snapshot(portfolio_data))) => // Handle portfolio snapshot
        Ok(MessageTopic::Portfolio(MessageEvent::Update(portfolio_data))) => // Handle portfolio update
        Err(reason) => // Handle error
        _ => {} // Ignore other messages (if any)
    }
}

Check here for a runnable example.