// Snapshot of all markets
{
  "topic": "market",
  "event": "snapshot",
  "markets": [
    {
      "ticker": "BTC/USD",
      "price": 1000000,
      "volume": 1000
    },
    {
      "ticker": "ETH/USD",
      "price": 10000,
      "volume": 10000
    }
  ]
}

// Snapshot of user portfolio
{
  "topic": "portfolio",
  "event": "snapshot",
  "open_orders": [
    {
      "id": 0,
      "side": "sell",
      "ticker": "BTC/USD",
      "limit_price": 1100000
    }
  ]
}

// Individual updates to markets
{
  "topic": "market",
  "event": "update",
  "markets": [
    {
      "ticker": "BTC/USD",
      "price": 1000001,
      "volume": 1001
    }
  ]
}

// Individual updates to user portfolio
{
  "topic": "portfolio",
  "event": "update",
  "open_orders": [
    {
      "id": 1,
      "side": "buy",
      "ticker": "ETH/USD",
      "limit_price": 9000
    }
  ]
}

In message processor we want better readability and maintenance. Strong-typing allow us to catch potential bugs early at the build/compile stage, and codes written that way are usually more self-descriptive. The question is, how do we model the messages above to achieve such goals?

As the message processor, I want each type of the messages to have a clear and easy-to-follow code path. Something like this would be nice at the first glance:

while let Some(message) = message_stream.next() {
    match message {
        Ok(MarketSnapshot(data)) => // Handle market snapshot
        Ok(MarketUpdate(data)) => // Handle market update
        Ok(PortfolioSnapshot(data)) => // Handle portfolio snapshot
        Ok(PortfolioUpdate(data)) => // Handle portfolio update
        Err(reason) => // Handle error
        _ => {} // Ignore other messages (if any)
    }
}

Using pattern matching allow us to write concise codes and avoid layers of messy if…else syntax (especially when the message has complex structures). Moreover, pattern matching is exhaustive-enforced by the compiler so it eliminates unattended edge cases.

Going along with the initial idea, we define an enum with four variants, each representing a message:

struct MarketData {
  // ...
}

struct PortfolioData {
  // ...
}

enum Message {
    MarketSnapshot(MarketData),
    MarketUpdate(MarketData),
    PortfolioSnapshot(PortfolioData),
    PortfolioUpdate(PortfolioData),
}

The problem is that it wouldn’t map easily to the JSON schema above, because the message variants are determined by two fields: topic and event, which are parts of the JSON messages and we need extra logic for translation.

We can use Serde, the popular framework for data serialization on Rust, to do such translation by “tagging” enum representation internally, so that:

#[derive(Serialize, Deserialize)]
#[serde(tag = "topic", rename_all = "camelCase")]
enum Message {
    Market(MarketData),
    ...
}

would serialize to (and vise versa):

{
  "topic": "market", // i.e. camelCase of the enum variant "Market" thanks to the `rename_all` attribute
  ... // market data
}

But that’s not all! We have two fields to tag: topic and event, and both of them are at the root level of the JSON message. Unfortunately, Serde does not support multiple tags on one enum, so we have to do it by nesting data structures:

#[derive(Serialize, Deserialize)]
#[serde(tag = "topic", rename_all = "camelCase")]
enum MessageTopic {
    Market(MessageEvent<MarketData>),
    Portfolio(MessageEvent<PortfolioData>),
}

#[derive(Serialize, Deserialize)]
#[serde(tag = "event", rename_all = "camelCase")]
enum MessageEvent<T> {
    Snapshot(T),
    Update(T)
}

By having an enum inside another enum, we are able to tag each of them separately and still translate both at the root level of the JSON message. It would serialize to:

{
  "topic": "market",
  "event": "snapshot",
  ... // market data
}

With that, we can now write the processor like this:

while let Some(message) = message_stream.next() {
    match message {
        Ok(MessageTopic::Market(MessageEvent::Snapshot(market_data))) => // Handle market snapshot
        Ok(MessageTopic::Market(MessageEvent::Update(market_data))) => // Handle market update
        Ok(MessageTopic::Portfolio(MessageEvent::Snapshot(portfolio_data))) => // Handle portfolio snapshot
        Ok(MessageTopic::Portfolio(MessageEvent::Update(portfolio_data))) => // Handle portfolio update
        Err(reason) => // Handle error
        _ => {} // Ignore other messages (if any)
    }
}

Check here for a runnable example.

After hours of debugging, it turns out to be the term !CAMELLIA (i.e. don’t use Camellia-based algorithm) in node’s default cipher list, probably introduced after v18. It wasn’t clear to me why the remote endpoint requires such algorithm, but as soon as I remove the term everything was working again.

const agent = new https.Agent({
    // Note it is almost the same as node v20's default cipher list except without `!CAMELLIA`
    ciphers: 'TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!SRP'
})

// Override the http.agent with ours
axios.post("https://the-picky-endpoint.com", {}, { httpsAgent: agent })

Finally! Onwards!

2024-06-24 19:51:37.478+00:00 | vert.x-worker-thread-18 | TRACE | AbstractJsonRpcExecutor | {"jsonrpc":"2.0","id":22,"method":"eth_maxPriorityFeePerGas"}

2024-06-24 19:51:37.478+00:00 | vert.x-worker-thread-18 | DEBUG | JsonRpcExecutor | JSON-RPC request -> eth_maxPriorityFeePerGas null

2024-06-24 19:51:37.479+00:00 | vert.x-worker-thread-18 | TRACE | AbstractJsonRpcExecutor | {"jsonrpc":"2.0","id":22,"error":{"code":-32601,"message":"Method not found"}}

Given I don’t want to switch back to geth, and it costs too much to switch to a cloud infra provider, also I don’t want to hack the app’s Optimism dependency. What else can I do?

Due to the geth-specific RPC eth_maxPriorityFeePerGas is sparsely called. I could potentially redirect only those particular calls to a cloud infra provider and therefore keep the usage low under free tier, then send everything else to my own full node. For that I need a reverse proxy with conditional routing, which can be done with Caddy.

Prerequisites

Assume our system already have Go installed.

To install Caddy:

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

To install xcaddy (development tools):

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/xcaddy/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-xcaddy-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/xcaddy/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-xcaddy.list
sudo apt update
sudo apt install xcaddy

Implementation

Our goal is to run a reverse proxy at localhost:8545 that redirects ETH RPC calls to either of the two upstreams: (1) eth_maxPriorityFeePerGas to a cloud infra provider (assume Alchemy Ethereum mainnet), and (2) all other calls to our full node (assume running at local network 192.168.1.123:8545).

Unlike regular reverse proxies, we need to read the request body in order to determine the route. Such feature does not come built-in so we will build our own caddy with a custom plugin to match request body payloads.

In addition, reading request body may come with performance penalty given payload size is unbounded. Luckily, since we know the exact payload to look for, we can hard-code a size cap to avoid unnecessary loads.

Create and initialize the project folder:

mkdir -p /path/to/my/caddy-reqbody-matcher
cd /path/to/my/caddy-reqbody-matcher
go mod init github.com/Detoo/caddy-reqbody-matcher

Create the request body matcher plugin: reqbody-matcher.go with the following codes:

package keywordmatcher

import (
    "bytes"
    "io/ioutil"
    "net/http"
    "unicode/utf8"
    
    "github.com/caddyserver/caddy/v2"
    "github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
)

// KeywordMatcher implements caddyhttp.MatchInterface
type KeywordMatcher struct {
    Keyword string `json:"keyword,omitempty"`
}

// Match implements caddyhttp.MatchInterface
func (m KeywordMatcher) Match(r *http.Request) bool {
    // Read the request body
    body, err := ioutil.ReadAll(r.Body)
    if err != nil {
        return false
    }
    
    // Restore the io.ReadCloser to its original state
    r.Body = ioutil.NopCloser(bytes.NewReader(body))
    
    // Check if the body is a string and within the character limit
    if utf8.Valid(body) && len(body) < 1000 {
        // Check if the keyword exists in the request body
        return bytes.Contains(body, []byte(m.Keyword))
    }
    
    return false
}

// CaddyModule returns the Caddy module information.
func (KeywordMatcher) CaddyModule() caddy.ModuleInfo {
    return caddy.ModuleInfo{
        ID:  "http.matchers.reqbody_keyword",
        New: func() caddy.Module { return new(KeywordMatcher) },
    }
}

// UnmarshalCaddyfile implements caddyfile.Unmarshaler.
func (m *KeywordMatcher) UnmarshalCaddyfile(d *caddyfile.Dispenser) error {
    d.Next() // consume directive name
    
    // require an argument
    if !d.NextArg() {
        return d.ArgErr()
    }
    
    // store the argument
    m.Keyword = d.Val()
    return nil
}

func init() {
    // Register your matcher with Caddy
    caddy.RegisterModule(KeywordMatcher{})
}

Build caddy executable with the custom plugin. If everything compiles, we will find the artifact under the same folder.

xcaddy build --with github.com/Detoo/caddy-reqbody-matcher=/path/to/my/caddy-reqbody-matcher

Configure the reverse proxy and our custom plugin by creating the config file Caddyfile:

# Reverse proxy for ETH JSON RPC with geth backup
http://localhost:8545 {
    # Match geth-specific requests
    @reqbodyMatcher {
        reqbody_keyword "eth_maxPriorityFeePerGas"
    }

    # Default to our own node
    reverse_proxy 192.168.1.123:8545

    # Redirect geth-specific requests
    reverse_proxy @reqbodyMatcher {
        rewrite /v2/<your-api-key>{uri}
        to "https://eth-mainnet.g.alchemy.com"
        header_up Host eth-mainnet.g.alchemy.com
    }
}

Install it as a systemctl service so it runs 24/7. Create a new file caddy-reqbody-matcher.service with the following contents:

# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy-reqbody-matcher
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=detoo
Group=detoo
WorkingDirectory=/path/to/my/caddy-reqbody-matcher
ExecStart=/path/to/my/caddy-reqbody-matcher/caddy run --environ
ExecReload=/path/to/my/caddy-reqbody-matcher/caddy reload --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

Install and start the service:

sudo systemctl enable /path/to/my/caddy-reqbody-matcher/caddy-reqbody-matcher.service
sudo systemctl restart caddy-reqbody-matcher

Now the reverse proxy should be running at localhost:8545, to verify:

# This should go to our own full node
curl --header 'Content-Type: application/json' --data-raw '{"jsonrpc":"2.0","id":22,"method":"eth_blockNumber"}' http://localhost:8545

# This should go to cloud infra provider
curl --header 'Content-Type: application/json' --data-raw '{"jsonrpc":"2.0","id":22,"method":"eth_maxPriorityFeePerGas"}' http://localhost:8545

Now we have a proper backend to handle all RPCs!

As an Ethereum PoS Validator, we want to stay online as long as we can. We could do as much to build a reliable node that keeps itself available, but one thing we don’t have much control of is the internet providers.

Unfortunately, where I setup the node I have only Comcast, which is the nation’s most beloved provider and is famous for its highly reliable uptime as well as its hassle-free cancellation process (kek).

But no High Availability is too high, that’s why I have also subscribed to Starlink. Because hey, sending blocks to the space! How cool is that?

Challenges

The idea is to hot switch providers whenever it fails to reach the internet. Sounds simple, but it turns out to be not as straightforward as I thought.

For one, simply having both interfaces online is not enough. As when one fails, it usually fails at WAN level instead of LAN. From the node OS (ex. Ubuntu)’s point of view the connection is still on, so it will not switch to the other interface and will keep sending the packets through the failing one.

It seems failover on internet/DNS accessibility requires higher layer logic than a network manager would normally support. I also tried netplan bonding but it seemed to get stuck for the same reason.

Homemade Solutions

Although the network manager alone might not be suitable for our need, it is good at setting interface priority. We could use it in combination of writing ourselves a simple cronjob to check the internet connections periodically, and then tell the network manager to switch interface accordingly.

The following example assumes two internet-facing interfaces: eno1, which is ethernet-based; and wlp3s0, which is WiFi-based. Both of them are connected to regular consumer routers respectively with DHCP support.

Again we use netplan as the network manager. First create the following config files (they are separated into multiple files for easier management):

• /etc/netplan/00-installer-config-eno1.yaml: basic configs for eno1

  network:
    version: 2
    ethernets:
      eno1:
        dhcp4: true
  

• /etc/netplan/01-installer-config-wlp3s0.yaml: basic configs for wlp3s0 (remember to update the access-point name and password)

  network:
    version: 2
    wifis:
      wlp3s0:
        access-points:
          YourWiFiName:
            password: YourWiFiPassword
        dhcp4: true
  

• /etc/netplan/02-prioritize-eno1.yaml: configs for prioritizing eno1

  network:
    version: 2
    ethernets:
      eno1:
        dhcp4-overrides:
          route-metric: 100 # lower number = higher priority
    wifis:
      wlp3s0:
        dhcp4-overrides:
          route-metric: 200
  

• /etc/netplan/02-prioritize-wlp3s0.yaml.disabled: configs for prioritizing wlp3s0. Note the postfix “disabled” because we use eno1 by default

  network:
    version: 2
    ethernets:
      eno1:
        dhcp4-overrides:
          route-metric: 200
    wifis:
      wlp3s0:
        dhcp4-overrides:
          route-metric: 100 # lower number = higher priority
  

The first two files are basic configs that apply all the time. The last two files are mutually-exclusive and there should be only one enabled at any given time. This way we can tell the network manager to use either eno1 or wlp3s0, whichever is available. And then we run a cronjob to perform the check periodically.

Create the script for cronjob at ~/bin/internet-failover/internet-failover.sh or other places of your choice:

#!/bin/bash

# choose a realiable URL or IP to check against.
# we use URL here to also check against DNS failures
CHECK_URL="www.yahoo.com"

# remember to update the interface IDs if necessary
IF0="eno1"
IF1="wlp3s0"

PRI_PLAN_PATH_PREFIX="/etc/netplan/02-prioritize-"
PRI_PLAN_PATH_POSTFIX=".yaml"

# determine the current primary interface by finding the plan files
if [ -f "$PRI_PLAN_PATH_PREFIX$IF0$PRI_PLAN_PATH_POSTFIX" ]; then
    echo "current primary plan: \$IF0"
    CUR_IF="$IF0"
    NEXT_IF="$IF1"
else
    echo "current primary plan: \$IF1"
    CUR_IF="$IF1"
    NEXT_IF="$IF0"
fi

echo "checking if internet is available..."
ping -I $CUR_IF -q -c 2 $CHECK_URL > /dev/null 2>&1

if [ $? -ne 0]; then
    echo "[WARN] internet not available"
    echo "switching primary plan to: \$NEXT_IF ..."

    mv "$PRI_PLAN_PATH_PREFIX$CUR_IF$PRI_PLAN_PATH_POSTFIX" "$PRI_PLAN_PATH_PREFIX$CUR_IF$PRI_PLAN_PATH_POSTFIX.disabled"
    mv "$PRI_PLAN_PATH_PREFIX$NEXT_IF$PRI_PLAN_PATH_POSTFIX".disabled "$PRI_PLAN_PATH_PREFIX$NEXT_IF$PRI_PLAN_PATH_POSTFIX"
    /usr/sbin/netplan apply
fi

Create the cronjob and set it to run every minute:

sudo crontab -e

# add the following line (remember to update the path to the script if necessary):
*/1 * * * * /home/username/bin/internet-failover/internet-failover.sh 2>&1 | logger -t internet-failover

# save it

# check execution
tail -f /var/log/syslog | grep internet-failover

# after a minute or so we should see the logs
CRON: (root) CMD (/home/username/bin/internet-failover/internet-failover.sh 2>&1 | logger -t internet-failover)
internet-failover: current primary plan: eno1
internet-failover: checking if internet is available...

If we are feeling lucky, try to disable the internet for eno1 and we will see the cronjob detects it and switches the provider:

CRON: (root) CMD (/home/username/bin/internet-failover/internet-failover.sh 2>&1 | logger -t internet-failover)
internet-failover: current primary plan: eno1
internet-failover: checking if internet is available...
internet-failover: [WARN] internet not available
internet-failover: switching primary plan to: wlp3s0 ...

Stake long and prosper!

Merge™ is coming. Most of us have probably already heard of the renaming and the introduction of Execution Client (ex. Geth) and the Consensus Client (ex. Lighthouse Beacon Node). It also introduced a new interface called Engine API. As a pre-merge validator, what exactly has changed? How do I upgrade it and verify if it is ready for merge?

Below assumes we have existing (pre-merge) nodes running and will focus only on the changes.

Execution Client

There are multiple options for running an Execution Client. I’ll focus on geth here.

The current merge-ready geth version is v1.10.23 as of writing. Upgrading from a pre-merge geth version you need to add the following arguments:

--authrpc.addr=localhost
--authrpc.port=8551

Note if you are running Beacon nodes or Validator nodes in a container, you might want to add the extra arguments so that incoming requests from virtual hosts are accepted:

--authrpc.vhosts="*"

Restart geth, if everything runs properly you will see:

WARN Chain pre-merge, sync via PoW (ensure beacon client is ready)
...
WARN Engine API enabled                       protocol=eth
WARN Engine API started but chain not configured for merge yet
...
INFO Generated JWT secret                     path=/home/you/.ethereum/geth/jwtsecret
...
INFO WebSocket enabled                        url=ws://127.0.0.1:8551
INFO HTTP server started                      endpoint=127.0.0.1:8551 auth=true  prefix= cors=localhost vhosts=*

Note the pre-merge status and the generated JWT secret location in the logs.

Consensus Client (Beacon Node)

Again there are multiple options for running a Beacon Node, here I will use Lighthouse.

As of writing, the current merge-ready Lighthouse version is v3.1.0.

Add the following merge-related arguments:

--execution-endpoints http://localhost:8551
--execution-jwt="/home/you/.ethereum/geth/jwtsecret"

Note the port 8551 is different from the legacy RPC port (8545) because it is an exclusive interface for the Beacon Node only. Also note the --execution-jwt location should match the location shown in geth logs above.

Restart the Beacon Node. It will take a few minutes to migrate the database from the previous version. Once it is connected and synced with the Execution Client we’ll see:

INFO Ready for the merge current_difficulty: 57833190587477479896205, terminal_total_difficulty: 58750000000000000000000, service: slot_notifier

That means both clients are ready! The Engine API has started, the TTD (Terminal Total Difficulty) is nominal, we are ready for Merge™

One More Thing

Go to bordel.wtf and check out the estimated Merge time if you want to see it happen live!