Since the merge an ethereum node consists of two layers, the execution layer (EL) and the consensus layer (CL). Each one of these have their own networking stack for peer-to-peer (from now on, p2p) communication.

The CL and EL p2p stacks are similar, but in this post I’ll focus in the CL p2p networking. Some of the terms are common to many peer-to-peer network implementations, but if you’re not familiar with any of them I encourage you to take a look at the libp2p documentation, which is also the main library to use when implementing these specs. Now let’s begin by taking a look at the fundamentals.

The network configuration can be, and possibly is, quite different in every client implementation, so I’ll stick to the specs that all of them supports:

• Transport — All implementations support the TCP libp2p transport enabled for both dialing and listening. They’re capable of dialing both IPv4 and IPv6 addresses and listening is supported on at least one of IPv4 or IPv6.

• Multiplexing — Clients support mplex and may support yamux.

• Encryption and identification — Is based on noise, a libp2p transport secure channel handshake. It supports the XX handshake pattern, which provides mutual authentication and encryption of static keys and handshake payloads and is resistant to replay attacks.

• Protocol negotiation — Clients use exact equality when negotiating protocol versions to use. At present, multistream-select 1.0 is used for negotiation of:

  • Authentication/Encryption Mechanism: Noise, SecIO, TLS 1.3.

  • Overlay Multiplexer: mplex, Yamux, spdystream.

Why is some of this needed?

• TCP is natively incapable of multiplexing and encryption, thus these are applied atop and we need to negotiate on which one to use depending on the different implementations.

• Multiplexing is needed to conduct parallel conversations.

• Transport-level encryption secures the communication (authentication, content eavesdropping, tampered data) but none of the application layer related data.

The gossip domain includes all information that has to spread rapidly throughout the network, such as blocks, proofs, attestations, exits and slashings. For doing this gossipsub is used, a general purpose publish/subscribe protocol.

A pub/sub protocol is based on topics and messages. Peers only receive messages about a topic that they’re subscribed to, and send messages to specific topics.

Clients support gossipsub v1 libp2p protocol including the gossipsub v1.1 extension. However, only the configurable v1 parameters are used to control it’s behavior.

Topics are plain UTF-8 strings (e.g. proposer_slashing) and are encoded on the wire as determined by protobuf (gossipsub messages are wrapped in protobuf messages).

• Global topics: beacon_block - beacon_aggregate_and_proof - voluntary_exit - proposer_slashing - attester_slashing

• Attestation subnets: Are used to propagate unaggregated attestations to subsections of the network. These are defined by the beacon_attestation_{subnet_id} topic.

Topics are also post-fixed with an encoding that define how the payload of a gossipsub message is encoded. All objects are SSZ-encoded and then compressed with snappy.

The message proto structure is:

message Message {
    optional bytes from = 1;
    optional bytes data = 2;
    optional bytes seqno = 3;
    optional string topic = 4;
    optional bytes signature = 5;
    optional bytes key = 6;
}

Where the fields from, seqno, signature and key are omitted. Messages are identified by content, anonymous, and signed where necessary in the application layer.

The default libp2p pubsub message-id is overridden to use content-addressing, allowing to filter unnecessary duplicates before hitting the application layer. It is replaced by either:

• SHA256(MESSAGE_DOMAIN_VALID_SNAPPY + snappy_decompress(message.data))[:20]

• SHA256(MESSAGE_DOMAIN_INVALID_SNAPPY + message.data)[:20]

Depending if the data has a valid snappy decompression.

The request-response domain contains protocols for clients requesting specific information from their peers (e.g. blocks matching certain root hashes or within a range of slots). The responses are always returned as snappy-compressed SSZ encoded bytes.

Each message type is segregated into its own libp2p protocol ID, which is a case-sensitive UTF-8 string of the form:

• /ProtocolPrefix/MessageName/SchemaVersion/Encoding

  • ProtocolPrefix — libp2p protocol name prefix. e.g. /eth2/beacon_chain/req

  • MessageName — Request identifier

  • SchemaVersion — Schema version number

  • Encoding — Encoding strategy

This protocol segregation allows libp2p multistream-select 1.0 / multiselect 2.0 to handle the request type, version, and encoding negotiation before establishing the underlying streams.

Only one stream per request/response interaction is used. Streams are closed when the interaction finishes, whether in success or in error.

Req-res messages have the following structure:

request   ::= <encoding-dependent-header> | <encoded-payload>
response  ::= <response_chunk>*
response_chunk  ::= <result> | <encoding-dependent-header> | <encoded-payload>
result    ::= “0” | “1” | “2” | [“128” ... ”255”]

• encoding-dependent-header carry metadata or assertions such as the encoded payload length, for integrity and attack proofing purposes.

• response is formed by zero or more response_chunks.

• result is a single byte that details the result of the request

  • 0x0 - Success

  • 0x1 - InvalidRequest

  • 0x2 - ServerError

  • 0x3 - ResourceUnavailable

Only one strategy is allowed at this time:

• ssz_snappy: The contents are first ssz-encoded and then compressed with snappy.

Let’s take a look at the Status message, which is sent upon connection:

• Protocol ID: /eth2/beacon_chain/req/status/1/

• Request/Response Content:

(
  fork_digest: ForkDigest
  finalized_root: Root
  finalized_epoch: Epoch
  head_root: Root
  head_slot: Slot
)

• Where:

  • fork_digest: 4-byte fork digest for the current_version and genesis_validators_root.

  • finalized_root: state.finalized_checkpoint.root of the current head block.

  • finalized_epoch: state.finalized_checkpoint.epoch of the current head block.

  • head_root: hash_tree_root root of the current head block.

  • head_slot: The slot of the block corresponding to the head_root.

Other messages are:

• Goodbye

• BeaconBlocksByRange

• BeaconBlocksByRoot

• Ping

• GetMetaData

Before a peer can subscribe to a topic or interact with another network participant it must find other peers and establish network connections with them. CL clients use the Node Discovery Protocol v5 (discv5).

To facilitate peer discovery and content routing a distributed hash table (DHT) is used. A regular hash table is a set of records in which every record holds a key -> value mapping. In a distributed hash table, the records are spread across the network, and every node holds a subset of them.

Discv5 is inspired by the Kademlia DHT, but in this case the DHT stores and relays 'node records', which are signed documents that provide information about nodes in the network. It acts as a database of all live nodes in the network and performs three functions:

• Sampling the set of all live participants.

• Searching for participants providing a certain service (topic advertisements).

• Authoritative resolution of node records with a node’s ID.

It runs on UDP on a dedicated port that it is meant for peer discovery only and supports self-certified, flexible peer records (ENRs).

The CL discv5 implementation differs from that of the EL only in that it includes an adaptor to connect it with libp2p , deprecating devP2P.

• Operation inputs include peer IDs or capabilities (such as topics).

• Outputs are multiaddress converted from the ENRs returned by the discv5 backend.

This integration enables the libp2p stack to subsequently form connections and streams with discovered peers.

The ethereum node record, as defined by the EIP-778, is a flexible format for connectivity-related information. It improves cryptographic agility and the handling of protocol upgrades. A record can contain information about arbitrary transport protocols and public key material associated with them.

The required entries for CL clients are:

• signature — cryptographic signature of record contents

• seq — Sequence number (uint64). Nodes increase the number whenever the record changes and republish the record.

• id — Identity scheme, e.g. v4

• secp256k1 — Compressed secp256k1 public key - 33 bytes.

• attnets — SSZ of Bitvector[ATTESTATION_SUBNET_COUNT]

• eth2 — SSZ of ENRForkID (fork_digest, next_fork_version,next_fork_epoch)

But some nodes also include:

• ip — IPv4 address

• ip6 — IPv6 address

• tcp — TCP local libp2p listening port.

• udp — UDP local discv5 listening port.

The textual form of a node record is the base64 encoding of its RLP encoded representation. Let’s take a sample ENR and decode it using the ENR Viewer. I will grab one of the EF Bootnodes.

• Encoded: enr:-Ku4QHqVeJ8PPICcWk1vSn_XcSkjOkNiTg6Fmii5j6vUQgvzMc9L1goFnLKgXqBJspJjIsB91LTOleFmyWWrFVATGngBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpC1MD8qAAAAAP__________gmlkgnY0gmlwhAMRHkWJc2VjcDI1NmsxoQKLVXFOhp2uX6jeT0DvvDpPcU8FWMjQdR4wMuORMhpX24N1ZHCCIyg

• Decoded:

{
  "signature": "0x7a95789f0f3c809c5a4d6f4a7fd77129233a43624e0e859a28b98fabd4420bf331cf4bd60a059cb2a05ea049b2926322c07dd4b4ce95e166c965ab1550131a78",
  "seq": "1",
  "attnets": "0x0000000000000000",
  "eth2": "0xb5303f2a00000000ffffffffffffffff",
  "id": "v4",
  "ip": "3.17.30.69",
  "secp256k1": "0x028b55714e869dae5fa8de4f40efbc3a4f714f0558c8d0751e3032e391321a57db",
  "udp": "9000"
}

We’ve seen the building blocks of the peer-to-peer consensus layer networking, why are they needed and how they interact with each other. It’s a bit too technical and without any code to test some of their workings! So I will upload some code to test these specs, mostly discv5 and libp2p.

• https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/p2p-interface.md

• https://github.com/ethereum/devp2p/blob/master/discv5/discv5.md

• https://ethereum.org/en/developers/docs/networking-layer/

• https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md

• https://github.com/sigp/discv5

Since the merge an ethereum node consists of two layers, the execution layer (EL) and the consensus layer (CL). Each one of these have their own networking stack for peer-to-peer (from now on, p2p) communication.

The CL and EL p2p stacks are similar, but in this post I’ll focus in the CL p2p networking. Some of the terms are common to many peer-to-peer network implementations, but if you’re not familiar with any of them I encourage you to take a look at the libp2p documentation, which is also the main library to use when implementing these specs. Now let’s begin by taking a look at the fundamentals.

The network configuration can be, and possibly is, quite different in every client implementation, so I’ll stick to the specs that all of them supports:

• Transport — All implementations support the TCP libp2p transport enabled for both dialing and listening. They’re capable of dialing both IPv4 and IPv6 addresses and listening is supported on at least one of IPv4 or IPv6.

• Multiplexing — Clients support mplex and may support yamux.

• Encryption and identification — Is based on noise, a libp2p transport secure channel handshake. It supports the XX handshake pattern, which provides mutual authentication and encryption of static keys and handshake payloads and is resistant to replay attacks.

• Protocol negotiation — Clients use exact equality when negotiating protocol versions to use. At present, multistream-select 1.0 is used for negotiation of:

  • Authentication/Encryption Mechanism: Noise, SecIO, TLS 1.3.

  • Overlay Multiplexer: mplex, Yamux, spdystream.

Why is some of this needed?

• TCP is natively incapable of multiplexing and encryption, thus these are applied atop and we need to negotiate on which one to use depending on the different implementations.

• Multiplexing is needed to conduct parallel conversations.

• Transport-level encryption secures the communication (authentication, content eavesdropping, tampered data) but none of the application layer related data.

The gossip domain includes all information that has to spread rapidly throughout the network, such as blocks, proofs, attestations, exits and slashings. For doing this gossipsub is used, a general purpose publish/subscribe protocol.

A pub/sub protocol is based on topics and messages. Peers only receive messages about a topic that they’re subscribed to, and send messages to specific topics.

Clients support gossipsub v1 libp2p protocol including the gossipsub v1.1 extension. However, only the configurable v1 parameters are used to control it’s behavior.

Topics are plain UTF-8 strings (e.g. proposer_slashing) and are encoded on the wire as determined by protobuf (gossipsub messages are wrapped in protobuf messages).

• Global topics: beacon_block - beacon_aggregate_and_proof - voluntary_exit - proposer_slashing - attester_slashing

• Attestation subnets: Are used to propagate unaggregated attestations to subsections of the network. These are defined by the beacon_attestation_{subnet_id} topic.

Topics are also post-fixed with an encoding that define how the payload of a gossipsub message is encoded. All objects are SSZ-encoded and then compressed with snappy.

The message proto structure is:

message Message {
    optional bytes from = 1;
    optional bytes data = 2;
    optional bytes seqno = 3;
    optional string topic = 4;
    optional bytes signature = 5;
    optional bytes key = 6;
}

Where the fields from, seqno, signature and key are omitted. Messages are identified by content, anonymous, and signed where necessary in the application layer.

The default libp2p pubsub message-id is overridden to use content-addressing, allowing to filter unnecessary duplicates before hitting the application layer. It is replaced by either:

• SHA256(MESSAGE_DOMAIN_VALID_SNAPPY + snappy_decompress(message.data))[:20]

• SHA256(MESSAGE_DOMAIN_INVALID_SNAPPY + message.data)[:20]

Depending if the data has a valid snappy decompression.

The request-response domain contains protocols for clients requesting specific information from their peers (e.g. blocks matching certain root hashes or within a range of slots). The responses are always returned as snappy-compressed SSZ encoded bytes.

Each message type is segregated into its own libp2p protocol ID, which is a case-sensitive UTF-8 string of the form:

• /ProtocolPrefix/MessageName/SchemaVersion/Encoding

  • ProtocolPrefix — libp2p protocol name prefix. e.g. /eth2/beacon_chain/req

  • MessageName — Request identifier

  • SchemaVersion — Schema version number

  • Encoding — Encoding strategy

This protocol segregation allows libp2p multistream-select 1.0 / multiselect 2.0 to handle the request type, version, and encoding negotiation before establishing the underlying streams.

Only one stream per request/response interaction is used. Streams are closed when the interaction finishes, whether in success or in error.

Req-res messages have the following structure:

request   ::= <encoding-dependent-header> | <encoded-payload>
response  ::= <response_chunk>*
response_chunk  ::= <result> | <encoding-dependent-header> | <encoded-payload>
result    ::= “0” | “1” | “2” | [“128” ... ”255”]

• encoding-dependent-header carry metadata or assertions such as the encoded payload length, for integrity and attack proofing purposes.

• response is formed by zero or more response_chunks.

• result is a single byte that details the result of the request

  • 0x0 - Success

  • 0x1 - InvalidRequest

  • 0x2 - ServerError

  • 0x3 - ResourceUnavailable

Only one strategy is allowed at this time:

• ssz_snappy: The contents are first ssz-encoded and then compressed with snappy.

Let’s take a look at the Status message, which is sent upon connection:

• Protocol ID: /eth2/beacon_chain/req/status/1/

• Request/Response Content:

(
  fork_digest: ForkDigest
  finalized_root: Root
  finalized_epoch: Epoch
  head_root: Root
  head_slot: Slot
)

• Where:

  • fork_digest: 4-byte fork digest for the current_version and genesis_validators_root.

  • finalized_root: state.finalized_checkpoint.root of the current head block.

  • finalized_epoch: state.finalized_checkpoint.epoch of the current head block.

  • head_root: hash_tree_root root of the current head block.

  • head_slot: The slot of the block corresponding to the head_root.

Other messages are:

• Goodbye

• BeaconBlocksByRange

• BeaconBlocksByRoot

• Ping

• GetMetaData

Before a peer can subscribe to a topic or interact with another network participant it must find other peers and establish network connections with them. CL clients use the Node Discovery Protocol v5 (discv5).

To facilitate peer discovery and content routing a distributed hash table (DHT) is used. A regular hash table is a set of records in which every record holds a key -> value mapping. In a distributed hash table, the records are spread across the network, and every node holds a subset of them.

Discv5 is inspired by the Kademlia DHT, but in this case the DHT stores and relays 'node records', which are signed documents that provide information about nodes in the network. It acts as a database of all live nodes in the network and performs three functions:

• Sampling the set of all live participants.

• Searching for participants providing a certain service (topic advertisements).

• Authoritative resolution of node records with a node’s ID.

It runs on UDP on a dedicated port that it is meant for peer discovery only and supports self-certified, flexible peer records (ENRs).

The CL discv5 implementation differs from that of the EL only in that it includes an adaptor to connect it with libp2p , deprecating devP2P.

• Operation inputs include peer IDs or capabilities (such as topics).

• Outputs are multiaddress converted from the ENRs returned by the discv5 backend.

This integration enables the libp2p stack to subsequently form connections and streams with discovered peers.

The ethereum node record, as defined by the EIP-778, is a flexible format for connectivity-related information. It improves cryptographic agility and the handling of protocol upgrades. A record can contain information about arbitrary transport protocols and public key material associated with them.

The required entries for CL clients are:

• signature — cryptographic signature of record contents

• seq — Sequence number (uint64). Nodes increase the number whenever the record changes and republish the record.

• id — Identity scheme, e.g. v4

• secp256k1 — Compressed secp256k1 public key - 33 bytes.

• attnets — SSZ of Bitvector[ATTESTATION_SUBNET_COUNT]

• eth2 — SSZ of ENRForkID (fork_digest, next_fork_version,next_fork_epoch)

But some nodes also include:

• ip — IPv4 address

• ip6 — IPv6 address

• tcp — TCP local libp2p listening port.

• udp — UDP local discv5 listening port.

The textual form of a node record is the base64 encoding of its RLP encoded representation. Let’s take a sample ENR and decode it using the ENR Viewer. I will grab one of the EF Bootnodes.

• Encoded: enr:-Ku4QHqVeJ8PPICcWk1vSn_XcSkjOkNiTg6Fmii5j6vUQgvzMc9L1goFnLKgXqBJspJjIsB91LTOleFmyWWrFVATGngBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpC1MD8qAAAAAP__________gmlkgnY0gmlwhAMRHkWJc2VjcDI1NmsxoQKLVXFOhp2uX6jeT0DvvDpPcU8FWMjQdR4wMuORMhpX24N1ZHCCIyg

• Decoded:

{
  "signature": "0x7a95789f0f3c809c5a4d6f4a7fd77129233a43624e0e859a28b98fabd4420bf331cf4bd60a059cb2a05ea049b2926322c07dd4b4ce95e166c965ab1550131a78",
  "seq": "1",
  "attnets": "0x0000000000000000",
  "eth2": "0xb5303f2a00000000ffffffffffffffff",
  "id": "v4",
  "ip": "3.17.30.69",
  "secp256k1": "0x028b55714e869dae5fa8de4f40efbc3a4f714f0558c8d0751e3032e391321a57db",
  "udp": "9000"
}

We’ve seen the building blocks of the peer-to-peer consensus layer networking, why are they needed and how they interact with each other. It’s a bit too technical and without any code to test some of their workings! So I will upload some code to test these specs, mostly discv5 and libp2p.

• https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/p2p-interface.md

• https://github.com/ethereum/devp2p/blob/master/discv5/discv5.md

• https://ethereum.org/en/developers/docs/networking-layer/

• https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md

• https://github.com/sigp/discv5