Welcome to another AllCoreDevs update 👋🏻

This one is a bit different from the others. First, I’ve decided to move these to Mirror! Using the platform for the 1559 NFT project was a really positive experience, and I’ve been looking for an excuse to use it again, so here we are. I’ll keep maintaining a list of updates on HackMD so they re easy to find in one place.

Second, given our current focus on The Merge, this post isn’t an “update” but a deep dive into the roadmap evolution which led to the current architecture choice. In a few weeks, expect a similar post going over the details of how the Ethereum network will operate post-merge.

The Merge will swap out Ethereum’s current proof of work consensus algorithm for the proof of stake mechanism running on the beacon chain. This design is the result of several iterations on the “Ethereum 2.0” roadmap. Let's recap how things changed over the past few years.

For several years, proof of stake and sharding R&D were both pretty independent from each other. During a 2018 research workshop in Taipei, a decision was made to unify both research initiatives into a single, three phrased, "Ethereum 2.0" roadmap. The workshop and general philosophy at the time is well summarized by Ben Edginton here. It was during this event that Hsiao-Wei Wang presented this now famous diagram of what "Ethereum 2.0" would look like:

This diagram pictures what each phase of the then new roadmap would deliver:

• Phase 0 would bring the beacon chain;

• In Phase 1, data shards would be added;

• In Phase 2, virtual machines would be added to each shard to enable computation in the system.

The number of shards was initially set to 100, then raised to 1024, and most recently lowered back to 64.

As work on the beacon chain started in 2018, it became clear that this three-phased Ethereum 2.0 roadmap would take several years to fully deliver. This, along with the growing pains from the rapid adoption of Ethereum, led to a revival of research initiatives on the proof of work chain. The umbrella term "Ethereum 1.x" was coined for these initiatives at Devcon IV, in 2018. The biggest one of these was research into Stateless Ethereum, a paradigm that would remove untouched state from the network in order to bound its growth rate.

The increased focus on making the PoW chain long-term sustainable paired with the realization that the beacon chain would be ready much earlier than other components of the Ethereum 2.0 roadmap led to an “Early Merge” proposal. This proposal would have the existing EVM chain launch as "Shard 0" of the Ethereum 2.0 system. Not only would this expedite the move to proof of stake, but it would also make for a much smoother transition for applications, as the move to proof of stake could happen without any migration on their end.

Shortly after this proposal, Danny Ryan explored how we could accomplish all this by leveraging the existing Eth1 clients in his Eth1+Eth2 client relationship post. This would massively reduce the development work to deliver a post-merge system and leverage clients which had been battle tested for years on mainnet. Following this path would also give researchers more time to figure out the open questions around Phases 1 & 2, explored here, as well as Stateless Ethereum, which is also still an active area of research.

Fast forward to late 2020: Phase 0 is now 99% done with the beacon chain launching soon. While the work on Phase 1 is progressing nicely, Phase 2, which would enable computation on the shards introduced in Phase 1, still has many unsolved questions:

• How to transition smoothly from the current EVM chain to the sharded VMs?

• What alternative VMs can actually be deployed, performance-wise?

• How would we ensure that various VMs are indeed safe?

• How would state and balances be reconcilled across all the VMs?

• … and more

At the same time, progress on rollups (i.e. Layer 2 scaling) is rapidly happening. Several teams announce testnets, with promising early results.

Around this time, Vitalik wrote a long Ethereum Magicians post arguing that we should focus our short and medium term scaling efforts around rollups. Not only would they be on mainnet before Phase 2 would be done, but they would also be the largest beneficiaries of Phase 1. Rollups generate a lot of data, and shards can offer cheaper storage for it than the EVM chain. From the post itself:

This approach would have several impacts on the Phase 0-2 roadmap. Specifically:

• Phase 0 (the beacon chain) which was now running on testnets ahead of mainnet launch, could become Ethereum’s consensus engine as soon as the PoW chain was ready, rather than after Phase 2;

• Phase 1 (data sharding), still several years away, went from being a critical blocker to scaling to an incremental improvement which would reduce the cost of already developed scaling solutions (e.g. rollups);

• Phase 2 (sharded execution), the more complex feature with the most open research questions, could be pushed out several years or cancelled entirely without any impact on the scaling roadmap.

The research community quickly rallied around this proposal and within a month had produced an ethresear.ch post detailing the current plan for The Merge!

The current architecture for the merge was first detailed in the "Executable Beacon Chain" post by Mikhail Kalinin in November 2020. In short, it combined the following insights from the various iterations to the Ethereum 2.0 roadmap:

• The beacon chain is live, and can now be used as a consensus engine;

• Rollups are the best short term solution for scaling computation;

• The current Eth1 clients form the best basis for the execution layer post-merge;

• The move to proof of stake can be done with minimal impact on the currently running applications.

The one big change Mikhail proposed was that instead of having the current EVM chain be "Shard 0," it could be coupled to the beacon chain directly.

This is a simple but important insight: Eth1 execution clients are already built in a way where the consensus algorithm can be swapped. Mainnet uses proof of work, but testnets and private Ethereum networks use various proof of authority consensus algorithms (clique, IBFT, etc.).

In his post, Mikhail proposed to have proof of stake "simply" be a new consensus algorithm that clients use. In other words, to merge the current proof of work chain with the beacon chain. The following diagram by Trent Van Epps illustrates the transition:

This approach would minimize the work that client teams on the PoW chain need to do, while still delivering all the benefits of the Early Merge and Rollup Centric Roadmap [1].

To validate that the Executable Beacon Chain architecture was viable, clients prototyped it over the course of a month-long hackathon named Rayonism.

Within a few weeks, all combinations of Eth1/Eth2 clients were turned into hybrid, post-merge, clients running transactions in the EVM and coming to consensus via the beacon chain.

At a high level, the current Eth2 nodes were repurposed as the consensus layer of the network, and the current Eth1 nodes as its execution layer. The consensus layer's functionality was expanded to send the latest chain head information to, and request blocks from, the execution layer. The execution layer was still in charge of processing blocks, gossiping transactions, storing and managing the state and handling JSON RPC requests.

This experiment validated that the Executable Beacon Chain architecture was sound and could be use as the basis for the transition to proof of stake, now dubbed The Merge.

One thing that was not tested as part of Rayonism was the transition from a live PoW network to one running under PoS. After a few iterations, a specification for this is now roughly finalized.

To transition from proof of work to proof of stake, a TERMINAL TOTAL DIFFICULTY will be set in clients [2]. Once a block is seen exceeding that difficulty on the proof of work chain, clients will enter a transition mode where they begin listening to the proof of stake layer for consensus. As soon as the consensus layer has finalized a block whose difficulty exceeds the TERMINAL TOTAL DIFFICULTY, the execution layer will stop listening for and gossiping PoW blocks entirely. And then, voilà, The Merge is done 🎉!

For applications, this won't cause any disruption to contracts or users. A few opcodes will be updated and that's about it. For beacon chain node operators, the merge will require choosing an execution engine . Similarly, if you are running a node on the proof of work network, the merge will require choosing a consensus client. You can expect multiple devnets, tutorials and calls to discuss this as the work on the merge progresses!

While the general approach for The Merge is now set, there is still a long list of things that client teams need to do over the next few months. The bulk of the TODOs are tracked here, and notable ones include finalizing post-merge sync protocols, setting up integration tests for the entire process, launching devnets & running them under adverse conditions, and planning for a variety of contingencies during the transition.

If you'd like to dig into the actual specs, EIP-3675 details the changes that will be required for the execution-layer clients, and the merge folder in the consensus specs details the changes to the consensus layer.

Once things are a bit more settled, expect another post going into the details of how Ethereum clients will work post-Merge. Thanks for reading, and see you next time 👋🏻!

Thanks to Danny Ryan, Trent Van Epps and Mikhail Kalinin for feedback on drafts fo this post.

[1] One notable change is that "Stateless Ethereum" is no longer a prerequisite for the move to proof of stake.

[2] Relying on the total difficulty rather than the block or slot number allows for better management of re-orgs during the transition phase. For more on this, see "Transition process" section of EIP-3675.

Welcome to another AllCoreDevs update 👋🏻

This one is a bit different from the others. First, I’ve decided to move these to Mirror! Using the platform for the 1559 NFT project was a really positive experience, and I’ve been looking for an excuse to use it again, so here we are. I’ll keep maintaining a list of updates on HackMD so they re easy to find in one place.

Second, given our current focus on The Merge, this post isn’t an “update” but a deep dive into the roadmap evolution which led to the current architecture choice. In a few weeks, expect a similar post going over the details of how the Ethereum network will operate post-merge.

The Merge will swap out Ethereum’s current proof of work consensus algorithm for the proof of stake mechanism running on the beacon chain. This design is the result of several iterations on the “Ethereum 2.0” roadmap. Let's recap how things changed over the past few years.

For several years, proof of stake and sharding R&D were both pretty independent from each other. During a 2018 research workshop in Taipei, a decision was made to unify both research initiatives into a single, three phrased, "Ethereum 2.0" roadmap. The workshop and general philosophy at the time is well summarized by Ben Edginton here. It was during this event that Hsiao-Wei Wang presented this now famous diagram of what "Ethereum 2.0" would look like:

This diagram pictures what each phase of the then new roadmap would deliver:

• Phase 0 would bring the beacon chain;

• In Phase 1, data shards would be added;

• In Phase 2, virtual machines would be added to each shard to enable computation in the system.

The number of shards was initially set to 100, then raised to 1024, and most recently lowered back to 64.

As work on the beacon chain started in 2018, it became clear that this three-phased Ethereum 2.0 roadmap would take several years to fully deliver. This, along with the growing pains from the rapid adoption of Ethereum, led to a revival of research initiatives on the proof of work chain. The umbrella term "Ethereum 1.x" was coined for these initiatives at Devcon IV, in 2018. The biggest one of these was research into Stateless Ethereum, a paradigm that would remove untouched state from the network in order to bound its growth rate.

The increased focus on making the PoW chain long-term sustainable paired with the realization that the beacon chain would be ready much earlier than other components of the Ethereum 2.0 roadmap led to an “Early Merge” proposal. This proposal would have the existing EVM chain launch as "Shard 0" of the Ethereum 2.0 system. Not only would this expedite the move to proof of stake, but it would also make for a much smoother transition for applications, as the move to proof of stake could happen without any migration on their end.

Shortly after this proposal, Danny Ryan explored how we could accomplish all this by leveraging the existing Eth1 clients in his Eth1+Eth2 client relationship post. This would massively reduce the development work to deliver a post-merge system and leverage clients which had been battle tested for years on mainnet. Following this path would also give researchers more time to figure out the open questions around Phases 1 & 2, explored here, as well as Stateless Ethereum, which is also still an active area of research.

Fast forward to late 2020: Phase 0 is now 99% done with the beacon chain launching soon. While the work on Phase 1 is progressing nicely, Phase 2, which would enable computation on the shards introduced in Phase 1, still has many unsolved questions:

• How to transition smoothly from the current EVM chain to the sharded VMs?

• What alternative VMs can actually be deployed, performance-wise?

• How would we ensure that various VMs are indeed safe?

• How would state and balances be reconcilled across all the VMs?

• … and more

At the same time, progress on rollups (i.e. Layer 2 scaling) is rapidly happening. Several teams announce testnets, with promising early results.

Around this time, Vitalik wrote a long Ethereum Magicians post arguing that we should focus our short and medium term scaling efforts around rollups. Not only would they be on mainnet before Phase 2 would be done, but they would also be the largest beneficiaries of Phase 1. Rollups generate a lot of data, and shards can offer cheaper storage for it than the EVM chain. From the post itself:

This approach would have several impacts on the Phase 0-2 roadmap. Specifically:

• Phase 0 (the beacon chain) which was now running on testnets ahead of mainnet launch, could become Ethereum’s consensus engine as soon as the PoW chain was ready, rather than after Phase 2;

• Phase 1 (data sharding), still several years away, went from being a critical blocker to scaling to an incremental improvement which would reduce the cost of already developed scaling solutions (e.g. rollups);

• Phase 2 (sharded execution), the more complex feature with the most open research questions, could be pushed out several years or cancelled entirely without any impact on the scaling roadmap.

The research community quickly rallied around this proposal and within a month had produced an ethresear.ch post detailing the current plan for The Merge!

The current architecture for the merge was first detailed in the "Executable Beacon Chain" post by Mikhail Kalinin in November 2020. In short, it combined the following insights from the various iterations to the Ethereum 2.0 roadmap:

• The beacon chain is live, and can now be used as a consensus engine;

• Rollups are the best short term solution for scaling computation;

• The current Eth1 clients form the best basis for the execution layer post-merge;

• The move to proof of stake can be done with minimal impact on the currently running applications.

The one big change Mikhail proposed was that instead of having the current EVM chain be "Shard 0," it could be coupled to the beacon chain directly.

This is a simple but important insight: Eth1 execution clients are already built in a way where the consensus algorithm can be swapped. Mainnet uses proof of work, but testnets and private Ethereum networks use various proof of authority consensus algorithms (clique, IBFT, etc.).

In his post, Mikhail proposed to have proof of stake "simply" be a new consensus algorithm that clients use. In other words, to merge the current proof of work chain with the beacon chain. The following diagram by Trent Van Epps illustrates the transition:

This approach would minimize the work that client teams on the PoW chain need to do, while still delivering all the benefits of the Early Merge and Rollup Centric Roadmap [1].

To validate that the Executable Beacon Chain architecture was viable, clients prototyped it over the course of a month-long hackathon named Rayonism.

Within a few weeks, all combinations of Eth1/Eth2 clients were turned into hybrid, post-merge, clients running transactions in the EVM and coming to consensus via the beacon chain.

At a high level, the current Eth2 nodes were repurposed as the consensus layer of the network, and the current Eth1 nodes as its execution layer. The consensus layer's functionality was expanded to send the latest chain head information to, and request blocks from, the execution layer. The execution layer was still in charge of processing blocks, gossiping transactions, storing and managing the state and handling JSON RPC requests.

This experiment validated that the Executable Beacon Chain architecture was sound and could be use as the basis for the transition to proof of stake, now dubbed The Merge.

One thing that was not tested as part of Rayonism was the transition from a live PoW network to one running under PoS. After a few iterations, a specification for this is now roughly finalized.

To transition from proof of work to proof of stake, a TERMINAL TOTAL DIFFICULTY will be set in clients [2]. Once a block is seen exceeding that difficulty on the proof of work chain, clients will enter a transition mode where they begin listening to the proof of stake layer for consensus. As soon as the consensus layer has finalized a block whose difficulty exceeds the TERMINAL TOTAL DIFFICULTY, the execution layer will stop listening for and gossiping PoW blocks entirely. And then, voilà, The Merge is done 🎉!

For applications, this won't cause any disruption to contracts or users. A few opcodes will be updated and that's about it. For beacon chain node operators, the merge will require choosing an execution engine . Similarly, if you are running a node on the proof of work network, the merge will require choosing a consensus client. You can expect multiple devnets, tutorials and calls to discuss this as the work on the merge progresses!

While the general approach for The Merge is now set, there is still a long list of things that client teams need to do over the next few months. The bulk of the TODOs are tracked here, and notable ones include finalizing post-merge sync protocols, setting up integration tests for the entire process, launching devnets & running them under adverse conditions, and planning for a variety of contingencies during the transition.

If you'd like to dig into the actual specs, EIP-3675 details the changes that will be required for the execution-layer clients, and the merge folder in the consensus specs details the changes to the consensus layer.

Once things are a bit more settled, expect another post going into the details of how Ethereum clients will work post-Merge. Thanks for reading, and see you next time 👋🏻!

Thanks to Danny Ryan, Trent Van Epps and Mikhail Kalinin for feedback on drafts fo this post.

[1] One notable change is that "Stateless Ethereum" is no longer a prerequisite for the move to proof of stake.

[2] Relying on the total difficulty rather than the block or slot number allows for better management of re-orgs during the transition phase. For more on this, see "Transition process" section of EIP-3675.