In this post, we analyze Lido operators’ client diversity in the Ethereum consensus layer over time, observing the number of blocks proposed using each client starting from genesis until now. We also give an introduction to what client diversity is, what Lido is, how diversity can be studied, and last but not least, we present our findings on Lido operators.

🔗 All data can be accessed here 🔗

2. Client Diversity: Validator vs Beacon

The Ethereum blockchain is made of thousands of nodes running across the world and controlled by multiple entities, ranging from companies to individuals. Each node follows a set of common rules that make all of them interoperable. These rules are defined in the so-called specifications and are split into execution and consensus specifications.

But note that the Ethereum Foundation does not provide the software per se, it just provides the rules that must be respected. Anyone is free to implement their software (referred to as client from now on) in the programming language they prefer and join the network.

Currently, there are 4 different software implementing the execution layer specifications, and 5 implementing the consensus layer:

• Clients implementing the execution specifications: geth, erigon, besu, and nethermind.

• Clients implementing the consensus specifications: lighthouse, teku, prysm, nimbus, and lodestar.

So each node of the network can run the combination execution/consensus that they prefer: geth/nimbus, erigon/prysm, etc. Note that in this post, we focus on consensus diversity.

Let’s say that the Ethereum network has 5000 beacon nodes. Ideally, we would like that each client has a 20% share of the network so that 1000 nodes run lighthouse, 1000 teku, 1000 prysm, 1000 nimbus and 1000 lodestar.

However, this is not the case. Anyone is free to pick the client they want, and some clients are over-represented in the network. Now you may wonder, why that many software implementations of Ethereum. There are multiple reasons behind this, but the main ones are:

• They are written in different programming languages with different use cases in mind

• If a software bug affects a given client, the network can continue working. This adds a new level of redundancy, similar to avionics systems in the aerospace industry.

It’s important to highlight that diversity can be studied from two different perspectives: beacon vs validator.

As a quick recap, the beacon node is the software that takes care of following the chain, staying in sync with the latest block, and constantly asking other nodes what’s their view of the chain.

On the other hand, a validator represents a staked position of 32 Eth, linked to a pair of public/private keys, that can create blocks and attestations when it’s its turn. Validators do not directly interact with the chain, they use the beacon node to send their blocks/attestations. So multiple validators can use the same beacon node.

So we can estimate client diversity from two perspectives:

• From a beacon-node perspective, estimating which client each node runs. According to migalabs, there are around 6000 beacon nodes (21st November 2022)

• From a validator perspective, estimating the client each validator runs. According to beaconchain, there are 472.000 validators (21st November 2022) and each one of them runs a given client.

You can find more on client diversity at clientdiversity.org, both on beacon node and validator level. Unfortunately, both metrics are aggregated for the whole network, so we don’t know the share for specific entities like exchanges, pools, or whales.

In this post, we go one step forward, studying Lido’s client diversity over time since genesis, on a validator level.

3. What is Lido?

Lido is a company that offers liquid staking solutions for different blockchains, including Ethereum. One of the benefits of liquid staking is that you can unstake at any time, and of course, you don’t need to run your validators.

Currently, Lido contains 29 operators that are in charge of running the validators that hold all user’s funds. Since all keys controlled by these 29 operators are publicly known and recognized by Lido, we can query the beacon chain with these keys and hold them accountable for what they do. In this post, we focus on client diversity, but you can refer to ethereumpools.info for their performance.

Note that Lido and its operators currently control around 30% of the Ethereum network (21 November 2022) so if you are thinking about staking, we recommend you to either solo stake (run your node) or use more decentralized solutions with less representation like RocketPool.

4. Methodology

For this study we focus on validator client diversity rather than beacon diversity, since for estimating the latter we would need to know the beacon nodes that Lido runs, and that’s not an easy task.

Lido publicly recognizes the validator indexes that they run. And fortunately, sigmaprime developed a tool called blockprint that estimates the client that a given validator uses. Without going into details, it uses a machine learning classification algorithim that classifies the validators using subtle differences in how blocks are proposed.

Disclaimer and Limitations:

• It’s impossible to estimate the client a validator uses with 100% accuracy. Blockprint is not perfect and in some cases, it can mislabel the client.

• The client type a given validator runs can only be estimated after a successful block proposal. In other words, we can’t know the client of a validator that hasn’t proposed any blocks.

• Lido has acknowledged that some of their operators run vouch, a software developed by Attestant that sits between the beacon node and signer and proposes “the best block” coming from any of the clients, which adds another dimension for diversity.

5. Conclussions

In the next section you can find all plots backing our conclusions, but if you want something more interactive, you can find the data here. Let’s see what happened from genesis until November 2022:

• DSRV, Figment, Chorusone, and Bridgetower are starting to use Nimbus. All of them have used Lighthouse for the whole time but a few weeks before The Merge they started using some Nimbus validators.

• We see that Stakely, Nethermind, Simplystaking, CryptoManufaktur, and StakingFacilities are very diverse. They have used most of the clients over time, which makes us think they are using Vouch.

• On the other hand Blockscape used to be very diverse, but right now they heavily rely on Lighthouse.

• Everstake, Hashquark and RockX are not diverse right now, relying only on Prysm client.

• Some operators like RockX massively decreased its Nimbus usage prior The Merge, perhaps they thought it was risky to run a minority client. But its been proven that they perform just like the others, and are well integrated with tools like mev-boost.

• Rockx, Skillz, Everstake, Hashquark, Infstones, Blockdaemon, Anyblockanalytics have a high usage of Prysm.

• Stakin and Allnodes have a high usage of Teku.

• Chainsafe run 100% of Lodestar validators, perhaps expected since its the company behind Lodestar client.

• SigmaPrime operator runs a high percent of Lighthouse, expected since its the company behind that client, but we can see that they recently started using Nimbus.

• PrysmaticLabs run only of Prysm, expected since its the company behind that client. Note that the are a few blocks labeled as Nimbus, but looks like an outlier, which can be due to a blockprint misslabel.

• Rockx, Skillz, Stakely, Everstake, Hashquark, Infstones, Certusone, Anyblockanalyitcs, P2Porg decreased their Nimbus usage before the merge.

6. Data

We like epoch-based timestamps, but if you still measure time in days, months and years, here’s some help. Data can be accessed interactively here.

• Epch 0: Dec 2020.

• Epoch 60k: August 2021.

• Epoch 100k: February 2022.

• Epoch 120k: May 2022.

• Epoch 140k: August 2022.

• Epoch 146k: The Merge 🐼, September 2022

DSRV

• Always used Lighthouse client, but recently before the merge started using some Nimbus validators.

Rockx

• Before the merge they had a good client diversity using mainly Prysm and Nimbus but they currently rely too much on just one client, Prysm.

Skillz

• Before the merge they had a good split beetween Prysm, Nimbys and Teku, but after the merge they almost stopped using Nimbus.

• We suspect they use vouch.

Stakin

• Started using only Teku but notably increased using Nimbus over 2022.

Figment

• Since the begining they relied 100% in Prysm, but few months before the merge they started running some Nimbus validators.

Stakely

• From the begining they always had a very diverse set of validators, proposing blocks with all of them but Lodestar.

• Started with a really high usage of Nibus client, but really decreased after June 2022.

• After the merge, they practically stopped using Prysm validators.

• We suspect they use vouch.

Allnodes

• Started using only Teku during more than a year, but we can see how they are starting to use some Nimbus validators.

Chainsafe

• All of the blocks they have proposed are from Lodestar client, which is kind of expected since Chainsafe its the company behind that client.

Everstake

• We can observe some client diversity in the past, but they currently heavily rely on just Prysm.

Hashquark

• They started using Prysm and Nimbus, but its Nimbus usage suddenly decreased few months before the merge.

• Currently they heavily rely on just Prysm.

Infstones

• Very similar pattern to hashquark operator. Started with Prysm and Nimbus with a decrease in Nimbus validators few months before the merge.

• We suspect infstones and hashquark are running the exact same setup, since the diversity over time exactly matches.

Stakefish

• Started using only Lighthouse but started to use Teku few months before the merge. Interesting to see the massive drop in Teku usage few weeks before the merge.

• Few months before the merge, they started using some Nimbus client, but still with a low share.

Blockscape

• During almost the first year they used only lighthouse, but after that we can see that their setup was very diverse.

• However, after the merge they went back to heavily rely on Lighthouse.

• We suspect they use vouch, which justifies some spikes in client type.

Certusone

• Started using Prysm and Lighthouse.

• Had an increase in Nimbus validators, but they stopped using it before the merge.

Chainlayer

• Lighthouse is the majority client but we can see some validators running Teku.

• Their diversity is quite stable, and didn’t change much with the merge.

• In Q2 2022 they started using some Nimbus clients.

Chorusone

• Uses only Lighthouse, but a bit before the merge we can see how they started using some Nimbus clients.

Nethermind

• Fairly new Lido operator, started with Lighthouse.

• After the merge they started using Teku for most of the blocks, and more recently Prysm.

• We suspect they use vouch.

Blockdaemon

• Very dependant on Prysm, but started using some Nimbus client from the begining of 2022.

Bridgetower

• Very dependant on Lighthouse.

• Started using some Nimbus validators a bit before the merge.

Sigmaprime

• Fairly new Lido operator.

• High Lighhouse usage, which is expected since Sigma Prime is the company building Lighthouse.

• We can see some Nimbus validators.

Kukisglobal

• High usage of Lighthouse, with some Nimbus clients over the last year.

Prysmaticlabs

• They started proposing blocks quite recently, few weeks after the merge.

• High usage of Prysm client, which is expected since its the company behind this client.

• We can one Nimbus block, but looks like an outlayer value, which might be due to blockprint misslabeling proposals.

Rocklogicgmbh

• Few weeks before the merge they changed from using only Lighthouse to having a more diverse setup, with Teku and Nimbus.

• More recently, they started using also Prysm.

Simplystaking

• Had a huge drop in Nimbus usage around the merge.

• They currently use only Prysm and Lighthouse, with very few blocks using Nimbus.

Consensyscodefi

• Always used Teku and Lighthouse, but we can observe some Nimbus usage started few months before the merge.

Cryptomanufaktur

• They mainly use Teku and Lighthouse but we can also see some blocks being proposed with Nimbus.

• We suspect they are using vouch.

Anyblockanalytics

• Always used Prysm and some Nimbus, but its Nimbus usage went to almost zero few weeks before the merge.

Stakingfacilities

• During the first year and a half, they only used Lighthouse.

• Over 2022 they started using a more diverse setup, perhaps using vouch.

• Its Prysm and Nimbus usage decreased after the merge.

P2POrg

• High usage of Prysm.

• Nimbus usage was increasing over 2022 but almost went to zero around the merge.

• Currently, it mainly relies on Prysm and Lighthouse.

For more on real-time Ethereum validator monitoring:

• 🐦 Follow us on Twitter.

• 📉 Check our dashboard.

Al. Rev.

In this post we analyze the performance of different Ethereum staking pools/operators/stakers during The Merge (migration from Proof of Work to Proof of Stake), taking 12 hours before, and 12 hours after it happened.

We start with some context on what The Merge is, and then we introduce ethereumpools.info a real-time monitoring tool whose data was used for this analysis. After that, we present and analyze multiple plots showing how this transition affected different entities, ending up with the methodology we followed for the analysis. To conclude, we present our conclussions.

2. The Merge

Last 15th of September 2022 at 08:42:59 UTC block 15537394 time something changed in the Ethereum blockchain forever. Proof of Work ⛏️ was switched off in favor of Proof of Stake 🌱, making Ethereum greener and reducing its energy consumption by 99.988%, among other implications.

This put an end to several years of research and development, finally delivering the GASPER consensus protocol to a production environment securing billions of dollars in assets. Note that while this was a huge milestone in the Ethereum blockchain, it’s just one upgrade of the roadmap, that continues with withdrawals, proto-dansksharding, PBS, account abstractions, or single secret leader election among others.

And you may think, why is this transition from PoW to PoS referred to as The Merge? Well, it’s actually a merge (or fusion if you are a Dragon Ball Z fan) because two chains were actually merging together:

• On one side we had Ethereum Mainnet, live since July 2015, what everyone calls Ethereum.

• On the other one we had the Ethereum Beacon Chain, which launched on the 1st of December but that didn’t process any transaction until The Merge was due.

This event was huge and was followed by thousands of people around the globe, with live streams, and local parties in multiple capitals. Even Google acknowledged it with a funny animation when searching for The Merge. If these 🐻🐼 emojis mean something to you, you know what we are talking about.

The Merge was a transition that was required to be monitored carefully since the network couldn’t stop, some people even compared it to changing the engine of an aircraft while flying without the passengers even noticing. Note that the Ethereum blockchain is used by hundreds of thousands of people and secures hundreds of millions of dollars in value, so it can just stop.

On top of that, there was an extra added complexity. The Ethereum blockchain is defined by a set of execution and consensus common specifications, but there are a total of 9 different teams implementing its software, with more than a dozen combinations that are required to be tested. It was not an easy task and parithosh_j can tell you more about that. We call this client-diversity:

• geth, erigon, besu, and nethermind on the execution side

• lighthouse, teku, prysm, nimbus, and lodestar on the consensus side.

You may wonder, why that many software implementations of Ethereum. There are multiple reasons behind this, but the main ones are:

• They are written in different programming languages with different use cases in mind

• If a software bug affects a given client, the network can continue working. This adds a new level of redundancy, similar to avionics systems in the aerospace industry.

Moreover, we have the so-called operator-diversity, meaning that this software is operated by different entities such as companies, exchanges, operators, and solo stakers, each one in a different location, with different hardware.

3. Real-Time Metrics

At ethereumpools we have spent almost a year analyzing real-time the metrics of groups of validators belonging to whales, exchanges, pools, operators, and even some small stakers. Our main goal is to monitor their performance and alert them if we detect problems. Over this time, we have publicly alerted on issues with ChorusOne, Whales, Figment, StakeWise, P2P Lido, Poloniex, BitcoinSuisse, and Stakefish among others. We also like writing about the technicalities behind staking and talking about rewards and penalties among other topics.

We want to hold them accountable for their performance, using publicly available data present in the beacon chain.

These are some of the consensus metrics that we look at to detect possible issues:

• Source and target votes, see Casper FFG.

• Votes for LMD GHOST, the fork-choice rule, aka head votes.

• Block proposal duties, and check the proposed and missed blocks.

• Slashings.

• Earned and lost rewards.

For example, this is one of the metrics we monitor in real-time, which measures the rewards/penalties in peta-wei that each pool earns per active validator every epoch, which is around six minutes. Since it’s a normalized metric, we can use it to compare how capital efficient each pool is.

We can use it to extrapolate the yearly APR% that each pool would get in consensus rewards. Since a year has 82125 epochs and each validator get on average 0.0175 peta-wei, one validator would earn 1.43718 Eth/year or 4.49%/year. This number doesn’t include execution rewards, and it decreases as number of active validators increase.

But how did this kind of graphs looked like during The Merge transition?

• Did professional operators performed better than solo stakers?

• Did any pool had issues?

• Was any pool offline?

We are here to answer these questions, extending the data avaialble on ethereumpools.info with extra metrics, plots and explanations.

4. Validators Performance During Merge Transition

Hereafter, we present a set of 24 hours plots, 12 hours before and 12 hours after The Merge. Our goal is to see which pools had issues during the transition, using several metrics to back our thesis. We divide the analysis in different groups:

• 1️⃣ Well known exchanges and pools.

• 2️⃣ Lido operators.

• 3️⃣ Solo stakers and whales.

The emoji 🐼 shown in the following diagrams denotes the boundary between PoW and PoS, or more precisely block 15537394, which was proposed early morning 15 September 2022 UTC time.

Each plot shows the performance 12 hours before/after The Merge with the following metrics:

• Percent of incorrect source votes for each epoch. A pool with 1000 validators and 100 validators voting wrong to source will have a 10% of wrong votes. The ideal value is 0%.

• Percent of incorrect head votes for each epoch. A pool with 1000 validators and 100 validators voting wrong to the head of the blockchain with have a 10% of wrong votes. The ideal value is 0%.

• Number of proposed blocks per epoch. Out of the 32 blocks that are proposed in a slot, indicates how many blocks a given pool proposed. The ideal value depends on the number of validators the pool controls.

• Number of missed proposer blocks per epoch. Amount of blocks that were missed. Ideal value is 0.

• Earned rewards per epoch per active validators. Amount of rewards that the pool earns per active validator on each epoch in peta-wei. This is a normalized metric that can be compared across different pools. Ideal value depends on the amount of active validators and varies with block proposals. Just consensus rewards are considered.

• Lost rewards per epoch per active validator. Amount of rewards that the pools is missing every epoch for each active validator. Ideal value is 0 peta-wei.

4.1 Exchanges and Pools 🏊

4.1.1 Coinbase

We can observe a small increase in wrong votes but its almost neglectable. On the other hand just one block proposal was missed, since Coinbase proposes 6-8 blocks per epoch this is neglectable. Rewards stayed constant and there were almost no penalties.

4.1.2 Stakewise

We can observe that around 20% of stakewise validators voted wrong to source and head, but it was quickly fixed within an hour. We can observe that no block proposals were missed, the rewards decreased a bit during some minutes and the penalties also increased.

Beyond performance, note how the earned rewards variance is higher than for Coinbase, but its expected. This is because of block proposals. Since Stakewise contains less validators, it proposes blocks less frequently.

4.1.3 Stakedus

We can observe a similar pattern as in Stakewise, but with less validators affected. Rewards and penalties were impacted but it lasted less than an hour.

4.1.4 Bloxstaking

Quite interestingly, Bloxstaking performed great during The Merge transition, 9 hours later, 40% of their validators had an issue during a small period of time. This lead to a missed proposal and some missed rewards, but was quickly fixed.

4.2 Lido Operators 💧

Since Lido has around 30 different operators controling their validator, we will just share few examples of performance: some of them good, and others not so good.

4.2.1 Stakely (Lido)

Stakely validators had no issues during The Merge transition. We can observe a small increase in missed votes, but neglectable. There were no missed blocks and we can observe how rewards stayed constant. Note that the spikes in rewards are due to block proposals.

4.2.2 Stakin (Lido)

We can clearly see how Stakin had some issues after The Merge. Wrong source votes started to increase up to 18%, and 40% of their votes to the head of the blockchain were wrong, which can be an indicator that their beacon node was having troubles staying in sync with the latest head.

Its also clear how the rewards decreased from a moving average of 0.016 peta-wei per epoch to 0.012. And beyond this, some of their validators were slightly penalised. Last but not least we can observe a missed proposal few hours after.

4.2.3 P2POrg (Lido)

For P2POrg we can see that the wrong votes increased a bit, but within reasonable limits when compared to other pools. However the amount of incorrect head votes increased notoriously, to almost 20% during few hours after The Merge. We can also see two missed proposals and a clear increase in the penalties.

4.3 Solo Stakers 🏠

4.3.1 RocketPool

First and foremost note that RocketPool is not a unique entity. We group here all solo stakers that run validatos for their pool, so the metrics we present here are an average of all of them.

We can observe how after The Merge, there was a spike to 10% in the number of incorrect source votes. Its not as crazy as Stakin (Lido) but enough to see a small impact in the rewards. Note that there were no missed proposals after The Merge.

4.3.2 DappNode

We group here all validators that we estimate run on DappNode based on the graffiti they use. These are thousands of validators running across the world in different hardwares and maintaned by non professional operators.

We can observe that The Merge had almost no impact in the wrong votes, staying at a similar level as before. Without being perfect, they performed better than some Lido operators or well known staking pools as shown above. No blocks were missed, and the impact in rewards was neglectable.

4.3.3 Avado

Similar to the previous ones, here we group all validators that run Avado as per identified by their graffiti. We can see that after The Merge there was a spike in the amount of wrong votes, to 22% and then stabilized to 20%. We can clearly see how the rewards were impacted a bit and few hours later we had 3 missed block proposals.

Methodology

• All data is fetched directly from the beacon chain, using a Teku as Consensus Client and Geth as Execution Client.

• All metrics are calculated on different groups of validators every epoch, which translates to datapoints every six minutes.

• Validators are grouped per exchange, pool or solo staker according to different criteria. Some operators publicly recognise the validators they run. In other cases we rely on onchain analytics and different heuristics to label them. For solo stakers such as DappNode and Avado, we filter them out based on the graffiti they use.

• The software eth-metrics is used to calculate all the metrics, and were exported into csv through Grafana for its analysis.

• Python was used for the data analysis and plots. Note that the % of missed votes is a moving average of 10 epochs.

• All these data and more can be accessed live at ethereumpools.info.

Disclaimer:

While the analysis was done with strict scientific rigor, bear in mind that some of the validators labeling might be wrong or incomplete. Note that it was crosschecked with other members of the community, but its imposible to have a 100% accurate estimation.

We are an independent entity that is not affiliated with any of the companies, operators or pools analysed here. If you think any data might be wrong, feel free to challenge us.

Conclusions

• We presented an analysis of how different staking entities performed during 12 hours before and after The Merge. While this is an interesting analysis, don’t use it to judge a given entity. Real-time data with the latest performance can be found at ethereumpools.info. Bad performance during this transition mean nothing in the long term.

• Its interesting to see that groups of solo stakers such as DappNode have a really good performance, taking into account that their nodes are not operated by profesionals in the cloud, but by individuals usually running the setup at their homes.

• On the other hand, we can see that even profesional operators like the ones Lido has can have issues that take hours to be resolved.

• The Merge went great, and you can refer to official sources for a different analysis on clients level rather than pools.

• Needless to say, there was no slashing during The Merge, something that some people were really afraid of.

• Performance over short periods of time is cool, but what really matter is a constant okey-ish performance month after month. Bear in mind when setting your KPIs, specially if you are a solo staker.

• Solo stakers performed great during The Merge, specially when compared to some profesional operators.

For more on real-time Ethereum validator monitoring

• 🐦 Follow us on Twitter.

• 📉 Check our dashboard.

Al. Rev.