It is not news that the Ethereum ecosystem lacks diversity in terms of both, execution and consensus client usage. Decentralized ecosystems do not have an authorized centralized point which can govern the whole network. Therefore, it is our intention to encourage users to use minority software clients, in order to build a more diverse and robust ecosystem.

Usually, once the setup is ready and all validators are running correctly, users are not encouraged to change to another client, as it may require additional effort and troubleshooting, specially when economic incentives play an important role in the formula.

With this report, we aim to describe and document how to migrate the Eth2 validators across the 6 main clients. We have performed two experiments which we hope brings some more light and helps anyone out there better understand the process of migrating validators from one client to another in an automated way.

This experiment is meant to demonstrate that it is possible to transition validators between the 6 main clients in an automated way.

For this experiment we have used a Raspberry PI 4b with 8GB RAM, connected via wired Internet connection. As storage, we have usFed a 256 GB class A Samsung microSD card and a 248 GB SSD.

The study was performed in the Kiln network with 250 validators.

https://hackmd.io/@philknows/By5qahdZc

https://ethereum-on-arm-documentation.readthedocs.io/en/latest/kiln/kiln-testnet.html#

https://github.com/migalabs/eth2-client-analyzer/tree/feature/validator_management/validator_management

https://web.postman.co/workspace/My-Workspace~888f987a-8e6c-4be7-886a-eb9031148c67/request/5618734-14680f42-dffc-4d58-a67b-85a55e0562ae

https://hackmd.io/@xXnoWNBFTWGjXn1S1bpFNA/B153kW19c

Once we have our mnemonics we can generate the validator keys and secrets, which will allow us to import all our validators in each of the clients.

It is important to mention that the key generation process can be done in a bunch of different ways. In this case we have used eth2-val-tools, which already creates keys and secrets in the corresponding format for each client. But there are other alternatives like staking-deposit-cli.

The slashing protection is a database containing the list of performed duties per validator per epoch. This is meant to be imported in the next client, so as to be aware of the already performed duties, which should reduce the slashing risk during the migration of validators.

In order to test the above described use case, we have synced all 6 main clients to the head of the Kiln network and we have imported the same 250 validators in all of them. To do this:

1. Sync the next client to try and stop it.

2. Import the recently exported slashing protection db from the previous client.

3. Import 250 validators.

4. Run both beacon and validator node for 10 epochs, let the validators perform their duties.

5. Stop the beacon and validator node and export slashing protection db.

6. Back to point 1.

Migrating the validators was only done the first time, as after this they were already imported. Syncing to the head of the blockchain only takes long the first time, but after this it just takes some minutes, depending on the client.

We have let each client run during 10 epochs, so there is enough time for them to sync to the head of the chain and continue attesting.

All clients, both the beacon node and validator node, were running as services, so we did not have to worry about any client crashing and remaining stopped.

In order to perform all the below described actions, we have used each client’s execution command parameters, following each team’s additional indications when needed. It is important to mention that these experiments can be done in many different ways which could mean that some extra steps that we performed may not be needed.

As a first approach, we wanted to understand the whole process and test how each client would behave when managing validator keys. Therefore, we have manually run each client, imported/exported both validator keys and slashing protection, and run the next client. After every time we successfully imported them, we have let the client run and check that indeed all validators are active and attesting.

This first attempt helped us develop all the scripts and folder structure to perform the below explained experiments. During this whole process, we have been in contact with the developer teams in case there was an issue or error we could not solve.

After the first approach, we had successfully migrated all 250 validators across all clients with no more big issues than reading documentation and a little troubleshooting of each client’s errors.

It is worth mentioning that, for safety reasons, it is recommended to wait at least 2 epochs before running your validators in the next client, so as to avoid performing duties twice, resulting in a possible slashing.

Therefore, for this first experiment, we have not only imported the slashing protection from the previous client, but also waited the suggested time of 2 epochs, showing the recommended scenario when migrating your validators.

The goal was to show that it is possible that 250 validators run in the 6 main clients, one after another, without the need of human interaction. This should demonstrate that all 6 main clients have implemented the needed features to migrate from / to the other ones.

In Figure 2 we can observe the rewards of the 250 validators during the experiment 1. We can appreciate that rewards are very similar across all clients, which proves that all validators continued attesting as expected.

We can also observe that after each transition, we still get rewards from the previous duties and then a gap, as a result of stopping 2 epochs between each migration.

In order to test the effectiveness of the slashing protection, we have performed a second experiment where we do not wait the recommended two epochs, but we migrate straight away and probably run the validators before 2 epochs (the client needs to sync to the head).

This experiment does not represent the recommended scenario, but proves that importing the slashing protection avoids the next client executing duties twice.

In Image 3 we can observe the rewards of the 250 validators during the experiment 2. We can appreciate that rewards are very similar across all clients, which proves that all validators continued attesting as expected. However, this time we can observe there is no gap between each migration, as during this experiment we did not wait between each transition.

It is important to mention that, even though we have used the client’s command parameters to perform the experiments, there are other alternatives to run your validators in different clients.

In an effort to standarize validator key manager API, the client developer teams have implemented a standard API which enables a common way to handle validator keys across all clients. We must mention that this API is not yet implemented by Grandine.

We have again performed a first approach to import validators and slashing protection, but using the API in this case. We must mention that all clients expose this standard API but it is usually disabled by default, so it needs to be activated by adding a specific parameter to the command line execution, plus in some cases there are still methods in development.

Table 1: Pros and cons of the Key Manager API

It is important to mention that a JSON file must be created with a specific command, whose content is the output of the key generation tool. Therefore, the keystore files cannot be used directly in the API. However, building this JSON file can be automated through a custom script.

There are also alternatives that abstract the signing layer, like the Web3 signer or Vouch, and have all the needed implementations to sign the requested duties and manage the validators. However, they are out of scope for the proposed experiments as the goal was to show that it is easy to transition from one client to another.

During both experiments we encountered several minor issues.

First of all, we were unable to import the slashing protection from Lodestar in any of the clients, possibly due to a version issue. We have communicated this to the Lodestar team and we are investigating the root cause of it.

Also, we were unable to use the Eth2 Key Manager API with Teku and Lodestar, whose teams we are in discussions with to figure out the solution.

In many cases, the Eth2 Key Manager API could not handle importing a large number of validator and the client would crash.

After performing the described experiments, we have come to the conclusion that it is possible to migrate validators from one client to another in an automated way.

It is worth mentioning that some documentation improvements are needed and also the Eth2 Key Manager API needs some more development to make it easier for the user, but we must say we are impressed by the effort all clients teams have done to standardize both the import and export of validators and slashing protection.

We were able not only able to migrate validators in an automated way, but without the need to alter or create new files, as the same export format from one client could be used as an input in the next one. This not only makes users experience much smoother, but also transmits more confidence, as the user can completely trust each clients format.

It is not news that the Ethereum ecosystem lacks diversity in terms of both, execution and consensus client usage. Decentralized ecosystems do not have an authorized centralized point which can govern the whole network. Therefore, it is our intention to encourage users to use minority software clients, in order to build a more diverse and robust ecosystem.

Usually, once the setup is ready and all validators are running correctly, users are not encouraged to change to another client, as it may require additional effort and troubleshooting, specially when economic incentives play an important role in the formula.

With this report, we aim to describe and document how to migrate the Eth2 validators across the 6 main clients. We have performed two experiments which we hope brings some more light and helps anyone out there better understand the process of migrating validators from one client to another in an automated way.

This experiment is meant to demonstrate that it is possible to transition validators between the 6 main clients in an automated way.

For this experiment we have used a Raspberry PI 4b with 8GB RAM, connected via wired Internet connection. As storage, we have usFed a 256 GB class A Samsung microSD card and a 248 GB SSD.

The study was performed in the Kiln network with 250 validators.

https://hackmd.io/@philknows/By5qahdZc

https://ethereum-on-arm-documentation.readthedocs.io/en/latest/kiln/kiln-testnet.html#

https://github.com/migalabs/eth2-client-analyzer/tree/feature/validator_management/validator_management

https://web.postman.co/workspace/My-Workspace~888f987a-8e6c-4be7-886a-eb9031148c67/request/5618734-14680f42-dffc-4d58-a67b-85a55e0562ae

https://hackmd.io/@xXnoWNBFTWGjXn1S1bpFNA/B153kW19c

Once we have our mnemonics we can generate the validator keys and secrets, which will allow us to import all our validators in each of the clients.

It is important to mention that the key generation process can be done in a bunch of different ways. In this case we have used eth2-val-tools, which already creates keys and secrets in the corresponding format for each client. But there are other alternatives like staking-deposit-cli.

The slashing protection is a database containing the list of performed duties per validator per epoch. This is meant to be imported in the next client, so as to be aware of the already performed duties, which should reduce the slashing risk during the migration of validators.

In order to test the above described use case, we have synced all 6 main clients to the head of the Kiln network and we have imported the same 250 validators in all of them. To do this:

1. Sync the next client to try and stop it.

2. Import the recently exported slashing protection db from the previous client.

3. Import 250 validators.

4. Run both beacon and validator node for 10 epochs, let the validators perform their duties.

5. Stop the beacon and validator node and export slashing protection db.

6. Back to point 1.

Migrating the validators was only done the first time, as after this they were already imported. Syncing to the head of the blockchain only takes long the first time, but after this it just takes some minutes, depending on the client.

We have let each client run during 10 epochs, so there is enough time for them to sync to the head of the chain and continue attesting.

All clients, both the beacon node and validator node, were running as services, so we did not have to worry about any client crashing and remaining stopped.

In order to perform all the below described actions, we have used each client’s execution command parameters, following each team’s additional indications when needed. It is important to mention that these experiments can be done in many different ways which could mean that some extra steps that we performed may not be needed.

As a first approach, we wanted to understand the whole process and test how each client would behave when managing validator keys. Therefore, we have manually run each client, imported/exported both validator keys and slashing protection, and run the next client. After every time we successfully imported them, we have let the client run and check that indeed all validators are active and attesting.

This first attempt helped us develop all the scripts and folder structure to perform the below explained experiments. During this whole process, we have been in contact with the developer teams in case there was an issue or error we could not solve.

After the first approach, we had successfully migrated all 250 validators across all clients with no more big issues than reading documentation and a little troubleshooting of each client’s errors.

It is worth mentioning that, for safety reasons, it is recommended to wait at least 2 epochs before running your validators in the next client, so as to avoid performing duties twice, resulting in a possible slashing.

Therefore, for this first experiment, we have not only imported the slashing protection from the previous client, but also waited the suggested time of 2 epochs, showing the recommended scenario when migrating your validators.

The goal was to show that it is possible that 250 validators run in the 6 main clients, one after another, without the need of human interaction. This should demonstrate that all 6 main clients have implemented the needed features to migrate from / to the other ones.

In Figure 2 we can observe the rewards of the 250 validators during the experiment 1. We can appreciate that rewards are very similar across all clients, which proves that all validators continued attesting as expected.

We can also observe that after each transition, we still get rewards from the previous duties and then a gap, as a result of stopping 2 epochs between each migration.

In order to test the effectiveness of the slashing protection, we have performed a second experiment where we do not wait the recommended two epochs, but we migrate straight away and probably run the validators before 2 epochs (the client needs to sync to the head).

This experiment does not represent the recommended scenario, but proves that importing the slashing protection avoids the next client executing duties twice.

In Image 3 we can observe the rewards of the 250 validators during the experiment 2. We can appreciate that rewards are very similar across all clients, which proves that all validators continued attesting as expected. However, this time we can observe there is no gap between each migration, as during this experiment we did not wait between each transition.

It is important to mention that, even though we have used the client’s command parameters to perform the experiments, there are other alternatives to run your validators in different clients.

In an effort to standarize validator key manager API, the client developer teams have implemented a standard API which enables a common way to handle validator keys across all clients. We must mention that this API is not yet implemented by Grandine.

We have again performed a first approach to import validators and slashing protection, but using the API in this case. We must mention that all clients expose this standard API but it is usually disabled by default, so it needs to be activated by adding a specific parameter to the command line execution, plus in some cases there are still methods in development.

Table 1: Pros and cons of the Key Manager API

It is important to mention that a JSON file must be created with a specific command, whose content is the output of the key generation tool. Therefore, the keystore files cannot be used directly in the API. However, building this JSON file can be automated through a custom script.

There are also alternatives that abstract the signing layer, like the Web3 signer or Vouch, and have all the needed implementations to sign the requested duties and manage the validators. However, they are out of scope for the proposed experiments as the goal was to show that it is easy to transition from one client to another.

During both experiments we encountered several minor issues.

First of all, we were unable to import the slashing protection from Lodestar in any of the clients, possibly due to a version issue. We have communicated this to the Lodestar team and we are investigating the root cause of it.

Also, we were unable to use the Eth2 Key Manager API with Teku and Lodestar, whose teams we are in discussions with to figure out the solution.

In many cases, the Eth2 Key Manager API could not handle importing a large number of validator and the client would crash.

After performing the described experiments, we have come to the conclusion that it is possible to migrate validators from one client to another in an automated way.

It is worth mentioning that some documentation improvements are needed and also the Eth2 Key Manager API needs some more development to make it easier for the user, but we must say we are impressed by the effort all clients teams have done to standardize both the import and export of validators and slashing protection.

We were able not only able to migrate validators in an automated way, but without the need to alter or create new files, as the same export format from one client could be used as an input in the next one. This not only makes users experience much smoother, but also transmits more confidence, as the user can completely trust each clients format.