I have been diving into Solana and its Validator code realizing the incredible design behind the selection of Validators and block producers, so here is a small rundown

• What are Blocks?

• What are POW and POS?

• Solana Leader Schedule

• Special Considerations to produce the Tastiest Block

• Open Questions

• Some Resources

Blocks containing transactions and state transition data lined on top of one another with cryptography securing their computational integrity is the foundation of blockchain aka chain of blocks.

Bitcoin was a breakthrough in this architecture where distributed parties could mine blocks by brute forcing and computing the hash for the block, the miner getting the hash got the reward and built the block. Because of this bitcoin was a pure game of computational resources where the miners with higher hash power or better hardware had >>> chances than any normal user. This was a great security model as if any malicious party tried to overturn a hashed block or change the history they would have to challenge the compute resources of the whole blockchain or wanted to add a rogue transaction in a future block they would have to build blocks faster than everyone combined which was kinda impossible giving bitcoin the better security.

Though this is a great model it’s not very feasible if you wanna get a more decentralized system without draining immense amounts of energy and resources that’s where POS was pioneered.

In Proof of Stake, a builder is selected ahead of schedule to build a block and only if they have staked some amount of native token to get "Skin in the game". It’s a basic game of reinforcement learning where there is positive feedback aka block reward if they build a block with right hashes and get their stake slashed if they do any malicious activity that hurts the chain. Proof of stake systems is just a solution to the byzantine problem of how decentralized parties could coordinate without trusting much. As in a proof-of-work system like Bitcoin, the chain is commanded by hash power, POS system is stake controlled

• If 1/3rd or 33% of the stake goes rogue then user transactions can be censored and the state changes be halted

• To roll back history and append hashed state changes it requires 2/3rd (67%) of the stake.

In Solana validators that stake Solana and build blocks are called Leaders that have a leader schedule based on which they build blocks. Leaders aka validators are selected based on the amount of their stake aka stake weight and the higher it is more are the chances of a staker being picked to build up a block.

Because Solana is so fast it needs to track leaders and time well so that blocks are not delayed or become stale. Solana ledger has its own internal clock which maps the time it takes to run an iterative SHA-256 hash function per second, as a minimal ceiling has been reached on how fast SHA256 can be hashed it gives us a uniform time for how long it takes for a general machine to generate one hash.

// GCP n1-standard hardware and also a xeon e5-2520 v4 are about this rate of hashes/s
pub const DEFAULT_HASHES_PER_SECOND: u64 = 2_000_000;

On every hash they produce a tick and these ticks can vary from machine to machine but to the ledger, the ticks are the main source of time and 2 ticks look the same.
There are 12,500 hashes in every single tick and 2 million hashes per second giving 800k hashes for every single block.
2,000,000/12500 = 160 ticks per second

The leader schedule is created for each epoch which is ~2 days. Solana has block slots of 400 milliseconds making 432,000 slots in a single epoch. During each slot, the leader gets 0.4s seconds to build the block and submit for voting before the schedule shifts for the next leader but every leader gets a pack of 4 slots, the time and schedule just passes even if the leader doesn't submit the block. Built blocks are turned into shreds and sent over Turbine, and the transactions are executed before they are voted upon for validity that’s why it’s called Optimistic execution freeing up a lot of time from back-and-forth gossip.

Dive into the clock code

Transactions have 3 simple parts →

1. Signatures → Ed25519 signature which verifies that the transaction is sent by user holding the private key + the signature from validators

2. Blockhash → To make a transaction, wallet pulls the latestBlockHash and uses it prove that the transaction is recent, transactions containing blockhash older than 152 blocks is rejected so if you fetch a block hash now it takes around 60 seconds for it become stale.

3. Instructions → Solana utilising the parallel execution needs all the programs and invocations prehand so all the cpis and programs + their addresses tuched by your transaction need to be specified before hand.

**But what about invalid transactions or out-of-bound txns?**If the transactions are incompatible they will fail anyway during their bytecode compilation in the Berekly packer filter virtual machine and safely halt the execution of that specific contract without hurting other transactions in that block.

Leader nodes "timestamp" blocks with cryptographic proofs that some duration of time has passed since the last proof

There are no limits to the no. of slots a single leader could be selected for, even if a validator has <50% stake there are chances could build more than half of all the blocks in the epoch because of the nature of the schedule.

Now lets dive into some codeeeee

impl LeaderSchedule {
    // Note: passing in zero stakers will cause a panic.
    pub fn new(ids_and_stakes: &[(Pubkey, u64)], seed: [u8; 32], len: u64, repeat: u64) -> Self {
        let (ids, stakes): (Vec<_>, Vec<_>) = ids_and_stakes.iter().cloned().unzip();
        let rng = &mut ChaChaRng::from_seed(seed);
        let weighted_index = WeightedIndex::new(stakes).unwrap();
        let mut current_node = Pubkey::default();
        let slot_leaders = (0..len)
            .map(|i| {
                if i % repeat == 0 {
                    current_node = ids[weighted_index.sample(rng)];
                }
                current_node
            })
            .collect();
        Self::new_from_schedule(slot_leaders)
    }

This code block shows how the leader's schedule is generated. As on the top, each validator’s id and stake are serialized into vectors and then a weighted index is created to calculate the stake weight of validators.

If you want to learn more about Solana Block production feel free to check out →

https://madhavgoyal.substack.com/p/async-program-execution-and-simd

https://www.umbraresearch.xyz/writings/lifecycle-of-a-solana-transaction

Block production and propagation is a complex process involving several parameters and considerations to achieve the best outcome.

Leaders have to optimize for the Uncle rate which is a parameter of the time it takes to build & forward a block and reach a consensus on it, the higher the Uncle rate higher the chances of the block becoming stale and not accepted by other nodes on the network. While Earlier a single Leader was producing and hashing the block with the implementation of PBS there are N of builders and even more searchers competing to get their blocks included and if the builders with the highest gas fee are farther it could lead to higher latency. Leaders have to keep

• Priority fee and Block reward from the block inclusion

• Latency from the block builder

• Price of inclusion of transactions as if they get too greedy and try to add too many transactions in the same block it could lead to longer propagation time and higher uncle rate

  • Block_size is directly proportional to Stale_blockrate

Decentralization →

https://twitter.com/Madhav_goyal_/status/1661225300825460736

so block builders need to find the sweet spot where the size of the block is good right and the gas fee is attractive enough for the leader. Solana's data throughput is upwards of 100kb per second so this also loosens up a bit for better block packaging.

As transaction order flow and supply chain gets more completed with time due to the introduction of new actors using complex strategies to extract profit and self-interests, this could change a lot and there are already new Protocols in work to improve the leader schedules on Ethereum to maintain the anonymity of the builders Solana has high chances of following the footsteps that’s why nothing is concrete even the most fundamental parts of a blockchain.

PS: Thank you for reading till the end! 🫡Added some resources if you wanna dive deeper and check out ideas

Resources used->

• Biggest inspiration for this post https://www.shinobi-systems.com/primer.html

• https://github.com/solana-labs/solana/blob/master/ledger/src/leader_schedule.rs

• https://github.com/solana-labs/solana/blob/master/ledger/src/leader_schedule_utils.rs

• https://docs.solana.com/ar/cluster/synchronization#relationship-to-vdfs

• https://docs.solana.com/cluster/turbine-block-propagation

I have been diving into Solana and its Validator code realizing the incredible design behind the selection of Validators and block producers, so here is a small rundown

• What are Blocks?

• What are POW and POS?

• Solana Leader Schedule

• Special Considerations to produce the Tastiest Block

• Open Questions

• Some Resources

Blocks containing transactions and state transition data lined on top of one another with cryptography securing their computational integrity is the foundation of blockchain aka chain of blocks.

Bitcoin was a breakthrough in this architecture where distributed parties could mine blocks by brute forcing and computing the hash for the block, the miner getting the hash got the reward and built the block. Because of this bitcoin was a pure game of computational resources where the miners with higher hash power or better hardware had >>> chances than any normal user. This was a great security model as if any malicious party tried to overturn a hashed block or change the history they would have to challenge the compute resources of the whole blockchain or wanted to add a rogue transaction in a future block they would have to build blocks faster than everyone combined which was kinda impossible giving bitcoin the better security.

Though this is a great model it’s not very feasible if you wanna get a more decentralized system without draining immense amounts of energy and resources that’s where POS was pioneered.

In Proof of Stake, a builder is selected ahead of schedule to build a block and only if they have staked some amount of native token to get "Skin in the game". It’s a basic game of reinforcement learning where there is positive feedback aka block reward if they build a block with right hashes and get their stake slashed if they do any malicious activity that hurts the chain. Proof of stake systems is just a solution to the byzantine problem of how decentralized parties could coordinate without trusting much. As in a proof-of-work system like Bitcoin, the chain is commanded by hash power, POS system is stake controlled

• If 1/3rd or 33% of the stake goes rogue then user transactions can be censored and the state changes be halted

• To roll back history and append hashed state changes it requires 2/3rd (67%) of the stake.

In Solana validators that stake Solana and build blocks are called Leaders that have a leader schedule based on which they build blocks. Leaders aka validators are selected based on the amount of their stake aka stake weight and the higher it is more are the chances of a staker being picked to build up a block.

Because Solana is so fast it needs to track leaders and time well so that blocks are not delayed or become stale. Solana ledger has its own internal clock which maps the time it takes to run an iterative SHA-256 hash function per second, as a minimal ceiling has been reached on how fast SHA256 can be hashed it gives us a uniform time for how long it takes for a general machine to generate one hash.

// GCP n1-standard hardware and also a xeon e5-2520 v4 are about this rate of hashes/s
pub const DEFAULT_HASHES_PER_SECOND: u64 = 2_000_000;

On every hash they produce a tick and these ticks can vary from machine to machine but to the ledger, the ticks are the main source of time and 2 ticks look the same.
There are 12,500 hashes in every single tick and 2 million hashes per second giving 800k hashes for every single block.
2,000,000/12500 = 160 ticks per second

The leader schedule is created for each epoch which is ~2 days. Solana has block slots of 400 milliseconds making 432,000 slots in a single epoch. During each slot, the leader gets 0.4s seconds to build the block and submit for voting before the schedule shifts for the next leader but every leader gets a pack of 4 slots, the time and schedule just passes even if the leader doesn't submit the block. Built blocks are turned into shreds and sent over Turbine, and the transactions are executed before they are voted upon for validity that’s why it’s called Optimistic execution freeing up a lot of time from back-and-forth gossip.

Dive into the clock code

Transactions have 3 simple parts →

1. Signatures → Ed25519 signature which verifies that the transaction is sent by user holding the private key + the signature from validators

2. Blockhash → To make a transaction, wallet pulls the latestBlockHash and uses it prove that the transaction is recent, transactions containing blockhash older than 152 blocks is rejected so if you fetch a block hash now it takes around 60 seconds for it become stale.

3. Instructions → Solana utilising the parallel execution needs all the programs and invocations prehand so all the cpis and programs + their addresses tuched by your transaction need to be specified before hand.

**But what about invalid transactions or out-of-bound txns?**If the transactions are incompatible they will fail anyway during their bytecode compilation in the Berekly packer filter virtual machine and safely halt the execution of that specific contract without hurting other transactions in that block.

Leader nodes "timestamp" blocks with cryptographic proofs that some duration of time has passed since the last proof

There are no limits to the no. of slots a single leader could be selected for, even if a validator has <50% stake there are chances could build more than half of all the blocks in the epoch because of the nature of the schedule.

Now lets dive into some codeeeee

impl LeaderSchedule {
    // Note: passing in zero stakers will cause a panic.
    pub fn new(ids_and_stakes: &[(Pubkey, u64)], seed: [u8; 32], len: u64, repeat: u64) -> Self {
        let (ids, stakes): (Vec<_>, Vec<_>) = ids_and_stakes.iter().cloned().unzip();
        let rng = &mut ChaChaRng::from_seed(seed);
        let weighted_index = WeightedIndex::new(stakes).unwrap();
        let mut current_node = Pubkey::default();
        let slot_leaders = (0..len)
            .map(|i| {
                if i % repeat == 0 {
                    current_node = ids[weighted_index.sample(rng)];
                }
                current_node
            })
            .collect();
        Self::new_from_schedule(slot_leaders)
    }

This code block shows how the leader's schedule is generated. As on the top, each validator’s id and stake are serialized into vectors and then a weighted index is created to calculate the stake weight of validators.

If you want to learn more about Solana Block production feel free to check out →

https://madhavgoyal.substack.com/p/async-program-execution-and-simd

https://www.umbraresearch.xyz/writings/lifecycle-of-a-solana-transaction

Block production and propagation is a complex process involving several parameters and considerations to achieve the best outcome.

Leaders have to optimize for the Uncle rate which is a parameter of the time it takes to build & forward a block and reach a consensus on it, the higher the Uncle rate higher the chances of the block becoming stale and not accepted by other nodes on the network. While Earlier a single Leader was producing and hashing the block with the implementation of PBS there are N of builders and even more searchers competing to get their blocks included and if the builders with the highest gas fee are farther it could lead to higher latency. Leaders have to keep

• Priority fee and Block reward from the block inclusion

• Latency from the block builder

• Price of inclusion of transactions as if they get too greedy and try to add too many transactions in the same block it could lead to longer propagation time and higher uncle rate

  • Block_size is directly proportional to Stale_blockrate

Decentralization →

https://twitter.com/Madhav_goyal_/status/1661225300825460736

so block builders need to find the sweet spot where the size of the block is good right and the gas fee is attractive enough for the leader. Solana's data throughput is upwards of 100kb per second so this also loosens up a bit for better block packaging.

As transaction order flow and supply chain gets more completed with time due to the introduction of new actors using complex strategies to extract profit and self-interests, this could change a lot and there are already new Protocols in work to improve the leader schedules on Ethereum to maintain the anonymity of the builders Solana has high chances of following the footsteps that’s why nothing is concrete even the most fundamental parts of a blockchain.

PS: Thank you for reading till the end! 🫡Added some resources if you wanna dive deeper and check out ideas

Resources used->

• Biggest inspiration for this post https://www.shinobi-systems.com/primer.html

• https://github.com/solana-labs/solana/blob/master/ledger/src/leader_schedule.rs

• https://github.com/solana-labs/solana/blob/master/ledger/src/leader_schedule_utils.rs

• https://docs.solana.com/ar/cluster/synchronization#relationship-to-vdfs

• https://docs.solana.com/cluster/turbine-block-propagation