Note: Originally published on September 7, 2023 on Mirror.
We are on the cusp of a revolutionary shift in the Ethereum landscape. Applications once considered unfeasible due to the network's limited throughput are about to become viable. The tradeoff between scalability, decentralization, and security, known as the blockchain trilemma, has long been a characteristic of blockchain networks. Ethereum chose to prioritize decentralization and security, amassing nearly 1 million validators and making a 51% attack prohibitively expensive (it would cost over $40 billion). However, this focus came at the cost of scalability. During peak usage, Ethereum transaction fees have soared to over $300. Even in the depths of a bear market, interacting with a smart contract could cost several dollars. Vitalik Buterin once famously stated that "the internet of money should not cost more than 5 cents per transaction." Fortunately, Ethereum has been striving for scalable solutions since its inception.
The Ethereum Foundation has been contemplating various sharding designs since 2016, with the first phase of data sharding slated for implementation in the coming months. According to Justin Drake, a researcher at the Ethereum Foundation, Layer 2 rollups may soon be so inexpensive that companies operating their sequencers could absorb the costs, making transactions free for users.
Note: Assuming you're familiar with how rollups work, this article will dig into Proto-Danksharding (PDS), its role in scaling Ethereum, and why its implications are not yet fully understood. If you're not familiar with rollups, you may want to pause and read my earlier post on the topic here.
Ethereum's final form envisions a data layer where rollups can post temporary data, secure in the knowledge that a global network of nodes and cryptoeconomic security protecting it. The word temporary might have made you pause, but yes, the days of a freshly initiated Ethereum node having access to the complete transaction history are numbered. PDS introduces a new space for rollups to post data and the pruning historical data is crucial for achieving low cost rollup fees and is a feature of PDS. This data pruning occurs after approximately 18 days (4096 epochs), which is more than adequate for both major types of rollups to fully function. Specifically, optimistic rollups only require data to be available for one week to finalize the state, while zk-rollups allow for immediate fund withdrawals. This 18 day period ensures that anyone can download the data if they want to. Additionally, it's worth noting that preserving historical data operates on a 1-of-n assumption, meaning you only need one entity among all possible options, be it rollups or block explorers, to retain this data to make it accessible in the future.
Currently rollups use calldata to post their data, and each Ethereum block contains about 10 KB used for calldata. In PDS, each message of data that a rollup wants to post is termed a blob. Each blob holds approximately 125 KB of data. With a targeted inclusion of 3 blobs per block and a maximum capacity of 6 blobs, this implies that on average, each block will contain about 375 KB of data. This is a substantial increase, a 37.5x jump, compared to the 10 KB used for calldata today and it's worth noting that not all of that calldata is utilized by rollups. The reason for this dramatic data increase, without significantly ramping up hardware requirements, lies in the transitory nature of data storage. Nodes will need slightly more storage to store these data blobs for the 18 days, but the storage doesn't perpetually expand with the progression of the chain due to the pruning mechanism.
In PDS, a new transaction type is introduced within the execution client (EVM), known as a blob-carry transaction. When rollups aim to post new data, they must perform two separate steps. First, they initiate a blob-carry transaction that includes the conventional transaction components like the sender, receiver, nonce, and gas bid along with two new fields specifically designed to facilitate data posting. The first of these new fields is max_fee_per_blob_gas, which is the bid that the sender is willing to pay to post their data blob. The second field, called blob_version_hashes, is essentially a list of hashes of the blobs to be posted. A single transaction can post multiple blobs.
These data blobs are a new transaction type on the execution chain. However, they don't add extra tasks for the execution process. The EVM looks only at the commitment tied to these blobs. In PDS, consensus clients fully download these data blobs. Instead of being fully packed into the Beacon block body, the blobs are simply referenced there. Their contents are sent separately, like a “sidecar” accompanying the main data. Each block in PDS has one such blob sidecar that is downloaded in full.
Instead of using a straightforward hashing approach to commit to the data, PDS employs a more intricate process. The data is first transformed into a polynomial, and then a Kate commitment (KZG) is generated for it. Finally, this commitment is hashed. While this procedure might sound complicated, it's not merely for robust cryptographic assurance. It's a necessary step for future upgrades aimed at facilitating Danksharding. The polynomial and KZG processes lay the groundwork for these upcoming features. If you're intrigued by the specifics of these cryptographic techniques, additional resources will be provided at the end of this article.
Sample Blob-Carry Transaction
{
"transaction_type": "blob-carry",
"sender": "0xAbcd1234...",
"receiver": "0xDef5678...",
"nonce": 42,
"gas_bid": 100000,
"max_fee_per_blob_gas": 2000,
"blob_version_hashes": [ "0x019f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a", "0x011679091c5a880faf6fb5e6087eb1b3c3a77f5a4d7f8f0a4d4a478e0484368f"
],
"signature": "0x1f67d0..."
} PDS introduces a separate fee market exclusively for blobs, independent of the primary Ethereum fee market. This means that if the main Ethereum layer is congested, let's say due to a surge in transactions for the latest meme coin AnimeButtPirate69Inu, the blob fee market remains unaffected. Blob pricing follows a mechanism resembling EIP-1559, using a unit called blob gas. The cost of blob gas is dynamically calculated based on a running tally of excess blobs, those exceeding the target of 3 blobs per block. While each block aims to include 3 blobs, there's a hard limit of 6 blobs per block. A header field in each block keeps track of how many blobs have been included over or under the target. This tally then serves as an input to an exponential function that determines the price of blob gas for the subsequent block. In essence, the function is designed to make it increasingly costly to include additional blobs, helping to maintain an average close to the intended target of 3 blobs per block.
Let's consider a simplified Ethereum universe where we have just 5 blocks to look at:
3 blobs included
Running Tally of Excess Blobs: 0 (since it matches the target of 3)
Blob Gas Price: Low (determined by a function of the tally, which is 0)
4 blobs included
Running Tally of Excess Blobs: 1 (1 more than the target)
Blob Gas Price: Slightly Higher (the tally is now 1, so the function increases the price a bit)
6 blobs included (the maximum burst limit)
Running Tally of Excess Blobs: 4 (3 more than the target, adding to the existing 1)
Blob Gas Price: Higher (now the tally is 4, and the function makes it much more expensive)
2 blobs included
Running Tally of Excess Blobs: 3 (1 less than the target, reducing the existing 4)
Blob Gas Price: Moderately High (the tally is reduced to 3, so the function adjusts the price downward but it's still high)
1 blob included
Running Tally of Excess Blobs: 1 (2 less than the target, reducing the existing 3)
Blob Gas Price: Lower (the tally is now 1, so the function makes it cheaper)
PDS introduces an entirely new resource, complete with its own separate fee market. Initially, this resource is unlikely to be fully consumed, leading to exceptionally low fees required for rollups to access it. As a result, transactions could potentially be free for the first 6 to 12 months following its introduction. This is because posting a blob may be so inexpensive that rollups can afford to subsidize the fees for their users.
The impact of PDS goes far beyond just making transactions cheaper. This isn't some minor adjustment, it’s unlocking a whole new level of possibilities on the blockchain. Think about it: tipping someone a few cents for a helpful online comment or leaving a review (attestation) could become as easy as clicking a button. Artists won't have to worry about high fees when they create new NFTs. But that's just scratching the surface. PDS makes it possible for completely new types of apps to exist. Decentralized social media platforms could become far more user friendly, removing the financial barriers that kept people away. Real time voting systems on the blockchain could turn into a reality, making decentralized governance more feasible than ever. Even in the world of finance, we could see the rise of platforms for high frequency trading that are fully decentralized. Remarkably, all of this can be achieved on a highly decentralized network, safeguarded by millions globally including solo stakers and full nodes operating on devices as simple as Raspberry Pis. Just a short while ago, such a vision was deemed unattainable.
I often reflect on a talk by Vitalik where he emphasizes the same point I'm highlighting. We should begin conceptualizing and creating apps today that might seem impractical now, but will become viable in a few months when transactions become 100x (or more or free) affordable.
But let's not forget, this is just the beginning. PDS is a stepping stone towards the ambitious goal of full danksharding, which aims to further increase the amount of data that can be posted. In essence, some of the promises of blockchain technology that seemed too ambitious in 2017 and 2018 may have simply been ahead of their time.
Introduction
Ethereum's roadmap is a lot to keep track of. This is your technical crash-course on everything from danksharding to KZGs to statelessness and so much more.
Proto-Danksharding FAQ [TOC] ## What is Danksharding? Danksharding is the new sharding design p
Over 100 subscribers
