The Indexing CompanyIndexing Mirror.xyz
If you read content online you’ve probably at least heard of publishing services like Medium and Substack. These are centralized, web2 companies that make money with views; subscriptions, ads, etc. Thankfully, as we transition into the web3 space we are already seeing some promising alternatives. The largest of these web3 publishers is Mirror. The beauty of protocols like Mirror is that they don’t own any of the data. They still have a login, a clean text editor, and shareable links just like...
The Indexing CompanyAccessing Data 3.0: Indexing 101
This is an entry in our long running series, “Accessing Data 3.0”, where we talk about the “whats” and the “hows” of working with data in web3. Enjoy! Remember libraries? The walls of books and the fearless librarians somehow always knowing exactly where everything is. Well, two things: 1) libraries still exist, 2) those libraries are each indexed. Librarians around the world categorize all of the books under their purview into what are known as a “library catalogs”. These catalogs serve as a...
The Indexing CompanyAccessing Data 3.0: Storage Options
This is an entry in our long running series, “Accessing Data 3.0”, where we talk about the “whats” and the “hows” of working with data in web3. Enjoy! There’s an often forgotten question in Data 3.0 - where do we actually put all our large data? That image of your favorite cat, videos from the last family trip, the unpublished book you’re working on - what is “home” for all that data? It’s easy to think “well if it’s web3, then it must be on-chain”, but that’s not always true, nor does it nee...
Paving the road to Data 3.0 one index at a time
The Indexing CompanyIndexing Mirror.xyz
If you read content online you’ve probably at least heard of publishing services like Medium and Substack. These are centralized, web2 companies that make money with views; subscriptions, ads, etc. Thankfully, as we transition into the web3 space we are already seeing some promising alternatives. The largest of these web3 publishers is Mirror. The beauty of protocols like Mirror is that they don’t own any of the data. They still have a login, a clean text editor, and shareable links just like...
The Indexing CompanyAccessing Data 3.0: Indexing 101
This is an entry in our long running series, “Accessing Data 3.0”, where we talk about the “whats” and the “hows” of working with data in web3. Enjoy! Remember libraries? The walls of books and the fearless librarians somehow always knowing exactly where everything is. Well, two things: 1) libraries still exist, 2) those libraries are each indexed. Librarians around the world categorize all of the books under their purview into what are known as a “library catalogs”. These catalogs serve as a...
The Indexing CompanyAccessing Data 3.0: Storage Options
This is an entry in our long running series, “Accessing Data 3.0”, where we talk about the “whats” and the “hows” of working with data in web3. Enjoy! There’s an often forgotten question in Data 3.0 - where do we actually put all our large data? That image of your favorite cat, videos from the last family trip, the unpublished book you’re working on - what is “home” for all that data? It’s easy to think “well if it’s web3, then it must be on-chain”, but that’s not always true, nor does it nee...
Paving the road to Data 3.0 one index at a time
Subscribe to The Indexing Company
Subscribe to The Indexing Company
Share Dialog
Share Dialog
<100 subscribers
<100 subscribers
By all measures, Devcon VI was a huge success. Over 6000 participants from around the world met in Bogota, Colombia to build, network, and celebrate together in the official Ethereum conference.
This is coming about a month after The Merge in which Ethereum switched from Proof of Work (PoW) to Proof of Stake (PoS). That transition worked far better than any could have hoped and has lead to ETH even being deflationary at times 🔥
So what’s next for Ethereum and its ecosystem? In short, far too much is happening to cover in a single post, but we’re going to touch on “the state of data”.
Let’s take a step back and remember how we got to today. Ethereum launched in 2015 with a vision of being the world’s computer. At its core, Ethereum processes transactions in its Ethereum Virtual Machine (EVM) and reaches consensus with its nodes (network of servers). In order to do this, each node in the ecosystem must keep track of a history of all blocks and transactions that have ever existed.
Fast forward to today, there are well over 15 million blocks and over 1 billion transactions on the Ethereum mainnet. And these numbers don’t even reflect the growing ecosystem of secondary blockchains on and around Ethereum such as Polygon, Optimism, Arbitrum, Starknet, etc. Point being, there’s a lot of data out there and it’s only continuing to grow.
In order for Ethereum, and its ecosystem, to truly reach “internet scale”, we need to drastically increase adoption. That adoption, though, inevitably comes with a sharp increase in data and we need to be ready for this.
Focusing primarily on solving for adoption, some of the common themes in Ethereum today are:
Too few transactions per second - not enough support for simultaneous users
Too few [independent] node operators - not enough decentralization
On-chain storage is expensive - and standards are missing for off-chain
Thankfully, the sharp minds of the industry have already been working on solutions; many of which should be rolling out in the coming months and years. Let’s dig into a few of these Ethereum Improvement Proposals (EIPs):
EIP-4488: Working on Throughput
https://eips.ethereum.org/EIPS/eip-4488
The leading way to increase throughput on Ethereum is simply to move transactions off of Ethereum. This may sound counter-intuitive, but bear with me.
Layer 2’s such as Optimism and Arbitrum offer developers and consumers lower gas fees, fast transaction times, and the full security of the Ethereum blockchain itself. How? By allowing transactions to use their own set of nodes, entirely independent of Ethereum, and then adding proof of those transactions to the Ethereum. Effectively, Layer 2’s keep their data self-contained except for the proof that something has happened (this is all a rough approximation, but close enough for our purposes here).
Circling back to EIP-4488, these Layer 2s frequently leverage what’s known as calldata to batch add these proofs to Ethereum. calldata is a specific type of data in the EVM that’s particularly cheap. That being said, if we’re hoping to reach internet scale, paying $0.30+ for something as simple as transferring ETH is still too much. EIP-4488 introduces an explicitly lower cost for calldata on Ethereum, which will decrease the cost of Layer 2s, and ultimately save users money.
EIP-4444: Everyone Gets a Node
https://eips.ethereum.org/EIPS/eip-4444
The biggest problem with running your own Ethereum node at home generally isn’t the technical complexity involved. The terminal commands are simple and there are even products like Dappnode offer plug’n’play ease. What gets tricky to solve for though is the sheer amount of storage costs you need to run a node. Each node must remember the entire history of the Ethereum blockchain. The storage requirements for that currently sit at ~1 TB and are closer to 6 TBs if you want to run what’s known as an “archive” node. And with proposals like EIP-4488 above, the size requirements are likely to increase even quicker (up to 3 TB per year in the extreme case).
EIP-4444 aims to address this by introducing a prune limit on historical data. Ethereum nodes would no longer have to remember the entire history of the blockchain. Instead, they can keep only the last year of data. This makes running Ethereum nodes at home considerably less resource intensive. And, importantly, more nodes directly translates into better decentralization for Ethereum as a whole.
If you’re like us, you’re probably wondering where all that old data is going to be stored? That is a fantastic question without an answer unfortunately. There seems to be consensus around some ideas though:
Have a separate “historical” node that people can choose to run
Introduce a P2P protocol for downloading past data (remember the BitTorrent days?)
Rely on centralized authorities to remember all past data - and make it available to the rest of us (likely for a fee…)
In any case, more nodes and more decentralization is a net positive for the ecosystem. With time we’ll find an appropriate solution to accessing historical data.
EIP-3668: Accessing Off-Chain Data
https://eips.ethereum.org/EIPS/eip-3668
Spearheaded by ENS and Chainlink, EIP-3668 introduces a standard for Ethereum developers to securely incorporate off-chain data into the ecosystem. This is a potentially huge win for everyone. Ethereum is meant to the world’s computer and it makes sense that we’d want the ability to “plug-in” an external hard-drive to that. This is roughly what the Cross-Chain Interoperability Protocol (CCIP) introduces.
CCIP works by allowing a smart contract to say “I don’t have the data, but I can verify it from Source X”. A client (e.g. browser) asking that contract for data can then reach out to Source X, receive raw data, and offer it back to the contract to verify. In this way, the data is safe to use (because it’s validated by the contract), but also cheap to store + access because it’s not stored on-chain. Even better, Source X could be anything - another blockchain, an API gateway, etc - CCIP simply provides us with a standard way of implementing this sort of off-Ethereum communication.
Early concepts around CCIP are promising. Chainlink and SWIFT are partnering on a way to bridge the web2 banking world with web3. ENS is expanding beyond primarily .eth to support any domain, on any chain; Coinbase has already implemented this.
In the end, growing pains are a great sign for the Ethereum ecosystem and the future looks bright 🌕 See you all at the next Devcon!
By all measures, Devcon VI was a huge success. Over 6000 participants from around the world met in Bogota, Colombia to build, network, and celebrate together in the official Ethereum conference.
This is coming about a month after The Merge in which Ethereum switched from Proof of Work (PoW) to Proof of Stake (PoS). That transition worked far better than any could have hoped and has lead to ETH even being deflationary at times 🔥
So what’s next for Ethereum and its ecosystem? In short, far too much is happening to cover in a single post, but we’re going to touch on “the state of data”.
Let’s take a step back and remember how we got to today. Ethereum launched in 2015 with a vision of being the world’s computer. At its core, Ethereum processes transactions in its Ethereum Virtual Machine (EVM) and reaches consensus with its nodes (network of servers). In order to do this, each node in the ecosystem must keep track of a history of all blocks and transactions that have ever existed.
Fast forward to today, there are well over 15 million blocks and over 1 billion transactions on the Ethereum mainnet. And these numbers don’t even reflect the growing ecosystem of secondary blockchains on and around Ethereum such as Polygon, Optimism, Arbitrum, Starknet, etc. Point being, there’s a lot of data out there and it’s only continuing to grow.
In order for Ethereum, and its ecosystem, to truly reach “internet scale”, we need to drastically increase adoption. That adoption, though, inevitably comes with a sharp increase in data and we need to be ready for this.
Focusing primarily on solving for adoption, some of the common themes in Ethereum today are:
Too few transactions per second - not enough support for simultaneous users
Too few [independent] node operators - not enough decentralization
On-chain storage is expensive - and standards are missing for off-chain
Thankfully, the sharp minds of the industry have already been working on solutions; many of which should be rolling out in the coming months and years. Let’s dig into a few of these Ethereum Improvement Proposals (EIPs):
EIP-4488: Working on Throughput
https://eips.ethereum.org/EIPS/eip-4488
The leading way to increase throughput on Ethereum is simply to move transactions off of Ethereum. This may sound counter-intuitive, but bear with me.
Layer 2’s such as Optimism and Arbitrum offer developers and consumers lower gas fees, fast transaction times, and the full security of the Ethereum blockchain itself. How? By allowing transactions to use their own set of nodes, entirely independent of Ethereum, and then adding proof of those transactions to the Ethereum. Effectively, Layer 2’s keep their data self-contained except for the proof that something has happened (this is all a rough approximation, but close enough for our purposes here).
Circling back to EIP-4488, these Layer 2s frequently leverage what’s known as calldata to batch add these proofs to Ethereum. calldata is a specific type of data in the EVM that’s particularly cheap. That being said, if we’re hoping to reach internet scale, paying $0.30+ for something as simple as transferring ETH is still too much. EIP-4488 introduces an explicitly lower cost for calldata on Ethereum, which will decrease the cost of Layer 2s, and ultimately save users money.
EIP-4444: Everyone Gets a Node
https://eips.ethereum.org/EIPS/eip-4444
The biggest problem with running your own Ethereum node at home generally isn’t the technical complexity involved. The terminal commands are simple and there are even products like Dappnode offer plug’n’play ease. What gets tricky to solve for though is the sheer amount of storage costs you need to run a node. Each node must remember the entire history of the Ethereum blockchain. The storage requirements for that currently sit at ~1 TB and are closer to 6 TBs if you want to run what’s known as an “archive” node. And with proposals like EIP-4488 above, the size requirements are likely to increase even quicker (up to 3 TB per year in the extreme case).
EIP-4444 aims to address this by introducing a prune limit on historical data. Ethereum nodes would no longer have to remember the entire history of the blockchain. Instead, they can keep only the last year of data. This makes running Ethereum nodes at home considerably less resource intensive. And, importantly, more nodes directly translates into better decentralization for Ethereum as a whole.
If you’re like us, you’re probably wondering where all that old data is going to be stored? That is a fantastic question without an answer unfortunately. There seems to be consensus around some ideas though:
Have a separate “historical” node that people can choose to run
Introduce a P2P protocol for downloading past data (remember the BitTorrent days?)
Rely on centralized authorities to remember all past data - and make it available to the rest of us (likely for a fee…)
In any case, more nodes and more decentralization is a net positive for the ecosystem. With time we’ll find an appropriate solution to accessing historical data.
EIP-3668: Accessing Off-Chain Data
https://eips.ethereum.org/EIPS/eip-3668
Spearheaded by ENS and Chainlink, EIP-3668 introduces a standard for Ethereum developers to securely incorporate off-chain data into the ecosystem. This is a potentially huge win for everyone. Ethereum is meant to the world’s computer and it makes sense that we’d want the ability to “plug-in” an external hard-drive to that. This is roughly what the Cross-Chain Interoperability Protocol (CCIP) introduces.
CCIP works by allowing a smart contract to say “I don’t have the data, but I can verify it from Source X”. A client (e.g. browser) asking that contract for data can then reach out to Source X, receive raw data, and offer it back to the contract to verify. In this way, the data is safe to use (because it’s validated by the contract), but also cheap to store + access because it’s not stored on-chain. Even better, Source X could be anything - another blockchain, an API gateway, etc - CCIP simply provides us with a standard way of implementing this sort of off-Ethereum communication.
Early concepts around CCIP are promising. Chainlink and SWIFT are partnering on a way to bridge the web2 banking world with web3. ENS is expanding beyond primarily .eth to support any domain, on any chain; Coinbase has already implemented this.
In the end, growing pains are a great sign for the Ethereum ecosystem and the future looks bright 🌕 See you all at the next Devcon!
No activity yet