Farcaster's architecture is a blend of on-chain and off-chain systems, which includes registry contracts deployed on the Ethereum network via Optimism. Messages are cast to a network of servers called 'Hubs', ensuring the reliability of user data. This hybrid architecture is spurring some innovative ideas like Frames. Frames allow developers to embed interactive experiences within Farcaster posts, known as Casts. There are fun and different frames being created, including one for Girl Scout cookies. This allows you to shop within the Frame and then checkout via Coinbase without ever leaving Warpcast.

Of course we had to create our own frame : )

We’re committed to the Farcaster ecosystem and are excited about its potential. We integrated Farcaster into our indexing pipelines, positioning it alongside prominent chains like Base, Syndicate, and Solana. We're proud to offer this dataset entirely free as a public good, fueled by our passion for easy and open access to data.

Our approach with this dataset has been to take a semi-opinionated stance on normalizing hub events into views that more easily work for products and analytics (e.g. "user_data" from Hubs maps to "profiles" in BigQuery).

To start, we have 5 tables available: casts, links, reactions, verifications, and profiles.

Looking where to start? Try seeing who the most active reply guys are.

Currently, BigQuery is updated every hour with the latest snapshot of data. Depending on feedback, we'll likely try to make this truly real-time using some materialized views.

A huge shout out and thanks to @pinatacloud for their public hub support!

Looking where to start? Try seeing who the most active reply guys are.

If you have questions or other data needs, contact us!

Warpcast: Indexing Co - Brock - Stephen

Email: hello@indexing.co

https://www.flowcapacitor.xyz/

In the ever-evolving world of Web3, NFTs (Non-Fungible Tokens) have taken center stage. But as with any gold rush, there's an inevitable oversaturation. Many NFTs now sit dormant in digital wallets, their value diminished or even non-existent. This posed a question to our team: How can we breathe new life into these forgotten digital assets?

Enter "Mint to the Future," a project that emerged from our desire to rejuvenate the NFT landscape and introduce users to the Flow blockchain. The concept was simple: a recycle contract that would allow users to repurpose their Ethereum-based NFTs, which had lost their value, and in return, receive brand-new, vibrant NFTs minted on the Flow blockchain.

The Perfect Blend of Design and Tech

Our team, a blend of design and technical expertise, was uniquely positioned to bring this vision to life. We weren't just looking to create another NFT platform; we aimed to revolutionize the user experience, making it seamless, enjoyable, and rewarding.

Why Flow and Cadence?

Flow, with its robust documentation and user-friendly approach, was a natural choice. We crafted a smart contract using Cadence, Flow's native programming language, ensuring a smooth and secure transition for NFTs from one blockchain to another.

Sustainability

Flow’s commitment to energy efficiency and sustainability is admirable. By using a proof-of-stake consensus mechanism, Flow significantly reduces its energy consumption. To put it into perspective, the network’s annual energy usage is astoundingly low at 0.18 GWh. To compare, minting an NFT on Flow requires less energy than making a simple Instagram post, making it a leader in energy efficiency, especially when juxtaposed with other blockchains. This conscious choice not only bolsters the platform's commitment to a greener environment but also empowers users and developers to make an eco-friendly choice.

In the context of “Mint to the Future,” this alignment with sustainability means that while users are rejuvenating their NFTs, they are also actively participating in a greener and more responsible digital ecosystem. The assurance that each NFT minted isn't draining vast amounts of energy or contributing significantly to carbon emissions is invaluable in today's conscientious digital age.

Embracing AI, a tool that will greatly enhance web3

The tech world is currently witnessing an AI explosion, but the actual value can be clouded by mainstream headlines. Headlines aside, there are practical things AI excels at, a major one being generative art. We leveraged OpenAI's DALLE via a feature that generates unique NFT images in real-time. The new image is generated by collecting keywords words from the NFT’s being recycled and combining them into a prompt that generates a new image. This not only enhances the user experience but also significantly reduces our design costs. It was a win-win!

A Glimpse into the Future

Imagine a world where your once-forgotten NFTs are given a fresh lease of life. With "Mint to the Future," that's precisely what we offer. As our website aptly puts it, "Help Marty McFlow move to a more decentralized, secure, and innovative future by transforming your old NFTs into something dynamic and new."

We were excited to create Flow to the Future because it shows how AI will impact the web3 ecosystem while also demonstrating new and creative user onboarding tools.

By combining strengths from Wovn & Indexing Co, we were able to design and deploy the app in about a month. We had a great experience learning about and building in the Flow ecosystem. Overall, the Flow ecosystem was the perfect choice for Mint to the Future and we’re excited follow its continued growth & success!

An In-depth Look into the World of Web3 Data

Welcome to the forefront of the digital revolution. In the world of web3, with its vast data, blockchain networks, protocols, and tokens, accessing and understanding information can be challenging. Developers often build custom solutions due to the lack of accessible options. This piece addresses the infrastructure problem underlying web3 data accessibility, exploring strategies for transformative success. Let's dive in.

The Challenges of Accessing Web3 Data

The world of web3 data is vast and ever-growing, but it can be incredibly frustrating to access. With all the different blockchains, protocols, and tokens, navigating and understanding this complex data ecosystem requires an upfront investment of time, patience, and money. The data is raw, unindexed, and brutal to merge with third-party data sets. Dapp providers often find themselves building custom tools over leveraging open source and paid solutions.

The current attempts to address these issues are futile as they focus on treating the symptom instead of the disease.

Identifying the Real Problem

The data accessibility problem in web3 is not a data problem, it’s an infrastructure problem.

Now that we’ve properly defined the problem, we can implement a strategy that leads to success.

Throughout the last two years, we used several web3 indexing products. Some, open-source, provided value in the short term. Over time the value was reversed through unannounced deprecated services and the additional resources we invested in making the product work. We later found centralized products that charge for their APIs. It was like having two versions of our town library next to each other. One makes you pay but gives you access to their card catalog. The other is free but requires more than a decade bit more time and creativity to find your book. After a few times navigating and getting comfortable in the free library, there is zero value in paying for admission next door. Like the soon-bankrupt library, companies charging for API access will soon reevaluate their business models.

The Future: An Ecosystem with Free Indexing APIs and Transformers

As we transition into an ecosystem with free indexing APIs, there is an additional hurdle that needs to be addressed. Simplifying access to indexed data is great but not useful if it's not easily configurable into different forms. Please welcome, transformers.

Transformers transform the raw, indexed data into configurable forms. The result is a composable infrastructure that any individual, company, industry, or government building in web3 can configure to their needs.

Looking at data from a single NFT project via an indexed API can give us insight into transaction data. Who is the largest holder? When did they buy? Where did they buy? Powerful, yes. However, scaling across thousands of transactions or adding web2 data into the model is hard. Transformers give builders the ability to overcome these challenges, create, and provide valuable datasets to their users.

The Promise of Composable Infrastructure

Leveraging this infrastructure, let's use the Ethereum, Tezos and Polygon NFT Indexers to get a few baseline stats on chain data. Next, we’ll configure the transformer to compare net new purchases from September 2022-December 2022 on Solona, Ethereum and Polygon using wallet addresses.

1. There are over 7.5 million wallets on Ethereum that have ever owned an NFT

2. The average NFT price in 2022 on Ethereum was $343

3. There are 45,000 ERC721 and 30,000 ERC1155 NFT contracts on Ethereum.

4. During 2022, Tezos NFT sales (XTZ) were up 115%

5. NFT Market place volume peaked at 1.7 billion on Ethereum in August 2021

6. NFT wash trading peaked in Jan 2022 on Ethereum with 4.1 billion in wash trades versus 1.1 billion in organic trading volume.

7. From Sept 2022-Dec 2022, new users buying NFTs on Solona dropped 63% and 36% on Ethereum. During that same time, new users buying NFTs on Polygon grew more than 500%.

8. With composable infrastructure, developers reduce superfluous costs and technical debt while providing more value to their users.

9. To realize Web3's goal of surpassing web2, the applications must be significantly superior to their web2 counterparts. Developers require the same, if not better, tools and infrastructure as those available in web2. With seven years of experience building custom data solutions and paying for what should be free, we’ve gone all in on solving web3’s data infrastructure problem.

The Real-World Impact of Configurable Data

Taking our understanding of configurable infrastructure a step further, we can begin to see its profound implications on real-world blockchain applications. Consider, for example, the way in which DeFi applications can leverage this technology to provide their users with highly personalized and detailed financial metrics. In fact, a study from the University of Cambridge highlights how adopting a more configurable data approach can lead to enhanced user experiences and improved decision-making capabilities in financial markets.

Looking Ahead: The Future of Web3 Data Infrastructure

Moving forward, we must remember that the full potential of web3 data will only be unlocked when we prioritize building and maintaining a robust, scalable, and flexible data infrastructure. This sentiment is echoed in a recent report by Deloitte, which emphasizes the crucial role of a strong data infrastructure in harnessing the transformative power of blockchain technology. By fostering an ecosystem of free and efficient indexing APIs, we are setting the stage for digital transformation and a future where web3 data is not only more accessible but also more impactful.

The Best Web3 Indexing Tools for Strategic Advantage

In today's increasingly decentralized digital landscape, effective web3 indexing tools can provide a strategic edge. These web 3 indexing tools not only provide access to vast amounts of blockchain data, but they also simplify the process of filtering and interpreting this data. Here, we highlight some of the top indexing tools designed to navigate the web3 data universe effectively:

The Graph

The Graph is a powerful protocol that revolutionizes blockchain by enabling easy access to data through a network of open APIs called subgraphs.

Covalent

Covalent is a unified API that offers visibility into vast amounts of blockchain data, providing comprehensive insights for web3 strategies the blockchain network.

Nansen

Nansen offers blockchain analytics for Ethereum-based finance protocols, providing insights into DeFi, NFTs, and more. Their sophisticated tools simplify navigation, understanding, and decision-making based on blockchain data.

QuickNode

QuickNode is an exceptional web3 indexing tool that enhances blockchain development efficiency. It offers reliable, fast, and scalable access to indexes, to Ethereum, Bitcoin, and other blockchain networks. With QuickNode, there's no need to run your own nodes, eliminating operational hassle and overhead.

Embracing these innovative web3 indexing tools can help stakeholders derive meaningful insights from blockchain data in real time, fueling informed strategies and decisions in the web3 ecosystem.

Overcoming Challenges in Web3 Data Indexing

Despite the significant advantages of web3 indexing tools, it's important to acknowledge the challenges that remain. One key hurdle is the sheer volume of blockchain data. As blockchains grow and proliferate, storing, indexing, and querying this data become increasingly resource-intensive. As a result, developing scalable solutions multiple blockchains remains a top priority.

Moreover, issues of data privacy and security in the distributed ledger and web3 space pose significant concerns. Public blockchains are inherently transparent, which may lead to unwanted disclosures of sensitive information. Ensuring the privacy-preserving computation and storage of blockchain data is thus a crucial area of ongoing research and development.

Final thoughts

In conclusion, the world of web3 data presents immense opportunities and challenges. While accessing and understanding this vast and complex web of data landscape may seem daunting, solutions offered by indexing companies like "The Indexing Company" provide a way forward. By simplifying complex data systems and offering tailored indexing tools, businesses can harness the power of web3 data for strategic advantage.

"The Indexing Company" stands out as a leading provider in this space, offering innovative solutions that enable faster build times, cost savings, and streamlined side chain workflows. Their expertise in web3 data indexing empowers businesses to navigate the intricacies of blockchain networks, protocols, and tokens with ease, unlocking valuable insights and informed decision-making.

For businesses seeking to stay at the forefront of the digital revolution, embracing the capabilities and technologies of "The Indexing Company" is a call to action. By partnering with them, businesses can simplify their data systems, gain a competitive edge, and unleash the transformative potential of web3 data. Don't miss out on the opportunity to revolutionize your data strategy and drive success in the evolving landscape of web3. Contact "The Indexing Company" today to embark on this journey towards streamlined and impactful web3 data utilization.

Chainlink is one of the most ambitious projects in web3, providing everything from price oracles to external API adapters. With the ability for anyone to easily connect to any web or off-chain API and use it as an input or output for smart contracts on the blockchain, Chainlink has become a go-to for many organizations in the crypto industry.

At the Indexing Company, we love Chainlink and indexing systems, so we decided to index their price oracles and run them through our transformation engine.

Understanding Chainlink's Price Oracles

First, let's cover the basics. What are Chainlink's price oracles, and why are they important for the crypto industry?

In the simplest terms, a price oracle is a source that provides up-to-date, real-time pricing data to software running smart contracts on a blockchain network. It can be compared to stock exchange tickers of companies in the past that supplied real-time pricing data to people. Without accurate and reliable sources for crypto prices, smart contract software would not be able to execute correctly.

Chainlink's price oracles offer a reliable alternative to traditional methods of data collection and market assessment. The decentralized, secure protocol provides access to a range of leading sources that can be used to make informed price predictions. Prices are managed via an open-source aggregation algorithm, which minimizes the risk of falsified or inaccurate results.

This is extremely beneficial compared to other price oracle solutions on the market, as it ultimately reduces the need for manual response checks and lowers financial exposure in potential errors. Chainlink's security data privacy and compliance measures also maintain top-level protection against external threats such as hacking and performance issues, giving users confidence in their pricing decisions.

The Technology Behind Chainlink's Price Oracles

Chainlink's Price Oracles aren't just a manifestation of advanced technology, they also embody a system of record that's been painstakingly curated for its reliability and seamless functionality. By engineering this intricate system, Chainlink ensures that the transaction data it sources maintains a high level of accuracy. The superior infrastructure underpinning Chainlink's Price Oracles plays a pivotal role in their operation, making them an invaluable asset within the broader cryptocurrency ecosystem.

Chainlink's Decentralized Data Aggregation

Chainlink has harnessed the power of decentralized data aggregation to fortify the reliability of its price oracles. This method dramatically enhances data accuracy and reliability, providing an unimpeachable source of information for users. The decentralized nature of blockchain data aggregation serves as a buffer against data manipulation, further strengthening the credibility of the data Chainlink oracles deliver.

Chainlink's Oracle Reputation System

To ensure the consistent performance of its network, Chainlink has implemented an Oracle Reputation System. This system objectively evaluates the performance of various oracles, assigning ratings based on reliability and accuracy. These ratings offer valuable insights into the quality of data each oracle provides and directly contribute to the overall performance and credibility of Chainlink's network.

Data Quality and Chainlink

In the realm of operations of price oracles, data quality is non-negotiable. Chainlink has established stringent protocols to guarantee high-quality data from its price oracles. By placing an unwavering emphasis and focus on data quality, Chainlink sets a high standard for the reliability and usability of the data it provides.

Understanding Price Oracle Attacks

The cryptosphere is not immune to security concerns and challenges, and price oracle attacks are one such issue that warrants attention. These attacks can have significant implications for the integrity of a blockchain network, reinforcing the importance of Chainlink's commitment to security and robust design. By understanding these attacks, we can better appreciate the safeguards Chainlink has in place to prevent such vulnerabilities.

Chainlink's Role in the DeFi Ecosystem

Chainlink's price oracles have carved out a pivotal role in the burgeoning DeFi ecosystem. As a key player in the DeFi landscape, Chainlink has made significant strides in enhancing the reliability and accessibility of decentralized financial services. Its influence and importance in shaping the DeFi market are unequivocal, marking it as a game-changer in the world of decentralized finance.

The Process and Advantages of Indexing Chainlink's Price Oracles

So, why index Chainlink's price oracles?

By indexing Chainlink's price oracles, we can configure the Indexing Company's transformation engine to generate some interesting data points.

First, we indexed the feeds and merged labels (e.g. ETH/USD) to compare and contrast price data. Next, using real time data we ran different transformations to get things like spot, 1-day, 7-day, 15-day, and 30-day moving averages. Lastly, to visualize the above data points, we exposed an API via GraphQL and routed it into ReTool.

https://indexingco.retool.com/embedded/public/49529312-e961-455a-8e76-a298e730b8c0

Testing and Comparing with Traditional Exchange

Test it yourself!

You might ask yourself, "I can already get this data on Coinbase. What's the big deal?"

The difference is in how the data is obtained and disseminated. for example, Exchanges like Coinbase get crypto prices by matching buyers and sellers in transactions on their platform, and the prices on the exchange reflect the supply and demand of the crypto assets being traded on that particular platform. The prices of crypto assets on different exchanges can vary due to differences in trading volume, order book depth, and other factors.

Chainlink price oracles, on the other hand, obtain crypto prices from multiple, decentralized data sources, such as other decentralized exchanges (DEXs), off-chain data aggregators, and oracles. The Chainlink network is decentralized, meaning that it is a distributed ledger not controlled by any single entity or organization, making it resistant to manipulation and tampering. The Chainlink network uses a consensus mechanism to ensure that the data provided by the oracles is accurate, and it uses smart contracts to ensure that the data is tamper-proof.

If the goal of web3 is decentralization, using tools like Chainlink's price oracles is pivotal.

Upcoming Advancements and Opportunities in Blockchain Data Indexing

We're finishing documentation this week then we'll make the current version of the API publicly available to developers (for free!)

The above exercise, in addition to whatsmynameagain.xyz and mirrormirror.page, are examples of what can be created with The Indexing Company's technology. Our mission is to provide the infrastructure so you can become the data company or surface to your business, partners and end users.

Email us today at hello@indexing.co, and we'll get you set up with your own configurable indexers and transformations to create fun tools like this.

Final Thoughts

The indexing Chainlink's price oracles facilitates digital transformation by leveraging advanced technologies like artificial intelligence and multiple blockchains. This process generates valuable data points, enabling informed decision-making and enhancing the reliability of blockchain data. By utilizing decentralized data aggregation, Chainlink's price oracles play a crucial role in the decentralized finance (DeFi) ecosystem, providing reliable and accessible pricing information. This advancement opens new opportunities and drives innovation in the evolving landscape of DeFi and beyond.

At The Indexing Company, we're excited to contribute to this journey by offering our indexing technology to enterprises around the world. As we look forward to what's next, we're not just observers; more than a decade in we're active participants shaping the future of data in a decentralized world. Join us on this journey - the future of data awaits you.

This is the same API that powers our ENS Profile search tool, What’s my name again, and is backed by our real-time indexing engine. You can access it via GraphQL at https://query.indexing.co/graphql with the following schema:

type ENSProfileAddresses {
  address: String
  coinType: Int
}

type ENSProfileAttributes {
  textKey: String
  textValue: String
}

type ENSProfile {
  addresses: [ENSProfileAddresses!]
  attributes: [ENSProfileAttributes!]
  contenthash: String
  name: String
  node: String
  owner: String
  tokenId: String
}

input ENSProfileFilter {
  name: String
  node: String
  owner: String
  textValue: String
  tokenId: String
}

type Query {
  ensProfiles(filters: ENSProfileFilter): [ENSProfile!]!
}

For example, to grab the current profile for vitalik.eth you could simply do:

query {
  ensProfiles(
    filters: {
      name: "vitalik.eth"
    }
  ) {
    addresses {
      address
      coinType
    }
    attributes {
      textKey
      textValue
    }
    contenthash
    name
    owner
  }
}

Which would return:

{
  "data": {
    "ensProfiles": [
      {
        "addresses": [
          {
            "address": "0xd8da6bf26964af9d7eed9e03e53415d37aa96045",
            "coinType": 60
          }
        ],
        "attributes": [
          {
            "textKey": "avatar",
            "textValue": "eip155:1/erc1155:0xb32979486938aa9694bfc898f35dbed459f44424/10063"
          },
          {
            "textKey": "url",
            "textValue": "https://vitalik.ca"
          }
        ],
        "contenthash": "0xe3010170122081e99109634060bae2c1e3f359cda33b2232152b0e010baf6f592a39ca228850",
        "name": "vitalik.eth",
        "owner": "0xd8dA6BF26964aF9D7eEd9e03E53415D37aA96045"
      }
    ]
  }
}

Give it a try and let us know how it goes! And if you find yourself needing anything extra / different, don’t hesitate to reach out - we’re always looking for ways to improve our offerings and empower the builders in web3.

In short, this is a search engine for Mirror.xyz. For those that don’t know, Mirror is a web3 publishing platform akin to Medium in the web2 world. Unlike Medium though, Mirror is merely an interface to help writers put their content out into Data 3.0. That is, everything written on Mirror is ultimately stored on Arweave.

(For those curious, we spoke in more depth on Data 3.0 storage options like Arweave here)

This intentional decentralization of data has important implications:

1. Writers own their own data - only they can add, modify, and sign their data

2. Mirror does not control the data - theoretically someone else could come along and build a competitor to Mirror, leveraging the exact same underlying data source

3. All of the posts from Mirror are publicly available, forever thanks to Arweave

And we at Indexing Co have been able to leverage the 3rd point there to successfully index every Mirror post written - i.e. creating a “mirror” of Mirror if you will 😎 And now that we’ve got a running index of posts (here’s the skinny on how that’s done), we can expose them via an API and ultimately create the simple UI that you see today.

https://www.mirrormirror.page/

Let us know what you think!

Because these profiles are stored on-chain, they’re publicly accessible yet fully controlled by you, the owner. We could compare this to Facebook or Twitter profiles, but without the ads, centralized control, and authentication requirements that these platforms require in service of their bottom line. In short, the potential power of ENS Profiles is enormous.

One thing that’s been missing though is the ability to search these profiles. Sure, tools like ens.domains and ens.vision exist, but these are focused on managing and purchasing ENS names, respectively. Neither service is intended to help ENS users communicate with one another or use the power of a “profile”.

Meanwhile, we’ve been working hard at The Indexing Company to build out our Indexing as a Service infrastructure. Recognizing the recent adoption of ENS and being long-time supporters ourselves, we decided to tune our indexing service towards ENS. As a result, we’re happy to introduce What’s My Name Again? 🎉

This is a freely available service to search both ENS names and entire ENS Profiles. Some examples:

• Wildcard name matches like nick*.eth

• ENS name by email

• ENS names with Mirror.xyz links

• Who wants to be contacted?

• Anyone putting "coffee" on-chain

(hint: searches not starting with a wildcard, *, are much faster!)

Give it a whirl and let us know what you think!

This is coming about a month after The Merge in which Ethereum switched from Proof of Work (PoW) to Proof of Stake (PoS). That transition worked far better than any could have hoped and has lead to ETH even being deflationary at times 🔥

So what’s next for Ethereum and its ecosystem? In short, far too much is happening to cover in a single post, but we’re going to touch on “the state of data”.

Today

Let’s take a step back and remember how we got to today. Ethereum launched in 2015 with a vision of being the world’s computer. At its core, Ethereum processes transactions in its Ethereum Virtual Machine (EVM) and reaches consensus with its nodes (network of servers). In order to do this, each node in the ecosystem must keep track of a history of all blocks and transactions that have ever existed.

Fast forward to today, there are well over 15 million blocks and over 1 billion transactions on the Ethereum mainnet. And these numbers don’t even reflect the growing ecosystem of secondary blockchains on and around Ethereum such as Polygon, Optimism, Arbitrum, Starknet, etc. Point being, there’s a lot of data out there and it’s only continuing to grow.

In order for Ethereum, and its ecosystem, to truly reach “internet scale”, we need to drastically increase adoption. That adoption, though, inevitably comes with a sharp increase in data and we need to be ready for this.

Today’s Problems

Focusing primarily on solving for adoption, some of the common themes in Ethereum today are:

1. Too few transactions per second - not enough support for simultaneous users

2. Too few [independent] node operators - not enough decentralization

3. On-chain storage is expensive - and standards are missing for off-chain

Looking Forward

Thankfully, the sharp minds of the industry have already been working on solutions; many of which should be rolling out in the coming months and years. Let’s dig into a few of these Ethereum Improvement Proposals (EIPs):

EIP-4488: Working on Throughput

https://eips.ethereum.org/EIPS/eip-4488

The leading way to increase throughput on Ethereum is simply to move transactions off of Ethereum. This may sound counter-intuitive, but bear with me.

Layer 2’s such as Optimism and Arbitrum offer developers and consumers lower gas fees, fast transaction times, and the full security of the Ethereum blockchain itself. How? By allowing transactions to use their own set of nodes, entirely independent of Ethereum, and then adding proof of those transactions to the Ethereum. Effectively, Layer 2’s keep their data self-contained except for the proof that something has happened (this is all a rough approximation, but close enough for our purposes here).

Circling back to EIP-4488, these Layer 2s frequently leverage what’s known as calldata to batch add these proofs to Ethereum. calldata is a specific type of data in the EVM that’s particularly cheap. That being said, if we’re hoping to reach internet scale, paying $0.30+ for something as simple as transferring ETH is still too much. EIP-4488 introduces an explicitly lower cost for calldata on Ethereum, which will decrease the cost of Layer 2s, and ultimately save users money.

EIP-4444: Everyone Gets a Node

https://eips.ethereum.org/EIPS/eip-4444

The biggest problem with running your own Ethereum node at home generally isn’t the technical complexity involved. The terminal commands are simple and there are even products like Dappnode offer plug’n’play ease. What gets tricky to solve for though is the sheer amount of storage costs you need to run a node. Each node must remember the entire history of the Ethereum blockchain. The storage requirements for that currently sit at ~1 TB and are closer to 6 TBs if you want to run what’s known as an “archive” node. And with proposals like EIP-4488 above, the size requirements are likely to increase even quicker (up to 3 TB per year in the extreme case).

EIP-4444 aims to address this by introducing a prune limit on historical data. Ethereum nodes would no longer have to remember the entire history of the blockchain. Instead, they can keep only the last year of data. This makes running Ethereum nodes at home considerably less resource intensive. And, importantly, more nodes directly translates into better decentralization for Ethereum as a whole.

If you’re like us, you’re probably wondering where all that old data is going to be stored? That is a fantastic question without an answer unfortunately. There seems to be consensus around some ideas though:

• Have a separate “historical” node that people can choose to run

• Introduce a P2P protocol for downloading past data (remember the BitTorrent days?)

• Rely on centralized authorities to remember all past data - and make it available to the rest of us (likely for a fee…)

In any case, more nodes and more decentralization is a net positive for the ecosystem. With time we’ll find an appropriate solution to accessing historical data.

EIP-3668: Accessing Off-Chain Data

https://eips.ethereum.org/EIPS/eip-3668

Spearheaded by ENS and Chainlink, EIP-3668 introduces a standard for Ethereum developers to securely incorporate off-chain data into the ecosystem. This is a potentially huge win for everyone. Ethereum is meant to the world’s computer and it makes sense that we’d want the ability to “plug-in” an external hard-drive to that. This is roughly what the Cross-Chain Interoperability Protocol (CCIP) introduces.

CCIP works by allowing a smart contract to say “I don’t have the data, but I can verify it from Source X”. A client (e.g. browser) asking that contract for data can then reach out to Source X, receive raw data, and offer it back to the contract to verify. In this way, the data is safe to use (because it’s validated by the contract), but also cheap to store + access because it’s not stored on-chain. Even better, Source X could be anything - another blockchain, an API gateway, etc - CCIP simply provides us with a standard way of implementing this sort of off-Ethereum communication.

Early concepts around CCIP are promising. Chainlink and SWIFT are partnering on a way to bridge the web2 banking world with web3. ENS is expanding beyond primarily .eth to support any domain, on any chain; Coinbase has already implemented this.

In the end, growing pains are a great sign for the Ethereum ecosystem and the future looks bright 🌕 See you all at the next Devcon!

The beauty of protocols like Mirror is that they don’t own any of the data. They still have a login, a clean text editor, and shareable links just like Medium or Substack. But the content that flows through Mirror is entirely decentralized on a network they don’t control, Arweave. We’ve spoken briefly about Arweave previously, but the gist is that any content stored on it is stored forever thanks to unique incentive models built into the network. This has two important ramifications:

1. There are 0 paywalls in Mirror. You can still choose to support individual creators, and Mirror helps facilitate this, but it’s entirely optional.

2. You don’t even have to use Mirror to participate in the broader ecosystem.

That second point is what we’re going to focus on in this post. Specifically, because the data is freely available forever, we can choose to use this data in any way we want. For instance, this very blog post is available on Mirror, but it’s also available on our company’s website. Any updates to this post are instantly available on both sites because they both use the exact same data source, Arweave. Let’s unpack this a bit:

1. Content is written in an editor such as Mirror

2. That content is added to a transaction on Arweave

3. A user visits a related link on either mirror.xyz or indexing.co

4. The site fetches the content from Arweave, formats it, and displays it to the end user

5. Content is shared from creator to reader

The Nitty Gritty

Now that we have an overview of the steps required to load an individual post, let’s look at the technical details for indexing all of a given user’s posts from Mirror. For this, we’re going to focus on using Typescript alongside the arweave package on npm.

Mirror helps structure the content stored on Arweave with what are known as tags. These are pretty much what you might expect: key <> value pairs representing arbitrary strings tied to a piece of content. For instance, these are the tags for our post on web3 storage options:

{
  'Content-Type': 'application/json',
  'App-Name': 'MirrorXYZ',
  Contributor: '0x0317d91C89396C65De570c6A1A5FF8d5485c58DC',
  'Content-Digest': 'B1ytOURSn75aACoOHmVHrV31bl0tL4ffWHEtl4JeGUE',
  'Original-Content-Digest': 'FDyv8i8c15ATs_KIpAtEdeMP20WZ00FfssPYOj3EZRY'
}

For our purposes we’re most interested in the App-Name and Contributor tags. We’ll use the combination of these two to pull all of the content published on Mirror by our given writer.

Alright, time for some code. We’re first defining our arweave instance and pointing it at the publicly hosted arweave.net provider. If you want to run your own node, check out their docs here.

import Arweave from "arweave";

const arweave = Arweave.init({
  host: "arweave.net",
  port: 443,
  protocol: "https",
});

Since we’re using Typescript, we can define our Post structure. This roughly reflects what we’ll get from Arweave directly with the addition of the originalDigest key. That originalDigest will be pulled from the Original-Content-Digest and is important because that’s what Mirror uses in their URLs (i.e. why you can edit a post without having to share a new link).

type Post = {
  authorship: {
    contributor: string;
  };
  content: {
    body: string;
    timestamp: string;
    title: string;
  };
  digest: string;
  originalDigest: string;
};

Finally, we get to the meat of this whole shebang. We first query Arweave for the transactions matching our given Contributor tag and then fetch the full transaction for each identifier, including its data. Since the current arweave package only allows us to search by one tag, we filter by the App-Name: MirrorXYZ piece further down after we parse out the tags.

Now that we’ve filtered down to only those transactions that match our Contributor and App-Name, we can pull out the data and turn it into a Post. Mirror adds all of their content as structured JSON strings, so we can readily parse that out and typecast to our Post type. Of course, null checks and error handling would be welcomed additions as well.

async function getPostsForContributor(address: string): Promise<Post[]> {
  const arweaveTransactionIds = await arweave.transactions.search(
    "Contributor",
    address
  );

  const arweaveTransactions = await Promise.all(
    arweaveTransactionIds.map((txId) => arweave.transactions.get(txId))
  );

  const postsByOriginalDigest: Record<string, Post> = {};

  for (const transaction of arweaveTransactions) {
    const tags: Record<string, string> = {};

    for (const tag of transaction.tags) {
      const name = tag.get("name", { decode: true, string: true });
      const value = tag.get("value", { decode: true, string: true });
      tags[name] = value;
    }

    const appName = tags["App-Name"];
    if (appName !== "MirrorXYZ") {
      continue;
    }

    const originalDigest = tags["Original-Content-Digest"];
    if (postsByOriginalDigest[originalDigest]) {
      continue;
    }

    const rawData = transaction.get("data", { decode: true, string: true });
    postsByOriginalDigest[originalDigest] = {
      ...JSON.parse(rawData),
      originalDigest,
    };
  }

  return Object.values(postsByOriginalDigest);
}

And that’s really all there is to it! You can view all of the code above, together in this gist. It’s worth noting that the current arweave package does not support subscriptions. Because of this, we have to regularly check for new Arweave transactions in a manual way. This can be done via polling, or in the case of indexing.co, simply at request time.

Lastly, if you want to render a given post, the Post.content.body parameter is stored as markdown and can be roughly converted to HTML using a package like markdown-it.

import md from "markdown-it";

function PostView(post: Post) {
  return (
    <div dangerouslySetInnerHTML={{ __html: md().render(post.content.body) }} />
  );
}

Happy indexing!

There’s an often forgotten question in Data 3.0 - where do we actually put all our large data? That image of your favorite cat, videos from the last family trip, the unpublished book you’re working on - what is “home” for all that data? It’s easy to think “well if it’s web3, then it must be on-chain”, but that’s not always true, nor does it need to be! There’s a whole, growing world of decentralized data that has no tie back to a blockchain.

The simplest explanation is that putting data on-chain is expensive. Blockchains are, well, chains of “blocks”. Each of those blocks has a set of transactions, which in turn can include some amount of data. Each server participating in the network must then store all data for the blocks they help decentralize. For example, in Ethereum the default for many servers is to store the last one year of blocks. Furthermore, all data added to a block gets hashed to secure the given blockchain. The combination of these two requirements leads to limits on the amount of raw data within a block. This in turn creates a competitive effect for “block space”. Resulting in web3 users having to pay fees in correlation to the total number of users (often referred to as “gas”).

Now, it’s important to circle back to that “large” term we used at the beginning. What is does it mean to have a large piece of data? Take for instance the data required to reference moving funds from one account to another. This is often measured in a unit called “bytes” and is plenty small enough to keep on-chain. After all, this is the original use case blockchains! As you move into say a school paper though, you begin measuring data in “kilobytes”, or thousands of bytes. An image often runs into “megabytes” (millions) and videos get well into the “gigabyte” (billions) range. As a web3 user, it’s safe to assume that anything measured in more than bytes is too large. It's either impossible to store on-chain (due to block limits) or it's too expensive to do so.

Soo, where do we store all those cat photos then?

Thankfully, the innovation in Data 3.0 hasn’t left us high and dry. Let’s take a look at a few of the popular solutions for storing “large” data today:

IPFS

https://ipfs.io/

The Interplanetary File System is a free, “peer-to-peer” protocol for decentralizing data. IPFS was one of the earliest adopted solutions for storing data in web3 and continues to be a favorite for many. Getting started is easy (they even have a browser extension!) and the broader network improves the more users it has.

That reliance on adoption has been both a defining factor and a sort of achilles heel for the project. Users are only required to store data that they actually want to use themselves. For instance, there’s likely no [good] reason for me wanting to store your family videos, and so I won’t. But! if a meme is going viral and shared via IPFS, then every single viewer of that meme would also be sharing that data. From a practical perspective, this makes IPFS decentralized, but only temporarily so. In short, IPFS provides an easy, decentralized way to share data with others who want it.

Explore the desktop application and other ways to get started here. Or try out a hosted “pinning” provider like Pinata.

Filecoin

https://filecoin.io/

Also built by Protocol Labs, Filecoin aims to solve the "temporary" nature of IPFS by providing "contract-based" storage. Servers offer their storage capacity to the network and users pay to host their data for a fixed period of time. Fees get determined by the size of the data stored and the length of the “contract”. This storage market is then powered by a dedicated blockchain and currency, $FIL.

And behind this marketplace, the servers paid to store your data are all doing so via IPFS. That means that adoption of Filecoin is also adoption of IPFS.

Check out web3.storage or Fleek to explore early consumer applications for Filecoin.

Arweave

https://www.arweave.org/

Much like Filecoin, Arweave has a storage market with a dedicated blockchain and token ($AR). Rather than doing fixed term contracts though, Arweave promises permanent data storage. One upfront fee, storage forever.

Arweave accomplishes this permanence by gamifying data storage (details in the yellow paper). Each server of the network can choose to store whatever data they want. For instance, they could avoid storing illegal content by censoring what's stored. But those servers are also incentivized to store data that isn’t sufficiently decentralized. In other words, it's worth more to store data decentralized to only a few servers vs thousands. And over the span of the network, this results in all data always stored.

Check out their ArDrive to give it a spin.

(fun fact - this blog is hosted on Arweave via Mirror.xyz)

Storj

https://www.storj.io/

Storj is another competitor in the web3 storage space, but focuses on developers. It boasts full compatibility with AWS S3 so most developers can leverage decentralized storage out of the box. The end result being fast, reliable cloud storage that's also decentralized.

In general, servers in solutions like Filecoin and Arweave are only rewarded if they store the full chunk of data (e.g. an image). Storj is different. It takes a given chunk of data, encrypts it, and then shares smaller pieces with its network of servers. When a user wants to retrieve data, only 29 of those pieces are required to reconstruct the full chunk of data.

In this way, servers are incapable of being aware of the data they’re storing. This in turn allows Storj to control data at a network level; optimizing for speed and privacy along the way.

Storj does have a hosted interface for consumers to leverage their network. And of course there's documentation for developers to get started as well.

What should I use?

Here’s the skinny on when to use different Data 3.0 storage solutions today:

1. IPFS - free, easy to use, and temporary file sharing

2. Filecoin - fixed-length storage for a fee

3. Arweave - permanent storage for an upfront cost

4. Storj - developer-centric alternative to AWS S3

(For those wanting to dive in, feel free to scroll past this to “The DIY Section” below)

The internet today (web2) works, and more importantly gained adoption, in part thanks to the magical world of the Domain Name System (DNS for short). At the highest level, DNS effectively allows users of web2 to say “I want to visit longdogechallenge.com” and reliably be taken to the content (website) that lives there. Let’s break down how that works a bit:

1. A user enters the domain name, longdogechallenge.com, in their browser’s address bar

2. A DNS lookup occurs to resolve that domain name to a server’s IP address (an Internet Protocol address is a way to uniquely identify every device on the internet)

3. The browser is given the IP address for the requested domain name and asks the server behind that IP address for content

4. The server returns the content and the browser renders that to the end user (aka the resulting website)

(The above is admittedly a major simplification of DNS, and for those wishing to dive deeper, Cloudflare has a great technical write up)

For our purposes though, the general idea is that users of web2 can use DNS to get content behind an IP address while only knowing a domain name. This abstraction is enormously powerful. For starters, IP addresses are not easy to remember - the “common” IPv4 format looks something like 192.168.1.1 and the newer IPv6 format, 2001:0db8:85a3:0000:0000:8a2e:0370:7334, certainly isn’t any easier. IP addresses are designed for uniquely identifying the billions of internet devices and facilitating communication between them; not with humans. And of equal importance, because humans are accessing content via domain names, the server providing that content can be swapped out simply by changing which IP address a domain name points to.

Okay, but what does this have to do with web3?

Allowing mere humans to remember just google.com instead of 142.250.113.102 was a massive accessibility win for web2. The Ethereum Name System, ENS, provides a similar system for the Ethereum ecosystem. And while Ethereum is of course not equivalent to all of web3, it does represent a large portion of the current web3 user base in some capacity (even if you aren’t directly using Ethereum, many protocols and services are built atop the same basic technologies - more on that another time).

ENS, in short, allows a web3 user to say “I want to send funds to alice.eth” without actually knowing the wallet address behind alice.eth.

The DIY Section

Alright, we’ve covered what ENS is and why you should have one. Now let’s dig into how to acquire and use ENS; for both web3 users and developers.

For Users

Go buy an ENS! Seriously, go right now and buy one if you haven’t already. Some steps to get you started:

Setup your wallet

If you haven’t already, make sure you have a web3 wallet. MetaMask is a great place to get started if you haven’t already joined web3.

From there, make sure you have “some” Ethereum in your wallet. A quarter (0.25) ETH should be plenty to get started (unless you’re trying to get a highly prized ENS). If you don’t have any ETH yet, look towards exchanges like Coinbase to get you started.

Buy an ENS

Visit a site like https://app.ens.domains to look up names you’re interested in. There’s no wrong answer here - it’s a lot like choosing an email or a Twitter handle. If your name is available, then great! Follow the registration process (yes, this will involve 2 transactions) and secure your shiny new ENS.

If the name you like is already registered, then you have a few options for purchasing it on the “secondary markets”. ENS, at least in part, is officially a standard NFT collection. That means you can use your favorite NFT marketplaces to trade registered ENS items. Some options in no particular order:

• https://opensea.io/

• https://nft.coinbase.com/

• https://www.ens.vision/

Tie your ENS to your wallet address

This is the important piece. Now that you own your ENS, you need to make sure that it’s setup to point to your wallet address. There are two different pieces to this. Go back to https://app.ens.domains/ once again and click “My Account” in the upper right-hand corner and then do the following:

Define the ENS => wallet address lookup -- Select the ENS you just purchased and set the “Record” for “ETH” address to your wallet address. During this step, feel free to set as many of the other text records you see there as you want - this is all part of your new web3 profile.

Define the wallet address => ENS reverse lookup -- Go back to your “My Account” page and you’ll have an option to select your “Primary ENS Name”. Submit the corresponding transaction to make it permanent 🔥

NOTE: Both of the above steps will also require submitting Ethereum transactions and paying a small amount of ETH in “gas”. This is to pay the network to store the new data you just added.

And just like that, you’ll now have an official, named identity in web3 🎉 A simple way you can try it out is by looking up your name or wallet address on Etherscan and seeing that the two are linked in the results.

BONUS: Add an NFT as your avatar

Thanks to the wonderful world of web3, you can also link pieces of decentralized data together. Specifically in this case, you can say that the photo for your shiny new ENS name should be another NFT that you own. For instance, the avatar field for runninyeti.eth is set to `eip155:1/erc1155:0x495f947276749ce646f68ac8c248420045cb7b5e/87433597745683365960201176492736871205018189775129059226749288698845216112641` which points to this lovely image:

For a good overview of how to do this, check out this post by the ENS team.

For Developers

Also go buy an ENS! For yourself, for your project, whatever it might be, just start participating. See the “For Users” section above and follow along.

Now that that’s done, here’s some how-to’s for adding ENS functionality to your projects. We’re going to focus on using web3.js since it’s usable on both frontend and backend codebases, but these same principles can be used with any language.

We’re also going to focus on some of the low-hanging fruit based on what is available to the developer community today, but we encourage everyone to go far beyond this (e.g. “ENS Profiles” or deploying ENS on a private chain). There are also existing packages to obfuscate working with ENS (mainly JavaScript based, like ensjs), but for this post we’ll be looking at how we can do it ourselves and handle all data directly.

Look up the current owner of an ENS

tl;dr - ask the NFT contract for ENS who the current owner is

We’ll start with an easy one. To grab the address of the current owner of a given ENS name:

1. Define a web3.js contract instance with the ownerOf ABI input

2. Convert the name to a tokenId

3. Ask the contract who owns that tokenId

async function getOwnerAddressForENSName(name) {
  // do a basic null check
  if (!name) {
    return null;
  }

  // define a new contract instance for ENS NFTs
  const nftContract = new web3.eth.Contract(
    [
      {
        inputs: [{ internalType: 'uint256', name: 'tokenId', type: 'uint256' }],
        name: 'ownerOf',
        outputs: [{ internalType: 'address', name: '', type: 'address' }],
        stateMutability: 'view',
        type: 'function',
      },
    ],
    ENS_NFT_ADDRESS
  );

  // convert the name to a tokenId
  const tokenId = new BigNumber(Web3.utils.sha3(name.replace(/\.eth$/, ''))).toString();

  // ask the contract for the current owner and return the result
  return nftContract.methods
    .ownerOf(tokenId)
    .call()
    .catch(() => null);
}

Swap ENS names for wallet addresses

tl;dr - ask the ENS resolver for the address of a given ENS name

Let’s kick things up a notch. In order to get the resolved address for a given ENS name, we’ll need a few things:

1. First we want to define a contract instance for the ENS resolver with the addr method. The current contract address is 0x4976fb03C32e5B8cfe2b6cCB31c09Ba78EBaBa41

2. We then leverage the namehash algorithm to convert the given name to a sha3 node

3. Ask the contract for the current addr value of the given node

async function getAddressFromENSName(name) {
  // some basic validity checks
  // @NOTE: non .eth names are valid, but may require a different resolver contract address
  if (!name || !name.endsWith('.eth')) {
    return null;
  }

  // define a new contract instance for our ENS resolver
  const resolverContract = new web3.eth.Contract(
    [
      {
        constant: true,
        inputs: [{ internalType: 'bytes32', name: 'node', type: 'bytes32' }],
        name: 'addr',
        outputs: [{ internalType: 'address', name: '', type: 'address' }],
        payable: false,
        stateMutability: 'view',
        type: 'function',
      },
    ],
    ENS_RESOLVER_ADDRESS
  );

  // convert our name to a node hash
  const node = namehash(name);

  // ask the contract for the current address and return the result
  return resolverContract.methods
    .addr(node)
    .call()
    .catch(() => null);
}

Swap wallet addresses for ENS names

tl;dr - ask the ENS reverse resolver for the ENS name of a given address; double check against the name => address method above

A final example here, getting the ENS name for a given wallet address is almost identical to the other two methods above:

1. Again, start by defining a new contract instance for the ENS reverse resolver with the getNames method. The current contract address is 0x3671aE578E63FdF66ad4F3E12CC0c0d71Ac7510C

2. Ask the contract for the name that corresponds to our given address

3. Double check that the name => address mapping matches. While this step isn’t necessary, it’s best practice to ensure no one is manipulating the reverse resolver - more info from ENS here

async function getENSNameFromAddress(address) {
  // some basic validity checks
  if (!address || address.length !== 42) {
    return null;
  }

  // make sure our address is the checksum version
  address = Web3.utils.toChecksumAddress(address);

  // define a new contract instance for our ENS reverse resolver
  const reverseResolverContract = new web3.eth.Contract(
    [
      {
        inputs: [{ internalType: 'address[]', name: 'addresses', type: 'address[]' }],
        name: 'getNames',
        outputs: [{ internalType: 'string[]', name: 'r', type: 'string[]' }],
        stateMutability: 'view',
        type: 'function',
      },
    ],
    ENS_REVERSE_RESOLVER_ADDRESS
  );

  // ask the contract for the name that maps from the address
  // @NOTE: you can pass multiple addresses in a single call with this method
  const [name] = await reverseResolverContract.methods
    .getNames([address])
    .call()
    .catch(() => []);

  // @NOTE: ideally we double check that the reverse resolver is correct
  //        this can be done by comparing against the name => address mapping
  if (address !== (await getAddressFromENSName(name))) {
    return null;
  }

  return name;
}

Wrapping up

There you have it, that’s a quick run down on working with ENS! Both as a web3 user and as a developer in the space. For deeper dives, definitely check out the official ENS documentation and feel free to explore the code from this post on Github. And please reach out if you find this post interesting or just want to chat more about ENS, indexing, and Data 3.0.

Remember libraries? The walls of books and the fearless librarians somehow always knowing exactly where everything is. Well, two things: 1) libraries still exist, 2) those libraries are each indexed. Librarians around the world categorize all of the books under their purview into what are known as a “library catalogs”. These catalogs serve as a means of quickly finding books by given keywords: genres, authors, titles, etc.

The internet works in much the same way. There are billions of websites on the internet (our books in the example above) and each contains some amount of content. In order to find anything online we ask our almighty catalogs for direction - we “Google” a question or search someone’s name on Facebook. And for any of that to be possible, our trusted librarians (Google, Facebook, etc) had to first sort through the sea of content, categorize it, and then build intelligent indexes (catalogs). Now when you search “weather today”, Google is aware of sites that are relevant to the keyword “weather” and presents those. Of course, Google is also aware of your physical location, today’s date, your previous search history, and which website is paying the most to be matched on the keyword “weather” … but we’ll leave the darker details of the indexing industry for another day.

The point is, indexing is everywhere - a book’s table of contents, your phone’s set of contacts, the grocery list pinned to the fridge … you get the gist. Any time a given data set is too large to be easily consumed, indexing of some sort is employed to aid in its digestion.

Let’s refocus with some definitions:

1. Data: bits of information

2. Decentralization: the distribution of control and ownership away from a central authority

3. Index: a catalog to help find data more quickly

4. Web3: the decentralized web; empowering individuals to own their data

From the above we can surmise that “indexing”, as it relates to web3, is the act of cataloging decentralized information. Simple as that.

Generally speaking, indexing, in all scenarios, is done to make data more easily searched, and therefore to make that data more accessible. When the data itself is decentralized though, it opens the door to entirely new models of indexing. Take for instance the basic flow of indexed data in web2:

In the current, web2 world, centralized authorities control the flow of data. They are responsible for discovering, aggregating, indexing, and ultimately serving data. When a user performs a “search” for instance, Google chooses which information is relevant and provides it back to the end user. Historically this has been “okay”, but when those centralized authorities start to drop their “don’t be evil” mottos, it’s worth pausing to rethink this centralized model.

Enter the world of web3 and decentralized data:

There’s two important pieces to call out in this web3 scenario:

1. Users are adding their data directly to decentralized networks (Ethereum, IPFS, Arweave, etc)

2. Because these networks are decentralized, anybody can go through and index the data

That second point has some significant ramifications. For starters, that means that the individual user, whether that’s a single human or a company, can ultimately index and access their own data directly from the network; no middlemen deciding what information is “right”. Furthermore, this model doesn’t stop centralized authorities from also indexing that data and providing it to users - and that’s also good!

By enabling open data access, decentralized networks effectively create an incentive strategy for truly providing what’s best for the users. Because the centralized authorities no longer control the influx of data, they must cater to the needs of their users. Otherwise, a new competitor will come along to meet those needs. And, the best part of all of this is, that new competitor could simply be the users themselves.

Although this access isn’t always simple today in many decentralized networks, the barriers to entry are lowering and the potential continues to increase. For those interested in following along, this series on “Accessing Data 3.0” will dive deeper into the various aspects of data in web3 and how we can all start participating in it.

If you are reading this, congratulations on being a part of the next generation of innovation we like to call “Web 3.0” (or simply “web3”). You could be an artist looking to leverage the decentralized economy via an NFT launch, a builder working to further on-chain protocols with smart contracts, a business looking to leverage blockchain technologies to level up your internal logistics, or even just an innocent bystander looking to learn more about the next big thing - whatever the reason, we are glad you are here!

As the 2022 State of Crypto Report by a16z summarizes, the adoption of web3 is growing, here to stay, and is empowering the next wave of creators. And while all of that is extremely exciting, the road to web2 scale (i.e. billions of users) is far from paved. For instance, the estimated 7-50 million Ethereum users have all had to learn about concepts like private keys, wallets, gas fees … and that’s just to get started in the ecosystem.

Creators in web3 don’t fair any better. Building Decentralized Applications (dApps) often requires a working knowledge of blockchain data types, smart contract deployment and management strategies (hint: it’s not like traditional software development), gas fees, tokenomics, pseudo-anonymous “users", etc, etc. There is a silver lining for creators and users alike though: the growth of web3 can largely be attributed to trailblazing individuals and companies *continuing* to work on solving everything from user identity to transaction throughput of the underlying networks.

Data 3.0

One often overlooked aspect of the entire ecosystem though is the accessibility of data itself. As builders we’ve grown [relatively] accustomed to sending data out into the ether (🥁) without knowing how we will get it back in any sort of usable form. This leads to reliance on centralized sources to figure that out on our behalf, and charge us for that service - at least partially defeating the original purpose of decentralizing the data to begin with. To truly enable the future of decentralized data we must rethink our approach to Data 3.0.

An Example

Let’s take a look at one of the most common examples in the space today: tracking NFT project owners. NFT (Non-Fungible Token) projects have significantly contributed to the rise of adoption in web3 over the last couple of years. OpenSea for instance, the largest NFT marketplace, has facilitated up to $4.8 billion USD in sales in a single month at the height of the last crypto cycle (source).

What makes NFT projects particularly interesting for our Data 3.0 example though is their relative ease to develop (relative to web3 in general that is) combined with the repeated centralization these projects utilize. At their base layer, NFTs are smart contracts (permanent, unchangeable software) on a given blockchain (often Ethereum) that enable users to own, and transfer, a fixed number of items. NFT project creators can write and deploy their smart contracts, distribute the initial set of items (whether through giving them away or letting web3 users buy them from the smart contract directly), and then leverage a marketplace like OpenSea to enable trades between web3 users.

An overly simplified lifecycle of an NFT project may go something like this:

1. Creator A launches the highly successful NFT Project X

2. Thousands of web3 users trade NFT Project X items as owners

3. … time passes …

4. Creator A now wants to give each of the current owners of Project X a gift for being such a great community

5. Creator A realizes they can’t readily get a list of current owners from the blockchain directly and is therefore presented with the following choices:

   1. Scrap the gift idea

   2. Write a script to check the current owner of every item against the blockchain. This requires web3 development knowledge and would only generate point in time snapshots making it difficult to keep up with items that may be rapidly switching hands.

   3. Learn how to build an indexer to crawl and subscribe to updates from the blockchain, replaying every trade sequentially, to determine the current owner of each item. This solves the problem, but requires even more web3 development knowhow and the ongoing maintenance of the indexer itself.

   4. Use a readily available API from a centralized source (that has built their own indexer) to provide this data. This is by far the easiest solution, but requires reliance on a 3rd party provider that may or may not be generally reliable (looking at you OpenSea…) and often incurs a service cost.

6. Creator A chooses a 3rd party API provider

7. Owners of NFT Project X get their gifts and everyone is happy

What’s the catch?

This all sounds fine and dandy right? And it mostly is. Generally the 3rd party providers can be trusted to provide real time, consistent data. But what happens when that 3rd party isn’t reliable or changes the way it serves that data? Or more importantly, what happens when Creator A has a use case no longer supported by the standard offering (e.g. they want to only give gifts to owners that have held their NFTs more than 90 days)? And all of this is just the tip of the iceberg.

While providers are rapidly appearing to fill the holes in Data 3.0, most of them are doing so in use case specific ways; OpenSea for NFT data, Alchemy for raw blockchain data, etc. Ethereum’s developer docs - which are honestly great - don’t even mention “indexing” except to point at a 3rd party provider, The Graph. And The Graph, which describes itself as “an indexing protocol for querying networks like Ethereum and IPFS”, has limited support for customizations and forces participation in something akin to an indexing marketplace.

There’s a missing link in the Data 3.0 lifecycle and the impacts are only just starting to be felt. Web3 creators should be empowered to own, leverage, and define their data as they see fit. Join us in shaping the future of Data 3.0.