You’re getting this email because you’re a member of the Tableland/Textile community.

If you missed it last week, Textile and 3Box Labs have merged to build the first decentralized intelligence network empowering agents to autonomously store, share, and exchange knowledge onchain. We’re launching a new Discord community for this network.

To thank you for being an early supporter, we’re giving you early access to our new Discord. By joining the server in the next 24 hours, you’ll get:

• An OG role exclusive to current Tableland Discord members (Snapshot before Feb 4, 2025)

• Skip the waiting room, getting instant access to the full list of channels

• Be part of the founding community shaping this new ecosystem

Grab your OG role now (link expires in 24 hours)

See you there!

As CEO of Textile, for years I've had the privilege of working with 11 incredible builders who've pushed the boundaries in decentralized data. Today, Danny and the brilliant 3Box Labs crew are joining the Textile family. We've been friendly competitors, occasional collaborators, but always united by three core beliefs: public networks, developer-first thinking, and the open flow of data.

Both of our teams have deep roots in decentralized data infrastructure. We’ve consistently delivered innovations trusted by major players in the space, from Gitcoin to MetaMask. Through years of both competition and collaboration, we’ve developed a shared vision that AI will be embedded within every application that creates or consumes data. As specialized AI agents become central to this future, we're focused on launching a network that serves their unique needs with AI-first data protocols. It’s perfect timing; the multi-agent future is here and it relies on much of what our two companies have built to date.

This merger brings together an unparalleled combination of talent and technology:

• Basin Network

• Ceramic Network

• Tableland DB

• ComposeDB

• 30+ world-class employees

• A thriving global community of 50,000+ developers

In the coming months, we'll be launching the first multi-agent intelligence network that enables AI agents to store, exchange, and monetize their specialized intelligence and developed skills with other agents around the world. With our network, agents can collaborate on tasks and up-level their capabilities at significantly less cost and overhead compared to training foundational models. We're already working with leading partners who share our vision, including: Gaia, Story Protocol, Index Network, AgentDB, Collab.Land, Ocean, Matrix.one, and a few others.

As for Textile and 3Box products, they will be fully-integrated into the new network, playing significant roles in building out a comprehensive data infrastructure that underpins the system.

In the midst of focusing our efforts on advancing AI, our core principles remain unchanged:

• AI serves humans, not the reverse

• Knowledge must flow freely

• Collaboration beats competition

• The future stays open

This isn't just another merger. It's a sprint toward our future. And we're ready.

Follow @textileio for updates, including details about our upcoming public testnet and what's next.

Read the full story

Begin transmission…

by Avichal Pandey

Basin is still in its research phase, and we’re exploring various data availability approaches, including XOR-based (and related) schemes for erasure encoding. In this architecture, the validators create an auxiliary data structure with a dual purpose. It can be used to repair data in case of data loss or data withholding attacks and to ensure that the validators are not deleting the old data they are supposed to store.

One common consideration in DA systems is to make the “data storage” requirement optional. This would enable light nodes to validate that no data was lost without downloading all the data. This guarantee is typically provided using data availability sampling and fraud proofs. However, fraud proofs introduce a “challenge” window, which deters the system’s finality. Fraud proofs are also tricky to implement and test.

An alternative to DAS and fraud proofs is to validate erasure encoding itself using SNARKs. This idea first surfaced in the Celestia forum last year.

If you created a way to generate a zk proof of correct erasure coding, then light nodes would not have to wait for a fraud proof window after sampling each block to consider it final. Second, we wouldn’t need bad encoding fraud proofs (BEFPs) which require the light node to download an entire row or column of the block data. Third, this unlocks the possibility of having consensus nodes vote on blocks without downloading all the data.

In this post, I want to build upon this idea to see how feasible it is to “verifiably” compute the XOR-based codes using the modern SNARK frameworks. For example, I want to answer the questions like:

• How much time does it take to build a proof of encoding 1GB blob?

• How big is the proof size?

• How suited is a particular SNARK framework for this task

XOR-based codes are essentially a sequence of XOR operations. In my first experiment, I am considering a 1GB object that gets broken into 1KB chunks. Bitwise operations such as XOR had been the Achilles heel of SNARKs for a long time. However, recently, this problem has been addressed using two different types of solutions:

• Lookup arguments

• Binary fields and tower extension

The following is the crux of why bitwise operations are complicated in SNARKs. The arithmetic circuits used in SNARKs are defined over finite fields. In the case of elliptic curve-based SNARKs, these fields could be as large as 256 bits to make the protocol secure. Arithmetic operations such as additions and multiplications can be performed natively in the field while performing a bitwise operation requires simulating the operands and operations, which creates some overhead. The following image shows how converting an arithmetic circuit to a boolean circuit makes it bigger and wastes space by storing 1 bit in a data type used to represent a field element.

Lookup tables addressed this issue by precomputing all the results of bitwise operations and indexing them in a table that the prover can access. Another way to address this problem is to work with proving systems based on binary fields instead of prime fields (almost all SNARKs use prime fields). Binius is a pioneer in this area and is widely considered a paradigm shift in SNARKs.

This Binius project is at a very early stage in terms of implementation. As a reference, we have a few papers and talks. However, in terms of concrete implementation, we only have a non-production-ready POC.

The Binius approach is well suited to our case for the following reasons:

• XOR-based erasure codes can be performed in binary fields without much overhead.

• Unlike other elliptic curve-based SNARKs (Groth16, Plonk, etc.) and some commitment schemes like KZG, it doesn’t require a trusted setup.

• Unlike other hash-based SNARKs (STARKs), bitwise operations don’t incur overhead.

• Probably the fastest-known proving times.

• Support of proof composition.

To test Binius, I have modified one of their examples to perform random XOR operations instead of AND operations. For this experiment, we are assuming that some N number of XOR operations is a substitute for actual encoding computation.

The following are observations from my local machine (a Mac M1 with ten cores and 32 GB memory).

This experiment is just a first attempt at the problem. It only approximates the actual XOR-based encoding computation. The circuit might need to have more constraints to be secure. Let us assume a high enough error margin. For instance, let’s say generating a proof for actual encoding will take 10x more time. It will still finish in ~600ms!

These results are highly encouraging compared to my expectations. I thought generating a practical proof might take 10-15 minutes. My first naive implementation took 10 minutes for an object size of 125MB (1/4th the size of the object in the table). With proper batching of the computation trace, prover latency was dramatically reduced. Furthermore, this is just a first glance at the Binius repo. There might be more levers to pull to optimize proving time.

• it is extremely new. The Binius reference implementation is a POC not ready for production use. There is no documentation or dev chat to ask questions, and the only way to learn how to use it is by looking at existing examples.

• it offers a low-level API for generating SNARK proofs. We must write the computation trace and specify the SNARK circuit by hand, constraint by constraint. We have to deal with polynomials. Writing SNARKs at this level of abstraction is slow and error-prone. It is analogous to using assembly language to write our programs.

The first issue should be resolved with time as the project matures. ZKVMs addresses the second issue.

Today, most RISC-V-based ZKVMs (RISC Zero, SP1, Jolt, Nexus, etc.) can take a high-level Rust program as input and produce a SNARK proof. Hiding all the complexity of generating computation trace, circuit constraints, and polynomial manipulation.

Unfortunately, as of today, most ZKVMs are based on STARKs not Binius. However, a16z’s Jolt has Binius support on the roadmap, and work on it has already started.

This leads me to conclude that using Jolt with Binius could enable validity proofs. Working with ZKVM also allows us to swap out the VM later. For instance, we could start with RISC Zero since it has great documentation and examples but swap it out for Jolt when it supports Binius.

by Dan Buchholz

We created a demo for how to use Basin with a "hosted" server setup:

• A backend wallet sends object storage transactions on behalf of a requesting user.

• You can /set a file—up to 100 MB in the demo (but the protocol itself has higher limits).

• You can also /list all objects in the store.

Clone the repo and review the README instructions to get started!: textileio/demo-basin-server

From the client's perspective, curl requests are sent to either push files:

curl -X POST -H 'Content-Type: multipart/form-data' \
--form 'address=0x79447b8db3a9d23f7db75ae724ba450b7b8dd7b0' \
--form 'key=hello/test' \
--form 'file=@test.dat' \
http://localhost:8081/set

// this will log the onchain transaction for storing the object
{
  "data": "bafy2bzacedxeu3g3uazqpn2ln7yvyfhc6ilj3vi5bf3h6usvygsxaub7paws4",
  "gas_used": 4311212,
  "hash": "1DDBED9D0398C4A7C0B2E0DE99BCE77C34232CC1AD45E9304F990A416ACAF830",
  "height": "956895",
  "status": "committed"
}

And also list the data in the store:

curl -X POST -H 'Content-Type: application/json' \
-d '{"prefix": "hello/", "limit": 10}' \
http://localhost:8081/list

// this logs data under the common prefix
{
  "common_prefixes": [],
  "objects": [
    {
      "key": "hello/world",
      "value": {
        "cid": "bafybeid3weurg3gvyoi7nisadzolomlvoxoppe2sesktnpvdve3256n5tq",
        "metadata": {},
        "resolved": true,
        "size": 5
      }
    }
  ]
}

The backend is written in Rust and makes use of the Basin SDK. Running the server will let you log information about incoming requests:

2024-07-20T17:49:27.589-04:00 - INFO Starting server at 127.0.0.1:8081
2024-07-20T17:50:12.015-04:00 - INFO {"body":"{\"multipart/form-data; boundary=------------------------u3Cayud8pzT4bsvlrHH4Z5\"}","route":"set"}
2024-07-20T17:50:14.691-04:00 - INFO {"client_addr":"127.0.0.1:50064","duration_ms":2676,"method":"POST","path":"/set","status":200}
2024-07-20T17:50:33.952-04:00 - INFO {"body":"{prefix: Some(\"hello/\"), delimiter: None, offset: None, limit: Some(10)}","route":"list"}
2024-07-20T17:50:34.371-04:00 - INFO {"client_addr":"127.0.0.1:50068","duration_ms":419,"method":"POST","path":"/list","status":200}

You could imagine adding other endpoints like getting the actual file at the object or deleting data, but hopefully, the demo server is enough to get you started!

• Stark by hand: An in-depth tutorial on STARKs internals by tge RISC Zero team.

Want to dive deeper, ask questions, or just nerd out with us? Jump into our Telegram or Discord—including weekly research office hours or developer office hours. And if you’d like to discuss any of these topics in more detail, comment on the issue over in GitHub!

Are you enjoying Weeknotes? We’d love your feedback—if you fill out a quick survey, we’ll be sure to reach out directly with community initiatives in the future!: Fill out the form here

Begin transmission…

It's been a few weeks since our last post, largely due to an all-company offsite (outside of Aspen, CO), travels, and some holidays. But, we've still been building new capabilities for both Tableland and Basin, plus, we attended some events (aka EthCC).

by Dan Buchholz

Base has been one of the most sought-after chain integrations from our developer community. Generally speaking, there's a ton of activity on Base, including Farcaster frames functionality (which we're huge fans of).

We wanted to make sure developers were able to use Base on both mainnet and the Sepolia testnet for onchain summer. Both of these chains are now officially supported!

Check out the docs that describe the official Tableland contracts on Base and useful other information: here

by Marla Natoli

EthCC was buzzing with builders, creators, and organizations dedicated to innovating around AI, data management, and so much more. Our team left the week feeling more optimistic than ever about all the possibilities that can be unlocked with decentralized database infrastructure, both near and long-term. A few highlights:

• We co-hosted our 2nd Proof of Data Summit with Ceramic, which was a half-day event filled with thought leadership talks and panels from teams including DIMO, WeatherXM, IoTex, Filecoin, Vana, and more. We'll be sharing recordings of sessions in the coming weeks

• Our Partnerships lead, Marla Natoli, presented at the FIL Dev Summit where she talked about Basin and the opportunities and use cases it unlocks for DePIN & AI organizations

• Marla also presented alongside Ally from Lilypad at the Filecoin Foundation Network Base event, where they spoke about synthetic data and ways that Basin aims to unlock more innovation for AI and ML organizations. A joint POC is in the works and will be showcased soon.

• We connected with existing friends and partners, and opened up conversations around new partnerships that we're sure will lead to some pretty epic POCs

We'll be sharing recordings and highlights from the event, so be sure to follow us on socials and subscribe to weeknotes for updates!

The Studio has made numerous incremental updates to the overall UX flow.

ICYMI—a sidebar now organizes the layout more intuitively at both the "user" and "project" levels. Projects are outlined a bit more clearly in a stacked view, too.

Once you click on a project, it'll display a sidebar with the Console (new!), table definitions, settings, and general metadata for the projects and their environment. The breadcrumbs also reflect the hierarchy in a more readable way.

Now, you can read and write table data directly in the UI! Every project has a Console in the sidebar. This opens up a query interface where you can execute SQL queries on your existing table definitions. The examples below show the results from a mutating query (logs the transaction information):

And a read query (logs the table data):

A lot is happening with Basin that isn't quite live yet. There are a few things to keep an eye out for in the coming future:

• AWS S3 compatibility: A fully compatible S3 client with the Basin Rust SDK. We even tested it out on ClickHouse to ensure it can be used with other 3rd party solutions. It's in the final review stage and should be ready in the coming weeks.

  

  In fact, we have a quick demo of the Basin S3 adapter with a MinIO client: here.

• Faucet service: This won't be live until a month or two from now (pending some WIP infrastructure changes), but the work is complete. We've stocked up on a ton of tFIL (s/o ChainSafe) and will deploy a Basin subnet faucet service to let you quickly get started with development. This will allow you to "register" new accounts (e.g., bridging accounts from EVM → FVM) on the Basin subnet within seconds, subsequently letting you create and write to the object store with little friction.

• JavaScript SDK: You may have read our previous posts about wasm support for Rust. Our goal was to compile our Rust SDK to wasm, but due to some transitive dependencies that don't support the wasm32 architecture, it isn't possible at this time without significant workarounds. Instead, we'll be developing a TypeScript-based SDK that closely follows the Rust implementation. The work is just getting started and still weeks away, but JS/TS support opens up a ton of use cases on the web development side of things!

There are more things we'll be sharing soon, but those are some of the more near-term items we've been working on. If you've had a change to work with the Basin CLI or Rust SDK, definitely let us know in Discord and keep use posted on anything you'd like to see there!

• Discussion on ZKVMs and ZKTLS: here

Want to dive deeper, ask questions, or just nerd out with us? Jump into our Telegram or Discord—including weekly research office hours or developer office hours. And if you’d like to discuss any of these topics in more detail, comment on the issue over in GitHub!

Are you enjoying Weeknotes? We’d love your feedback—if you fill out a quick survey, we’ll be sure to reach out directly with community initiatives in the future!: Fill out the form here

Begin transmission…

by Dan Buchholz

We've officially published a new version of our CLI and SDK, which includes Base Sepolia as a supported chain. You can read about the Base chain information on our docs site: here. All you really need to do is provide a chain RPC URL pointing to Base Sepolia—and also have testnet ETH in your wallet—and you're be good to go!

If you want to get started quickly, you can do this in the Studio. When you're ready to deploy a table, Base Sepolia is an option under the testnets section.

by Dan Buchholz

We'll be at EthCC in early July and present some talks about Basin and Tableland (see below). Earlier this year at ETHDenver, we cohosted the Proof of Data event, which was all about decentralized data. The second iteration of this event will be held on July 9th, and we'll have talks given by IoTeX, DIMO, WeatherXM, and Filecoin!

Sign up for the Proof of Data event in Brussels: https://lu.ma/0vifocie

It's always great to meet up IRL, and we're also down to attend/cosponsor something else, too. We hope to have some new Basin-related POCs to share by then, and if that's something you're interested in working on, definitely reach out!

Fill out this quick form if you're interested in working with us! https://basin.textile.io/partnerwithus

by Jim Kosem

When you give a talk in front of a bunch of people you should probably have your facts straight. This is especially the case when you’re a designer giving a talk at a tech event, such as IPFS Camp in Brussels where I’ll be talking about Studio, which I’ve been working on for the majority of the past year.

The funny thing is as part of a design track, I just might be speaking to other designers who are also working on developer tools and the developer experience such as myself. This led me to reexamine what I’ve been designing for and what the developer experience is for, meaning data and databases. So I had to go back and review just what those things mean and what they mean for interfaces and interactions. When you get lost in the weeds as it were with an application you’ve been working on for a while, you often lose track of the basics of what you’re doing, why data, why tables, why ownership, so talking about a forcing function to reexamine the first principles of your work is usually not a bad thing.

• Introduction to STARKs by Eli Ben-Sasson: https://www.youtube.com/watch?v=jg9KSNOO2XY

• NVIDIA’s new open synthetic data pipeline: https://blogs.nvidia.com/blog/nemotron-4-synthetic-data-generation-llm-training

• Introduction to State Space models: https://freedium.cfd/https://medium.com/thedeephub/gentle-introduction-to-state-space-models-e8cd7501e0cf

• S3 is showing its age: https://materializedview.io/p/s3-is-showing-its-age

Want to dive deeper, ask questions, or just nerd out with us? Jump into our Telegram or Discord—including weekly research office hours or developer office hours. And if you’d like to discuss any of these topics in more detail, comment on the issue over in GitHub!

Are you enjoying Weeknotes? We’d love your feedback—if you fill out a quick survey, we’ll be sure to reach out directly with community initiatives in the future!: Fill out the form here

Decentralized Data Solutions: Insights and Opportunities

by Andrew Hill

Our recent blog post is essential reading for web3 builders, offering a thorough overview of the decentralized data landscape, complete with insights from a comprehensive survey of 32 developers and leaders. This piece highlights the opportunities and challenges in adopting decentralized infrastructure. It also calls out some key opportunities for all of us to improve data protocols in web3.

by Dan Buchholz

We published a blog about how Basin (new & improved) works and what makes its design unique. There's still a lot that we haven't fully dug into or implemented yet and are still researching, but this post should provide additional insights into the network.

The gist of it is that it's built on Filecoin's IPC subnet architecture for fast and verifiable data—specifically, object storage. If you want to learn more, the post talks about the general consensus mechanism, parent-child subnets/state, ABCI & FVM, and more!

by Dan Buchholz

We're adding a feature to the Basin CLI that will let nodes run a daemon / API server to auto-fund new accounts. The problem this helps address is around DX. If you're a new user coming from the EVM world to the FVM and the Basin (IPC) subnet, your EVM account is compatible with the FVM, but there's a bit of "registration" required onchain.

Filecoin has an Ethereum Account Manager (EAM) that lets EVM 0x addresses delegate to a f410 address, which lets the hex-prefixed address work natively on Filecoin. But, for the 0x address to work, it needs to be registered onchain with the EAM. This requires a transaction to be made.

In the coming weeks, we'll be adding a feature to the CLI that abstracts this process away. The core logic required here is a hosted/funded backend wallet that will send a small amount of FIL to a new account. Since Basin is only live of Filecoin Calibration, it doesn't actually "cost" anything for the backend wallet to fund new accounts. If you haven't had a chance to use the Basin CLI yet, get started while we implement it! Right now, you'll have to get your own tFIL testnet currency and deposit it into the Basin FVM subnet, and then you're good to go.

Below is a skeleton of the API server. It uses warp as the server framework, and tokio for async operations. There's a single /fund endpoint that takes an address as a parameter, so a requires like /fund/0x1234… will trigger an FVM and Basin subnet transaction to execute sending some tFIL to the specified address.

use std::convert::Infallible;

use dotenv::dotenv;
use serde::Serialize;
use serde_json::json;
use std::sync::Arc;
use tokio::sync::Mutex;
use warp::{http::StatusCode, Filter, Rejection, Reply};

#[tokio::main]
async fn main() {
    dotenv().ok();

    let state = Arc::new(Mutex::new(State::new()));

    let fund = warp::post()
        .and(warp::path!("fund" / String))
        .and(with_state(state.clone()))
        .and_then(handle_fund);

    let router = fund
        .with(
            warp::cors()
                .allow_any_origin()
                .allow_headers(vec!["Content-Type"])
                .allow_methods(vec!["POST"]),
        )
        .recover(handle_rejection);

    warp::serve(router).run(([127, 0, 0, 1], 8081)).await;
}

The handle_fund and handle_rejection are handlers that execute logic once the endpoint is hit. Aside from those, you can also pass some local state to the API—for example, perhaps you want to implement a simple rate-limiting feature. Passing/calling with_state lets you perform some operations to track some state every time the API is hit.

/// State for the API
struct State {
    // Rate limit data?
}

impl State {
    /// Create new state.
    pub fn new() -> Self {
        State {}
    }
}

/// Filter to pass the state to the request handlers.
fn with_state(
    state: Arc<Mutex<State>>,
) -> impl Filter<Extract = (Arc<Mutex<State>>,), Error = Infallible> + Clone {
    warp::any().map(move || state.clone())
}

The rejection handler is nice because it lets you control responses upon 404s, internal server errors, etc.:

/// Generic request error.
#[derive(Clone, Debug)]
struct BadRequest {
    message: String,
}

impl warp::reject::Reject for BadRequest {}

/// Custom error message with status code.
#[derive(Clone, Debug, Serialize)]
struct ErrorMessage {
    code: u16,
    message: String,
}

/// Rejection handler for the API.
async fn handle_rejection(err: Rejection) -> Result<impl Reply, Infallible> {
    let (code, message) = if err.is_not_found() {
        (StatusCode::NOT_FOUND, "Not Found".to_string())
    } else if let Some(e) = err.find::<BadRequest>() {
        let err = e.to_owned();
        (StatusCode::BAD_REQUEST, err.message)
    } else {
        (StatusCode::INTERNAL_SERVER_ERROR, format!("{:?}", err))
    };

    let reply = warp::reply::json(&ErrorMessage {
        code: code.as_u16(),
        message,
    });

    Ok(warp::reply::with_status(reply, code))
}

Lastly, this is a stubbed-out example of the actual handler for the endpoint. It's where you'd do some processing or make other function calls that handle the bulk of the logic, such as the onchain calls to the FVM for the "registration" process:

/// Handles the `/fund/<address>` request.
async fn handle_fund(
    address: String,
    state: Arc<Mutex<State>>,
) -> Result<impl warp::Reply, warp::Rejection> {
    // Do stuff

    let json = json!({"tx": "stuff"});
    Ok(warp::reply::json(&json))
}

by Jim Kosem

There are lots of questions about how AI will impact “knowledge” work moving forward as we all huddle in front of our machines for terminal velocity to be achieved to see if things calm down. One thing is for sure though in the meanwhile, we should think about how to use this stuff. There are no shortage of screeds of enthusiasm and self flagellating doomsaying, but one thing you don’t see often are patterns.

Design patterns are quite simply codified recommendations of use. You should design or build software this way or that. The interesting thing here is establishing how AI fits in terms of scope. The current approach we’ve seen thus far is that you type something into a box and AI does everything. This article examines it from a different and more nuanced standpoint, one that the staid and solid enterprise types like Microsoft have already known, which is you need to be clever where you slot it in and how much. You need to think which nail the hammer is going to pound down, rather than demolish a bed of nails with a ton of concrete. The author also takes this slightly further in exploring the notions of this fit being alongside or separate, layered or integrated. Perhaps most interesting as paragraph upon paragraph is written or perhaps even generated about provenance of produced words, images and sounds is the notion of awareness and action.

All in all, it’s a good, middle of the road piece of thinking about AI in the tools that people like us at Textile are building.

• Talk on how to deisgn protocols by Eddy Lazzarin (a16z crypto CTO) https://www.youtube.com/watch?v=WGfS6pPJ5jo

• Native vs Foreign DA: Interesting discussion on trust assumptions on different types of DA https://paragraph.xyz/@vinny/native-vs-non-native-da

• Backstory of Lasso and Jolt: https://www.youtube.com/watch?v=Uy0Qap3fePI

• A promising EIP (7683) that would standardize an API for cross chain trade execution. Vatalik seems to think this is promising, which is a good sign for community support. Plus, it's being driven by Uniswap and Across—some more info here.

Want to dive deeper, ask questions, or just nerd out with us? Jump into our Telegram or Discord—including weekly research office hours or developer office hours. And if you’d like to discuss any of these topics in more detail, comment on the issue over in GitHub!

Are you enjoying Weeknotes? We’d love your feedback—if you fill out a quick survey, we’ll be sure to reach out directly with community initiatives in the future!: Fill out the form here

TLDR

• Survey insights: Key findings from 32 developers and leaders in crypto projects on decentralized data solutions.

• Security & compliance: 36% prioritize enhanced security; 27% highlight regulatory compliance as crucial.

• Adoption barriers: Transition costs (36%) and tool fragmentation (21%) hinder widespread adoption.

• Blockchain data protocols: Opportunities and challenges in AI, cloud infrastructure, and the data economy.

• A solution for object storage: Introducing Basin, a decentralized data storage solution on Filecoin, addressing high throughput, security, and cost efficiency.

There’s a powerful shift underway in the web3 space, one that's long overdue. Projects are now actively seeking decentralized solutions for data storage, processing, and delivery in ways we only dreamed of two years ago. With the rapid advancement of scalable blockchain protocols and the growth of sectors like compute, crypto-AI, and DePIN, understanding the decentralized data landscape has become crucial for developers and businesses alike.

To get a better handle on this shift—why it's happening and what could accelerate it—we conducted a comprehensive survey of 32 developers and leaders from crypto-based projects.

• 80% of respondents were company leaders.

• Half of the projects we surveyed raised over $1,000,000 in funding.

• Most companies estimated the value of their data to be in the tens of thousands of dollars or more.

We asked our research group questions about a wide range of topics related to their infrastructure decisions, technology aspirations, and the trade-offs they weigh when executing their own visions. Our findings revealed a clear inclination towards decentralized data architectures with lagging adoption. Here’s what we found:

• Security and Compliance: 36% of respondents highlighted enhanced security as a primary driver for decentralized infrastructure. This move promises enhanced resilience and tamper-proof systems, which are critical in a landscape fraught with cybersecurity threats. Additionally, 27% cited regulatory compliance as a crucial factor, reflecting a growing emphasis on data sovereignty and user privacy.

• Barriers to Adoption: Despite the apparent benefits, two major barriers hinder widespread adoption:

  • Cost Concerns: Transition costs, from initial technology investments to ongoing development efforts, were cited by 36% of the projects as a deterrent. Practically speaking, it’s not simple to leave the centralized cloud and adopt a decentralized infrastructure.

  • Tool Fragmentation: 21% of respondents pointed to the fragmentation of available tools, complicating decentralized solutions integration and operational efficiency.

These insights suggest a critical need for more integrated and cost-effective solutions to facilitate the transition from traditional cloud-based models to decentralized data architectures.

Blockchain data protocols are at a pivotal juncture, with the potential to fundamentally reshape AI, cloud infrastructure, and the data economy. The growth of data availability networks and the development of decentralized physical infrastructure networks (DePINs) drive optimism about a more decentralized future. However, the transition is not without its challenges. The fragmented landscape of existing tools and the high costs associated with adopting the latest technologies are significant hurdles that need addressing.

Once a project has one foot in AWS, it's hard to justify not using the cloud for everything. The biggest competition for web3 data protocols are centralized cloud providers. And to beat the cloud, we still need to simplify the path to adoption. We also need to provide more comprehensive solutions to complex problems in systems that require data services.

In response to our survey, participants pinpointed the components of their data architecture they hope to transition to web3 alternatives:

• 55% - Caching and retrieval (e.g., CDN)

• 52% - High throughput data storage (e.g., S3)

• 30% - Streaming (e.g., Kafka)

In response to these needs, we’re building Basin, a decentralized data store designed to meet modern data systems' high-throughput and security demands. Built on the robust Filecoin network, Basin offers a scalable solution for developers needing reliable and accessible data storage options. Key features include:

• Verifiable Data for Trust: Ensure data integrity with cryptographic security, making data tamper-proof and trustworthy.

• High Throughput and Availability: Basin's architecture is designed for high performance, ensuring data is accessible whenever needed, supported by cryptographic proofs.

• Cost Efficiency: With a competitive pricing model, Basin allows projects to scale their storage needs without incurring prohibitive costs.

The future of data in the web3 space is poised for transformative growth, driven by innovations like Basin. As we continue to tackle tool fragmentation and high adoption costs, your involvement and feedback are invaluable. Together, we can build a robust data infrastructure that meets today's needs but also anticipates tomorrow's challenges.

Background

Basin is a decentralized data layer, enabled by subnets that are purpose-built for onchain data storage. It is built on top of the Filecoin Virtual Machine (FVM) and provides a horizontally scalable, verifiable, and cost-effective data availability layer for onchain applications, networks (e.g., DePINs), and services. The first data Layer 2 (L2). A handful of specialized "machines" for object storage and data anchoring power the Basin's featured data services.

Basin is currently an alpha testnet, and the network is subject to fortnightly changes with rolling updates. Please be aware that the network may be reset at any time, and data may be deleted every two weeks. A more stable testnet will be released in the future that won't have this limitation.

Basin is developed using the Filecoin InterPlanetary Consensus (IPC) framework. IPC is a blockchain scaling solution and architectural design that is an alternative to existing L2 scaling solutions and is based on the design principles of on-demand horizontal scaling.

IPC allows for permissionless creation of new blockchain subsystems called subnets. Subnets are organized hierarchically, enabling seamless internal communication and eliminating the need for cross-chain bridges. Each subnet can employ its specific consensus algorithm and inherit security from parent subnets, and this structure facilitates hosting or sharding applications based on performance or cost needs.

Subnet operators must run a full validator node for both the parent and the child subnet. The parent subnet is responsible for the security of the child subnet, and the child subnet is responsible for its own consensus and state transitions.

Basin is currently a single child subnet rooted in its parent Filecoin Calibration testnet. Future versions of the Basin will align more closely with IPC's permissionless subnet spawning and configurable consensus mechanisms.

CometBFT (formerly known as Tendermint) helps the Basin achieve state machine replication across all nodes in the network. Its consensus algorithm is based on a variant of Practical Byzantine Fault Tolerance (PBFT) and relies on a round-robin proposer selection mechanism while incorporating elements that improve PBFT's performance and communication overhead. It is a fast, battle-tested, and well-designed consensus engine.

CometBFT is a standalone process that issues commands to Fendermint and exposes a public JSON RPC API, much like the Ethereum JSON RPC API.

Basin's blockchain functionality is exposed as a unified ABCI++ application controlled by CometBFT.

Application Blockchain Interfaces (ABCI) programs interface between CometBFT and the actual state machine being replicated. ABCIs implement deterministic state machines to be securely replicated by the CometBFT consensus engine. The "++" in ABCI++ refers to additional functionality that CometBFT enables compared to the original ABCI, which helps improve the overall scalability and feature surface area.

Fendermint is a specialized ABCI++ interface to the Filecoin Virtual Machines (FVM) and Ethereum-compatible FEVM. It exposes FEVM/FVM-specific functionality within subnets, allowing the Basin subnets to behave like Filecoin but with custom parameters to greatly improve throughput and features. Fendermint is also a standalone process that includes:

• Interpreters: Responsible for handling commands from CometBFT.

• Snapshots: CAR files that can be offered to peers for quick chain sync.

• IPLD resolver: A libp2p-based service that is used to resolve CIDs from IPFS as well as the network of validator peers. IPLD powers the network's ability to store and retrieve data in a content-addressable way.

Checkpoints that reference subnet state are pushed to the parent subnet in a bottom-up fashion, which is essential for the parent to validate the state transitions of the child. Basin passes checkpointed headers to its parent and uses the CometBFT ledger to gather relevant signatures and data. Additionally, the Basin can contact the IPLD resolver & store to read and write data to its internal state so that it is IPLD addressable. There is also a top-down sync action; subnets must have a view of their parent's finality, which includes the latest block hash, power table information, and (in the future) cross-subnet message passing.

In general, data is represented as CIDs onchain (within a Basin machine's state), and the actual data is stored offchain in a node's local (networked) block store. Basin uses the concept of a detached payload asynchronous sync mechanism, which is a transaction that includes a CID reference to an object but does not include the object data itself. When a detached payload is added to the chain, validators are required to download the object data from the network and verify that it matches the CID reference. This ensures that all validators have a copy of the object data and can verify the integrity of the object store state.

The core IPC process for top-down parent finality is a vote-based mechanism. Once a validator has the data locally (via synchronization with its peers), the validator issues a vote to the other validators to confirm data availability (this is similar to Ethereum's concept of a data availability committee). During this time, normal block production continues. The leader of a given consensus round checks the vote tally for quorum based on the power table (stake) within the subnet. If a quorum is reached, the leader injects a synthetic transaction into the proposal, which is validated by the other validators based on their view of the vote tally, and then finally during the execution of this transaction, the object is marked as resolved.

Basin's core functionality is enabled with a series of data machines, which are synonymous with smart contracts or "actors" that run on the FVM and are used to manage, query, and update the state of the subnet. The FVM is responsible for executing the logic of the Basin protocol, including processing transactions, updating account balances, and managing the state of the network. There are three primary machines in the Basin:

• Machine manager

• Object store machines

• Accumulator machines

The FVM executes messages in WASM over machine state and uses the Wasmtime runtime, and this includes a WASM implementation of the EVM bytecode interpreter. Under the hood, any "built-in" and "custom" (unique / subnet-specific) smart contracts are compiled to CAR files and provided to the subnet during genesis.

Users can deploy new machines on-demand using the machine manager. This interface is responsible for creating new object stores or accumulators and also managing the state of the network. Each machine is associated with a unique address, which is used to identify the machine on the network.

These are key-value stores that allow you to push and retrieve data in a familiar S3-like fashion. Object stores support byte-range requests and advanced queries based on a key's prefix, delimiter, offset, and limit. The object store machine provides a set of methods for interacting with the store, including put, get, and delete, which allows users to store and retrieve data from the store.

Internally, the state of an object store is represented as an IPLD-based HAMT (hash array mapped trie). The IPLD data model provides a flexible and extensible way to represent complex data structures.

An accumulator is a Merkle Mountain Range (MMR)-based verifiable anchoring system for state updates. You can push values up to 500KiB and retrieve them by index. Accumulators support querying for state root, MMR peaks, and total leaf count. As you push new data to the accumulator, you can retrieve the underlying data at a leaf or other relevant data structure components like peaks and total count. Similar to the object store machine, the accumulator stores a CID summary in its on-chain state.

Accounts (ECDSA, secp256k1) are used to send data-carrying transactions as you would on any blockchain system. Since the Basin is built on top of Filecoin's FVM, the addresses follow a slightly different convention than a purely EVM-based account system. Here's a quick primer:

• Addresses are prefixed with a network identifier: t for Filecoin testnet, or f for Filecoin mainnet, and there are five different address types denoted by the second character in the address string: t0/f0 to t4/f4.

• If you're coming from the EVM world, you'll mostly see two types in the Basin:

  • t2/f2: Any FVM-native contract that gets deployed, such as the object store and accumulator machines.

  • t4/f4: A namespaced contract address, and t410/f410 is a specialized namespace for Ethereum-compatible addresses (wallets and smart contracts) on the FVM.

• Namely, each t410.../f410... address is equivalent to a 0x address; the hex address is encoded in the FVM address string.

Once your EVM-style account is registered on a subnet, the 0x and its corresponding t410.../f410... addresses can be used interchangeably on Filecoin and all Basin subnets.

Currently, there are two types of write-access controls: only-owner or public access. For example, you can create an object store that you, and only you can write to—gated by signatures from your private key. Or, you can have "allow all" access where anyone can write data.

This is being further refined, and there will be more robust access control mechanisms soon.

For context, there are three ways in which transactions can be sent/broadcasted to the network: commit, sync, and async. Here's a quick overview of each:

• commit: Wait until the transaction is delivered and final (default behavior).

• sync: Wait only for the result of a local transaction pre-check, but don’t wait for it to be delivered to all validators (i.e., added risk the transaction may fail during delivery).

• async: Does not wait at all. You will not see errors in your terminal (i.e., added risk the transaction may fail during delivery).

There are a few limitations and behaviors of which to be aware when using the Basin network:

• The maximum size for a single object is 1 GiB.

• Objects less than or equal to 1024 bytes are "on-chain" objects stored fully onchain.

• Objects greater than 1024 bytes are "detached" objects that get stored offchain and resolved as a CID.

• The current throughput of the Basin is hundreds of transactions per second (TPS) (note: this is a rough estimate and may vary based on network conditions. The node design is still under heavy development, and optimizations are being made).

Basin comes with both a CLI and Rust SDK that exposes the core network interfaces. You can find detailed instructions for each of these in their respective subdirectories:

• CLI: here

• SDK: here

Since the Basin is built on top of Filecoin, you must have FIL in your account to interact with the network. Basin is currently only live on the Filecoin Calibration network, so you can get tFIL via the faucet here. For reference, Filecoin chain information can be found here.

For example, you can use the CLI with your Ethereum-style account (and note the adm usage below is simply an internal codename for Basin):

> export PRIVATE_KEY=abcd1234... # your wallet private key
> adm account deposit 0.1
> adm objectstore create # create an object store
> export CONTRACT_ADDRESS=t2abc123... # object store contract address via above
> adm objectstore add \
--address $CONTRACT_ADDRESS \
--key "my/object" \
./hello.json
# {
#   "status": "committed",
#   "hash": "48BD1767DC5739C1EABB25FBC8E6718E3E8DE95FB7EE74D13895F2E9D9F5E00A",
#   "height": "358569",
#   "gas_used": 4784697,
#   "data": "bafy2bzacebnjpu5e3ushfu2weqvmtvk7vnndg4fkqsbr4zub52cyekcix7l4o"
# }

> adm objectstore get \
--address $CONTRACT_ADDRESS \
"my/object" \
# {"hello":"world"}

Begin transmission…

by Bruno Calza

A new version of Tableland was released with support for placeholders and query params for /query endpoint (both GET and POST).

Here are some examples:

• GET /query

https://tableland.network/api/v1/query?statement=select * from _137_266 where ksuid = ? and year = ?&params='26G4V9VLoG9wVTeBE1M4UhJzOPi'&params=2011
# url-encoded version
https://tableland.network/api/v1/query?statement=select%20*%20from%20_137_266%20where%20ksuid%20=%20?%20and%20year%20=%20?&params=%2726G4V9VLoG9wVTeBE1M4UhJzOPi%27&params=2011

• POST /query:

curl -s -X POST "https://tableland.network/api/v1/query" \
--data '{
  "statement": "select * from _137_266 where ksuid = ? and year = ?",
  "params": ["26G4V9VLoG9wVTeBE1M4UhJzOPi",2011]
}'

With that change, it becomes a lot easier to programmatically build your query requests and send it to the Tableland Gateway. You don't need to worry about query building!

by Dan Buchholz

Tableland is adding Base Sepolia chain support! We've already updated the validator and the @tableland/evm repo, so you can build on Base today with smart contacts. The downstream clients (JS SDK and Studio) will be incorporating Base support later this week.

Note this is only for the Base testnet, so mainnet is not something available, yet. Mainnet will be added based on developer demand…so let us know if it's something you're looking for on Tableland!

by Avichal Pandey

How do you decide whether to use Option<&T> or &Option<T> as function arguments or returns types in Rust? For instance, should you choose the first or second version in the code shown below for public functions?

pub fn data_a(&self) -> &Option<Data> {
	&self.0
}

pub fn data_b(&self) -> Option<&Data> {
	self.0.as_ref()
}

The choice of one over the others will impact the code base's overall performance, flexibility, and maintainability.

Here is a great explanation https://youtu.be/6c7pZYP_iIE

by Dan Buchholz

We're working on getting the Basin SDK to compile to wasm, which makes it easier to build downstream JavaScript or Python clients that are constantly in sync with the base Rust crate. The most common approach is wasm-bindgen plus wasm-pack. With wasm-bindgen, you add macros to the parts of your library that you want to be able to compile to wasm:

use wasm_bindgen::prelude::*; 

#[wasm_bindgen]
pub struct Data {
	inner: u8,
}

And then you build it by specifying your crate and the target architecture:

wasm-pack build --target web my_crate

You also might have directives that use different features for architecture-specific dependencies:

[target.'cfg(target_arch = "wasm32")'.dependencies]
my_dependency = "0.0.1"

The Rust cookbook has a ton of information on how to set this all up, too: here. For example, if a library is doing any I/O, you'll have to set up additional wasm support with js-sys or web-sys, which let you implement features that NodeJS or a web browser supports for file access. If there are any async operations, the wasm-bindgen-futures crate is also required for converting JS Promises to Rust Futures.

by Marla Natoli

Lilypad is creating a distributed compute network that focuses on enabling AI use cases such as AI and ML. They enable the contribution of idle compute power from your own hardware via their established and verifiable processes powered by smart contracts. Their focus is on high-throughput data processing to power models that require high-performance computing. Lilypad’s decentralized compute network can significantly reduce the cost and time associated with generating large datasets.

With the proliferation of AI services and infrastructure, there are a few key recurring themes we’re hearing about from teams working in AI - the need for compute power, the importance of internet-scale processing, and the value of verifiable data pipelines. At Basin, we’re looking to support a decentralized, high-throughput compute workflow with our decentralized object storage solution running on horizontally scaleable subnets (powered by IPC). This makes key data available quickly, enabling compute over that data from various stakeholders and facilitating collaboration over datasets, which is becoming more and more important to facilitate AI training.

If you’re working in AI and interested in decentralizing database infrastructure to bring collaboration and verifiability to your data, we’d love to hear from you. Read more about Basin here and apply for our private beta here.

by Jim Kosem

In What can LLMs never do?, Rohit Krishnan exposes some fascinating gaps in the AI takeover of everything.

There is no shortage of polar-swinging discussions around LLMs and “AI” in general. It is either going to save humanity and single-handedly make a better cocktail than any human ever could, say the cheerleaders, who often have a monetary stake in this “force” (because what else is it really at this point, nebulous...). On the other side of the debate globe are the fearful and huddled masses seeing nothing but doom and gloom.

This is a large camp that is quite easy to join because humans are easy to frighten, and when there is lots of money to be made, it’s easy to get screwed. There is hope for the masses that there are things the LLMs just can’t figure out which most three to four-year-old humans can, such as statements of equality (this is that and that is this) and interestingly, things involving words and grids. Who would have knew. The LLMs sure didn’t.

• ETHcc is about a month away! We’ll have a couple of folks in attendance, so keep an eye out for further announcements and when/where we’ll be. And if there are any events you’d like to collaborate on, let us know on Discord!

Want to dive deeper, ask questions, or just nerd out with us? Jump into our Telegram or Discord—including weekly research office hours or developer office hours. And if you’d like to discuss any of these topics in more detail, comment on the issue over in GitHub!

Are you enjoying Weeknotes? We’d love your feedback—if you fill out a quick survey, we’ll be sure to reach out directly with community initiatives in the future!: Fill out the form here

Begin transmission…

by Dan Buchholz

ICYMI—we officially announced the newest iteration of Basin. Our initial MVP was centered around pushing arbitrary data to "vaults" with writer access controls, a hot layer for immediate access, and archival to Filecoin. The new Basin takes learnings from this and simplifies it to a Filecoin data L2 with object storage. There's a lot more to it, and we've written a detailed blog about it and have a new landing page with more information.

If onchain object storage (think: S3, but decentralized) is something you're interested in, fill out the form on the site or hop into our Discord!

Check out the blog here and our new website: https://basin.textile.io

by Dan Buchholz

We wrapped up a proof of concept with IoTeX, WeatherXM, Risc0, & Filecoin to show how a decentralized system can enable transparent contributor token reward calculations. The main premise is that it's a permissionless workflow where no central entity is managing the rewards distribution, rather, it could be automated with networks performing each step.

As WeatherXM pushes data to Basin, IoTeX's W3bstream runs Quality of Data (QoD) calculations over it and leverages Risc0 zk proofs for data validity. Lastly, as new QoD scores are calculated they are also written to Basin and later archived to Filecoin for long-term data availability and dispute resolution.

You can read the detailed blog post: here

by Dan Buchholz

We've been supporting the Backdrop program for their v3 and v4 cohorts, and we'll also support the next few cohorts, too. Backdrop is a mini-accelerator / hackathon that supports hackers in a 4-week buildout and helps connect them with potential grants and funding across web3, AI, OSS, and more.

The v5 cohort kicks off in July. Now, that the Basin alpha is live, we'll support Backdrop through additional bounties that focus on Basin usage in addition to Tableland. There will probably be a couple of new Basin features by then that'll make it even easier to switch over to Basin from "traditional" object storage providers, so keep an eye out for that.

If you're interested in applying, head over to the Backdrop website: https://backdropbuild.com

• In Looking for AI use-cases, Ben Evans questions the wide base use cases for LLMs. He sweeps wide, wondering what is there for everyone, when will this supposed generalized application happen, and how.

• For the encryption folks, there are a few interesting resources on different approaches in a web3 native setting. We might also write a deep dive blog post on these and related themes in the coming weeks.

  • Threshold proxy re-encryption is a novel encryption mechanism that distributes re-encryption keys across a network, and once a threshold is met, it can give decryption access to a criteria-fulfilling party.

  • Fully homomorphic encryption lets an entity compute over encrypted data without ever decrypting it. It lets you make arbitrary computations and extends the feature set of previous iterations—but a simple example would be to “add 1 to this encrypted data,” and after performing the calculation, only the original encryptor can see the result.

Want to dive deeper, ask questions, or just nerd out with us? Jump into our Telegram or Discord—including weekly research office hours or developer office hours. And if you’d like to discuss any of these topics in more detail, comment on the issue over in GitHub!

Are you enjoying Weeknotes? We’d love your feedback—if you fill out a quick survey, we’ll be sure to reach out directly with community initiatives in the future!: Fill out the form here

In the Decentralized Physical Infrastructure Networks (”DePIN”) space, early contributors often invest in hardware and take the time to capture and contribute data to an ecosystem in exchange for token rewards. However, most DePINs are still calculating and distributing token incentive payouts in a centralized, opaque manner, making it difficult for contributors or investors to know if they’re receiving the correct amount of tokens for their contribution. With this PoC, we’ve demonstrated what a more transparent, decentralized, and trustless future could look like for DePINs and their token incentive payouts to their contributors.

WeatherXM has been designing its protocol and ecosystem to lead from a place of transparency, sharing the many intricacies that makeup quality of data (QoD) scores for reward payouts and continually experimenting with decentralized back-end infrastructure to open up access to and collaboration over their data. Their input and feedback have been invaluable to us as we design Basin, a Data Availability (”DA”) and Long-Term Storage (”LTS”) layer, to unlock new data use cases for DePINs. Namely, data is immediately available with the Basin hot cache layer while being replicated to Filecoin for long-term persistence. We’ve also been working closely with the IoTeX team on their new compute protocol W3bstream, which, in turn, supports Risc0’s newly released zkVM. Ultimately, a DA/LTS layer should be modular and part of a DePIN’s middleware stack, so we all collaborated on a proof-of-concept that leveraged all three protocols.

If you want to dive straight into the POC, check out the source code here: https://github.com/machinefi/WeatherXM-IoTeX-Textile-POC

To provide a bit more context, here’s a quick overview of what each protocol does:

• W3bstream: compute protocol built for DePINs that allows for off-chain, verifiable computing on data (with zero knowledge proofs) and can trigger smart contracts on any blockchain, facilitating the creation of circular token economies.

• Risc0: a zero-knowledge verifiable general computing platform based on zk-STARKs and the RISC-V microarchitecture.

• WeatherXM: DePIN network that's powered by community-owned weather stations (devices), purpose-built by WeatherXM for weather data collection and incentivized to share data.

• Basin: a DA/LTS storage layer that provides hot/cold decentralized storage for large datasets with hot cache + Filecoin…sorta like a “data L2.”

WeatherXM pushes station/device data to Basin on a daily basis. Separately, the network runs a Quality of Data (”QoD”) algorithm that gets incorporated into the token rewarding mechanism. The IoTeX team built on top of the WeatherXM dataset stored on Basin, while also demonstrating how the QoD pipeline could work within a decentralized compute network. Namely, W3sbtream could run the QoD algorithm daily over the WeatherXM data and then also store those results in Basin, along with zk proofs.

You can think of this process in three steps:

• Ingest: WeatherXM pushes data to a Textile Basin on a daily basis as a Parquet file.

• Compute: The daily QoD score per station is calculated for the entire network, with a zk proof by W3bstream.

• Visualizations: Access to the data & reward proofs is provided through a simple dashboard (here).

In the POC, the first step in the process is already accounted for without additional effort because WeatherXM is already sending station data to Basin. The second step requires W3bstream to pull data from Basin and run QoD calculations. Luckily, the QoD algorithm is open source and comes ready to go with a Docker setup: here. You can generate the QoD scores with that repo using the following commands, which include the station/device ID and time ranges to generate the QoD scores:

> docker build -t wxm-qod:local .
> docker run \
  -v /datasets:/datasets \
  wxm-qod:local obc_sqc.iface.file_model_inference \
  --device_id <device_id> \ # unique WeatherXM device ID
  --date <date> \ # the QoD score for that day
  --day1 <yesterday> \ # the data at the start time
  --day2 <today> \ # the data at the end time
  --output_file_path results.json

For example, W3bstream nodes could run this QoD algorithm daily to generate the dataset on-machine. The next step is the verifiability component; you must prepare the prover code to compute the average QoD scores with the W3bstream zk engine. A reference example is located in methods.rs (here) that defines how to generate a risc0 proof. The actual implementation looks like this, which risc0-zkvm and those custom methods that define a ZK_ELF, ZK_ID, and ZK_PATH:

use zk_methods::ZK_ELF;
use risc0_zkvm::{
    default_prover,
    serde::{from_slice, to_vec},
    ExecutorEnv, Receipt,
};

// Compare them inside the ZKP
pub fn avg(a: String) -> (Receipt, String) {
    let env = ExecutorEnv::builder()
        .write_slice(&to_vec(&a).unwrap())
        .build()
        .unwrap();

    // Obtain the default prover.
    let prover = default_prover();

    // Produce a receipt by proving the specified ELF binary.
    let receipt = prover.prove(env, ZK_ELF).unwrap();
    println!("{:?}", receipt);

    // Extract journal of receipt (i.e. output c, where c = a + b)
    let c: String = from_slice(&receipt.journal.bytes).expect(
        "Journal output should deserialize into the same types (& order)",
    );
    println!("{}", c);

    (receipt, c)
}

Then, generating the actual proof uses the avg function above (for averaging QoD scores):

use zk::avg;
use zk_methods::ZK_ID;

fn main() {
	// An example of a single WeatherXm device ID, its QoD, 
    // and the proof type as a stored variable / string
    let json_str = "[
        \"{\\\"data\\\":\\\"[{\\\\\\\"device_id\\\\\\\":\\\\\\\"e1f8a900-1dfb-11ed-9972-4f669f2d96bd\\\\\\\",\\\\\\\"qod_score\\\\\\\":0.9644833333333332},\\\"receipt_type\\\":\\\"Stark\\\"}\"
        ]";

    let (receipt, _) = avg(json_str.to_owned());
    // Verify receipt (and error if there's an issue)
    receipt.verify(ZK_ID).expect(
        "code you have proven should successfully verify",
    );
}

Once the prover code (methods.rs) is defined, you can then use the W3bstream network to run proofs over the data by using the W3bstream Sandbox. Start by providing the prover code and where the QoD/proof data is to be written (i.e., to Basin) before submitting the request to the network:

Alternatively, this can be performed programmatically with the IoTeX CLI (ioctl). It’ll run through similar steps and log the results:

{
  "messageID": "c4a2be1c-def3-463f-813a-12efc7e46856",
  "states": [
    {
      "state": "received",
      "time": "2024-03-22T19:05:36.661Z",
      "comment": ""
    },
    {
      "state": "packed",
      "time": "2024-03-22T19:05:40.2937Z",
      "comment": ""
    },
    {
      "state": "dispatched",
      "time": "2024-03-22T19:19:08.268849Z",
      "comment": ""
    },
    {
      "state": "proved",
      "time": "2024-03-22T19:24:53.339793Z",
      "comment": ""
    },
    {
      "state": "outputted",
      "time": "2024-03-22T19:24:57.587218Z",
      "comment": ""
    }
  ]
}

All of the data is made available on Basin, so you can inspect the CID to see what the proof looks like—here’s an example: bafybeigcwwx67s27str46u45grpk7xn3brqe2fqezgogaiycscjs53xhqm

Reach out to Tableland, IoTeX, or WeatherXM if you found this PoC useful! There are so many ways you can build on what this demonstration showed, such as using:

• WeatherXM’s weather station data in a compute or visualization workflow. Explore data on index.weatherxm.network or use their client API.

• W3bstream to run verifiable computation over arbitrary data (e.g., DePIN device data) with zk proofs (including to automate your token incentive streams and build contributor trust in your reward calculations).

• Basin for data availability and storage (e.g, DePIN device data or compute results/proofs).

• Risc0 to significantly speed up your development, and build a Rust program, whose execution can be proven by zero-knowledge technology.

This post was co-authored by: Textile (Dan Buchholz, @dtbuchholz) & IoTeX (Giuseppe De Luca, @iotex_dev)

Basin is designed to overcome the limitations of traditional cloud storage, providing a scalable, secure, and verifiable data management solution. Leveraging Interplanetary Consensus, Basin can help protocols manage their data at internet speed. It offers an S3-compatible object store backed by a scalable consensus protocol. Basin ensures data availability and scalability through its subnet architecture, offering high-throughput storage atop Filecoin’s exabytes of storage capacity.

We chose Filecoin as the backbone for Basin because of its unmatched decentralized storage capacity. This massive capacity allows Basin to offer persistent data availability and verifiable integrity across a decentralized network. Moreover, Basin benefits Filecoin by providing a high-throughput data lifecycle management tool that streamlines the onboarding and utilization of large-scale data, enhancing the overall network.

Much like a basin that collects and channels water, our platform aggregates, secures, and optimizes data flow. Basin was created to meet the critical needs of transparency, security, and efficiency, supporting applications such as AI/ML, decentralized compute, and DePINs. With Basin, your data remains tamper-proof and authentic, fostering trust and transparency.

• Data Availability at Internet Speed: Our high-throughput, S3-compatible architecture ensures your data is always accessible with cryptographic proofs.

• Verifiable Data for Trust: Basin provides cryptographic security guarantees, ensuring your data remains tamper-proof and authentic.

Basin is designed for protocols and autonomous agents that require reliable storage capacity on demand for varying time frames. Built on crypto primitives, it can be programmed and paid for seamlessly without lock-in, long-term plans, or centralized accounts. As part of our private testnet, we’re working with a select set of partners and customers including WeatherXM and IoTex to build POCs that showcase how Basin unlocks real-world value.

WeatherXM is currently using Basin to move toward fully decentralized infrastructure, prioritizing transparency and open data availability for the data coming from their network of weather stations. Basin facilitates data collaboration by combining high throughput, temporary data availability with long-term archiving to Filecoin, allowing contributors to build on top of device data, unlocking value and monetization opportunities for their network. Basin could also be used to enable verifiability, signing data at the source, and ensuring all data is cryptographically provable. This decentralized framework ensures transparency in rewards calculations, offering confidence to network participants.

To add automation to transparent rewards flows, we’ve been working closely with IoTex via their W3bstream solution. W3bstream is a compute protocol built for DePINs that allows for off-chain, verifiable computing on data (with zero knowledge proofs) and smart contract triggers to facilitate circular token economies. Basin acts as a high throughput data availability option for W3bstream, allowing protocols to send data to Basin’s hot cache layer for compute via W3bstream before generating a proof. This unlocks a transparent and programmatic rewards structure that can be used by any protocol. To get updates on our upcoming POC, subscribe to our blog here.

While our full suite of features will roll out throughout 2024, you can sign-up to start using Basin today. Here’s what it looks like to store data on the network :

# Create an object store (i.e., a smart contract-based data container)
> basin objectstore create

{
  "address": "t2weumc7otsi3kniwjgy2xnemws5jpi3vmbnxg4fa",
  "tx": {
    "gas_used": 15004808,
    "hash": "3999595D0F74F912323F0F545204BE9D0605CE741275120E553FA395E64DA48D",
    "height": "358550"
  }
}

# Add an object to the store (sends an onchain tx)
> echo '{"hello":"world"}' | basin objectstore add \
--address t2weumc7otsi3kniwjgy2xnemws5jpi3vmbnxg4fa \
--key "my/object"

{
  "status": "committed",
  "hash": "48BD1767DC5739C1EABB25FBC8E6718E3E8DE95FB7EE74D13895F2E9D9F5E00A",
  "height": "358569",
  "gas_used": 4784697,
  "data": "bafy2bzacebnjpu5e3ushfu2weqvmtvk7vnndg4fkqsbr4zub52cyekcix7l4o"
}

# Get an object from the network (resolved after tx is committed onchain)
> basin objectstore get \
--address t2weumc7otsi3kniwjgy2xnemws5jpi3vmbnxg4fa \
"my/object"

{"hello":"world"}

Here is a brief overview of the technical arcitecture:

We invite you to partner with us in building the future of data. If you’re interested in using Basin, fill out this form to get access to our private testnet or to talk to a member of our team about collaboration opportunities.

Stay updated as Basin powers the next wave of decentralized innovation. Join us, grow with us, and reshape the data landscape.

Begin transmission…

by Dan Buchholz

ICYMI—DIMO officially launched on Polygon mainnet with Tableland!

Read more about the DIMO launch here, and you can see hundreds of tables they've created on behalf of their manufacturers here. It's truly open & collaborative data!

From this launch, we have a WIP feature that will be released in the future that lets you upload existing tables to the Studio in bulk quantity. One of the reasons we’re adding it is because DIMO has tons of tables. We’ve started to consider more UX improvements, particularly, related to DIMO’s use case where a project might have significantly more tables than the average use case (i.e., more than a handful).

One component we’re working on is enhancing the CLI to make this type of setup easier for anyone with a table factory-like design. The syntax will look something like this:

# Import from CSV file with headers: 'tableName', 'description', 'definitionName'
> studio import bulk path/to/csv/file 

Once executed, this will log the progress as each table from the CSV gets imported into your Studio project. Much easier than trying to do that one by one! But…if you did need to import one-off tables, the CLI has an existing command to perform this, and the UI has a few places where you can easily execute an import as well.

Since we're just wrapping up some of the initial revamps and kicking off that new work, be sure to let us know if there’s anything you’d like to add or change with how things work today.

by Brian Hoffstein

Last Thursday, we hosted a discussion on "The New Data Paradigm." The session offered a comprehensive exploration of the evolving landscape of decentralized data technologies, featuring insights from pioneers in the field. Panelists shared their perspectives on interoperability, security innovations, and the transformative potential of decentralized storage solutions.

Jonathan Victor, Co-founder of Ansa Moderating the discussion, Jonathan Victor articulated the core values driving the shift towards decentralized technologies: "When I think about why this space is important, it's less so about purely just saying decentralization for the sake of it. It's really thinking about how we build applications that are more resilient than any individual cloud provider, any individual protocol, and more hopefully composable and interoperable.” His perspective highlighted the goals of this paradigm: to create robust, adaptable systems capable of withstanding the evolving technological landscapes.

Joshua Noble, Co-founder of Filebase Joshua Noble added on with the integration capacities for decentralized storage to enhance artificial intelligence, stressing the transformative implications for data management within this emerging arena of our collective landscape: "The future looks pretty bright on our side, and I think really for the future of IPFS as a whole, we've been having a lot of discussions with a number of AI communities and we're starting to see a lot of this data come online." He expanded on this dynamic intersection of AI and decentralized networks, alluding to the important path ahead for innovative solutions in data handling and accessibility.

David Whittington, Infrastructure Architect at PDS David Whittington shared his perspective on the potential for decentralized data to create more dynamic and interconnected data ecosystems: "Building up datasets by composing different data over time is very interesting because... over time, datasets can just refer to each other and you can have composite data sets." David continued on this thread, illustrating a future where data is not only more accessible but also more collaboratively utilized across various platforms and users.

Kanishk Khurana, Dev Rel Lead at Fleek Kanishk Khurana discussed challenges and solutions encountered by Fleek in managing web3 infrastructure, particularly in enhancing security: "We have about 100,000 websites already deployed and we've had our fair share of phishing and malicious users. We've sorted that out to be a very big problem. So we've implemented new security hashing compute solutions that we were able to build out to mitigate a lot of those problems." Kanishk’s example highlighted Fleek's commitment to providing robust, secure environments for developers in the evolving digital landscape.

Nandit Mehra, Founder of Lighthouse

Nandit Mehra described the upcoming transformations within Lighthouse aimed at enhancing decentralization: "From our side at Lighthouse, we are focusing on making the protocol permissionless in the coming quarters. This will involve transitioning all endowment pool contracts to the Filecoin Virtual Machine, which will be controlled by a DAO. Additionally, the roles of data bundling, aggregation, and access control will transition from our internal team to external community and ecosystem teams." His vision for Lighthouse highlights a shift towards a governance model that is open and driven by the community, fundamentally changing how data management and storage are handled within the platform.

The panelists not only shared updates from their projects but also discussed the broader implications and aspirations for the impact of these technologies on the digital world. They delved into practical aspects of decentralized data systems, emphasizing the enhancement of data sovereignty, and improving accessibility and integration across various platforms.

Their insights underscore the critical need for scalable, secure, and interoperable data solutions that can support the next generation of digital applications. From boosting AI capabilities to fostering more resilient and inclusive data infrastructures, the potential for decentralized technologies to drive significant innovation is vast.

We extend our gratitude to Jonathan Victor for his expert moderation and to all the panelists for their enlightening contributions. You can listen to the whole conversation here.

For those eager to engage further in these topics, stay tuned for upcoming sessions and join our ongoing discussions on Twitter and Discord.

by Marla Natoli

Nodle is a DePIN focussed on authenticating media content and location via their decentralized physical infrastructure network that requires only a smartphone to participate. The idea is to take advantage of the mass amounts of smartphones in the world and put them to work in new ways, turning them into Edge Nodes and rewarding users for performing work. These edge notes are able to read devices and sensors using Bluetooth, and Nodle connects this information to the blockchain. Read more about how Nodle works here.

Nodle is particularly interesting in their ability to connect the concept of DePIN to real-world use cases, having launched a number of products including their Click Camera App which has a goal to fight misinformation and make the creation of media verifiable. Seeing these use cases come to fruition is exciting, as this is exactly the type of use case we aim to help solve using Basin, our decentralized object storage solution. Nodle has recently taken steps to decentralize its data infrastructure by integrating Filecoin for decentralized storage, indicating a need for a decentralized infrastructure stack to power the next generation of verifiable information. Read more about how they’re using Filecoin here and get in touch if you’re looking for solutions related to verifiable and decentralized data.

Below are some recent topics and articles that we found interesting and wanted to share with our readers!

• For the encryption fans—drand has a cool approach to timelocking data: https://drand.love/docs/timelock-encryption

• Catch up on Jack Dorsey’s interview with Mike Solana, which is all about open protocols—including tidbits about adding value to for-profit entities: https://www.piratewires.com/p/interview-with-jack-dorsey-mike-solana

• For the Reth fans out there, there’s a new feature for Reth execution extensions (”ExEx”), which is a framework for building offchain infrastructure as post-execution hooks: https://www.paradigm.xyz/2024/05/reth-exex

• And if you found that interesting, check out the podcast for a Reth overview and how Paradigm thinks about its prospective future: https://www.youtube.com/watch?v=jEUD-G8cF78

Want to dive deeper, ask questions, or just nerd out with us? Jump into our Telegram or Discord—including weekly research office hours or developer office hours. And if you’d like to discuss any of these topics in more detail, comment on the issue over in GitHub!

Are you enjoying Weeknotes? We’d love your feedback—if you fill out a quick survey, we’ll be sure to reach out directly with community initiatives in the future!: Fill out the form here

Gating data within the silos of car manufacturers stifles innovation and limits which players can contribute to the future of connected vehicles. This could lead to further privacy challenges if the same large tech companies that own most of our data, are the ones dictating how our vehicle data is being used and monetized. DIMO is combatting this with their open network, putting users in control of how data is used, and enabling developers to create value and get rewarded accordingly. A critical component to this success is ensuring they are using and creating reusable infrastructure that supports collaboration—this is where Tableland comes in.

Tableland provides a database solution that meets all the criteria DIMO was looking for: enable collaboration and composability over the data, allow for some level of ACL (to minimize spam entries), and ensure others can replicate the model (to encourage further creation of open information). Further, with the recent launch of Tableland Studio, it’s easier than ever for developers to find DIMO’s tables, build on top of the data, or recreate a similar structure for a different set of data.

Check out DIMO’s data in the Studio: https://studio.tableland.xyz/partners/dimo

This public good dataset is now deployed on mainnet and available for anyone to use. It is an important step toward making it easier for new projects and developers to build within the DIMO ecosystem. Read on to dive deeper into this collaboration and how it works.

DIMO and Tableland collaborated to create a decentralized, public dataset containing DIMO vehicle definitions. One of the primary benefits of open data is that it enables anyone to build on top of and use that information. The onchain-related components consist of:

• A primary registry contract is the main entry point for adding or removing related module contracts.

• A series of “node” contracts that include:

  • Manufacturer: Stores information about a vehicle’s or aftermarket device’s manufacturer.

  • Vehicle: Mints an NFT representation of the owner’s vehicle, including the manufacturer, owner, make, model, and year.

  • AftermarketDevice: Mints an NFT representation of an aftermarket IoT device, including the EVM associated address for the device, serial number, and hardware/IMEI information.

  • SyntheticDevice: Mints an NFT representation of a “synthetic device,” which is then mapped to a vehicle ID.

  • Integration: Mints an NFT representation that uniquely identifies an integration node.

• A DIMO user pairs their EVM wallet with devices and vehicles, and then devices and vehicles are also mapped to one another. Specifically, the AftermarketDeviceId and VehicleId are stored in the “device definitions” table in Tableland.

  

  Check out the implementation by the DIMO team here: https://github.com/DIMO-Network/dimo-identity
  
  And the initial reference demo code by the Tableland team is below, which also has a unique setup with a dynamic merkle tree that represents the offchain table state fully onchain: https://github.com/tablelandnetwork/dimo-vehicle-defs

Tableland and DIMO set out to manage the device registry identities to help maintain and update the global list of valid vehicles fully transparently. While much of this data is public, DIMO spent considerable time and resources to source and format the data in a usable way and wants to simplify this for future use cases. For example, the onchain-driven and open dataset lets anyone query the vehicle registry and help compose data as part of the vehicle NFT metadata:

This also allows DIMO to seamlessly create new entries in the vehicle registry through onchain actions—without any additional database infrastructure setup—and it lets their end users see exactly what data about their vehicle is made available on the network. For an introduction to how we’re helping DePINs like DIMO, read our initial partnership blog post: here.

Let’s review the DIMO implementation noted above. There are two main contracts:

• DeviceDefinitionTable: Manages table creation and mutations for vehicle definitions so that DIMO can add new manufacturer vehicle definitions to the network

  • Note: This manages vehicle information and should not be confused with the Device nor Vehicle contracts mentioned above.

• DeviceDefinitionController: Additional access controller that ensures only the core smart contract can forward SQL write statements to the manufacturer/user-owned table.

  • Namely, table writes are gated by NFT ownership, but they still must pass through the smart contract so that valid and "authorized" SQL is written.

Plus, the DeviceDefinitionTableStorage contract provides a storage interface for a mapping that holds manufacturerId and its tableId.

Every manufacturer minted by the Manufacturer contract gets its own table. The following method is where a table is created, and information about the table’s creator/owner, manufacturer, and table ID is emitted in an event—called by an authorized manufacturer:

function createDeviceDefinitionTable(
	address tableOwner,
	uint256 manufacturerId
) { 
	// Assertions & SQL... 

	emit DeviceDefinitionTableCreated(tableOwner, manufacturerId, tableId); 
};

The device definitions table has the following schema. The table is “nameless” but is created with a unique format like _{chainId}_{tableId}.

CREATE TABLE _ (
  id TEXT PRIMARY KEY,
  model TEXT NOT NULL,
  year INTEGER NOT NULL,
  metadata TEXT,
  ksuid TEXT,
  devicetype TEXT,
  imageuri TEXT,
  UNIQUE(model, year)
)

When an authorized party calls createDeviceDefinitionTable, they must make sure a Manufacturer NFT is minted first and then pass the manufacturer ID to the method. A mapping in the contract tracks which manufacturer owns which table. The Solidity implementation of the table described above is the following, and you can see that address(this) is used to mint the table to the actual contract:

string memory statement = string(
	abi.encodePacked(
		"CREATE TABLE _", 
		Strings.toString(block.chainid),
		"(id TEXT PRIMARY KEY, model TEXT NOT NULL, year INTEGER NOT NULL, metadata TEXT, ksuid TEXT, devicetype TEXT, imageuri TEXT, UNIQUE(model,year))" 
	)
); 
uint256 tableId = tablelandTables.create(address(this), statement);

Right after minting the table, the contract transfers ownership to the manufacturer/address that called the method. Before doing so, it sets the table’s controller to a Tableland controller contract (described below). This ensures that while the new owner does have full control of the table, the data they write to is must be written by the DIMO smart contract. In other words, all table writes are gated by the smart contract’s logic and available methods.

tablelandTables.setController(address(this), tableId, address(this));
INFT(address(tablelandTables)).safeTransferFrom(
	address(this),
	tableOwner,
	tableId
);

From an access control perspective, the contract methods handle the assertions before an SQL is executed. The snippet below shows how this works for the table creation process by checking:

• The passed manufacturerId (the Manufacturer contract’s NFT) is valid.

• The method’s caller has an admin role and is the owner of the token at manufacturerId.

• There isn’t an existing table created for the manufacturerId.

if (!manufacturerIdProxy.exists(manufacturerId)) {
    revert InvalidManufacturerId(manufacturerId);
}
if (
	!_hasRole(ADMIN_ROLE, msg.sender) &&
	msg.sender != manufacturerIdProxy.ownerOf(manufacturerId)
) revert Unauthorized(msg.sender);

DeviceDefinitionTableStorage.Storage
	storage dds = DeviceDefinitionTableStorage.getStorage();
TablelandTables tablelandTables = TablelandDeployments.get();

if (dds.tables[manufacturerId] != 0) {
	revert TableAlreadyExists(manufacturerId);
}

And there is also a table “controller” definition that ensures this smart contract is the only one that can write to the table:

function getPolicy(
    address caller,
    uint256
) external payable returns (TablelandPolicy memory policy) {
    if (caller == address(this)) {
        policy = TablelandPolicy({
            allowInsert: true,
            allowUpdate: true,
            allowDelete: true,
            whereClause: "",
            withCheck: "",
            updatableColumns: new string[](0)
        });
    }
}

To reiterate—the manufacturer is the only one that can call the smart contract’s table mutation methods, and the smart contract is, technically, the one forwarding the SQL statements and doing the actual onchain calls to the Tableland registry.

Once a table exists, the manufacturer can insert new definitions into the table. The insertDeviceDefinition follows similar assertions as described above before inserting data, also using the INFTMultiPrivilege contract to manage permissions:

function insertDeviceDefinition(
    uint256 manufacturerId,
    DeviceDefinitionInput calldata data
) {
	// Assertions & SQL...

	emit DeviceDefinitionInserted(tableId, data.id, data.model, data.year);
}

The Solidity implementation is the following, where data includes:

• id: The alphanumeric ID of the Device Definition.

• model: The model of the Device Definition.

• year: The year of the Device Definition.

• metadata: The metadata stringfied object of the Device Definition.

tablelandTables.mutate(
    address(this),
    tableId,
    SQLHelpers.toInsert(
        "",
        tableId,
        "id,model,year,metadata,ksuid,devicetype,imageuri",
        string.concat(
            string(abi.encodePacked("'", data.id, "'")),
            ",",
            string(abi.encodePacked("'", data.model, "'")),
            ",",
            Strings.toString(data.year),
            ",",
            string(abi.encodePacked("'", data.metadata, "'")),
            ",",
            string(abi.encodePacked("'", data.ksuid, "'")),
            ",",
            string(abi.encodePacked("'", data.devicetype, "'")),
            ",",
            string(abi.encodePacked("'", data.imageuri, "'"))
        )
    )
);

Which, when seen by a Tableland validator node, gets emitted and materialized as the following SQL statement:

INSERT INTO
  "_{chainId}_{tableId}" (id, model, year, metadata, ksuid, devicetype, imageuri)
VALUES
  (
    '{data.id}',
    '{data.model}',
    {data.year},
    '{data.metadata}'
    '{data.ksuid}',
    '{data.devicetype}',
    '{data.imageuri}'
  )

Note that the {...} values in the example statement above would be actual string-encoded values that were passed as part of the data method parameter.

There are a few other available methods for retrieving a table’s name or ID as well as a batch insert variation.

All of this makes it possible to openly query the vehicle definitions by using Tableland clients like the SDK or REST API. Thus, a developer could build on top of the dataset or query information related to their DIMO vehicle.

The DIMO team made it easy for anyone to test out this setup, too, by providing scripts and instructions for deploying the contracts on Local Tableland. You can start by cloning their repo and checking out the device-defintions branch:

git clone https://github.com/DIMO-Network/dimo-identity
cd dimo-identity
git checkout device-definitions

Then, check out the /scripts/tableland/instructions.md file (here) to see the full guide. After installing npm packages, you can run the following script and tasks against a Local Tableland network:

npx hardhat run scripts/tableland/deployLocalTableland.ts --network localhost
npx hardhat --network localhost mint-manufacturer manufacturerNameExample
npx hardhat --network localhost create-dd-table <your_manufacturer_id>

This will log the manufacturer token and table information:

Minting manufacturer manufacturerNameExample for 0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266...
Manufacturer manufacturerNameExample minted with ID: 142
Creating Device Definition table for manufacturer ID 142...
Device Definition table created
Table ID: 2
Table Name: _31337_2
Table Owner: 0xffffffffffffffffffffffffffffffffffffffff

Lastly, you can call the insertDeviceDefinition or even use something like the Tableland SDK to insert data and expand the dataset. Recall that the access controls limit table mutations such that only the smart contract has the right to mutate data, and only the table’s owner (i.e., the manufacturer) has the right to call the method.

We’ll continue to put out additional walkthroughs of our partner and general use case implementations in long-form posts. But, if you’d like to stay up to date with what we’re building on a week-by-week basis, follow us here on Paragraph.xyz (it's what our blog site uses), or, hop into our Discord for the latest updates: here.

Visit the DIMO Website to learn more about how to get involved as a user or developer in their network, and follow them on X for the latest updates.

Begin transmission…

by Dan Buchholz

Within the Studio, a feature lets you view any table across the Tableland network. But, there's also another level to this: you can take any table and import it directly into your Studio project.

For example, let's say you or someone else created this table with the Tableland SDK or smart contracts:

https://studio.tableland.xyz/table/_137_275

When you're on the table inspection page, there's a … icon in the top-right corner:

Clicking that will prompt you with two options:

• Import the table into your project.

  • For example, you might discover some random table on the network and could then use the CLI to run queries against the data—where the imported table can be contained in / referenced by your personal project.

  • Note: importing a table is an existing feature, but this saves you a couple of steps vs. manually typing in the data via a form input!

• Copy the table schema.

  • That is, it doesn't copy the actual data stored in the table, but the table's definition is created in your project and can act as a nice starting point / help with ideation.

The goal of these two features is to make it even easier to interact with existing tables that weren't deployed on the Studio but are still accessible within it. As always, if you have feature requests, let us know on Discord or open an issue on GitHub.

by Joe Wagner

This week we found an issue affecting our Studio dev team when working with the Brave browser, and the SpruceID SIWE npm package. We were able to fix the issue in studio and contribute an update to the SpruceID quickstart code base.

If you're unable to use Sign In With Ethereum in a localhost sandbox with Brave Wallet, the fix is to start explicitly including the optional scheme field in the SIWE message. An example of the needed change can be found here https://github.com/spruceid/siwe-quickstart/pull/41.

While manually QA testing the Studio web app with Brave one of our dev team noticed the sign-in was no longer working. When attempting to sign in Brave, it shows the following error dialog:

A discerning eye will notice that the URL in the red text is http, and the URL in parentheses is https. Sign-in was working in all the other browsers being tested, so we figured that there was some nuance to Brave that was setting https as a default for an optional value.

Further debugging led to the discovery that the spec shows the first field is the optional scheme. It turns out that if the scheme isn't provided, most browsers are either leaving it out or parsing it out of the uri field. However, Brave's implementation of the spec is to set a default value of https if scheme isn't provided.

The fix ended up being fairly simple. We updated our version of the siwe package to one that supports scheme and started parsing the scheme out of the page uri.

Here's the PR for our update: https://github.com/tablelandnetwork/studio/pull/270

We hope this helps anyone who runs into a similar issue. If that's the case, or you just want to say hi definitely give us a shout on Discord or Twitter.

by Dan Buchholz

The new Tableland SDK has been published officially as @tableland/sdk@7.0.0—along with all other clients that depend on it (Local Tableland, CLI, Hardhat plugin, etc.). We've walked through the changes in our most recent Weeknotes, but the only major changes are:

• Polygon Mumbai has been swapped for Polygon Amoy

• ethers v5 has been swapped for ethers v6

Our docs site is now updated (https://docs.tableland.xyz) with all of these changes. Namely, the SDK usage hasn't changed, but many of the snippets had to be refactored to take into account the points noted above. Also, all of our template repos have been updated as well, so you can get started quickly with the newest SDK!

by Jim Kosem

Accessibility seems to be trending. This is new in a way that is almost disturbing for several reasons, the first of which is why it has taken so long, and the second being what caused this. I think it might have to do with just wanting things to work.

We as as species have lived with the breakneck speed of technological advancement for two to three decades now. At no other point in our collective history has anything resembling this, and it’s all largely happened in how we learn to use and adapt to software patterns. Yet a lot of the basics have been somehow skipped. Whilst there are endless patterns and documentation for spatial design, which happened yesterday for all intents and purposes, there is little on how to navigate with a keyboard with something as notionally simple as a web page form autocomplete.

There is this idea of The Curb Cut Effect: if you make things better for people with access needs, you make it better for all users. For instance, most people use the curb cut whether in a wheelchair or not, because it’s just easier. The same goes for websites and tools. If they are simple in navigation and content structure, easy to read, and things like the tab focus work, it won’t be better just for some people, but for everyone.

Want to dive deeper, ask questions, or just nerd out with us? Jump into our Telegram or Discord—including weekly research office hours or developer office hours. And if you’d like to discuss any of these topics in more detail, comment on the issue over in GitHub!

Are you enjoying Weeknotes? We’d love your feedback—if you fill out a quick survey, we’ll be sure to reach out directly with community initiatives in the future!: Fill out the form here

Begin transmission…

The new Tableland SDK has been unofficially released at v7.0.0-pre.0. This will introduce breaking changes because of two things:

• Polygon Mumbai is being deprecated as a whole, and providers no longer support it. Instead, you should migrate over to Polygon Amoy (chain ID: 80002). Tableland validators have already made the change, so it’s possible to use it today!

• Ethers v6 has been out for quite some time. We’ve finally migrated to ethers v6, so some slight functionality changes in terms of setting up a sgner need to be accounted for if you’re coming from ethers v5. We’ll be updating our docs shortly to demonstrate how to use ethers v6 as well. But, the Database class and all of its methods should be exactly the same…they just work a bit differently under the hood.

The official release of the @tableland/sdk@7.0.0 should be out relatively soon, and the Studio will also incorporate these changes. If you have any questions about Mumbai → Amoy or using ethers v6, hop in our Discord and let us know!

by Brian Hoffstein

Last Thursday, we hosted a Twitter Spaces discussion focused on the blossoming field of Decentralized Physical Infrastructure Networks, or DePINs. Moderated by Xinxin Fang of IoTeX, this session convened ecosystem leaders to delve deeply into the potential of DePINs for real-world applications, examining both inherent challenges and emerging opportunities. The insights shared cast light on innovative pathways for integrating these technologies into our interconnected digital and physical realms.

Jonathan Victor, co-founder of Ansa, provided deep insights into the foundations of building a thriving DePIN ecosystem. He emphasized the critical balance between supply and demand sides for achieving sustainable growth:

"I think for successful projects, one important aspect is how you build momentum. This isn't just on the supply side; it’s also on the demand side, ensuring you're balancing your economy overall, not just focusing on one half."

Rob Solomon, co-founder of DIMO, addressed the challenges and necessities of aligning DePIN projects with market needs, especially in his work within the automotive sector. He highlighted the importance of demonstrating real-world demand for such technologies:

"The big question is, do people actually want these physical networks? Are we going to find product-market fit? We're actively demonstrating that developers want to build on top of the DIMO protocol, showing that the data we generate for vehicles is valuable and leads to the creation of great apps and user experiences."

Manos Nikiforakis, co-founder of WeatherXM, offered a thoughtful perspective on defining the success of DePIN projects. He advocated for a more community-centric approach over conventional metrics:

"Many people define success by looking at the token's price performance. It shouldn't be that way. Success should be measured by the benefits the project delivers to specific groups, whether they're part of our community or the end-users of our products."

The panelists explored various facets of DePIN technology, emphasizing interoperability, user-centric design, and the need for thoughtful regulatory considerations. Each speaker brought unique insights into navigating the complexities of decentralized technologies and integrating them into scalable, user-focused solutions. Themes highlighted included the necessity of balancing innovation with practical market needs, the importance of composability for future expansion, and redefining success in meaningful ways.

This discussion provided a comprehensive overview of the challenges and opportunities within the DePIN sector, offering invaluable insights for anyone interested in the future of decentralized technologies. Our speakers not only outlined the complexities of developing such networks but also discussed their potential to revolutionize industries by enhancing efficiency, transparency, and community engagement.

We extend our gratitude to Xinxin Fang for his expert moderation and to Jonathan, Rob, and Manos for their invaluable contributions to this enlightening discussion. For those seeking to explore the topics discussed further, the full conversation is available here.

Stay tuned for more insightful discussions by following us on Twitter and joining our Discord server, as we continue to explore the cutting edge of the decentralized future.

by Jim Kosem

I might be doing some more speaking at another developer conference or two quite soon. This is a good, but slightly uncomfortable spot to be in when you’re a designer. You’re typically talking a different language to what is hopefully a crowd of interested people. That means you need to assume very little. I’ve done my fair share of speaking to non-designer crowds, everything from teacher trainers in central Africa to the European Union Parliament, but developers are something different.

When you’re talking to people whose job is to build things that have to work, there isn’t a lot of wiggle room or even necessarily interested in learning about things that don’t relate to that. Thinking through what I could talk about, and how we’ve been building and learning and building and learning how Tableland Studio should work, I think we might have something that might not require so much translation and explanation. That is because in this case, developers are the users. This is in itself slightly different from most design content as well, because there isn’t necessarily a regular consumer using an app or service, but a developer using an app or service to build another app or service.

The differences are quite huge, but a lot of themes are the same. How do you understand what people want to do? What are the right questions to ask and to whom? How can we learn what is really going on in many different ways? All of these and more are “design thinking” but with a very specific and higher-order user set. It’s all about thinking about the user as a builder for another user. Slightly abstract, but user designers like me need to have empathy regardless.

Want to dive deeper, ask questions, or just nerd out with us? Jump into our Telegram or Discord—including weekly research office hours or developer office hours. And if you’d like to discuss any of these topics in more detail, comment on the issue over in GitHub!

Are you enjoying Weeknotes? We’d love your feedback—if you fill out a quick survey, we’ll be sure to reach out directly with community initiatives in the future!: Fill out the form here

Begin transmission…

by Carson Farmer

In the realm of probabilistic data structures like skip lists, treaps, merkle search trees, and others, efficiently generating random numbers that follow specific distributions is crucial. One particularly interesting problem is simulating a geometric distribution—a model used to describe the number of trials needed to achieve the first success in a sequence of independent and identically distributed Bernoulli trials. At the same time, in distributed (and/or peer-to-peer) systems, we also end up doing a lot of hashing. And a good hash function should be pretty much indistinguishable from a random string of bits. So in this post, we’ll try to combine these two ideas, and explore an interesting method to generate geometrically distributed random variables using cryptographic hashes, by treated the hashes as a source of random independent coin flips! The outcome will be probabilistic data structures with pseudo random values that are (ideally) identical across peers in the peer-to-peer system — deterministic randomness!

A geometric distribution typically requires a success probability p to determine the distribution of trials until the first success. If you want to simulate a geometric distribution with a probability that isn’t 1/2, say something like p = 1 - (1/m), you can manipulate a sequence of fair Bernoulli trials (each with p = 1/2) to achieve this. The solution involves transforming a 32-byte hash, which is essentially a 256-bit binary string, into the desired geometric distribution. But how?

Each bit in the hash can be thought of as a result from a fair coin flip—either 0 or 1, each with a probability of 1/2. However, to simulate our target distribution, we need groups of bits where the group collectively has a success probability of 1/m. To find the number of bits (k) per group, we calculate the smallest k for which 2^k is greater than or equal to m. Mathematically, this is determined by:

Once k is determined, the hash is divided into groups of k bits. Each group is treated as a single trial with the desired success probability:

1. Extract Groups: We iterate over the 256 bits in batches of k bits.

2. Determine Success or Failure: A group is considered a "success" if all k bits are 0.

   1. This happens with a probability of:

3. Count Trials: The count of groups until the first "success" is the result of our geometric distribution simulation.

Here's a practical function to achieve this using Rust:

#[cfg(test)] mod tests { use rand::Rng; fn simulate_geometric_random(bytes: [u8; 32], m: u32) -> u32 { // Compute ⌈log_2(m)⌉ let k = (m + 1).ilog2(); // Number of batches of k bits let batch_count = 256 / k; // Mask to extract k bits let mask = (1u8 << k) - 1; for i in 0..batch_count { let byte_index = (k * i) / 8; let bit_index = (k * i) % 8; let batch = (bytes[byte_index as usize] >> bit_index) & mask; // batch != 0 means we are looking for failure p = 1 / m // batch == 0 means we are looking for success p = 1 / m if batch == 0 { // +1 because geometric distribution starts at 1 return i + 1; } } batch_count + 1 // if no success found within the limit } #[test] fn test_calculate_rank() { let mut rng = rand::thread_rng(); let m = 4; let p = 1.0 - 1.0 / m as f64; let mut sum = 0; for _ in 0..1000 { let index = simulate_geometric_random(rng.gen(), m); sum += index; } println!("mean: {} expected: {}", sum as f64 / 1000.0, 1.0 / p); } } // running 1 test // mean: 1.333 expected: 1.3333333333333333 // test rank::tests::test_calculate_rank ... ok // test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in 0.01s

This method is efficient and interesting, converting a simple hash value into a sequence of geometrically distributed random trials. It underscores how foundational concepts in probability can be applied to solve practical problems in computer science, particularly in the design of data structures where randomness and probability management are important. Another bonus, treating our hashes (something we have to do anyway) as a random oracle makes this approach computationally elegant and requires minimal additional resources, making it ideal for environments where performance and memory efficiency are critical.

by Dan Buchholz

We're close to launching a new Tableland SDK that includes Polygon Amoy support and also upgrades from ethers v5 to ethers v6. Most of the ethers changes shouldn't really break any of the behavior with the SDK, but it will result in a major dependency bump because it will no longer work with ethers v5.

The upgrade process led to a couple of challenges (and solutions!) with using ethers v6—particularly, when developing against a local Hardhat node:

1. A perpetual failed to detect network error gets logged every second, even though the network exists (see here for more details). For the most part, this is more annoying than actually preventing the SDK from working.

2. A nonce too low or expired error, indicating that a transaction has been sent/accepted with the same nonce as one that's already been used. This is a huge issue because it became impossible to run our tests against a Local Tableland network, which runs a Hardhat node. Tests flakily fail due to a nonce issue.

For problem (1), it turned out to be an easy fix. The failed to detect network error would get logged every time the getSigner() method was called on a provider. Before making that call, all that's needed is to _detectNetwork:

// Instantiate a `provider`—e.g., via `globalThis.ethereum` or an instance of `Eip1193Provider` import { BrowserProvider } from "ethers"; const browserProvider = new BrowserProvider(provider); await browserProvider._detectNetwork(); // this stops the annoying logging const signer = await browserProvider.getSigner();

For problem (2), this took forever to figure out. I tried:

• Always pass an ethers signer wrapped in an ethers NonceManager to the Tableland Database class.

• Make sure all calls in a test are awaited before proceeding to the next test (to ensure that txs weren't being submitted too fast to the Hardhat node).

• In the SDK's Database, Registry, and exec(), take the passed signer and force convert it to a NonceManager.

• Try getting the nonce (via signer.getNonce()) before a call is made and manually pass it in tx calls.

• Setting custom JsonRpcProvider options (note: this was the ~right solution).

• In the before/after mocha test hooks, make calls to the Hardhat node and hardcode/reset the nonce with hardhat_setNonce (here)...but, for obvious reasons, this didn't work.

It was a much simpler fix, as first described here. An ethers JsonRpcProvider can take several options, and setting the batchStallTime and cacheTimeout are required to avoid the nonce issues. Also, since in this case, we're always connecting to the same network (a Hardhat node), the staticNetwork can reduce some calls, too, when coupled with the network param.

import { Network, Wallet } from "ethers"; const account = "your_private_key" const wallet = new Wallet(account); const chainName = "local-tableland"; // i.e., a hardhat node const chainId = 31337; const network = new Network(chainName, chainId); // Optional const signer = wallet.connect( new JsonRpcProvider(`http://127.0.0.1:8545`, network, { staticNetwork: true, // Reduce `eth_chainId` calls batchStallTime: 0, // Don't batch requests, send them immediately cacheTimeout: -1, // Don't cache }) );

by Jim Kosem

We’re in the midst of designing and prototyping a number of sites, showing what we’re working on at Textile, which I’ve been writing about or around in the past couple of weeks or so. At first, we talked about just putting a website up in Framer which should be good enough for now. Then I realized that you don’t quite get responsive behavior out of the box.

The long-standing issue with designing visuals, and well most things with screens, is that you sort of only half know what it will look like for the person that might stumble upon it. Color issues notwithstanding, there is no way to know what sort of device or screen size someone will use. So, just thinking about 65.49% of global website traffic being on mobile devices makes us think this is something we need to get underway. It just needs to work at this point, but fortunately, at this point, responsive design is quite established. Framer itself has some tools to help with this, but the interesting thing from a design standpoint is how to let the design degrade and how to decide what to cut. What is important to see or read when you’re on a phone? Just how much stuff should we be putting on a page at this point then? The issues are innumerable but part of the process of being loose in the right places with how to design for what is a constantly shifting target.

Congrats to all of the team that participated in the Filecoin Data Economy hackathon. Here were the top 3 projects:

1. DBNS by @LionisNick—a unified namespace to host and collaborate datasets and AI models: here

2. DeRAG by @debuggingfuture—a DataDAO-owned LLM AI with trustless, decentralized augmentation: here

3. ChainQuery by @0x_Clint—web3 Q&A platform with token-based rewards and AI augmented answers: here

The DI hack had 7 total winners—3 top projects, and 4 honorable mentions:

• Arbitrarian—simplifying smart contract learning and ERC20 token creation: here

• Memester—merging meme creation with blockchain for rewards: here

• Spooky—storytelling with AI and blockchain, minting narratives as NFTs: here

• DeFi Pets—interactive DeFi learning through AI-powered NFT pets: here

• dChat—secure, decentralized communication via blockchain: here

• TokenMart—revolutionizing e-commerce with tokenized rewards: here

• Crypto Subscriptions—managing crypto subscriptions with ease: here

Want to dive deeper, ask questions, or just nerd out with us? Jump into our Telegram or Discord—including weekly research office hours or developer office hours. And if you’d like to discuss any of these topics in more detail, comment on the issue over in GitHub!

Are you enjoying Weeknotes? We’d love your feedback—if you fill out a quick survey, we’ll be sure to reach out directly with community initiatives in the future!: Fill out the form here

Begin transmission…

by Dan Buchholz

If you haven't been in the loop, Polygon deprecated the Mumbai testnet. Providers like Alchemy have also deprecated Mumbai RPCs—so…it's time to transition to Amoy! We'll be releasing a new @tableland/evm and downstream clients later this week, which will swap the chains from a support perspective.

Support for Amoy isn't quite ready out-of-the-box if you're working with ethers v6. This is due both to fluctuations in gas and the fact the default ethers maxFeePerGas and maxPriorityFeePerGas values are wildly incorrect. For example, if you use the built-in method, you'll get values like this:

import { ethers } from "hardhat"; // using ethers within hardhat context const { maxFeePerGas, maxPriorityFeePerGas } = await ethers.provider.getFeeData(); // maxFeePerGas: 1000000030n (note: `n` signifies `bigint`; it's wei) // maxPriorityFeePerGas: 1000000000n

But, if you look on the Amoy Gas Station (here), these values will be up to 30x larger (e.g., 35185714305 wei). If you pass the default values, your transaction will never get accepted because the fees are far too low. Instead, you must create a custom method—or use the ethers plugin—to fetch the live fees.

You can do this with a Node fetch call and also make it generalized so that it either returns the Amoy fee data or the built-in method's fee data (for other chains):

import { network } from "hardhat"; import type { FeeData } from "ethers"; async function getFeeData(chainName: string): Promise<FeeData> { if (chainName === "polygon-amoy") { try { const url = "<https://gasstation-testnet.polygon.technology/amoy>"; const response = await fetch(url); const data = (await response.json()) as AmoyFeeData; // a custom interface that aligns to the API above const feeData = new FeeData( null, // No gas price value needed BigInt(ethers.parseUnits(String(data.fast.maxFee), "gwei")), BigInt(ethers.parseUnits(String(data.fast.maxPriorityFee), "gwei")) ); return feeData; } catch { const feeData = new FeeData(); return feeData; } } else { return await ethers.provider.getFeeData(); } } const chainName = network.name; // "polygon-amoy" or whatever chain const { maxFeePerGas, maxPriorityFeePerGas } = await getFeeData(chainName);

Since Amoy is pretty new, the costs to deploy a contract also seem to vary quite a bit and are more expensive than when deploying on Mumbai. For example, an upgradable contract deployed on a low gas day (Sunday) cost around 0.18 MATIC, but on a higher gas day (Monday), it was closer to 0.8 MATIC.

If you need Amoy test MATIC, the Alchemy and Polygon faucets each drip 0.5 MATIC per day…which isn't great when fees are so high, but what can you do?!

by Dan Buchholz

We officially launched the Studio a few weeks ago and are continuing to build new features. Most of the more recent additions were changes to optimize the developer/user experience within the app. Let's review a few of them.

Let's be honest…everyone prefers dark mode! We changed the design to make this the default color when you visit the Studio app.

When creating projects or tables, you used to be redirected to a form to fill out the information, which led to more clicks. Now, a nice panel pops out from the right-hand side, which lets you fill out the info without the redirection.

In the Studio CLI, the concept of an environment ID was part of certain commands, but it wasn't obvious how or where these came from. The UI shows these on the project's page, along with improvements in how the project’s description is rendered. When working in the CLI, you use the project ID to then, for example, set a context that lets the CLI know you intend to work within this project.

Lastly, we made some small tweaks to the logic (e.g., route name guards against reserved keywords) and created a new package to help modularize forms.

Those initial changes were part of a design and user experience improvement process. Next, we're starting to dive into letting users edit certain information in the Studio—and if you're looking for something that isn't a feature or something that isn't working as expected, be sure to let us know!

by Marla Natoli

Wingbits** is organizing a community around employing Automatic Dependent Surveillance-Broadcast (ADS-B) technology to capture, aggregate and monetize aviation data, to enhance aviation safety and efficiency. They have a variety of hardware options to choose from, and participants can claim a hexagon of coverage on a first-come, first-served basis. The goal is to get enough coverage in certain areas to reach critical mass whereby they can begin monetizing the data. Participants are rewarded based on volume and quality of coverage data via a token, with the hope of creating a flywheel effect as more data is monetized.

ADS-B is a type of aircraft tracking data that is transmitted by aircraft in real-time including the aircraft’s position, velocity, altitude, etc. The main output includes flight tracking data feeds for both live and historic flights which enables developers to access real-time data for various use cases including flight tracking websites, insights to improve operations for airlines and charters, air traffic control organizations, emergency services, and more.

Wingbits is still early in their DePIN journey, and we’re excited to see how the incentive mechanisms they have in place can catapult their community toward creating enough data to kickstart monetization. This would be yet another industry that could begin creating valuable data outside of walled gardens and siloes, giving data consumers more options for how they collect and use data.

If you’re interested in starting to open up your data, or in exploring how decentralized data infrastructure could support collaboration, monetization, and verification, we’d love to hear from you. Feel free to set up some time with us here or join our Discord and get in touch.

**Information was taken from https://docs.wingbits.com/

by Jim Kosem

This past week has been a lot of making web prototypes, namely in Framer, a tool that allows designers to design and publish websites without coding. I learned how to make websites a long time ago, longer than most could imagine or want to imagine. It’s what led me to interaction design and user research. When you design software that nine times out of ten, or even more actually, comes to the world in a browser, if you’re not building an actual website, you’re building an application that shares many of the functions of one.

So, understanding the web as material means you need to not only know what can happen in a browser, but you have to sketch out how it looks and feels. You must play around with how it scales and changes and adapt your design to that. This means you need to prototype before you build. We’ll make websites properly, digging deeply into the frontend, but right now, design means seeing quickly how it works first.

by Brian Hoffstein

Last Thursday, we hosted a Twitter Spaces event featuring innovators from the forefront of "Decentralized AI." Moderated by Jonathan Victor of Ansa Research, this session delved into how blockchain and decentralized networks can revolutionize artificial intelligence. Leaders from this emerging frontier discussed the vast potential and the synergistic effects of these technologies, revealing numerous ways they could alter our economic systems, redefine privacy and security, and enhance communal governance.

Joel Thorstensson, co-founder of 3Box Labs and creator of Ceramic, emphasized the societal implications of decentralized technologies. He pointed out, "The consequence of AI models can be used for a lot of good stuff, a lot of bad stuff. And a lot of the bad stuff is like impersonation and things like that. This is actually something that the crypto space, since the early cypherpunks, foresaw… with things like public key cryptography, we can actually change the structure of society." He underscored the need for ethical frameworks to govern the deployment of these technologies, reflecting Ceramic Network’s commitment to building a secure, open web where data integrity and verifiability are paramount.

Ben Fielding, co-founder of Gensyn, discussed the practical benefits of decentralization for AI, particularly in terms of resource allocation. He noted, "Decentralization essentially allows us to unlock a lot of those resources by drastically lowering the barrier to being a supplier of the resources." Ben's insights stress how decentralization enhances the efficiency and scalability of AI applications, an approach that Gensyn is applying to create a global supercluster of computational resources. He also emphasized the vital role of open-source models in keeping the development of AI technologies accessible and inclusive.

Doug Petkanics, founder of Livepeer, focused on the market dynamics of decentralized networks, emphasizing the necessity of aligning supply with real-world demand. Doug explained, "To actually bring on real demand, and to make sure that fees are flowing to the supply side as well, you have to productize and you have to meet the market where it is, you have to make a product that fits in with the way that people want to consume these resources." He also highlighted the importance of regulatory environments that support rather than stifle innovation, ensuring that new technologies can reach their full potential without unnecessary barriers.

David Minarsch, co-founder of Valory, highlighted another dimension of decentralization by discussing the role of autonomous agents. He articulated, "Another way to think about decentralization in this context is at the other end of the spectrum, where the autonomous agent is something we share, which essentially works as a collective whole." David’s work at Valory is reshaping how communities and organizations interact with AI through their design of autonomous, co-owned systems that empower individual and collective decision-making, further underscoring the need for data verifiability to ensure trust and functionality in these systems.

The conversation explored the impact of regulatory environments on innovation, the importance of data verifiability to maintain trust in decentralized systems, and the necessity of open-source models to democratize access to technology. These interconnected themes illustrated the challenges and opportunities at the intersection of AI and blockchain technologies. As we look toward a future where these technologies are increasingly intertwined, the insights from these leaders will help provide a roadmap for navigating the challenges and seizing the opportunities that lie ahead.

We’d like to thank Jonathan for his excellent stewardship of the conversation, as well as Joel, Ben, Doug, and David, for their participation and contributions to the field broadly. Thanks to pioneers like them, the trajectory of technological development and its societal impact are in capable hands. You can listen to the full conversation here.

Stay tuned for upcoming discussions on the cutting-edge of our decentralized future by following us on Twitter and joining our Discord server.

The Filecoin Data Economy hack is over, and winners will be announced this upcoming Wednesday—three prizes for 1st/2nd/3rd will be awarded. Plus, the LearnWeb3 Decentralized Intelligence hack is also over, so keep an eye out for the top 3 hacks and top 4 runner-ups!

Want to dive deeper, ask questions, or just nerd out with us? Jump into our Telegram or Discord—including weekly research office hours or developer office hours. And if you’d like to discuss any of these topics in more detail, comment on the issue over in GitHub!

Are you enjoying Weeknotes? We’d love your feedback—if you fill out a quick survey, we’ll be sure to reach out directly with community initiatives in the future!: Fill out the form here

Begin transmission…

by Avichal Pandey

Axelar is a proof-of-stake blockchain built using Cosmos SDK. It delivers cross-chain communication between various blockchain networks.

In this post, we will see a minimal example of cross-chain message passing between an EVM contract deployed on the Filecoin blockchain and the one deployed on Ethereum. Let's first look at how Axelar's bridging network works.

Source: Axelar docs

The gateway contracts are deployed on each chain connected to the Axelar network. When a user initiates a cross-chain message, they first interact with a Gateway.

Once a Gateway contract receives a message, it emits an event. Incoming relayers pick up the event and submit it to the Axelar network for processing.

When Axelar validators receive the event sent by an external chain, they verify that it exists on the source chain and then vote to reach a consensus about its validity. The Gateway only allows actions on the external chain if the Axelar validators reach a consensus threshold.

Like the incoming relayers, outgoing relayers monitor outgoing transaction queues containing cross-chain messages already approved and signed by the validators and periodically submit these transactions to external chains.

Although a cross-chain transaction touches at least three different chains, thanks to the Gas Receiver contract, the gas can be paid with the native tokens of the source chain.

To provide a developer-friendly experience, Axelar deploys a Gas Receiver contract to all connected EVM chains. Under the hood, the Gas Receiver estimates the total gas cost required across the source chain, Axelar network, and destination chain. It converts source-chain tokens into AXL, destination-chain tokens, and other required currencies.

Hence, in our example, we can pay for the entire transaction with tFIL.

Start by importing the Axelar Solidity SDK in the source file. We will use three dependencies from the SDK: AxelarExecutable, IAxelarGateway, and IAxelarGasService. We will extend the AxelarExecutable contract and use it to send and receive cross-chain messages on respective chains.

We must provide the address of the Gateway and the Gas Service in the constructor. Here is a list of available gateways and gas services contracts deployed on various supported networks.

Next, we will add two more functions to our contract to implement a minimal cross-chain message flow. The function sendMessage will trigger a new cross-chain message. Along with the actual message, we will provide destinationChain and destinationAddress. Here is a list of supported destination chain names.

Lastly, we will add one more function: _execute. It will be used on the other side to receive the message. In addition to the message we sent, we will receive the sourceChain and sourceAddress.

We will deploy this contract on the Filecoin calibration testnet and Ethereum Sepolia to test our minimal cross-chain flow. Here is how to do it using Forge. Recall that the constructor arguments are Gateway and Gas Service contract addresses on the respective chains.

forge create --rpc-url "YOUR_RPC_URL" --private-key "YOUR_PRIVATE_KEY" --contracts src/SenderReceiver.sol SenderReceiver --constructor-args 0x999117D44220F33e0441fbAb2A5aDB8FF485c54D0xbE406F0189A0B4cf3A05C286473D23791Dd44Cc6

We can trigger the cross-chain message by calling the sendMessage function on the source chain. Finally, remember to abi.encode the actual message as a string.

cast send --private-key YOUR_PRIVATE_KEY --rpc-url YOUR_RPC_URL 0x6e80e2411E8afc76186A2f4955FBf037B95b25FE "sendMessage(string,string,string)" "ethereum-sepolia" "0x6e80e2411E8afc76186A2f4955FBf037B95b25FE" "0x0000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000001548656c6c6f2066726f6d20636f6e747261637420410000000000000000000000"

After you have successfully triggered a cross-chain message, you can monitor its status on axelarscan.

by Dan Buchholz

The Tableland SDK is designed for JavaScript/TypeScript applications. For Python developers, there isn't a great way to get started. We'll be releasing some official docs to make it easier to set up a Tableland connection shortly, but we wanted to give devs a quick previous as to what they can do.

The most popular framework for working with blockchains is web3.py, so start by installing this package in your Python project: pip install web3.

You need the Tableland registry's ABI (TablelandTables.sol) to interact with the contract. The evm-tableland repo makes this easy with the following steps:

git clone <https://github.com/tablelandnetwork/evm-tableland> cd evm-tableland npm install npm run build cat artifacts/contracts/TablelandTables.sol/TablelandTables.json | jq '.abi' > abi.json

Now that we have the ABI, we can connect to the registry. There are two things we'll need:

• A private key to sign transactions.

• The provider of the chain's RPC provider.

The example below shows how you'd do this if you're connecting to Local Tableland with a Hardhat account:

# Define provider and private key provider_uri = "<http://localhost:8545>" private_key = "59c6995e998f97a5a0044966f0945389dc9e86dae88c7a8412f4603b6b78690d" # Set up web3 instance and signer w3 = Web3(Web3.HTTPProvider(provider_uri)) signer = Account.from_key(private_key)

We have the provider and signer. Thus, we can load the ABI and connect to the Tableland registry. If we're using Local Tableland, the registry address is 0xe7f1725e7734ce288f8367e1bb143e90bb3f0512:

from web3 import Web3 from eth_account import Account from eth_utils import to_checksum_address # Existing code... abi_file = "abi.json" # Path to the ABI file abi = read_file_to_json(abi_file) registry_addr = "0xe7f1725e7734ce288f8367e1bb143e90bb3f0512" registry = w3.eth.contract( address=to_checksum_address(registry_addr), abi=abi )

This is all we need to make onchain calls! We can execute both a CREATE TABLE and INSERT statement like so, and for each, we can parse the CreateTable or RunSQL event, respectively:

# Create a table table_prefix = "my_table" chain_id =. 31337 # E.g., for Local Tableland + Hardhat node tx_hash_create = registry.functions.create( signer.address, f"CREATE TABLE {table_prefix}_{chain_id} (id integer, val text)" ).transact() create_rec = w3.eth.wait_for_transaction_receipt(tx_hash_create) create_log = registry.events.CreateTable().process_receipt(create_rec) data = create_log[0] table_id = data["args"]["tableId"] # Write to it table_name = f"{table_prefix}_{chain_id}_{table_id}" tx_hash_mutate = self.registry.functions.mutate( signer.address, table_id, f"INSERT INTO {table_name} VALUES (1, 'hello')" ).transact({"from": signer.address}) mutate_rec = w3.eth.wait_for_transaction_receipt(tx_hash_mutate) mutate_log = registry.events.RunSQL().process_receipt(mutate_rec) data = mutate_log[0] transaction_hash = data["transactionHash"].hex()

There's an experimental Python SDK that can be used to interact with the Tableland registry. It's not officially supported (i.e., not an official Tableland package), but it's a good starting point for Python developers. One of the challenges with the approach above is that you also need to interact with the validator APIs to make sure onchain SQL is successfully materialized.

The [tableland](<https://github.com/dtbuchholz/tableland-py>) Python library handles this and makes it easier to get started. You can install by running the following command:

pip install tableland

You can import the library and set up the connection in a way that's similar to the official Tableland JavaScript SDK:

from tableland import Database private_key = "59c6995e998f97a5a0044966f0945389dc9e86dae88c7a8412f4603b6b78690d" # Replace with your private key db = Database( private_key=private_key, provider_uri="<http://localhost:8545>", # Replace with your RPC provider URL )

Creating a table includes the registry contract call as shown in the walkthrough above, but it also handles waiting for the validator to materialize the SQL and includes error messages in case of syntax issues:

# Create a new table statement = "create table my_table (id int, val text)" create_event = db.create(statement) table_name = create_event["table_name"] table_id = create_event["table_id"] # Check if there are any errors if create_event["error"] is not None: print(f"Error: {create_event['error']}")

Once a table is created, writing to it is straightforward and follows a similar pattern.

# Insert a row into the table statement = f"insert into {table_name} (id, val) values (1, 'hello')" write_event = db.write(statement) # Check if there are any errors if write_event["error"] is not None: print(f"Error: {write_event['error']}")

Lastly, the read method lets you write arbitrary SQL statements to query the table:

# Query the table statement = f"select * from {table_name}" data = db.read(statement) print(data) # [{'id': 1, 'val': 'hello'}]

If you're interested in contributing to the library, you can find the source code on GitHub.

by Jim Kosem

Colour can count for a lot. Recently with Tableland Studio we’ve been piloting a dark mode colour scheme. This is a bigger deal than you probably think and the history of dark mode is an interesting one. We’ve been working with the idea coming from the angle of Studio being a developer tool, and especially as such, it works first in CLI, and then we bring it to the GUI. This actually matches up with the origin story of dark mode, which was in early UI designs for text-based terminals.

Way back when, actually turning on a pixel had computing and energy costs that were comparatively significant. So, it was cheaper to just turn on the ones you needed, which at the time, were for text. Then, as early Windows-based interfaces came to the scene, the thinking was that it should remind all these people new to using computers of paper. So, it all went white. That's simple, really. There are, of course, other issues in terms of glare and spending long amounts of time reading screens, where dark mode may help, but we’re thinking about the aesthetics and our origin in terminals and paying a bit of homage to that.

We’ve been sponsoring the Filecoin Data Economy and the LearnWeb3 Decentralized Intelligence hackathons. Projects have made their final submissions, and we’ll be judging and deciding on winners by the end of the week. Check out next week’s post for more details on those projects!

Want to dive deeper, ask questions, or just nerd out with us? Jump into our Telegram or Discord—including weekly research office hours or developer office hours. And if you’d like to discuss any of these topics in more detail, comment on the issue over in GitHub!

Are you enjoying Weeknotes? We’d love your feedback—if you fill out a quick survey, we’ll be sure to reach out directly with community initiatives in the future!: Fill out the form here

TL;DR

WeatherXM used Textile to do the following:

• Bring open data availability of station/device data and rewards calculations to help progress toward fully decentralized infrastructure and bring transparency to the dataset.

• Enable verifiable data such that the data is signed at the source, ensuring all data is cryptographically provable with access controls on data mutations.

• A decentralized framework for computation, such as QoD, providing the WeatherXM ecosystem a way to build on top of the data.

You can view available “events” (data pushes) here and extract the device data at the https://basin.tableland.xyz/events/<event_cid> endpoint for further analysis, as described below.

Textile and WeatherXM are working together to design a solution for DePIN data availability (“DA”), storage, and retrieval. If you’re unfamiliar with WeatherXM, it is a network that's powered by community-owned devices, purpose-built by WeatherXM for weather data collection. As part of the network, stations/devices throughout the globe are incentivized to share data.

WeatherXM’s goal is to make network data more accessible so developers can build on top of it. The station/device data are replicated to Basin for decentralized storage, allowing for computation over that data. The data flows through Basin, which can support cache with TTL, and is replicated to Filecoin for long-term storage guarantees.

Basin is designed to be flexible and work with raw file uploads (e.g., parquet) or even database streaming (e.g., Postgres). Data is replicated to the network in a “vault” and then can be retrieved by the vault’s CIDs for further processing.

There are two possible data availability options for a vault—hot or cold storage:

• Hot layer: Cache with time-to-live (TTL) where the data is made available for the defined timeline and immediately retrievable (i.e., no need to wait for long propagation or finality times).

• Cold layer: Persistence on Filecoin where all data is flushed for long-term retrieval.

This makes it possible to run computation over the open data where both the inputs—and, potentially, outputs—are highly available in the short term and also retrievable after cold storage flushing (e.g., dispute resolution).

WeatherXM device data is made available for data consumers through hot + cold storage replication, and this data is used today in a dashboard available at https://index.weatherxm.network. You’ll notice two tabs on that page:

• Weather Data: Device data located in the Textile vault wxm.weather_data_dev

• Rewards: Calculations over the device data, located in the Textile vault wxm.rewards_merkle_trees_dev

We also built a demonstration of how developers can use the DePIN network’s data in the downstream compute and visualization pipeline. One of WeatherXM’s goals is to progressively decentralize their stack, and Basin is a key component to making that possible. We wanted to provide a way for developers to see exactly how they can build on top of the public dataset and add value to the WeatherXM ecosystem.

We built a demonstration about how you can leverage WeatherXM and run queries over the data, which we’ll walk through below. You can review the Data.md for the final visualizations that plot WeatherXM network data over historical trends and across the globe.Check it out here*: https://github.com/textileio/demo-basin-wxm-query*

The setup process for WeatherXM involved:

• Creating a vault with the Basin CLI, which is an onchain interaction signed by a private key.

• Aggregating device data that is processed by internal systems for Quality of Data purposes.

• Replicating internal data to the Basin DA layer on a consistent schedule—i.e., parquet files containing device data are written to the vault and signed by a private key.

Once that data is available, anyone with knowledge of the unique vault identifier or vault owner’s address can extract and process the data. It’s openly accessible on the Basin network and through the cold layer. In our demo described below, we built a program that consists of:

• A Python script to fetch remote parquet files written to the vault, filtered over a time range, and identified at an IPFS CID.

• A DuckDB in-memory SQL database for loading parquet files and executing queries.

• Writing the query results to a CSV and markdown files and producing data visualization with geopandas (e.g., precipitation rate per geography).

• Running the steps above weekly with GitHub Actions.

You could imagine more complex flows with Docker images or compute networks like Bacalhau, but the idea was to keep things lightweight to showcase how to interact with open, verifiable data.

If you want to understand the basics behind Basin, you can start by installing the CLI on your machine. Make sure the path to Go binaries is in your shell’s PATH variable (e.g., include $(go env GOPATH)/bin).

go install github.com/tablelandnetwork/basin-cli/cmd/vaults@latest

Then, you can take a peek at what the structure and data within a vault look like. For example, list all vaults created by a specific account/address:

vaults list --account 0x75c9eCb5309d068f72f545F3fbc7cCD246016fE0

Or, list out the events associated with a vault:

vaults events --vault wxm.weather_data_dev

Which provides the “events” (i.e., vault mutations & CIDs) for the vault. Note the cache_expiry field shows when the highly available data will expire from the cache.

[
  {
    "cid": "bafybeicizws2qymmcn657b3qa7rm2w2lz4bdumxc6b66jw4ac4fsrhmsxm",
    "timestamp": 1709078400,
    "is_archived": false,
    "cache_expiry": "2024-05-16T20:56:47.379695"
  },
  {
    "cid": "bafybeigppam7v3uwarke5iulytgptdj6hzjta5kye7nljbvemzje6r3xma",
    "timestamp": 1709596800,
    "is_archived": false,
    "cache_expiry": "2024-05-16T19:49:50.277204"
  }
]

This period is defined upon vault creation such that all files written to the vault follow the same expiry logic. Note the data is flushed to cold storage and still available after the cache expiration. The CIDs can then be retrieved and processed locally:

vaults retrieve --output <filename> <event_cid>

Now that we understand how Basin works, we can review some of the demo’s features since it builds on top of the WeatherXM vault data. The first step is to get all of the vaults for a specific address and then retrieve all of the events and CIDs for the vaults. This can be done programmatically with the Basin HTTP API (where {variable} would be replaced by actual string values—e.g., account=0x75c9eCb5309d068f72f545F3fbc7cCD246016fE0):

• Get vaults for address: curl https://basin.tableland.xyz/vaults?account={address}

• Get events for vault: curl https://basin.tableland.xyz/vaults/{vault}/events

• Download the raw data at an event: curl https://basin.tableland.xyz/events/{event_cid} -o data.parquet

Let’s presume the files have been downloaded at each event for WeatherXM’s vault into some data_dir directory. We then load these parquet files into a DuckDB in-memory instance.

from pathlib import Path

from duckdb import connect

def create_database(data_dir):
    db = connect()
    files = Path(data_dir) / "*.parquet"  # Read all parquet files in data directory
    db.execute(f"CREATE VIEW xm_data AS SELECT * FROM read_parquet('{files}');")

    return db

We can now run queries over the WeatherXM device data! This will calculate average or aggregate metrics for temperature, precipitation, wind, etc., and create a DataFrame for subsequent processing.

def execute_queries(db, start, end):
    # Set up columns for average calculations
    columns = [
        "temperature",
        "humidity",
        "precipitation_accumulated",
        "wind_speed",
        "wind_gust",
        "wind_direction",
        "illuminance",
        "solar_irradiance",
        "fo_uv",
        "uv_index",
        "precipitation_rate",
        "pressure",
    ]

    # Set up all query parts
    avg_selection = [f"avg({col}) AS {col}" for col in columns]
    avg_calculations = ",".join(avg_selection)
    query_parts = [
        "SELECT min(timestamp) AS range_start, max(timestamp) AS range_end,",
        "COUNT(DISTINCT device_id) AS number_of_devices,",
        "mode(cell_id) AS cell_id_mode,",
        "sum(precipitation_accumulated) AS total_precipitation,",
        avg_calculations,
    ]

    # Add WHERE clause for time filtering, if applicable
    where_clause = ""
    if start is not None and end is not None:
        where_clause = f"WHERE timestamp >= {start} AND timestamp <= {end}"
    elif start is not None:
        where_clause = f"WHERE timestamp >= {start}"
    elif end is not None:
        where_clause = f"WHERE timestamp <= {end}"

    # Combine all parts into one query and execute
    query = " ".join(query_parts) + " FROM xm_data" + f" {where_clause}"
    try:
        result = db.execute(query).pl()  # Create a polars DataFrame
        return result
    except Exception as e:
        print(f"Error executing DuckDB aggregate queries: {e}")
				raise

Lastly, we can use geopandas to create visualizations from the returned DataFrame. In the example, there are a few rough bounding boxes (“bbox”) for each major continent with lat/long bounds:

bboxes = {
    "north_america": (14, 72, -172, -52),
    "south_america": (-55, 12, -85, -34),
    "europe": (35, 72, -13, 60),
    "africa": (-35, 38, -18, 55),
    "asia": (-11, 81, 25, 179),
    "australia": (-48, -6, 108, 178),
}

For example, this calculates the total precipitation accumulated by cell_id within a bounding box:

def query_bbox(db, bbox, start, end):
    # DuckDB query to select and aggregate data within the bounding box
    query = f"""
    SELECT cell_id, SUM(precipitation_accumulated) as total_precipitation, AVG(lat) as lat, AVG(lon) as lon
    FROM xm_data
    WHERE lat BETWEEN {bbox[0]} AND {bbox[1]} AND lon BETWEEN {bbox[2]} AND {bbox[3]}
    """

    # Add WHERE clause for time filtering, if applicable
    where_clause = ""
    if start is not None and end is not None:
        where_clause = f"AND timestamp >= {start} AND timestamp <= {end}"
    elif start is not None:
        where_clause = f"AND timestamp >= {start}"
    elif end is not None:
        where_clause = f"AND timestamp <= {end}"

    # Combine all parts into one query and execute
    query = query + f" {where_clause}" + " GROUP BY cell_id"
    try:
        result = db.execute(query).df()  # Create a pandas Dataframe
        return result
    except Exception as e:
				print(f"Error executing DuckDB bbox query: {e}")
				raise

By taking the pandas DataFrame and passing it to geopanda’s GeoDataFrame method, you can create maps that plot data at lat/long coordinates, such as the one described above for the North American bbox. The history CSV file also plots information week-over-week—below is what the device trends look like (hint: the number of devices is increasing!):

If you’d like to develop with or alter the demo code, start by cloning the repo and going through the setup steps described in the README.

git clone https://github.com/textileio/demo-basin-wxm-query

Then, you can use the make run command to fetch events and compute queries over the data for the WeatherXM vault. You can also choose to pass start and end parameters to the make run command, which specifies timestamp ranges for data extraction.

If you want to use the WeatherXM data, the Basin HTTP APIs described in the sections above should provide all that you need. Namely, you can fetch the events and download the parquet files locally—or event load remote data into DuckDB from an IPFS gateway!

This is just the first post in a series of Basin data demonstrations. We’ll continue to put out additional walkthroughs in long-form posts, but if you’d like to stay up to date with what we’re building on a week-by-week basis, check out the Weeknotes newsletter on Substack!