<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/">
    <channel>
        <title>Tableland</title>
        <link>https://paragraph.com/@tableland-2</link>
        <description>Tableland is a permissionless database that allows developers to use relational data and SQL from any contract, wallet, or app. </description>
        <lastBuildDate>Fri, 20 Nov 2026 18:12:34 GMT</lastBuildDate>
        <docs>https://validator.w3.org/feed/docs/rss2.html</docs>
        <generator>https://github.com/jpmonette/feed</generator>
        <language>en</language>
        <image>
            <title>Tableland</title>
            <url>https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/06adcd34ab302457fa99093ca133d4e880391af77fab9677ddbf79ef5e7ed0da.png</url>
            <link>https://paragraph.com/@tableland-2</link>
        </image>
        <copyright>All rights reserved</copyright>
        <item>
            <title><![CDATA[We're moving to Paragraph.xyz!]]></title>
            <link>https://paragraph.com/@tableland-2/we-re-moving-to-paragraph-xyz</link>
            <guid>24raPiipWNAk6Ud3F826</guid>
            <pubDate>Tue, 07 May 2024 01:47:27 GMT</pubDate>
            <description><![CDATA[We started this blog and our Weeknotes newsletter on Substack to give the community insight into the latest happenings with Tableland. Since then, we began research and experiments around web3-native data needs—which led to the MVP for Basin. Our blogs started to incorporate topics outside of the core Tableland database, but everything we’ve shared has been built by the team behind the protocol—Textile. ICYMI—Mirror and Paragraph are joining forces (see here), so moving from Mirror to Paragra...]]></description>
            <content:encoded><![CDATA[<p>We started this blog and our <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tableland.substack.com/">Weeknotes newsletter on Substack</a> to give the community insight into the latest happenings with Tableland. Since then, we began research and experiments around web3-native data needs—which led to the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/basin-cli">MVP for Basin</a>. Our blogs started to incorporate topics <em>outside</em> <em>of</em> the core Tableland database, but everything we’ve shared has been built by the team behind the protocol—<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/textileio">Textile</a>.</p><p>ICYMI—<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://mirror.xyz/tableland.eth/">Mirror</a> and <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://paragraph.xyz/">Paragraph</a> are joining forces (see <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://x.com/paragraph_xyz/status/1786068122895790340">here</a>), so moving from Mirror to Paragraph was inevitable. For reference, Paragraph is a web3-native blog + newsletter service. And since we’ve been publishing about Textile-built products under “Tableland” handles, we decided it was time to consolidate everything from both Mirror and Substack to Paragraph under a single <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://paragraph.xyz/@textileio">@textileio</a> handle.</p><blockquote><p>You can find all of our blogs at <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://blog.textile.io/">https://blog.textile.io</a> (it’s using Paragraph under the hood).</p></blockquote><h3 id="h-what-do-i-need-to-do" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">What do I need to do?</h3><p>Nothing! We’ll export all of our Substack and Mirror subscribers and import them to Paragraph. Starting next week, you’ll automatically receive the Weeknotes newsletter in your inbox!</p><h3 id="h-for-the-devs-and-content-creators" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">For the devs &amp; content creators</h3><p>If you’re in a similar position, Paragraph <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.paragraph.xyz/docs/migrating-to-paragraph">makes it super simple to migrate</a> your existing articles and subscribers from Substack and Mirror. After signing up, you’ll have a dashboard page to edit your settings—including the import/export page. Here, you can just drop in an RSS link.</p><p>For example, with our existing Tablealnd Substack and Mirror, the RSS feed can be found by appending a specific URI on the domain:</p><ol><li><p><strong>Substack</strong>: append <code>/feed</code>—for example, <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tableland.substack.com/feed">https://tableland.substack.com/feed</a></p></li><li><p><strong>Mirror</strong>: append <code>/feed/atom</code>—for example, <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://mirror.xyz/tableland.eth/feed/atom">https://mirror.xyz/tableland.eth/feed/atom</a></p></li></ol><p>And that’s (basically) it! There also might be some steps for exporting subscribers or content regarding who published the article, but that’s the gist of it.</p>]]></content:encoded>
            <author>tableland-2@newsletter.paragraph.com (Tableland)</author>
            <enclosure url="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/345ad61e0fce43e835de59bbfb9c39a476fc864f4365828ccd7fbcd99f908c1f.png" length="0" type="image/png"/>
        </item>
        <item>
            <title><![CDATA[Data Availability: Reviewing  proof & storage approaches]]></title>
            <link>https://paragraph.com/@tableland-2/data-availability-reviewing-proof-storage-approaches</link>
            <guid>9fWcvGKG7PQFQOt4aw7K</guid>
            <pubDate>Thu, 04 Apr 2024 19:31:49 GMT</pubDate>
            <description><![CDATA[BackgroundData Availability (”DA”) is a hot topic now that EIP-4844 is live! There are a few major players in the space, and all of them take similar approaches but with a few differences. We&apos;ll take a look at EigenDA, Celestia, Avail, and Arbitrum AnyTrust. But first, let&apos;s start off with a bit of background information. DA layers ensure that block data is provably published so that applications and rollups can know what the state of the chain is—but once the data is published, DA ...]]></description>
            <content:encoded><![CDATA[<h2 id="h-background" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Background</h2><p>Data Availability (”DA”) is a hot topic now that EIP-4844 is live! There are a few major players in the space, and all of them take similar approaches but with a few differences. We&apos;ll take a look at EigenDA, Celestia, Avail, and Arbitrum AnyTrust. But first, let&apos;s start off with a bit of background information.</p><p>DA layers ensure that block data is provably published so that applications and rollups can know what the state of the chain is—but once the data is published, DA layers <em>do not</em> guarantee that historical data will be permanently stored and remain retrievable. DAs either deploy a validity proof <em>or</em> a fraud/fault proof (validity proofs are more common). Data availability sampling (”DAS”) by light clients is also a common feature to ensure data can be recovered. However, the term <em>DA</em> typically refers to simple blocks/transaction data, so it differs from large, arbitrary, long-term data availability and storage.</p><h2 id="h-proof-types-and-related-terms" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Proof types &amp; related terms</h2><p><em>The following section outlines common terms across DA protocols. Skip if familiar.</em></p><p>To clarify, here&apos;s a quick recap of what DAs implement:</p><ul><li><p><em>Validity proofs</em>: ensure that all data and transactions are valid before they are included onchain via zk-SNARKs/STARKs.</p><ul><li><p>Computationally intensive but provides strong security guarantees.</p></li></ul></li><li><p><em>Fraud/fault proofs</em>: allow data to be posted onchain before guaranteed valid—and use a challenge period for tx dispute resolution.</p><ul><li><p>Less computationally intensive but lower security guarantees (i.e., requires the network to actively generate fraud proofs).</p></li></ul></li><li><p><em>KZG commitment scheme</em>: data redundancy via erasure encoding—and correctness thereof without needing a fraud proof.</p><ul><li><p>E.g., full nodes to prove transaction inclusion to light nodes using succinct proof.</p></li></ul></li><li><p><em>Erasure encoding</em>: reduce per-node storage requirements by splitting data up across many nodes while ensuring the original data can be recovered if lost.</p><ul><li><p>This involves decreasing an individual node’s storage requirement by increasing the total size of a piece of data (splitting into blocks &amp; adding additional redundancy/erasure encoded blocks).</p></li><li><p>Then, distribute the blocks across many nodes. If you need the original data, it should be recoverable by piecing blocks back together from the network—assuming some defined tolerance threshold is held.</p></li></ul></li><li><p><em>Data availability sampling</em>: ensure data availability without requiring nodes to hold the entire dataset; complements erasure encoding to he</p></li><li><p>lp guarantee data is available.</p><ul><li><p>I.e., randomly sampled pieces of erasure-coded block data to assure the entire block is available in the network for reconstruction—else, slash nodes.</p></li></ul></li><li><p><em>Data availability committee</em>: a trusted set of nodes—or validators in a DA PoS network—that store full copies of data and publish onchain attestations to prove ownership.</p></li></ul><h2 id="h-layer-2s-and-da-approaches" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Layer 2s &amp; DA approaches</h2><p><em>The following section outlines common L2s and how they approach DA.</em></p><p>According to <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.availproject.org/docs/the-avail-trinity/avail-da#taking-layer-2-data-off-chain">Avail</a>, there are a few different approaches that L2s take to DA. Note this is in the sense of <em>block/transaction</em> DA and differs from the “arbitrary” / large DA approach Textile focuses on with Basin and object storage:</p><h3 id="h-zk-and-optimistic-rollups" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0"><strong>ZK &amp; optimistic rollups</strong></h3><ul><li><p>Post proofs (validity or fraud) onchain along with state commitments.</p></li></ul><h3 id="h-plasma-rollup-offchain-da" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0"><strong>Plasma</strong>: Rollup + offchain DA</h3><ul><li><p>All data and computation, except for deposits, withdrawals, and Merkle roots, to be kept offchain.</p></li></ul><h3 id="h-optimiums-optimistic-rollups-offchain-da-subclass-of-plasma" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0"><strong>Optimiums:</strong> Optimistic rollups + offchain DA (subclass of Plasma)</h3><ul><li><p>Adaptation of Optimistic rollups that also take data availability offchain while using fraud proofs for verification.</p><ul><li><p>I.e., differs from traditional rollups in that transaction data is entirely in offchain storage.</p></li></ul></li><li><p>E.g., Optimism offers a “<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://specs.optimism.io/experimental/plasma.html#da-storage">plasma mode</a>” where data is uploaded to the DA storage layer via plain HTTP calls</p></li></ul><blockquote><br></blockquote><h3 id="h-validiums-zk-rollups-offchain-da-subclass-of-plasma" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0"><strong>Validiums:</strong> ZK rollups + offchain DA (subclass of Plasma)</h3><ul><li><p>Adaptation of ZK rollups that shift data availability offchain while continuing to use validity proofs.</p></li><li><p>E.g., Starknet posts a <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/starkware-libs/ethSTARK/blob/master/rescue_stark_documentation.pdf">STARK</a> validity proof and also sends a <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.starknet.io/documentation/architecture_and_concepts/Network_Architecture/on-chain-data/"><em>state diff</em></a>, which represents the changes in the L2 state since the last validity proof was sent (updates/modifications made to the network&apos;s state).</p></li></ul><h3 id="h-volitions-zk-rollups-da-ie-the-user-picks-da-location-dual-mode-operation-for-either-onchain-or-offchain-da-opt-for-zk-rollup-mode-with-zk-proofs-to-ensure-the-integrity-and-validity-of-transactions-where-transaction-data-is-stored-onchain-opt-for-validium-mode-which-stores-transaction-data-offchain-enhancing-scalability-and-throughput-while-maintaining-robust-validity-proofs-sovereign-rollups-independent-rollups-maintain-autonomy-over-security-and-data-availability-models-protocol-decides-if-data-availability-is-either-onchain-or-offchain-theres-no-standard-approach-here-da-solutions-overview-the-following-section-outlines-common-da-layers-and-how-they-implement-their-solution-eigenda-validity-proofs-data-attestations-kzg-commitments-and-reed-solomon-erasure-encoding-while-other-systems-such-as-celestia-and-danksharding-planned-also-make-use-of-reed-solomon-encoding-they-do-so-only-for-the-purpose-of-supporting-certain-observability-properties-of-data-availability-sampling-das-by-light-nodes-on-the-other-hand-all-incentivizedfull-nodes-of-the-system-download-store-and-serve-the-full-system-bandwidth-source-eigenda-produces-a-da-attestation-that-asserts-that-a-given-blob-or-collection-of-blobs-is-available-have-both-a-liveness-and-safety-threshold-enable-consensus-about-whether-a-given-blob-of-data-is-fully-within-the-custody-of-a-set-of-honest-nodes-attestations-are-anchored-to-one-or-more-quorums-each-of-which-defines-a-set-of-eigenlayer-stakers-that-underwrite-the-security-of-the-attestation-architecture-see-here-operator-eigenda-full-nodesthe-service-providers-of-eigendastore-chunks-of-blob-data-for-a-predefined-period-and-serve-these-chunks-upon-request-disperser-currently-centralized-is-responsible-for-encoding-blobs-distributing-them-to-the-da-nodes-and-aggregating-their-digital-signatures-into-a-da-attestation-retriever-a-service-that-queries-eigenda-operators-for-blob-chunks-verifies-that-blob-chunks-are-accurate-and-reconstructs-the-original-blob-for-the-user-prune-data-after-some-predefined-period-for-referenceeigenlayer-vs-eigenda-eigenlayer-serves-as-a-platform-connecting-stakers-and-infrastructure-developers-where-stakers-have-the-option-to-restake-their-stake-and-contribute-to-the-security-of-other-infrastructures-while-earning-native-eth-rewards-ie-pool-security-vs-fragmenting-it-avs-stands-for-an-actively-validated-servicesoftware-that-uses-its-own-set-of-nodes-that-are-doing-something-that-requires-verificationvalidation-eg-consensus-da-tee-etc-the-core-eigenlayer-protocol-lets-people-take-their-eth-restake-it-liquid-staking-tokens-and-then-take-these-tokens-and-stake-to-eigenlayer-nodes-an-eigenlayer-operator-node-can-run-arbitrary-softwareie-developers-can-create-an-avs-of-their-choice-such-as-a-database-load-balancer-log-service-etc-an-example-of-an-avs-implementation-is-the-eigendaie-eigenlayer-is-dogfooding-their-base-protocol-celestia-fraud-proofs-das-fraud-proofs-via-namespaced-merkle-trees-and-reed-solomon-erasure-encoding-light-clients-verify-headers-through-dasonly-sample-blocks-within-a-30-day-window-instead-of-sampling-all-blocks-from-genesis-while-performing-das-for-a-block-header-every-light-node-queries-celestia-nodes-for-a-number-of-random-data-shares-from-the-extended-matrix-and-the-corresponding-merkle-proofs-if-all-the-queries-are-successful-then-the-light-node-accepts-the-block-header-as-valid-from-a-da-perspective-assumptions-there-is-a-minimum-number-of-light-nodes-depends-on-the-block-size-that-are-conducting-data-availability-sampling-for-a-given-block-size-light-nodes-assume-they-are-connected-to-at-least-one-honest-full-node-and-can-receive-fraud-proofs-for-incorrectly-erasure-coded-blocks-prune-data-after-30-day-window-they-recommend-l2-rollups-that-want-to-use-celestia-to-implement-their-own-long-term-storage-plan-here-avail-validity-proofs-das-kzg-commitments-rs-erasure-encoding-light-clients-access-and-sample-data-and-ensure-correct-block-encoding-and-provide-data-availability-guarantees-upon-the-finalization-of-new-blocks-have-3-node-types-full-nodes-download-and-verify-the-correctness-of-blocks-but-do-not-partake-in-the-consensus-process-validator-nodes-responsible-for-generating-blocks-deciding-on-transaction-inclusion-and-maintaining-the-order-incentivized-through-consensus-participation-light-clients-das-query-full-nodes-to-check-kzg-polynomial-openings-against-the-commitments-in-the-block-header-for-each-sampled-cell-full-nodes-prune-data-after-some-cutoff-period-arbitrum-anytrust-variant-of-arbitrum-nitro-data-availability-committees-that-post-data-availability-certificates-dacert-dacert-is-a-proof-that-the-blocks-data-will-be-available-from-at-least-one-honest-committee-member-until-an-expiration-time-anytrust-sounds-like-a-volitionthe-sequencer-either-posts-data-blocks-on-the-l1-chain-as-calldatablobs-or-it-posts-dacerts-dac-members-run-da-servers-with-two-endpoints-sequencer-api-to-submit-data-blocks-for-storage-rest-api-that-allows-data-blocks-to-be-fetched-by-hash-related-resources-da-protocol-deep-dives-httpschroniclecastlecapitalvcpdeepdive-data-availability-avail-da-comparison-httpsblogavailprojectorga-guide-to-selecting-the-right-data-availability-layer-avail-da-overview-httpsdocsavailprojectorgdocsthe-avail-trinityavail-da-avail-rfp-for-storage-httpsgithubcomavailprojectavail-unchartedblobmaingrantsrfpsrfp-003md-eigenlayer-design-httpswwwblogeigenlayerxyzycie-ethereum-da-basics-httpsethereumorgendevelopersdocsdata-availability-starknet-da-overview-httpsbookstarknetioch03-01-03-data-availabilityhtmlrecreating-starknets-state-optimism-da-overview-httpsspecsoptimismioexperimentalplasmahtmlda-storage-vitaliks-da-and-plasma-overview-httpsvitalikethlimogeneral20231114neoplasmahtml-arbitrum-anytrust-da-overview-httpsdocsarbitrumioinside-anytrustdata-availability-certificates-celestia-da-and-storage-httpsdocscelestiaorglearnretrievability-fraud-and-da-proofs-via-celestia-httpsarxivorgabs180909044" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0"><strong>Volitions:</strong> ZK rollups + DA (i.e., the user picks DA location) Dual-mode operation for either onchain or offchain DA: Opt for zk rollup mode with zk proofs to ensure the integrity and validity of transactions where transaction data is stored onchain. Opt for Validium mode, which stores transaction data offchain, enhancing scalability and throughput while maintaining robust validity proofs <strong>Sovereign Rollups:</strong> Independent rollups Maintain autonomy over security and data availability models; protocol decides if data availability is either onchain or offchain. There’s no standard approach here. DA solutions overview <em>The following section outlines common DA layers and how they implement their solution.</em> <strong>EigenDA</strong> Validity proofs + data attestations <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/Layr-Labs/eigenda/blob/master/docs/spec/attestation/encoding.md#trustless-encoding-via-kzg-and-reed-solomon">KZG commitments &amp; Reed-Solomon erasure encoding</a> While other systems such as Celestia and Danksharding (planned) also make use of Reed Solomon encoding, they do so only for the purpose of supporting certain observability properties of Data Availability Sampling (DAS) by light nodes. On the other hand, all incentivized/full nodes of the system download, store, and serve the full system bandwidth. Source <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/Layr-Labs/eigenda/blob/master/docs/spec/overview.md#attestation-layer">EigenDA produces a DA attestation</a> that asserts that a given blob or collection of blobs is available. Have both a liveness &amp; safety threshold Enable consensus about whether a given blob of data is fully within the custody of a set of honest nodes Attestations are anchored to one or more &quot;Quorums,&quot; each of which defines a set of EigenLayer stakers that underwrite the security of the attestation. Architecture (see <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.eigenlayer.xyz/eigenda/overview">here</a>): <strong>Operator:</strong> EigenDA full nodes—the service providers of EigenDA—store chunks of blob data for a predefined period and serve these chunks upon request. <strong>Disperser:</strong> (currently centralized) is responsible for encoding blobs, distributing them to the DA nodes, and aggregating their digital signatures into a DA attestation. <strong>Retriever:</strong> a service that queries EigenDA operators for blob chunks, verifies that blob chunks are accurate, and reconstructs the original blob for the user Prune data after <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/Layr-Labs/eigenda/blob/master/docs/spec/overview.md#core-components">some predefined period</a>. <em>For reference—EigenLayer vs. EigenDA:</em> <em>EigenLayer</em> serves as a platform connecting stakers and infrastructure developers where stakers have the option to restake their stake and contribute to the security of other infrastructures while earning native ETH rewards (i.e., pool security vs. fragmenting it). AVS stands for an &quot;actively validated service&quot;—software that uses its own set of nodes that are doing something that requires verification/validation (e.g., consensus, DA, TEE, etc.). The core <strong><em>EigenLayer</em></strong> protocol lets people take their ETH, restake it (liquid staking tokens), and then take <em>these</em> tokens and stake to EigenLayer nodes. An EigenLayer <em>operator</em> node can run arbitrary software—i.e., developers can create an AVS of their choice, such as a database, load balancer, log service, etc. An example of an AVS implementation <em>is</em> the <strong>EigenDA</strong>—i.e., EigenLayer is dogfooding their base protocol. <strong>Celestia</strong> Fraud proofs + DAS Fraud proofs via <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/celestiaorg/nmt">Namespaced Merkle Trees</a> &amp; Reed-Solomon erasure encoding Light clients verify headers through DAS—only sample <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/celestiaorg/CIPs/blob/main/cips/cip-4.md">blocks within a 30-day window</a> instead of sampling all blocks from genesis While performing DAS for a block header, every light node queries Celestia Nodes for a number of random data shares from the extended matrix and the corresponding Merkle proofs. If all the queries are successful, then the light node accepts the block header as valid (from a DA perspective). Assumptions: There is a minimum number of light nodes (depends on the block size) that are conducting data availability sampling for a given block size Light nodes assume they are connected to at least one honest full node and can receive fraud proofs for incorrectly erasure-coded blocks. Prune data after 30-day window They recommend L2 rollups that want to use Celestia to implement their own long-term storage plan: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.celestia.org/learn/retrievability#suggested-practices-for-rollups">here</a> <strong>Avail</strong> Validity proofs + DAS <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.availproject.org/docs/operate-a-node/run-a-light-client/Overview#light-client">KZG commitments + RS erasure encoding</a> Light clients access and sample data and ensure correct block encoding and provide data availability guarantees upon the finalization of new blocks Have 3 node types: <strong>Full Nodes</strong>: download and verify the correctness of blocks but do not partake in the consensus process. <strong>Validator Nodes</strong>: responsible for generating blocks, deciding on transaction inclusion, and maintaining the order; incentivized through consensus participation <strong>Light Clients</strong>: (DAS) query full nodes to check KZG polynomial openings against the commitments in the block header for each sampled cell Full nodes prune data after <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/availproject/avail-uncharted/blob/main/grants/RFPs/RFP-003.md#contextbackground">some cutoff period.</a> <strong>Arbitrum AnyTrust (variant of Arbitrum Nitro)</strong> Data Availability Committees that post <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.arbitrum.io/inside-anytrust#data-availability-certificates">Data Availability Certificates</a> (DACert) DACert is a proof that the block&apos;s data will be available from at least one honest Committee member until an expiration time. AnyTrust sounds like a volition—the sequencer either posts data blocks on the L1 chain as calldata/blobs, or it posts DACerts DAC members run <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.arbitrum.io/inside-anytrust#data-availability-servers">DA servers</a> with two endpoints: Sequencer API to submit data blocks for storage. REST API that allows data blocks to be fetched by hash. Related resources DA protocol deep dives: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://chronicle.castlecapital.vc/p/deepdive-data-availability">https://chronicle.castlecapital.vc/p/deepdive-data-availability</a> Avail DA comparison: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://blog.availproject.org/a-guide-to-selecting-the-right-data-availability-layer/">https://blog.availproject.org/a-guide-to-selecting-the-right-data-availability-layer/</a> Avail DA overview: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.availproject.org/docs/the-avail-trinity/avail-da">https://docs.availproject.org/docs/the-avail-trinity/avail-da</a> Avail RFP for storage: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/availproject/avail-uncharted/blob/main/grants/RFPs/RFP-003.md">https://github.com/availproject/avail-uncharted/blob/main/grants/RFPs/RFP-003.md</a> EigenLayer design: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.blog.eigenlayer.xyz/ycie/">https://www.blog.eigenlayer.xyz/ycie/</a> Ethereum DA basics: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://ethereum.org/en/developers/docs/data-availability">https://ethereum.org/en/developers/docs/data-availability</a> Starknet DA overview: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://book.starknet.io/ch03-01-03-data-availability.html#recreating-starknets-state">https://book.starknet.io/ch03-01-03-data-availability.html#recreating-starknets-state</a> Optimism DA overview: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://specs.optimism.io/experimental/plasma.html#da-storage">https://specs.optimism.io/experimental/plasma.html#da-storage</a> Vitalik’s DA &amp; Plasma overview: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://vitalik.eth.limo/general/2023/11/14/neoplasma.html">https://vitalik.eth.limo/general/2023/11/14/neoplasma.html</a> Arbitrum AnyTrust DA overview: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.arbitrum.io/inside-anytrust#data-availability-certificates">https://docs.arbitrum.io/inside-anytrust#data-availability-certificates</a> Celestia DA &amp; storage: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.celestia.org/learn/retrievability">https://docs.celestia.org/learn/retrievability</a> Fraud &amp; DA proofs via Celestia: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://arxiv.org/abs/1809.09044">https://arxiv.org/abs/1809.09044</a></h3>]]></content:encoded>
            <author>tableland-2@newsletter.paragraph.com (Tableland)</author>
            <enclosure url="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/a795d2ad2b5a3edd72ebd04419076f51c4205652250fa4a451b18deded8b4d96.png" length="0" type="image/png"/>
        </item>
        <item>
            <title><![CDATA[Tableland Studio Launch: Manage, Deploy, & Discover Onchain Datasets]]></title>
            <link>https://paragraph.com/@tableland-2/tableland-studio-launch-manage-deploy-discover-onchain-datasets</link>
            <guid>zP4kWp3BaiTweOndnMUI</guid>
            <pubDate>Fri, 15 Mar 2024 17:57:55 GMT</pubDate>
            <description><![CDATA[The Tableland Studio is officially launching! We opened up beta access to the Studio around 6 months ago, and since then, we’ve been working with developers to enhance its functionality.Studio landing page linking to your teams and associated projects.If you’re unfamiliar with the Studio, it’s a web application that lets you create collaborative teams, projects, and tables on the Tableland network. A primary benefit is a lower lift to getting started developing on Tableland. Additionally, any...]]></description>
            <content:encoded><![CDATA[<p>The <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://studio.tableland.xyz/">Tableland Studio</a> is officially launching! We opened up beta access to the Studio around 6 months ago, and since then, we’ve been working with developers to enhance its functionality.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/4806cdcf0e0644996f95801854acad119b9f39732cf904cd3c3d40d1bba0e685.png" alt="Studio landing page linking to your teams and associated projects." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Studio landing page linking to your teams and associated projects.</figcaption></figure><p>If you’re unfamiliar with the Studio, it’s a web application that lets you create collaborative teams, projects, and tables on the Tableland network. A primary benefit is a lower lift to getting started developing on Tableland. Additionally, any tables deployed on the Studio are fully interoperable with other Tableland clients like the SDK or smart contracts!</p><blockquote><p>Review the Studio documentation for how to get started with both the web app and CLI: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.tableland.xyz/studio/">here</a></p></blockquote><h2 id="h-studio-features" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Studio features</h2><h3 id="h-featured-and-trending-projects" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Featured &amp; trending projects</h3><p>The Studio landing page displays a few things to help anyone discover network trends:</p><ul><li><p><strong>Featured projects</strong>: A set of projects being actively worked on, helping newcomers see a handful of top use cases being built.</p></li><li><p><strong>Latest &amp; active tables</strong>: Filter across all supported chains to see the latest tables being created and which ones are the most active regarding write frequency.</p></li></ul><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/8a97e9b71f9443e57da0ddd89644b1c2b427b888f09a77b93d4af266f6d37d4a.png" alt="Studio landing page with featured projects, latest tables created, and most active tables." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Studio landing page with featured projects, latest tables created, and most active tables.</figcaption></figure><h3 id="h-create-teams-and-projects" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Create teams &amp; projects</h3><p>Define teams and projects that manage one or more tables. You can even invite others to collaborate on your team or project…or simply share a read-only link for viewing deployment information.</p><h3 id="h-create-and-deploy-on-any-supported-chain" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Create &amp; deploy on any supported chain</h3><p>The web app and CLI give you flexibility for how you want to manage and deploy your projects. Initially, you create an account, a team, and a project. Then, you can stage a table and then deploy it on a live network:</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/be55a14c5fc6a9c268e07953631dc49b8d3d29025dbee3e77d3abca71b737eb3.png" alt="Table creation and import flow." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Table creation and import flow.</figcaption></figure><p>If you have existing tables on the network (e.g., created via the SDK or smart contracts), you can import your tables into a project, too. The Studio CLI brings an additional set of features that aren’t part of the web app:</p><ul><li><p><strong>Queries</strong>: Open a shell to write or read table data.</p></li><li><p><strong>File writes</strong>: Take a local CSV file and write it to a table.</p></li><li><p><strong>Contexts</strong>: Easily switch between different teams and projects to execute the features noted above.</p></li></ul><h3 id="h-inspect-table-data-and-logs" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Inspect table data &amp; logs</h3><p>Once data is written to a table, you can inspect it as well! Note that <em>any</em> table on the network can be inspected in the Studio by simply appending it to the following URL: <code>https://studio.tableland.xyz/table/&lt;prefix_chainId_tableId&gt;</code>. For example, you <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://studio.tableland.xyz/table/pilot_sessions_1_7">can check out the Rigs</a> <code>pilot_sessions</code> table and filter through the columns:</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/15179e73772e2c0f8675b2757b3fafe62a0c68be71caea0805ecad687195c0a4.png" alt="Table inspection with table metadata, filtering, and pagination." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Table inspection with table metadata, filtering, and pagination.</figcaption></figure><p>One of the more useful features is the ability to inspect SQL logs. Namely, for any onchain operation, you can view the raw SQL log and any potential syntax or access control errors that might have occurred. The example below shows both a successful and unsuccessful log:</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/b851f24bc2510e56a479afcb506dda6e068c64da85f1d95ecf817a7061fc6258.png" alt="SQL logs for both successful and unsuccessful table materialization." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">SQL logs for both successful and unsuccessful table materialization.</figcaption></figure><h2 id="h-how-it-was-built" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">How it was built</h2><p>First and foremost, the Studio <em>uses</em> Tableland to store information for users, teams, projects, tables, environments, and more! Our docs site walks through each of these <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.tableland.xyz/playbooks/blueprints/studio-app-design">here</a>, including descriptions and links to the deployed table. <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://lerna.js.org/">Lerna</a> is used for monorepo package management across the following:</p><ul><li><p><code>@tableland/studio-api</code>: Defines tRPC API endpoints for writing and reading data with a Next.js router.</p></li><li><p><code>@tableland/studio-chains</code>: Simple internal package that stitches together wagmi and viem to configure the supported chains with RPC providers like Alchemy and Infura.</p></li><li><p><code>@tableland/studio-cli</code>: The Studio CLI tool.</p></li><li><p><code>@tableland/studio-client</code>: HTTP client set up to connect to the tRPC API server.</p></li><li><p><code>@tableland/studio-mail</code>: Encapsulates email sending in Studio (e.g., team invites).</p></li><li><p><code>nonce</code>: An internal package that manages a nonce with an ethers.js <code>NonceManager</code> implementation and Redis (needed for Vercel deployments since the Tableland team pays for transactions with a single wallet when writing data to the tables noted above).</p></li><li><p><code>@tableland/studio-store</code>: Studio-specific data store that contains the schemas of the data used in the Studio.</p></li><li><p><code>web</code>: The web application code, which includes:</p><ul><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://lerna.js.org/">Next.js</a> and <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://vercel.com/">Vercel</a>: Website framework and deployment.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tailwindcss.com/">Tailwind</a> and <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.radix-ui.com/">Radix</a>: Styling and components.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.ethers.org/">Ethers.js</a>, <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://wagmi.sh/">wagmi</a>, and <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://viem.sh/">viem</a>: Managing chain connections and interactions.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://orm.drizzle.team/">Drizzle</a>: Handles Tableland table migrations (with some customizations, described <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.tableland.xyz/sdk/walkthroughs/orm#drizzle-migrations">here</a>).</p></li></ul></li></ul><p>Overall, a ton went into building this site that uses web3 on the backend and web2 on the frontend to help ensure an optimal user experience.</p><h2 id="h-contribute-to-open-source" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Contribute to open source</h2><p>We kept the Studio code close sourced during the open beta period since it made it easier to iterate. We believe in open source, and with the official launch, we’re officially open sourcing the code! We’re <em>definitely</em> open to contributions, so if you want to tackle any open issues or have ideas for new features, feel free to check out the source code:</p><blockquote><p>Open sourced Tableland Studio repo: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/studio">here</a></p></blockquote><p>We’d also love to hear what you think, so <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tableland.xyz/discord">hop into our Discord</a>, too, and let us know if you have any ideas or questions. We’ll continue to build out the Studio and have plenty lined up in the future to make it even easier to build on Tableland!</p>]]></content:encoded>
            <author>tableland-2@newsletter.paragraph.com (Tableland)</author>
            <enclosure url="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/41d70ab5ac0216c822175e480375bd95c043d85a2012d6769b3b9e279ec922a8.png" length="0" type="image/png"/>
        </item>
        <item>
            <title><![CDATA[Blobs, transient storage, & data availability: Tableland just became your cheap onchain SQL database]]></title>
            <link>https://paragraph.com/@tableland-2/blobs-transient-storage-data-availability-tableland-just-became-your-cheap-onchain-sql-database</link>
            <guid>YKFHqChVrQxNOJPwd0pJ</guid>
            <pubDate>Wed, 13 Mar 2024 20:07:04 GMT</pubDate>
            <description><![CDATA[Dencun goes liveToday, the Dencun upgrade went live! This brings EIP-4844 (and others) to Ethereum mainnet and the long-awaited Proto-Danksharding aka “blob” support. Blobs are a new transient storage option that expires after ~2 weeks. Up to this point, L2 chains would post transaction data from the L2 back to Ethereum L1 as calldata at some interval to ensure the data is made available onchain. The Dencun changes how L2s interact with the L1; instead of posting rolled-up data as calldata, i...]]></description>
            <content:encoded><![CDATA[<h2 id="h-dencun-goes-live" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Dencun goes live</h2><p>Today, the Dencun upgrade went live! This brings <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.eip4844.com/">EIP-4844</a> (and others) to Ethereum mainnet and the long-awaited Proto-Danksharding aka “blob” support. Blobs are a new transient storage option that <em>expires</em> after ~2 weeks. Up to this point, L2 chains would post transaction data from the L2 back to Ethereum L1 as calldata at some interval to ensure the data is made available onchain. The Dencun changes how L2s interact with the L1; instead of posting rolled-up data as calldata, it’s now posted as a blob.</p><p>Since blobs are transient, they are <em>significantly</em> cheaper than calldata—around 10 times less. For example, <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/Rjected/status/1767928246614499812/photo/1">someone posted the entire Bee Movie script</a> to Ethereum L1 and only paid <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://etherscan.io/tx/0x110d6d8888ced3615a7ca07d91acd9eebc4e61f669d83fd2e7f42de1ac7d39a3">around $14 for it in transaction fees</a>. Quite a significant cost reduction!</p><h2 id="h-transient-storage-and-l2s" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Transient storage &amp; L2s</h2><p>Let’s take a step back and think about Tableland before the Dencun upgrade. The basic flow is the following:</p><ul><li><p>A user or contract creates a table and has onchain ownership of it (ERC721 token), and the data is emitted as an event so that offchain Tableland validator nodes can index the data.</p></li><li><p>SQL writes are made onchain to the table, and the SQL write data is passed <em>only</em> as calldata to a smart contract method—and the event is emitted for indexing purposes.</p></li><li><p>This flow can occur on Ethereum L1 or any of the L2s that Tableland supports.</p></li></ul><p>Now that Dencun is live, the flow is the same; however, since L2s are using blobs, that means the data L2s post onchain <em>will</em> expire. If it’s not indexed, the L2 data will go away…forever! The benefit is that L2 transactions get cheaper, thus, writing table data <em>also</em> gets cheaper.</p><h2 id="h-the-need-for-data-availability" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">The need for data availability</h2><p>Tableland naturally plays a role in making sure that any SQL data has been materialized and properly indexed. Whenever you write SQL to an L2 Tableland table, all event logs are stored in the validator’s local SQLite database. You can take any transaction hash of a Tableland SQL event and get the receipt of what happened for that transaction. Any successfully materialized SQL event will mutate the validator’s database—meaning once the L2 blob expires on the L1, the data is <em>still</em> available in Tableland.</p><p>Additionally, the indexing portion will become critical in the future once blobs have greater support (they’re currently only possible in low-level interactions or Solidity assembly. Let’s say you have a smart contract that’s using blob storage. In ~2 weeks, that data is gone. There’s no way to ever retrieve it <em>unless</em> someone has indexed the data.</p><p>Fundamentally, the idea behind blobs is the approach we took when designing Tableland. Onchain storage is too expensive, and offchain storage is, well, <em>only</em> offchain. The optimal middle ground was to use calldata so that the data is still executed and “stored” onchain, but the actual availability and functionality are available offchain at the Tableland validator/indexing layer. A dual-pronged approach.</p><h2 id="h-whats-next" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">What’s next?</h2><p>If you’re using any supported L2 (Arbitrum, Optimism, or Polygon), you can take advantage of reduced write costs today!</p><p>Note that Tableland is still using calldata on its Ethereum L1 contracts, but over time, you could imagine how the protocol could adopt blob support for table writes to offer developers even more cost savings. The protocol’s native indexing feature would guarantee that data is available even after the transient storage period has ended!</p>]]></content:encoded>
            <author>tableland-2@newsletter.paragraph.com (Tableland)</author>
            <enclosure url="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/78c5ee9f412699c55a3b55909dd6cf83ba65c8b3730080154ce074023ea8b6f4.png" length="0" type="image/png"/>
        </item>
        <item>
            <title><![CDATA[WeatherXM + Textile: DePIN Data Storage, Retrieval, & Rewards Computation]]></title>
            <link>https://paragraph.com/@tableland-2/weatherxm-textile-depin-data-storage-retrieval-rewards-computation</link>
            <guid>0LokSJgKQInsKhpUIH64</guid>
            <pubDate>Mon, 11 Mar 2024 20:53:49 GMT</pubDate>
            <description><![CDATA[TL;DRWeatherXM used Textile to do the following:Bring open data availability of station/device data and rewards calculations to help progress toward fully decentralized infrastructure and bring transparency to the dataset.Enable verifiable data such that the data is signed at the source, ensuring all data is cryptographically provable with access controls on data mutations.A decentralized framework for computation, such as QoD, providing the WeatherXM ecosystem a way to build on top of the da...]]></description>
            <content:encoded><![CDATA[<h2 id="h-tldr" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">TL;DR</h2><p>WeatherXM used Textile to do the following:</p><ul><li><p>Bring open data availability of station/device data and rewards calculations to help progress toward fully decentralized infrastructure and bring transparency to the dataset.</p></li><li><p>Enable verifiable data such that the data is signed at the source, ensuring all data is cryptographically provable with access controls on data mutations.</p></li><li><p>A decentralized framework for computation, such as <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/weatherxm-network/qod">QoD</a>, providing the WeatherXM ecosystem a way to build on top of the data.</p></li></ul><blockquote><p>You can view available “events” (data pushes) <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://basin.tableland.xyz/vaults/wxm.weather_data_dev/events?limit=5">here</a> and extract the device data at the <code>https://basin.tableland.xyz/events/&lt;event_cid&gt;</code> endpoint for further analysis, as described below.</p></blockquote><h2 id="h-what-is-it" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">What is it?</h2><p>Textile and WeatherXM are working together to design a solution for DePIN data availability (“DA”), storage, and retrieval. If you’re unfamiliar with WeatherXM, it is a network that&apos;s powered by community-owned devices, purpose-built by WeatherXM for weather data collection. As part of the network, stations/devices throughout the globe are incentivized to share data.</p><p>WeatherXM’s goal is to make network data more accessible so developers can build on top of it. The station/device data are replicated to Basin for decentralized storage, allowing for computation over that data. The data flows through Basin, which can support cache with TTL, and is replicated to Filecoin for long-term storage guarantees.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/613faab8a72596cf95d8b3bb0145a6e787ff39923677f0f8a6ac2f1c4568e4a7.png" alt="Flow from WeatherXM stations (devices) to internal systems to Textile Basin...to downstream compute operations that anyone can run over the data." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Flow from WeatherXM stations (devices) to internal systems to Textile Basin...to downstream compute operations that anyone can run over the data.</figcaption></figure><p>Basin is designed to be flexible and work with raw file uploads (e.g., parquet) or even database streaming (e.g., Postgres). Data is replicated to the network in a “vault” and then can be retrieved by the vault’s CIDs for further processing.</p><p>There are two possible data availability options for a vault—hot or cold storage:</p><ul><li><p><strong>Hot layer</strong>: Cache with time-to-live (TTL) where the data is made available for the defined timeline and immediately retrievable (i.e., no need to wait for long propagation or finality times).</p></li><li><p><strong>Cold layer</strong>: Persistence on Filecoin where all data is flushed for long-term retrieval.</p></li></ul><p>This makes it possible to run computation over the open data where both the inputs—and, potentially, outputs—are highly available in the short term and also retrievable after cold storage flushing (e.g., dispute resolution).</p><h2 id="h-how-does-weatherxm-use-the-basin-data" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">How does WeatherXM use the Basin data?</h2><p>WeatherXM device data is made available for data consumers through hot + cold storage replication, and this data is used today in a dashboard available at <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://index.weatherxm.network/">https://index.weatherxm.network</a>. You’ll notice two tabs on that page:</p><ul><li><p><strong>Weather Data</strong>: Device data located in the Textile vault <code>wxm.weather_data_dev</code></p></li><li><p><strong>Rewards</strong>: Calculations over the device data, located in the Textile vault <code>wxm.rewards_merkle_trees_dev</code></p></li></ul><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/9fd0cdd6daee5f31d70e5e7b70b5e40d1bf6c2e6ee59b790440ceb65c8e75e9a.png" alt="WeatherXM&apos;s dashboard displays the CIDs created when data is pushed to Textile Basin and provides a link to the Basin HTTP API." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">WeatherXM&apos;s dashboard displays the CIDs created when data is pushed to Textile Basin and provides a link to the Basin HTTP API.</figcaption></figure><p>We also built a demonstration of how developers can use the DePIN network’s data in the downstream compute and visualization pipeline. One of WeatherXM’s goals is to progressively decentralize their stack, and Basin is a key component to making that possible. We wanted to provide a way for developers to see exactly how they can build on top of the public dataset and add value to the WeatherXM ecosystem.</p><blockquote><p><em>We built a demonstration about how you can leverage WeatherXM and run queries over the data, which we’ll walk through below. You can review the</em> <code>Data.md</code> for the final visualizations that plot WeatherXM network data over historical trends and across the globe.<strong><em>Check it out here</em></strong>*: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/textileio/demo-basin-wxm-query">https://github.com/textileio/demo-basin-wxm-query</a>*</p></blockquote><h2 id="h-how-does-basin-work" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">How does Basin work?</h2><h3 id="h-background" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Background</h3><p>The setup process for WeatherXM involved:</p><ul><li><p>Creating a vault with the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/basin-cli">Basin CLI</a>, which is an onchain interaction signed by a private key.</p></li><li><p>Aggregating device data that is processed by internal systems for <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://blog.weatherxm.com/weatherxm-qod-quality-of-data-simplified-b5a42aee7367">Quality of Data</a> purposes.</p></li><li><p>Replicating internal data to the Basin DA layer on a consistent schedule—i.e., parquet files containing device data are written to the vault and signed by a private key.</p></li></ul><p>Once that data is available, anyone with knowledge of the unique vault identifier or vault owner’s address can extract and process the data. It’s openly accessible on the Basin network and through the cold layer. In our demo described below, we built a program that consists of:</p><ul><li><p>A Python script to fetch remote parquet files written to the vault, filtered over a time range, and identified at an IPFS CID.</p></li><li><p>A DuckDB in-memory SQL database for loading parquet files and executing queries.</p></li><li><p>Writing the query results to a CSV and markdown files and producing data visualization with geopandas (e.g., precipitation rate per geography).</p></li><li><p>Running the steps above weekly with GitHub Actions.</p></li></ul><p>You could imagine more complex flows with Docker images or compute networks like <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.bacalhau.org/">Bacalhau</a>, but the idea was to keep things lightweight to showcase how to interact with open, verifiable data.</p><h3 id="h-viewing-and-retrieving-weatherxm-data" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Viewing &amp; retrieving WeatherXM data</h3><p>If you want to understand the basics behind Basin, you can start by installing the CLI on your machine. Make sure the path to Go binaries is in your shell’s <code>PATH</code> variable (e.g., include <code>$(go env GOPATH)/bin</code>).</p><pre data-type="codeBlock" text="go install github.com/tablelandnetwork/basin-cli/cmd/vaults@latest
"><code>go install github.com/tablelandnetwork<span class="hljs-operator">/</span>basin<span class="hljs-operator">-</span>cli<span class="hljs-operator">/</span>cmd<span class="hljs-operator">/</span>vaults@latest
</code></pre><p>Then, you can take a peek at what the structure and data within a vault look like. For example, list all vaults created by a specific account/address:</p><pre data-type="codeBlock" text="vaults list --account 0x75c9eCb5309d068f72f545F3fbc7cCD246016fE0
"><code>vaults list <span class="hljs-operator">-</span><span class="hljs-operator">-</span>account <span class="hljs-number">0x75c9eCb5309d068f72f545F3fbc7cCD246016fE0</span>
</code></pre><p>Or, list out the events associated with a vault:</p><pre data-type="codeBlock" text="vaults events --vault wxm.weather_data_dev
"><code>vaults events <span class="hljs-operator">-</span><span class="hljs-operator">-</span>vault wxm.weather_data_dev
</code></pre><p>Which provides the “events” (i.e., vault mutations &amp; CIDs) for the vault. Note the <code>cache_expiry</code> field shows when the highly available data will expire from the cache.</p><pre data-type="codeBlock" text="[
  {
    &quot;cid&quot;: &quot;bafybeicizws2qymmcn657b3qa7rm2w2lz4bdumxc6b66jw4ac4fsrhmsxm&quot;,
    &quot;timestamp&quot;: 1709078400,
    &quot;is_archived&quot;: false,
    &quot;cache_expiry&quot;: &quot;2024-05-16T20:56:47.379695&quot;
  },
  {
    &quot;cid&quot;: &quot;bafybeigppam7v3uwarke5iulytgptdj6hzjta5kye7nljbvemzje6r3xma&quot;,
    &quot;timestamp&quot;: 1709596800,
    &quot;is_archived&quot;: false,
    &quot;cache_expiry&quot;: &quot;2024-05-16T19:49:50.277204&quot;
  }
]
"><code><span class="hljs-punctuation">[</span>
  <span class="hljs-punctuation">{</span>
    <span class="hljs-attr">"cid"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"bafybeicizws2qymmcn657b3qa7rm2w2lz4bdumxc6b66jw4ac4fsrhmsxm"</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"timestamp"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">1709078400</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"is_archived"</span><span class="hljs-punctuation">:</span> <span class="hljs-literal"><span class="hljs-keyword">false</span></span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"cache_expiry"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"2024-05-16T20:56:47.379695"</span>
  <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span>
  <span class="hljs-punctuation">{</span>
    <span class="hljs-attr">"cid"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"bafybeigppam7v3uwarke5iulytgptdj6hzjta5kye7nljbvemzje6r3xma"</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"timestamp"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">1709596800</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"is_archived"</span><span class="hljs-punctuation">:</span> <span class="hljs-literal"><span class="hljs-keyword">false</span></span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"cache_expiry"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"2024-05-16T19:49:50.277204"</span>
  <span class="hljs-punctuation">}</span>
<span class="hljs-punctuation">]</span>
</code></pre><p>This period is defined upon vault creation such that all files written to the vault follow the same expiry logic. Note the data is flushed to cold storage and still available after the cache expiration. The CIDs can then be retrieved and processed locally:</p><pre data-type="codeBlock" text="vaults retrieve --output &lt;filename&gt; &lt;event_cid&gt;
"><code>vaults retrieve <span class="hljs-operator">-</span><span class="hljs-operator">-</span>output <span class="hljs-operator">&#x3C;</span>filename<span class="hljs-operator">></span> <span class="hljs-operator">&#x3C;</span>event_cid<span class="hljs-operator">></span>
</code></pre><h2 id="h-demo-how-to-use-weatherxm-data" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Demo: How to use WeatherXM data</h2><p>Now that we understand how Basin works, we can review some of the demo’s features since it builds on top of the WeatherXM vault data. The first step is to get all of the vaults for a specific address and then retrieve all of the events and CIDs for the vaults. This can be done programmatically with the Basin HTTP API (where <code>{variable}</code> would be replaced by actual string values—e.g., <code>account=0x75c9eCb5309d068f72f545F3fbc7cCD246016fE0</code>):</p><ul><li><p>Get vaults for address: <code>curl https://basin.tableland.xyz/vaults?account={address}</code></p></li><li><p>Get events for vault: <code>curl https://basin.tableland.xyz/vaults/{vault}/events</code></p></li><li><p>Download the raw data at an event: <code>curl https://basin.tableland.xyz/events/{event_cid} -o data.parquet</code></p></li></ul><p>Let’s presume the files have been downloaded at each event for WeatherXM’s vault into some <code>data_dir</code> directory. We then load these parquet files into a DuckDB in-memory instance.</p><pre data-type="codeBlock" text="from pathlib import Path

from duckdb import connect

def create_database(data_dir):
    db = connect()
    files = Path(data_dir) / &quot;*.parquet&quot;  # Read all parquet files in data directory
    db.execute(f&quot;CREATE VIEW xm_data AS SELECT * FROM read_parquet(&apos;{files}&apos;);&quot;)

    return db
"><code><span class="hljs-keyword">from</span> pathlib <span class="hljs-keyword">import</span> Path

<span class="hljs-keyword">from</span> duckdb <span class="hljs-keyword">import</span> connect

<span class="hljs-keyword">def</span> <span class="hljs-title function_">create_database</span>(<span class="hljs-params">data_dir</span>):
    db = connect()
    files = Path(data_dir) / <span class="hljs-string">"*.parquet"</span>  <span class="hljs-comment"># Read all parquet files in data directory</span>
    db.execute(<span class="hljs-string">f"CREATE VIEW xm_data AS SELECT * FROM read_parquet('<span class="hljs-subst">{files}</span>');"</span>)

    <span class="hljs-keyword">return</span> db
</code></pre><p>We can now run queries over the WeatherXM device data! This will calculate average or aggregate metrics for temperature, precipitation, wind, etc., and create a DataFrame for subsequent processing.</p><pre data-type="codeBlock" text="def execute_queries(db, start, end):
    # Set up columns for average calculations
    columns = [
        &quot;temperature&quot;,
        &quot;humidity&quot;,
        &quot;precipitation_accumulated&quot;,
        &quot;wind_speed&quot;,
        &quot;wind_gust&quot;,
        &quot;wind_direction&quot;,
        &quot;illuminance&quot;,
        &quot;solar_irradiance&quot;,
        &quot;fo_uv&quot;,
        &quot;uv_index&quot;,
        &quot;precipitation_rate&quot;,
        &quot;pressure&quot;,
    ]

    # Set up all query parts
    avg_selection = [f&quot;avg({col}) AS {col}&quot; for col in columns]
    avg_calculations = &quot;,&quot;.join(avg_selection)
    query_parts = [
        &quot;SELECT min(timestamp) AS range_start, max(timestamp) AS range_end,&quot;,
        &quot;COUNT(DISTINCT device_id) AS number_of_devices,&quot;,
        &quot;mode(cell_id) AS cell_id_mode,&quot;,
        &quot;sum(precipitation_accumulated) AS total_precipitation,&quot;,
        avg_calculations,
    ]

    # Add WHERE clause for time filtering, if applicable
    where_clause = &quot;&quot;
    if start is not None and end is not None:
        where_clause = f&quot;WHERE timestamp &gt;= {start} AND timestamp &lt;= {end}&quot;
    elif start is not None:
        where_clause = f&quot;WHERE timestamp &gt;= {start}&quot;
    elif end is not None:
        where_clause = f&quot;WHERE timestamp &lt;= {end}&quot;

    # Combine all parts into one query and execute
    query = &quot; &quot;.join(query_parts) + &quot; FROM xm_data&quot; + f&quot; {where_clause}&quot;
    try:
        result = db.execute(query).pl()  # Create a polars DataFrame
        return result
    except Exception as e:
        print(f&quot;Error executing DuckDB aggregate queries: {e}&quot;)
                raise
"><code><span class="hljs-keyword">def</span> <span class="hljs-title function_">execute_queries</span>(<span class="hljs-params">db, start, end</span>):
    <span class="hljs-comment"># Set up columns for average calculations</span>
    columns = [
        <span class="hljs-string">"temperature"</span>,
        <span class="hljs-string">"humidity"</span>,
        <span class="hljs-string">"precipitation_accumulated"</span>,
        <span class="hljs-string">"wind_speed"</span>,
        <span class="hljs-string">"wind_gust"</span>,
        <span class="hljs-string">"wind_direction"</span>,
        <span class="hljs-string">"illuminance"</span>,
        <span class="hljs-string">"solar_irradiance"</span>,
        <span class="hljs-string">"fo_uv"</span>,
        <span class="hljs-string">"uv_index"</span>,
        <span class="hljs-string">"precipitation_rate"</span>,
        <span class="hljs-string">"pressure"</span>,
    ]

    <span class="hljs-comment"># Set up all query parts</span>
    avg_selection = [<span class="hljs-string">f"avg(<span class="hljs-subst">{col}</span>) AS <span class="hljs-subst">{col}</span>"</span> <span class="hljs-keyword">for</span> col <span class="hljs-keyword">in</span> columns]
    avg_calculations = <span class="hljs-string">","</span>.join(avg_selection)
    query_parts = [
        <span class="hljs-string">"SELECT min(timestamp) AS range_start, max(timestamp) AS range_end,"</span>,
        <span class="hljs-string">"COUNT(DISTINCT device_id) AS number_of_devices,"</span>,
        <span class="hljs-string">"mode(cell_id) AS cell_id_mode,"</span>,
        <span class="hljs-string">"sum(precipitation_accumulated) AS total_precipitation,"</span>,
        avg_calculations,
    ]

    <span class="hljs-comment"># Add WHERE clause for time filtering, if applicable</span>
    where_clause = <span class="hljs-string">""</span>
    <span class="hljs-keyword">if</span> start <span class="hljs-keyword">is</span> <span class="hljs-keyword">not</span> <span class="hljs-literal">None</span> <span class="hljs-keyword">and</span> end <span class="hljs-keyword">is</span> <span class="hljs-keyword">not</span> <span class="hljs-literal">None</span>:
        where_clause = <span class="hljs-string">f"WHERE timestamp >= <span class="hljs-subst">{start}</span> AND timestamp &#x3C;= <span class="hljs-subst">{end}</span>"</span>
    <span class="hljs-keyword">elif</span> start <span class="hljs-keyword">is</span> <span class="hljs-keyword">not</span> <span class="hljs-literal">None</span>:
        where_clause = <span class="hljs-string">f"WHERE timestamp >= <span class="hljs-subst">{start}</span>"</span>
    <span class="hljs-keyword">elif</span> end <span class="hljs-keyword">is</span> <span class="hljs-keyword">not</span> <span class="hljs-literal">None</span>:
        where_clause = <span class="hljs-string">f"WHERE timestamp &#x3C;= <span class="hljs-subst">{end}</span>"</span>

    <span class="hljs-comment"># Combine all parts into one query and execute</span>
    query = <span class="hljs-string">" "</span>.join(query_parts) + <span class="hljs-string">" FROM xm_data"</span> + <span class="hljs-string">f" <span class="hljs-subst">{where_clause}</span>"</span>
    <span class="hljs-keyword">try</span>:
        result = db.execute(query).pl()  <span class="hljs-comment"># Create a polars DataFrame</span>
        <span class="hljs-keyword">return</span> result
    <span class="hljs-keyword">except</span> Exception <span class="hljs-keyword">as</span> e:
        <span class="hljs-built_in">print</span>(<span class="hljs-string">f"Error executing DuckDB aggregate queries: <span class="hljs-subst">{e}</span>"</span>)
                <span class="hljs-keyword">raise</span>
</code></pre><p>Lastly, we can use geopandas to create visualizations from the returned DataFrame. In the example, there are a few rough bounding boxes (“bbox”) for each major continent with lat/long bounds:</p><pre data-type="codeBlock" text="bboxes = {
    &quot;north_america&quot;: (14, 72, -172, -52),
    &quot;south_america&quot;: (-55, 12, -85, -34),
    &quot;europe&quot;: (35, 72, -13, 60),
    &quot;africa&quot;: (-35, 38, -18, 55),
    &quot;asia&quot;: (-11, 81, 25, 179),
    &quot;australia&quot;: (-48, -6, 108, 178),
}
"><code>bboxes = {
    <span class="hljs-string">"north_america"</span>: (14, 72, -172, -52),
    <span class="hljs-string">"south_america"</span>: (-55, 12, -85, -34),
    <span class="hljs-string">"europe"</span>: (35, 72, -13, 60),
    <span class="hljs-string">"africa"</span>: (-35, 38, -18, 55),
    <span class="hljs-string">"asia"</span>: (-11, 81, 25, 179),
    <span class="hljs-string">"australia"</span>: (-48, -6, 108, 178),
}
</code></pre><p>For example, this calculates the total precipitation accumulated by <code>cell_id</code> within a bounding box:</p><pre data-type="codeBlock" text="def query_bbox(db, bbox, start, end):
    # DuckDB query to select and aggregate data within the bounding box
    query = f&quot;&quot;&quot;
    SELECT cell_id, SUM(precipitation_accumulated) as total_precipitation, AVG(lat) as lat, AVG(lon) as lon
    FROM xm_data
    WHERE lat BETWEEN {bbox[0]} AND {bbox[1]} AND lon BETWEEN {bbox[2]} AND {bbox[3]}
    &quot;&quot;&quot;

    # Add WHERE clause for time filtering, if applicable
    where_clause = &quot;&quot;
    if start is not None and end is not None:
        where_clause = f&quot;AND timestamp &gt;= {start} AND timestamp &lt;= {end}&quot;
    elif start is not None:
        where_clause = f&quot;AND timestamp &gt;= {start}&quot;
    elif end is not None:
        where_clause = f&quot;AND timestamp &lt;= {end}&quot;

    # Combine all parts into one query and execute
    query = query + f&quot; {where_clause}&quot; + &quot; GROUP BY cell_id&quot;
    try:
        result = db.execute(query).df()  # Create a pandas Dataframe
        return result
    except Exception as e:
                print(f&quot;Error executing DuckDB bbox query: {e}&quot;)
                raise
"><code><span class="hljs-keyword">def</span> <span class="hljs-title function_">query_bbox</span>(<span class="hljs-params">db, bbox, start, end</span>):
    <span class="hljs-comment"># DuckDB query to select and aggregate data within the bounding box</span>
    query = <span class="hljs-string">f"""
    SELECT cell_id, SUM(precipitation_accumulated) as total_precipitation, AVG(lat) as lat, AVG(lon) as lon
    FROM xm_data
    WHERE lat BETWEEN <span class="hljs-subst">{bbox[<span class="hljs-number">0</span>]}</span> AND <span class="hljs-subst">{bbox[<span class="hljs-number">1</span>]}</span> AND lon BETWEEN <span class="hljs-subst">{bbox[<span class="hljs-number">2</span>]}</span> AND <span class="hljs-subst">{bbox[<span class="hljs-number">3</span>]}</span>
    """</span>

    <span class="hljs-comment"># Add WHERE clause for time filtering, if applicable</span>
    where_clause = <span class="hljs-string">""</span>
    <span class="hljs-keyword">if</span> start <span class="hljs-keyword">is</span> <span class="hljs-keyword">not</span> <span class="hljs-literal">None</span> <span class="hljs-keyword">and</span> end <span class="hljs-keyword">is</span> <span class="hljs-keyword">not</span> <span class="hljs-literal">None</span>:
        where_clause = <span class="hljs-string">f"AND timestamp >= <span class="hljs-subst">{start}</span> AND timestamp &#x3C;= <span class="hljs-subst">{end}</span>"</span>
    <span class="hljs-keyword">elif</span> start <span class="hljs-keyword">is</span> <span class="hljs-keyword">not</span> <span class="hljs-literal">None</span>:
        where_clause = <span class="hljs-string">f"AND timestamp >= <span class="hljs-subst">{start}</span>"</span>
    <span class="hljs-keyword">elif</span> end <span class="hljs-keyword">is</span> <span class="hljs-keyword">not</span> <span class="hljs-literal">None</span>:
        where_clause = <span class="hljs-string">f"AND timestamp &#x3C;= <span class="hljs-subst">{end}</span>"</span>

    <span class="hljs-comment"># Combine all parts into one query and execute</span>
    query = query + <span class="hljs-string">f" <span class="hljs-subst">{where_clause}</span>"</span> + <span class="hljs-string">" GROUP BY cell_id"</span>
    <span class="hljs-keyword">try</span>:
        result = db.execute(query).df()  <span class="hljs-comment"># Create a pandas Dataframe</span>
        <span class="hljs-keyword">return</span> result
    <span class="hljs-keyword">except</span> Exception <span class="hljs-keyword">as</span> e:
                <span class="hljs-built_in">print</span>(<span class="hljs-string">f"Error executing DuckDB bbox query: <span class="hljs-subst">{e}</span>"</span>)
                <span class="hljs-keyword">raise</span>
</code></pre><p>By taking the pandas DataFrame and passing it to geopanda’s <code>GeoDataFrame</code> method, you can create maps that plot data at lat/long coordinates, such as the one described above for the North American bbox. The history CSV file also plots information week-over-week—below is what the device trends look like (hint: the number of devices is increasing!):</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/31413a47d5db6fd126a4651d058de2b7f297c0d5e3af1717f8c5c4d4bbf228fa.png" alt="A simple demo of computing the number of devices and precipitation plot over the WeatherXM vault." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">A simple demo of computing the number of devices and precipitation plot over the WeatherXM vault.</figcaption></figure><h2 id="h-how-can-someone-use-it" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">How can someone use it?</h2><p>If you’d like to develop with or alter the demo code, start by cloning the repo and going through the setup steps described in the README.</p><pre data-type="codeBlock" text="git clone https://github.com/textileio/demo-basin-wxm-query
"><code>git <span class="hljs-built_in">clone</span> https://github.com/textileio/demo-basin-wxm-query
</code></pre><p>Then, you can use the <code>make run</code> command to fetch events and compute queries over the data for the WeatherXM vault. You can also choose to pass <code>start</code> and <code>end</code> parameters to the <code>make run</code> command, which specifies timestamp ranges for data extraction.</p><p>If you want to use the WeatherXM data, the Basin HTTP APIs described in the sections above should provide all that you need. Namely, you can fetch the events and download the parquet files locally—or event load remote data into DuckDB from an IPFS gateway!</p><h2 id="h-next-steps" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Next steps</h2><p>This is just the first post in a series of Basin data demonstrations. We’ll continue to put out additional walkthroughs in long-form posts, but if you’d like to stay up to date with what we’re building on a week-by-week basis, check out the Weeknotes newsletter on Substack!</p><blockquote><p><strong>Follow us on Substack: </strong><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tableland.substack.com/"><strong>https://tableland.substack.com/</strong></a></p></blockquote>]]></content:encoded>
            <author>tableland-2@newsletter.paragraph.com (Tableland)</author>
            <enclosure url="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/e6ae8cfbd783941a2be35c96c5664a7b74eed00b9928559cdb8b4bcb8a85d944.png" length="0" type="image/png"/>
        </item>
        <item>
            <title><![CDATA[Synthetic foundations: the future of data in web3]]></title>
            <link>https://paragraph.com/@tableland-2/synthetic-foundations-the-future-of-data-in-web3</link>
            <guid>KrcIRpk6xJJGJzsIXgoz</guid>
            <pubDate>Fri, 23 Feb 2024 15:59:54 GMT</pubDate>
            <description><![CDATA[In 2012, Chris Dixon wrote that just as oil powered the Industrial Age, data is the fuel of the Information Age. I believe that by the end of 2024, we will all realize that synthetic data is the clean energy.Understanding synthetic dataSynthetic data is created by first training a model on real data to learn its distribution. Then, the model&apos;s generative capabilities are used to produce new, synthetic data. This synthetic data can then train other models to learn new functions or abiliti...]]></description>
            <content:encoded><![CDATA[<p>In 2012, Chris Dixon wrote that <em>just as oil powered the Industrial Age, data is the fuel of the Information Age</em>. I believe that by the end of 2024, we will all realize that synthetic data is the clean energy.</p><h1 id="h-understanding-synthetic-data" class="text-4xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0"><strong>Understanding synthetic data</strong></h1><p>Synthetic data is created by first training a model on real data to learn its distribution. Then, the model&apos;s generative capabilities are used to produce new, synthetic data. This synthetic data can then train other models to learn new functions or abilities.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/81dc301f1112eb3dc124b7a9a4391dcedfea60a9dc7797f03a3d41fa08e66a89.png" alt="source: https://tinyurl.com/yymtp4hh" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">source: https://tinyurl.com/yymtp4hh</figcaption></figure><blockquote><p>In shadowed veil &apos;neath star-crossed sky, our tale begins with whisper&apos;d sigh. <br>GPT-4, 2024</p></blockquote><p>We&apos;ve all seen the impressive capabilities of Large Language Models (LLMs). For instance, GPT-4 can effortlessly produce new lines of Shakespearean dialogue, despite being trained on less than 1,000,000 original words from Shakespeare’s works. Synthetic data utilizes these capabilities to generate new data that imitates the characteristics of real data. Synthetic data will unlock a scale and breadth of data that may be impossible to collect in the real-world.</p><p>Synthetic data is more than just &apos;fake&apos; data; it is crafted to mimic the complexity of real-world data. However, since the data is synthetic, it overcomes the ethical and logistical limitations associated with traditional data collection and sharing methods. Moreover, it can be produced faster, more complete (pre-labeled, without sample biases, etc), and cheaper than real data. Synthetic data promises a future of accelerated innovation, allowing us to train enhanced and entirely new models with data that would otherwise be unattainable.</p><h2 id="h-synthetic-data-in-the-real-world" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Synthetic data in the real-world</h2><blockquote><p>Generally, the arc of technology has been about reducing randomness and increasing our control over the world. At some point in the next century, we are going to have the most randomness ever injected into the system. <br><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://blog.samaltman.com/machine-intelligence-part-2">Sam Altman, 2015</a></p></blockquote><p>In 2024, synthetic data is set to become more accessible and broadly applicable across many fields. The use-cases for synthetic data are starting to proliferate, from text-to-image pipelines for creating domain-specific image libraries, to generating datasets for <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://arxiv.org/abs/2109.00591">hate speech detection</a>, and <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://arxiv.org/abs/2207.14502">formulating coding challenges for model training</a>.</p><p>Let’s look at some examples of how synthetic data is being used in the real-world.</p><ul><li><p>Medical researchers are using synthetic data, like images made by GANs (Generative Adversarial Networks), to help detect diseases without using real patient data. This approach lets them train AI models to spot conditions such as eye diseases from fake but realistic images, keeping patient information private. This technology speeds up research while sticking to privacy laws, pushing medical advancements forward. See here for discussion of implications and concerns <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.nature.com/articles/s41551-021-00751-8">https://www.nature.com/articles/s41551-021-00751-8</a>.</p></li></ul><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/1d4056d6952ce95d84ed07f3310ff22524399aea585bfa32f3992037866b5ce0.png" alt="" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="hide-figcaption"></figcaption></figure><ul><li><p>The generation of over 100,000 synthetic faces to enhance facial recognition technologies. The project then uses the synthetic faces to generate synthetic eye motion, aiming to train even better eye-tracking models, as demonstrated in their presentation. In short, create fake faces, create fake eye movement, create all the labels for both datasets automatically. Then train models to do facial labeling and eye tracking on real world faces. Important to note is the scale and quality of automated data labeling generated for both the faces and the eye-tracking examples. <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.youtube.com/watch?v=wlOMpQe8luQ">Watch full video here</a>.</p></li></ul><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/a028cebc6115cb045293d5aabde97fc54cde67352158875e5c22a84542422dda.gif" alt="" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="hide-figcaption"></figcaption></figure><ul><li><p>NVIDIA is building the Omniverse, a platform that facilitates the creation of extensive synthetic datasets based on a simulated real-world environment with real-world physics. The vast amounts of synthetic data being produced, and the nuance in potential parameters generated is wild. Video link below:</p></li></ul><div data-type="youtube" videoId="IHbbOnN1bAE">
      <div class="youtube-player" data-id="IHbbOnN1bAE" style="background-image: url('https://i.ytimg.com/vi/IHbbOnN1bAE/hqdefault.jpg'); background-size: cover; background-position: center">
        <a href="https://www.youtube.com/watch?v=IHbbOnN1bAE">
          <img src="{{DOMAIN}}/editor/youtube/play.png" class="play"/>
        </a>
      </div></div><p>It&apos;s clear the impact of synthetic data extends far beyond just the creation of random data. Here are some key advantages of synthetic data in today&apos;s data-driven world:</p><ul><li><p><strong>Enhanced data</strong>: Synthetic data enables the creation of expansive, training-ready datasets, significantly alleviating the time and financial burdens associated with traditional data collection methods.</p></li><li><p><strong>Scalable data</strong>: By removing the need for real-world training data and accessing these pre-labeled data, we substantially reduce the need for labor-intensive human annotation.</p></li><li><p><strong>Impossible data</strong>: The simulation of otherwise inaccessible data, providing solutions to challenges posed by biases, incompleteness, scale, accessibility, or security constraints inherent in real-world data collection.</p></li><li><p><strong>Sensitive data</strong>: Synthetic data facilitates the sharing or publishing of data for training purposes, circumventing the privacy and sensitivity concerns tied to actual data.</p></li></ul><h2 id="h-the-web3-opportunity" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">The web3 opportunity</h2><p>We are approaching a significant change in how data is created. Synthetic data introduces both opportunities and challenges for all of us. On the one hand, synthetic data is still tricky to get right (and <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.vanderschaar-lab.com/privacy-challenge/">when dealing with privacy, comes with serious pitfalls</a>). On the other hand, synthetic data will rapidly grow more valuable and easy to create. Web3 needs to look at the concrete opportunities that come in the synthetic data future. Let&apos;s explore some of the problem areas where web3 and synthetic data can grow together.</p><h3 id="h-bootstrapping-data-economies" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0"><strong>Bootstrapping data economies</strong></h3><p>The path to establishing data economies within web3 ecosystems (or any ecosystem) comes with challenges, particularly in bootstrapping real-world data to fuel these new economies. Here, synthetic data can serve as a bridge to overcoming the initial scarcity of real-world data. Consider these examples for how to use synthetic data to help bootstrap web3 data economies:</p><ul><li><p><strong>Rapidly scale DePIN data</strong>: Synthetic data can be used to improve analysis and prediction models for decentralized sensor networks. Synthetic data can be used by a project like <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://weatherxm.com/">WeatherXM</a> to improve the detection of bad data faster than it appears in the wild, enhance the reward models, and build classification models for high-value events (e.g., floods, heat-waves, etc.) not yet seen at high-levels in real data.</p></li><li><p><strong>Privacy-conscious data sharing</strong>: Projects like <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://dimo.zone/">Dimo</a> could generate synthetic trip datasets, allowing developers to enhance applications without compromising user privacy. While real trip data on the network is sensitive and private to only the drivers and the apps they share it with, synthetic data can be shared (<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.vanderschaar-lab.com/privacy-challenge/">with attention to risks</a>) for all developers to train new models and build novel applications to run in their ecosystem.</p></li><li><p><strong>Reputation and future threat detection</strong>: Projects like <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://passport.gitcoin.co/">Gitcoin Passport</a> could utilize synthetic data to refine models identifying fraudulent behavior, bolstering security across Web3 platforms. Help expand beyond heuristic approaches to threat detection into advanced models that may perform better at detecting attacks versus real people.</p></li><li><p><strong>Generating more map data</strong>: Synthetic data derived from dash cams is helping to create an open dataset of accurate, high-detail map data, crucial for navigation and geographic information systems. On a network like <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://hivemapper.com/">Hivemapper</a>, synthetic data would help developers create innovative models (e.g., new categories of object detection) to extract greater amounts of data from images. See <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.youtube.com/watch?v=4OYhYDziklg">here for examples</a> on how to use synthetic data for real-world object detection in imagery and what a single developer can achieve.</p></li><li><p><strong>Decentralized ride-share &amp; delivery predictions</strong>: Bootstrapping the sharing economy has well documented challenges, among them is the gap between data that incumbents have versus what any upstart can access. Synthetic data facilitates the creation of a comprehensive training dataset to optimize driver dispatching and forecast demand in various regions, enhancing service efficiency and user satisfaction. These capabilities could allow these networks to compete more quickly with the incumbents.</p></li><li><p><strong>DeSci for healthcare research and drug discovery</strong>: DeSci projects often involve disease research, where the sharing of original data on open networks is restricted or prohibited. This limitation can be overcome through synthetic patient data, conforming to privacy regulations, to accelerate clinical research and personalized medicine, driving forward healthcare innovations.</p></li></ul><p>While generating synthetic data isn’t trivial, and any example above would take investment, the potential of synthetic data in web3 can’t be understated. I believe it could become a cornerstone of the web3 data landscape.</p><h3 id="h-maximize-access-and-democratize-innovation" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Maximize access &amp; democratize innovation</h3><blockquote><p>The issue of figuring out which people are best to work on a problem is totally different from the issue of figuring out which problems to solve. <br><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.brasstacks.blog/explain-im/">Munn, 2024</a></p></blockquote><p>By publishing data (e.g., benchmark data for competition or standardized data in place of private data for composability) on web3 platforms and leveraging crypto-based incentives, we can power more open innovation. Open &amp; permissible data encourages collaboration and iteration and also (through incentives) can create new models of rewarding innovation. All together, the combination can propel the development of novel solutions to big problems.</p><p>Here are two examples for how synthetic data might be used to expand the innovation on open networks in web3.</p><ul><li><p><strong>Tailored forecasting solutions</strong>: The weather prediction and event classification needs of a farmer in Brazil are different from a city dweller in India. By offering customizable, region-specific synthetic datasets, WeatherXM and other networks could empower local developers and researchers to rapidly innovate and deploy weather prediction and classification tools that are tuned to the unique environmental challenges and requirements of their communities. This innovation, and showcasing the value of the results, can potentially happen faster than the bootstrapping of hardware installation to completely cover those regions.</p></li><li><p><strong>Collaborative research breakthroughs</strong>: A similar promise exists within DeSci, where interested parties can pool resources to solve specific challenges or unlock new resources not always addressed by institutions or larger industry players. Synthetic patient data bypasses the privacy and sharing constraints of traditional research data. Allowing for innovation on data to begin prior to solving regulatory challenges, or even knowing who you are collaborating with.</p></li></ul><h3 id="h-provenance-and-verifiability" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0"><strong>Provenance and verifiability</strong></h3><blockquote><p>… AI needs data and blockchains are good for storing and tracking data. <br><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://vitalik.eth.limo/general/2024/01/30/cryptoai.html">Vitalik, 2024</a></p></blockquote><p>Synthetic data is going to exacerbate all problems around trusting and using data on the Internet. It will grow increasingly difficult for (both humans and machines) to discern real-data from synthetic data. I have no doubt that costly mistakes will be made from processing synthetic data in the place of real data (or vice versa). This is an area where blockchain can help.</p><p>Certifying data origins is crucial as the distinction between synthetic and real data becomes blurred. With every piece of data, be it a dataset for training machine learning models or records for transactional purposes, blockchain can provide a verifiable trail that confirms its source, modifications, and ownership over time. This is true for both public and private data.</p><p>Comprehensive traceability will be vital for compliance with regulatory standards and privacy laws (e.g., synthetic PII). By creating an ecosystem where data&apos;s origins are transparent and verifiable, we can mitigate risks associated with data misuse and enhance the reliability of data.</p><p>Advanced data structures, proof carrying formats, inclusion proofs, and other cryptographic tools can further strengthen the integrity and verifiability of data. These mechanisms could verify any data&apos;s accuracy and completeness (often without exposing the underlying data).</p><p>The future of data usage and sharing, particularly in sensitive and regulated domains, will increasingly rely on such cryptographic verification methods to maintain trust, security, and privacy in our increasingly data-driven world.</p><h2 id="h-conclusions" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Conclusions</h2><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/5f6642c7fffe1b5e61b57fb88300bdcd272283262fb0b3e6be5978ec7a67c4ed.png" alt="source: http://tinyurl.com/5ajfppbc" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">source: http://tinyurl.com/5ajfppbc</figcaption></figure><p>The gains made in a future GPT-5 wont come from newly crawled Internet content as much as they will from innovations in synthetic data. Likewise, our entire field should be looking at how to capture and incorporate synthetic data into our open networks.</p><p>Web3’s opportunities with synthetic data are big. From certifying data origins to nurturing burgeoning data economies that thrive on open innovation and collaboration. But synthetic data isn’t magical, creating it isn’t easy and it will take a <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.youtube.com/watch?v=DS_ZpXBbx0w">lot of effort to get it right</a>. Large, open questions remain: how can we capture the full potential and value? How can we ensure responsible development and equitable access? Our future is not just for AI, but for collaboration, discovery, and innovation itself.</p><p>I’m excited for where we can go and to see how we use the current wave of technology to build bigger, more valuable, and more decentralized networks of innovative data products. I hope and believe we can play an important role in catalyzing and supporting this path. For our part, we aim to build a joint, shared protocol for connecting and sharing high-value synthetic datasets for all. Our latest work brings together storage networks, attribution &amp; provenance, and data delivery to solve many challenges that will become more pressing in the synthetic data future. Connect with me online or at <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/tableland/status/1755657293939589176">EthDenver next week</a> if you want to talk more about synthetic data or our solutions.</p>]]></content:encoded>
            <author>tableland-2@newsletter.paragraph.com (Tableland)</author>
            <enclosure url="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/cdb33f8b9e82d08abf2f855dfb020e94941cb481ebd1cb7bb1ec250563777220.jpg" length="0" type="image/jpg"/>
        </item>
        <item>
            <title><![CDATA[Gearing Up for a Dynamic Week at ETHDenver with Tableland!]]></title>
            <link>https://paragraph.com/@tableland-2/gearing-up-for-a-dynamic-week-at-ethdenver-with-tableland</link>
            <guid>xym42etdjstl9TMb8oCR</guid>
            <pubDate>Thu, 22 Feb 2024 20:20:34 GMT</pubDate>
            <description><![CDATA[We’re looking forward to a vibrant week of conversations, presentations, and activations with our ecosystem partners and friends. We’d love to see you there! Here are some of the gatherings on our agenda:Tuesday, February 27thDePIN DaySet to collaborate with our companions at Fluence for a day replete with thought-provoking talks, immersive workshops, and engaging panels exploring the depths of DePIN innovation. Location: Green Spaces, 2950 Walnut St, 80205 Time: 13:00 - 17:00 MSTWednesday, F...]]></description>
            <content:encoded><![CDATA[<p>We’re looking forward to a vibrant week of conversations, presentations, and activations with our ecosystem partners and friends. We’d love to see you there! Here are some of the gatherings on our agenda:</p><h2 id="h-tuesday-february-27th" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0"><strong>Tuesday, February 27th</strong></h2><h3 id="h-depin-day" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0"><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://depinday.xyz/?ref=blog.ceramic.network"><strong>DePIN Day</strong></a></h3><p>Set to collaborate with our companions at Fluence for a day replete with thought-provoking talks, immersive workshops, and engaging panels exploring the depths of <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://depinday.xyz/">DePIN innovation</a>.</p><p><strong>Location</strong>: Green Spaces, 2950 Walnut St, 80205<br><strong>Time</strong>: 13:00 - 17:00 MST</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/ca2a2ffdf8eacd4d0016e4ab2abab978f05accf2d8eeaffe23b7a3c80aad84e5.png" alt="" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="hide-figcaption"></figcaption></figure><h2 id="h-wednesday-february-28th" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0"><strong>Wednesday, February 28th</strong></h2><h3 id="h-r3al-world" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0"><strong>R3AL WORLD</strong></h3><p>Dive into the nexus of innovation at <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://lu.ma/r3al-world-ethdenver">R3al World</a>, powered by IoTeX. This gathering spotlights pioneers turning devices into shared economies. Catch Textile/Tableland co-founder and CEO Andrew Hill’s presentation and visit us at our booth to see web3&apos;s real-world merge through tech like mobile, wearables, and more.</p><p><strong>Location</strong>: Green Spaces, 2950 Walnut St, 80205<br><strong>Time</strong>: 10:00 - 20:00 MST</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/26b1e5221e3d03a3d2d29ac03ebe8f20b1f5db0b9ef5fff43349626de449ee47.png" alt="" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="hide-figcaption"></figcaption></figure><h3 id="h-desci-summit" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0"><strong>DeSci Summit</strong></h3><p>Join a pivotal gathering of open science infrastructure leaders committed to shaping a collaborative, fair, and AI-enhanced scientific ecosystem. We’ll be participating in the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://scios.desci.community/#day-three-february-28th">Open State Data Networks Workshop</a> to help tackle challenges and explore opportunities to integrate computation and data seamlessly in an open network.</p><p><strong>Location</strong>: 2601 Walnut St., 80205<br><strong>Time</strong>: 13:00 - 16:00 MST</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/a8fddd25cce32670e39e078957dcab5101fc5a81a8736d489022b6111e9e9a13.png" alt="" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="hide-figcaption"></figcaption></figure><h2 id="h-thursday-february-29th" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0"><strong>Thursday, February 29th</strong></h2><h3 id="h-fil-dev-summit" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0"><strong>FIL Dev Summit</strong></h3><p>Join the forefront of Filecoin development at the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://lu.ma/FDS-EthDenver">FIL Dev Summit</a>, a confluence for developers, builders, and the Filecoin community, aimed at shaping the future of the protocol. Featuring deep discussions and collaborative efforts to advance Filecoin, we’ll be certain to attend the Decentralized Compute &amp; Decentralized Storage session.</p><p><strong>Location</strong>: The Slate, 1250 Welton St, 80204<br><strong>Time</strong>: 10:00 - 18:00 MST</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/9ded210304a24c0ff8d5d359d730cffde6a82522a9be5e1426edd96c5903ac4b.png" alt="" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="hide-figcaption"></figcaption></figure><h3 id="h-filecoin-foundation-booth" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0"><strong>Filecoin Foundation Booth</strong></h3><p>Carson Farmer will explore the adoption of cloud-native object storage APIs in Web3 to improve scalability and attract developers, detailing strategies for integrating cloud storage standards into the decentralized web.</p><p><strong>Location</strong>: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://hub.fil.org/events/ethdenver/menu/see-the-schedule">Devtopia Booth #204</a>, ETH Denver Spork Castle<br><strong>Time</strong>: 16 - 16:30 MST</p><h2 id="h-friday-march-1st" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0"><strong>Friday, March 1st</strong></h2><h3 id="h-proof-of-data-summit" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0"><strong>Proof of Data Summit</strong></h3><p>We’re hosting the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://proofofdata.love">Proof of Data Summit</a> with our friends at Ceramic, a pivotal event where the data management landscape is redefined for the decentralized age. This summit brings together the brightest minds in DePIN, decentralized AI, compute over data, reputation, and identity to leverage our collective infrastructures for data-driven innovation, trust, and community growth.</p><p><strong>Location</strong>: Denver Art Museum, 100 W 14th Ave Pkwy, 80204<br><strong>Time</strong>: 09:00 - 17:00 MST</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/0d71c304d4ece9a500667dd7b69975880da9e8c9027af07aff51eda5fd1a9110.png" alt="" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="hide-figcaption"></figcaption></figure><h3 id="h-filecoin-orbit-ecosystem-showcase-and-happy-hour" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0"><strong>Filecoin Orbit Ecosystem Showcase &amp; Happy Hour</strong></h3><p>Delve into the forefront of innovation with the Filecoin community at this engaging <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://filecoindenver.io/">showcase and happy hour</a>. Join Textile/Tableland&apos;s Head of Research, Carson Farmer, as he unveils the future of data infrastructure with Textile and Filecoin.</p><p><strong>Location</strong>: The Spur, 4777 National Western Dr, 80216<br><strong>Time</strong>: 17:30 - 19:30 MS</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/fd371ddf84415ba08645f800e0b4d2a56d8db9194ea1ca12fe93312bb885b4b2.png" alt="" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="hide-figcaption"></figcaption></figure><p>Stay tuned for more details and follow our <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://x.com/tableland">Twitter</a> for real-time updates. Excited to connect with you in Denver!</p>]]></content:encoded>
            <author>tableland-2@newsletter.paragraph.com (Tableland)</author>
            <enclosure url="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/94a021cd2e614c05eca4877668e2007d366e26fc22931c96639f4e8457f54a79.png" length="0" type="image/png"/>
        </item>
        <item>
            <title><![CDATA[Farcaster Replicator: How to Further Decentralize the Stack with Textile]]></title>
            <link>https://paragraph.com/@tableland-2/farcaster-replicator-how-to-further-decentralize-the-stack-with-textile</link>
            <guid>uClBGN20JbCyGeriUDIg</guid>
            <pubDate>Thu, 15 Feb 2024 20:40:07 GMT</pubDate>
            <description><![CDATA[Make Farcaster data available on decentralized storage with replication from Hubs to Postgres to Textile & Filecoin.BackgroundFarcaster is a “sufficiently” decentralized social network that’s modeled off of Twitter/X but rooted in web3. Part of its success has been the team’s approach toward building a scalable and UX-focused application, or as the co-founder puts it… https://twitter.com/dwr/status/1755306700486381825 What’s unique with Textile’s approach to web3 data is that we agree with Fa...]]></description>
            <content:encoded><![CDATA[<blockquote><p><em>Make Farcaster data available on decentralized storage with replication from Hubs to Postgres to Textile &amp; Filecoin.</em></p></blockquote><h2 id="h-background" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Background</h2><p>Farcaster is a “sufficiently” decentralized social network that’s modeled off of Twitter/X but rooted in web3. Part of its success has been the team’s approach toward building a scalable and UX-focused application, or as the co-founder puts it…</p><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/dwr/status/1755306700486381825">https://twitter.com/dwr/status/1755306700486381825</a></p><p>What’s unique with Textile’s approach to web3 data is that we agree with Farcaster’s sentiment: web2 isn’t going away and is more performant for certain parts of the stack. Thus, Textile’s <code>vaults</code> tooling lets developers take data from common web2 environments, such as a Postgres database, and write that data to decentralized storage with almost no additional effort! This allows protocols and applications to progressively decentralize and make certain data open for fair use.</p><h3 id="h-demonstrating-replication-from-farcaster-to-textile" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Demonstrating replication from Farcaster to Textile</h3><p>The Farcaster replicator app takes data from Hubs (e.g., “Hubble” is a Hub implementation) and stores it in a Postgres database. This allows developers to query Farcaster data and easily build applications on top of the network&apos;s data. You can check out the Farcaster <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.farcaster.xyz/developers/guides/apps/replicate">docs for more information</a> on the existing functionality, but make sure you have <em>at least</em> 2GB of memory and 4GB of free disk space to run the replicator app. (Farcaster&apos;s original resource estimates are now outdated due to increased activity; expect slightly higher memory and storage needs for the replicator app.)</p><p>To make this possible, a slight adjustment was made to Farcaster’s replicator app that makes the data also available via the Textile Network&apos;s novel caching layer and cold storage adapter (to Filecoin). The <code>vaults</code> CLI (<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/basin-cli">here</a>) made this process easy to implement. It lets you create vaults—web3-native data containers—and stream data from a Postgres database to decentralized storage, retrievable by &quot;event&quot; <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.ipfs.tech/concepts/content-addressing/">CIDs</a> which are formatted as Parquet files. Every event is written to both a hot cache layer with TTL (time-to-live) and a cold storage layer. Thus, anyone can extract openly replicated data from a vault and load it into their applications instead of running the replicator app themselves.</p><blockquote><p><em>Check out the source code (a fork of Farcaster’s monorepo): </em><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/dtbuchholz/hub-monorepo/tree/main/apps/replicator"><em>dtbuchholz/hub-monorepo</em></a><em>. Note: There were some demo-related imperfections in the setup, explained below. But, a decent amount of data was replicated to Textile and should show how it can be used in tandem with existing web2 tech stacks.</em></p></blockquote><h3 id="h-project-structure" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Project structure</h3><p>In our demonstration, the Textile replication process is handled using shell scripts that execute CLI commands and a custom Docker image. To use the <code>vaults</code> CLI tool, the <code>wal2json</code> Postgres extension must be installed/configured, and the <code>vaults</code> binary must also be installed. You can review <code>dtbuchholz/textile-vaults-farcaster</code> (<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://hub.docker.com/repository/docker/dtbuchholz/textile-vaults-farcaster">here</a>) for the published Docker image, which supports the <code>linux/arm64</code> architecture and handles these setup steps.</p><p>Since the replicator app already uses Docker Compose, the original source code is, largely, untouched. The <code>docker-compose.yml</code> file has been updated to include the new Docker image as well as the necessary environment variables for the <code>vaults</code> CLI (<code>TEXTILE_PRIVATE_KEY</code> &amp; <code>TEXTILE_VAULT_NAMESPACE</code>).</p><p>Then, a couple of custom scripts were added:</p><ul><li><p><code>custom-entrypoint.sh</code>: Entrypoint for the Docker image, which sets up Postgres to use logical replication, increases the max replication slots, and increases WAL senders to account for 12 total Farcaster tables via <code>postgres -c wal_level=logical -c max_replication_slots=15 -c max_wal_senders=15</code>. Then, it runs the <code>vaults</code> commands defined in <code>textile_vaults.sh</code>.</p></li><li><p><code>textile_vaults.sh</code>: Polls for the Farcaster tables to be created after the replicator app&apos;s migration process, and then starts replicating the data to Textile as it gets inserted into the Postgres database/tables.</p></li></ul><p>More details on this code are described below, or check out the repo for the full implementation.</p><h2 id="h-installation-and-setup" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Installation &amp; setup</h2><p>Start by cloning this forked repository, navigating to the root directory, installing/building, and navigating to the <code>apps/replicator</code> directory. Note that one of the apps <code>@farcaster/hubble</code> requires <code>cargo</code> to be installed on your machine (but isn&apos;t used in the replicator app).</p><pre data-type="codeBlock" text="git clone https://github.com/dtbuchholz/hub-monorepo
cd hub-monorepo
yarn install
yarn build
cd apps/replicator
"><code>git <span class="hljs-built_in">clone</span> https://github.com/dtbuchholz/hub-monorepo
<span class="hljs-built_in">cd</span> hub-monorepo
yarn install
yarn build
<span class="hljs-built_in">cd</span> apps/replicator
</code></pre><h3 id="h-set-up-environment-variables" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Set up environment variables</h3><p>Once you install &amp; build, you need to copy the <code>.env.example</code> file (in <code>apps/replicator</code>) to a new <code>.env</code> file and fill in the necessary environment variables. All of the Farcaster-related ones are pre-configured for you. For context, <em>most</em> of these come from the <code>.env</code> file that gets created if you were to run the official replicator bootstrapping process, described <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.farcaster.xyz/developers/guides/apps/replicate#installation">here</a>. The only change is that <code>nemes.farcaster.xyz:2283</code> is used as the <code>FARCASTER_HUB_URL</code>, which lets us skip the setup step for Hubble and use a public Hubble API, provided by the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.farcaster.xyz/hubble/hubble#public-instances">Farcaster team</a>.</p><p>The <code>TEXTILE_PRIVATE_KEY</code> and <code>TEXTILE_VAULT_NAMESPACE</code> environment variables are the only new ones you need to add. You can use an existing Ethereum-style (secp256k1) private key for <code>TEXTILE_PRIVATE_KEY</code> (note: <em>do not</em> hex-prefix the string in the <code>.env</code> file). Or, you can run the following command to generate a new one...just make sure the <code>vaults</code> CLI is installed if you run this on your machine!:</p><pre data-type="codeBlock" text="vaults account create .wallet
"><code>vaults account create .wallet
</code></pre><p>This will create a private key and store it locally in a file called <code>.wallet</code> (or whatever file name you choose). The <code>TEXTILE_VAULT_NAMESPACE</code> can be any string you want to use to namespace your vaults—the example below is for <code>demo_farcaster_2</code>. When a vault is created, it will be prefixed with this string, and each table that gets replicated will exist under this namespace.</p><p>For example, the <code>casts</code> table would be replicated to a vault called <code>my_namespace.casts</code>. The following defines all of the tables that get replicated:</p><ul><li><p>casts</p></li><li><p>chain_events</p></li><li><p>fids</p></li><li><p>fnames</p></li><li><p>links</p></li><li><p>messages</p></li><li><p>reactions</p></li><li><p>signers</p></li><li><p>storage_allocations</p></li><li><p>user_data</p></li><li><p>username_proofs</p></li><li><p>verifications</p></li></ul><h3 id="h-replication-process-and-docker" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Replication process &amp; Docker</h3><p>To run the replicator app, you&apos;ll need to make sure you have <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.docker.com/engine/install/">Docker installed</a> and running. Then, you can start the replicator app by running the following command from the <code>apps/replicator</code> directory:</p><pre data-type="codeBlock" text="docker compose up
"><code></code></pre><p>This will kick off a process that does the following:</p><ul><li><p>Spins up 5 total Docker containers: one for the core replicator app, Redis (caching), statsd (for stats aggregation), Grafana (to view stats), and the Postgres database (altered with Textile functionality).</p></li><li><p>Fetches the latest Farcaster data from the Nemes Hub and runs through a migration process that writes backfilled data to the Postgres database.</p></li><li><p>Streams Postgres changes to Textile via the <code>vaults</code> CLI, which creates vaults for each table and replicates the data to Textile&apos;s decentralized storage.</p></li><li><p>The replicated data goes to both a caching layer and cold storage layer, so it can be retrieved ~immediately for a pre-configured TTL and ~indefinitely from Filecoin—all using the <code>vaults retrieve &lt;cid&gt;</code> command.</p></li></ul><blockquote><p>Note: The Textile script is designed naively to run only on startup, so if you exit the Docker process and restart it, the script will start from the beginning and try to create new vaults for each table.</p></blockquote><h2 id="h-textile-vaults-and-farcaster-data" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Textile vaults &amp; Farcaster data</h2><h3 id="h-vaults-script-and-streaming" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Vaults script &amp; streaming</h3><p>If you review the <code>textile_vaults.sh</code> script, two important steps occur. First, new vaults are created for every Farcaster table with the <code>vaults create</code> command:</p><pre data-type="codeBlock" text="vaults create \
--account &quot;$(vaults account address &quot;${private_key_file}&quot;)&quot; \
--cache 10080 \
--dburi &quot;postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}&quot; \
--window-size 1800 \
&quot;${TEXTILE_VAULT_NAMESPACE}.${table}&quot;
"><code>vaults create \
<span class="hljs-operator">-</span><span class="hljs-operator">-</span>account <span class="hljs-string">"$(vaults account address "</span>${private_key_file}<span class="hljs-string">")"</span> \
<span class="hljs-operator">-</span><span class="hljs-operator">-</span>cache <span class="hljs-number">10080</span> \
<span class="hljs-operator">-</span><span class="hljs-operator">-</span>dburi <span class="hljs-string">"postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}"</span> \
<span class="hljs-operator">-</span><span class="hljs-operator">-</span>window<span class="hljs-operator">-</span>size <span class="hljs-number">1800</span> \
<span class="hljs-string">"${TEXTILE_VAULT_NAMESPACE}.${table}"</span>
</code></pre><ul><li><p><code>--account</code>: Sets the account address to use for the vault, which is the public key of the private key you set in the <code>.env</code> file (provided by the <code>vaults account address</code> command).</p></li><li><p><code>--cache</code>: Sets the TTL in minutes, so <code>10080</code> is equivalent to 7 days of cached data storage &amp; retrieval.</p></li><li><p><code>--dburi</code>: Defines the Postgres database to stream data from.</p></li><li><p><code>--window-size</code>: Sets the window size in seconds, which is the time period to batch changes before sending them to Textile—in this example, every 30 minutes.</p></li><li><p>Lastly, the <code>${TEXTILE_VAULT_NAMESPACE}.${table}</code> part of the script defines the vault name. This is the environment variable you set in the <code>.env</code> file and the name of the table in Postgres that&apos;s being replicated, separated by a period (<code>.</code>). The <code>table</code> is defined in a loop where the command is executed, which is each table name in the bulleted list above.</p></li></ul><p>Then, the script starts replicating the data to Textile with the <code>vaults stream</code> command:</p><pre data-type="codeBlock" text="vaults stream \
--private-key &quot;${TEXTILE_PRIVATE_KEY}&quot; \
&quot;${TEXTILE_VAULT_NAMESPACE}.${table}&quot;
"><code>vaults stream \
--<span class="hljs-keyword">private</span>-key <span class="hljs-string">"<span class="hljs-subst">${TEXTILE_PRIVATE_KEY}</span>"</span> \
<span class="hljs-string">"<span class="hljs-subst">${TEXTILE_VAULT_NAMESPACE}</span>.<span class="hljs-subst">${table}</span>"</span>
</code></pre><ul><li><p><code>--private-key</code>: The private key you set in the <code>.env</code> file is used to sign all data that gets sent to Textile, which ensures data integrity and authenticity.</p></li><li><p><code>${TEXTILE_VAULT_NAMESPACE}.${table}</code>: The name of the vault (formatted as <code>namespace.table_name</code>) to stream data to, which gets created in the preceding <code>vaults create</code> command.</p></li></ul><h3 id="h-farcaster-textile-events" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Farcaster → Textile events</h3><p>Because Textile makes the data publicly available, you can work with data that&apos;s already been backfilled in an existing vault. First, you&apos;ll want to make sure the <code>vaults</code> CLI is installed on your machine (note: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://go.dev/doc/install">make sure Go is installed</a> on your machine first):</p><pre data-type="codeBlock" text="go install github.com/tablelandnetwork/basin-cli/cmd/vaults@latest
"><code>go install github.com/tablelandnetwork<span class="hljs-operator">/</span>basin<span class="hljs-operator">-</span>cli<span class="hljs-operator">/</span>cmd<span class="hljs-operator">/</span>vaults@latest
</code></pre><p>For the sake of this demonstration, we&apos;ve created a vault with the namespace <code>demo_farcaster_2</code> (ignore <code>demo_farcaster</code> as this was a test). The owner of this vault is at the account <code>0x5e50Bb5A0fA30c4b9B6a809687CDB90A776D8bB7</code>. You can start by checking out all of the vaults owned by this account:</p><pre data-type="codeBlock" text="vaults list --account 0x5e50Bb5A0fA30c4b9B6a809687CDB90A776D8bB7
"><code>vaults list <span class="hljs-operator">-</span><span class="hljs-operator">-</span>account <span class="hljs-number">0x5e50Bb5A0fA30c4b9B6a809687CDB90A776D8bB7</span>
</code></pre><blockquote><p><em>Note: Due to some demo script issues and constraints when running Docker, the following tables did not fully replicate data to the</em> <code>demo_farcaster_2</code> <em>namespace:</em> <code>fids</code><em>,</em> <code>fnames</code><em>,</em> <code>casts</code><em>,</em> <code>signers</code><em>, and</em> <code>messages</code>. If you run the <code>vaults events</code> <em>command, only a subset of the full dataset will be listed. Additionally, after about 6 hours, the Docker image maxed out on memory and caused Redis to fail, which then caused the replication process to halt. In total, the database reached 2.867 GB at the time the processes were (manually) killed.</em></p></blockquote><p>This will list out each of those described above, such as <code>demo_farcaster_2.user_data</code>, <code>demo_farcaster_2.reactions</code>, etc. You can then retrieve the data from any of these vaults by running the following command to see each event CID (the example shows data from the table <code>user_data</code>). An event gets created anytime new data is written to the vault and the window size is reached.</p><pre data-type="codeBlock" text="vaults events --vault demo_farcaster_2.user_data --format json
"><code>vaults events <span class="hljs-operator">-</span><span class="hljs-operator">-</span>vault demo_farcaster_2.user_data <span class="hljs-operator">-</span><span class="hljs-operator">-</span>format json
</code></pre><p>Which will log out a list of CIDs that you can use to retrieve the data from the vault:</p><pre data-type="codeBlock" text="[
  {
    &quot;cid&quot;: &quot;bafkreietgpbkfvdjyqlpwph75eajcp2pix3uw5dajjrfxvzvxv6gti7duy&quot;,
    &quot;timestamp&quot;: 1707970383,
    &quot;is_archived&quot;: false,
    &quot;cache_expiry&quot;: &quot;2024-02-22T04:13:08.355842&quot;
  }
]
"><code><span class="hljs-punctuation">[</span>
  <span class="hljs-punctuation">{</span>
    <span class="hljs-attr">"cid"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"bafkreietgpbkfvdjyqlpwph75eajcp2pix3uw5dajjrfxvzvxv6gti7duy"</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"timestamp"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">1707970383</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"is_archived"</span><span class="hljs-punctuation">:</span> <span class="hljs-literal"><span class="hljs-keyword">false</span></span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"cache_expiry"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"2024-02-22T04:13:08.355842"</span>
  <span class="hljs-punctuation">}</span>
<span class="hljs-punctuation">]</span>
</code></pre><h3 id="h-retrieving-and-querying-vault-events" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Retrieving &amp; querying vault events</h3><p>If the Textile cache feature is enabled—and the TTL has not expired—the data will be available immediately. If not, it will be routed from IPFS/Filecoin and retrieved from cold storage.</p><pre data-type="codeBlock" text="vaults retrieve bafkreietgpbkfvdjyqlpwph75eajcp2pix3uw5dajjrfxvzvxv6gti7duy
"><code></code></pre><p>Once you download this data, you can then load it into your application. The file gets downloaded with a filename in the format <code>&lt;cid&gt;-&lt;timestamp&gt;.db.parquet</code>, so you can use any Parquet reader to load the data into your application—for example, <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://duckdb.org/#quickinstall">DuckDB</a>.</p><p>First, make sure the DuckDB CLI is installed on your machine, such as using Homebrew:</p><pre data-type="codeBlock" text="brew install duckdb
"><code></code></pre><p>Then, you can load the data within a DuckDB in-memory database using the <code>read_parquet</code> function:</p><pre data-type="codeBlock" text="&gt; duckdb
SELECT
  *
FROM
  read_parquet(
    &apos;bafkreietgpbkfvdjyqlpwph75eajcp2pix3uw5dajjrfxvzvxv6gti7duy-1707948730199375716.db.parquet&apos;
  )
LIMIT
  2;
"><code><span class="hljs-operator">></span> duckdb
<span class="hljs-keyword">SELECT</span>
  <span class="hljs-operator">*</span>
<span class="hljs-keyword">FROM</span>
  read_parquet(
    <span class="hljs-string">'bafkreietgpbkfvdjyqlpwph75eajcp2pix3uw5dajjrfxvzvxv6gti7duy-1707948730199375716.db.parquet'</span>
  )
LIMIT
  <span class="hljs-number">2</span>;
</code></pre><p>Note that DuckDB also lets you load multiple files at once or even remote URLs by passing an array instead of a single string. This is useful if a vault has multiple events and you want to load all of the data at once:</p><pre data-type="codeBlock" text="SELECT
  *
FROM
  read_parquet([
    &apos;bafy123...&apos;,
    &apos;bafy456...&apos;,
    &apos;bafy789...&apos;
  ])
LIMIT
  2;
"><code>SELECT
  *
<span class="hljs-selector-tag">FROM</span>
  read_parquet(<span class="hljs-selector-attr">[    <span class="hljs-string">'bafy123...'</span>,    <span class="hljs-string">'bafy456...'</span>,    <span class="hljs-string">'bafy789...'</span>  ]</span>)
LIMIT
  <span class="hljs-number">2</span>;
</code></pre><p>Once you execute the query, it logs the data from that specific table&apos;s updates and the Textile event created in the Hubble → Postgres → Textile replication process. For example, the <code>user_data</code> table&apos;s first 2 rows of the Parquet file look like this:</p><pre data-type="codeBlock" text="[
  {
    &quot;id&quot;: &quot;018da9ad-5b90-8ec2-15e7-b8df0b2a80db&quot;,
    &quot;created_at&quot;: &quot;2024-02-14 14:12:07.126457-08&quot;,
    &quot;updated_at&quot;: &quot;2024-02-14 14:12:07.126457-08&quot;,
    &quot;timestamp&quot;: &quot;2023-03-15 13:09:51-07&quot;,
    &quot;deleted_at&quot;: null,
    &quot;fid&quot;: 1,
    &quot;type&quot;: 1,
    &quot;hash&quot;: &quot;481d5e80de1b2803e89c6ad96ab349979a689358&quot;,
    &quot;value&quot;: &quot;https://i.imgur.com/I2rEbPF.png&quot;
  },
  {
    &quot;id&quot;: &quot;018da9ad-5ba6-34a8-95f7-373ef6d9cfdd&quot;,
    &quot;created_at&quot;: &quot;2024-02-14 14:12:07.201302-08&quot;,
    &quot;updated_at&quot;: &quot;2024-02-14 14:12:07.201302-08&quot;,
    &quot;timestamp&quot;: &quot;2023-07-21 18:20:44-07&quot;,
    &quot;deleted_at&quot;: null,
    &quot;fid&quot;: 1,
    &quot;type&quot;: 3,
    &quot;hash&quot;: &quot;0edc9e1164b7d743517e6c0df4ec9f539ab4252b&quot;,
    &quot;value&quot;: &quot;A sufficiently decentralized social network. farcaster.xyz&quot;
  }
]
"><code><span class="hljs-punctuation">[</span>
  <span class="hljs-punctuation">{</span>
    <span class="hljs-attr">"id"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"018da9ad-5b90-8ec2-15e7-b8df0b2a80db"</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"created_at"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"2024-02-14 14:12:07.126457-08"</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"updated_at"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"2024-02-14 14:12:07.126457-08"</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"timestamp"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"2023-03-15 13:09:51-07"</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"deleted_at"</span><span class="hljs-punctuation">:</span> <span class="hljs-literal"><span class="hljs-keyword">null</span></span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"fid"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">1</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"type"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">1</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"hash"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"481d5e80de1b2803e89c6ad96ab349979a689358"</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"value"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"https://i.imgur.com/I2rEbPF.png"</span>
  <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span>
  <span class="hljs-punctuation">{</span>
    <span class="hljs-attr">"id"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"018da9ad-5ba6-34a8-95f7-373ef6d9cfdd"</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"created_at"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"2024-02-14 14:12:07.201302-08"</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"updated_at"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"2024-02-14 14:12:07.201302-08"</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"timestamp"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"2023-07-21 18:20:44-07"</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"deleted_at"</span><span class="hljs-punctuation">:</span> <span class="hljs-literal"><span class="hljs-keyword">null</span></span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"fid"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">1</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"type"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">3</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"hash"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"0edc9e1164b7d743517e6c0df4ec9f539ab4252b"</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"value"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"A sufficiently decentralized social network. farcaster.xyz"</span>
  <span class="hljs-punctuation">}</span>
<span class="hljs-punctuation">]</span>
</code></pre><blockquote><p>*Fun fact: As shown above, the first ever user data was a link to an image of the Farcaster logo: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://i.imgur.com/I2rEbPF">https://i.imgur.com/I2rEbPF</a>. Plus, the fifth piece of user data was a message from/about Brian Armstrong (the *<code>value</code> <em>is</em> <code>barmstrong</code>)!</p></blockquote><h3 id="h-example-logs" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Example logs</h3><p>Below is an abbreviated version of the logs. You can see that once the Farcaster replicator starts to backfill the FID registrations, it kicks off the Textile vault creation and streaming processes. Everything before the last section of the logs is part of the setup process, and once new data is inserted into the table, the streaming process begins.</p><pre data-type="codeBlock" text="replicator-1  | [22:36:36.847] INFO (7): Enqueuing jobs for backfilling FID registrations for 346480 FIDs...
...
postgres-1    | All required tables exist; proceeding with Textile vaults
postgres-1    | Creating vaults for tables...
postgres-1    | Vault demo_farcaster_2.chain_events created.
...
postgres-1    | Streaming tables...
postgres-1    | Streaming to vault: &apos;demo_farcaster_2.user_data&apos;
...
2024/02/14 21:20:45 INFO created new db at=/root/.vaults/demo_farcaster_2.chain_events/1707945645693026302.db
2024/02/14 21:20:45 INFO applying create query=&quot;CREATE TABLE IF NOT EXISTS chain_events (id uuid NOT NULL,created_at timestamp with time zone NOT NULL,block_timestamp timestamp with time zone NOT NULL,fid bigint NOT NULL,chain_id bigint NOT NULL,block_number bigint NOT NULL,transaction_index smallint NOT NULL,log_index smallint NOT NULL,type smallint NOT NULL,block_hash blob NOT NULL,transaction_hash blob NOT NULL,body varchar NOT NULL,raw blob NOT NULL,PRIMARY KEY (id))&quot;
...
2024-02-14 21:20:45.700 UTC [475] LOG:  starting logical decoding for slot &quot;basin_chain_events&quot;
2024-02-14 21:20:45.700 UTC [475] DETAIL:  Streaming transactions committing after 0/1A16208, reading WAL from 0/1A161D0.
2024-02-14 21:20:45.700 UTC [475] STATEMENT:  START_REPLICATION SLOT basin_chain_events LOGICAL 0/1A16208 (&quot;pretty-print&quot; &apos;false&apos;, &quot;include-transaction&quot; &apos;true&apos;, &quot;include-lsn&quot; &apos;true&apos;, &quot;include-timestamp&quot; &apos;true&apos;, &quot;include-pk&quot; &apos;true&apos;, &quot;format-version&quot; &apos;2&apos;, &quot;include-xids&quot; &apos;true&apos;, &quot;add-tables&quot; &apos;public.chain_events&apos;)
2024-02-14 21:20:45.700 UTC [475] LOG:  logical decoding found consistent point at 0/1A161D0
2024-02-14 21:20:45.700 UTC [475] DETAIL:  There are no running transactions.
2024-02-14 21:20:45.700 UTC [475] STATEMENT:  START_REPLICATION SLOT basin_chain_events LOGICAL 0/1A16208 (&quot;pretty-print&quot; &apos;false&apos;, &quot;include-transaction&quot; &apos;true&apos;, &quot;include-lsn&quot; &apos;true&apos;, &quot;include-timestamp&quot; &apos;true&apos;, &quot;include-pk&quot; &apos;true&apos;, &quot;format-version&quot; &apos;2&apos;, &quot;include-xids&quot; &apos;true&apos;, &quot;add-tables&quot; &apos;public.chain_events&apos;)
2024/02/14 21:20:45 INFO Logical replication started slot=basin_chain_events
...
2024/02/14 21:21:03 INFO new transaction received
2024/02/14 21:21:03 INFO replaying query=&quot;insert into chain_events (id, created_at, block_timestamp, fid, chain_id, block_number, transaction_index, log_index, type, block_hash, transaction_hash, body, raw) values (&apos;018da97e-9c50-bd1d-0a3b-8c71bb062e74&apos;, &apos;2024-02-14 21:21:03.508914+00&apos;, &apos;2023-11-07 19:42:51+00&apos;, 22, 10, 111893697, 7, 24, 3, &apos;73eaf3e303afa0d8e48ab3ac043ce60130f5061beb33314efa70d7bde5941b76&apos;, &apos;286dd35ce467bfb3db7e8e4b036909a5e2769d0efadb6f342ef1d8970ea1e264&apos;, &apos;{\&quot;to\&quot;:\&quot;0x411c9d8a1788c5764b32b44e6c381a541bfdce65\&quot;,\&quot;eventType\&quot;:1,\&quot;from\&quot;:\&quot;0x\&quot;,\&quot;recoveryAddress\&quot;:\&quot;0x00000000fcb080a4d6c39a9354da9eb9bc104cd7\&quot;}&apos;, &apos;0803100a18c1b9ad35222073eaf3e303afa0d8e48ab3ac043ce60130f5061beb33314efa70d7bde5941b7628bba6aaaa063220286dd35ce467bfb3db7e8e4b036909a5e2769d0efadb6f342ef1d8970ea1e264381840165a2e0a14411c9d8a1788c5764b32b44e6c381a541bfdce651001221400000000fcb080a4d6c39a9354da9eb9bc104cd768077002&apos;)&quot;
...
"><code>replicator<span class="hljs-number">-1</span>  <span class="hljs-operator">|</span> [<span class="hljs-number">22</span>:<span class="hljs-number">36</span>:<span class="hljs-number">36.847</span>] INFO (<span class="hljs-number">7</span>): Enqueuing jobs <span class="hljs-keyword">for</span> backfilling FID registrations <span class="hljs-keyword">for</span> <span class="hljs-number">346480</span> FIDs...
...
postgres<span class="hljs-number">-1</span>    <span class="hljs-operator">|</span> All required tables exist; proceeding with Textile vaults
postgres<span class="hljs-number">-1</span>    <span class="hljs-operator">|</span> Creating vaults <span class="hljs-keyword">for</span> tables...
postgres<span class="hljs-number">-1</span>    <span class="hljs-operator">|</span> Vault demo_farcaster_2.chain_events created.
...
postgres<span class="hljs-number">-1</span>    <span class="hljs-operator">|</span> Streaming tables...
postgres<span class="hljs-number">-1</span>    <span class="hljs-operator">|</span> Streaming to vault: <span class="hljs-string">'demo_farcaster_2.user_data'</span>
...
<span class="hljs-number">2024</span><span class="hljs-operator">/</span>02<span class="hljs-operator">/</span><span class="hljs-number">14</span> <span class="hljs-number">21</span>:<span class="hljs-number">20</span>:<span class="hljs-number">45</span> INFO created <span class="hljs-keyword">new</span> db at<span class="hljs-operator">=</span><span class="hljs-operator">/</span>root<span class="hljs-operator">/</span>.vaults/demo_farcaster_2.chain_events/<span class="hljs-number">1707945645693026302</span>.db
<span class="hljs-number">2024</span><span class="hljs-operator">/</span>02<span class="hljs-operator">/</span><span class="hljs-number">14</span> <span class="hljs-number">21</span>:<span class="hljs-number">20</span>:<span class="hljs-number">45</span> INFO applying create query<span class="hljs-operator">=</span><span class="hljs-string">"CREATE TABLE IF NOT EXISTS chain_events (id uuid NOT NULL,created_at timestamp with time zone NOT NULL,block_timestamp timestamp with time zone NOT NULL,fid bigint NOT NULL,chain_id bigint NOT NULL,block_number bigint NOT NULL,transaction_index smallint NOT NULL,log_index smallint NOT NULL,type smallint NOT NULL,block_hash blob NOT NULL,transaction_hash blob NOT NULL,body varchar NOT NULL,raw blob NOT NULL,PRIMARY KEY (id))"</span>
...
<span class="hljs-number">2024</span><span class="hljs-operator">-</span>02<span class="hljs-number">-14</span> <span class="hljs-number">21</span>:<span class="hljs-number">20</span>:<span class="hljs-number">45.700</span> UTC [<span class="hljs-number">475</span>] LOG:  starting logical decoding <span class="hljs-keyword">for</span> slot <span class="hljs-string">"basin_chain_events"</span>
<span class="hljs-number">2024</span><span class="hljs-operator">-</span>02<span class="hljs-number">-14</span> <span class="hljs-number">21</span>:<span class="hljs-number">20</span>:<span class="hljs-number">45.700</span> UTC [<span class="hljs-number">475</span>] DETAIL:  Streaming transactions committing after <span class="hljs-number">0</span><span class="hljs-operator">/</span>1A16208, reading WAL <span class="hljs-keyword">from</span> <span class="hljs-number">0</span><span class="hljs-operator">/</span>1A161D0.
2024-02<span class="hljs-number">-14</span> <span class="hljs-number">21</span>:<span class="hljs-number">20</span>:<span class="hljs-number">45.700</span> UTC [<span class="hljs-number">475</span>] STATEMENT:  START_REPLICATION SLOT basin_chain_events LOGICAL <span class="hljs-number">0</span><span class="hljs-operator">/</span>1A16208 (<span class="hljs-string">"pretty-print"</span> <span class="hljs-string">'false'</span>, <span class="hljs-string">"include-transaction"</span> <span class="hljs-string">'true'</span>, <span class="hljs-string">"include-lsn"</span> <span class="hljs-string">'true'</span>, <span class="hljs-string">"include-timestamp"</span> <span class="hljs-string">'true'</span>, <span class="hljs-string">"include-pk"</span> <span class="hljs-string">'true'</span>, <span class="hljs-string">"format-version"</span> <span class="hljs-string">'2'</span>, <span class="hljs-string">"include-xids"</span> <span class="hljs-string">'true'</span>, <span class="hljs-string">"add-tables"</span> <span class="hljs-string">'public.chain_events'</span>)
<span class="hljs-number">2024</span><span class="hljs-operator">-</span>02<span class="hljs-number">-14</span> <span class="hljs-number">21</span>:<span class="hljs-number">20</span>:<span class="hljs-number">45.700</span> UTC [<span class="hljs-number">475</span>] LOG:  logical decoding found consistent point at <span class="hljs-number">0</span><span class="hljs-operator">/</span>1A161D0
<span class="hljs-number">2024</span><span class="hljs-operator">-</span>02<span class="hljs-number">-14</span> <span class="hljs-number">21</span>:<span class="hljs-number">20</span>:<span class="hljs-number">45.700</span> UTC [<span class="hljs-number">475</span>] DETAIL:  There are no running transactions.
2024-02<span class="hljs-number">-14</span> <span class="hljs-number">21</span>:<span class="hljs-number">20</span>:<span class="hljs-number">45.700</span> UTC [<span class="hljs-number">475</span>] STATEMENT:  START_REPLICATION SLOT basin_chain_events LOGICAL <span class="hljs-number">0</span><span class="hljs-operator">/</span>1A16208 (<span class="hljs-string">"pretty-print"</span> <span class="hljs-string">'false'</span>, <span class="hljs-string">"include-transaction"</span> <span class="hljs-string">'true'</span>, <span class="hljs-string">"include-lsn"</span> <span class="hljs-string">'true'</span>, <span class="hljs-string">"include-timestamp"</span> <span class="hljs-string">'true'</span>, <span class="hljs-string">"include-pk"</span> <span class="hljs-string">'true'</span>, <span class="hljs-string">"format-version"</span> <span class="hljs-string">'2'</span>, <span class="hljs-string">"include-xids"</span> <span class="hljs-string">'true'</span>, <span class="hljs-string">"add-tables"</span> <span class="hljs-string">'public.chain_events'</span>)
<span class="hljs-number">2024</span><span class="hljs-operator">/</span>02<span class="hljs-operator">/</span><span class="hljs-number">14</span> <span class="hljs-number">21</span>:<span class="hljs-number">20</span>:<span class="hljs-number">45</span> INFO Logical replication started slot<span class="hljs-operator">=</span>basin_chain_events
...
<span class="hljs-number">2024</span><span class="hljs-operator">/</span>02<span class="hljs-operator">/</span><span class="hljs-number">14</span> <span class="hljs-number">21</span>:<span class="hljs-number">21</span>:03 INFO <span class="hljs-keyword">new</span> transaction received
<span class="hljs-number">2024</span><span class="hljs-operator">/</span>02<span class="hljs-operator">/</span><span class="hljs-number">14</span> <span class="hljs-number">21</span>:<span class="hljs-number">21</span>:03 INFO replaying query<span class="hljs-operator">=</span><span class="hljs-string">"insert into chain_events (id, created_at, block_timestamp, fid, chain_id, block_number, transaction_index, log_index, type, block_hash, transaction_hash, body, raw) values ('018da97e-9c50-bd1d-0a3b-8c71bb062e74', '2024-02-14 21:21:03.508914+00', '2023-11-07 19:42:51+00', 22, 10, 111893697, 7, 24, 3, '73eaf3e303afa0d8e48ab3ac043ce60130f5061beb33314efa70d7bde5941b76', '286dd35ce467bfb3db7e8e4b036909a5e2769d0efadb6f342ef1d8970ea1e264', '{\"to\":\"0x411c9d8a1788c5764b32b44e6c381a541bfdce65\",\"eventType\":1,\"from\":\"0x\",\"recoveryAddress\":\"0x00000000fcb080a4d6c39a9354da9eb9bc104cd7\"}', '0803100a18c1b9ad35222073eaf3e303afa0d8e48ab3ac043ce60130f5061beb33314efa70d7bde5941b7628bba6aaaa063220286dd35ce467bfb3db7e8e4b036909a5e2769d0efadb6f342ef1d8970ea1e264381840165a2e0a14411c9d8a1788c5764b32b44e6c381a541bfdce651001221400000000fcb080a4d6c39a9354da9eb9bc104cd768077002')"</span>
...
</code></pre><h2 id="h-next-steps" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Next steps</h2><p>The script demonstrated a simple yet important concept: streaming data from a Postgres database to Textile&apos;s decentralized storage via the <code>vaults</code> CLI is a powerful way to make data publicly available and easily retrievable. From a programming perspective, the code could be improved <em>quite a bit</em> to handle failures, retries, and other edge cases. For example, the replicator app uses many <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://nodejs.org/api/worker_threads.html">workers</a> to execute tasks that migrate tables and backfill data to Postgres. instead of shell scripts, perhaps workers could be used for the Textile vaults logic.</p><p>Now, if you were to load <em>all</em> of the events/CIDs into something like DuckDB, you could then query the data across the entire Farcaster network. In other words, the replicated data on Textile is now available for you to use in your applications, and you can query it as you would any other SQL (or other) database!</p><p>As a developer, this could facilitate a much quicker onboarding time. For example, if the Farcaster team were to run the <code>vaults</code> streaming process indefinitely, as a public good, then any developer continually could load all of the data into their application and start building on top of the Farcaster network without having to run the replicator app themselves. This would save not only time but also costs and resources for the community—while also increasing how quickly a dev can start building on top of the network&apos;s data.</p><h3 id="h-further-development" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Further development</h3><p>If you&apos;d like to alter the Textile scripts or Docker image, you can use <code>docker build</code> to produce the Postgres + Textile image with the following commands:</p><pre data-type="codeBlock" text="docker build -t &lt;your_username&gt;/textile-vaults-farcaster:latest .
docker image push &lt;your_username&gt;/textile-vaults-farcaster
"><code>docker build <span class="hljs-operator">-</span>t <span class="hljs-operator">&#x3C;</span>your_username<span class="hljs-operator">></span><span class="hljs-operator">/</span>textile<span class="hljs-operator">-</span>vaults<span class="hljs-operator">-</span>farcaster:latest .
docker image push <span class="hljs-operator">&#x3C;</span>your_username<span class="hljs-operator">></span><span class="hljs-operator">/</span>textile<span class="hljs-operator">-</span>vaults<span class="hljs-operator">-</span>farcaster
</code></pre><p>Just be sure to update the <code>docker-compose.yml</code> file with the new image name under the <code>postgres</code> service. To support other architectures, check out <code>docker buildx</code>.</p>]]></content:encoded>
            <author>tableland-2@newsletter.paragraph.com (Tableland)</author>
            <enclosure url="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/ac3307a3809a125db81918bbd583432ae4f872d5177abfc518173f3f9084d152.jpg" length="0" type="image/jpg"/>
        </item>
        <item>
            <title><![CDATA[Basin Vaults Tracker: web3.py & GitHub Actions to Parse Contract Events for New Vaults]]></title>
            <link>https://paragraph.com/@tableland-2/basin-vaults-tracker-web3-py-github-actions-to-parse-contract-events-for-new-vaults</link>
            <guid>0AAJvwlIGkXTvVjbf6WN</guid>
            <pubDate>Thu, 01 Feb 2024 21:45:02 GMT</pubDate>
            <description><![CDATA[What is it?Textile has built an experimentally verifiable data layer called Basin, which lets anyone create “vaults” that act as a data container for arbitrary data. A vault is created by an EVM-compatible wallet (on Filecoin FVM), thus, granting the account write ownership to the vault. As data is pushed to the vault, it’s registered onchain as an append-only log for all mutations associated with the vault. We built a simple demonstration of how you can:Parse all vault creation events from t...]]></description>
            <content:encoded><![CDATA[<h2 id="h-what-is-it" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">What is it?</h2><p>Textile has built an experimentally verifiable data layer called <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/basin-cli">Basin</a>, which lets anyone create “vaults” that act as a data container for arbitrary data. A vault is created by an EVM-compatible wallet (on Filecoin FVM), thus, granting the account write ownership to the vault. As data is pushed to the vault, it’s registered onchain as an append-only log for all mutations associated with the vault.</p><p>We built a simple demonstration of how you can:</p><ul><li><p>Parse all vault creation events from the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/basin-storage/blob/main/evm/basin_storage/src/BasinStorage.sol">Basin storage</a> smart contract.</p></li><li><p>Fetch vault information from the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/basin-provider/pull/17">Basin HTTP API</a>.</p></li><li><p>Store the state for each run and a markdown file that summarizes the activity.</p></li><li><p>Run this as a GitHub action daily.</p></li></ul><blockquote><p>Check out the source code and output data files, which lets you inspect various vaults, view their event CIDs, and extract the raw data pushed to the vault as parquet:</p><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/textileio/demo-vault-tracker"><strong><em>https://github.com/textileio/demo-vault-tracker</em></strong></a></p></blockquote><h2 id="h-how-does-it-work" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">How does it work?</h2><p>The program is built in python. It uses <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://web3py.readthedocs.io/en/stable/">web3.py</a> to connect to Filecoin Calibration and then parse vault creation events from the logs at the Basin storage contract (at the address <code>0xaB16d51Fa80EaeAF9668CE102a783237A045FC37</code> ). For each log, there is information emitted that includes:</p><ul><li><p><em>owner</em>: The address of the vault creator.</p></li><li><p><em>vault</em>: The bytes of a keccak hash of the vault’s actual name.</p></li><li><p><em>block number</em>: The block in which the event/vault was created.</p></li></ul><p>To retrieve the logs, a block range is specified to get all of the logs during that period, and then a few processing steps happen:</p><ul><li><p>Parse the logs for event and block information (described above).</p></li><li><p>Fetch all vaults for the owner addresses seen during the run via the Basin HTTP API.</p></li><li><p>Store the run and vault information in a <code>state.json</code> file.</p></li><li><p>Write summary results to a data markdown file.</p></li></ul><p>These steps run with a GitHub action every day. The action runs the logic and then commits the changes automatically so that anyone can inspect all vaults and related information about the underlying data on the network.</p><h2 id="h-how-did-we-make-it" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">How did we make it?</h2><h3 id="h-overview" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Overview</h3><p>To kickstart the project, <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/cookiecutter/cookiecutter">cookiecutter</a> and <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/sourcery-ai/python-best-practices-cookiecutter">sourcery-ai</a>’s template helped set everything up with <code>pipenv</code>, linting, formatting, type hints with <code>mypy</code>, and pre-commit/push hooks. Here’s the gist of the project’s structure:</p><pre data-type="codeBlock" text=".
├── Data.md
├── abi.json
├── state.json
├── .github
│   └── workflows
│       └── daily.yaml
└── vaults_tracker
    ├── __init__.py
    ├── __main__.py
    ├── fetch.py
    └── write.py
"><code>.
├── Data.md
├── <span class="hljs-built_in">abi</span>.json
├── state.json
├── .github
│   └── workflows
│       └── daily.yaml
└── vaults_tracker
    ├── __init__.py
    ├── __main__.py
    ├── fetch.py
    └── write.py
</code></pre><ul><li><p><code>Data.md</code>: Stores all vaults ever created with relevant links to the Basin HTTP API that points to the actual data in the vault.</p></li><li><p><code>abi.json</code>: The Basin storage contract ABI, used with web3.py to connect to the contract.</p></li><li><p><code>state.json</code>: A simple datastore for each run and cumulative vault information per owner.</p></li><li><p><code>.github/workflows/daily.yaml</code>: The GitHub actions workflow file that runs the script and commits the data once per day.</p></li><li><p><code>__main__.py</code>: The entry point for the script</p></li><li><p><code>fetch.py</code>: Connects the Basin storage contract, parses events, and makes calls to the Basin HTTP API.</p></li><li><p><code>write.py</code>: Writes run information to both the <code>state.json</code> file and <code>Data.md</code> file.</p></li></ul><p>If you’d like to better understand the actual source code and setup steps, clone the repo and read over the README:</p><pre data-type="codeBlock" text="git clone https://github.com/textileio/demo-vault-tracker
"><code>git <span class="hljs-built_in">clone</span> https://github.com/textileio/demo-vault-tracker
</code></pre><h3 id="h-fetching-data-with-web3py" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Fetching data with <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="http://web3.py">web3.py</a></h3><p>The first step is to instantiate the <code>Web3</code> class with an RPC URL (Filecoin Calibration), the contract’s ABI, and the contract’s deployed address:</p><pre data-type="codeBlock" text="from json import loads
from pathlib import Path

from web3 import Web3

url = &quot;https://rpc.ankr.com/filecoin_testnet&quot;
w3 = Web3(Web3.HTTPProvider(url))

def get_contract_create_events(start_block, end_block):
    # The ABI file is in the root directory
    abi_file = Path(__file__).parent.parent / &quot;abi.json&quot;
    with open(abi_file, &quot;r&quot;) as basin_abi:
        abi = loads(basin_abi.read())
        
    # Creation events occur when `PubCreated` is emitted
    new_vault_event = &quot;PubCreated&quot;
    basin_address = Web3.to_checksum_address(
        &quot;0xaB16d51Fa80EaeAF9668CE102a783237A045FC37&quot; # On Filecoin Calibration
    )
        
    # Create a Basin contract connection
    contract = w3.eth.contract(address=basin_address, abi=abi)
"><code><span class="hljs-keyword">from</span> json <span class="hljs-keyword">import</span> <span class="hljs-title">loads</span>
<span class="hljs-title"><span class="hljs-keyword">from</span></span> <span class="hljs-title">pathlib</span> <span class="hljs-title"><span class="hljs-keyword">import</span></span> <span class="hljs-title">Path</span>

<span class="hljs-title"><span class="hljs-keyword">from</span></span> <span class="hljs-title">web3</span> <span class="hljs-title"><span class="hljs-keyword">import</span></span> <span class="hljs-title">Web3</span>

<span class="hljs-title">url</span> <span class="hljs-operator">=</span> <span class="hljs-string">"https://rpc.ankr.com/filecoin_testnet"</span>
<span class="hljs-title">w3</span> <span class="hljs-operator">=</span> <span class="hljs-title">Web3</span>(<span class="hljs-title">Web3</span>.<span class="hljs-title">HTTPProvider</span>(<span class="hljs-title">url</span>))

<span class="hljs-title">def</span> <span class="hljs-title">get_contract_create_events</span>(<span class="hljs-title">start_block</span>, <span class="hljs-title">end_block</span>):
    # <span class="hljs-title">The</span> <span class="hljs-title">ABI</span> <span class="hljs-title">file</span> <span class="hljs-title"><span class="hljs-keyword">is</span></span> <span class="hljs-title">in</span> <span class="hljs-title">the</span> <span class="hljs-title">root</span> <span class="hljs-title">directory</span>
    <span class="hljs-title">abi_file</span> <span class="hljs-operator">=</span> <span class="hljs-title">Path</span>(<span class="hljs-title">__file__</span>).<span class="hljs-title">parent</span>.<span class="hljs-title">parent</span> <span class="hljs-operator">/</span> <span class="hljs-string">"abi.json"</span>
    <span class="hljs-title">with</span> <span class="hljs-title">open</span>(<span class="hljs-title">abi_file</span>, <span class="hljs-string">"r"</span>) <span class="hljs-title"><span class="hljs-keyword">as</span></span> <span class="hljs-title">basin_abi</span>:
        <span class="hljs-title"><span class="hljs-built_in">abi</span></span> <span class="hljs-operator">=</span> <span class="hljs-title">loads</span>(<span class="hljs-title">basin_abi</span>.<span class="hljs-title">read</span>())
        
    # <span class="hljs-title">Creation</span> <span class="hljs-title">events</span> <span class="hljs-title">occur</span> <span class="hljs-title">when</span> `<span class="hljs-title">PubCreated</span>` <span class="hljs-title"><span class="hljs-keyword">is</span></span> <span class="hljs-title">emitted</span>
    <span class="hljs-title">new_vault_event</span> <span class="hljs-operator">=</span> <span class="hljs-string">"PubCreated"</span>
    <span class="hljs-title">basin_address</span> <span class="hljs-operator">=</span> <span class="hljs-title">Web3</span>.<span class="hljs-title">to_checksum_address</span>(
        <span class="hljs-string">"0xaB16d51Fa80EaeAF9668CE102a783237A045FC37"</span> # <span class="hljs-title">On</span> <span class="hljs-title">Filecoin</span> <span class="hljs-title">Calibration</span>
    )
        
    # <span class="hljs-title">Create</span> <span class="hljs-title">a</span> <span class="hljs-title">Basin</span> <span class="hljs-title"><span class="hljs-keyword">contract</span></span> <span class="hljs-title">connection</span>
    <span class="hljs-title"><span class="hljs-keyword">contract</span></span> <span class="hljs-operator">=</span> <span class="hljs-title">w3</span>.<span class="hljs-title">eth</span>.<span class="hljs-title"><span class="hljs-keyword">contract</span></span>(<span class="hljs-title"><span class="hljs-keyword">address</span></span><span class="hljs-operator">=</span><span class="hljs-title">basin_address</span>, <span class="hljs-title"><span class="hljs-built_in">abi</span></span><span class="hljs-operator">=</span><span class="hljs-title"><span class="hljs-built_in">abi</span></span>)
</code></pre><p>In our case, we’re only interested in parsing log data for <code>PubCreated</code>. This is what signals that a new vault has been created and includes the owner and vault information. Now, there is a bit of a challenge when using the <code>get_logs</code> method (shown below). Each Filecoin Calibration RPC provider implements their own limits for how far in the past you can query, and the block range for retrieving events also has a limit.</p><p>Thus, if you want to query a range greater than the block range limit, you have to chunk it up into multiple <code>get_logs</code> queries. The following shows how the result from chunking is used. An <code>events</code> list holds all events that are seen between the <code>start_block</code> and <code>end_block</code> parameters:</p><pre data-type="codeBlock" text="def get_contract_create_events(start_block, end_block):
# ...
    chunks = chunk_block_range(start_block, end_block)
    events = []
    for chunk in chunks:
        new_events = contract.events[&quot;PubCreated&quot;].get_logs(
            fromBlock=chunk[&quot;start_block&quot;], toBlock=chunk[&quot;end_block&quot;]
        )
        if new_events:
            events.append(new_events)
"><code>def get_contract_create_events(start_block, end_block):
# ...
    chunks <span class="hljs-operator">=</span> chunk_block_range(start_block, end_block)
    events <span class="hljs-operator">=</span> []
    <span class="hljs-keyword">for</span> chunk in chunks:
        new_events <span class="hljs-operator">=</span> <span class="hljs-keyword">contract</span>.events[<span class="hljs-string">"PubCreated"</span>].get_logs(
            fromBlock<span class="hljs-operator">=</span>chunk[<span class="hljs-string">"start_block"</span>], toBlock<span class="hljs-operator">=</span>chunk[<span class="hljs-string">"end_block"</span>]
        )
        <span class="hljs-keyword">if</span> new_events:
            events.append(new_events)
</code></pre><p>The chunking process involves looking at the <code>start_block</code> and <code>end_block</code>, and if these are greater than the 2880 block limit imposed by the RPC provider, then multiple batches must be created. The logic below isn’t the <em>most</em> effective way to accomplish this, but it gets the job done!</p><pre data-type="codeBlock" text="def chunk_block_range(start_block, end_block):
    block_range = end_block - start_block
    if (block_range) &gt; 2880:
        start_chunk = start_block
        end_final = end_block
        chunks = []
        while block_range &gt; 0:
            end_chunk = start_chunk + 2880  # Block range max is 2880
            if end_chunk &gt; end_final:
                end_chunk = end_final
            chunks.append({
              &quot;start_block&quot;: start_chunk, 
              &quot;end_block&quot;: end_chunk
            })
            start_chunk = end_chunk + 1
            block_range = end_final - start_chunk
        return chunks
"><code>def chunk_block_range(start_block, end_block):
    block_range <span class="hljs-operator">=</span> end_block <span class="hljs-operator">-</span> start_block
    <span class="hljs-keyword">if</span> (block_range) <span class="hljs-operator">></span> <span class="hljs-number">2880</span>:
        start_chunk <span class="hljs-operator">=</span> start_block
        end_final <span class="hljs-operator">=</span> end_block
        chunks <span class="hljs-operator">=</span> []
        <span class="hljs-keyword">while</span> block_range <span class="hljs-operator">></span> <span class="hljs-number">0</span>:
            end_chunk <span class="hljs-operator">=</span> start_chunk <span class="hljs-operator">+</span> <span class="hljs-number">2880</span>  # Block range max <span class="hljs-keyword">is</span> <span class="hljs-number">2880</span>
            <span class="hljs-keyword">if</span> end_chunk <span class="hljs-operator">></span> end_final:
                end_chunk <span class="hljs-operator">=</span> end_final
            chunks.append({
              <span class="hljs-string">"start_block"</span>: start_chunk, 
              <span class="hljs-string">"end_block"</span>: end_chunk
            })
            start_chunk <span class="hljs-operator">=</span> end_chunk <span class="hljs-operator">+</span> <span class="hljs-number">1</span>
            block_range <span class="hljs-operator">=</span> end_final <span class="hljs-operator">-</span> start_chunk
        <span class="hljs-keyword">return</span> chunks
</code></pre><p>Once all of the events are retrieved, we can then parse the data to get the vault owner’s address, a hash of the vault’s name, and the block number it occurred at.</p><pre data-type="codeBlock" text="def get_data_from_events(contract_events):
    data = []
    for events in contract_events:
        for event in events:
            args = event[&quot;args&quot;]
            owner = args[&quot;owner&quot;]
            vault = args[&quot;pub&quot;]
            block_num = event[&quot;blockNumber&quot;]
            data.append(
                {
                  &quot;owner&quot;: owner,
                  &quot;vault_hash&quot;: vault.hex(), 
                  &quot;block_num&quot;: block_num
                }
            )

    return data
"><code>def get_data_from_events(contract_events):
    <span class="hljs-attr">data</span> = []
    for events in contract_events:
        for event in events:
            <span class="hljs-attr">args</span> = event[<span class="hljs-string">"args"</span>]
            <span class="hljs-attr">owner</span> = args[<span class="hljs-string">"owner"</span>]
            <span class="hljs-attr">vault</span> = args[<span class="hljs-string">"pub"</span>]
            <span class="hljs-attr">block_num</span> = event[<span class="hljs-string">"blockNumber"</span>]
            data.append(
                {
                  "owner": owner,
                  "vault_hash": vault.hex(), 
                  "block_num": block_num
                }
            )

    return data
</code></pre><h3 id="h-calling-the-basin-http-api" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Calling the Basin HTTP API</h3><p>Now that we have the event data, we know the value of each <code>owner</code>&apos;s address from the event logs. It’s possible that the same owner created more than one vault, so we’ll want to deduplicate these values. In our <code>__main__.py</code> file, we’ll set everything up and call the methods above.</p><pre data-type="codeBlock" text="# Note: the start/end block would be defined by application logic

# Get the vault creators from the Basin contract&apos;s `PubCreated` event
create_events = get_contract_create_events(start_block, end_block)

# Get data from the logs, including vault creator, vault hash, &amp; block
data = get_data_from_events(create_events)

# Remove duplicate vault creators/owners
owners = {data[&quot;owner&quot;] for data in data}

# Get all vaults for each vault creator via Basin API
vaults = []
for owner in owners:
    vault = get_vaults(owner)
    vaults.append({owner: vault})
"><code><span class="hljs-comment"># <span class="hljs-doctag">Note:</span> the start/end block would be defined by application logic</span>

<span class="hljs-comment"># Get the vault creators from the Basin contract's `PubCreated` event</span>
<span class="hljs-attr">create_events</span> = get_contract_create_events(start_block, end_block)

<span class="hljs-comment"># Get data from the logs, including vault creator, vault hash, &#x26; block</span>
<span class="hljs-attr">data</span> = get_data_from_events(create_events)

<span class="hljs-comment"># Remove duplicate vault creators/owners</span>
<span class="hljs-attr">owners</span> = {data[<span class="hljs-string">"owner"</span>] for data in data}

<span class="hljs-comment"># Get all vaults for each vault creator via Basin API</span>
<span class="hljs-attr">vaults</span> = []
for owner in owners:
    <span class="hljs-attr">vault</span> = get_vaults(owner)
    vaults.append({owner: vault})
</code></pre><p>This gives us a way to fetch all vaults that have been created by the unique set of owners with the <code>get_vaults</code> method.</p><pre data-type="codeBlock" text="def get_vaults(address):
    url = &quot;https://basin.tableland.xyz/vaults&quot;
    params = {&quot;account&quot;: address}
    response = get(url, params=params)

    if response.status_code == 200:
        vaults = response.json()
        return vaults
    else:
        raise Exception(
            f&quot;Failed to fetch vault data: {response.status_code}&quot;,
        )
"><code>def get_vaults(<span class="hljs-keyword">address</span>):
    url <span class="hljs-operator">=</span> <span class="hljs-string">"https://basin.tableland.xyz/vaults"</span>
    params <span class="hljs-operator">=</span> {<span class="hljs-string">"account"</span>: <span class="hljs-keyword">address</span>}
    response <span class="hljs-operator">=</span> get(url, params<span class="hljs-operator">=</span>params)

    <span class="hljs-keyword">if</span> response.status_code <span class="hljs-operator">=</span><span class="hljs-operator">=</span> <span class="hljs-number">200</span>:
        vaults <span class="hljs-operator">=</span> response.json()
        <span class="hljs-keyword">return</span> vaults
    <span class="hljs-keyword">else</span>:
        raise Exception(
            f<span class="hljs-string">"Failed to fetch vault data: {response.status_code}"</span>,
        )
</code></pre><h3 id="h-storing-state-and-writing-files" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Storing state &amp; writing files</h3><p>The last step is to take the information from above and store it in a <code>state.json</code> and <code>Data.md</code> file. We’ll do the following in our <code>__main__.py</code> file:</p><ul><li><p>Create a <code>snapshot</code> dictionary with the run info (starting block, ending block, and the parsed/formatted events seen during that period).</p></li><li><p>Take the <code>snapshot</code> and index the run by the <code>end_block</code>, which will be used to ensure that each run has a unique identifier because runs are spaced out over a long period.</p></li><li><p>Write the new state, read the new/full state, and then write to a summary markdown file.</p><ul><li><p>Note: The <code>get_saved_state</code> logic just reads the <code>state.json</code> file.</p></li></ul></li></ul><p>This is what it looks like:</p><pre data-type="codeBlock" text="# Store the current snapshot/run
snapshot = {
    &quot;start_block&quot;: start_block,
    &quot;end_block&quot;: end_block,
    &quot;events_data&quot;: data,
}
# Index the run by the `end_block` number
new_run = {end_block: snapshot}

# Write the new run and cumulative vaults files
write_to_state(new_run, vaults)
updated_state = get_saved_state()
write_to_markdown(updated_state, end_block)
"><code><span class="hljs-comment"># Store the current snapshot/run</span>
<span class="hljs-attr">snapshot</span> = {
    "start_block": start_block,
    "end_block": end_block,
    "events_data": data,
}
<span class="hljs-comment"># Index the run by the `end_block` number</span>
<span class="hljs-attr">new_run</span> = {end_block: snapshot}

<span class="hljs-comment"># Write the new run and cumulative vaults files</span>
write_to_state(new_run, vaults)
<span class="hljs-attr">updated_state</span> = get_saved_state()
write_to_markdown(updated_state, end_block)
</code></pre><p>You can check out the source code for the details behind the last couple of steps—there are plenty of inline comments and docs to explain what’s going on. The resulting state file will resemble the following:</p><pre data-type="codeBlock" text="{
    &quot;vaults&quot;: [
        {
            &quot;0x635285e3d83ba6723E7b23840b324b2a7A6532bC&quot;: [
                &quot;wxm_t1.data&quot;,
            ...
            ]
        },
        {
            &quot;0x00FEEc1fC91074f5F38a8FC5129dbc4FD204eca6&quot;: [
                &quot;bruno2.test&quot;,
            ...
            ]
        },
        {
            &quot;0x0c9CE72E9c30a0ebC61f976e8254a4F58D276248&quot;: [
                &quot;avichalp.test_01_24_24&quot;,
            ...
            ]
        }
    ],
    &quot;runs&quot;: {
        &quot;1296221&quot;: {
            &quot;start_block&quot;: 1076346,
            &quot;end_block&quot;: 1296221,
            &quot;events_data&quot;: [
                {
                    &quot;owner&quot;: &quot;0x635285e3d83ba6723E7b23840b324b2a7A6532bC&quot;,
                    &quot;vault_hash&quot;: &quot;a49518b3dfc33a3c7091c0821c2b9979b3bfd6f8d3f7f354a1747eedfb19d0c4&quot;,
                    &quot;block_num&quot;: 1289156
                },
                ...
            ]
        }
    }
}
"><code>{
    <span class="hljs-string">"vaults"</span>: [
        {
            <span class="hljs-string">"0x635285e3d83ba6723E7b23840b324b2a7A6532bC"</span>: [
                <span class="hljs-string">"wxm_t1.data"</span>,
            ...
            ]
        },
        {
            <span class="hljs-string">"0x00FEEc1fC91074f5F38a8FC5129dbc4FD204eca6"</span>: [
                <span class="hljs-string">"bruno2.test"</span>,
            ...
            ]
        },
        {
            <span class="hljs-string">"0x0c9CE72E9c30a0ebC61f976e8254a4F58D276248"</span>: [
                <span class="hljs-string">"avichalp.test_01_24_24"</span>,
            ...
            ]
        }
    ],
    <span class="hljs-string">"runs"</span>: {
        <span class="hljs-string">"1296221"</span>: {
            <span class="hljs-string">"start_block"</span>: <span class="hljs-number">1076346</span>,
            <span class="hljs-string">"end_block"</span>: <span class="hljs-number">1296221</span>,
            <span class="hljs-string">"events_data"</span>: [
                {
                    <span class="hljs-string">"owner"</span>: <span class="hljs-string">"0x635285e3d83ba6723E7b23840b324b2a7A6532bC"</span>,
                    <span class="hljs-string">"vault_hash"</span>: <span class="hljs-string">"a49518b3dfc33a3c7091c0821c2b9979b3bfd6f8d3f7f354a1747eedfb19d0c4"</span>,
                    <span class="hljs-string">"block_num"</span>: <span class="hljs-number">1289156</span>
                },
                ...
            ]
        }
    }
}
</code></pre><h2 id="h-how-can-someone-use-it" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">How can someone use it?</h2><p>There are a couple of ways anyone can work with this information. You could:</p><ul><li><p>Inspect the <code>Data.md</code> file (<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/dtbuchholz/vaults-tracker/blob/main/Data.md">here</a>) <em>Events</em> columns and use the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/basin-cli">Basin CLI</a> or API to retrieve the vault mutation events/CIDs—these can be extracted as parquet files to get the raw data pushed to the vault.</p></li><li><p>Produce your own code that runs through similar logic and lets you build an automated index of onchain events with GitHub actions.</p></li></ul><h2 id="h-learn-more" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Learn more</h2><p>If you found this interesting, dive into our <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tableland.xyz/discord">Discord</a> and let us know! We also have a weekly newsletter called Weeknotes, which features the latest updates about the protocol. You can check it out <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tableland.substack.com/">here</a>.</p>]]></content:encoded>
            <author>tableland-2@newsletter.paragraph.com (Tableland)</author>
            <enclosure url="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/f967b8e8a63e77c20b3368ea2b439548e821b2123f9d192b624a2e616c11a073.png" length="0" type="image/png"/>
        </item>
        <item>
            <title><![CDATA[Solutions for DePIN Data Management & Collaboration]]></title>
            <link>https://paragraph.com/@tableland-2/solutions-for-depin-data-management-collaboration</link>
            <guid>IinMZ6xUuTCSapEii8yz</guid>
            <pubDate>Mon, 18 Dec 2023 17:04:29 GMT</pubDate>
            <description><![CDATA[DePINs are transforming the landscape of physical infrastructure, but managing their vast and diverse data sets efficiently and securely remains a major challenge. At Tableland, we&apos;re tackling this head-on, pioneering innovative solutions that can reshape DePIN data management, and enable enable new, more useful ways of utilizing data.Tableland Basin v2: Scaling Data in TablelandIf you’ve followed the progress of Basin development you are aware that we are excited about the potential of ...]]></description>
            <content:encoded><![CDATA[<p>DePINs are transforming the landscape of physical infrastructure, but managing their vast and diverse data sets efficiently and securely remains a major challenge. At Tableland, we&apos;re tackling this head-on, pioneering innovative solutions that can reshape DePIN data management, and enable enable new, more useful ways of utilizing data.</p><h3 id="h-tableland-basin-v2-scaling-data-in-tableland" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Tableland Basin v2: Scaling Data in Tableland</h3><p>If you’ve followed the progress of <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/basin-cli">Basin development</a> you are aware that we are excited about the potential of making large scale, community driven data available, verifiable, and secure. Our recent Phase 2 updates address key DePIN data challenges:</p><ul><li><p><strong>Enhanced Deal Addition:</strong> Robust data transactions even under network stress, thanks to a retry mechanism with incremented nonce.</p></li><li><p><strong>Strategic Transition to PostgreSQL:</strong> Enhanced data storage and retrieval capabilities, reflecting foundational improvements in scaling.</p></li><li><p><strong>Advanced HTTP API and Endpoints:</strong> Streamlined data interaction, simplifying how clients find, fetch, and verify data availability and completeness.</p></li><li><p><strong>Flexible Timestamped Data &amp; Novel Hashing Technique:</strong> Fortified data verifiability and traceability, essential for DePIN developers, resonating with strides in verifiable data technology.</p></li><li><p><strong>Advanced Caching Feature</strong>: The final, and perhaps most important feature we’ve introduced is an advanced caching feature that significantly improves data retrieval efficiency. This feature allows users to set a period of time where the data will be warmed for retrieval, ensuring quicker response times for critical applications and protocol functions. Designed to support large-scale data operations, this caching mechanism is particularly beneficial for DePIN projects with substantial and frequent data interactions (e.g., reward functions), streamlining the overall data management process.</p></li></ul><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/search?q=topic%3Abasin+org%3Atablelandnetwork+fork%3Atrue&amp;type=repositories">Follow development here</a>.</p><h3 id="h-weatherxm-a-case-study-in-data-management" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">WeatherXM: A Case Study in Data Management</h3><p>Our partnership with <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://weatherxm.com/">WeatherXM</a>, a community-driven weather network, showcases how Basin&apos;s solutions can efficiently manage community-sourced data. We ensure the availability, integrity, and longevity of critical weather data through hot and cold storage solutions. This collaboration paves the way for future DePIN partnerships. <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://mirror.xyz/tableland.eth/jxWM36j7OaW4plfTlpQpgRrIDuth75lJhVjj0BiFwBA">Read more here</a>.</p><p>We’re kicking off work with new partners, so stay tuned for new projects in the coming months.</p><h3 id="h-the-basin-demo-showcasing-data-usage" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">The Basin Demo: Showcasing Data Usage</h3><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/Filecoin/status/1731745941751820682">https://twitter.com/Filecoin/status/1731745941751820682</a></p><p>More than just a technical showcase, a new <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/dtbuchholz/basin-wxm-fetch/">Basin demo</a> is a glimpse into how DePINs can open their data to real users, easily. This Python and DuckDB-based demonstration illustrates how developers can leverage Basin to efficiently analyze WeatherXM data.</p><p><strong>Demo Highlights:</strong></p><ul><li><p><strong>Workflow Integration:</strong> WeatherXM seamlessly integrates Basin&apos;s signing and replication of large data files containing vital weather data into their workflow.</p></li><li><p><strong>Python and Polars Utilization:</strong> The project leverages Python and Polars for fast and efficient analysis, showcasing Basin&apos;s compatibility with familiar web2 workflows.</p></li><li><p><strong>Basin CLI Implementation:</strong> The script uses Basin CLI commands to access information about signed data.</p></li><li><p><strong>Optimized Data Processing:</strong> Utilizing DuckDB (or Polars&apos; &quot;lazy&quot; and &quot;streaming&quot; features), the demonstration achieves significant performance improvements, showcasing the opportunity for managing large data sets over a network.</p></li><li><p><strong>GitHub Actions:</strong> The demo uses a weekly cron-based GitHub action, automatically analyzing the latest network data to calculate new summary statistics. Using actions helps us showcase how web3 protocols can help power common web2 workflows that leverage authentic and independently verifiable data.</p></li></ul><h3 id="h-the-roadmap-empowering-depin-developers" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">The Roadmap: Empowering DePIN Developers</h3><p>Our vision is a future where DePIN data is not just stored but is verifiable, traceable, and collaborative, aligning with the broader trends in blockchain.</p><p><strong>Join the DePIN Data Revolution</strong></p><p>We invite DePIN developers to explore Basin&apos;s solutions, feel free to <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://calendly.com/marlatextile/30min">book some time</a> to discuss our solutions and how they can fit in your stack (<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://calendly.com/marlatextile/30min">calendar</a>). Stay updated by <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tableland.substack.com/">subscribing to our week notes</a>, following us on <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/tableland">social media</a> (<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/tableland">twitter</a>, <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://warpcast.com/tableland">farcaster</a>), or <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tableland.xyz/discord">joining our Discord</a> community. Discover WeatherXM&apos;s journey by following them on <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/WeatherXM">social media</a> and <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://weatherxm.com/discord">joining their Discord</a>.</p><p><strong>A Glimpse of the Future</strong></p><p>Recent innovative applications and enhanced smart contract controls, highlighted in the Basin demo and underscored by a recent tweet from Tableland, showcase the transformative potential of Basin and Tableland, and open new doors in the blockchain world.</p><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/tableland/status/1732453296361615483">https://twitter.com/tableland/status/1732453296361615483</a></p><p><strong>Join us in building a more connected, transparent, and trusted digital world, where data management empowers innovation and growth in the DePIN sector.</strong></p>]]></content:encoded>
            <author>tableland-2@newsletter.paragraph.com (Tableland)</author>
            <enclosure url="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/78fe721c66ec06d8fec92084985be454ff268774eaf67298257b03451aeaef9c.png" length="0" type="image/png"/>
        </item>
        <item>
            <title><![CDATA[WeatherXM Partner Update]]></title>
            <link>https://paragraph.com/@tableland-2/weatherxm-partner-update</link>
            <guid>RKA1r1aI5tyybzWOKdHD</guid>
            <pubDate>Tue, 31 Oct 2023 14:47:47 GMT</pubDate>
            <description><![CDATA[In our continuous endeavor to revolutionize how decentralized peer-to-peer projects manage and utilize data, we&apos;re proud to shine a light on our collaboration with WeatherXM, a forward-thinking community-driven weather network. In late August we shared a bit about the work we’re doing with DePIN organizations to build solutions that simplify data aggregation, verifiability, and collaboration. Since then, we’ve laid the foundation for something much bigger for DePINs, distributed networks...]]></description>
            <content:encoded><![CDATA[<p>In our continuous endeavor to revolutionize how decentralized peer-to-peer projects manage and utilize data, we&apos;re proud to shine a light on our collaboration with WeatherXM, a forward-thinking community-driven weather network.</p><p>In late August <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://mirror.xyz/tableland.eth/UJMDvy8vAqekOGpPmspRGzdl2-osjPeGVaIv2Lv4Oh0">we shared a bit about the work we’re doing with DePIN organizations</a> to build solutions that simplify data aggregation, verifiability, and collaboration. Since then, we’ve laid the foundation for something much bigger for DePINs, distributed networks, and open data projects alike.</p><p>WeatherXM stands as a beacon for community empowerment, rewarding contributors for sharing pivotal weather data. But with global contributions come the challenges of storing vast amounts of data, ensuring its availability when needed, and safeguarding it for posterity. Here&apos;s where our partnership comes in.</p><h2 id="h-hot-and-cold-storage" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Hot &amp; Cold Storage</h2><p>Our initial pilot with WeatherXM leverages the power of our availability solutions paired with Filecoin for long-term storage. Every byte of data, from a temperature spike in Cairo to a drizzle in Seattle, is systematically categorized and stored where all can see. While the hot storage ensures immediate access for real-time applications and urgent analyses, our Filecoin component is the vault, preserving historical and archival data, all underpinned by the reliability of the Filecoin blockchain.</p><p>In this demo, you can check out an early version of what we’re building, showcasing how simple it is to automatically sync and store changes from your existing database to Filecoin:</p><div data-type="youtube" videoId="-VenMvrBt5A">
      <div class="youtube-player" data-id="-VenMvrBt5A" style="background-image: url('https://i.ytimg.com/vi/-VenMvrBt5A/hqdefault.jpg'); background-size: cover; background-position: center">
        <a href="https://www.youtube.com/watch?v=-VenMvrBt5A">
          <img src="{{DOMAIN}}/editor/youtube/play.png" class="play"/>
        </a>
      </div></div><h2 id="h-where-were-going" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Where we&apos;re going</h2><p>Data availability is just the beginning. As we integrate these storage solutions, we&apos;re laying the foundation for a broader vision. A future where every piece of data in the WeatherXM network is not just stored, but verifiable, traceable, and collaborative. We&apos;re envisioning a world where:</p><ul><li><p>Every data point can be certified, ensuring its authenticity.</p></li><li><p>The authenticity of data sets can be verified without direct access to the raw data.</p></li><li><p>Collaboration becomes second nature, with data effortlessly flowing between stakeholders.</p></li></ul><p>In our partnership with WeatherXM, we&apos;re not merely offering a storage solution; we&apos;re laying down the tracks for a future express train of data verifiability, transparency, and collaboration.</p><p>Stay tuned as we journey further, crafting a decentralized world that&apos;s not just interconnected but also holds the promise of unparalleled trust and collaboration.</p><h2 id="h-connect-with-us" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Connect with us</h2><h3 id="h-tableland" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Tableland</h3><p>If you want to discuss data challenges or opportunities you’re exploring, or simply want to learn more, <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://calendly.com/marlatextile/30min">set up a call with us here</a>. To stay up to date on what we’re up to, <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tableland.substack.com/">subscribe to our weekly updates here</a>, give us a <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/tableland">follow us on X</a> and drop into our <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tableland.xyz/discord">Discord</a>.</p><h3 id="h-weatherxm" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">WeatherXM</h3><p>To learn more about <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://weatherxm.com">WeatherXM</a> and join their community, <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/WeatherXM">follow them on X</a> or <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://discord.gg/DqpStMC5">jump into their Discord</a>.</p>]]></content:encoded>
            <author>tableland-2@newsletter.paragraph.com (Tableland)</author>
        </item>
        <item>
            <title><![CDATA[Fostering Verifiable Trust: Showcasing the Open Data Hack Winner 'Tableland Attestation Service']]></title>
            <link>https://paragraph.com/@tableland-2/fostering-verifiable-trust-showcasing-the-open-data-hack-winner-tableland-attestation-service</link>
            <guid>PWR3NDatTM02SoR9CGPw</guid>
            <pubDate>Wed, 18 Oct 2023 19:57:24 GMT</pubDate>
            <description><![CDATA[The recent Open Data Hack hosted by Encode Club served as a crucible for innovative solutions addressing pivotal challenges within the decentralized landscape. Among the groundbreaking projects unveiled was the Tableland Attestation Service (TAS), led by Nick Lionis. TAS has been introduced to significantly bolster trust and verification within the Filecoin network. TAS emerges as a structured attestation service, elegantly anchored on the robust infrastructure of Tableland. Its objective is ...]]></description>
            <content:encoded><![CDATA[<p>The recent Open Data Hack hosted by Encode Club served as a crucible for innovative solutions addressing pivotal challenges within the decentralized landscape. Among the groundbreaking projects unveiled was the Tableland Attestation Service (TAS), led by Nick Lionis. TAS has been introduced to significantly bolster trust and verification within the Filecoin network.</p><p>TAS emerges as a structured attestation service, elegantly anchored on the robust infrastructure of Tableland. Its objective is to instill a framework of trust by adeptly indexing user attestations through Tableland, thus paving the way for seamless interactions across various EVM chains. By catering to the pressing need for a reliable attestation service within the Filecoin network, TAS aspires to establish a new benchmark for trust-enhanced protocols in the blockchain realm.</p><p>The discussion below with Nick delves into the core objectives, development, and broader vision of TAS. Through his insights, we explore the potential of TAS as a cornerstone for trust-enhanced applications within the Filecoin network and beyond, reflecting the ethos of innovation that the Open Data Hack event sought to inspire.<br><br><strong>Where are you from?</strong><br>I am Nick Lionis from Greece, specifically from Athens, where my journey into the fascinating world of web3 and decentralized technologies began three years ago.</p><p><strong>How did you get interested in crypto?</strong><br>My interest in crypto stemmed from a desire to explore decentralization and its potential to reshape existing systems. This curiosity drove me to delve deeper into various protocols.</p><p><strong>What got you interested in Tableland?</strong><br>My fascination with Tableland began when I started researching protocols related to IPFS and Filecoin for my thesis. Discovering Textile, the creators of Tableland, and diving into their first YouTube video on their stack ignited my interest in Tableland’s protocol.</p><p><strong>What pain point are you trying to solve with your hack?</strong><br>TAS addresses a significant pain point by introducing a robust attestation service on the Filecoin network. Tableland, the main protocol leveraged by TAS, facilitates efficient indexing of user attestations using SQL. This enables complex queries and easy expansion to all EVM chains supported by Tableland, making querying user attestations across various chains seamless and efficient.</p><p><strong>What is your solution?</strong><br>My solution addresses the need for a robust attestation service within the Filecoin network. The Tableland Attestation Service (TAS) acts as this solution, providing a structured and efficient way to handle attestations, index user data, and allow seamless expansion across various EVM chains. TAS utilizes the power of Tableland protocol to ensure data organization and accessibility, contributing to a more trustable and efficient Filecoin ecosystem.</p><p><strong>Why do you think users might be interested in your project?</strong><br>TAS significantly enriches the Filecoin ecosystem by serving as a foundational protocol for other applications within the network. It brings immense value by addressing challenges related to Storage providers and Filecoin Actors&apos; reputation. Additionally, it enables verifiable credentials and opens doors to a multitude of exciting use cases.</p><p><strong>What are you planning to work on next?</strong><br>My immediate plans involve developing an SDK to empower others to build on top of TAS. Simultaneously, I aim to create comprehensive documentation to facilitate an easier understanding and adoption of TAS by developers and enthusiasts.<br><br>Get started with TAS at their <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tas.vercel.app/">website</a>, connect with them on <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/TAS-Protocol/TAS">GitHub</a> and <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/TAS_Protocol">X</a>, and say hey to the founder Nick on <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/LionisNick">X</a> or in <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://discord.gg/NHZjTyzG">Discord</a>.</p>]]></content:encoded>
            <author>tableland-2@newsletter.paragraph.com (Tableland)</author>
            <enclosure url="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/3fd330da2b5606a24f82a773affc40f512b577c67353298cbbaa0aeae1ae0879.png" length="0" type="image/png"/>
        </item>
        <item>
            <title><![CDATA[Bridging Real and Virtual Worlds: Showcasing the Open Data Hack Winner '3D Print Near Me']]></title>
            <link>https://paragraph.com/@tableland-2/bridging-real-and-virtual-worlds-showcasing-the-open-data-hack-winner-3d-print-near-me</link>
            <guid>soquWpZqcE1Xp1H1y9kx</guid>
            <pubDate>Wed, 18 Oct 2023 18:32:36 GMT</pubDate>
            <description><![CDATA[In the ever-evolving world of decentralized applications (dApps), the recent Open Data Hack hosted by Encode Club bore witness to the emergence of a novel project, “3D Print Near Me,” conceived by Dominic from Trinidad and Tobago, and Karthikeyan from Singapore. Their initiative showcases what becomes possible when imaginative minds intersect with the enabling environment provided by Tableland. Dominic highlighted the value of Tableland, stating, "This is a needed technology to make dApps wor...]]></description>
            <content:encoded><![CDATA[<p>In the ever-evolving world of decentralized applications (dApps), the recent Open Data Hack hosted by Encode Club bore witness to the emergence of a novel project, “3D Print Near Me,” conceived by Dominic from Trinidad and Tobago, and Karthikeyan from Singapore. Their initiative showcases what becomes possible when imaginative minds intersect with the enabling environment provided by Tableland.</p><p>Dominic highlighted the value of Tableland, stating, &quot;This is a needed technology to make dApps work better,&quot; a testament to how “3D Print Near Me” harnessed our platform to simplify 3D printing. Through their marketplace, they aim to connect individuals in need of 3D printing services with local printer owners, thereby breaking down the technical and financial barriers associated with 3D printing.</p><p>In the following Q&amp;A, Dominic and Karthikeyan share insights into their backgrounds, the inception of their project, and their ambitious plans for the future, offering a glimpse into the expansive potential that lies at the intersection of decentralized data solutions and real-world challenges.</p><p><strong>Where are you guys from?</strong> <br>Dominic hails from the twin-island country of Trinidad and Tobago. Karthikeyan is based in Singapore.</p><p><strong>How did you get interested in crypto?</strong> <br>Originally through a mutual friend, and then in researching the space.</p><p><strong>What got you interested in Tableland?</strong> <br>Tableland was the ideal tool for our dApp. We needed a data store with SQL-like capabilities to store our print orders and marketplace listings. It made it easier to query and display all of this information on our dApp.</p><p><strong>What pain point are you trying to solve with your hack?</strong> <br>3D printing is fun but it can be technically difficult. Users can enjoy the benefits of 3D printing without the hassle. Some users may not want to purchase a 3D printer and not fully utilize it.</p><p><strong>What is your solution?</strong> <br>3D Print Near Me is a marketplace for 3D objects. Our platform facilitates printing 3D objects in your local area, acting as the Uber and Airbnb of 3D printing. It allows anyone with a 3D printer to earn money by offering it as a service.</p><p><strong>Why do you think users might be interested in your project?</strong> <br>We believe users will be interested in this project because 3D printing is fun and there is a lot of utility in it. 3D Print Near Me aims to give people the ability to utilize the limitless possibilities of 3D printing without having to bear the upfront expense of purchasing a printer.</p><p><strong>What are you planning to work on next?</strong> <br>We would like to get into an accelerator program to develop the product further, and then to get funding and launch it!</p><p>Connect with 3D Print Near Me on <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/dominichackett/3dprintnearme/tree/opendatahack">GitHub</a> and <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/Dominichackett">X</a>, and explore more on <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.tableland.xyz/fundamentals/use-cases#application-data">Tableland&apos;s role</a> in the dApp ecosystem.</p>]]></content:encoded>
            <author>tableland-2@newsletter.paragraph.com (Tableland)</author>
            <enclosure url="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/32a4dac99b2b17007e947467c192d9611349c59554e2e30c9fceee15965dd96f.png" length="0" type="image/png"/>
        </item>
        <item>
            <title><![CDATA[SQLite & Database Tooling]]></title>
            <link>https://paragraph.com/@tableland-2/sqlite-database-tooling</link>
            <guid>owct9SRRw7miBZ7sCo1E</guid>
            <pubDate>Fri, 29 Sep 2023 19:08:06 GMT</pubDate>
            <description><![CDATA[The Tableland team has been doing deep research the past few months. We’re amidst some new protocol capabilities and recently wrote about how DIMO and WeatherXM plan to use it—check out the blog post we wrote for a little alpha! Throughout this process, we looked at a number of SQLite tools and wanted to pass along some high-level research as we found them quite interesting!SQLite ToolsThe following list defines the projects we came across and various notes about them, including consensus typ...]]></description>
            <content:encoded><![CDATA[<p>The Tableland team has been doing deep research the past few months. We’re amidst some new protocol capabilities and recently wrote about how DIMO and WeatherXM plan to use it—check out the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://mirror.xyz/tableland.eth/UJMDvy8vAqekOGpPmspRGzdl2-osjPeGVaIv2Lv4Oh0">blog post we wrote</a> for a little alpha! Throughout this process, we looked at a number of SQLite tools and wanted to pass along some high-level research as we found them quite interesting!</p><h2 id="h-sqlite-tools" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">SQLite Tools</h2><p>The following list defines the projects we came across and various notes about them, including consensus type (<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://developer.hashicorp.com/consul/docs/architecture/consensus">Raft</a>, <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://developer.hashicorp.com/consul/docs/intro">Consul</a>, or N/A), features (sync, replication, recovery, hosted, publish, or GUI), whether or not it’s open sourced, and potential usage highlights/limitations for our use case. It’s not meant to be fully comprehensive, but it just might be useful to folks looking for database products/solutions.</p><h3 id="h-litesync" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">LiteSync</h3><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://litesync.io/en/">LiteSync</a> allows your applications to easily keep their SQLite databases synchronized.</p><ul><li><p><strong>Features</strong>: replication, sync</p></li><li><p><strong>Open Source?</strong>: no</p></li><li><p><strong>Consensus</strong>: Raft</p></li><li><p><strong>Usage</strong>: Simple to integrate, does replication, sync, and encryption. Full sync between replicas. Directly operate via SQLite.</p></li></ul><h3 id="h-marmot" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Marmot</h3><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://maxpert.github.io/marmot/">Marmot</a> is a distributed SQLite replicator.</p><ul><li><p><strong>Features</strong>: replication, sync</p></li><li><p><strong>Open Source?</strong>: yes</p></li><li><p><strong>Consensus</strong>: Raft</p></li><li><p><strong>Usage</strong>: Change detection running on NATS (<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://nats.io/">https://nats.io</a>). Appears to support sharding. Uses RAFT via NATS for conflict resolution. Last writer wins. NATS appears to be something like LibP2P, but with connection and scalability tooling built in.</p></li></ul><h3 id="h-rqlite" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">rqlite</h3><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/rqlite/rqlite">rqlite</a> is an easy-to-use, lightweight, distributed relational database, which uses <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.sqlite.org/">SQLite</a> as its storage engine.</p><ul><li><p><strong>Features</strong>: replication, sync</p></li><li><p><strong>Open Source?</strong>: yes</p></li><li><p><strong>Consensus</strong>: Raft</p></li><li><p><strong>Usage</strong>: Full replication across replicas. Each replica can write, uses RAFT for consistency. However, the API wraps SQLite.</p></li></ul><h3 id="h-dqlite" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">DQLite</h3><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://dqlite.io">Dqlite</a> is a fast, embedded, persistent SQL database with Raft consensus that is perfect for fault-tolerant IoT and Edge devices.</p><ul><li><p><strong>Features</strong>: replication, sync</p></li><li><p><strong>Open Source?</strong>: yes</p></li><li><p><strong>Consensus</strong>: Raft</p></li><li><p><strong>Usage</strong>: Supported by Canonical (folks behind Ubuntu). Wraps SQLite and includes networking and RAFT consensus. While it does sync, it isn’t bi-directional like other projects. This is purely for scaling and still uses a client-server architecture. However, the API wraps SQLite.</p></li></ul><h3 id="h-litefs" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">LiteFS</h3><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/superfly/litefs">LiteFS</a> is a FUSE-based file system for replicating SQLite databases across a cluster of machines.</p><ul><li><p><strong>Features</strong>: replication, sync</p></li><li><p><strong>Open Source?</strong>:</p></li><li><p><strong>Consensus</strong>: Raft, Consul</p></li><li><p><strong>Usage</strong>: Replicating database state across a cluster of replicas. Uses <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.consul.io/">Consul</a> for leader election, and projects like <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/walterwanderley/litefs-raft">litefs-raft</a> provide things like RAFT.</p></li></ul><h3 id="h-litestream" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Litestream</h3><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://litestream.io/">Litestream</a> is a standalone disaster recovery tool for SQLite.</p><ul><li><p><strong>Features</strong>: replication, recovery</p></li><li><p><strong>Open Source?</strong>: yes</p></li><li><p><strong>Consensus</strong>: N/A</p></li><li><p><strong>Usage</strong>: Litestream is a streaming replication tool for SQLite databases. It runs as a separate background process and continuously copies write-ahead log pages from disk to one or more replicas. It does not do sync.</p></li></ul><h3 id="h-sqlite3-sync" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">SQLite3 Sync</h3><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/moisseev/sqlite3-sync">SQLite3 Sync</a> is a live SQLite3 database master-slave replication with <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/moisseev/sqlite3-rdiff">sqlite3-rdiff</a> using rsync over SSH.</p><ul><li><p><strong>Features</strong>: replication, recovery, sync</p></li><li><p><strong>Open Source?</strong>: yes</p></li><li><p><strong>Consensus</strong>: N/A</p></li><li><p><strong>Usage</strong>: Live SQLite3 database master-slave replication using rsync over SSH.</p></li></ul><h3 id="h-litereplica" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Litereplica</h3><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://litereplica.io/">Litereplica</a> allows your applications to easily keep replicas of SQLite databases</p><ul><li><p><strong>Features</strong>: replication, recovery</p></li><li><p><strong>Open Source?</strong>:</p></li><li><p><strong>Consensus</strong>: no</p></li><li><p><strong>Usage</strong>: Litereplica is a library that provides master-slave replication with automatic failover. Appears to be related to the Litesync project.</p></li></ul><h3 id="h-symmetricds" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">SymmetricDS</h3><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.symmetricds.org/">SymmetricDS</a> is open source database replication software that focuses on features and cross platform compatibility.</p><ul><li><p><strong>Features</strong>: replication, sync</p></li><li><p><strong>Open Source?</strong>: yes</p></li><li><p><strong>Consensus</strong>: N/A</p></li><li><p><strong>Usage</strong>: Generic database replication software that focuses on features and cross platform compatibility. This isn’t specific to SQLite per se, but does replicate SQLite as a feature.</p></li></ul><h3 id="h-mycelial" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Mycelial</h3><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://mycelial.com/">Mycelial</a> is the one-click data backbone for Edge AI/ML, Cloud ETL, and everything in between.</p><ul><li><p><strong>Features</strong>: replication, sync</p></li><li><p><strong>Open Source?</strong>: yes</p></li><li><p><strong>Consensus</strong>: N/A</p></li><li><p><strong>Usage</strong>: SQLite extension that connects a remote hub that can sync many readers on your DB.</p></li></ul><h3 id="h-sqlite-cloud" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">SQLite Cloud</h3><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://sqlitecloud.io/">SQLite Cloud</a> is a is a distributed relational database system that ensures strong consistency across all nodes in a cluster.</p><ul><li><p><strong>Features</strong>: replication, hosted</p></li><li><p><strong>Open Source?</strong>: no</p></li><li><p><strong>Consensus</strong>: N/A</p></li><li><p><strong>Usage</strong>: Fully hosted SQLite DBs with some auto-scaling, auto-replication things built into the dash.</p></li></ul><h3 id="h-dbhub" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">DBHub</h3><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://dbhub.io/">DBHub</a> is a Cloud for SQLite databases.</p><ul><li><p><strong>Features</strong>: publish</p></li><li><p><strong>Open Source?</strong>: yes</p></li><li><p><strong>Consensus</strong>: N/A</p></li><li><p><strong>Usage</strong>: Create private and public datasets. Integration for <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="http://sqlitebrowser.org/">DB Browser for SQLite</a>. CLI interface. Automatic alerts for data changes. Collaborative development.</p></li></ul><h3 id="h-sqlite-browser" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">SQLite Browser</h3><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://sqlitebrowser.org/">SQLite Browser</a> is a high quality, visual, open source tool to create, design, and edit database files compatible with SQLite.</p><ul><li><p><strong>Features</strong>: GUI, hosted</p></li><li><p><strong>Open Source?</strong>: yes</p></li><li><p><strong>Consensus</strong>: N/A</p></li><li><p><strong>Usage</strong>: Online viewer that integrates with DB Hub. Simple visualization tool for SQLite databases.</p></li></ul><h3 id="h-turso" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Turso</h3><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://turso.tech/">Turso</a> is an edge-hosted, distributed database based on <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://libsql.org/">libSQL</a>, an open-source and open-contribution fork of <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://sqlite.org/">SQLite</a>.</p><ul><li><p><strong>Features</strong>: replication, hosted</p></li><li><p><strong>Open Source?</strong>: yes</p></li><li><p><strong>Consensus</strong>: N/A</p></li><li><p><strong>Usage</strong>: Turso is an edge-hosted, distributed database based on SQLite. Designed to minimize query latency via intelligent geo-replication. A major selling feature is its integration with platforms like CloudFlare, Netlify, and Vercel. <em>Some</em> of it is open source.</p></li></ul><h3 id="h-cloudflare-d1" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Cloudflare D1</h3><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://developers.cloudflare.com/d1">Cloudflare D1</a> is Cloudflare’s native serverless SQL database.</p><ul><li><p><strong>Features</strong>: replication, hosted</p></li><li><p><strong>Open Source?</strong>: no</p></li><li><p><strong>Consensus</strong>: N/A</p></li><li><p><strong>Usage</strong>: D1 is Cloudflare’s native serverless database. It is based on SQLite, and integrates with Cloudflare Workers. (In fact, Tableland’s <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.tableland.xyz/sdk/">SDK</a> is fully D1-compatible!)</p></li></ul><h2 id="h-awesome-decentralized-database" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Awesome Decentralized Database</h2><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/awesome-decentralized-database">Awesome Decentralized Database</a> is a curated list of the best decentralized database solutions, tools, resources, and learning materials that empower developers to build data-driven applications on the decentralized web. (Note: if you see anything missing, PRs are welcome!)****</p><p>You can check out the repo linked above for the details, but a current snapshot of what we’ve put together is outlined below.</p><h2 id="h-relational-databases" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Relational Databases</h2><h3 id="h-blockchain" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Blockchain</h3><ul><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/aergoio/aergolite">AergoLite</a> - AergoLite: SQLite with Blockchain.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/pixelspark/catena">Catena</a> - Catena is a distributed database based on a blockchain, accessible using SQL.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/ivoras/daisy">Daisy</a> - Private proof of authority blockchain where blocks are SQLite databases, in Go.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/go-tableland/">Tableland</a> - Decentralized SQLite database that provides provides wallet-based authentication, row-level permissions, and high performance at a low cost.</p></li></ul><h3 id="h-ipfs" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">IPFS</h3><ul><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/ipfs/go-bs-sqlite3">go-bs-sqlite3</a> - IPFS blockstore.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/Actyx/ipfs-sqlite-block-store">ipfs-sqlite-block-store</a> - IPFS blockstore.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/jtsmedley/ipfs-sqlite-cli">ipfs-sqlite-cli</a> - CLI utility to backup and restore a database from IPFS. Requires local instance of GO-IPFS running.</p></li></ul><h3 id="h-peer-to-peer" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Peer-to-Peer</h3><ul><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/vlcn-io/cr-sqlite">CR-SQLite</a> - CR-SQLite is a run-time loadable extension for SQLite and libSQL. It allows merging different SQLite databases together that have taken independent writes.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/IceFireDB/IceFireDB/tree/main/IceFireDB-SQLite">IceFireDB</a> - IceFireDB-SQLite database is a decentralized SQLite database.</p></li></ul><h3 id="h-raft" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Raft</h3><ul><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/chiselstrike/chiselstore">ChiselStore</a> - ChiselStore is an embeddable, distributed SQLite for Rust, powered by Little Raft.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/uglyer/ha-sqlite">ha-sqlite</a> - High-availability sqlite database service based on raft.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/shettyh/raft-sqlite">raft-sqlite</a> - Raft backend using SQLite.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/jervisfm/resqlite">ReSqlite</a> - ReSqlite is an extension of Sqlite that aims to add basic replication functionality to Sqlite database.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/minghsu0107/tqlite">tqlite</a> - Distributed SQL database with replication, fault-tolerance, tunable consistency and leader election.</p></li></ul><h2 id="h-real-time-databases" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Real-time Databases</h2><h3 id="h-peer-to-peer" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Peer-to-Peer</h3><ul><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/fireproof-storage/fireproof">Fireproof</a> - Fireproof is a decentralized realtime database that stores documents using prolly-trees.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/redwood/redwood">Redwood</a> - Highly-configurable, distributed, realtime database that manages a state tree shared among many peers.</p></li></ul><h2 id="h-nosql-databases" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">NoSQL Databases</h2><h3 id="h-blockchain" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Blockchain</h3><ul><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/bigchaindb/bigchaindb">BigchainDB</a> Blockchain database.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/bluzelle/curium">Bluzelle</a> - Decentralized database for Web 3.0 using blockchain and edge computing for greater performance and security.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/ceramicstudio/js-composedb">ComposeDB</a> - Decentralized, composable graph database to build interoperable applications on Ceramic.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/graphprotocol/contracts">The Graph</a> - Indexing protocol for querying networks like Ethereum, IPFS, Polygon, and other blockchains.</p></li></ul><h3 id="h-peer-to-peer" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Peer-to-Peer</h3><ul><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/dbpunk-labs/db3">db3</a> - Open-source and decentralized firebase firestore alternative for building fully decentralized dApps quickly with minimal engineering effort.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/orbitdb/orbit-db">OrbitDB</a> - Serverless, distributed, peer-to-peer database.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/weavedb/weavedb">WeaveDB</a> - NoSQL database powered by Warp Contracts (SmartWeave) on the Arweave blockchain.</p></li></ul><h2 id="h-encryption-and-security" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Encryption and Security</h2><ul><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/shenghe/FreeSQLiteEncryption">FreeSQLiteEncryption</a> - Add-on to the public domain version of SQLite that allows an application to read and write encrypted database files.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/LIT-Protocol/js-sdk">Lit</a> - Create and manage distributed cryptographic key-pairs for condition-based encryption and programmatic signing.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.nucypher.com/">Threshold Network</a> - Access control with end-to-end encrypted data sharing and communication without the need to trust a centralized authority.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/sqlcipher/sqlcipher">sqlcipher</a> - Fork of the SQLite database library that adds 256 bit AES encryption of database files and other security features.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/abc34/sqlite_codec">sqlite_codec</a> - sqlite3 with built-in transparent encryption based on mbedtls library.</p></li></ul><h2 id="h-learning" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Learning</h2><ul><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/pingcap/awesome-database-learning">Awesome Database Learning</a> - Another <em>awesome list</em> for database learning.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/hasura/learn-graphql">Learn GraphQL</a> - Learn real world GraphQL, quickly.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/WebDevSimplified/Learn-SQL">Learn SQL</a> - Practice and learn SQL from the basics.</p></li></ul><h2 id="h-lets-chat" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Let’s Chat</h2><p>If you found this interesting, give out a follow on <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/tableland">Twitter</a>, check out our <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork">GitHub</a>, and drop into our <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tableland.xyz/discord">Discord</a>. We’re always down to discuss the latest and greatest in database and blockchain tech! And soon, we’ll be launching our Studio web application to make it easier to develop with Tableland, so <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tableland.xyz/studio">sign up for the waitlist</a> if you’re interested in trying it out.</p>]]></content:encoded>
            <author>tableland-2@newsletter.paragraph.com (Tableland)</author>
            <enclosure url="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/1aa92953d1e5603e1f24efb3a4ca2d384d8640caa812aa08ab3764ad7a2be8ba.png" length="0" type="image/png"/>
        </item>
        <item>
            <title><![CDATA[Building a Secure, Verifiable Future for DePIN data: Introducing Our Partnership with DIMO and WeatherXM]]></title>
            <link>https://paragraph.com/@tableland-2/building-a-secure-verifiable-future-for-depin-data-introducing-our-partnership-with-dimo-and-weatherxm</link>
            <guid>f4r5Rf1YTZSCxfrA6P9V</guid>
            <pubDate>Thu, 31 Aug 2023 17:20:22 GMT</pubDate>
            <description><![CDATA[Welcome to an exciting moment in Tableland&apos;s journey. The majority of DePINs (Decentralized Physical Infrastructure Networks) are searching for better solutions for organizing, verifying, and using data across their networks. Today, we&apos;re excited to announce a partnership with DIMO and WeatherXM to tackle this problem head-on. Both projects have collaborated with us to develop and now rigorously test a Tableland solution for verifiable, real-time databases in the Web3 ecosystem. Let...]]></description>
            <content:encoded><![CDATA[<p>Welcome to an exciting moment in Tableland&apos;s journey. The majority of DePINs (Decentralized Physical Infrastructure Networks) are searching for better solutions for organizing, verifying, and using data across their networks. Today, we&apos;re excited to announce a partnership with DIMO and WeatherXM to tackle this problem head-on. Both projects have collaborated with us to develop and now rigorously test a Tableland solution for verifiable, real-time databases in the Web3 ecosystem. Let&apos;s dive in.</p><h3 id="h-a-depin-collaboration" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0"><strong>A DePIN Collaboration</strong></h3><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="http://dimo.zone">DIMO</a> and <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://weatherxm.com/">WeatherXM</a> are quite different, but share common challenges in decentralized data. DIMO is a mobility data network, powered by drivers that are rewarded for contributing to the network. WeatherXM is a community-owned weather network powered by individuals installing weather stations. Both face the challenge of how to connect globally distributed data to the network, how to move data from nodes on the edge into apps or pipelines that consume it, how to verify data anyplace it is used, and how to track that movement to reward network participants accurately.</p><p>This is where <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tableland.xyz/">Tableland</a> comes in. Our novel Tableland features for DePINs aim to simplify the data production, aggregation, and consumption process on open networks. Our DePIN-specific features will allow many contributors on the network to participate in the creation of common, verifiable, and customizable views of data on the network. Consumers can query data from any number of network participants, spanning the entire history of the network. We believe this solution can ultimately benefit DePINs across various sectors.</p><h3 id="h-enabling-novel-use-cases" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0"><strong>Enabling Novel Use-Cases</strong></h3><p>For WeatherXM we offer a way to seamlessly store data generated by <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://explorer.weatherxm.com/stats">thousands of weather stations</a> into an open, decentralized verifiable database. The database can be queried directly or integrated into the processes that manage the protocol&apos;s reward functions which includes quality of data evaluations and other computation heavy tasks. Using the same tools, businesses, scientists and weather enthusiasts worldwide can query or replicate the data.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/c2f060a35c54ab1f2ef633ddcdd8e558ae7ef8b0c625f69c2d16845910f398fe.png" alt="WeatherXM&apos;s Station Explorer" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">WeatherXM&apos;s Station Explorer</figcaption></figure><p>With DIMO, we&apos;re helping to simplify the tricky problem of managing a registry of device identities, and more specifically, how to maintain and update a universal list of valid vehicles so apps on the network can quickly turn 17-digit VIN numbers like <em>1FTCR15X0TTA01050</em>  into a more complete picture (see below). This allows DIMO users to know exactly what they can expect when connecting their vehicle to the network, and enables the seamless addition of new entries as new models are created.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/b51e419ba87dffbb5195a4bef324e0c98bd6c00d0556aba5c493d2e6c7b74efc.png" alt="Example of DIMO&apos;s vehicle information page" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Example of DIMO&apos;s vehicle information page</figcaption></figure><p>Tableland is helping to build data infrastructure that can turn painful paper-based experiences like a visit to the DMV or transactions with used car dealerships into simple digital transactions powered by smart contracts. DIMO contributors in any region will be able to contribute to this knowledge base and add vehicle and telematics devices without a central authority.</p><h3 id="h-simplified-verifiable-data-tools" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0"><strong>Simplified, Verifiable Data Tools</strong></h3><p>Beyond just storing data, data authenticity matters. Tableland is focused on powering a cryptographic verification layer that ensures data integrity. In the future, we will share some of our novel approaches to handling verifiability in these network-derived databases. Most importantly though, those approaches still power familiar and easy-to-use SQL interfaces and can connect directly into popular databases (e.g., PostgreSQL) to read and write data.</p><p>We understand the value of historical data. Furthermore, we&apos;ve learned from our partners the value of providing guarantees for historical data availability, providing point-in-time data recovery, and more. We&apos;re building an early integration with <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://filecoin.io">Filecoin</a>, where data moving across Tableland will be securely stored for long-term retrieval.</p><h3 id="h-conclusion" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0"><strong>Conclusion</strong></h3><p>Our focus is on interoperability and utility for a wide array of DePINs and networks. Our work with DIMO and WeatherXM is pivotal in shaping a novel solution for all DePINs. Imagine automated data aggregation, validation, and notifications — all built with decentralized computing in mind. We&apos;re enthusiastic about the vast potential this collaboration unlocks. Together with DIMO and WeatherXM, we&apos;re setting the groundwork for a more secure, efficient, and transparent decentralized infrastructure.</p><p>Curious to learn more? <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tinyurl.com/yc8zh6zj">Chat with one of our team members here</a>.</p>]]></content:encoded>
            <author>tableland-2@newsletter.paragraph.com (Tableland)</author>
        </item>
        <item>
            <title><![CDATA[The Quest for a Content Addressable SQLite]]></title>
            <link>https://paragraph.com/@tableland-2/the-quest-for-a-content-addressable-sqlite</link>
            <guid>Fk3QcdeLI456t1q86chc</guid>
            <pubDate>Thu, 17 Aug 2023 21:51:15 GMT</pubDate>
            <description><![CDATA[Happy Birthday SQLite! SQLite 1.0 was released 23 years ago today, so to celebrate this momentous occasion, join us on an exciting quest as we blend the old with the new, the familiar with the cutting-edge, and dare to dream of a future where edge databases and content addressable storage work together. Our destination? An experimental project to combine SQLite&apos;s features with the high performance, scalability, and deduplication capabilities of Content Addressable Storage (CAS). We’ll pr...]]></description>
            <content:encoded><![CDATA[<p>Happy Birthday SQLite! <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.sqlite.org/changes.html#version_1_0">SQLite 1.0 was released 23 years ago today</a>, so to celebrate this momentous occasion, join us on an exciting quest as we blend the old with the new, the familiar with the cutting-edge, and dare to dream of a future where edge databases and content addressable storage work together.</p><p>Our destination? An experimental project to combine SQLite&apos;s features with the high performance, scalability, and deduplication capabilities of Content Addressable Storage (CAS). We’ll probe new directions for scaling replicated SQLite databases like those on edge computing platforms, and even build a working prototype along the way.</p><p>In a nutshell, our experiment will leverage a novel SQLite Virtual File System (VFS) written in Rust, which will store database pages in a content addressable storage layer using a content-addressed Array Mapped Trie as our underlying persistent data structure. To top it all off, we’ll use off-the-shelf crates and libraries from the Filecoin ecosystem for maximum data interoperability!</p><h2 id="h-sqlite-is-right-now" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">SQLite is 🔥 Right Now</h2><p>SQLite is an in-process library that implements a self-contained, serverless, zero-configuration, and transactional SQL database engine. Although it&apos;s not new, SQLite has been gathering attention in the developer community, and for good reason.</p><p>In the current landscape of edge computing platforms and edge networks like Cloudflare Workers and <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="http://fly.io/">Fly.io</a>, developers are seeking lightweight and flexible database solutions that are easy to set up, deploy, and run. SQLite, with its ability to run as an embedded engine, fits this requirement perfectly. We’re talking about a complete compliant SQL database with all the bells and whistles, contained within a single binary 🤯.</p><p>For these reasons and more, SQLite is seeing a lot of interest from the developer world, with SQLite-specific products like <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://blog.cloudflare.com/introducing-d1/">Cloudflare’s D1</a>, and <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://turso.tech">turso.tech</a> leveraging the Lite in SQLite to simplify replication and improve query latencies on the edge, while at the same time folks like <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="http://Fly.io">Fly.io</a> are doubling down on <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://fly.io/blog/all-in-on-sqlite-litestream/">server-side SQLite</a> offerings.</p><h2 id="h-exploring-the-benefits-of-content-addressing" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0"><strong>Exploring the Benefits of Content Addressing</strong></h2><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://en.wikipedia.org/wiki/Content-addressable_storage">Content Addressable Storage</a>, or CAS, is a unique storage mechanism that allows data retrieval based on <em>content</em>, rather than storage <em>location</em>. CAS can be thought of as a scalable, high-performance library that never allows two identical books to be stored. <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://dweb-primer.ipfs.io/avenues-for-access/power-of-content-addressing">Projects like IPFS</a>, Git, the Google File System, blockchains, and even Amazon’s S3 use CAS concepts under the hood.</p><p>CAS works by assigning a unique identifier (a cryptographic hash) to each piece of information, facilitating rapid data retrieval and ensuring data immutability. This immutability feature guarantees a high level of <em>data integrity</em>. The unique id is often called a content id, due to it being derived from the content itself. Also, since the id is unique and content-based, CAS storage systems get deduplication super powers pretty much for free!</p><blockquote><p>A CID (content identifier) is simply a hash-based label used to link to some data. It is going to be our CA in CAS. You can <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.ipfs.tech/concepts/content-addressing/">read all about CIDs</a> in the IPFS documentation, which is a practical implementation of a content id.</p></blockquote><h2 id="h-an-experimental-database-for-the-edge" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">An Experimental Database for the Edge</h2><p>Imagine we have a database running on an edge computing platform (or even IoT/mobile devices), where we have extremely high throughput, limited compute/network resources, and a need to sync state between a number of replicas. Ideally, we want these replicas close to the users that are interacting with the data. But we don’t want the replicas to diverge too much (or at all really), and we don’t want one replica to have a totally different view from another. <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://zknill.io/posts/edge-database/">Here’s a great discussion</a> of some of the tradeoffs with putting data on the edge like this. At the end of the day, we’re going to have to decide at what layer of our system we’d like to deal with data consistency.</p><blockquote><p>We initially explored this concept in the context of blockchain-based state machine replication (SMR) across a set of decentralized nodes operating in a Byzantine Fault-Tolerant (BFT) environment. In that context, throughput is low. However, decentralized compute networks conceptually share many threat models and issues with traditional edge compute networks. <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/sanderpick/builtin-actors/blob/sander/tableland-actor2/actors/tableland/interface/src/state.rs">Here’s a POC</a> of an actor (smart contract)-based implementation that uses the Filecoin VM to maintain consensus on database state.</p></blockquote><p><em>So let’s try an experiment.</em> Instead of writing to a single database file (or WAL file), let’s see if we can have SQLite store database pages in our content addressable storage layer instead! Then we can let the storage layer handle the hard part of data consistency. To test this idea out, we built a proof of concept (POC) application, which consists of a SQLite Virtual File System (VFS) that leverages the power of a CAS backend.</p><p>Cloning or forking databases also becomes as easy as copying a tiny cryptographic hash (like with Git). We also get extremely high concurrency because our SQLite database is now a persistent, immutable data structure. When a modification is made, a new (extremely lightweight) version of the database itself is created that shares most of the structure with the original (and related) databases, and only the changes are added to CAS. This is the essence of structural sharing and <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://en.wikipedia.org/wiki/Persistent_data_structure">persistent data structures</a>!</p><h2 id="h-virtual-file-system-design" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Virtual File System Design</h2><p>Now comes the fun part! We are going to implement this POC in Rust to help illustrate the concept. Why Rust? Why not? Also, our team has been exploring Rust more and more lately, especially as we try out some cool experiments with extending SQLite. So first things first, let’s define how we’ll actually implement this thing diagrammatically, before we actually start building the thing.</p><p>The <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://fly.io/blog/introducing-litefs/">easiest way</a> to extend SQLite in this way is to implement a <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.sqlite.org/vfs.html">Virtual File System or VFS</a>. You can think of VFS as SQLite’s defacto portability layer. It is the module at the bottom of the SQLite implementation stack that provides portability across operating systems. SQLite ships with a number of default VFSes, but we’re going to implement our own to leverage CAS, rather than any native filesystem. Here’s roughly how we can conceptualize it:</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/a7ab41f2b5dc03c0a1bbe4bddf636b4b66da387cd8fa19fe4ea507c4ad6612d5.png" alt="Custom VFS" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Custom VFS</figcaption></figure><p>We want the normal SQLite application to communicate with the standard SQLite engine. At this layer, we want zero customization. From here, the default SQLite engine interacts with its internal B-Tree data structure, and uses a “Pager” to decide which pages to write out to our custom storage layer. Our custom SQLite VFS layer communicates with the underlying CAS, providing all the features of CAS to the SQLite engine. Nice.</p><h3 id="h-b-trees-and-pages" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">B-Trees and Pages</h3><p>Ok, what do I mean by B-trees and Pages? Well, if you didn’t know already, the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.sqlite.org/fileformat2.html">SQLite file format</a> is a stable and well-defined file format for specifying the structure and content of a SQLite database. In practice, it is a relatively simple binary format that describes a series of <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://en.wikipedia.org/wiki/B-tree">b-tree pages</a>. It is portable, efficient, and easy to implement. You can read more about some of the internals in this <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://fly.io/blog/sqlite-internals-btree/">great blog post</a> from our friends at <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="http://Fly.io">Fly.io</a>, but here’s a quick summary: “SQLite groups rows together into […] chunks called ‘pages’.” These pages are by default 4KBs, but can be any power of two between 512 and 65536 inclusive. Why 4KB by default? That&apos;s what file systems typically use as their page size…</p><p>So, we’re going to store b-tree pages, and we’ll just take whatever the Pager layer gives us, and stick these “atomic units” into our CAS. But the CAS is based on <em>content</em> addresses, and we want to be able to fetch them later using only their <em>location</em> address (page id). So we need some mapping between page id and content addresses, and we want this mapping to be pretty lightweight and efficient so it doesn’t add a ton of overhead to our CAS layer.</p><h3 id="h-authenticated-data-structures" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Authenticated Data Structures</h3><p>This is where we can leverage content address aware data structures to help us out! We’ll use an <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://en.wikipedia.org/wiki/Hash_array_mapped_trie">Array Mapped Trie</a> (AMT) that allows for the insertion and persistence of data, that itself is serializable to a CID. This is a data structure that looks and feels like an array, but actually stores the underlying data and its own structure within an underlying CAS system.</p><blockquote><p>What about something other than a AMT? For sure there are other solutions for actually representing the array structure. In fact, one really nice optimization that <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/sanderpick">Sander Pick</a> used in our early FVM POC was to build a deterministic B-tree like structure over the pages, grouping them into “buckets” of equal size as we move from the bottom up.</p></blockquote><p>This structure is an authenticated data structure (ADS), with efficient digests (fingerprinting) and a mechanism for efficient inclusion (or related) proofs. <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://en.wikipedia.org/wiki/Merkle_tree">Merkle Trees</a> provide a very useful starting point to talk about ADS, and we’re going to leverage a Merkle-DAG based AMT data structure for our experiment. If you aren’t familiar with Merkle trees and DAGs, <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://decentralizedthoughts.github.io/2020-12-22-what-is-a-merkle-tree/">here’s a pretty nice primer</a> to get you started. The digest is going to be in the form of a CID, so that we can reference an entire database structure with a single reference/link. A generic Merkle structure looks something like this:</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/902a614d01db76eaedc888cc730d7148e37a643afa8cfc296e3d589ca38ece58.png" alt="Basic Merkle Tree" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Basic Merkle Tree</figcaption></figure><p>As you can see in the above figure, the pages are grouped to form intermediate nodes, which are hashed and form interior nodes, and so on up to a root Merkle DAG node that can be used to represent the entire structure. This is perfect for “mapping” array indices to CIDs, and this structure itself can also be encoded and mapped to a CID. This gives us pretty much everything we need to turn our SQLite database pages into a persistent data-structure that supports structural sharing across a shared CAS layer.</p><h2 id="h-implementation" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Implementation</h2><p>The “full” (lightweight example version) implementation is <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://gist.github.com/carsonfarmer/852d4a520462b0be496354aaa2b9614e">available in this gist</a>, and borrows heavily <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/rkusa/wasm-sqlite/blob/main/wasm/src/vfs.rs">from this example</a> The details are a bit involved for a blog post (and this one is getting pretty long as it is), but here’s the test module to give you an idea of how you’d interact with this CAS-based SQLite VFS via the very popular <code>rusqlite</code><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/rusqlite/rusqlite"> library</a>:</p><pre data-type="codeBlock" text="#[cfg(test)]
mod tests {
    use crate::PagesVfs;
        // We&apos;ll use this to actually diff the db!
    use fvm_ipld_amt::{diff, Amt};
    use fvm_ipld_blockstore::MemoryBlockstore;
    use rusqlite::{Connection, OpenFlags};
    use sqlite_vfs::register;

    pub const SQLITE_PAGE_SIZE: usize = 4096;

    #[test]
    fn basic_get_set() {
        let mem = MemoryBlockstore::default();
        let initial_bytes: &amp;&apos;static [u8] = include_bytes!(&quot;file.db&quot;);
        let vfs = PagesVfs::&lt;_&gt;::new(mem, initial_bytes);
        let other = vfs.clone();

        register(&quot;vfs&quot;, vfs, true).unwrap();
        let mut conn = Connection::open_with_flags_and_vfs(
            &quot;main.db&quot;,
            OpenFlags::SQLITE_OPEN_READ_WRITE
                | OpenFlags::SQLITE_OPEN_CREATE
                | OpenFlags::SQLITE_OPEN_NO_MUTEX,
            &quot;vfs&quot;,
        )
        .unwrap();

        // If we set this to memory, then we don&apos;t need the &quot;cache&quot; cursor,
        // but then we also don&apos;t get the &quot;delete&quot; hook to work with...
        let journal_mode: String = conn
            .query_row(&quot;pragma journal_mode = delete&quot;, [], |row| row.get(0))
            .unwrap();

        assert_eq!(journal_mode, &quot;delete&quot;);

                // We don&apos;t need to enforce this, our VFS will adjust based on
                // the initial_bytes
        let page_size: usize = conn
            .query_row(&quot;PRAGMA page_size&quot;, [], |row| row.get(0))
            .unwrap();
        assert_eq!(page_size, SQLITE_PAGE_SIZE);

        let c = other.root().unwrap();
        println!(&quot;{:?}&quot;, c.to_string().as_str());

        let tx = conn.transaction().unwrap();
        tx.execute(
            &quot;create table my_table(id integer primary key, msg text);&quot;,
            [],
        )
        .unwrap();
        tx.execute(&quot;insert into my_table(msg) values(&apos;hello&apos;);&quot;, [])
            .unwrap();
        tx.execute(&quot;insert into my_table(msg) values(&apos;world&apos;);&quot;, [])
            .unwrap();
        tx.commit().unwrap();

        let c = other.root().unwrap();

        let mut stmt = conn.prepare(&quot;select * from my_table;&quot;).unwrap();
        let mut rows = stmt.query([]).unwrap();
        while let Some(r) = rows.next().unwrap() {
            let id: i32 = r.get(0).unwrap();
            let msg: String = r.get(1).unwrap();
            println!(&quot;found in db: {} = {}&quot;, id, msg);
        }

        let db = MemoryBlockstore::new();
        let new_amt: Amt&lt;Vec&lt;u8&gt;, _&gt; = Amt::new(&amp;db);
                // This actually computes a diff of the two structures
        let results = diff(&amp;other.state.lock().unwrap(), &amp;new_amt).unwrap();
                // We&apos;d expect there to be two pages that are different
        assert_eq!(results.len(), 2);
                // Deterministic structure here should lead to this cid
        assert_eq!(
            c.to_string().as_str(),
            &quot;bafy2bzaced3vtba4kcg4w5q4dnfdstxfom6t4cv72uuq4jgvkeav24ujdcgtg&quot;
        );
    }
}
"><code>#[cfg(test)]
mod tests {
    use crate::PagesVfs;
        <span class="hljs-comment">// We'll use this to actually diff the db!</span>
    use fvm_ipld_amt::{diff, Amt};
    use fvm_ipld_blockstore::MemoryBlockstore;
    use rusqlite::{Connection, OpenFlags};
    use sqlite_vfs::register;

    pub const SQLITE_PAGE_SIZE: usize <span class="hljs-operator">=</span> <span class="hljs-number">4096</span>;

    #[test]
    fn basic_get_set() {
        let mem <span class="hljs-operator">=</span> MemoryBlockstore::default();
        let initial_bytes: <span class="hljs-operator">&#x26;</span><span class="hljs-string">'static [u8] = include_bytes!("file.db");
        let vfs = PagesVfs::&#x3C;_>::new(mem, initial_bytes);
        let other = vfs.clone();

        register("vfs", vfs, true).unwrap();
        let mut conn = Connection::open_with_flags_and_vfs(
            "main.db",
            OpenFlags::SQLITE_OPEN_READ_WRITE
                | OpenFlags::SQLITE_OPEN_CREATE
                | OpenFlags::SQLITE_OPEN_NO_MUTEX,
            "vfs",
        )
        .unwrap();

        // If we set this to memory, then we don'</span>t need the <span class="hljs-string">"cache"</span> cursor,
        <span class="hljs-comment">// but then we also don't get the "delete" hook to work with...</span>
        let journal_mode: String <span class="hljs-operator">=</span> conn
            .query_row(<span class="hljs-string">"pragma journal_mode = delete"</span>, [], <span class="hljs-operator">|</span>row<span class="hljs-operator">|</span> row.get(<span class="hljs-number">0</span>))
            .<span class="hljs-built_in">unwrap</span>();

        assert_eq<span class="hljs-operator">!</span>(journal_mode, <span class="hljs-string">"delete"</span>);

                <span class="hljs-comment">// We don't need to enforce this, our VFS will adjust based on</span>
                <span class="hljs-comment">// the initial_bytes</span>
        let page_size: usize <span class="hljs-operator">=</span> conn
            .query_row(<span class="hljs-string">"PRAGMA page_size"</span>, [], <span class="hljs-operator">|</span>row<span class="hljs-operator">|</span> row.get(<span class="hljs-number">0</span>))
            .<span class="hljs-built_in">unwrap</span>();
        assert_eq<span class="hljs-operator">!</span>(page_size, SQLITE_PAGE_SIZE);

        let c <span class="hljs-operator">=</span> other.root().<span class="hljs-built_in">unwrap</span>();
        println<span class="hljs-operator">!</span>(<span class="hljs-string">"{:?}"</span>, c.to_string().as_str());

        let <span class="hljs-built_in">tx</span> <span class="hljs-operator">=</span> conn.transaction().<span class="hljs-built_in">unwrap</span>();
        <span class="hljs-built_in">tx</span>.execute(
            <span class="hljs-string">"create table my_table(id integer primary key, msg text);"</span>,
            [],
        )
        .<span class="hljs-built_in">unwrap</span>();
        <span class="hljs-built_in">tx</span>.execute(<span class="hljs-string">"insert into my_table(msg) values('hello');"</span>, [])
            .<span class="hljs-built_in">unwrap</span>();
        <span class="hljs-built_in">tx</span>.execute(<span class="hljs-string">"insert into my_table(msg) values('world');"</span>, [])
            .<span class="hljs-built_in">unwrap</span>();
        <span class="hljs-built_in">tx</span>.commit().<span class="hljs-built_in">unwrap</span>();

        let c <span class="hljs-operator">=</span> other.root().<span class="hljs-built_in">unwrap</span>();

        let mut stmt <span class="hljs-operator">=</span> conn.prepare(<span class="hljs-string">"select * from my_table;"</span>).<span class="hljs-built_in">unwrap</span>();
        let mut rows <span class="hljs-operator">=</span> stmt.query([]).<span class="hljs-built_in">unwrap</span>();
        <span class="hljs-keyword">while</span> let Some(r) <span class="hljs-operator">=</span> rows.next().<span class="hljs-built_in">unwrap</span>() {
            let id: i32 <span class="hljs-operator">=</span> r.get(<span class="hljs-number">0</span>).<span class="hljs-built_in">unwrap</span>();
            let <span class="hljs-built_in">msg</span>: String <span class="hljs-operator">=</span> r.get(<span class="hljs-number">1</span>).<span class="hljs-built_in">unwrap</span>();
            println<span class="hljs-operator">!</span>(<span class="hljs-string">"found in db: {} = {}"</span>, id, <span class="hljs-built_in">msg</span>);
        }

        let db <span class="hljs-operator">=</span> MemoryBlockstore::<span class="hljs-keyword">new</span>();
        let new_amt: Amt<span class="hljs-operator">&#x3C;</span>Vec<span class="hljs-operator">&#x3C;</span>u8<span class="hljs-operator">></span>, <span class="hljs-keyword">_</span><span class="hljs-operator">></span> <span class="hljs-operator">=</span> Amt::<span class="hljs-keyword">new</span>(<span class="hljs-operator">&#x26;</span>db);
                <span class="hljs-comment">// This actually computes a diff of the two structures</span>
        let results <span class="hljs-operator">=</span> diff(<span class="hljs-operator">&#x26;</span>other.state.lock().<span class="hljs-built_in">unwrap</span>(), <span class="hljs-operator">&#x26;</span>new_amt).<span class="hljs-built_in">unwrap</span>();
                <span class="hljs-comment">// We'd expect there to be two pages that are different</span>
        assert_eq<span class="hljs-operator">!</span>(results.len(), <span class="hljs-number">2</span>);
                <span class="hljs-comment">// Deterministic structure here should lead to this cid</span>
        assert_eq<span class="hljs-operator">!</span>(
            c.to_string().as_str(),
            <span class="hljs-string">"bafy2bzaced3vtba4kcg4w5q4dnfdstxfom6t4cv72uuq4jgvkeav24ujdcgtg"</span>
        );
    }
}
</code></pre><p>It is actually pretty simple. We bring our CAS system to the VFS implementation, here it is a shared <code>MemoryBlockstore</code>. This could be anything that implements the <code>Blockstore</code> trait that is provided by <code>fvm_ipld_blockstore</code>. Then we just register the VFS, and from there we essentially just use SQLite as per normal. Nothing fancy to think about. The only difference is, we can now reference the root hash of the database at any time. What’s cool about that is, an entire database can be defined by a single CID and access to a (possibly remote) blockstore!</p><p>We don’t really do much in terms of serious locking infrastructure, or even write ahead logging or rollback journals. With our setup, we don’t <em>really</em> need them, because if something goes wrong, we’ll just rollback the underlying AMT to the previous root hash and go from there. Additionally, to “fork” or otherwise bootstrap a new database from our existing one is as easy as calling its <code>clone</code> method. The only things we really need to clone are the references to the blockstrore and the root CID. This works because “loading” an AMT from an existing root and blockstore is extremely lightweight:</p><pre data-type="codeBlock" text="let mut new_amt: Amt&lt;Vec&lt;u8&gt;, _&gt; = Amt::load(&amp;cid, &amp;mem).unwrap();
assert_eq!(
    new_amt.flush().unwrap().to_string().as_str(),
    &quot;bafy2bzaced3vtba4kcg4w5q4dnfdstxfom6t4cv72uuq4jgvkeav24ujdcgtg&quot;
);
"><code>let mut new_amt: Amt<span class="hljs-operator">&#x3C;</span>Vec<span class="hljs-operator">&#x3C;</span>u8<span class="hljs-operator">></span>, <span class="hljs-keyword">_</span><span class="hljs-operator">></span> <span class="hljs-operator">=</span> Amt::load(<span class="hljs-operator">&#x26;</span>cid, <span class="hljs-operator">&#x26;</span>mem).<span class="hljs-built_in">unwrap</span>();
assert_eq<span class="hljs-operator">!</span>(
    new_amt.flush().<span class="hljs-built_in">unwrap</span>().to_string().as_str(),
    <span class="hljs-string">"bafy2bzaced3vtba4kcg4w5q4dnfdstxfom6t4cv72uuq4jgvkeav24ujdcgtg"</span>
);
</code></pre><p>Other nice features here are that, unlike a normal SQLite database setup, our CAS can be networked, so we don’t have to have <em>any</em> real local storage if we don’t want. This makes it quite easy to use in a shared (or multi-tenant) storage environment like Amazon&apos;s EC2 or Cloudflare&apos;s R2.</p><h2 id="h-stay-tuned" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Stay Tuned…</h2><p>So that’s pretty much it for today!</p><p><strong>Here’s a summary so far</strong>: <em>We presented an experimental project to support SQLite databases on edge computing platforms, combining SQLite&apos;s features with the scalability and deduplication capabilities of Content Addressable Storage (CAS). We detailed the process of creating a SQLite Virtual File System (VFS) in Rust, which stores database pages in a CAS layer using a content addressable Array Mapped Trie as our underlying persistent data structure. To top it all off, we used off-the-shelf crates and libraries from the Filecoin ecosystem for the ultimate in data interoperability!</em></p><p><strong>In a future post</strong>: <em>We’ll be</em> d<em>oing some benchmarking and measuring the potential benefits and tradeoffs in terms of storage and network traffic. After that,</em> stay tuned for more of these types of posts as we continue to explore the design space around databases, peer-to-peer networks, blockchain, and a whole lot more. <em>In particular,</em></p><p><strong>In the mean time</strong>: <em>if you liked this post, and/or if you are a Rust person and know how to improve the above code/design, we’re all ears! Let us know either way in </em><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tableland.xyz/discord"><em>our Discord</em></a><em>, via email, </em><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/tableland"><em>Twitter/X</em></a><em>, etc.</em></p>]]></content:encoded>
            <author>tableland-2@newsletter.paragraph.com (Tableland)</author>
            <enclosure url="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/80069acaeed3405b371f2a868bd30af670753d0fc268851469d7bc3524133751.jpg" length="0" type="image/jpg"/>
        </item>
        <item>
            <title><![CDATA[Tableland + Filecoin: FEVM Support]]></title>
            <link>https://paragraph.com/@tableland-2/tableland-filecoin-fevm-support</link>
            <guid>AxWu0q7JzUj3js032VcG</guid>
            <pubDate>Wed, 16 Aug 2023 18:49:23 GMT</pubDate>
            <description><![CDATA[What Is It?Tableland developers can now build on Filecoin! We actually added Filecoin FEVM support a couple of months ago, but we wanted to dive into this in a bit more detail since it opens up a ton of new possibilities that weren’t available before. One demonstrated example is the Rigs NFT with Filecoin deals for its assets, which it described in more detail below.If you’re currently building with or interested in exploring a Filecoin + Tableland integration, apply to the Pilot Program for ...]]></description>
            <content:encoded><![CDATA[<h2 id="h-what-is-it" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">What Is It?</h2><p>Tableland developers can now build on Filecoin! We actually added Filecoin FEVM support a couple of months ago, but we wanted to dive into this in a bit more detail since it opens up a ton of new possibilities that weren’t available before. One demonstrated example is the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://opensea.io/collection/tableland-rigs">Rigs NFT</a> with Filecoin deals for its assets, which it described in more detail below.</p><blockquote><p>If you’re currently building with or interested in exploring a Filecoin + Tableland integration, apply to the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tableland.xyz/pilot-program">Pilot Program</a> for microgrant funding, support, and access to our team’s network.</p></blockquote><h2 id="h-why-did-we-do-it" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Why Did We Do It?</h2><p>Tableland is a decentralized database network. Filecoin is a decentralized storage network. The two work quite well together, especially, in the context of what traditional databases can support. Plus, Filecoin adds verifiable, persisted storage, which isn’t a feature with IPFS alone.</p><p>For context, Filecoin launched their virtual runtime called the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://fvm.filecoin.io/">FVM</a> to bring programmability to its network. The FVM <em>also</em> contains an Ethereum Virtual Machine (EVM) runtime, called the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.filecoin.io/smart-contracts/fundamentals/filecoin-evm-runtime/">FEVM</a>. The FEVM lets you use familiar languages like Solidity to write smart contracts (also called “actors” on the FVM), which makes building on top of Filecoin easier for existing web3 developers.</p><p>Tableland focuses on EVM-compatible chains, so when the FEVM launched, we were very excited to support it as soon as possible! But unlike the other blockchains (Ethereum, L2s), Filecoin offers novel features like storing large files and using actors/smart contracts to orchestrate storage deals for persistence. There are <strong>tons</strong> of use cases for using Tableland as well as Filecoin that, quite simply, aren’t possible on other blockchains, so it was an easy decision to bring Tableland to the Filecoin ecosystem!</p><blockquote><p>Fun fact: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://textile.io/">Textile</a>, the team ushering forward Tableland, spent a number of years building Filecoin tooling and other infrastructure for the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://protocol.ai/">Protocol Labs</a> ecosystem, prior to launching Tableland.</p></blockquote><h2 id="h-what-does-it-do-and-how-does-it-work" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">What Does it Do &amp; How Does It Work?</h2><p>You can think of Tableland as an additional database layer on top of Filecoin. This enables web3-native mutability and queries to data written by FEVM smart contracts, such as deal data or file metadata. Recall that Tableland works in two parts:</p><ul><li><p><em>Onchain</em>: a registry smart contract that mints SQL tables, allows you to write SQL statements against them, and use additional access control checks for shared table mutations.</p></li><li><p><em>Offchain</em>: a network of validator nodes that materialize onchain SQL events and materialize them in a per-validator SQLite database, allowing for read queries to be made at a gateway.</p></li></ul><p>By supporting the FEVM, developers can build applications on Filecoin. Existing Tableland tooling—like smart contracts, SDK, and CLI—fully support Filecoin without requiring any changes for builders…aside from pointing to a new chain! There are some architectural and naming convention differences (transactions, address format, etc.), but these shouldn’t impact the developer experience (e.g., you can still use an EVM account/wallet).</p><h2 id="h-how-can-someone-use-it" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">How Can Someone Use It?</h2><h3 id="h-tableland-tooling" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Tableland Tooling</h3><p>You can set the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.tableland.xyz/fundamentals/chains/filecoin">chain</a> name to <code>filecoin</code> (mainnet) or <code>filecoin-calibration</code> (testnet) and leverage Tableland tooling as you would on any other chain. For example, the SDK or CLI:</p><pre data-type="codeBlock" text="// SDK
const db = new Database({
  signer,
  chain: &quot;filecoin&quot;, // or `filecoin-calibration`
});

// CLI
tableland read &quot;SELECT ...&quot; --chain filecoin --providerUrl ...
"><code><span class="hljs-comment">// SDK</span>
const db <span class="hljs-operator">=</span> <span class="hljs-keyword">new</span> Database({
  signer,
  chain: <span class="hljs-string">"filecoin"</span>, <span class="hljs-comment">// or `filecoin-calibration`</span>
});

<span class="hljs-comment">// CLI</span>
tableland read <span class="hljs-string">"SELECT ..."</span> <span class="hljs-operator">-</span><span class="hljs-operator">-</span>chain filecoin <span class="hljs-operator">-</span><span class="hljs-operator">-</span>providerUrl ...
</code></pre><p>If you’re using the Hardhat framework to deploy smart contracts, the <code>hardhat.config.js</code> file should include Filecoin networks. There are a number of RPC providers you can choose from, but we recommend <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.ankr.com/">Ankr</a> or <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.glif.io/">Glif</a>. When you run a script, pass the <code>--network</code> flag with one of the following:</p><pre data-type="codeBlock" text="const config: HardhatUserConfig = {
  // ...
  networks: {
    filecoin: {
      url: `https://rpc.ankr.com/filecoin/your_api_key`,
      accounts: YOUR_PRIVATE_KEY,
    },
    &quot;filecoin-calibration&quot;: {
      url: `https://api.calibration.node.glif.io/rpc/v1/`,
      accounts: YOUR_PRIVATE_KEY,
    },
  },
  // ...
};
"><code><span class="hljs-attr">const config:</span> <span class="hljs-string">HardhatUserConfig</span> <span class="hljs-string">=</span> {
  <span class="hljs-string">//</span> <span class="hljs-string">...</span>
  <span class="hljs-attr">networks:</span> {
    <span class="hljs-attr">filecoin:</span> {
      <span class="hljs-attr">url:</span> <span class="hljs-string">`https://rpc.ankr.com/filecoin/your_api_key`</span>,
      <span class="hljs-attr">accounts:</span> <span class="hljs-string">YOUR_PRIVATE_KEY</span>,
    },
    <span class="hljs-attr">"filecoin-calibration":</span> {
      <span class="hljs-attr">url:</span> <span class="hljs-string">`https://api.calibration.node.glif.io/rpc/v1/`</span>,
      <span class="hljs-attr">accounts:</span> <span class="hljs-string">YOUR_PRIVATE_KEY</span>,
    },
  },
  <span class="hljs-string">//</span> <span class="hljs-string">...</span>
}<span class="hljs-string">;</span>
</code></pre><p>One callout—Filecoin <strong>block times average ~30 seconds</strong>, so keep that in mind since it’s a <em>little</em> different than other EVM chains’ shorter inclusion periods. For the SQL to materialize in the Tableland network, there’s also a bit of latency due to how Filecoin operates; you can expect <strong>up to a 4 minute wait</strong> until the data is reflected and queryable.</p><p>Lastly, let’s review what a Tableland <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.tableland.xyz/validator">validator node</a> operator can do to add Filecoin support. In your <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/go-tableland">source code</a>, navigate to <code>docker/deployed/mainnet/api/config.json</code> and/or <code>docker/deployed/testnet/api/config.json</code>; this is where the configuration files are located. Under the <code>&quot;Chains&quot;</code> array, you can add an entry for these networks. For demonstration purposes, Filecoin mainnet and testnet settings are shown below, but keep in mind these would be in the proper mainnet or testnet config file, not in the same one. Also note Glif doesn’t require authentication, so you won’t need to set up an API key in your <code>.env_...</code> files but should still keep the variable in there.</p><pre data-type="codeBlock" text="{
    // other config info
    &quot;Chains&quot;: [
        // in the mainnet config file
        {
            &quot;Name&quot;: &quot;Filecoin Mainnet&quot;,
            &quot;ChainID&quot;: 314,
            &quot;Registry&quot;: {
                &quot;EthEndpoint&quot;: &quot;https://node.glif.io/fvm-archive/lotus/rpc/v1&quot;,
                &quot;ContractAddress&quot;: &quot;0x59EF8Bf2d6c102B4c42AEf9189e1a9F0ABfD652d&quot;,
                &quot;ProviderAuthToken&quot;: &quot;${VALIDATOR_GLIF_FILECOIN_MAINNET_API_KEY}&quot;
            },
            &quot;EventFeed&quot;: {
                &quot;ChainAPIBackoff&quot;: &quot;15s&quot;,
                &quot;NewBlockPollFreq&quot;: &quot;15s&quot;,
                &quot;MinBlockDepth&quot;: 5,
                &quot;PersistEvents&quot;: true
            },
            &quot;EventProcessor&quot;: {
                &quot;BlockFailedExecutionBackoff&quot;: &quot;10s&quot;,
                &quot;DedupExecutedTxns&quot;: true,
            },
            &quot;HashCalculationStep&quot;: 60
        },
        // in the testnet config file
        {
            &quot;Name&quot;: &quot;Filecoin Calibration&quot;,
            &quot;ChainID&quot;: 314159,
            &quot;Registry&quot;: {
                &quot;EthEndpoint&quot;: &quot;https://api.calibration.node.glif.io/rpc/v1&quot;,
                &quot;ContractAddress&quot;: &quot;0x030BCf3D50cad04c2e57391B12740982A9308621&quot;
            },
            &quot;EventFeed&quot;: {
                &quot;ChainAPIBackoff&quot;: &quot;15s&quot;,
                &quot;NewBlockPollFreq&quot;: &quot;15s&quot;,
                &quot;MinBlockDepth&quot;: 5,
                &quot;PersistEvents&quot;: true
            },
            &quot;EventProcessor&quot;: {
                &quot;BlockFailedExecutionBackoff&quot;: &quot;10s&quot;,
                &quot;DedupExecutedTxns&quot;: true
            },
            &quot;HashCalculationStep&quot;: 60
        }
    ]
}
"><code>{
    <span class="hljs-comment">// other config info</span>
    <span class="hljs-string">"Chains"</span>: [
        <span class="hljs-comment">// in the mainnet config file</span>
        {
            <span class="hljs-string">"Name"</span>: <span class="hljs-string">"Filecoin Mainnet"</span>,
            <span class="hljs-string">"ChainID"</span>: <span class="hljs-number">314</span>,
            <span class="hljs-string">"Registry"</span>: {
                <span class="hljs-string">"EthEndpoint"</span>: <span class="hljs-string">"https://node.glif.io/fvm-archive/lotus/rpc/v1"</span>,
                <span class="hljs-string">"ContractAddress"</span>: <span class="hljs-string">"0x59EF8Bf2d6c102B4c42AEf9189e1a9F0ABfD652d"</span>,
                <span class="hljs-string">"ProviderAuthToken"</span>: <span class="hljs-string">"<span class="hljs-subst">${VALIDATOR_GLIF_FILECOIN_MAINNET_API_KEY}</span>"</span>
            },
            <span class="hljs-string">"EventFeed"</span>: {
                <span class="hljs-string">"ChainAPIBackoff"</span>: <span class="hljs-string">"15s"</span>,
                <span class="hljs-string">"NewBlockPollFreq"</span>: <span class="hljs-string">"15s"</span>,
                <span class="hljs-string">"MinBlockDepth"</span>: <span class="hljs-number">5</span>,
                <span class="hljs-string">"PersistEvents"</span>: <span class="hljs-literal">true</span>
            },
            <span class="hljs-string">"EventProcessor"</span>: {
                <span class="hljs-string">"BlockFailedExecutionBackoff"</span>: <span class="hljs-string">"10s"</span>,
                <span class="hljs-string">"DedupExecutedTxns"</span>: <span class="hljs-literal">true</span>,
            },
            <span class="hljs-string">"HashCalculationStep"</span>: <span class="hljs-number">60</span>
        },
        <span class="hljs-comment">// in the testnet config file</span>
        {
            <span class="hljs-string">"Name"</span>: <span class="hljs-string">"Filecoin Calibration"</span>,
            <span class="hljs-string">"ChainID"</span>: <span class="hljs-number">314159</span>,
            <span class="hljs-string">"Registry"</span>: {
                <span class="hljs-string">"EthEndpoint"</span>: <span class="hljs-string">"https://api.calibration.node.glif.io/rpc/v1"</span>,
                <span class="hljs-string">"ContractAddress"</span>: <span class="hljs-string">"0x030BCf3D50cad04c2e57391B12740982A9308621"</span>
            },
            <span class="hljs-string">"EventFeed"</span>: {
                <span class="hljs-string">"ChainAPIBackoff"</span>: <span class="hljs-string">"15s"</span>,
                <span class="hljs-string">"NewBlockPollFreq"</span>: <span class="hljs-string">"15s"</span>,
                <span class="hljs-string">"MinBlockDepth"</span>: <span class="hljs-number">5</span>,
                <span class="hljs-string">"PersistEvents"</span>: <span class="hljs-literal">true</span>
            },
            <span class="hljs-string">"EventProcessor"</span>: {
                <span class="hljs-string">"BlockFailedExecutionBackoff"</span>: <span class="hljs-string">"10s"</span>,
                <span class="hljs-string">"DedupExecutedTxns"</span>: <span class="hljs-literal">true</span>
            },
            <span class="hljs-string">"HashCalculationStep"</span>: <span class="hljs-number">60</span>
        }
    ]
}
</code></pre><h3 id="h-use-cases" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Use Cases</h3><p>A programmable Filecoin opens up quite a broad spectrum of opportunities. One example is automated storage deals: store large media (images, files, etc.) and guarantee it’s persisted with the smart contract managing the negotiation process. You can even build reputation-based services that help track storage provider efficacy.</p><p>Data DAOs are another great example. Tableland is designed for storing “small” data in cells, up to 1kb. So, you can use Filecoin to store large datasets and choose where/how that data gets placed, incentivized, and governed. Although the FVM doesn’t directly interact with the data stored, you can add metadata or other access control rules around the orchestration, or even tokenize datasets. Tableland can help with storing dataset metadata, deal data, or other relevant DAO information.</p><p>There are plenty of other use cases, but those should help contextualize what you can do!</p><h2 id="h-example-rigs-ipfs-retrieval-and-filecoin-storage" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Example: Rigs, IPFS Retrieval, &amp; Filecoin Storage</h2><p>The <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://opensea.io/collection/tableland-rigs">Tableland Rigs NFT</a> is a collection that uses the core Tableland protocol to store its metadata; all of this code is <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/rigs">open source</a>. In fact, there are <em>10 tables</em> across Ethereum, Arbitrum, and Filecoin that are composed into a single <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.tableland.xyz/fundamentals/concepts/nft-metadata">metadata</a> object and fully <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.tableland.xyz/playbooks/walkthroughs/nft-metadata">ERC-721 compliant</a>. Rigs images were originally only stored on IPFS, so we wanted to persist these images by creating Filecoin deals but also use IPFS for faster retrieval. Each NFT’s media files are stored in a single folder, packaged as a <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://ipld.io/specs/transport/car/">CAR</a> (Content Addressable aRchives) file. Then, a final CAR file is pushed to <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://web3.storage/products/w3link/">w3link</a> that maps out all Rig folders, giving a single CID/path.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/f82255dab7d4ffb6661c7e495a29a66829c3864cc1237be443f318abaaa73373.png" alt="Structure of a CAR file." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Structure of a CAR file.</figcaption></figure><p>Each Rig has a deal made to store its media on Filecoin, and deal information is written back to Tableland metadata. A viewer will fetch a single media file with a CID/folder/path, and if it is not cached, the request will reach the Filecoin Storage Provider to serve the data.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/c1c55acf3ff32425d7bac74e852b41af31a575ab0e9dc169f12ed77e6e25d4b7.png" alt="Tableland Rigs architecture." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Tableland Rigs architecture.</figcaption></figure><p>Our table schema is the following, and upon making a deal, each Rig’s metadata is inserted into the table (see <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tableland.network/api/v1/query?statement=select%20*%20from%20deals_314_4%20limit%2010">here</a> for what this looks like in production):</p><pre data-type="codeBlock" text="CREATE TABLE deals(
  rig_id integer not null,
  deal_id integer not null,
  storage_provider text not null,
  data_model_selector text not null,
  deal_number integer not null,
  primary key(rig_id, deal_id)
)
"><code><span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">TABLE</span> deals(
  rig_id <span class="hljs-type">integer</span> <span class="hljs-keyword">not</span> <span class="hljs-keyword">null</span>,
  deal_id <span class="hljs-type">integer</span> <span class="hljs-keyword">not</span> <span class="hljs-keyword">null</span>,
  storage_provider text <span class="hljs-keyword">not</span> <span class="hljs-keyword">null</span>,
  data_model_selector text <span class="hljs-keyword">not</span> <span class="hljs-keyword">null</span>,
  deal_number <span class="hljs-type">integer</span> <span class="hljs-keyword">not</span> <span class="hljs-keyword">null</span>,
  <span class="hljs-keyword">primary</span> key(rig_id, deal_id)
)
</code></pre><p>A Rig’s metadata will now include some additional attributes that describe the deal data. If you navigate to <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://filfox.info/en/deal/36739326">one of the deal IDs</a>, you can further <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://filfox.info/en/message/bafy2bzaced7psypolrji4ld7dmche4lhtqxocgfvn4uzshgpjtwsjrxf5y5u6">inspect the deal information</a>.</p><pre data-type="codeBlock" text="{
  &quot;display_type&quot;: &quot;string&quot;,
  &quot;trait_type&quot;: &quot;Filecoin Deal 1&quot;,
  &quot;value&quot;: &quot;https://filfox.info/deal/36739326&quot;
},
{
  &quot;display_type&quot;: &quot;string&quot;,
  &quot;trait_type&quot;: &quot;Filecoin Deal 2&quot;,
  &quot;value&quot;: &quot;https://filfox.info/deal/37000535&quot;
},
{
  &quot;display_type&quot;: &quot;string&quot;,
  &quot;trait_type&quot;: &quot;Filecoin Deal 3&quot;,
  &quot;value&quot;: &quot;https://filfox.info/deal/38987515&quot;
},
{
  &quot;display_type&quot;: &quot;string&quot;,
  &quot;trait_type&quot;: &quot;Filecoin Deal 4&quot;,
    &quot;value&quot;: &quot;https://filfox.info/deal/39012246&quot;
},
"><code><span class="hljs-punctuation">{</span>
  <span class="hljs-attr">"display_type"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"string"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"trait_type"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Filecoin Deal 1"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"value"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"https://filfox.info/deal/36739326"</span>
<span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span>
<span class="hljs-punctuation">{</span>
  <span class="hljs-attr">"display_type"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"string"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"trait_type"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Filecoin Deal 2"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"value"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"https://filfox.info/deal/37000535"</span>
<span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span>
<span class="hljs-punctuation">{</span>
  <span class="hljs-attr">"display_type"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"string"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"trait_type"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Filecoin Deal 3"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"value"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"https://filfox.info/deal/38987515"</span>
<span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span>
<span class="hljs-punctuation">{</span>
  <span class="hljs-attr">"display_type"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"string"</span><span class="hljs-punctuation">,</span>
  <span class="hljs-attr">"trait_type"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Filecoin Deal 4"</span><span class="hljs-punctuation">,</span>
    <span class="hljs-attr">"value"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"https://filfox.info/deal/39012246"</span>
<span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span>
</code></pre><p>Now, each Rig has a dedicated and verifiably persisted deal for its media!</p><h2 id="h-get-in-touch" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Get In Touch!</h2><p>Be sure to give us a <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/tableland">follow on Twitter</a> and <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tableland.xyz/discord">hop into our Discord</a> if you’d like to discuss this design or the protocol in more detail. And keep an eye out for upcoming hackathons—both Filecoin and Tableland should be at a couple of them!</p>]]></content:encoded>
            <author>tableland-2@newsletter.paragraph.com (Tableland)</author>
            <enclosure url="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/b1d83787f450485fff2cc6ecf14cfe1902a32d3db38f848f875bb172c44d03ba.png" length="0" type="image/png"/>
        </item>
        <item>
            <title><![CDATA[Discord Roles from Chain-driven Application Data]]></title>
            <link>https://paragraph.com/@tableland-2/discord-roles-from-chain-driven-application-data</link>
            <guid>aDlFBif4jkb1x4eMHkNv</guid>
            <pubDate>Thu, 10 Aug 2023 23:44:06 GMT</pubDate>
            <description><![CDATA[What Is It?The Tableland team is excited to introduce our new Discord<>Tableland bot integration—linking on/off-chain retribution back to Discord roles! Namely, developers can create/deploy this bot as an extension to Vulcan using its native features. It allows the bot to read data from an application’s Tableland tables and use it in Discord user/role management—all with a decentralized cloud database!Why Did We Do It?Vulcan is great for checking Discord member NFT ownership and creating role...]]></description>
            <content:encoded><![CDATA[<h2 id="h-what-is-it" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">What Is It?</h2><p>The Tableland team is excited to introduce our new Discord&lt;&gt;Tableland bot integration—linking on/off-chain retribution back to Discord roles!</p><p>Namely, developers can create/deploy this bot as an extension to <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.vulcan.xyz/">Vulcan</a> using its native features. It allows the bot to read data from an application’s Tableland tables and use it in Discord user/role management—all with a decentralized cloud database!</p><h2 id="h-why-did-we-do-it" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Why Did We Do It?</h2><p>Vulcan is great for checking Discord member NFT ownership and creating roles based on NFT traits—but what if you want <strong><em>other</em></strong> data to be part of that role attribution process, including conditional checks on that data? With a custom webhooks, you can dynamically update a member&apos;s role by <strong>reading application data from Tableland</strong>. For example, a smart contract can write user data (e.g., address &amp; session data) to a set of tables, and you can then make queries to them using Vulcan.</p><p>This extension will automatically attach new roles to users predicated on on-chain data, enhancing your overall Discord role management &amp; community experience!</p><h2 id="h-what-does-it-do" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">What Does it Do?</h2><p>The bot will give different Discord roles/badges to every member who has gone through the initial Vulcan authentication steps.</p><p>That is, <em>only</em> members who&apos;ve verified will have roles associated with them, and the roles are based on data read from application-controlled Tableland tables. A role might even unlock additional Discord channels or capabilities for that subset of users!</p><h2 id="h-how-does-it-work" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">How Does It Work?</h2><p>This feature relies on <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.vulcan.xyz/">Vulcan</a> to provide user wallet information; it’s a great tool for web3 community management, as it provides the wallet-to-member attribution needed in our workflow. Although, the general approach could be used with other Discord bots, too! Recall that Tableland is EVM compatible with <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.tableland.xyz/fundamentals/chains/">support for a number of L1/L2 chains</a> and works in two parts:</p><ol><li><p><strong>On-chain</strong>: table creation and writes go through a registry smart contract—i.e., these are signed transactions that correspond to a table ERC721 token, allowing for on-chain data like a user’s account address to be part of any written data context.</p></li><li><p><strong>Off-chain</strong>: table reads to validator nodes running the Tableland protocol—i.e., this enables open APIs for running read-only queries across any set of tables (including cross-chain), such as querying for the user’s address &amp; other useful relational data.</p></li></ol><p>Each developer’s Discord server will have their own Vulcan verification requirements and rules for the role attribution, but let’s assume you’ve developed a web3 application that stores your connected users’ EVM account addresses. With Vulcan’s <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://help.vulcan.xyz/custom-webhook">custom webhooks</a>, we’re creating an extension that queries dynamic, chain-driven table data. Developers can make SQL queries to the Tableland network and use this user data alongside Vulcan by controlling the webhook’s response. You can even compose data across one or multiple tables to determine what types of roles should be shown on each Discord member’s profile.</p><p>For example, a common application pattern is to store user data. Your app might capture user data upon logging in and write it to a Tableland table. Since Vulcan lets you associate each Discord member with their EVM account/address, you can combine Vulcan verification data with your own application’s user table data.</p><h2 id="h-how-can-you-use-this" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">How Can You Use This?</h2><h3 id="h-table-setup" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Table Setup</h3><p>There are endless ways to build this feature and add it to your Discord server! But, let’s walk through an example with user session data. First, you’ll want to create a table called <code>sessions</code> that tracks user data, perhaps with <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.tableland.xyz/smart-contracts/">smart contracts</a> or the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.tableland.xyz/sdk/">Tableland SDK</a>.</p><pre data-type="codeBlock" text="# Track user data in a `sessions` table
CREATE TABLE sessions (
  id INTEGER PRIMARY KEY,
  owner TEXT NOT NULL,
  start_block TEXT,
  end_block TEXT
);
"><code># Track <span class="hljs-keyword">user</span> data <span class="hljs-keyword">in</span> a `sessions` <span class="hljs-keyword">table</span>
<span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">TABLE</span> sessions (
  id <span class="hljs-type">INTEGER</span> <span class="hljs-keyword">PRIMARY</span> KEY,
  owner TEXT <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">NULL</span>,
  start_block TEXT,
  end_block TEXT
);
</code></pre><p>Let’s also assume our app only works if the user owns a specific NFT; a token-gated workflow. When users connect to your app, you’ll check for NFT ownership and then insert data into this table with the <code>start_block</code> (i.e., the time at which the session begins). Upon exiting the session, you’ll update the user’s session row with an <code>end_block</code>. So, you’ll execute various INSERT INTO and UPDATE statements that will change the user table’s data.</p><p>This provides us with a simple setup where we can check how long the user was using the app—Discord roles will be awarded to the most active community members!</p><h3 id="h-discord-setup" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Discord Setup</h3><p>Our Discord bot will connect to Vulcan through their <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://help.vulcan.xyz/custom-webhook">custom webhooks</a> to retrieve the associated wallet address for verified community members. It’s a pretty simple implementation. Vulcan will post to an endpoint with the following:</p><pre data-type="codeBlock" text="{
  &quot;wallet&quot;: [wallet_address],
}
"><code><span class="hljs-punctuation">{</span>
  <span class="hljs-attr">"wallet"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span>wallet_address<span class="hljs-punctuation">]</span><span class="hljs-punctuation">,</span>
<span class="hljs-punctuation">}</span>
</code></pre><p>If the user/address should be given a role, the endpoint needs to respond with a JSON object containing <code>&apos;success&apos;: true</code> and a 200 response code. Before returning a response, you will want to implement functionality to determine what the response should be. For example, you might use a query like the following with a <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.tableland.xyz/gateway-api">Gateway API call</a> or the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.tableland.xyz/sdk/">Tableland SDK</a>:</p><pre data-type="codeBlock" text="SELECT
  SUM(coalesce(end_time, BLOCK_NUM(1)) - start_time)
FROM
  &lt;sessions&gt; # The full table name that stores all user sessions
WHERE
  lower(owner) = lower(&apos;&lt;wallet_address&gt;&apos;) # Passed wallet address
"><code><span class="hljs-keyword">SELECT</span>
  <span class="hljs-built_in">SUM</span>(<span class="hljs-built_in">coalesce</span>(end_time, BLOCK_NUM(<span class="hljs-number">1</span>)) <span class="hljs-operator">-</span> start_time)
<span class="hljs-keyword">FROM</span>
  <span class="hljs-operator">&#x3C;</span>sessions<span class="hljs-operator">></span> # The <span class="hljs-keyword">full</span> <span class="hljs-keyword">table</span> name that stores <span class="hljs-keyword">all</span> <span class="hljs-keyword">user</span> sessions
<span class="hljs-keyword">WHERE</span>
  <span class="hljs-built_in">lower</span>(owner) <span class="hljs-operator">=</span> <span class="hljs-built_in">lower</span>(<span class="hljs-string">'&#x3C;wallet_address>'</span>) # Passed wallet address
</code></pre><p>If the query returns <strong><em>any</em></strong> session data for this wallet address, there could be a “basic” Discord role added for this member that indicates they’ve at least logged into the app. If a user has above some threshold of time spent on the app, this could help identify them as a “power user” and give them a more powerful role, such as accessing new channels. If no session data exists, then there is no role given to the user.</p><p>As for <strong><em>creating</em></strong> your webhook, again, there are many different to do this. Let’s say you have a web app built with <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://vitejs.dev/">Vite</a> + NextJS, deployed with Vercel. You could use <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://vercel.com/docs/concepts/functions/edge-functions">Vercel Edge Functions</a> and create a custom API route to return the required data to determine the Vulcan response we outlined above!</p><h2 id="h-example-tableland-rigs-community" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Example: Tableland Rigs Community</h2><p>In general, we’ve been thinking a lot about what <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tableland.xyz/rigs"><em>Flight Time (”FT”)</em></a> can do for Rigs owners—if you’ve haven’t checked out our deep dive on <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://mirror.xyz/tableland.eth/q95HrGD4n-5fkGgm8ONjA2Wmvz-dosQX7kAHLZYCIiQ">token reputation</a>, this will give you some additional insight. Recall that if you own a Rig, you can actively “use” it via actions like piloting and parking. When you pilot your Rig, you start earning FT (which disables marketplace trades), and upon parking you stop accruing FT (but it re-enables trading).</p><blockquote><p><strong>Important</strong>: You <strong>must</strong> re-verify with Vulcan in order to access this feature in the Tableland Discord.</p></blockquote><p>The Tableland server’s Discord bot uses FT as a way to attach roles to community members, based on their total FT. A quick reminder on what the Tableland Rigs and FT encompass:</p><ul><li><p>The <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://opensea.io/collection/tableland-rigs">Rigs NFT</a> uses the Tableland protocol for its metadata, enabling a dynamic and fully web3-driven NFT.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://garage.tableland.xyz/">The Garage application</a> provides a way to interact with your owned Rigs, including the ability to <em>pilot</em> or <em>park</em> your NFTs and even <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://mirror.xyz/tableland.eth/L7POIOyHAx6A_BczcOQ_h_bY5iz1UzWAXKkEcEebtDU">vote on proposals, weighted by FT</a>.</p></li></ul><p>In other words, building up a bunch of FT shows your commitment to the Rigs project by displaying a Discord badge and reflects your dedication as a community member!</p><h3 id="h-implementation" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Implementation</h3><p>You can see this logic in our <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/rigs/tree/main/garage/api/ft/%5Bfrom%5D/%5Bto%5D">Rigs repo</a> under the <code>garage/api/ft/[from]/[to]</code> directory. There, a simple query is made to Tableland tables:</p><pre data-type="codeBlock" text="SELECT
  SUM(ft) as &quot;ft&quot;
FROM
  (
    SELECT
      (coalesce(end_time, BLOCK_NUM(1)) - start_time) as &quot;ft&quot;
    FROM
      &lt;pilot_sessions&gt; # The table that stores all piloting/parking sessions
    WHERE
      lower(owner) = lower(&apos;&lt;wallet_address&gt;&apos;) # Passed wallet address
    UNION ALL
    SELECT
      amount as &quot;ft&quot;
    FROM
      &lt;ft_rewards&gt; # The table that stores *additional* FT rewards for addresses
    WHERE
      lower(recipient) = lower(&apos;&lt;wallet_address&gt;&apos;) # Passed wallet address
  )
"><code><span class="hljs-keyword">SELECT</span>
  <span class="hljs-built_in">SUM</span>(ft) <span class="hljs-keyword">as</span> "ft"
<span class="hljs-keyword">FROM</span>
  (
    <span class="hljs-keyword">SELECT</span>
      (<span class="hljs-built_in">coalesce</span>(end_time, BLOCK_NUM(<span class="hljs-number">1</span>)) <span class="hljs-operator">-</span> start_time) <span class="hljs-keyword">as</span> "ft"
    <span class="hljs-keyword">FROM</span>
      <span class="hljs-operator">&#x3C;</span>pilot_sessions<span class="hljs-operator">></span> # The <span class="hljs-keyword">table</span> that stores <span class="hljs-keyword">all</span> piloting<span class="hljs-operator">/</span>parking sessions
    <span class="hljs-keyword">WHERE</span>
      <span class="hljs-built_in">lower</span>(owner) <span class="hljs-operator">=</span> <span class="hljs-built_in">lower</span>(<span class="hljs-string">'&#x3C;wallet_address>'</span>) # Passed wallet address
    <span class="hljs-keyword">UNION</span> <span class="hljs-keyword">ALL</span>
    <span class="hljs-keyword">SELECT</span>
      amount <span class="hljs-keyword">as</span> "ft"
    <span class="hljs-keyword">FROM</span>
      <span class="hljs-operator">&#x3C;</span>ft_rewards<span class="hljs-operator">></span> # The <span class="hljs-keyword">table</span> that stores <span class="hljs-operator">*</span>additional<span class="hljs-operator">*</span> FT rewards <span class="hljs-keyword">for</span> addresses
    <span class="hljs-keyword">WHERE</span>
      <span class="hljs-built_in">lower</span>(recipient) <span class="hljs-operator">=</span> <span class="hljs-built_in">lower</span>(<span class="hljs-string">'&#x3C;wallet_address>'</span>) # Passed wallet address
  )
</code></pre><p>This makes queries to two tables:</p><ul><li><p>One that stores <strong><em>all</em></strong> of the piloting/parking session data, which is what gets displayed in The Garage and holds the majority of FT data. This comes from the Rigs/Pilots contract itself.</p></li><li><p>Another that stores FT rewards—additional FT that can be rewarded to holders, outside of the piloting/parking actions (e.g., off-chain rewards).</p></li></ul><p>The query is made using <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://vercel.com/docs/concepts/functions/edge-functions">Vercel Edge Functions</a>. Then, our API allows a custom <code>from</code> and <code>to</code> FT amount range to be specified, which is used in determining the different roles for each user. For example, you can make a query in The Garage to check if a wallet has earned between 1M to 5M FT:</p><pre data-type="codeBlock" text="https://garage.tableland.xyz/api/ft/1000000/5000000
"><code><span class="hljs-symbol">https:</span>/<span class="hljs-regexp">/garage.tableland.xyz/api</span><span class="hljs-regexp">/ft/</span><span class="hljs-number">1000000</span>/<span class="hljs-number">5000000</span>
</code></pre><h3 id="h-community-usage" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Community Usage</h3><p>Let’s review the steps you need to use this feature. If you’re already a community member and <strong>own a Rig</strong>, it’s likely you’ve already done this but <strong>will need to verify again</strong>:</p><ol><li><p>Join/enter the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tableland.xyz/discord">Tableland Discord server</a>.</p></li><li><p>Verify your account using Vulcan in the <code>#rigs-verfiy</code> <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://discord.com/channels/592843512312102924/996917151208329317">channel</a>.</p></li><li><p>That’s it!</p></li></ol><p>Once you’re (re-)verified, the logic described above will make the proper queries and add roles/badges to your Discord account. That is, as your Rigs continue to earn FT, you’ll automatically have the roles added/upgraded—no additional actions are required!</p><h2 id="h-how-can-someone-use-it" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">How Can Someone Use It?</h2><p>Above, we walked through how developers can build this functionality into their Discord servers. We used Vulcan to attribute Discord members with their on-chain accounts/addresses. Then, we combined that information with user application data stored in Tableland tables, and custom webhooks return a boolean that attaches a user’s Discord role.</p><p>Developers can also reference the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/rigs/tree/main/garage">Rigs GitHub repo</a> and the API we’ve built in <code>garage</code> directory; the <code>ethereum</code> directory is where the tables are created. In general, this concept could be recreated with <em>the same</em> FT-related tables we’ve noted above, or you could create a similar setup but with your own reputation-related data.</p><p>In the Rigs setup, the Discord bot uses an existing <code>pilot_sessions</code> table that stores core reputation data from user-initiated smart contract piloting and parking methods. The secondary <code>ft_rewards</code> table gives you the ability to add one-off FT that rewards users for something they’ve done, such as an on-chain action for voting on a proposal or an off-chain action like contributing to open source code. Depending on your needs, you might have a smart contract writing data on-chain to the <code>ft_rewards</code> table, or you might have an off-chain-only use case where you can simply use the Tableland SDK to mutate the table after a community member completes some task.</p><h2 id="h-get-in-touch" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Get in Touch!</h2><p>Be sure to give us a <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/tableland">follow on Twitter</a> and <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tableland.xyz/discord">hop into our Discord</a> if you’d like to discuss this design or the protocol in more detail—and give the <code>rigs</code> <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/rigs">repo</a> a star on GitHub if you found this useful.</p><p>And if you’re not familiar with Tableland Rigs, check out the collection on <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://opensea.io/collection/tableland-rigs">OpenSea</a>, and <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://garage.tableland.xyz/">The Garage</a> app is open for anyone to view!</p>]]></content:encoded>
            <author>tableland-2@newsletter.paragraph.com (Tableland)</author>
            <enclosure url="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/8d5f07907c485eca0855ddf6e86e0d946e6b108409b1a6fedb680f9bb0a0dcf8.png" length="0" type="image/png"/>
        </item>
        <item>
            <title><![CDATA[Garage Voting: NFT Owner Reputation & Rewards]]></title>
            <link>https://paragraph.com/@tableland-2/garage-voting-nft-owner-reputation-rewards</link>
            <guid>P3H2lKJUTSM1ULluAidk</guid>
            <pubDate>Wed, 09 Aug 2023 20:36:25 GMT</pubDate>
            <description><![CDATA[What Is It?Tableland has launched an NFT-based voting mechanism for Rigs! If you’re new to Tableland, there are a few components to be aware of:The protocol itself—a decentralized cloud database, built on SQLite, that allows on-chain SQL to drive off-chain database changes.Tableland Rigs NFT, which uses the protocol for its metadata and is the basis for The Garage.The Garage application, which provides a way to interact with your owned Rigs, browse others’ activity, and reads data stored on t...]]></description>
            <content:encoded><![CDATA[<h2 id="h-what-is-it" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">What Is It?</h2><p>Tableland has launched an NFT-based voting mechanism for Rigs! If you’re new to Tableland, there are a few components to be aware of:</p><ul><li><p>The protocol itself—a decentralized cloud database, built on SQLite, that allows on-chain SQL to drive off-chain database changes.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://opensea.io/collection/tableland-rigs">Tableland Rigs NFT</a>, which uses the protocol for its metadata and is the basis for The Garage.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://garage.tableland.xyz/">The Garage application</a>, which provides a way to interact with your owned Rigs, browse others’ activity, and reads data stored on the protocol.</p></li></ul><p>Rig owners have the ability to <em>pilot</em> or <em>park</em> their NFTs. Upon piloting, the Rig starts accruing block-based <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tableland.xyz/rigs"><strong>Flight Time (”FT”)</strong></a> and stops accruing it when you park.</p><p>The Garage’s new <em>Voting</em> feature provides a way to reward those who have earned FT! When voting proposals are created, it’s not simply Rig ownership that gates the ability to vote. It’s the <em>FT earned</em> that dictates voting abilities and power, and casting a vote also increases FT as a reputation reward for participating!</p><blockquote><p>A proposal’s outcome is no longer solely tied to the <em>size of your bag</em> but <em>how much you care about</em> what’s in it!</p></blockquote><h2 id="h-why-did-we-do-it" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Why Did We Do It?</h2><h3 id="h-background" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Background</h3><p>We launched Tableland Rigs <em>just</em> over a year ago, and a few months later, we released The Garage. From this point forward, any Rig token owner could choose to pilot their Rig, putting it “<em>in flight.</em>” Users could even set a custom pilot—your other NFTs could “operate” your Rigs, displayed via the metadata. Rigs started to earn FT! That initial release also included a “training” mechanism where you had to earn 172800 blocks (~30 days) of FT before you could first operate a Rig with a custom pilot.</p><p>This whole idea of piloting your Rig is a parallel to “soft” staking, where the Rigs owner never gives up ownership, but the token is blocked from transfers while it’s in flight. In this way, FT is earned as a reputation reward for your belief and commitment to the project. Parking is synonymous to unstaking in this comparison.</p><p>So, we started thinking more and more about <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://mirror.xyz/tableland.eth/q95HrGD4n-5fkGgm8ONjA2Wmvz-dosQX7kAHLZYCIiQ">token-based reputation</a> and how it can be used in web3, more broadly. How could we turn FT into something that we could actually use to drive engagement within the community, and reward owners appropriately?</p><p>We were already using community votes in various ways within the Tableland ecosystem, and Rig holders were our primary voters, so we decided to start there!</p><h3 id="h-alternatives" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Alternatives</h3><p>Garage voting will now use FT as a core feature. The more FT you’ve earned, the more impact you can have on a vote! But, why would we develop our own voting mechanism instead of using a tool like <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://snapshot.org/#/">Snapshot</a>?</p><p>With Snapshot, you can definitely create token-gated voting in a similar fashion, but the weights are <strong>only</strong> based on the <strong>number of NFTs</strong> you own. There isn’t a way to automatically pull in additional data that could be valuable to the vote—in this case, the Rig’s total FT earned as a reputation proxy.</p><p>The limitation around Snapshot’s proposal/vote features was a key driver for developing our own voting mechanism. Not only that, but rewarding users for their participation is not a native Snapshot feature, which is another capability that the Garage’s voting enables!</p><h2 id="h-what-does-it-do" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">What Does it Do?</h2><p>Voting gives Rigs owners a say in the decision-making process. A proposal’s votes are a product of <strong>voting power</strong> based on total FT. It gives the most committed community members—those who’ve been piloting their Rigs for the longest—a greater say in the vote’s outcome. Additionally, voting will reward those who cast a vote by increasing their total FT by a set amount!</p><p>For example:</p><ul><li><p>Holder <em>A</em> owns 5 Rigs, but never piloted any of them.</p></li><li><p>Holder <em>B</em> owns 3 Rigs, but only starting flying them yesterday.</p></li><li><p>Holder <em>C</em> owns 1 Rig, and they’ve been flying since day one.</p></li></ul><p>Here, holder <em>A</em> has <strong>zero</strong> voting power—they never earned any FT! Holder <em>B</em> has <strong>some</strong> voting power, but holder <em>C</em> has <strong><em>the most</em></strong>. Upon casting their votes, holder <em>C</em> will have the greatest impact on the outcome; reputation driven results!</p><h2 id="h-how-does-it-work" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">How Does It Work?</h2><p>Upon logging into The Garage, you’ll see any newly created proposals in the dashboard. An admin from the Tableland team will create a proposal, including:</p><ul><li><p>Proposal’s name</p></li><li><p>Description</p></li><li><p>Vote options</p></li><li><p>Weighted or non-weighted</p></li><li><p>Participation reward (FT)</p></li><li><p>Timeline (starting/ending block number)</p></li></ul><p>Creating a new proposal will initially take a snapshot of all owner’s current FT. Users are then able to connect their wallets and cast their votes. If weighted votes are enabled, you can also distribute your voting power across various options. When the proposal timeline ends, the defined participation reward is then added to the owner’s total FT balance.</p><h2 id="h-how-did-we-make-it" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">How Did We Make It?</h2><p>For the developers in the house, you might be interested to know more about how we built this, and how you might be able to use Tableland and similar mechanisms for your own community reputation based voting!</p><h3 id="h-smart-contracts" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Smart Contracts</h3><p>You can check out the Voting (and other) contracts in the Rigs GitHub repo’s <code>VotingRegistry</code> <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/rigs/blob/main/ethereum/contracts/VotingRegistry.sol">contract</a> for the full implementation. The <code>IVotingRegistry</code> <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/rigs/blob/main/ethereum/contracts/IVotingRegistry.sol">interface contract</a> is pretty straightforward—a few callouts:</p><ul><li><p><code>Proposal</code>: struct that holds information about a proposal (name, timeline, rewards, etc.).</p></li><li><p><code>createProposal</code>: method for some to create a proposal—e.g., a contract admin set through an inherited OpenZeppelin <em>AccessControl</em> contract.</p></li><li><p><code>vote</code>: a user’s casted vote(s) for a proposal ID, option weights, and comments.</p></li><li><p><code>distributeVoterRewards</code>: called externally and <strong>only</strong> distributes rewards if the ending block number for the vote has occurred.</p></li></ul><p>There are also a number of Tableland tables created or used in the design, shown in the <code>VotingRegistry.sol</code> contract and set in the constructor. These are shown below with some slight variable name tweaks for readability purposes and notes the table’s column names:</p><pre data-type="codeBlock" text="// Store the proposal information (ID, name, description CID, etc.)
proposals
// Store the options available for a given proposal (proposal ID, ID, description)
options
// When a proposal is created, take a snapshot of current FT for owners (address, FT, proposal ID)
ftSnapshot
// Track the votes for a proposal (address, proposal ID, option ID, weight)
votes
// When a vote closes, track rewards for participation (block #, recipient, reason, amount, proposal ID)
ftRewards
// This is a reference to the Rigs/Pilots contract—it&apos;s where FT is stored for piloting/parking sessions
pilotSessions 
"><code><span class="hljs-comment">// Store the proposal information (ID, name, description CID, etc.)</span>
proposals
<span class="hljs-comment">// Store the options available for a given proposal (proposal ID, ID, description)</span>
options
<span class="hljs-comment">// When a proposal is created, take a snapshot of current FT for owners (address, FT, proposal ID)</span>
ftSnapshot
<span class="hljs-comment">// Track the votes for a proposal (address, proposal ID, option ID, weight)</span>
votes
<span class="hljs-comment">// When a vote closes, track rewards for participation (block #, recipient, reason, amount, proposal ID)</span>
ftRewards
<span class="hljs-comment">// This is a reference to the Rigs/Pilots contract—it's where FT is stored for piloting/parking sessions</span>
pilotSessions 
</code></pre><p>When a proposal is created, the following occurs:</p><ol><li><p>Insert a new row in the <em>proposals</em> table.</p></li><li><p>Insert options in the <em>options</em> table.</p></li><li><p>Snapshot all voting power and store it in <strong><em>ftSnaphot</em></strong>.</p></li><li><p>Insert the “cross product” of all eligible voter addresses and options and <strong><em>votes</em></strong>.</p></li></ol><p>Later, voting will make updates to the <strong><em>votes</em></strong> table, which looks something like this:</p><pre data-type="codeBlock" text="UPDATE
  votes
SET
  weight = CASE
    option_id
    WHEN X1 THEN Y1
    WHEN X2 THEN Y2
    ELSE 0
  END
WHERE
  lower(address) = lower(msg.sender);
"><code>UPDATE
  votes
SET
  weight <span class="hljs-operator">=</span> CASE
    option_id
    WHEN X1 THEN Y1
    WHEN X2 THEN Y2
    ELSE <span class="hljs-number">0</span>
  END
WHERE
  lower(<span class="hljs-keyword">address</span>) <span class="hljs-operator">=</span> lower(<span class="hljs-built_in">msg</span>.<span class="hljs-built_in">sender</span>);
</code></pre><p>Once voting has ended, the <strong>ftRewards</strong> table gets updated:</p><pre data-type="codeBlock" text="INSERT INTO
  ft_rewards (block_num, recipient, reason, amount, vote_id)
SELECT
  DISTINCT BLOCK_NUM(),
  address,
  &apos;participated in vote&apos;,
  amount,
  proposal_id
FROM
  votes
WHERE
  weight &gt; 0
  and proposal_id = ?
"><code><span class="hljs-keyword">INSERT</span> <span class="hljs-keyword">INTO</span>
  ft_rewards (block_num, recipient, reason, amount, vote_id)
<span class="hljs-keyword">SELECT</span>
  <span class="hljs-keyword">DISTINCT</span> BLOCK_NUM(),
  address,
  <span class="hljs-string">'participated in vote'</span>,
  amount,
  proposal_id
<span class="hljs-keyword">FROM</span>
  votes
<span class="hljs-keyword">WHERE</span>
  weight <span class="hljs-operator">></span> <span class="hljs-number">0</span>
  <span class="hljs-keyword">and</span> proposal_id <span class="hljs-operator">=</span> ?
</code></pre><p>There’s a lot more SQL and useful information in the actual contracts, to be sure to check them out!</p><h3 id="h-web-app" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Web App</h3><p>The Garage is a React app (via <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://vitejs.dev/">Vite</a>), deployed using <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://vercel.com/">Vercel</a>. You can check it out for yourself as well as <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/rigs/tree/main/garage">the source code</a> with voting-related pages/components; its purpose-built for Rigs and offers the following:</p><ul><li><p>Global dashboard—view stats about all Rigs, like the number piloted or FT across all of them.</p></li><li><p>Personal dashboard—view information about your owned Rigs.</p></li><li><p>Gallery—filter the Rigs collection and peak into the NFT details.</p></li><li><p>Proposals—enter to check out current or past proposals and cast your vote.</p></li></ul><p>The app uses wagmi + RainbowKit for the wallet connection, and the styles are set with Tailwind CSS. From there, a number of other packages help make connections to the Rigs and Voting contracts, execute methods, pull in NFT data, and query information directly from the Tableland network.</p><h3 id="h-a-note-on-ft-and-storage" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">A Note on FT &amp; Storage</h3><p>A core theme to understand is <strong><em>where</em></strong> FT is aggregated. The Tableland protocol uses on-chain actions to drive off-chain data materialization. Instead of storing total FT earned on the blockchain (here, it’s Ethereum), only a subset of that data is stored on-chain—essentially, the most recent piloting/parking action taken. That is, the total FT earned is <em>not</em> stored in the contract.</p><p>When a user pilots or parks, it runs SQL statements through a “registry” smart contract. This allows for significant gas and cost savings for the token owners. All piloting/parking still takes place on-chain and stored there as <em>calldata,</em> ensuring anyone can recreate the state of a Rig’s total FT. The <strong>full history</strong> for <em>all</em> FT data itself exists off-chain on the Tableland protocol!</p><h2 id="h-how-can-someone-use-it" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">How Can Someone Use It?</h2><h3 id="h-rigs-community" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Rigs Community</h3><p>Once a vote is live, if you’re a Rig owner (with earned FT), you can participate within the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://garage.tableland.xyz/">Garage</a>. Stay tuned for the first vote—we’ll make announcements on both Discord &amp; Twitter, but here’s a sneak peak!</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/5e29ea20bcedb8ca6be462879f3dfeab43bacd80467d09e6bd9edd60a8f94ade.png" alt="An example of a closed proposal with a few votes, dictating the outcome and rewarding FT for participation." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">An example of a closed proposal with a few votes, dictating the outcome and rewarding FT for participation.</figcaption></figure><h3 id="h-developers-and-their-own-community" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Developers &amp; Their Own Community</h3><p>For the developers out there, check out the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/rigs">Rigs GitHub repo</a>, including the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/rigs/tree/main/ethereum/contracts">ethereum</a> and <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/rigs/tree/main/garage">garage</a> directories to see the Rigs/voting smart contracts and web app, respectively. The implementation details described above can be replicated with your own design. For example, see the <code>VotingRegistry</code> <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/rigs/blob/main/ethereum/contracts/VotingRegistry.sol">contract</a> and <code>IVotingRegistry</code> <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/rigs/blob/main/ethereum/contracts/IVotingRegistry.sol">interface</a> for the voting/proposal layout.</p><p>Namely, the Voting hinges on looking up data from a <code>pilot_sessions</code> table that stores some sort of reputation-related information; this table was created when deploying the <code>TablelandPilots</code> <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/rigs/blob/main/ethereum/contracts/TablelandRigPilots.sol">contract</a>. In our case, the reputation calculation is the sum of FT for all sessions tied to an address. You might have a totally different way of collecting/storing that data. It could even be <strong><em>someone else’s table</em></strong>—e.g., you could use the FT data and tables in your own setup! Regardless, once you have a “core” reputation table, the rest of the Voting contract can be slightly tweaked to account for your specific setup.</p><h2 id="h-get-in-touch" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Get in Touch!</h2><p>If you’re not familiar with Tableland Rigs, check out the collection on <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://opensea.io/collection/tableland-rigs">OpenSea</a>, and <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://garage.tableland.xyz">The Garage app</a> is open for anyone to view—albeit, voting participation will only work for token owners.</p><p>And be sure to give us a <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/tableland">follow on Twitter</a> and <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://tableland.xyz/discord">hop into our Discord</a> if you’d like to discuss this design or the protocol in more detail, and give us a start on the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://github.com/tablelandnetwork/rigs">Rigs GitHub repo</a> if this walkthrough was beneficial!</p>]]></content:encoded>
            <author>tableland-2@newsletter.paragraph.com (Tableland)</author>
            <enclosure url="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/89bdd403791ffdf1c92deee94629214a794adb843dcff4188a995634a87ea0be.png" length="0" type="image/png"/>
        </item>
        <item>
            <title><![CDATA[Token Reputation: Chain-based Tracking using NFTs]]></title>
            <link>https://paragraph.com/@tableland-2/token-reputation-chain-based-tracking-using-nfts</link>
            <guid>XMJuuYTkVyJlq5fwKlQk</guid>
            <pubDate>Wed, 26 Jul 2023 16:54:15 GMT</pubDate>
            <description><![CDATA[Over the past few months, Tableland has been experimenting with quite a lot! One particular focus was thinking more broadly about token reputation, even so far as considering authoring an EIP that extends the base ERC-721. Ultimately, we’re sticking with our current design due to a few reasons that we’ll touch on, but we felt it was still worthwhile sharing the research! Generally, you can think of NFTs as a proxy for reputation.A brief of Rigs’ designWhen we first set down this path, the ide...]]></description>
            <content:encoded><![CDATA[<p>Over the past few months, Tableland has been experimenting with quite a lot! One particular focus was thinking more broadly about token reputation, even so far as considering authoring an EIP that extends the base <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://eips.ethereum.org/EIPS/eip-721">ERC-721</a>. Ultimately, we’re sticking with our current design due to a few reasons that we’ll touch on, but we felt it was still worthwhile sharing the research! Generally, you can think of NFTs as a proxy for reputation.</p><h2 id="h-a-brief-of-rigs-design" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">A brief of Rigs’ design</h2><p>When we first set down this path, the idea stemmed from our <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://opensea.io/collection/tableland-rigs">Tableland Rigs NFT</a>. There is a key feature wherein the token owner can currently do the following:</p><ul><li><p>“Train” a Rig for a period of 172800 blocks (~30 days).</p></li><li><p>Once trained, they can “pilot” the Rig with a custom NFT, where the NFT contract &amp; token ID pointers are stored only in the metadata.</p></li><li><p>While training or piloting, the token is earning “Flight Time” (or “FT”) and cannot be sold on marketplaces.</p></li><li><p>At any time, the owner can “park” the Rig and stop accruing flight time, allowing it to be sold.</p></li><li><p>These actions can be viewed as user sessions where a Tableland table stores the session ID, Rig token ID, starting block number, ending block number, and pilot contract address/token ID.</p></li></ul><p>Taking a step back, what’s really happening here is a concept of “soft” staking. The owner <em>always</em> maintains ownership of the token. The associated FT being earned is, essentially, a reputation for staking the token as it signifies belief in the core project since the token cannot be sold.</p><h2 id="h-thinking-reputation" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Thinking reputation</h2><p>Let’s review some of the technical specifics.</p><p>The Rigs contract has three primary methods: <code>trainRig</code>, <code>parkRig</code>, and <code>pilotRig</code>. In a more generalized approach, you can sort of combine the train and park methods into the soft stake. The parking method can be thought of as unstaking. When these methods are called, they emit information about the action, including the Rig token ID and pilot information. And for those curious, the transfer blocking simply comes from the <code>_beforeTokenTransfers</code> hook where there’s a check for if the token’s session is active (i.e., <code>pilotStartTime</code> greater than zero). Once a user parks the Rig, this is the end of the soft staking session.</p><p>This setup is a bit specialized for the Rigs use case because the staking action needs to take additional inputs for the owner’s custom pilot (contract address and token ID). In a standard staking case, it’s typically just the token ID. When this happens, that data is not only emitted in an event but also passes SQL statements to the Tableland Registry contract. Below is an example of how you can think about this setup, which disregards the piloting aspect for simplicity.</p><p>In the broader sense, you could imagine how you can use reputation for, really, anything by using ERC-721 tokens (and their metdata!) as the core primitive. The overarching actions the owner takes using that token are all on-chain, so all of that data can be used, as in the Rigs use case, a simple aggregation of the block-based FT earned. But, this could be extended to other use cases that aren’t about NFTs as collectibles (e.g., DeFi, AI/ML, etc.).</p><blockquote><p>Note: for a <em>full</em> reputation contract example, review the contract at the bottom of this writeup.</p></blockquote><h3 id="h-example-sql-setup" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Example SQL setup</h3><p>Upon contract deployment, you could set up a <code>token_reputation</code> table to track the aforementioned session data. With staking/unstaking actions, you either insert into the table with starting session data or update that row with ending session data. Thus, you can query the total reputation earned for a single owner/token and subsequently use that off-chain (e.g., stored in metadata, community building, etc.).</p><pre data-type="codeBlock" text="/* Example: tracking reputation using sessions */

/* Create a table to track reputation */
CREATE TABLE token_reputation (
  id INTEGER PRIMARY KEY, /* Session ID */
  token_id INTEGER NOT NULL, /* NFT token ID */
  owner TEXT NOT NULL, /* Token owner&apos;s address */
  start_time INTEGER NOT NULL, /* Starting block number */
  end_time INTEGER /* Ending block number */
);

/* Process event at `block.number` value of `100`, and also emit event */
INSERT INTO
  token_reputation (token_id, owner, start_time)
VALUES
  (1, &apos;0x1234...&apos;, 100);

/* Process event at `block.number` value of `175`, and also emit event */
UPDATE
  token_reputation SET end_time = 175
WHERE
  token_id = 1 AND
  owner = &apos;0x1234...&apos; AND
  end_time IS NULL;

/* Query reputation for address `0x1234...` */
SELECT
  SUM(end_time - start_time)
FROM
  token_reputation
WHERE
  owner = &apos;0x1234...&apos;;
/* Returns a reputation value of: 75 */
"><code><span class="hljs-comment">/* Example: tracking reputation using sessions */</span>

<span class="hljs-comment">/* Create a table to track reputation */</span>
<span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">TABLE</span> token_reputation (
  id <span class="hljs-type">INTEGER</span> <span class="hljs-keyword">PRIMARY</span> KEY, <span class="hljs-comment">/* Session ID */</span>
  token_id <span class="hljs-type">INTEGER</span> <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">NULL</span>, <span class="hljs-comment">/* NFT token ID */</span>
  owner TEXT <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">NULL</span>, <span class="hljs-comment">/* Token owner's address */</span>
  start_time <span class="hljs-type">INTEGER</span> <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">NULL</span>, <span class="hljs-comment">/* Starting block number */</span>
  end_time <span class="hljs-type">INTEGER</span> <span class="hljs-comment">/* Ending block number */</span>
);

<span class="hljs-comment">/* Process event at `block.number` value of `100`, and also emit event */</span>
<span class="hljs-keyword">INSERT</span> <span class="hljs-keyword">INTO</span>
  token_reputation (token_id, owner, start_time)
<span class="hljs-keyword">VALUES</span>
  (<span class="hljs-number">1</span>, <span class="hljs-string">'0x1234...'</span>, <span class="hljs-number">100</span>);

<span class="hljs-comment">/* Process event at `block.number` value of `175`, and also emit event */</span>
<span class="hljs-keyword">UPDATE</span>
  token_reputation <span class="hljs-keyword">SET</span> end_time <span class="hljs-operator">=</span> <span class="hljs-number">175</span>
<span class="hljs-keyword">WHERE</span>
  token_id <span class="hljs-operator">=</span> <span class="hljs-number">1</span> <span class="hljs-keyword">AND</span>
  owner <span class="hljs-operator">=</span> <span class="hljs-string">'0x1234...'</span> <span class="hljs-keyword">AND</span>
  end_time <span class="hljs-keyword">IS</span> <span class="hljs-keyword">NULL</span>;

<span class="hljs-comment">/* Query reputation for address `0x1234...` */</span>
<span class="hljs-keyword">SELECT</span>
  <span class="hljs-built_in">SUM</span>(end_time <span class="hljs-operator">-</span> start_time)
<span class="hljs-keyword">FROM</span>
  token_reputation
<span class="hljs-keyword">WHERE</span>
  owner <span class="hljs-operator">=</span> <span class="hljs-string">'0x1234...'</span>;
<span class="hljs-comment">/* Returns a reputation value of: 75 */</span>
</code></pre><h3 id="h-example-solidity-setup" class="text-2xl font-header !mt-6 !mb-4 first:!mt-0 first:!mb-0">Example Solidity setup</h3><p>On the Solidity side of things, there’s a bit more nuance. But, we can look at another simplification from the Rigs contract to understand how this SQL works on-chain. Take, for example, a method that ends the staking session—this is how the Rigs <code>parkRig</code> method works. This example makes use of OpenZeppelin’s strings library for casting data within the SQL statement and some additional Tableland-specific helpers:</p><ul><li><p>A <code>setters</code> string that stores the column to update.</p></li><li><p>A <code>filters</code> string to ensure only the correct session is updated—there <strong>is</strong> a bit of storage here with the <code>startTime</code> variable that stores when a reputation session begins (e.g., getter to a mapping or similar).</p></li><li><p>These are passed to <code>TablelandDeployments.get().mutate()</code>. The <code>get()</code> method returns an interface with the correct Tableland Registry address, allowing you to call the <code>mutate()</code> method that passes a mutating SQL statement—in this case, an UPDATE statement.</p></li><li><p>There are contract variables stored for <code>_TOKEN_REPUTATION_PREFIX</code> and <code>_tokenReputationTableId</code>. The prefix is a custom string the table creator sets, e.g., <code>token_reputation</code>. All tables have a globally unique name in the format <code>{prefix}_{chainId}_{tableId}</code>. The chain ID is inferred. Thus, the only missing piece is the unique table ID, stored as <code>_tokenReputationTableId</code>.</p></li><li><p>The <code>SQLHelpers</code> library takes these inputs and concatenates the inputs into an UPDATE statement to make the process a bit smoother.</p></li></ul><pre data-type="codeBlock" text="uint64 startTime = sessionStartTime(tokenId); // Getter for start time, stored in contract

string memory setters = string.concat(
    &quot;end_time=&quot;,
    Strings.toString(uint64(block.number))
);

// Only update the row with the matching `token_id` and `start_time`
string memory filters = string.concat(
    &quot;token_id=&quot;,
    Strings.toString(uint16(tokenId)),
    &quot; and &quot;,
    &quot;start_time=&quot;,
    Strings.toString(startTime) 
);

// Update the pilot information in the token_reputation sessions table
TablelandDeployments.get().mutate(
    address(this),
    _pilotSessionsTableId,
    SQLHelpers.toUpdate(
        _TOKEN_REPUTATION_PREFIX,
        _tokenReputationTableId,
        setters,
        filters
    )
);

// Emit some sort of event to indicate the reputation session has ended
emit Unstaked(tokenId);
"><code><span class="hljs-keyword">uint64</span> startTime <span class="hljs-operator">=</span> sessionStartTime(tokenId); <span class="hljs-comment">// Getter for start time, stored in contract</span>

<span class="hljs-keyword">string</span> <span class="hljs-keyword">memory</span> setters <span class="hljs-operator">=</span> <span class="hljs-built_in">string</span>.<span class="hljs-built_in">concat</span>(
    <span class="hljs-string">"end_time="</span>,
    Strings.toString(<span class="hljs-keyword">uint64</span>(<span class="hljs-built_in">block</span>.<span class="hljs-built_in">number</span>))
);

<span class="hljs-comment">// Only update the row with the matching `token_id` and `start_time`</span>
<span class="hljs-keyword">string</span> <span class="hljs-keyword">memory</span> filters <span class="hljs-operator">=</span> <span class="hljs-built_in">string</span>.<span class="hljs-built_in">concat</span>(
    <span class="hljs-string">"token_id="</span>,
    Strings.toString(<span class="hljs-keyword">uint16</span>(tokenId)),
    <span class="hljs-string">" and "</span>,
    <span class="hljs-string">"start_time="</span>,
    Strings.toString(startTime) 
);

<span class="hljs-comment">// Update the pilot information in the token_reputation sessions table</span>
TablelandDeployments.get().mutate(
    <span class="hljs-keyword">address</span>(<span class="hljs-built_in">this</span>),
    _pilotSessionsTableId,
    SQLHelpers.toUpdate(
        _TOKEN_REPUTATION_PREFIX,
        _tokenReputationTableId,
        setters,
        filters
    )
);

<span class="hljs-comment">// Emit some sort of event to indicate the reputation session has ended</span>
<span class="hljs-keyword">emit</span> Unstaked(tokenId);
</code></pre><p>With SQL mutating the off-chain Tableland database, all of the on-chain SQL can be read using Tableland’s Gateway API, SDK, CLI, etc. For example, maybe you want to compose this reputation data via a read query and return <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.tableland.xyz/fundamentals/concepts/nft-metadata">ERC-721 compatible</a> metadata. The SQL statement needed to generate this is a little dependent on each specific setup, but using the example SQL SELECT statement above along with SQLite’s built-in <code>json_group_array</code> and <code>json_object</code> functions. Generally, ERC-721 queries might look like <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.tableland.xyz/playbooks/walkthroughs/nft-metadata">this example</a> where you could add an additional JOIN to combine the standard NFT attributes with the reputation attribute.</p><p>Thus, your NFT’s attributes could be extended to include something like the following, which aggregates reputation data over all sessions.</p><pre data-type="codeBlock" text="{
  &quot;attributes&quot;: [
    {
      &quot;display_type&quot;: &quot;number&quot;,
      &quot;trait_type&quot;: &quot;Reputation&quot;,
      &quot;value&quot;: 75
    }
  ]
}
"><code><span class="hljs-punctuation">{</span>
  <span class="hljs-attr">"attributes"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span>
    <span class="hljs-punctuation">{</span>
      <span class="hljs-attr">"display_type"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"number"</span><span class="hljs-punctuation">,</span>
      <span class="hljs-attr">"trait_type"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Reputation"</span><span class="hljs-punctuation">,</span>
      <span class="hljs-attr">"value"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">75</span>
    <span class="hljs-punctuation">}</span>
  <span class="hljs-punctuation">]</span>
<span class="hljs-punctuation">}</span>
</code></pre><h2 id="h-existing-eips" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Existing EIPs</h2><p>When starting down the EIP path, we took a look at a few existing EIPs:</p><ul><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://eips.ethereum.org/EIPS/eip-5058">EIP-5058</a>: Designed with lock/unlock methods and events, and it includes other methods like setting time locked expiration and operator approvals.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://eips.ethereum.org/EIPS/eip-5192">EIP-5192</a>: Minimally designed for non-transferrable &quot;soulbound&quot; tokens (&quot;SBT&quot;) such that it’s largely event-based (emits <code>Locked</code> or <code>Unlocked</code>) and only has a <code>locked</code> getter method.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://eips.ethereum.org/EIPS/eip-5753">EIP-5753</a>: Although EIP-5753 is currently a draft and yet to be accepted, it is rather similar to EIP-5192 but adds additional features with <code>lock</code> and <code>unlock</code> methods, rather than only events and a getter.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://eips.ethereum.org/EIPS/eip-6982">EIP-6982</a>: This is a minimal interface that includes similar functionality to those mentioned but focuses on “default” locking (upon contract deployment) and a single <code>locked</code> method that emits true or false for the locked status.</p></li></ul><p>Ultimately, our potential EIP ended up being quite similar to these, so we felt it was better to, potentially, consider one of these in the future for composability purposes. Even though none of them were reputation-specific, they all speak to the core concept of staking/unstaking or locking/unlocking.</p><p>One interesting tidbit we learned here was about how marketplaces leverage these types of events. For example, <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.opensea.io/docs/metadata-standards#disable-trading-for-staked-or-locked-tokens">OpenSea</a> uses both of the events noted in EIP-5192 and EIP-5753 to <em>disable trading for staked/locked token</em>s. There are other accepted events by OpenSea, including the <code>Stake</code> and <code>Unstake</code> events. OpenSea noted the marketplace <strong>should</strong> respond to the <code>Stake</code> and <code>Unstake</code> event <strong>as long as they emit an indexed</strong> <code>tokenID</code> param. Namely, it appears their marketplace implements this functionality not with the event topic but the event name and arguments.</p><h2 id="h-key-differences-and-data-availability" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Key differences &amp; data availability</h2><p>Our approach was thinking about reputation from an on-chain driven, off-chain materialized perspective. We’ve designed Rigs in that we believe not all data <strong><em>needs</em></strong> to be <em>stored</em> on-chain in storage, as long as it exists in the Data Availability (”DA”) layer. If you take a look at the example SQL &amp; Solidity we walked through above, you might notice that the <strong><em>total</em></strong> reputation earned isn’t stored in the contract.</p><p>Whenever you start or end a session, this stores only the starting time of the session to help decrease storage costs. Instead, Tableland is materializing the SQL. Any valid SQL statement that’s sent to the Registry contract will be emitted as a <code>RunSQL</code> event, which includes pertinent information like the statement itself and access control rules. Here, the DA layer is capturing <strong><em>all</em></strong> of the token’s reputation such that <strong>anyone</strong> can recreate the token’s reputation state. This is an important callout because it’s a core principle with not only Rigs but the Tableland protocol, in general.</p><h2 id="h-final-thoughts" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Final thoughts</h2><p>Due to this design, it didn’t make the most sense to try and force an EIP into the mix since it was rather dependent on Tableland functionality. Albeit, the idea still holds! Taking some of the EIPs noted above, you could imagine implementing their <code>lock</code>/<code>unlock</code> or <code>stake</code>/<code>unstake</code> methods and associated events—and within these methods, use Tableland to store reputation-related data.</p><h2 id="h-generalized-reputation-contract" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Generalized reputation contract</h2><p>A final word—perhaps, implementing your own token reputation contract would look something like this! The example doesn’t implement a specific EIP but could use a custom interface with <code>stake</code>/<code>unstake</code> methods and events (with an <code>indexed tokenId</code> argument)—which, as mentioned, is something marketplaces like OpenSea actually use today.</p><pre data-type="codeBlock" text="// SPDX-License-Identifier: Unlicense
pragma solidity &gt;=0.8.10 &lt;0.9.0;

import &quot;@openzeppelin/contracts/token/ERC721/ERC721.sol&quot;;
import &quot;@openzeppelin/contracts/utils/Counters.sol&quot;;
import &quot;@openzeppelin/contracts/utils/Strings.sol&quot;;
import &quot;@tableland/evm/contracts/utils/SQLHelpers.sol&quot;;
import &quot;@tableland/evm/contracts/utils/TablelandDeployments.sol&quot;;
// You could implement some simply custom interface, such as stake/unstake methods and events
// Or, use one of the noted EIPs and alter the functionality accordingly
import &quot;./ITokenReputation.sol&quot;;

contract Reputatoin is ERC721, ITokenReputation {
    using Counters for Counters.Counter;

    Counters.Counter private _tokenIdCounter; // Incrementor for the token ID
    mapping(uint256 =&gt; bool) private _status; // Track the token&apos;s staking status: `true` (staked) or `false` (unstaked)
    uint256 private _tokenRepTableId; // Some reference used off-chain to store reputation data
    string private constant _REPUTATION_PREFIX = &quot;token_reputation&quot;; // Used off-chain but stored for interoperability purposes
    string private _baseURIString; // Base URI for metadata (set on deployment to https://tableland.network with URL params)

    constructor() ERC721(&quot;Tableland&quot;, &quot;TBL&quot;) {
        _tokenRepTableId = TablelandDeployments.get().create(
            address(this),
            SQLHelpers.toCreateFromSchema( // Some off-chain SQL database table schema
                &quot;id INTEGER PRIMARY KEY,&quot; // Track a staking session ID
                &quot;token_id INTEGER NOT NULL,&quot; // The specific token ID
                &quot;owner text NOT NULL,&quot; // The address that owns the token at that point in time
                &quot;start_time INTEGER NOT NULL,&quot; // Starting block for staking activity via `stake()`
                &quot;end_time INTEGER&quot;, // Ending block for staking activity via `unstake()`
                _REPUTATION_PREFIX // Some off-chain SQL database table name
            )
        );
        _baseURIString = &quot;https://tableland.network/api/v1/query?unwrap=true&amp;extract=true&amp;statement=&quot;;
    }

    function _baseURI() internal view override returns (string memory) {
        return
            string.concat(
                _baseURIString,
                &quot;select%20json_object(%27attributes%27,json_array(json_object(%27display_type%27,%27number%27,%27trait_type%27,%27Reputation%27,%27value%27,SUM(end_time-start_time))))%20from%20&quot;,
                SQLHelpers.toNameFromId(_REPUTATION_PREFIX, _tokenRepTableId),
                &quot;%20where%20token_id%20%3D%20&quot;
            );
    }

    function mint() external payable {
        uint256 tokenId = _tokenIdCounter.current();
        _tokenIdCounter.increment();
        _safeMint(_msgSender(), tokenId);
    }

    function stake(uint256 tokenId) external {
        address tokenOwner = _ownerOf(tokenId);
        require(_msgSender() == tokenOwner, &quot;UNAUTHORIZED&quot;); // Token owner or delegate can stake
        require(_status[tokenId] == false, &quot;INVALID STATUS&quot;); // Only currently unstaked tokens can be staked
        // Track the staking session with some off-chain metadata
        TablelandDeployments.get().mutate(
            address(this),
            _tokenRepTableId,
            SQLHelpers.toInsert(
                _REPUTATION_PREFIX,
                _tokenRepTableId,
                &quot;token_id,owner,start_time&quot;,
                string.concat(
                    Strings.toString(uint64(tokenId)), // Some implementation-specific casting
                    &quot;,&quot;,
                    SQLHelpers.quote(Strings.toHexString(tokenOwner)), // Track reputation for the token owner
                    &quot;,&quot;,
                    Strings.toString(uint64(block.number)) // Track the staking block number starting point
                )
            )
        );
        _status[tokenId] = true;
        emit Stake(tokenId, tokenOwner);
    }

    function unstake(uint256 tokenId) external {
        address tokenOwner = _ownerOf(tokenId);
        require(_msgSender() == tokenOwner, &quot;UNAUTHORIZED&quot;); // Token owner or delegate can unstake
        require(_status[tokenId] == true, &quot;INVALID STATUS&quot;); // Only currently staked tokens can be unstaked
        // Update the metadata&apos;s `end_time` to accrue block-based reputation, ending at the current `block.number`
        string memory setters = string.concat(
            &quot;end_time=&quot;,
            Strings.toString(uint64(block.number))
        );
        // Only update the row with the matching `token_id` and without an `end_time` (i.e., unfinished staking session)
        string memory filters = string.concat(
            &quot;token_id=&quot;,
            Strings.toString(uint64(tokenId)),
            &quot; AND &quot;,
            &quot;owner=&quot;,
            SQLHelpers.quote(Strings.toHexString(tokenOwner)),
            &quot; AND &quot;,
            &quot;end_time IS NULL&quot;
        );
        // Update the token reputation data by ending the current block-based staking session
        TablelandDeployments.get().mutate(
            address(this),
            _tokenRepTableId,
            SQLHelpers.toUpdate(
                _REPUTATION_PREFIX,
                _tokenRepTableId,
                setters,
                filters
            )
        );
        _status[tokenId] = false;
        emit Unstake(tokenId, tokenOwner);
    }

    function onERC721Received(
        address,
        address,
        uint256,
        bytes calldata
    ) public pure returns (bytes4) {
        return 0x150b7a02; // Allows this reputation contract to own NFTs, which may be useful in certain cases
    }
}
"><code><span class="hljs-comment">// SPDX-License-Identifier: Unlicense</span>
<span class="hljs-meta"><span class="hljs-keyword">pragma</span> <span class="hljs-keyword">solidity</span> >=0.8.10 &#x3C;0.9.0;</span>

<span class="hljs-keyword">import</span> <span class="hljs-string">"@openzeppelin/contracts/token/ERC721/ERC721.sol"</span>;
<span class="hljs-keyword">import</span> <span class="hljs-string">"@openzeppelin/contracts/utils/Counters.sol"</span>;
<span class="hljs-keyword">import</span> <span class="hljs-string">"@openzeppelin/contracts/utils/Strings.sol"</span>;
<span class="hljs-keyword">import</span> <span class="hljs-string">"@tableland/evm/contracts/utils/SQLHelpers.sol"</span>;
<span class="hljs-keyword">import</span> <span class="hljs-string">"@tableland/evm/contracts/utils/TablelandDeployments.sol"</span>;
<span class="hljs-comment">// You could implement some simply custom interface, such as stake/unstake methods and events</span>
<span class="hljs-comment">// Or, use one of the noted EIPs and alter the functionality accordingly</span>
<span class="hljs-keyword">import</span> <span class="hljs-string">"./ITokenReputation.sol"</span>;

<span class="hljs-class"><span class="hljs-keyword">contract</span> <span class="hljs-title">Reputatoin</span> <span class="hljs-keyword">is</span> <span class="hljs-title">ERC721</span>, <span class="hljs-title">ITokenReputation</span> </span>{
    <span class="hljs-keyword">using</span> <span class="hljs-title">Counters</span> <span class="hljs-title"><span class="hljs-keyword">for</span></span> <span class="hljs-title">Counters</span>.<span class="hljs-title">Counter</span>;

    Counters.Counter <span class="hljs-keyword">private</span> _tokenIdCounter; <span class="hljs-comment">// Incrementor for the token ID</span>
    <span class="hljs-keyword">mapping</span>(<span class="hljs-keyword">uint256</span> <span class="hljs-operator">=</span><span class="hljs-operator">></span> <span class="hljs-keyword">bool</span>) <span class="hljs-keyword">private</span> _status; <span class="hljs-comment">// Track the token's staking status: `true` (staked) or `false` (unstaked)</span>
    <span class="hljs-keyword">uint256</span> <span class="hljs-keyword">private</span> _tokenRepTableId; <span class="hljs-comment">// Some reference used off-chain to store reputation data</span>
    <span class="hljs-keyword">string</span> <span class="hljs-keyword">private</span> <span class="hljs-keyword">constant</span> _REPUTATION_PREFIX <span class="hljs-operator">=</span> <span class="hljs-string">"token_reputation"</span>; <span class="hljs-comment">// Used off-chain but stored for interoperability purposes</span>
    <span class="hljs-keyword">string</span> <span class="hljs-keyword">private</span> _baseURIString; <span class="hljs-comment">// Base URI for metadata (set on deployment to https://tableland.network with URL params)</span>

    <span class="hljs-function"><span class="hljs-keyword">constructor</span>(<span class="hljs-params"></span>) <span class="hljs-title">ERC721</span>(<span class="hljs-params"><span class="hljs-string">"Tableland"</span>, <span class="hljs-string">"TBL"</span></span>) </span>{
        _tokenRepTableId <span class="hljs-operator">=</span> TablelandDeployments.get().create(
            <span class="hljs-keyword">address</span>(<span class="hljs-built_in">this</span>),
            SQLHelpers.toCreateFromSchema( <span class="hljs-comment">// Some off-chain SQL database table schema</span>
                <span class="hljs-string">"id INTEGER PRIMARY KEY,"</span> <span class="hljs-comment">// Track a staking session ID</span>
                <span class="hljs-string">"token_id INTEGER NOT NULL,"</span> <span class="hljs-comment">// The specific token ID</span>
                <span class="hljs-string">"owner text NOT NULL,"</span> <span class="hljs-comment">// The address that owns the token at that point in time</span>
                <span class="hljs-string">"start_time INTEGER NOT NULL,"</span> <span class="hljs-comment">// Starting block for staking activity via `stake()`</span>
                <span class="hljs-string">"end_time INTEGER"</span>, <span class="hljs-comment">// Ending block for staking activity via `unstake()`</span>
                _REPUTATION_PREFIX <span class="hljs-comment">// Some off-chain SQL database table name</span>
            )
        );
        _baseURIString <span class="hljs-operator">=</span> <span class="hljs-string">"https://tableland.network/api/v1/query?unwrap=true&#x26;extract=true&#x26;statement="</span>;
    }

    <span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">_baseURI</span>(<span class="hljs-params"></span>) <span class="hljs-title"><span class="hljs-keyword">internal</span></span> <span class="hljs-title"><span class="hljs-keyword">view</span></span> <span class="hljs-title"><span class="hljs-keyword">override</span></span> <span class="hljs-title"><span class="hljs-keyword">returns</span></span> (<span class="hljs-params"><span class="hljs-keyword">string</span> <span class="hljs-keyword">memory</span></span>) </span>{
        <span class="hljs-keyword">return</span>
            <span class="hljs-built_in">string</span>.<span class="hljs-built_in">concat</span>(
                _baseURIString,
                <span class="hljs-string">"select%20json_object(%27attributes%27,json_array(json_object(%27display_type%27,%27number%27,%27trait_type%27,%27Reputation%27,%27value%27,SUM(end_time-start_time))))%20from%20"</span>,
                SQLHelpers.toNameFromId(_REPUTATION_PREFIX, _tokenRepTableId),
                <span class="hljs-string">"%20where%20token_id%20%3D%20"</span>
            );
    }

    <span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">mint</span>(<span class="hljs-params"></span>) <span class="hljs-title"><span class="hljs-keyword">external</span></span> <span class="hljs-title"><span class="hljs-keyword">payable</span></span> </span>{
        <span class="hljs-keyword">uint256</span> tokenId <span class="hljs-operator">=</span> _tokenIdCounter.current();
        _tokenIdCounter.increment();
        _safeMint(_msgSender(), tokenId);
    }

    <span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">stake</span>(<span class="hljs-params"><span class="hljs-keyword">uint256</span> tokenId</span>) <span class="hljs-title"><span class="hljs-keyword">external</span></span> </span>{
        <span class="hljs-keyword">address</span> tokenOwner <span class="hljs-operator">=</span> _ownerOf(tokenId);
        <span class="hljs-built_in">require</span>(_msgSender() <span class="hljs-operator">=</span><span class="hljs-operator">=</span> tokenOwner, <span class="hljs-string">"UNAUTHORIZED"</span>); <span class="hljs-comment">// Token owner or delegate can stake</span>
        <span class="hljs-built_in">require</span>(_status[tokenId] <span class="hljs-operator">=</span><span class="hljs-operator">=</span> <span class="hljs-literal">false</span>, <span class="hljs-string">"INVALID STATUS"</span>); <span class="hljs-comment">// Only currently unstaked tokens can be staked</span>
        <span class="hljs-comment">// Track the staking session with some off-chain metadata</span>
        TablelandDeployments.get().mutate(
            <span class="hljs-keyword">address</span>(<span class="hljs-built_in">this</span>),
            _tokenRepTableId,
            SQLHelpers.toInsert(
                _REPUTATION_PREFIX,
                _tokenRepTableId,
                <span class="hljs-string">"token_id,owner,start_time"</span>,
                <span class="hljs-built_in">string</span>.<span class="hljs-built_in">concat</span>(
                    Strings.toString(<span class="hljs-keyword">uint64</span>(tokenId)), <span class="hljs-comment">// Some implementation-specific casting</span>
                    <span class="hljs-string">","</span>,
                    SQLHelpers.quote(Strings.toHexString(tokenOwner)), <span class="hljs-comment">// Track reputation for the token owner</span>
                    <span class="hljs-string">","</span>,
                    Strings.toString(<span class="hljs-keyword">uint64</span>(<span class="hljs-built_in">block</span>.<span class="hljs-built_in">number</span>)) <span class="hljs-comment">// Track the staking block number starting point</span>
                )
            )
        );
        _status[tokenId] <span class="hljs-operator">=</span> <span class="hljs-literal">true</span>;
        <span class="hljs-keyword">emit</span> Stake(tokenId, tokenOwner);
    }

    <span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">unstake</span>(<span class="hljs-params"><span class="hljs-keyword">uint256</span> tokenId</span>) <span class="hljs-title"><span class="hljs-keyword">external</span></span> </span>{
        <span class="hljs-keyword">address</span> tokenOwner <span class="hljs-operator">=</span> _ownerOf(tokenId);
        <span class="hljs-built_in">require</span>(_msgSender() <span class="hljs-operator">=</span><span class="hljs-operator">=</span> tokenOwner, <span class="hljs-string">"UNAUTHORIZED"</span>); <span class="hljs-comment">// Token owner or delegate can unstake</span>
        <span class="hljs-built_in">require</span>(_status[tokenId] <span class="hljs-operator">=</span><span class="hljs-operator">=</span> <span class="hljs-literal">true</span>, <span class="hljs-string">"INVALID STATUS"</span>); <span class="hljs-comment">// Only currently staked tokens can be unstaked</span>
        <span class="hljs-comment">// Update the metadata's `end_time` to accrue block-based reputation, ending at the current `block.number`</span>
        <span class="hljs-keyword">string</span> <span class="hljs-keyword">memory</span> setters <span class="hljs-operator">=</span> <span class="hljs-built_in">string</span>.<span class="hljs-built_in">concat</span>(
            <span class="hljs-string">"end_time="</span>,
            Strings.toString(<span class="hljs-keyword">uint64</span>(<span class="hljs-built_in">block</span>.<span class="hljs-built_in">number</span>))
        );
        <span class="hljs-comment">// Only update the row with the matching `token_id` and without an `end_time` (i.e., unfinished staking session)</span>
        <span class="hljs-keyword">string</span> <span class="hljs-keyword">memory</span> filters <span class="hljs-operator">=</span> <span class="hljs-built_in">string</span>.<span class="hljs-built_in">concat</span>(
            <span class="hljs-string">"token_id="</span>,
            Strings.toString(<span class="hljs-keyword">uint64</span>(tokenId)),
            <span class="hljs-string">" AND "</span>,
            <span class="hljs-string">"owner="</span>,
            SQLHelpers.quote(Strings.toHexString(tokenOwner)),
            <span class="hljs-string">" AND "</span>,
            <span class="hljs-string">"end_time IS NULL"</span>
        );
        <span class="hljs-comment">// Update the token reputation data by ending the current block-based staking session</span>
        TablelandDeployments.get().mutate(
            <span class="hljs-keyword">address</span>(<span class="hljs-built_in">this</span>),
            _tokenRepTableId,
            SQLHelpers.toUpdate(
                _REPUTATION_PREFIX,
                _tokenRepTableId,
                setters,
                filters
            )
        );
        _status[tokenId] <span class="hljs-operator">=</span> <span class="hljs-literal">false</span>;
        <span class="hljs-keyword">emit</span> Unstake(tokenId, tokenOwner);
    }

    <span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">onERC721Received</span>(<span class="hljs-params">
        <span class="hljs-keyword">address</span>,
        <span class="hljs-keyword">address</span>,
        <span class="hljs-keyword">uint256</span>,
        <span class="hljs-keyword">bytes</span> <span class="hljs-keyword">calldata</span>
    </span>) <span class="hljs-title"><span class="hljs-keyword">public</span></span> <span class="hljs-title"><span class="hljs-keyword">pure</span></span> <span class="hljs-title"><span class="hljs-keyword">returns</span></span> (<span class="hljs-params"><span class="hljs-keyword">bytes4</span></span>) </span>{
        <span class="hljs-keyword">return</span> <span class="hljs-number">0x150b7a02</span>; <span class="hljs-comment">// Allows this reputation contract to own NFTs, which may be useful in certain cases</span>
    }
}
</code></pre>]]></content:encoded>
            <author>tableland-2@newsletter.paragraph.com (Tableland)</author>
        </item>
    </channel>
</rss>