Let’s be realistic, data is ephemeral, mostly when we decide to store data on cloud platforms and other places on the internet. Centralized data providers rely heavily on profit to sustain their operations, covering server expenses and upkeep. This strategy is usually not reliable long term. As I’m writing here on Medium, it might be wise to maintain a local copy of this post as a backup in case things don’t go so well for the Medium side.

The Arweave Web3 protocol is a blockchain-like data structure launched in 2018. It stands as a permanent data storage, and anyone can use it to store data with a single payment.

The protocol relies on miners, who are rewarded with AR tokens for mining new blocks, which requires them to store and serve data. The users pay with AR tokens to upload data, and the network is in charge of channelling the rewards to the miners.

Nowadays Arweave is a flourishing ecosystem, full of projects and decentralized applications.

You can explore different analytics gateways, development tools, email services, NFT platforms, storage solutions and of course different Arweave wallets.

I didn't have the time to explore all of them, but there are 2 main projects that deserve to be shared.

The first time I used Akord, was to upload Data Science reports to Arweave. They are now available as NFTs in the Ocean Market, which is a data marketplace on the Web3. Check out my article to know more about it:

Win Cryptos with Your Data Science Skills

Win Cryptos with Your Data Science Skills Use Ocean Protocol to win cryptocurrencies Have you heard about Ocean Protocol? If you have been surfing on both Data Science and Web3, You've probably ...

https://blog.cubed.run

To use Akord as a gateway to upload data to Arweave is very simple, it works much like a normal cloud provider. You can create an account here and you’ll get 250 MB of free storage. If you need more space, you pay for the amount you want, there’s no need for monthly or annual subscriptions. In addition, Akord accepts payments in USD in order to facilitate access.

The platform is organized in vaults, you can make them public or encrypted.

Inside the vaults, you can start uploading your files. To share them, there are two ways, either through an Akord shared link (take this example). Or through the permaweb link, which fetches the file directly from Arweave.

The permaweb link usually has the following format:

https://arweave.net/<file_tx_id>

This is important for Python integration later in this article.

Mirror is not a traditional NFT marketplace, as we are used to see like OpenSea or Rarible, it is more like a publishing platform, just like Medium, where posts can be converted into non-fungible tokens.

You can connect to the platform, through your Ethereum wallet and start writing entries (articles in Mirror).

The writing experience is outstanding, as good as Medium or even better in some situations. It allows for further customization and you finally get ownership of the articles you write. Users can mint other people’s entries as NFTs, and add them to their collection. To avoid huge gas fees, Mirror uses Optimism layer 2 solution.

Once you mint an entry, it will be available to read as an NFT on OpenSea and some wallets that allow this kind of integration.

Arweave uses GraphQL endpoints for querying transactions/block data. For this, we can use the GraphQL platform.

In this case, we’re using a query to extract the first 10 transactions on the Mirror platform. It outputs the transaction IDs, that can be used to extract further information.

As we’ve seen before, the permalink is composed of https://arweave.net/ and the transaction ID. In the case of Mirror, this takes us to a JSON file, which has all the content of the article. See the example below:

If we want to produce a more robust way of extracting data, we can use Python with GraphQL queries.

Let’s start by using the same query as before, with the tag App-Name and value MirrorXYZ.

import requests
import json
import time

query = """query {
    transactions(first:10,
        tags: {
            name: "App-Name",
            values: ["MirrorXYZ"]
        }
    ) {
        edges {
            node {
                id
            }
        }
    }
}"""

Now do the following to get the IDs and iterate over them:

url = 'https://arweave.net/graphql'
r = requests.post(url, json={'query': query})

if r.status_code == 200:
    # Parse the response JSON into a Python dictionary
    response_data = json.loads(r.text)

    # Access the data you need
    for edge in response_data['data']['transactions']['edges']:
        transaction_id = edge['node']['id']
        url = 'https://arweave.net/' + transaction_id
        response = requests.get(url)
        url_data = response.json()
        values = extract_from_json(url_data)
        print(values)
else:
    print("Error: HTTP Status Code", r.status_code)
    time.sleep(20)
    pass

The function extract_from_json() extracts the desired information from the JSON file, it can take the following shape for instance:

def extract_from_json(data):
    body = data['content']['body']
    timestamp = data['content']['timestamp']
    title = data['content']['title']
    contributor = data['authorship']['contributor']
    return {
        'body': body,
        'timestamp': timestamp,
        'title': title,
        'contributor': contributor,
    }

As for Akord, we can’t use the App-Name tag to look for content, but we can for example grab files that are .txt or .csv.

query = """query {
    transactions(first:10,
        tags: {
            name: "Content-Type",
            values: ["text/csv"]
        }
    ) {
        edges {
            node {
                id
            }
        }
    }
}"""

The output of this query is once again a list of transaction IDs, that can be extracted using the permalink.

At this point, you’re probably wondering, how do you get these tags? They are visible on the ViewBlock website. If we take for instance the transaction ID of my entry, we reach the following page:

On the left side, we see the different tags that compose this transaction, and on the right, we have the JSON file (permalink) of the Mirror entry.

In case you want to try out other GraphQL queries, you can find more here.

Other APIs are available inside the Arweave network, such as this one that lists the different projects that compose the ecosystem.

import requests

res = requests.get(
    'https://api.weavescan.dev/api/content/items/projects')

print(res.json())

The options are endless when it comes to analysing content inside Arweave. Two platforms were briefly described, and some Python scripts with GraphQL queries were introduced to interact with the content. By using these endpoints developers can keep on building solutions to enrich the ecosystem.

This entry is also available on Medium with paywall for non-members.

https://medium.com/coinmonks/extract-data-from-mirror-and-akord-platforms-on-arweave-377a7f7a70fb

👉 Join the Web3 revolution and store your articles on the Permaweb with Mirror.

Let’s be realistic, data is ephemeral, mostly when we decide to store data on cloud platforms and other places on the internet. Centralized data providers rely heavily on profit to sustain their operations, covering server expenses and upkeep. This strategy is usually not reliable long term. As I’m writing here on Medium, it might be wise to maintain a local copy of this post as a backup in case things don’t go so well for the Medium side.

The Arweave Web3 protocol is a blockchain-like data structure launched in 2018. It stands as a permanent data storage, and anyone can use it to store data with a single payment.

The protocol relies on miners, who are rewarded with AR tokens for mining new blocks, which requires them to store and serve data. The users pay with AR tokens to upload data, and the network is in charge of channelling the rewards to the miners.

Nowadays Arweave is a flourishing ecosystem, full of projects and decentralized applications.

You can explore different analytics gateways, development tools, email services, NFT platforms, storage solutions and of course different Arweave wallets.

I didn't have the time to explore all of them, but there are 2 main projects that deserve to be shared.

The first time I used Akord, was to upload Data Science reports to Arweave. They are now available as NFTs in the Ocean Market, which is a data marketplace on the Web3. Check out my article to know more about it:

Win Cryptos with Your Data Science Skills

Win Cryptos with Your Data Science Skills Use Ocean Protocol to win cryptocurrencies Have you heard about Ocean Protocol? If you have been surfing on both Data Science and Web3, You've probably ...

https://blog.cubed.run

To use Akord as a gateway to upload data to Arweave is very simple, it works much like a normal cloud provider. You can create an account here and you’ll get 250 MB of free storage. If you need more space, you pay for the amount you want, there’s no need for monthly or annual subscriptions. In addition, Akord accepts payments in USD in order to facilitate access.

The platform is organized in vaults, you can make them public or encrypted.

Inside the vaults, you can start uploading your files. To share them, there are two ways, either through an Akord shared link (take this example). Or through the permaweb link, which fetches the file directly from Arweave.

The permaweb link usually has the following format:

https://arweave.net/<file_tx_id>

This is important for Python integration later in this article.

Mirror is not a traditional NFT marketplace, as we are used to see like OpenSea or Rarible, it is more like a publishing platform, just like Medium, where posts can be converted into non-fungible tokens.

You can connect to the platform, through your Ethereum wallet and start writing entries (articles in Mirror).

The writing experience is outstanding, as good as Medium or even better in some situations. It allows for further customization and you finally get ownership of the articles you write. Users can mint other people’s entries as NFTs, and add them to their collection. To avoid huge gas fees, Mirror uses Optimism layer 2 solution.

Once you mint an entry, it will be available to read as an NFT on OpenSea and some wallets that allow this kind of integration.

Arweave uses GraphQL endpoints for querying transactions/block data. For this, we can use the GraphQL platform.

In this case, we’re using a query to extract the first 10 transactions on the Mirror platform. It outputs the transaction IDs, that can be used to extract further information.

As we’ve seen before, the permalink is composed of https://arweave.net/ and the transaction ID. In the case of Mirror, this takes us to a JSON file, which has all the content of the article. See the example below:

If we want to produce a more robust way of extracting data, we can use Python with GraphQL queries.

Let’s start by using the same query as before, with the tag App-Name and value MirrorXYZ.

import requests
import json
import time

query = """query {
    transactions(first:10,
        tags: {
            name: "App-Name",
            values: ["MirrorXYZ"]
        }
    ) {
        edges {
            node {
                id
            }
        }
    }
}"""

Now do the following to get the IDs and iterate over them:

url = 'https://arweave.net/graphql'
r = requests.post(url, json={'query': query})

if r.status_code == 200:
    # Parse the response JSON into a Python dictionary
    response_data = json.loads(r.text)

    # Access the data you need
    for edge in response_data['data']['transactions']['edges']:
        transaction_id = edge['node']['id']
        url = 'https://arweave.net/' + transaction_id
        response = requests.get(url)
        url_data = response.json()
        values = extract_from_json(url_data)
        print(values)
else:
    print("Error: HTTP Status Code", r.status_code)
    time.sleep(20)
    pass

The function extract_from_json() extracts the desired information from the JSON file, it can take the following shape for instance:

def extract_from_json(data):
    body = data['content']['body']
    timestamp = data['content']['timestamp']
    title = data['content']['title']
    contributor = data['authorship']['contributor']
    return {
        'body': body,
        'timestamp': timestamp,
        'title': title,
        'contributor': contributor,
    }

As for Akord, we can’t use the App-Name tag to look for content, but we can for example grab files that are .txt or .csv.

query = """query {
    transactions(first:10,
        tags: {
            name: "Content-Type",
            values: ["text/csv"]
        }
    ) {
        edges {
            node {
                id
            }
        }
    }
}"""

The output of this query is once again a list of transaction IDs, that can be extracted using the permalink.

At this point, you’re probably wondering, how do you get these tags? They are visible on the ViewBlock website. If we take for instance the transaction ID of my entry, we reach the following page:

On the left side, we see the different tags that compose this transaction, and on the right, we have the JSON file (permalink) of the Mirror entry.

In case you want to try out other GraphQL queries, you can find more here.

Other APIs are available inside the Arweave network, such as this one that lists the different projects that compose the ecosystem.

import requests

res = requests.get(
    'https://api.weavescan.dev/api/content/items/projects')

print(res.json())

The options are endless when it comes to analysing content inside Arweave. Two platforms were briefly described, and some Python scripts with GraphQL queries were introduced to interact with the content. By using these endpoints developers can keep on building solutions to enrich the ecosystem.

This entry is also available on Medium with paywall for non-members.

https://medium.com/coinmonks/extract-data-from-mirror-and-akord-platforms-on-arweave-377a7f7a70fb

👉 Join the Web3 revolution and store your articles on the Permaweb with Mirror.