# Extract Data from Mirror and Akord with GraphQL and Python

**Published by:** [Marco Rodrigues](https://paragraph.com/@macrodrigues/)
**Published on:** 2023-10-09
**URL:** https://paragraph.com/@macrodrigues/extract-data-from-mirror-and-akord-with-graphql-and-python

## Content

Let’s be realistic, data is ephemeral, mostly when we decide to store data on cloud platforms and other places on the internet. Centralized data providers rely heavily on profit to sustain their operations, covering server expenses and upkeep. This strategy is usually not reliable long term. As I’m writing here on Medium, it might be wise to maintain a local copy of this post as a backup in case things don’t go so well for the Medium side. The Arweave Web3 protocol is a blockchain-like data structure launched in 2018. It stands as a permanent data storage, and anyone can use it to store data with a single payment. The protocol relies on miners, who are rewarded with AR tokens for mining new blocks, which requires them to store and serve data. The users pay with AR tokens to upload data, and the network is in charge of channelling the rewards to the miners. 1 — Meet the ecosystem Nowadays Arweave is a flourishing ecosystem, full of projects and decentralized applications. List of projects on Arweave You can explore different analytics gateways, development tools, email services, NFT platforms, storage solutions and of course different Arweave wallets. I didn't have the time to explore all of them, but there are 2 main projects that deserve to be shared. Akord (Storage) The first time I used Akord, was to upload Data Science reports to Arweave. They are now available as NFTs in the Ocean Market, which is a data marketplace on the Web3. Check out my article to know more about it: Win Cryptos with Your Data Science Skills Win Cryptos with Your Data Science Skills Use Ocean Protocol to win cryptocurrencies Have you heard about Ocean Protocol? If you have been surfing on both Data Science and Web3, You've probably ... https://blog.cubed.run To use Akord as a gateway to upload data to Arweave is very simple, it works much like a normal cloud provider. You can create an account here and you’ll get 250 MB of free storage. If you need more space, you pay for the amount you want, there’s no need for monthly or annual subscriptions. In addition, Akord accepts payments in USD in order to facilitate access. Akord payment options The platform is organized in vaults, you can make them public or encrypted. Akord Vaults Inside the vaults, you can start uploading your files. To share them, there are two ways, either through an Akord shared link (take this example). Or through the permaweb link, which fetches the file directly from Arweave. Akord file information The permaweb link usually has the following format: https://arweave.net/<file_tx_id> This is important for Python integration later in this article. Mirror (NFTs) Mirror is not a traditional NFT marketplace, as we are used to see like OpenSea or Rarible, it is more like a publishing platform, just like Medium, where posts can be converted into non-fungible tokens. Mirror’s website You can connect to the platform, through your Ethereum wallet and start writing entries (articles in Mirror). The writing experience is outstanding, as good as Medium or even better in some situations. It allows for further customization and you finally get ownership of the articles you write. Users can mint other people’s entries as NFTs, and add them to their collection. To avoid huge gas fees, Mirror uses Optimism layer 2 solution. Once you mint an entry, it will be available to read as an NFT on OpenSea and some wallets that allow this kind of integration. Minted NFT on OpenSea 2 — Extract data from Arweave Arweave uses GraphQL endpoints for querying transactions/block data. For this, we can use the GraphQL platform. Arweave GraphQL API In this case, we’re using a query to extract the first 10 transactions on the Mirror platform. It outputs the transaction IDs, that can be used to extract further information. As we’ve seen before, the permalink is composed of https://arweave.net/ and the transaction ID. In the case of Mirror, this takes us to a JSON file, which has all the content of the article. See the example below: { "content": { "body": "I started the year 2023 highly focused on finding my next job. Having recently completed a contract as a Software Engineer at the European Spaceport in French Guiana, I wanted to pursue a career more towards data. I was actively looking for jobs related to Data Science, Data Analytics, Data Engineering, Machine Learning Engineer, you name it.\n\nI must say, I never had so many working interviews in my life, there are a lot of job offers in these fields, but they seem to be equally balanced with the demand. I\u2019ve reached the final step of some job processes, but for some reason, I was never the chosen one.\n\nIt is important to say, that I don\u2019t own a master\u2019s degree in Data Science or any other data field. I\u2019ve studied Engineering of Nanotechnology and Microelectronics, and I\u2019ve worked in this field for almost 3 years. However I have a proven record of projects and work related to Data Science, I even won some data challenges.\n\nI felt I was lonely in a rainforest, trying to survive. At that point, I knew I was not the predator. In French Guiana, nature taught me how small humans are, and doesn\u2019t matter how much you\u2019re willing to fight nature, it will somehow, somewhere defeat you. There\u2019s no point in trying to be the alfa in the rainforest, for that, I would need to have a diploma or at least some years of experience in the field, which was not the case.\n\nI kept on building projects, enrolling in challenges mainly building a portfolio and at some point the impossible happened, I started to get noticed in the immense rainforest.\n\n## Showing myself to the world\n\nIf you don\u2019t have a diploma or years of experience in the field. Chances are that nobody will hire you. But dedication and showcasing your work is valued by some companies, and by recruiters in general.\n\nThat\u2019s why owning a portfolio is key to showing your work and being emailed by future clients.\n\nThere are several ways of building portfolios nowadays. Some people use platforms like Notion, others prefer to use Wordpress, there are even AI solutions, that can build a website in 30 seconds like Durable. I took the long path, by making it all by myself, but I only recommend it if you have the time, the skills and a geeky mindset, otherwise, it can be very time-consuming. This is my beloved portfolio:\n\n\u003Chttps://www.macrodrigues.xyz/\u003E\n\nPortfolios are great, but they are not enough. If you\u2019re trying to survive in the rainforest. Having a fishing rod is good, but having a bow is even better. The same applies to our society, so I\u2019ve decided to start writing on Medium. It was a win-win situation because I was feeding my portfolio and at the same time getting noticed in the technical writing field.\n\n\u003Chttps://medium.com/@macrodrigues\u003E\n\nShowcasing yourself is never enough, and the next big move was social networks. The main one in terms of professional path is Linkedin. I had my account for some time already, but I was never active there, only when I was looking for a job. As soon as I started to publish my Medium articles, people started contacting me (recruiters among them) and I increased my followers exponentially.\n\nI\u2019m still trying to navigate the world of social media. As a Data Scientist, AI and Web3 enthusiast, Instagram didn\u2019t work as well as I thought, but depending on your field it might be a great source. I\u2019m now trying my way on Twitter (aka X), which is full of technical enthusiasts.\n\n\u003Chttps://twitter.com/marcoacavaco/status/1707385662536548534\u003E\n\n## Technical sources of income\n\nAfter marketing myself, I finally started to have some gigs. Writing on Medium, was the first jump to start writing on other platforms and consequently being contacted by clients through LinkedIn. See below an example of a paid article.\n\n\u003Chttps://techjury.net/blog/geo-targeting-with-proxies/\u003E\n\nI\u2019ve used all the technical content I\u2019ve written to start promoting myself on Upwork. Many times before I\u2019ve tried my way on these freelancing websites, but I\u2019ve not accomplished much, and clients would only accept me for very low prices. I know my place in the rainforest, I might not be a predator, but I know how much I\u2019m worth. Having a rich portfolio helped me to find clients more prone to pay fair wages, and slowly I started to get more and more working contracts.\n\n\u003Chttps://www.upwork.com/freelancers/~01185447baa36a1b77\u003E\n\nOther good sources to find freelancing work are surprisingly Discord, Whatsapp and Facebook groups.\n\nSome Discord communities have channels focused on job offers. Building a community is also a great way to showcase your work, but mostly when you\u2019re no longer a foreigner in the rainforest.\n\nWhatsApp groups are great when it comes to local communities. Maybe in your region, someone might need a developer to build a simple application, and that\u2019s where you show up.\n\n## Other sources of income\n\nWhen we are stepping in as freelancers, it's essential to shift away from a commodity mindset. Most likely no one will pay for your holidays and you must think about investing more than spending because you don\u2019t have your future secured.\n\nThat\u2019s why some years ago I decided to invest in crypto, at that time I was not working on my own, but now I can use it to have a passive income.\n\nDuring bear markets, there isn\u2019t much you can do besides buying more, trading, farming or staking. The latest option seemed the most effortless, so I jumped into these two solutions:\n\n* Stake ETH: I spent some time looking for the best alternative to stake some of my ETH. I even thought of making a node myself, but I wanted to avoid maintenance and being penalized due to a bad internet connection and so on. I looked into Lido but I found it too centralized, then I discovered StakeWise which was exactly what I was looking for.\n\n* Presearch node: To make a Presearch node I had to rent a server because once again I wanted to avoid having my own. To set up a node you can follow these instructions. The node provides a very good APY, and you receive the rewards in the form of PRE tokens. Besides Presearch is my main search engine along with Brave browser.\n\n## Future quests\n\nThe rainforest is immense, and there are thousands of ways to gain your place. I\u2019m still building my wooden boat, learning how to fish, hunting small mammals, and healing myself with very few plants. Getting to know your surroundings is key and then progressively expand to other areas of the forest.\n\nWorking on your own does not mean being a lone wolf, it just means that you need to deal with very different tribes and survive in all of them. In the future that can be a means of connecting those tribes together and owning an important place in the rainforest. Meanwhile, respect nature, its rhythms, and do good, you\u2019ll survive, just like me.", "timestamp": "1695980307", "title": "Chronicles of a Young Freelancer" }, "digest": "UZ1XM3xNRr9MFH0hK8TOwBTqgeEThXpXFI4H0GpKons", "authorship": { "contributor": "0xc25656Bd364e9DE6AB6dFC8E01e4fe9a7A574624", "signingKey": "{"crv":"P-256","ext":true,"key_ops":["verify"],"kty":"EC","x":"cUY2PCUAs4dSDas8K0VEGmGBz21eL9OUTi-YUvl2izI","y":"Mv7_XupCQugR2TiaWt03kVjQ825mmY6JhRV73-MKwko"}", "signature": "0x5af98af2b4171e87386aa0049f3439c72aac03f7ca3f09ffd299deb944a873991c9f8465130232376e14a7b2fef3b94f7ec85479950286d44dd757df8f2f5be01b", "signingKeySignature": "OMFxFvIvebg8XmZvOuGj0i4k9chBeLTh59rtnFjcMst-no6dadz-cZ-YUNGe1TGlDuLvsE1oh-KAYkw3GgAwCQ", "signingKeyMessage": "I authorize publishing on mirror.xyz from this device using:\n{"crv":"P-256","ext":true,"key_ops":["verify"],"kty":"EC","x":"cUY2PCUAs4dSDas8K0VEGmGBz21eL9OUTi-YUvl2izI","y":"Mv7_XupCQugR2TiaWt03kVjQ825mmY6JhRV73-MKwko"}", "algorithm": { "name": "ECDSA", "hash": "SHA-256" } }, "version": "04-25-2022" } If we want to produce a more robust way of extracting data, we can use Python with GraphQL queries. Let’s start by using the same query as before, with the tag App-Name and value MirrorXYZ. import requests import json import time query = """query { transactions(first:10, tags: { name: "App-Name", values: ["MirrorXYZ"] } ) { edges { node { id } } } }""" Now do the following to get the IDs and iterate over them: url = 'https://arweave.net/graphql' r = requests.post(url, json={'query': query}) if r.status_code == 200: # Parse the response JSON into a Python dictionary response_data = json.loads(r.text) # Access the data you need for edge in response_data['data']['transactions']['edges']: transaction_id = edge['node']['id'] url = 'https://arweave.net/' + transaction_id response = requests.get(url) url_data = response.json() values = extract_from_json(url_data) print(values) else: print("Error: HTTP Status Code", r.status_code) time.sleep(20) pass The function extract_from_json() extracts the desired information from the JSON file, it can take the following shape for instance: def extract_from_json(data): body = data['content']['body'] timestamp = data['content']['timestamp'] title = data['content']['title'] contributor = data['authorship']['contributor'] return { 'body': body, 'timestamp': timestamp, 'title': title, 'contributor': contributor, } As for Akord, we can’t use the App-Name tag to look for content, but we can for example grab files that are .txt or .csv. query = """query { transactions(first:10, tags: { name: "Content-Type", values: ["text/csv"] } ) { edges { node { id } } } }""" The output of this query is once again a list of transaction IDs, that can be extracted using the permalink. At this point, you’re probably wondering, how do you get these tags? They are visible on the ViewBlock website. If we take for instance the transaction ID of my entry, we reach the following page: ViewBlock transaction information On the left side, we see the different tags that compose this transaction, and on the right, we have the JSON file (permalink) of the Mirror entry. In case you want to try out other GraphQL queries, you can find more here. Other APIs are available inside the Arweave network, such as this one that lists the different projects that compose the ecosystem. import requests res = requests.get( 'https://api.weavescan.dev/api/content/items/projects') print(res.json()) Conclusion The options are endless when it comes to analysing content inside Arweave. Two platforms were briefly described, and some Python scripts with GraphQL queries were introduced to interact with the content. By using these endpoints developers can keep on building solutions to enrich the ecosystem. Subscribe This entry is also available on Medium with paywall for non-members. https://medium.com/coinmonks/extract-data-from-mirror-and-akord-platforms-on-arweave-377a7f7a70fb 👉 Join the Web3 revolution and store your articles on the Permaweb with Mirror.

## Publication Information

- [Marco Rodrigues](https://paragraph.com/@macrodrigues/): Publication homepage
- [All Posts](https://paragraph.com/@macrodrigues/): More posts from this publication
- [RSS Feed](https://api.paragraph.com/blogs/rss/@macrodrigues): Subscribe to updates
- [Twitter](https://twitter.com/marcoacavaco): Follow on Twitter