News Archiver Project for Arweave

andres-pirela@newsletter.paragraph.com (Andres Pirela) — Mon, 21 Feb 2022 17:32:34 GMT

Recently, an initiative started by the Arweave team with a big push on archiving everything related to the current Russia-Ukraine situation. The goal is simple: Store anything that’s related to it and let historians figure it out in the future.

By having this information in the permaweb, we can guarantee it will survive overtime, and censorship to the articles with different perspectives won’t be possible. The most important thing about this philosophy is that we don’t necessarily care about what we’re storing, we only care about providing historians enough information for researches in the future (who knows, maybe dozens of the generations ahead).

With this initiative, I created 'https://github.com/andreespirela/ukraine-russia-permaweb

https://github.com/andreespirela/ukraine-russia-permaweb

A script that was mainly to archive news about the situation, but it’s now possible to store any sort of news as long as you provide the needed keywords.

It works in a very simple way and you only need to follow the steps such as providing the environmental variables and running \`node index.js\`

With this script, you can migrate hundreds of thousands of news articles. As I write this blog, this script has already archived over 10500 articles with their respective PDF snapshot.

PDF Snapshots & Why?

Websites and their components such as CSS, HTML, JS and everything that makes a news website (or any website for instance) work are not meant to be stored in the permawe but if you take a PDF snapshot (Screenshot of the website saved in PDF format) you can replicate how the website looked at the moment it was archived in the permaweb. We believe this is important in order to provide historians with complete context alongside a full picture of the article, not only a title and description.

Checking the Archive Machine

You can check how many articles we have stored by running the following GraphQL query in https://arweave.net/graphql

query {
            transactions(
                tags: [{
                    name: "Initiative",
                    values: ["AndresPirelaUkraineRussia"]
                }],
                first: 100
            ) {
                edges {
                    node {
                        id,
                    },
                cursor
                }
            }
        }

This is also visible in the repository file fetch-all.js which you can run by doing deno run --allow-all fetch-all.js