By buidling on the shoulders of giants in the Farcaster community, Farchiver can focus on farchiving. Obsidian for visualization, DeForm for signups, Paragraph for blogging, POIDH for security bounties. The FE starter by @farcasteradmin.eth makes sane, cost-effective decisions everywhere. A powerful tool called fario by @vrypan.eth simplifies exploration of Hub data.

S/O wallet services by Dynamic and Rainbow. Never roll your own auth or crypto!

Hub, Replicator, or both

Hub maintains consensus on live state, as well as an auditable, off-chain and purge-able history. A node responds to queries via its own APIs.

Replicator is a set of services, primarily a Node app that listens to Farcaster protocol messages (via Hub) and updates a PostgreSQL instance.

Neither Hub nor Replicator guarantees full synchronization, or a complete snapshot.

The Farcaster community provided useful discussion and advice on the trade-offs, in the /Neynar channel.

How it started:

artlu.eth (🗝️,👑) - l/dau on Warpcast

strong opinion, weakly held: there are always better choices than Replicator - clients do not need complex joins - 2 Hubs handle concurrency as well as 1 Postgres constantly drip-fed by Replicator - tested APIs > rewriting logic in SQL - backfill is too complex: distinct state layers in Hub, bullmq, redis, postgres

How it's going:

artlu.eth (🗝️,👑) - l/dau on Warpcast

strong opinion, weakly held: client devs should not run Hubs (they should run a Replicator) - Hubs are expensive - point Replicator to nemes, StandardCrypto, Neynar - Replicator is copy of all (verified) data, without overhead of CRDT, syncing, network spam - SQL is more flexible+expressive than Hub HTTP API or gRPC

Farchiver currently runs on two Replicators: a stock instance pulled directly from the repository, and a magpie instance that grabs subsets of data from multiple data sources.

Charitably, the "eventually consistent" Hub network is not yet completely in sync. Practically, different Hubs provide slightly different slices of history, and consolidating it all into one data store is a quirky and undocumented process.

At some point in the future, Replicator will work perfectly out of the box. Or perhaps, something else like Teleport (a Hub + SQL database in one) may better serve our particular needs. Farchiver is buidling before everything is settled.

On boostrapping

Farchiver uses Neynar APIs for bootstrapping. They are a tremendous resource. We are also exploring the open-source Farcaster APIs from Wield.

However, none of the API providers provide exactly everything needed to produce a Farchive. This feels fine. For the project to offer something unique and valuable, it has to do more than bundle together a bunch of off-the-shelf services.

Furthermore, generating a Farchive can incur a significant load on the infra and bandwidth. We have timed out/maxxed out on various API provider limits, which of course can be relaxed on our own infra.

Farchiver runs its own Hub + Replicator(s) for long-term sustainability. This is a significant diversion of resources from the core product, but it gives us better visibility to steer around bottlenecks.

CONCLUSION

We have now reached the end of the 4-part series. Please check out farchiver.xyz and follow @farchiver on Farcaster for more!