Farcaster

Trish🫧 on Warpcast

I’m one of the 5-15 and this stresses me out. People might not care now, but nobody knows how valuable a fully archived Farcaster will be. Undoubtedly very valuable and possibly most to people who aren’t thinking of it. I’m pretty much in this space for watching history unfold. https://warpcast.com/dwr.eth/0xe06c1d17

Farcaster

Nounish Prof ⌐◧-◧🎩💩 on Warpcast

As of now, you save everything w/ @farchiver — maybe @artlu can elaborate?

TL;DR

• casts are kept live in Hubs

• casts are stored in databases

• casts are exportable as JSONs

• threads are savable as Farchives

• Farchiver hosts public and semi-private Farchives

Casts are kept live in Hubs

The Farcaster protocol achieves eventually-consistent consensus on live state. Each FID has a storage allocation, and activities that go over the allocation lead to pruning of older activities. More details elsewhere.

Farcaster

Dan Romero on Warpcast

A few of things: 1. You can keep paying for storage and as long as someone is running a valid Hub your content will be preserved. You could even make a smart contract that drips out payments for 100 years. 2. Anyone can pay for someone else's storage ($3 in OP ETH) 3. Back up off protocol on Arweave or onchain https://warpcast.com/dwr.eth/0xa7eb1dbb

Casts are stored in databases

A handful of individuals maintain public databases of cast history. Some will prune, some will not.

The best known example is searchcaster.xyz by @greg. Somebody else (pixel?) received early Purple grants to make their database snapshots public.

Neynar pushes their private custom Replicator dataset to Dune, where it is available to the public via Dune's gating. Airstack provides access to Farcaster data via their own private APIs. Etc.

Casts are exportable as JSONs

It is fast and low-cost (often free) to get your own cast history on your local machine! It is entirely yours, and you can do anything with it, including back it up.

Check out fcau.xyz by @iammathias for an MIT-licensed FOSS Node CLI utility.

Check out fario by @vrypan for a powerful CLI tool in python, also MIT-licensed.

Check out @pixel's guide. Neynar is not free, but the one-month cost is less than one year of Farchiver.

All of these are really fast. The actual process to download full history takes less than ~2 minutes for most users. The setup may be intimidating to some, but is not super difficult for experienced devs.

To find free, open Hub endpoints at which to point these utilities, check out the list we maintain at foss.farchiver.xyz.

Threads are savable as Farchives

Farchiver lets you download every thread you've ever touched, with all interactions, in a delightful package. We suggest Obsidian for browsing; it can be any Markdown-aware knowledge tool.

We believe this is more interesting than just one's own casts. We charge 0.03 ETH for one year of this unique service.

You may purchase a snapshot of any FID.

Farchiver hosts public Farchives

In the public interest, Farchiver hosts Farchives, pinned on IPFS and available for the community to preserve collectively.

Examples:

• the proto channels thread, NBA game 5

• an epic @keccers.eth thread, requested by @shazow.eth

• a month of @dwr.eth interactions, around the wowow meme

Farchiver also enables private, sharable Farchives of threads of interest. Each user has access to a section of private Farchives (possibly encrypted) behind a wallet login. This includes Farchives that others have paid for, to share with others.

As an example, one caster wanted to own the toxic interaction they had with a powerful account. It had been deleted, and only pieces of it had been screen-shotted without the full context. Now this user has a copy of the deleted abusive casts, and this Farchive can be shared with others as a semi-private, but transparent and permissionless record.

TL;DR

• Community.

• Hub, Replicator, or both.

• On bootstrapping.

Community

Farcaster and Hubble are FOSS. As mentioned in Part 1 of this series, the team provided the major unlock with "100% permissionless" in early 2023.

By buidling on the shoulders of giants in the Farcaster community, Farchiver can focus on farchiving. Obsidian for visualization, DeForm for signups, Paragraph for blogging, POIDH for security bounties. The FE starter by @farcasteradmin.eth makes sane, cost-effective decisions everywhere. A powerful tool called fario by @vrypan.eth simplifies exploration of Hub data.

S/O wallet services by Dynamic and Rainbow. Never roll your own auth or crypto!

Hub, Replicator, or both

Hub maintains consensus on live state, as well as an auditable, off-chain and purge-able history. A node responds to queries via its own APIs.

Replicator is a set of services, primarily a Node app that listens to Farcaster protocol messages (via Hub) and updates a PostgreSQL instance.

Neither Hub nor Replicator guarantees full synchronization, or a complete snapshot.

The Farcaster community provided useful discussion and advice on the trade-offs, in the /Neynar channel.

How it started:

artlu.eth (🗝️,👑) - l/dau on Warpcast

strong opinion, weakly held: there are always better choices than Replicator - clients do not need complex joins - 2 Hubs handle concurrency as well as 1 Postgres constantly drip-fed by Replicator - tested APIs > rewriting logic in SQL - backfill is too complex: distinct state layers in Hub, bullmq, redis, postgres

How it's going:

artlu.eth (🗝️,👑) - l/dau on Warpcast

strong opinion, weakly held: client devs should not run Hubs (they should run a Replicator) - Hubs are expensive - point Replicator to nemes, StandardCrypto, Neynar - Replicator is copy of all (verified) data, without overhead of CRDT, syncing, network spam - SQL is more flexible+expressive than Hub HTTP API or gRPC

Farchiver currently runs on two Replicators: a stock instance pulled directly from the repository, and a magpie instance that grabs subsets of data from multiple data sources.

Charitably, the "eventually consistent" Hub network is not yet completely in sync. Practically, different Hubs provide slightly different slices of history, and consolidating it all into one data store is a quirky and undocumented process.

At some point in the future, Replicator will work perfectly out of the box. Or perhaps, something else like Teleport (a Hub + SQL database in one) may better serve our particular needs. Farchiver is buidling before everything is settled.

On boostrapping

Farchiver uses Neynar APIs for bootstrapping. They are a tremendous resource. We are also exploring the open-source Farcaster APIs from Wield.

However, none of the API providers provide exactly everything needed to produce a Farchive. This feels fine. For the project to offer something unique and valuable, it has to do more than bundle together a bunch of off-the-shelf services.

Furthermore, generating a Farchive can incur a significant load on the infra and bandwidth. We have timed out/maxxed out on various API provider limits, which of course can be relaxed on our own infra.

Farchiver runs its own Hub + Replicator(s) for long-term sustainability. This is a significant diversion of resources from the core product, but it gives us better visibility to steer around bottlenecks.

CONCLUSION

We have now reached the end of the 4-part series. Please check out farchiver.xyz and follow @farchiver on Farcaster for more!

• No secrets.

• Serverless and Stateless.

• Security bounties.

No secrets

POV December 2023: if Ledger Connect can be supply-chain attacked, if X can get link-jacked, then a smol project like Farchiver should never manage secrets.

The contents of any Farchive are 100% public. Farcaster shares everything in real-time, across the decentralized and permissionless Hub network.

Meanwhile, a Farchive collects all your social activities into a neat package. The act of extracting one individual's activities, out of the swirling sea of global messages, can feel intrusive.

For most users, it is better to keep the files unencrypted. For those who are extremely concerned about privacy, we offer symmetric encryption of each Farchive. We will throw away the key right after encryption, making the contents of the Farchive unrecoverable if the key becomes lost.

Serverless and Stateless

The Farchiver website offers a private section, behind signed wallet authentication provided by Dynamic. Connection + signature takes two clicks, in one flow.

Each Farchive is a gzipped tarball with a random name. Cloudflare R2 protects these assets from scraper bots, providing access by exact filename only. Serverless Cloudflare Workers verify Dynamic-signed JWTs, and Cloudflare KV holds meta-data mappings at the edge.

Even if a malicious actor is able to present properly-structured spoofed credentials after reverse-engineering the flow, they would still need to break whitelisted communication channels, the JWT signature chain, and/or the adaptive Cloudflare network, in order to access unauthorized data.

Farchiver does not need to secure a large set of private secrets, only a handful of API keys across multiple trustworthy providers.

Security Bounties

One view on Farcaster: startups paying for smart contract audits before PMF ngmi

briang is live on unlonely on Warpcast

I’m sorry but if you’re paying $200k for a smart contract engineer you’re ngmi any solid dev should be able to write solidity with the help of chatgpt “no, the smart contract needs to be perfect and be audited” bro hit $100k in daily volume then worry about it being perfect

Our problem is different. Farchiver is never gonna hit $100k in daily volume. Nobody with skillz would waste their time on our tiny-stakes project. Yet security is of paramount importance to end users.

So we launched a 0.75 ETH bounty on POIDH, payable to anybody who could retrieve private filenames hidden with Vitalik's public ENS. To our vast surprise, a 21-year-old anon weightlifter in a small village in Germany hacked the site in ~45 minutes.

The smart contract paid 0.75 ETH [proof]. We learned many valuable lessons. Individual pieces may be highly secure, but a project combining battle-tested, open-source technologies can still have holes.

0xChew on Warpcast

The biggest lesson to take away is no matter how widely adopted and trusted contracts are, if they are secure independently, they still may not be secure when used together, no matter how trivial they may seem.

We immediately hardened endpoints and rotated secrets. The hacker is actually a member of a whitehat collective and worked with us responsibly. Very happy to recommend privately.

We implemented three features/migrations which were already on the roadmap, but got pulled forward from nice-to-haves to P0:

• random filenames + (optional) encryption + obfuscation

• migrated from DigitalOcean to Cloudflare

• switched from wagmi + WalletConnect to Dynamic (still wrapping WalletConnect)

Follow @farchiver on Forecaster for new security bounties in January 2024 👀 .

Up Next: How Farchiver

In contrast, Farchiver is building in public as a sustainable business. Hopefully, the choices we've made create a stronger foundation for the business, which provides more upfront confidence in the product.

TL;DR

• pricing

• costs

• roadmap

Pricing

KISS. Buy once, own the Farchive forever.

Base price is 0.005 ETH per quarter. Free samples + discounts for early-users.

Compare with Farcaster storage units, which cost ~0.0032 ETH on-chain, or ~0.0055 ETH via Apple/Google, and expire after one year. Most early Farcaster accounts have two storage units.

Costs

Farchiver primarily uses protocol-level primitives, rather than convenience API services. This is much slower, but also more robust and less dependent on business decisions by others.

Hub (~$50/month incl. compute + RPC access), Replicator (~$20/month). Neynar ($9/month as bootstrap). Domain name + hosting (~$1/month). Farcaster account ($1/month). These costs will fluctuate, probably increasing as usage + network value grows.

Dev and ops time + labor. Mostly front-loaded, with ongoing costs expected to whittle down over time, replaced with automation.

Roadmap

Public Goods: important threads saved as Public Farchives.

Farchives as building block for other Farcaster apps: useful for permanently saving content (with credible neutrality + promise of immortality). Displaying comments, testimonials, and providing permalinks that point to interactive spaces for perusal.

Farchives as building block for personal (self-owned) platforms: React components for publishing to a blog platform; UI-agnostic components that rely on Farchive data + metadata.

Farchiver as testbed for new crypto primitives: e.g., Quilibrium MPC for encryption, Mod Protocol for credible storage, Fabric Hypersub for subscriptions. Will be guided by customer interest.

Up Next: Farchiver Security

Nicholas Charriere on Warpcast

App request: farchiver Casts only last for 1 year by default in hubs. This is good generally, but as a user I probably would like to curate some of my (and others?) casts and mark them as persistent and backed up. Open-minded about the interface (could be a built-in button in a client, a bot, a dedicated app).

TL;DR

• the spark

• pain points

• opportunity for delight

The spark

POV early 2023: new Warpcast user looking for ideas, still trying to grow my previous project from its MVP stage. Way too early for me to think about cast purging. Wasn't ready yet to care about future coffee spit takes. Couldn't foresee anon frenships taking over my f/Shower Thoughts.

@pushix floated his thought bubble [thread], and got a ton of engagement. (Unfortunately, most of the likes disappeared around 3 months later, ~July 2023.)

Then @dwr.eth said the magic word: permissionless.

Dan Romero on Warpcast

Important: now with Hubs, 100% permissionless to do.

Pain points

Plenty of us are completionists, hoarders, compulsive archivers and Zettelkästen crafters. No need to justify this service to you! if the idea of having your casts deleted causes an inner twinge, you already know what you have to do.

There's obvious benefit to having a record of the public discourse. A way to celebrate people who do what they say will do. To admire and partake in personal growth. To provide a perspective on how challenges shift over time. Also, a way for the Third Estate to speak truth to power.

On the world wide web, someone else preserves public information (e.g., the Internet Archive). In web3, an immutable public blockchain means everyone gets the history for free.

Alas, Farcaster social activities are not on-chain (by design!). Only the sufficient pieces of identity are. Hubs de-centralize the data + metadata, as light consensus clients for global live state. Activity does not get stored on a slow, expensive blockchain.

Arweave immortalizes this blogpost. But every Warpcast link inside the post will break within a year, by protocol design.

Opportunity for Delight

@kepano shared some cool visualizations of his Obsidian knowledge base. It became obvious that a living view of Farcaster interactions would be amazing to play with! Even better that it can be beautiful, self-sovereign, offline-first, and future-proof/readable forever.

@valcoholics proved how mesmerizing Obsidian could make one's Farchiver interactions!

V on Warpcast

my farcaster history visualized with Obsidian (cc: @artlu.eth) https://i.imgur.com/nQDnJ80.gif

Check out this lost [thread] about this from over a year ago, courtesy of Searchcaster. Try to find it in Warpcast now, you can't!

If you want to own your own Farchive forever, we have multiple plans to suit your needs!

Up Next: Farchiver as a Business