Intro

In the first installment of this Compute Infrastructure series, I wrote about an AI company called Together that is building a decentralized cloud to fuel open-source AI development. I recommend checking it out if you are interested in a primer on the parallel frontiers of decentralized compute and open-source AI. In conducting this research I discovered that Together is partnering with a compute infrastructure company called Crusoe Energy to power its cloud. This article is focused on how Crusoe is building a new stack for compute to power energy-intensive applications like bitcoin mining and AI, and how this infrastructure accelerates clean energy abundance for a more prosperous future.

The Crusoe Approach

When you drill for oil, you create natural gas as a byproduct. Oil producers have the option to compress and transport the gas somewhere else to be sold, but the problem here is that it usually costs more in transport and storage than the producer would earn from the sale. Since this option would result in a net loss, producers flare the gas, which means they light it on fire. This process wastes energy and creates significant greenhouse gas emissions (GHGs). The worst impact to the environment is from the methane that escapes uncombusted in the course of the flaring process. Methane is 80X more potent than carbon dioxide so even in small quantities it contributes significantly to GHGs. In meeting the world’s demand for oil production, we currently flare 14 billion cubic feet of gas that is burned as waste per day.

Crusoe’s Digital Flare Mitigation (DFM) technology deploys on-site infrastructure to convert this natural gas into energy for computing applications. Being able to convert and use the gas on site eliminates the transaction costs that would otherwise make converting natural gas into power unfeasible. This process eliminates 99% of the methane that would have otherwise been released into the environment through flaring and also reduces emissions by capturing demand for compute that would otherwise tax the main grid. Pretty cool.

Crusoe installs and operates DFM modules on oil fields and taps into gas lines on site to fuel generators that produce electricity to power data centers. These data centers power energy-intensive computing systems that live in as cloud-based applications or internet protocols. It currently has data centers across the US with plans for international expansion. The global nature of operations will be critical to address the flaring problem on a greater scale, since 90% of the world’s flaring occurs outside of the US.

The other component of Crusoe’s approach is focused on accelerating the transition to renewables. This means converting stranded energy from carbon-free resources — like solar and wind — using its Digital Renewable Optimization (DRO) process. Renewable energy production currently has its own inefficiencies. Although you don’t have to deal with natural gas waste that is being flared and contributing significantly to greenhouse gas emissions, there is still a significant excess of power because there is relatively little demand near the sites where the renewable power is being produced. If you’ve ever driven by wind turbines on a road trip you’ll understand why — wind farms are generally located in remote and less inhabited, less industrial areas. And like all sources of energy — which inherently seeks to dissipate and reform —  it is difficult to catch and keep, i.e. transport and store. While there is plenty of renewable energy to be had in this vortex of water, wind and sun that we call home, it’s grabbing it that’s the challenge. Fossil fuels, by contrast, provide us with a source of energy that is already stored and much easier for us to release on demand, with the catch of course being the environmental hit we take in the process of extracting its limited supply.

 “Moving power is hard, storing power is even harder.”

- Chase Lochmiller, CEO & Co-Founder, Crusoe Energy

Crusoe is negotiating between the realities of our legacy infrastructure and the growing demand for compute to transition towards a future where clean energy has replaced fossil fuels. Building a new energy system takes time. While we build the clean energy system of the future, we also need to clean up our existing energy system to buy time along the way. We buy time to implement the renewable future by eliminating waste and generating more energy from the infrastructure that is already in place. Also, by the work of people who are committed to doing hard things.

“I’m attracted by difficult. Difficult is a fucking magnet for me. I go straight to difficult. And I think it probably goes back to this idea that there are lots of smart, really gifted, really talented filmmakers out there that just can’t do the difficult stuff. So that gives me a tactical edge to do something nobody else has ever seen, because the really gifted people don’t fucking want to do it.”

- James Cameron, Director (+ Renaissance Man Extraordinaire)

Bitcoin Mining

Crusoe began its operations by deploying modular data centers filled with Bitcoin mining rigs directly onto oil production sites, where it took possession of the otherwise wasted natural gas (via compensation for oil producers) and used DFM to convert it into electricity for mining Bitcoin.  The barrier to setting up bitcoin mining solutions is relatively permission-less to begin. You only need a processing unit (for Bitcoin mining, the most commonly used is a specialized hardware circuit called an ‘ASIC’), an internet connection and some cooling infrastructure to convert your energy source into bitcoin. Whether or not the setup can generate profit depends on your energy costs and scale, but the ability to set up and contribute to securing the bitcoin network by computing hashes is available to almost everyone. Obviously the lower the cost of energy, the greater the potential for profit. 

Crusoe set-up shipping containers filled with bitcoin mining rigs onto oil fields to use its DFM technology to convert the natural gas into power for mining bitcoin. The company discovered that there was more operational complexity than anticipated in maintaining containers spread out over a remote landscape. To help address these challenges and scale its operation, Crusoe acquired a competitor called Great American Mining in October of 2022. This acquisition provided the company with 4,000 ASIC miners, new facilities, experienced employees and established relationships with large-scale energy producers across multiple states. At this point it could integrate its technology across GAM’s facilities, which increased the scale of its operation. 

Crusoe also looked to vertical integration to help scale its mining business. For example, the company acquired a supplier that it was using to design its modular data centers for mining on site. This acquisition not only helped the company to reduce its manufacturing transaction costs but also gave it a platform to advance the electrical and data center infrastructure that it would need to build out its advanced cloud computing stack. 

“Bitcoin mining made it possible.”

- Chase Lochmiller, CEO, Crusoe

Bitcoin provides an opportunity to turn the abundance of low cost power into financial return. Crusoe was creative and ambitious in how it approached this opportunity, including by using different project financing mechanisms to advance growth without diluting equity. These efforts in turn created the foundation to enter the AI compute marketplace with the infrastructure and scale needed to provide a competitively-priced clean cloud solution. Following the company’s trajectory is yet another example of how the destinies of cryptocurrency and AI are intertwined through compute. 

From Bitcoin to AI

AI models run on accelerator cards otherwise known as GPUs. These cards use a large number of specialized cores (i.e, an A100 GPU has 512 “tensor cores) that perform a certain matrix multiplication (i.e., 4x4) per single cycle. Training large models must distribute the model parameters across multiple GPUs. This process involves significant coordination across multiple nodes that need to frequently exchange information to work together. In centralized data center environments, multiple devices or instances share computing tasks via a high-speed network. To conduct the same training in a distributed environment is a much harder problem to solve, but also reduces costs and democratizes access to building AI systems.

AI is compute-bound; compute access points and cost barriers are infrastructure-level concerns that influence the AI industry on the ground floor. Most AI startups will use a cloud provider (i.e., AWS) as a sort of middleman to access compute. It is more practical and cost-effective for most companies to contract for compute through the cloud rather than drawing that power from the grid themselves. In certain instances it might make sense for AI companies operating at a large scale to build their own data centers. But usually, this aspect of the AI stack is handled by cloud service providers, big and small.

In tandem to scaling its bitcoin mining business, Crusoe has spent the past few years building the cloud infrastructure that is required for high-performance AI computing workloads. This means infrastructure that connects state of the art hardware with the latest in networking infrastructure to deliver power for the most energy-intensive AI tasks. On a practical level this translates to supercomputers equipped with the latest NVIDIA GPUs along with cooling, networking and storage capabilities that provide the performance and scale required to train large (i.e., language and generative) AI models.

Downstream of chip design, the company is vertically integrating its own computing stack. Vertical integration here refers to working at the layer of the energy system first. Traditionally, cloud providers are working one step above the energy layer and one step below the product or application layer. They draw the electricity for their data centers from the carbon-intensive grid, which is expensive and generates new GHG emissions as do all energy intensive-applications powered by the grid. To disrupt this stack requires creating an energetic alternative to the grid. Crusoe does this by repurposing wasted energy and using it to power a virtual cloud-like AI platform.

Energy Abundance in Transition

Given the elegance of converting wasted energy into low-cost power, one of the most interesting questions to ask here is why hasn’t anyone done this before? To begin with, the energy sources that Crusoe is working with are most often produced in harsh conditions and remote locations that don’t support a workforce. Even if they did, the amount of energy that is wasted by a single site may itself not be enough to power an entire industry that would otherwise geolocate to take advantage of that power. The dynamics of this have altogether made it an unattractive value proposition for legacy industrial applications. It is, however, attractive for technological applications that do not need to be physically located by the source of energy. While it is difficult to move power, it is much easier to move data. This establishes a digital pipeline that can extract value from remote locations for globally situated computing workloads. 

Modern computing demands more energy but can also be powered more efficiently across space. All of a sudden the wasted energy that would have cost too much to turn into usable power can be absorbed via on-site data centers to power global, virtually-located applications. By understanding this new demand for virtual energy we begin to see how energy systems transform with technology.  

To live well, energy consumption is non-negotiable. The U.S. Department of Energy concludes: 

“The nation’s standard of living depends in part on energy consumption. Access to adequate energy is now and will continue to be required to achieve a high quality of life.”

- Nuclear Energy Research & Development Roadmap – Report to Congress, April 2010

 We need energy. For everything. To thrive, to learn, to innovate, to build, to dream, to create. The environmental impact of energy consumption cannot be solved by reducing the demand for energy as the fuel of life. In fact, you might even say that reducing the demand for energy is antithetical to life, and oppressive in its deprivation of life when sought to be instituted from above. 

We believe energy should be in an upward spiral. Energy is the foundational engine of our civilization. The more energy we have, the more people we can have, and the better everyone’s lives can be. We should raise everyone to the energy consumption level we have, then increase our energy 1,000x, then raise everyone else’s energy 1,000x as well. 

- Marc Andreesen, Techno-Optimist Manifesto

Technology makes access to an abundant supply of clean energy an economic reality. The demand for compute combined with flexibility in how it can be delivered incentivizes finding low cost points of conversion — such as resources that would otherwise be wasted and renewable sources of energy that are abundant in supply. 

As human beings, we have a universal energetic destiny irrespective of the state or status of our birth.  Solutions like Crusoe that operate on the premise of “always more, but constantly better” are not only efficient but also elegant in how they respect the equal and abundant energetic claim of every person. It is nearly impossible to have had an intimate experience of the developing world without coming to understand the inviolable current of demand for more energy to support life. Instead of looking at climate change as a matter of reducing energy consumption, we should approach it as a matter of harnessing more energy in harmony with the earth.

“Without energy, there is poverty and death.”

-  Cully Cavness, President & Co-Founder, Crusoe Energy

The fact that humans need energy is non-negotiable and our approach to mitigating environmental impact should begin with that constraint. The solution cannot be reduced to reducing greenhouse gas emissions because that is just one part of the story. Rather, how can that be accomplished while continuing to harness more energy to improve quality of life for everyone, today? That’s the game.

I’ll leave you with something I think about a lot. If you were to analogize the history of the Earth to a 26-mile marathon, just the first stride would land you 150,000 years before all of recorded human civilization. This means there could be an untold number of civilizations that have formed and been reduced to dust in the history of the world, and our era is no different. Whether it began 10 or 10,000 years ago (in the grand scheme of things, it’s all the same) we do not belong to the future of our geological epoch any more than we belong to each other, today. So let’s start there.

—————————————————————————————————————————

Indivisible is a media outlet focused on sovereignty and culture at the AI - crypto frontier. It is home to the Indivisible podcast and newsletter.

Subscribe on Substack

Twitter | Farcaster | Apple | Spotify

#npub15a570wxs8acaa8p6efh7fyuf40ynplzc7ufvlfk39nq285h46a9slced8v

—————————————————————————————————————————

Intro

I recently published a podcast episode called “Decentralized Compute for Machine Learning,” which focused on a decentralized protocol for machine learning called  Gensyn. 

https://open.spotify.com/episode/2G7YGHiE0JRq0T2vgdWBdo

This episode marked a first look into the developing market for decentralized AI. In this article, I want to build on that work by introducing a platform called Together alongside Gensyn, which is using a different strategy and technological approach to work towards similar goals. This article aims to provide an overview of both initiatives and how they compare in the overall landscape of decentralizing access to the resources needed to build the world’s most capable ML.

—————————————————————————————————————————

Gensyn recently completed a $43M Series A funding round led by a16z and Together recently announced a $20M seed funding round led by Lux Capital. These funding rounds confirm that venture is attracted to the thesis of decentralized AI and betting on it from a commercial perspective in alignment with its obvious benefits for sovereignty and free society. Things are moving fast, so let’s dive in!

—————————————————————————————————————————

The Closed API Paradigm

Today’s most powerful AI models are either completely closed or gated behind commercial APIs that grant limited access to select partners on preferred terms. API gatekeeping can happen in several ways. First, it usually locks in certain aspects of the model — which can’t be changed, for example on the basis of “safety” or some other political decision. The company can also restrict access up to a certain level of compute and/ or elect to grant access to select applicants for specific and usually non-commercial use.

This is the paradigm of centralized AI development that companies like OpenAI and Google are operating under today and both Gensyn and Together aim to disrupt.

If our current economic model has led to the development of centralized AI systems, then it means we will need to create a new financial incentive structure that supports AI systems that are owned by no one and accessible to everyone, instead.

—————————————————————————————————————————

The Together Platform

Together has created a decentralized cloud platform to train and publish large open-source models. It began by using aggregating resources from five large compute providers— Stanford, ETH Zurich, Open Science Grid, University of Wisconsin-Madison, and CrusoeCloud — into a decentralized cloud. It then reproduced 1.2 trillion tokens that were used to train Meta’s LLaMA to create a new base dataset and a group of base and instruction tuned models that are collectively referred to as The RedPajama Project. LLaMa is Meta’s foundational large language model that the company released earlier this year as open-source.

Together’s approach here was rooted in a research paper called Chinchilla suggesting that if you spend more time (and tokens) training a smaller model (with less parameters) you can get the same quality as one that is much larger.

RedPajama aims not only to produce state-of-the-art open models, but also to make the whole process radically transparent by including its creation recipe and training process along with the base dataset. The goal of these efforts is to accelerate innovation around LLMs, in part by creating a high-performing model that can be run on a device as small as a home computer or even a phone (like the RedPajama base model can).

A key characteristic of open-source AI is allowing developers to own their model weights. This is in contrast to the closed source big tech models, which are built around protecting the weights. This design for protection means that providers host their own models and provide access on an API basis.

API access for model training in turn raises privacy issues, which makes privacy difficult for parties downstream who are hoping to use their own data to further train models. These privacy concerns pose prohibitive limitations for many companies — which drives a greater commercial interest in advancing industry-leading open-source to allow these companies to build in-house ML systems in-house that steer clear of data privacy and related compliance concerns.

”AI is going to be a compute disaggregation event.”

Vipul Ved Prakash, Co-founder and CEO of Together

—————————————————————————

This might seem counterintuitive, but as you dive further into the momentum of decentralized machine learning, it starts to make sense. It is a powerful statement that invites us to think about AI more in terms of opportunity than risk and situates the present reality more as a reflection of the past than an indication of the future. On a technical level, this statement refers to how AI workloads are stateless, which means they do not need anchors like databases and file systems but rather, use a form of compressed data that can migrate to the compute.

RedPajama has been trained on repurposed compute from several sources (including GPU crypto mining farms) but they are currently all large data centers. In the future, Together hopes to use crypto tokens to incentivize a wider set of compute providers, like you and I with our devices, at home. If you listened to the podcast, this should sound familiar… because it would mirror the crypto-native compute-driven approach that Gensyn is leading with today.

—————————————————————————————————————————

The Gensyn Protocol

While Together is focusing on techniques to make high-performing open-source models that are more transparent and accessible, Gensyn is (arguably) disrupting the current AI paradigm further by creating the infrastructure for anyone to access large-scale compute and train their own models. Eventually this training will take place on top of a Gensyn-native global foundation model that many of the network’s participants have contributed to training.

“Everyone, train this foundation model. Design it together, we train it together on an infrastructure that nobody owns and at the end of the day we have a model we can all use that is global and hasn’t been biased by a specific company’s cache of data.”

Ben Fielding, CEO of Gensyn

—————————————

At this point, anyone can then find the hash of this model on-chain and continue training at a specific point on their dataset for their task. Gensyn’s approach centers on the question and challenges of decentralized verification of ML work. The core tension of blockchains (interwoven with what makes them so valuable) is that they are slow and expensive for multiple nodes in a network to run every instruction or every computation to ensure that transactions are accurate. This is also what makes the technology incredibly secure, and that security (when effectively implemented) is what allows them to operate as permission-less, decentralized, censorship-resistant…etc.

As a crypto protocol, a core challenge for Gensyn is to scale decentralized verification. This involves ensuring that both 1) a diversity of hardware devices will be compatible with providing compute to the network and 2) off-chain ML computations can be proven on-chain.

“The fundamental challenge in building this network is the verification of completed ML work. This is a highly complex problem that sits at the intersection of complexity theory, game theory, cryptography, and optimization.” 

Gensyn Litepaper

————————

Gensyn begins with securing a global network of compute that is capable of supplying power for the most computationally intensive machine learning models. One of the protocol’s core objectives is to be compatible with all of the world’s hardware devices that wish to contribute computing power to the network. This includes custom ASICs, consumer GPUs and even computer chips like Apple’s M1 and M2.

Once this size has been secured, the protocol ensures that computation can be verified without the need for any third party by using crypto to achieve consensus. It is on these two foundational components of computational liberty that the magic begins.

Its current approach uses several concepts to delegate verification tasks amongst different types of participants — Submitters, Solvers, Verifiers and Whistleblowers. Submitters are the users that provide tasks for completion and pay for units of completed work. The Solver does the work, the Verifier shows whether it’s correct and then the Whistleblower can confirm whether it’s correct. The Verifier will at times show an error to the Whistleblower on purpose just to keep them on their toes— this is all off-chain work completed by off-chain participants who are rewarded when this work is checked and pushed to the chain. The idea is that rewards are distributed to incentive financial rationality over the entire system.

There is a lot of development that is currently happening to advance zero-knowledge techniques across crypto. This is a cryptographic solution that works by generating smaller proofs to submit to the chain that we can rely on to represent the entire, compute-heavy transaction off-chain. ZK solutions also foster a critical measure of privacy, by being able to reveal that the transaction is valid without revealing the specific data that has been used for calculation. 

As ZK techniques advance, Gensyn will be able to continue optimizing its approach to verification. For example, one of Gensyn’s co-founders postulates that they might be able to eliminate the Whistleblower role as ZK technology advances in its ability to securely represent computational work performed off-chain.

“Having the Whistleblower is annoying because it’s overcomplicated but it is crucial given the size of the computations to ensure the honesty of the verifier. It is not certain we would always have to have that, there are certain ZK techniques that mean we could but we don’t want to get ahead of ourselves.”

Ben Fielding

——————

Fin

One useful, albeit simple distinction that emerges for me at this initial stage of the analysis is that a platform is not a protocol.

Together is designed as a platform. They have aggregated a selected set of decentralized compute sources to create a specialized cloud platform for efficiently training large models that are then made available with the base datasets and detailed explanations of process. They have started by releasing open-source models that are size-optimized for people to download, run and fine-tune on their home devices or commodity GPUs. Soon, they plan to open up access to their platform. Together’s roadmap reflects its central role in scaling access to provide compute and directly train models using their cloud.

Gensyn, by contrast, is building a protocol. They are working at the very base of the stack to create a decentralized layer of machine learning compute that anyone can access at any time. The vision is for everyone to participate in creating the global foundation models that live on Gensyn by contributing work that is verified through consensus. Developers can then uses these models to create further specialized models that are nonetheless powered by the generalized intelligence of the whole.

So far, Together has focused on efficiency in training its RedPajama models and its approach could drive innovation in techniques for achieving greater performance with less compute. Gensyn is focused on ensuring that everyone has access to a maximum quotient of compute to equalize the playing field for training models on the ground floor. Both approaches aim to provide developers with friction-less and foundational access to training the most advanced AI.

I’m sure my framing of the issues will change as I continue to spend more time with these concepts. Even in the week spent between releasing my solo pod on decentralized compute and today, it feels like my understanding of the field has grown in many ways. So here’s to an evergreen beginner’s mindset, learning in public and enjoying the ride.

Signing off for now! I hope you leave reading this more informed and slightly unhinged with excitement about the future of open source AI and decentralized compute.

—————————————————————————————————————————

Indivisible is a media outlet focused on sovereignty and culture at the AI - crypto frontier. It is home to the Indivisible podcast and newsletter.

Twitter | Farcaster | Apple | Spotify

#npub15a570wxs8acaa8p6efh7fyuf40ynplzc7ufvlfk39nq285h46a9slced8v

—————————————————————————————————————————

This episode features an in-depth conversation with crypto artist Patrick Amadon about life, truth, origin stories and the future of NFTs.

Collect the Indivisible x Patrick Amadon NFT Here

Episode Overview

This episode welcomes Patrick Amadon, a crypto artist who uses a distinctive glitch style to bring us provocative social commentary through his work. Our conversation covers Patrick’s background and road to becoming an artist. We also discuss the stories behind a few of his pieces, which I found to be a dynamic vehicle for adding new dimensions to the provenance of this work. 

Patrick is a real pro, so he had a lot of insight to share on understanding the game and strategies for playing it well, over the long term. Tl;dr: play long term games with long term people. 

My favorite thing about this conversation is its honesty; Patrick and I dive deep on several subjects and he is open with sharing real but thoughtful opinions. I believe this quality is also reflected by Patrick’s authenticity in his work, which in turn has become a driver for his success. As humans, there are few things we love so well as truth. 

If you are in the mood to hear from an OG leader and creative who is pioneering 1/1 art in the NFT space and who provides an antidote to the froth, then I am pretty sure you will enjoy this conversation as much as I did. 

xA

Listen

On Apple | On Spotify | On Zora

Time Stamps

• 1:47 - What is High-Frequency Glitch?

• 3:30 - The Story Behind “Rodeo Drive”

• 10:30 - Using Process to Achieve Mastery

• 14:50 - Patrick’s Sources of Inspiration 

• 19:20 - Establishing a Career as an Artist

• 21:30 - Authenticity as a Mode For Longevity

• 25:00 - How the Filters of Cryptoart Self-Select for Tribe

• 33:30 - Finding the Right Collectors

• 35:17 - Risks Inherent to the Artist Collector Relationship

• 37:24 - Art, Money and Power

• 40:00 - Economic Prospects for the Future of Crypto Art

• 44:55 - Approach to Distribution

• 47:50 - The Story Behind “WAGMI” and Transmuting Toxic Positivity

• 53:20 - Approach to Slow and Selected Sharing 

• 55:00 -  The Story Behind “Fiat” 

• 58:03 - Deep Truth and Message for Younger Self

Gallery | Work Mentioned

Additional Content

• Patrick on Twitter

• Patrick on Superrare 

• Episode Webpage

Indivisible is a podcast and composable media exploration of crypto, creativity and freedom by design. Follow @AainaxMirror on Twitter and subscribe to the Indivisible newsletter to stay up to date with the latest content

collect://