<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/">
    <channel>
        <title>Gensyn</title>
        <link>https://paragraph.com/@gensyn</link>
        <description>undefined</description>
        <lastBuildDate>Sun, 22 Nov 2026 03:19:51 GMT</lastBuildDate>
        <docs>https://validator.w3.org/feed/docs/rss2.html</docs>
        <generator>https://github.com/jpmonette/feed</generator>
        <language>en</language>
        <image>
            <title>Gensyn</title>
            <url>https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/89b68c857d55630f6c133ff24899b5140d5c6b5b3b35b25e7379505e4e2ba93b.png</url>
            <link>https://paragraph.com/@gensyn</link>
        </image>
        <copyright>All rights reserved</copyright>
        <item>
            <title><![CDATA[The Machine Learning Compute Protocol and our future]]></title>
            <link>https://paragraph.com/@gensyn/the-machine-learning-compute-protocol-and-our-future</link>
            <guid>qQ8smtuQCHdnIfA2pNZQ</guid>
            <pubDate>Sun, 11 Jun 2023 23:01:33 GMT</pubDate>
            <description><![CDATA[Machine learning is undergoing a renaissanceMachine learning compute is the most important market in the world and could surpass 1% of all US GDP in < 7 yearsThis market is controlled by a few players who limit access and extract rentA Machine Learning Compute Protocol provides an accessible and cost-effective alternative: securing the right to build AI and computational liberty for everyone.To be successful the protocol must meet the GHOSTLY criteriaIf AI is going to reach its full potential...]]></description>
            <content:encoded><![CDATA[<ul><li><p><strong>Machine learning is undergoing a renaissance</strong></p></li><li><p><strong>Machine learning compute is the most important market in the world and could surpass 1% of all US GDP in &lt; 7 years</strong></p></li><li><p><strong>This market is controlled by a few players who limit access and extract rent</strong></p></li><li><p><strong>A Machine Learning Compute Protocol provides an accessible and cost-effective alternative: securing the right to build AI and computational liberty for everyone.</strong></p></li><li><p><strong>To be successful the protocol must meet the GHOSTLY criteria</strong></p></li><li><p><strong>If AI is going to reach its full potential we must establish the right to build</strong></p></li></ul><hr><p>Machine learning models are<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://arxiv.org/pdf/1604.07316.pdf%5D"> driving our cars</a>, testing our<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://jamanetwork.com/journals/jama/fullarticle/2588763/"> eyesight</a>, detecting our<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.nature.com/articles/s41598-018-22437-z"> cancer</a>, giving sight to the<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.jci.org/articles/view/151331/pdf"> blind</a>, giving speech to the<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.biorxiv.org/content/10.1101/2023.01.21.524489v1"> mute</a>, and dictating what<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.nytimes.com/2021/12/05/business/media/tiktok-algorithm.html"> we consume and enjoy</a>. These models are a part of the human experience now and the way we create and govern them is going to shape our future as a species. Our belief is that an open system of computational resources, accessible by everyone, is the best way to ensure that we are all represented by the artificially intelligent systems that we create.</p><p>Soon, we will be able to conjure entertainment on a whim, moulded to our exact preferences and desires and shaped by every sentence we utter and every reaction we exhibit. We’ll be able to tune in to<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.twitch.tv/watchmeforever"> never-ending TV series</a> that feature ourselves and our loved ones, watch action films where our favourite heroes save any world we choose, watch <em>C-beams glitter in the dark near the Tannhäuser gate</em>, and live in digital worlds that constantly change and evolve around us.</p><p>Beyond entertainment, our lives will be enriched by constant, probabilistic feedback. We’ll have personalised foundation models that act as external memory, consuming the information that we consume and learning to approximate and augment our inner thoughts through <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://proceedings.neurips.cc/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf">RLHF</a>. We’ll adapt to new methods of storing information—much like we adapted to search engines—storing the keys for retrieval rather than the information itself. Historical figures will be re-awoken by the imprints they’ve left behind, our loved ones will live on through chat systems that know what they knew, and the lines between human and machine interaction will begin to blur without us even noticing. Biological evolution can be slow but social and intellectual evolution is fast, and through it we’ll become indistinguishable from the intelligent systems that we create.</p><p>This future requires <strong>massive</strong> computational power, an always-on network of raw processing resources, the electricity of a new age. Harnessing this power and providing it to every home, every device, and every person should be done through networks that we all govern. Owned by no one but accessible by everyone - with value streams that incentivise us all to provision the resources to them. Forever.</p><h2 id="h-0-machine-learning-compute-and-agi" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">0. Machine learning, compute, &amp; AGI</h2><ul><li><p><strong>Machine Learning (ML) is the branch of AI that deals with systems that learn from data</strong></p></li><li><p><strong>Compute is the the number of calculations a device can achieve per second</strong></p></li><li><p><strong>AGI is the concept of a generalised intelligence (i.e. able to make sense of – and exist within – the wider world) made from artificial means</strong></p></li></ul><p>Before continuing, it’s best to describe what we mean by ‘machine learning’, ‘compute’, and ‘AGI’.</p><p>Machine learning (ML) is a subfield of Artificial Intelligence (AI). Where AI broadly represents any computer program that makes decisions autonomously, ML typically refers to computer programs that can <strong>learn how to make those decisions</strong> from data they ingest. We call the programs that make these decisions <strong>models</strong> and the process of instilling knowledge within them as <strong>training</strong>.</p><p>AI has always been a mercurial concept as our shared definition of intelligence can vary significantly <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://en.wikipedia.org/wiki/AI_effect">over time</a>. For example, the AI models which made (relatively poor) weather predictions in the mid 1950s are unlikely to seem intelligent when contrasted with the AI models that can hold a fluid conversation with you today.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/478465fdcaae9a93d819a024737240e42340a03310661515f6496b1a31fbfeec.png" alt="Large Language Model, GPT-4, speaks with a user about its origins." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Large Language Model, GPT-4, speaks with a user about its origins.</figcaption></figure><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/33514abe0cbdc91225cc37f9ed030e56d39daf779df274f7b28b167e93cea1d8.png" alt="Weather prediction Neural Networks featured in a 1958 copy of the New York Times." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Weather prediction Neural Networks featured in a 1958 copy of the New York Times.</figcaption></figure><p>Computational power, or ‘compute’, is electricity transformed by a machine to perform a calculation; the most basic example of this is a FLOP (FLoating point OPeration), e.g. 1.0 + 1.0 = 2.0, which is usually measured in FLOPS (FLOPs per Second). For historical context, the two Apollo Guidance Computers that flew the Apollo 11 mission to the moon in 1969 both achieved <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://arxiv.org/pdf/2201.08230.pdf">14,245  FLOPS</a>, and today’s iPhone 14 achieves <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://appleinsider.com/articles/22/09/26/how-iphone-speeds-have-grown-in-the-last-5-years">~2 trillion FLOPS</a>.</p><p>FLOPs are the building blocks of computation and are particularly valuable for understanding the computational complexity (or ‘amount’) of compute a model requires to learn. However, the amount of compute used to train a model is also a liquid concept; partly because FLOPs are executed faster/more efficiently on state of the art devices (meaning the electrical ‘cost’ of a FLOP 10 years ago was more expensive than it is now), but also because a FLOP isn’t always the best measure of compute power. For example, the rate at which data can be transferred from memory to the processor might be the main bottleneck in a computational task. Notwithstanding these two issues, FLOPS are a convenient way to quantify the performance of computational hardware.</p><p>As AI has developed as a field, we’ve seen a progression of models from expert knowledge systems like <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.sciencedirect.com/science/article/abs/pii/S0167865599000537">template matching</a>, which required human knowledge to be manually hardcoded into the model, to ML models like <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://ieeexplore.ieee.org/document/990517">Viola-Jones</a> in 2001 that automatically discovered visual features in an image. Today, we typically use deep learning (DL) models when performing advanced ML tasks. These models are a special class of Artificial Neural Network (ANN or just NN) that (very loosely) resemble human neural networks and gained prominence in the 2010s when they showed an incredible ability to <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.image-net.org/challenges/LSVRC/">understand images</a>. DL models are simply NNs that contain a large number of intermediate (or hidden) layers of processing between the input data and the final output. These layers allow the model to reason about the input and build its own understanding of the world by building up a ‘memory’ (known as parameters) of things that it has seen before.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/538084b79457a1a2a72e672b79d2806366da9364e51ae6562cbbfc6c65e2805c.gif" alt="A tiny neural network ingests the image of a handwritten &apos;9&apos; during its training process. Source: 3blue1brown" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">A tiny neural network ingests the image of a handwritten &apos;9&apos; during its training process. Source: 3blue1brown</figcaption></figure><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/22de5bc623a7a6132a483bac100b57b47ed11301726a124a0ccc97bb5b4d3f4a.gif" alt="The same neural network begins to learn the relationship between the handwritten images and the true numbers. Each layer in the network traditionally learns a different concept.  E.g. &quot;does this number have a circle in it&quot;. Source: 3blue1brown." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">The same neural network begins to learn the relationship between the handwritten images and the true numbers. Each layer in the network traditionally learns a different concept. E.g. &quot;does this number have a circle in it&quot;. Source: 3blue1brown.</figcaption></figure><p>The rapid advances we see in AI are now almost entirely from the field of deep learning, and mostly as a consequence of the ever-increasing size (depth) of the models and the data used to train them. Put simply, more of everything generally results in models that 1. make fewer mistakes, and 2. can be applied to ever more complex use cases.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/753771be788b059f1008189ec926b8fe645f376cefc49ed87b512efb57fcf39a.png" alt="Bigger is better. Source: Kaplan (2020)." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Bigger is better. Source: Kaplan (2020).</figcaption></figure><p>As DL models have rapidly grown in size (roughly doubling every <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://arxiv.org/pdf/2202.05924.pdf">6 months</a>), their impressive ability to generalise (i.e. use their learning on tasks other than the one they originally undertook) has led to the creation of <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://arxiv.org/abs/2108.07258">foundation models</a>. These models take the concept of transfer learning (using a pre-trained model to quickly start training on a new task) and expand it into the idea of creating a model that can perform a large number of tasks through either multiple interfaces or a single general interface, often <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://chat.openai.com/">chat-based</a>. Foundation models and shared interfaces give AI a way to interact with the world that goes beyond the efficient performance of a single task, leading many to consider them the seeds of Artificial General Intelligence (AGI).</p><p>AGI <em>could</em> be a single, large model that is able to generalise incredibly well about the world, but it could also be a rich ecosystem of different models, all continuously learning and interacting with each other autonomously. The former represents a quick path towards impressive outcomes plagued by <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.forbes.com/sites/forbestechcouncil/2023/03/31/uncovering-the-different-types-of-chatgpt-bias/">fragile biases</a> and difficulty <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://openai.com/research/instruction-following">aligning objectives</a>. The latter allows for meta-optimisation through a free market of models, voted on by <strong>all of us</strong> through our usage. If you don’t think a model represents your way of thinking - create a new one!</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/f1ed440b4f5af206d2733155ac336b21fb4783ac6bcc791af8cd7a474f354501.jpg" alt="A decade of exponential growth in raining FLOPs. Sevilla et al (2022)." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">A decade of exponential growth in raining FLOPs. Sevilla et al (2022).</figcaption></figure><h2 id="h-1-the-most-important-market-in-the-world" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">1. The most important market in the world</h2><ul><li><p><strong>Compute is powering a new industrial revolution in intelligence</strong></p></li><li><p><strong>ML training compute costs alone could reach 1% of US GDP in 8 years</strong></p></li><li><p><strong>Hardware is hugely concentrated and price gouged by incumbents</strong></p></li></ul><p>We’ve undergone roughly three industrial revolutions since the 1760s: starting with steam, then electricity, and most recently the information revolution powered by the internet. It’s clear we’re on the cusp of something else - an intelligence revolution - and even clearer what the accelerant for it will be: compute.</p><p>Recall that we previously noted the main drivers of AI system performance are architecture, data, and compute. Both architectural advancements and data are (relatively) easy to procure. Academic publishing disseminates the knowledge required to build complex models, ensuring that new breakthroughs are understandable and useable by everyone soon after discovery. Data is now ubiquitous, and the requirement for complex human labelling is slipping away, replaced with <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.amacad.org/publication/human-language-understanding-reasoning">unsupervised and semi-supervised training methods</a> that can create rich embedding spaces simply by processing huge amounts of unlabelled open data.</p><p>So, what about compute? It is used during the training phase of models (when they ‘learn’) and also during inference (when they make predictions). However, the former is significantly more compute-intensive than the latter. As early as 2012, it became clear that training on Graphical Processing Units (GPUs) was vastly preferential to training on CPUs. This is because individual NN operations are highly parallelisable; for example a matrix multiplication can be split into many concurrent operations under the paradigm of Single Instruction Multiple Data (SIMD). The most common hardware used for training NNs is still GPUs but they now come with specialised cores for computing NN layers called Tensor Cores. Additionally, several companies have developed their own specialised Deep Learning ASICs that accelerate matrix multiplication and convolution operators under the terminology of [Intelligence/Neural/Tensor] Processing Units (IPU/NPU/TPU).</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/b9d321a3e45904b5f73234c8d77b17e8afe7b530bb8184db695b6399aa96e26e.jpg" alt="An IPU-M2000 (Intelligence Processing Unit) from Graphcore. A single IPU provides 1 petaFLOP of compute. Source: Graphcore" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">An IPU-M2000 (Intelligence Processing Unit) from Graphcore. A single IPU provides 1 petaFLOP of compute. Source: Graphcore</figcaption></figure><p>This hardware is expensive: Meta’s 2023 <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.reuters.com/technology/meta-launch-ai-language-model-llama-2023-02-24/"> LLaMA model</a>, a large language model (LLM), was trained on <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://news.ycombinator.com/item?id=35008694">$30m</a> of GPU hardware. Indeed, training models, and re-training them (‘fine-tuning’) for certain tasks, is the most capital-intensive part of the revolution. The cost to train models has been rising exponentially since 2009, as advances in <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.lesswrong.com/posts/midXmMb2Xg37F2Kgn/new-scaling-laws-for-large-language-models"> compute capacity</a> unlocked new model use-cases which, in turn, increased demand. Extrapolating growth trends, model training alone could <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://epochai.org/blog/trends-in-the-dollar-training-cost-of-machine-learning-systems">hit 1% of US GDP</a> in the next 8 years. This year the revolution was reflected in the share price of NVIDIA - the world’s most eminent GPU manufacturer: their market capitalization eclipsed $1 trillion for the first time in history.</p><p>However, it is the cloud giants who currently stand to truly benefit from this. As demand for HPC ballooned in 2010, AWS (Amazon Web Services) introduced their first GPU instance (the Cluster GPU instance - which had Nvidia M2050s). Consider AWS’s current <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://aws.amazon.com/ec2/instance-types/p4/">p4d.24xlarge</a> compute instance - a server you can rent by the hour which contains eight <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.nvidia.com/en-gb/data-center/a100/">Nvidia A100 40GB GPUs</a>. The cost for an Ohio instance is $4.10/hour/gpu. Per Amazon’s balance sheet and external research, the marginal Cost of Goods Sold for delivering the instance is $1.60 (equating to a<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.cnbc.com/2021/09/05/how-amazon-web-services-makes-money-estimated-margins-by-service.html"> 61%</a> gross margin) and the marginal operating expense amounts to $1.50 (equating to a<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://ir.aboutamazon.com/news-release/news-release-details/2023/Amazon.com-Announces-Fourth-Quarter-Results/default.aspx"> 24.3%</a> operating margin in Q4 2023). Microsoft has an even higher gross margin for its cloud services: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.microsoft.com/en-us/Investor/earnings/FY-2023-Q2/performance">72%</a>.</p><p>Other cloud giants have followed suit, leading to a “cloud war” over computation and a race to secure hardware. This is illustrated by ARK Invest’s <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://thoughtkast.com/2022/05/05/ark-big-ideas-2022-review-part-2-artificial-intelligence/">projected</a> compute-related hardware spend: $1.7T by 2030.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/7b2e152a1ef4780fcce480682e585b33c11391a492710045aa59ce5b7f97a397.png" alt="Training costs keep rising. Source: Epoch AI" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Training costs keep rising. Source: Epoch AI</figcaption></figure><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/62c6d7e009d1cf02119933834ae3e6e8691b2075dfc553065002fc2c79fcd92c.png" alt="At current growth rates, training costs could reach 1% of US GDP by 2031. Source: Epoch AI" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">At current growth rates, training costs could reach 1% of US GDP by 2031. Source: Epoch AI</figcaption></figure><p>This scramble to amass hardware is true even outside of cloud giants. Consider the aforementioned Nvidia A100 GPU and the newer <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.nvidia.com/en-gb/data-center/h100/">H100</a> - the uranium in the arms race to amass computational power. Ownership and concentration of these GPUs into High Performance Compute (HPC) clusters has a strong pareto relationship, and skews heavily to very well financed incumbent organisations. The benefit (&amp; consequences) of owning this compute cannot be understated. It’s why the US, EU, and Japan have repeatedly applied ‘Chip’ (semiconductor) <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://edition.cnn.com/2023/03/31/tech/japan-china-chip-export-curbs-intl-hnk/index.html">sanctions</a> to China - to slow down their ability to build AGI. It’s also why large AI companies are frantically <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.opensecrets.org/news/2023/05/big-tech-lobbying-on-ai-regulation-as-industry-races-to-harness-chatgpt-popularity/">lobbying</a> to maintain their edge over new market entrants: controlling compute is like having a monopoly on steam power, the electrical grid, or the internet (<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://interestingengineering.com/culture/james-watt-father-of-the-modern-steam-engine">people</a> <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.justice.gov/atr/technological-innovation-and-monopolization">have</a> <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.justice.gov/atr/case/us-and-plaintiff-states-v-google-llc">tried</a>).</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/a46897a11ad7ed3add10a524950defc88a9edda3a0e3bbf00bed766f38501c9d.png" alt="A100 concentration by entity. Source: State of AI Report" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">A100 concentration by entity. Source: State of AI Report</figcaption></figure><h2 id="h-2-the-protocol-for-machine-learning-compute" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">2.  The protocol for machine learning compute</h2><ul><li><p><strong>Previous decentralised compute and file sharing networks present an alternative perspective on the compute issue</strong></p></li><li><p><strong>Hardware access can be increased, and price decreased, by using a P2P network secured by a blockchain</strong></p></li><li><p><strong>For such a protocol to work it must address the GHOSTLY criteria</strong></p></li></ul><p>What can be done? The year is 2023, costs are spiralling out of control, there is a ground war by incumbent countries and companies to lock in (and expand) control via legislation, and some developers are struggling to even access a <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/Suhail/status/1643002186320994304">single GPU</a>.</p><p>If we were to pause and ask ‘how would we approach this differently’, what questions might we ask? Perhaps:</p><ul><li><p>Why are we paying cloud oligopolists 72 cents of every dollar we spend on compute?</p></li><li><p>Given the exaFLOPs of underutilised compute in the world (see below) - why are we only training models in centralised server farms?</p></li><li><p>If we’re willing to pay more for it, why can’t we access the state-of-the-art GPUs that cloud giants give to their <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://blogs.microsoft.com/blog/2023/01/23/microsoftandopenaiextendpartnership/">preferred clients</a>?</p></li><li><p>Why does OpenAI <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/ETNOWlive/status/1666460799093620738">tell us</a> that we won&apos;t be able to train an LLM from scratch?</p></li></ul><p>The answer to all of these questions, obviously, is <em>because of the economic model under which they operate</em> (closed source, rentier capitalism). Further, any attempt to solve these problems that use the same economic model are destined to only result in different companies with identical outcomes and a repeating cycle.</p><p>However, when we look at the history of HPC, we see the seeds of another path: there are teraFLOP or even exaFLOP projects (exa means 1018, or a 1 followed by 18 zeros) which operate over <em>crowdsourced compute</em>. As we’ve noted <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://mirror.xyz/gensyn.eth/0SHaOYVPhATdTfw8Ypixoln_G5HF_NcVz9gEI7AXLTw">before</a>, well before Google was stockpiling TPUs, projects like <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://setiathome.berkeley.edu/">SETI@home</a> attempted to discover alien life by using decentralised compute power provided on a voluntary basis. By the year 2000, SETI@home had a processing rate of<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://setiathome.berkeley.edu/sah_papers/CISE.pdf">17 teraflops</a>, which is more than double the performance of the best supercomputer at the time, the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://en.wikipedia.org/wiki/ASCI_White">IBM ASCI White</a>. SETI@home was sunsetted in 2020; however, in its place, <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://foldingathome.org/">Folding@home</a> rose to prominence - crowdsourcing 2 exaFLOPS of volunteer compute to run protein-folding programs (used to better understand and treat diseases).</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/093c6a7d5811971e7e80b2817ec1faf39f211ede463f6ef24070761f04532876.png" alt="Searching for aliens in 1999. Source: New York Times." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Searching for aliens in 1999. Source: New York Times.</figcaption></figure><p>The crowdsourced compute approach seems to address some of our questions above: there is no access limit (any computer can join the network, increasing supply); there is no margin charged here; money doesn’t even change hands. Indeed, mutual relationships like this have occurred elsewhere in the internet revolution. Sites like PirateBay and BitTorrent, which <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.computerworld.com/article/2523466/the-pirate-bay-four-found-guilty-in-sweden.html">came to prominence</a> in the early 2000s as a peer-to-peer (P2P) method to exchange (often copyrighted) media. Despite the legal issues, they demonstrated that P2P could exist at a large scale (in the past <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://citeseerx.ist.psu.edu/document?repid=rep1&amp;type=pdf&amp;doi=f5873b83702a6defb52ae2ba9cdc3e1b34259b60">65-70% of internet traffic was estimated to be P2P</a>) - as long as the users were properly incentivized.</p><p>The issue is what would happen if money were to change hands? People would have a strong incentive to cheat. Indeed, some users even <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.zdnet.com/article/seti-venture-plagued-by-cheating/">cheated on SETI@home</a> to get to the top of the volunteer leaderboard.  To solve this we require a core system that can secure the economic incentives of the network and verify that the resource provisioned at the lowest layer is provided correctly. <strong>We need a new protocol.</strong></p><p>This is similar to the issue that Bitcoin author(s) Satoshi Nakamoto had to overcome in 2008 when attempting to create a protocol that would enable virtual money to be securely transacted with no intermediaries. In this case, the core resource being provided is consensus itself - that is, agreement on the outcome of a state transition within the network. Irrespective of one’s views on Bitcoin, one thing is clear: it successfully incentivizes crowdsourced hardware to join a network and behave in a certain way. Indeed, approximately $13bn of hardware currently contributes to the network (at today’s Hashrate, assuming Antminer S19 Hydro devices) and has ultimately been <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.newsbtc.com/news/bitcoin/bitcoin-creator-satoshi-nakamoto-nominated-nobel-prize/">lauded</a> for its security and verification of transactions.</p><p>After Bitcoin proved that the protocol concept could create and secure an open financial system, Ethereum expanded the concept to arbitrary computational tasks with the inclusion of the Ethereum Virtual Machine (EVM), which is (technically <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://cypherpunks-core.github.io/ethereumbook/13evm.html">quasi</a>) <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://ethereum.org/en/whitepaper/">Turing-complete</a>. This demonstrated  that, through those same dynamics, we can build a trustless network of computing parties to perform <strong>any</strong> computation.</p><p>A downstream benefit of this protocol approach is that cryptonetworks like Bitcoin and Ethereum - which uses a more financialised and less energy-intensive consensus mechanism - <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/cdixon/status/1425645842552086532">do not charge margins</a>. Instead, a small tax from every payment is returned to the network, destined to be used to maintain the network itself as directed by community governance.</p><p>An additional advantage of P2P for hardware networking is that the devices don’t need to be located in traditional data centres. Indeed, beyond the data centre GPUs like the A100, the following devices become available:</p><ol><li><p><strong>Custom ASICs</strong>: Like Graphcore’s IPUs - custom-built for machine learning - which could be combined into small clusters similar to Bitcoin mining operations.</p></li><li><p><strong>Consumer GPUs</strong>: Like the Nvidia RTX series- mainly used for gaming but a mainstay of ML researchers in academia.</p></li><li><p><strong>Consumer SoCs</strong>: Like Apple M2 - general-purpose chips with neural processing abilities. Even M1 processors can be used for <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://sebastianraschka.com/blog/2022/pytorch-m1-gpu.html">training </a>with performance comparable to a mid-range Nvidia RTX.</p></li></ol><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/cc635298778eae5a52426b75d27ac01e920c3c8c6632f365855343fad1771f6e.png" alt="5 Nvidia GPUs combined into a rig. Source: Gensyn" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">5 Nvidia GPUs combined into a rig. Source: Gensyn</figcaption></figure><p>The greater supply and superior unit economics of such a protocol could reduce the price of compute by up to an order of magnitude. For example, the Nvidia A100 mentioned above can be purchased in bulk at $7,800/unit (US-customs cleared, December 2022). Hardware miners in the Bitcoin network typically rate opportunities on their ability to return the value of the hardware (note: the price of hardware can change in this time) over a certain time period, commonly 18 months. With a total power draw of 250w/h and assuming a constant price of electricity and maintenance of $0.10/Kwh in the United States, for a miner to be incentivised to provide deep learning computation they require a yield of $0.59/h to cover the machine and $0.025/h to cover electricity. This means they could provide compute for as little as ~$0.62/h → ~85% cheaper than AWS.</p><p>However, so far we’ve only considered protocols like Ethereum which work for small-scale computation - where every peer in the network performs the work (putting an upper cap on the computational complexity). If we wanted to perform large-scale computation using trustlessly-provisioned hardware to train a machine learning model, we would need to achieve one of two things:</p><ol><li><p>Make the computations more efficient so that they can feasibly be performed by the chain; or</p></li><li><p>Prove that large-scale computations, performed off-chain, were performed as-specified to the satisfaction of the chain.</p></li></ol><p>The former point is possible but results in heavy constraints on the model, owing to the scaling laws mentioned earlier; the latter is a compelling concept. In this setting, a blockchain secured by a trustless consensus mechanism and able to do arbitrary small computations would be able to take the place of a trusted third party in a verification dispute - as long as the final disputed portion of the computation can be reduced to a size that is reasonable for a chain to perform. This makes the verification system, and particularly the additional cost of verifying the work, core to the protocol’s success.</p><p>Indeed, such a protocol would need to satisfy six defining principles, GHOSTLY:</p><ol><li><p><strong>Generalisability</strong> - Specifically, can you submit a <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://machinelearningmastery.com/pytorch-tutorial-develop-deep-learning-models/">custom architecture, optimiser, and dataset</a> and receive a trained model? This is a core user requirement for ML developers; constraining a protocol to a single type of model training would only work if that mode of training became the default form of training (fragile/very unlikely).</p></li><li><p><strong>Heterogeneity (over supply)</strong> - Core to achieving maximum scale of supply is ensuring the ability to use different processor architectures and different operating systems. If the protocol requires hashing (as is highly common in decentralised protocols) then the computation performed on the different devices must result in identical outputs, i.e. it must be deterministic (or reproducible). Despite the fact that differences in output are often tiny, it is surprisingly hard to achieve reproducibility for deep learning because:</p><ol><li><p>Different processor architectures add together numbers differently and at varying levels of precision</p></li><li><p>Many parts of the training process involve random seeds that are often hard to control</p></li><li><p>The physical hardware is susceptible to cosmic ray radiation causing bit-flips - but this is widely solved by <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.researchgate.net/publication/262409861_An_investigation_of_the_effects_of_error_correcting_code_on_GPU-accelerated_molecular_dynamics_simulations">Error Correcting Codes</a></p></li><li><p>Stochasticity has actually proven <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://jmlr.org/papers/v15/srivastava14a.html">beneficial</a> for deep learning in avoiding overfitting, leaving little reason to address it in the face of <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://pytorch.org/docs/stable/notes/randomness.html">greater throughput</a></p></li></ol></li><li><p><strong>Overheads (low)</strong> - Ethereum blockspace could be theoretically used to train a model; however, this would <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.gensyn.ai/litepaper/#protocol-evaluation">incur</a> a ~7850x computational overhead vs an AWS instance using a Nvidia V100 GPU as a reference. Similarly, even a <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://arxiv.org/abs/1908.04756">Truebit</a>-style game-theoretic replication approach fails at achieving suitably low overhead (still ~7x). Any approach to verification must have a minor overhead lest it offset the unit economic gains achieved through decentralisation. Further, <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://blog.openzeppelin.com/thin-protocols-cc872258379f">Fat protocols</a> typically incur higher overheads as they attempt to allow every possible computation.</p></li><li><p><strong>Scalability</strong> - Protocols relying on specialised hardware that are distinct from those used by normal machine learning researchers invariably cannot scale to State-of-The-Art (SoTA) LLMs. For example, use of Trusted Execution Environments (TEEs) partially solve the trust problem at the expense of <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://petsymposium.org/popets/2022/popets-2022-0105.pdf">huge speed reductions</a>.</p></li><li><p><strong>Trustlessness</strong> - If the network requires trust, then:</p><ol><li><p>it incurs scaling limits as a trusted party is required to verify work; and/or</p></li><li><p>the network cannot scale above a certain point before trust is broken and the dominant economic outcome becomes cheating; and/or</p></li><li><p>an operational labour cost is borne by the network that puts upward pressure on price (e.g. KYC checks).</p></li></ol></li><li><p><strong>LatencY</strong> - Inference is highly sensitive to latency (training systems less so). Core to this problem is the notion of model parallelism; that is, if you are compute-rich and latency-poor, instead of sending a 1GB model to one device, could you send 1024 1 MB models to 1024 devices? Could you split it up even more? The short answer for most NN architectures is no. NN training requires context (state dependency) about what is happening elsewhere in the model (unlike <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.cs.ucdavis.edu/~matloff/matloff/public_html/158/Hwk/Becker_Peri_Final_Report.pdf">SETI which is embarrassingly parallel</a>) and therefore requires a system which can: a) optimally trade off compute capacity for network bandwidth; and b) somehow chunk a model into smaller components without violating point 3) above.</p></li></ol><p>In satisfying the GHOSTLY principles, a machine learning compute protocol would combine the crowdsourced grid computing of SETI@Home with the P2P verification and security of Ethereum to perform machine learning tasks over the entire world’s hardware.</p><h2 id="h-3-the-right-to-build" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">3. The right to build</h2><ul><li><p><strong>Computational liberty is best achieved via technological means, not legal means</strong></p></li><li><p><strong>The decentralised machine learning compute protocol is core to building AI systems that are aligned with society</strong></p></li><li><p><strong>Such a protocol also increases the rate of technological change by preventing oligopolies</strong></p></li></ul><p>For machine learning models to proliferate throughout our world in a fair and equitable manner, it is crucial that we establish the <strong>right to build</strong>. Not as a<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://en.wikipedia.org/wiki/Right_to_repair"> legal right</a> enforced by governments, but as a technological right enabled by building with a new perspective - decentralisation. This right is very simple - <strong>we should all be able to access the resources required to build new machine learning models</strong>. We’re in a race against competitors aiming to dominate our future by controlling an enabling technology that will radically alter our society.</p><p>Regulating this technology will create new monopolies, and those monopolies will do everything they can to maintain their position and increase their capture of value. Without the feedback loop of competition, we lose the direct representation of the voices of consumers and must rely on those same regulating institutions to uphold their positions. Why hand this task to a chain of organisations, each with their own incentives, when we can rely directly on the market to perform it?</p><p>Decentralised networks give us the ability to change the incentive structure itself, to choose a new set of guiding principles for the powerful optimising force of capitalism. By building protocols instead of companies we can counteract the behaviours that inevitably <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://onezero.medium.com/why-decentralization-matters-5e3f79f7638e">turn innovation into rent-seeking oligopolies</a>.</p><p>We see this trend with storage protocols like <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://ipfs.tech/">IPFS</a>, networking layers like<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.helium.com/"> Helium</a>, and niche computational networks like<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.rendertoken.com/"> Render Token</a>.  Crucially, unlike storage, which is incredibly cheap to scale in a centralised way; networking, which suffers from<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.oxfordreference.com/display/10.1093/oi/authority.20110810105406240"> Metcalfe’s Law</a>; and rendering, which is a relatively niche market, machine learning hardware has clear economic value at any scale. The first GPU on the network is just as valuable as the last.</p><p>Through <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://a16zcrypto.com/posts/article/a-taxonomy-of-tokens-distinctions-with-a-difference/">protocol economics</a>, we can ensure that the value flows are governed by a network of participants, with no barrier to entry other than owning a valuable resource. These networks do not isolate distribution or empower existing royalty, they manage a pool of participants as a shared resource to be optimised. Distributing value over the core resources in this way, the protocol incentivises improving the provision of the resource itself, not capturing more of the distribution.</p><p>Decentralised networks don’t merely improve economic incentives, they also free us from authoritarian control over the content that we consume and even our personal communications themselves. As per <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://a16zcrypto.com/posts/article/control-software-powers-internet/">Chris Dixon</a>, ‘But the benefits of such a shift will be immense. Instead of placing our trust in corporations, we can place our trust in community-owned and -operated software, transforming the internet’s governing principle from “don’t be evil” back to “can’t be evil”.’</p><h2 id="h-4-the-future" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">4. The future</h2><ul><li><p><strong>The world will be unrecognisable in 5 years</strong></p></li></ul><p>In 5 years time, a developer won’t think about where their model will be trained, how the server is configured, how many GPUs they can access, or any other system administration details. They’ll simply define their model architecture and hyperparameters and send it out to a protocol - where it could be trained on a single GPU, a cluster of TPUs, <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.theverge.com/2021/1/27/22253162/iphone-users-total-number-billion-apple-tim-cook-q1-2021">a billion iPhones</a>, or any combination thereof.</p><p>We’ll look back on human-guided training of AI models like we look back on ancestral hand-tools; a necessary, yet almost incomprehensibly expensive (in terms of human effort) step towards an automated future.</p><p>By building the decentralised rails to carry computation to the hardware and payments to the participants, a machine learning compute protocol would be an invisible network, supplying the compute revolution with the base resources it needs to move to the next level of scale. Right now we’re all relying on a small number of companies to drip-feed access to computation to the partners they choose. Through decentralisation we could create a free market over all compute, allowing anyone to pay a fair price to access as much of it as they’d like. Such a network would enable currently gatekept developers to try out new architectures, datasets, and even modalities - ultimately creating things that would otherwise never exist.</p><p>Conversely, we could all contribute resources to the network, getting paid for the compute power that we have when we’re not using it. When our MacBooks or iPhones are charging, when our gaming PCs are idle, when we’re sleeping but the other side of the earth is awake, our devices could earn us money. By spreading the load in this way we’d take advantage of idle efficiencies, greener sources of electricity, geographical downtimes, and make the most out of the resources we’ve already manufactured. As Bitcoin began to<a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.coindesk.com/policy/2021/10/11/bitcoin-mining-is-reshaping-the-energy-sector-and-no-one-is-talking-about-it/"> proxy the global energy markets</a>—this would do the same—all underpinned by real-economic-valued work, requested and paid for by others.</p><p>Ultimately, those others will be machine learning models themselves. Living on a self-organising substrate over the core resources required to sustain them, they’ll trigger training, inference, and even the creation of new models. Artificial General Intelligence won’t be a single large model that can answer questions, it will be a living ecosystem of models that can think for themselves, replicate, and one day surpass our understanding. It’s important that we build this in the open and for everyone - we can’t allow the future of AI to be in the hands of a small minority who think they know what’s best - we should all have a part in this future.</p>]]></content:encoded>
            <author>gensyn@newsletter.paragraph.com (Gensyn)</author>
            <enclosure url="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/977d794bbdd7eed6b8c5110dc86a563519bfc532695f26933d781b2593e328d4.png" length="0" type="image/png"/>
        </item>
        <item>
            <title><![CDATA[Deep Learning’s New Infrastructure]]></title>
            <link>https://paragraph.com/@gensyn/deep-learning-s-new-infrastructure</link>
            <guid>1IU2ErNmRMxwsCtq1oly</guid>
            <pubDate>Mon, 07 Mar 2022 17:29:33 GMT</pubDate>
            <description><![CDATA[Deep learning is changing the human experience but there is a battle raging for control of the underlying infrastructure.Meta, the company behind Facebook, recently revealed its own AI supercomputer, a cluster of approximately 6,000 GPUs. This makes it the ~5th fastest supercomputer in the world and underscores their ambition to own the infrastructure on which the world’s artificial intelligence is developed. As we’ve seen in the past decade, what’s good for Meta and other oligopolists genera...]]></description>
            <content:encoded><![CDATA[<h2 id="h-deep-learning-is-changing-the-human-experience-but-there-is-a-battle-raging-for-control-of-the-underlying-infrastructure" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Deep learning is changing the human experience but there is a battle raging for control of the underlying infrastructure.</h2><p>Meta, the company behind Facebook, recently revealed its own AI supercomputer, a cluster of approximately 6,000 GPUs. This makes it the ~5th fastest supercomputer in the world and underscores their ambition to own the infrastructure on which the world’s artificial intelligence is developed.</p><p>As we’ve seen in the past decade, what’s good for Meta and other oligopolists <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.eff.org/issues/big-tech">generally</a> hasn’t been good for us; the ease of use of their products typically trades off against price gouging and censorship/anti-competitive behaviour.</p><p>Thus far, most of the fight against Big Tech centralisation has been on age-old issues, principally, freedom of speech. In the coming decades, as AI becomes intractably integrated into society, a more fundamental concern will arise: freedom of compute.</p><p>As we discuss below, limitations and control on compute introduce  existential implications, so let&apos;s rewind a bit, unpack what&apos;s really at stake here, and examine a radically new way to think about the problem.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/0b750a6a491b1be0c8b90aa316a68e5eb555848390bce03dac1ef1fd4f37de3c.png" alt="Supercomputer: Facebook announces a cluster of 16k GPUs exclusively for model training." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Supercomputer: Facebook announces a cluster of 16k GPUs exclusively for model training.</figcaption></figure><h2 id="h-deep-learning-is-everywhere-in-your-life" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Deep learning is everywhere in your life</h2><p>Every face you see on a video call and all the audio you hear is manipulated. To improve call quality, neural networks <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.hbs.edu/faculty/Pages/item.aspx?num=58693">selectively</a> adjust the resolution in Zoom and <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://techcommunity.microsoft.com/t5/microsoft-teams-blog/reduce-background-noise-in-microsoft-teams-meetings-with-ai/ba-p/1992318">suppress background noise</a> in Microsoft Teams. More recent advances even see lower resolution video <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.youtube.com/watch?v=CRwqfSZaKOQ">‘dreamed’</a> into a higher resolution.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/3b6e267edb2e29baf59c7ae23bdde37c61c28769f3ff12ed93cef11f69504462.jpg" alt="Your video calls aren’t what they seem: Nvidia’s AI video compression technology can convincingly recreate your face from low resolution video." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Your video calls aren’t what they seem: Nvidia’s AI video compression technology can convincingly recreate your face from low resolution video.</figcaption></figure><p>Neural networks are the models used in the deep learning branch of artificial intelligence. They are loosely based on the structure of the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.ibm.com/uk-en/cloud/learn/neural-networks">human brain</a> and have myriad applications, perhaps ultimately creating human level artificial intelligence. Bigger models generally yield better results, and the hardware required for state-of-the-art development is doubling <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.nature.com/articles/d41586-021-00530-0">every three months</a>.</p><p>This explosion in development has made deep learning a fundamental part of the modern human experience. In 2020, a neural network <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.popularmechanics.com/military/aviation/a34978872/artificial-intelligence-controls-u2-spy-plane-air-force-exclusive/">operated the radar</a> on a US spy plane, language models now write <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.wired.com/story/ai-phishing-emails/">better scam emails</a> than humans, and self-driving car algorithms <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.theguardian.com/technology/2021/aug/16/teslas-autopilot-us-investigation-crashes-emergency-vehicles">outperform</a> humans in many environments.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/85d2d85f824fa6fb77d7140c8981aa8e5c72cc3cc773dbc224874f3f8916f76f.gif" alt="Hands free: A Tesla autonomously drives through California." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Hands free: A Tesla autonomously drives through California.</figcaption></figure><p>In the next decade, deep learning will become even more fundamental as jumps in model development become more frequent and also as hardware advances create greater sensory immersion. For example, Brain Machine Interfaces (BMI) use artificial intelligence to <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.biorxiv.org/content/10.1101/548081v1">decode</a> brain signals. BMIs, like <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://neuralink.com/applications/">Neuralink</a>, hold the near-term promise of allowing someone to access and browse the internet with their thoughts.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/705f0e1f6128033cf47cb9ada6fc241ca561cfb66bdacd63e60a179a8ece85bd.gif" alt="Brain Pong: Neuralink shows a real monkey playing Pong. (L) shows the monkey moving the joystick in its left hand. (R) shows the joystick unplugged - with the monkey still moving the joystick - but with control of the game happening entirely through the monkey’s thoughts via BMI." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Brain Pong: Neuralink shows a real monkey playing Pong. (L) shows the monkey moving the joystick in its left hand. (R) shows the joystick unplugged - with the monkey still moving the joystick - but with control of the game happening entirely through the monkey’s thoughts via BMI.</figcaption></figure><h2 id="h-current-deep-learning-progress-is-astounding-but-only-scratches-the-surface" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Current deep learning progress is astounding but only scratches the surface</h2><p>If you’re sceptical of grand visions for deep learning then you might have well-founded concerns. The space has been anthropomorphised and overhyped from the moment it entered the public eye.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/9d0f319332fe25ffed6c02e83f817ef5d66963b953b37245416e0f3f4d6b5ab7.png" alt="Bold claims: In a debut July 1958 NYT article we were told neural networks would become self aware." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Bold claims: In a debut July 1958 NYT article we were told neural networks would become self aware.</figcaption></figure><p>Yet in the past decade, some of the human-like promise of AI has indeed come to fruition. In 2016, AlphaGo, a neural network developed by DeepMind (an Alphabet subsidiary), <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://deepmind.com/research/case-studies/alphago-the-story-so-far">toppled Lee Sedol</a>, the world’s Go Champion. It accomplished this by analysing games played by Go masters and then playing itself to improve. Go is a game that is 10^100 times more complicated than chess.</p><p>Less well-known is the fact that the following year, a new version of AlphaGo learned to play Go without ever being shown games from the Go masters. Having never seen a real game of Go, it ended up beating the previous version that had beaten Sedol.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/9ae12f534412d4505b82c8848c2bbb0f2fa10fd99904ec92737fb1b617161e5e.png" alt="Lonely at the top: Lee Sedol battles AlphaGo, and loses (L). A year later, a new version which learns by playing itself (0 human guidance)" blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Lonely at the top: Lee Sedol battles AlphaGo, and loses (L). A year later, a new version which learns by playing itself (0 human guidance)</figcaption></figure><p>However, despite enormous gains in narrow areas like Go, the most basic and innate concepts of the human experience are the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.scientificamerican.com/article/will-machines-ever-become-conscious/">hardest</a> to replicate: self-awareness, morality, and ‘gut instinct’. This shortcoming was famously captured in Bladerunner’s Voight-Kampff test–loosely based on the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://plato.stanford.edu/entries/turing-test/">Turing Test</a>–in which AI replicants were asked a series of morally ambiguous questions to determine their humanity.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/d4ff6b5140c81028c82d4415730390e3ad8097b510d3663d16453b62f051ac33.png" alt="Do you like our owl? Rick Deckard tests the humanity of a suspected replicant in Bladerunner (1982)." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Do you like our owl? Rick Deckard tests the humanity of a suspected replicant in Bladerunner (1982).</figcaption></figure><p>For a system to convincingly answer these questions, it would probably need to satisfy a version of Artificial General Intelligence (AGI), aka ‘Strong AI’. An AGI is a <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.ibm.com/cloud/learn/strong-ai">system</a> that matches average human intelligence and has a sense of consciousness. Achieving this state is an area of passionate research and fundamental uncertainty (there is no universally accepted definition of consciousness, for example). Estimates for achieving AGI vary wildly; Bladerunner might have been set in 2019, but rough estimates for AGI creation span from <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.theverge.com/2018/11/27/18114362/ai-artificial-general-intelligence-when-achieved-martin-ford-book">2029 to 2220</a>!</p><h2 id="h-the-three-factors-that-get-us-to-agi-are-being-fiercely-centralised" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">The three factors that get us to AGI are being fiercely centralised</h2><p>If you’ve noticed that most of the above examples are produced by the same set of companies, that’s because the deep learning industry currently looks like a game of monopoly between Big Tech companies. At the state level, too, it often looks like a trade and talent war between China and the United States. These forces are resulting in huge centralisation of the key resources that get us to AGI: compute power, knowledge, and data.</p><p><strong>Compute power</strong>: access to superior processors enables increasingly large/complex models to be trained. In the past decade, transistor density gains and advances in memory access speed/parallelisation have dramatically reduced training times for large models. Virtual access to this hardware, via cloud giants like AWS and Alibaba, has simultaneously widened adoption.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/279a15258734038fb0f581af54274923836cf56637ab5c28c5ad63eb2f3031aa.png" alt="Look closely: Intel increased the density of transistors in 2011 with their 22nm Tri-Gate design. State-of-the-art transistor density is now \~5nm, reaching the physical limits of what can be built. For context, the width of a human hair is 100,000 nm." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Look closely: Intel increased the density of transistors in 2011 with their 22nm Tri-Gate design. State-of-the-art transistor density is now \~5nm, reaching the physical limits of what can be built. For context, the width of a human hair is 100,000 nm.</figcaption></figure><p>Accordingly, there is strong state interest in acquiring the means to produce state-of-the-art processors. Mainland China does not yet have the end-to-end capability to produce state-of-the-art semiconductors (namely, silicon wafers), an essential component in processors. They need to import these, particularly from TSMC (Taiwan Semiconductor Manufacturing Company). Chip vendors also attempt to block out other customers from accessing chip manufacturers <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.techradar.com/news/intel-locks-down-all-remaining-tsmc-3nm-production-capacity-boxing-out-amd-and-apple">by buying up supply</a>. At the state level, the US has been <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://foreignpolicy.com/2021/06/22/biden-semiconductors-south-korea-china-trade-war/">aggressively blocking</a> any move by Chinese companies to acquire this technology.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/4b131deabb745336b67fc667ee24ee52e2cb8b2dd1b5dc3a60b4b3ac3ee66d22.png" alt="Chip war: US-aligned countries dominate the sub-22nm chip production process." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Chip war: US-aligned countries dominate the sub-22nm chip production process.</figcaption></figure><p>Further up the tech stack, some companies have gone as far as creating their own deep learning specific hardware, like Google’s TPU clusters. These outperform standard GPUs at deep learning and aren’t available for sale, only for rent.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/066faf17a23b59b9146e0f183d37cb72c0764c46eff8fb3b39e16aefcc9b4228.png" alt="Pod race: A single Google TPU v3 pod has 100 petaFLOP performance. One billion times more than the best supercomputer of the 1960s." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Pod race: A single Google TPU v3 pod has 100 petaFLOP performance. One billion times more than the best supercomputer of the 1960s.</figcaption></figure><p><strong>Knowledge</strong>: many of the most public breakthroughs have stemmed from new model architectures developed by researchers, but there is a battle over the underlying IP and talent. The US has historically captured <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.nytimes.com/2020/06/09/technology/china-ai-research-education.html">over 50%</a> (!) of the talent emerging from China, and the companies that develop models with this talent are increasingly making the technology less accessible. GPT-3 by OpenAI was (as the name suggests) meant to be openly available. But, as of today, it controversially sits behind an API with only Microsoft having access to the source code.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/c8daa57c2d8c237e1c63a58dc04eef3b6b5b24a88d47eb2f8ba6c8c098828f8e.png" alt="Brain drain: historically, China has lost over 50% of its best AI researchers to the US." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Brain drain: historically, China has lost over 50% of its best AI researchers to the US.</figcaption></figure><p><strong>Data</strong>: deep learning models require huge volumes of data–both labelled and unlabeled–and generally improve as data quantity increases. GPT-3 was trained on 300 billion words. Labelled data is particularly important, and the industry has been steadily accruing it for years. A clandestine example: every time you solve a reCaptcha to access a website you are labelling training data <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.vox.com/22436832/captchas-getting-harder-ai-artificial-intelligence">to improve Google Maps</a>.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/dbfe48cd7468f638f27832a95e2d96f99afe3a186883c02eef314179ed63166f.png" alt="Big data: (L) you have likely labelled data for Google, (R) GPT-3 models make less mistakes as they are exposed to more words." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Big data: (L) you have likely labelled data for Google, (R) GPT-3 models make less mistakes as they are exposed to more words.</figcaption></figure><h2 id="h-decentralising-compute-improces-scale-and-access" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Decentralising compute improces scale and access</h2><p>The internet might have been born of the US Government in the 1960s, but by the 1990s it was an anarchic web of creativity, individualism, and opportunity. Well before Google was stockpiling TPUs, projects like SETI@home attempted to discover alien life by crowdsourcing decentralised compute power. By the year 2000, SETI@home had a processing rate of <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://setiathome.berkeley.edu/sah_papers/CISE.pdf">17 teraflops</a>, which is over double the performance of the best supercomputer at the time, the IBM ASCI White. This period of time is generally named ‘web1’, a moment before the hegemony of large platforms like Google or Amazon (web2).</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/8f3f5247a74316a74a061f57159a9b59913806d22c1fb1f0c0e18ff623116b1d.png" alt="Lonely planet: A SETI@home user 22 hours into an analysis of data from Puerto Rico’s old Arecibo Radio Observatory." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Lonely planet: A SETI@home user 22 hours into an analysis of data from Puerto Rico’s old Arecibo Radio Observatory.</figcaption></figure><p>The shift from web1 and the relative obscurity of SETI@home might have been due to some of the issues with decentralised infrastructure. For one, all of the data must be distributed, increasing bandwidth requirements. Equally important is how fast you can complete tasks that must be performed in sequence. For example, if you want to analyse the background radiation of the universe (and maybe find alien life) you can divide the sky into small parts and distribute it to everyone (the radiation in one part of the sky can be analysed independently of the other parts). This allows for perfect parallelisation of work; it also means that it’s relatively trivial to check if the work has been done correctly. After asking a third party to perform a computation, you could randomly select units of submitted work and check if they are correct.</p><p>In contrast, if you want to calculate the next best move in a game of chess (or Go), you must have access to the positions of all the pieces on the board. This makes it very difficult to efficiently decentralise the computation in chess and Go engines.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/2d7474ccdfda9bf3c76361826c123dbc5ab107e4e878c7e7fffa2ec81ffdf4ae.png" alt="“Easy” vs “Hard” problems: You can take any slice of the night sky and analyse it to determine results: this enables perfect parallelisation (L). However, you cannot reliably determine the next best move on a chessboard without seeing the whole board (R) " blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">“Easy” vs “Hard” problems: You can take any slice of the night sky and analyse it to determine results: this enables perfect parallelisation (L). However, you cannot reliably determine the next best move on a chessboard without seeing the whole board (R)</figcaption></figure><p>However, the current centralisation of web infrastructure into huge web2 platforms creates its own arguably larger issues:</p><ol><li><p><strong>Costs increase</strong>: AWS’ gross margin is an estimated <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.cnbc.com/2021/09/05/how-amazon-web-services-makes-money-estimated-margins-by-service.html">61%</a> - effectively a money printing licence.</p></li><li><p><strong>Control</strong>: AWS turned off the infrastructure of popular right-wing social media platform Parler with <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.cnbc.com/2021/01/09/amazon-drops-parler-from-its-web-hosting-service.html">one day’s notice</a> following the Jan 6th 2021 Capitol Riot. Many agreed with this decision, but the precedent is dangerous when AWS <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://kinsta.com/aws-market-share/">hosts 42%</a> of the top 10,000 sites on the internet. The propensity for government tyranny also increases, for instance, the CCP block Google (although it’s still –illegally–accessible via a bridge on the decentralised TOR network).</p></li></ol><p>There is a third way, however. Web3 can be thought of as a combination of the decentralised components of web1 and the capitalist components of web2. For example, decentralising compute with a blockchain and buying/selling processor cycles with tokens would circumvent the above issues with web2:</p><ol><li><p><strong>Low cost scale</strong>: Unlike web2 organisations, which extract value through margins, there are low (or no) margins in web3. Instead, as demand for the token increases (or supply deflates) the token price increases and investors realise a return. As long as the cost per processor cycle is dynamically priced, the value of the token increases without affecting the cost of the purchasing compute. This instantly removes the AWS-style margin applied on compute. Just as importantly, blockchains are a more scalable option as long as the blockchain has, at worst, a linear relationship between work verification time and work.</p></li><li><p><strong>Free access</strong>: For blockchains that are trustless, there is generally no oversight as to who can and cannot access the network. This circumvents state and private sector restrictions and enables users to build where they might not have otherwise been able to.</p></li></ol><h2 id="h-training-deep-learning-models-across-decentralised-hardware-is-difficult-due-to-the-verification-problem" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Training deep learning models across decentralised hardware is difficult due to the verification problem</h2><p>It’s clear that decentralising compute creates a cheaper and freer base from which to research and develop artificial intelligence. But the fundamental blocker to the decentralisation of deep learning training has been verification of work. Essentially, how do you know that another party has completed the computation that you requested? The two factors driving this blocker are:</p><p><strong>State dependency</strong>: neural networks are more like the chess board than the night sky. That’s because, generally, each layer in the network is connected to all the nodes in the layer before it. This means it requires the state of the previous layer (literally, ‘state dependent’). Worse still, all the weights in every layer are determined by the previous time step. So if you want to verify that someone has trained a model–say, by picking a random point in the network and seeing if you get the same state--you need to train the model all the way up to that point, which is very computationally expensive.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/6b1d06696a3308314fdff029ee27d77dfab35883668b5f971a8c43fabf4550d1.gif" alt="Learning to read: a simple neural network uses the MNIST handwriting dataset to learn the relationship between handwritten numbers and their true values. Each layer has nodes and each node is connected to all the nodes in the previous layer, creating state dependence." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Learning to read: a simple neural network uses the MNIST handwriting dataset to learn the relationship between handwritten numbers and their true values. Each layer has nodes and each node is connected to all the nodes in the previous layer, creating state dependence.</figcaption></figure><p><strong>High computational expense</strong>: It cost <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/eturner303/status/1266264358771757057/photo/1">~$12m</a> in 2020 for a single training run of GPT-3, &gt;270x more than the estimated <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.reddit.com/r/MachineLearning/comments/aqlzde/r_openai_better_language_models_and_their/eghxt8n/">~$43k</a> for the 2019 training of GPT-2. In general, model complexity (size) of the best neural networks is currently doubling every <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.nature.com/articles/d41586-021-00530-0">three months</a>. If neural networks were less expensive, and/or if the training represented less of the model development process, then perhaps the verification overhead stemming from state dependency would be acceptable.</p><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/58d472f7d563cce9ff76c1c1b0f05bd34e4773d50a5c53076e2884ef8c12048c.png" alt="Hold on tight: state-of-the-art neural networks are growing in size at an exponential rate." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">Hold on tight: state-of-the-art neural networks are growing in size at an exponential rate.</figcaption></figure><p>If we want to lower the price, and decentralise the control, of deep learning training, we need a system that trustlessly manages state dependent verification whilst also being inexpensive in terms of overhead and rewarding to those who contribute compute.</p><h2 id="h-gensyn-is-the-future-of-deep-learning-training" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Gensyn is the future of deep learning training</h2><p>The Gensyn protocol trustlessly trains neural networks at hyperscale and low cost. The protocol achieves the lower prices and higher scale by combining two things:</p><ol><li><p><strong>Novel verification system</strong>: a verification system which efficiently solves the state dependency problem in neural network training at any scale. The system combines model training checkpoints with probabilistic checks that terminate on-chain. It does all of this trustlessly and the overhead scales linearly with model size (keeping verification costs constant). If you’d like to go deeper into how this is technically possible, read our <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://docs.gensyn.ai/litepaper">Litepaper</a>.</p></li><li><p><strong>New supply</strong>: leveraging underutilised and underused/unoptimised compute sources. These range from presently unused gaming GPUs to sophisticated Eth1 mining pools about to detach from the Ethereum network. Better still, the protocol’s decentralised nature means it will ultimately be majority community governed and cannot be ‘turned off’ without community consent; this makes it censorship resistant, unlike its web2 counterparts.</p></li></ol><figure float="none" data-type="figure" class="img-center" style="max-width: null;"><img src="https://img.paragraph.com/cdn-cgi/image/format=auto/https://storage.googleapis.com/papyrus_images/1bdda0beeb6e6f975e651cd996cfb356c9dc38de0fea7830344facf3bfb24ab0.png" alt="High scale + low cost: the Gensyn protocol provides a cost similar to an owned GPU in a data centre at a scale which can surpass AWS. (Prices as at Nov 2021)." blurdataurl="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=" nextheight="600" nextwidth="800" class="image-node embed"><figcaption HTMLAttributes="[object Object]" class="">High scale + low cost: the Gensyn protocol provides a cost similar to an owned GPU in a data centre at a scale which can surpass AWS. (Prices as at Nov 2021).</figcaption></figure><h2 id="h-deep-learnings-cambrian-explosion" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Deep learning’s Cambrian Explosion</h2><p>Vastly increasing the scale of accessible compute, whilst simultaneously reducing its unit cost, opens the door to a completely new paradigm for deep learning for both research and industrial communities.</p><p>Improvements in scale and cost allow the protocol to build up a set of already-proven, pre-trained, base models–also known as <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://arxiv.org/abs/2108.07258">Foundation Models</a>–in a similar way to the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://pytorch.org/docs/stable/model_zoo.html#torch-utils-model-zoo">model zoos</a> of popular frameworks. This allows researchers and engineers to openly research and train superior models over huge open datasets, in a similar fashion to the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://www.eleuther.ai/">Eleuther</a> project. These models will solve some of humanity’s fundamental problems without centralised ownership or censorship.</p><p>Cryptography, particularly Functional Encryption, will allow the protocol to be leveraged over private data on-demand. Huge foundation models can then be fine-tuned by anyone using a proprietary dataset, maintaining the value/privacy in that data but still sharing collective knowledge in model design and research.</p><h2 id="h-join-us-at-the-frontiers-of-artificial-intelligence" class="text-3xl font-header !mt-8 !mb-4 first:!mt-0 first:!mb-0">Join us at the frontiers of artificial intelligence</h2><p>With the Gensyn protocol, we finally have hyperscale, cost-efficient, deep learning training that isn’t marshalled by institutions who play god with who gets access. The first version of the protocol, our testnet, will be deployed later this year.</p><p>If you’d like to use Gensyn to train models, get a yield on your GPU, or solve some hard engineering problems: join our <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://discord.com/invite/bCHyMQu4Qn">Discord</a>, join our team, or follow us on <a target="_blank" rel="noopener noreferrer nofollow ugc" class="dont-break-out" href="https://twitter.com/gensynai">Twitter</a>!</p><p>Finally, thank you to everyone in the Gensyn community who helped with this piece!</p>]]></content:encoded>
            <author>gensyn@newsletter.paragraph.com (Gensyn)</author>
        </item>
    </channel>
</rss>