I first started reading about decentralized training when it first popped up on my radar with Gensyn's seed round announcement in March 2022, so it's nice to write this as a reflection on how my thinking towards and understanding of AI development and consumer appetite for these tools writ large has evolved over the past two years.

Each time you query ChatGPT/Claude/Perplexity, you leverage a foundation model that delivers a response via a user interface. Your query is processed (infrastructure routes, authenticates, scales, monitors requests etc) and the system performs inference to generate a response. Foundation models generate responses by using learned patterns and relationships to predict the next item(s) in a sequence—responses are built one token at a time. This process continues until the system determines that a sufficiently complete answer has been generated.

Now that you understand the role of foundation models, the inevitable next question is how are these foundation models built?

Foundation model training process begins with data ingestion (raw data is collected from databases, files, APIs etc) from across the Internet. This data is then preprocessed to eliminate bad data.

Side note: Data is sometimes thought of as the bottleneck for improvements in foundation models but its high quality data that is actually the bottleneck, you achieve less noise and more signal that better inform the actual value created from these systems—namely, human decisions. Dive into synthetic data if you’re curious to learn more. Mary spent too much time in college processing and normalizing data for internships.

The next step is selection of some architecture that determines how the model will process information and learn patterns from the training data. This stage is intended to identify the most informative attributes for the learning algorithm and allows inherent structures in data types (natural language, images etc) to be represented by specialized techniques employed.

You may have heard of the paper Attention is All You Need. The paper proposed a deep learning model architecture comprised of encoders and decoders called the Transformer that is based on attention mechanisms (it enables the model to focus on dynamic associations between different positions, thus better capturing long-distance interdependent features in a sentence).

Pre-training is computationally intensive and requires multiple parallel GPUs and meaningful energy resources. The process itself can take weeks or months depending on model and size and available computing infrastructure. This scale of computation is considered one of the most significant barriers to foundation model development and previously only well-resourced organizations are able to train models from scratch (this stage is the focus of the article today). In this stage, the model learns patterns from a dataset through self-supervised learning. The model generates its own learning objectives from the data rather than requiring labeled examples—for example, predicting words in masked text or reconstructing corrupted images. 

As more and more major technology giants dedicate resources to develop foundation model systems, parameter size continues to grow (increased parameter count offers greater potential for complex tasks and better outputs) and this increase results in significant demand for computation and storage which consequently creates pressure on hardware resources and computational efficiency. Training these models takes a long time and an efficient utilization of computational resources. 

This necessary increase in workload has become a challenge to circumvent given it leads to issues for model serving systems (latency, performance decline, resource bottlenecks etc). Model training and serving are both being explored—decentralized training represents one such way teams are approaching scaling model training. 

Pre-training establishes a general understanding patterns in the data. Fine-tuning adapts foundation models for specific tasks or domains by using smaller domain-specific datasets to refine the model’s capabilities for particular applications. This process enables transfer learning in which knowledge gained during pre-training is applied to new, related tasks. 

Implementation prepares models for consumer use and involves optimizing model for deployment (considerations around latency, throughput, and resource efficiency). 

I’m including this section because I think it provides some pattern recognition and real world understanding on how different teams are attempting to build better foundation models while making all aspects of the development pipeline more efficient and more importantly, cheaper. DeepSeek was able to achieve competitive performance with its foundation models while using substantially less compute resources. 

The DeepSeek team generated training data that could be automatically verified and focused on deterministic domains like mathematics where correctness is certain and unambiguous. Prioritizing high quality verifiable data set DeepSeek apart from the rest of the pack (traditional approach is inhaling the expanse of the Internet). 

DeepSeek used a mixture of experts (MoE) architecture design that contains specialized neural network components that activate selectively based on specific input, in contrast to traditional transformer architectures that activate all parameters for each input. Thus, only a fraction of the model’s parameters actively compute, meaning a significant reduction in compute requirements.

DeepSeek developed a sophisticated reward mechanism that identified training examples that provided the most value to the model’s performance, which allowed them to be extremely selective on how they allocated compute resources. This meant wasting compute on redundant data that wouldn’t meaningfully improve the model.

All of these improvements were borne out of need given restrictions in compute resources that the US has imposed on private companies like NVIDIA in supplying Chinese companies. Despite all of these improvements that sidestep raw compute (which is essentially the approach that Western tech companies are taking right now), DeepSeek’s founder has acknowledged that they still require more computing power. 

This should give you a good understanding of how foundation models are built and a good start to understanding the next piece that dives into decentralized training that I will finish up tomorrow--I'm in rural Chile where wifi is sparse (this Internet cafe closes in an hour).

A Primer on Compute Carnegie Endowment

Attention is All You Need Various Authors

Training and Serving System of Foundation Models Various Authors

i travel a lot for work and last year it was so atrocious that i was in new york for maybe 2 weeks at a time then gone for 3-4 weeks. i'd do my best to spend time with my new york based friends and when i returned, i felt that no time at all had passed. but to them, i was someone who was more and more infrequently a part of their lives (this kills me and something i aim to fix this year).

they were stationary observers (by nature of staying in new york) and moved only moved on one dimension (time). but myself--constantly in a new city, a new country, then returning to new york--i felt that no time had passed and i had only moved through space. to return and find that a couple whose wedding i was a bridesmaid no longer considered me a close friend but to me nothing had changed (when i enter the space that is new york, the clockpiece starts again)...brutal.

anyways, this set me down a rabbit hole because i thought a lot about how development of time keeping pieces has really only accelerated in the past century and how that has impacted our collective understanding of the markets. terence @ hyperplex (an actual genius, one of the smartest guys i've met in my lifetime) inspired this piece because he talked about how monad's one second block times actually lead to more efficient markets (given the unnecessary friction in battling over micro and milli seconds).

have you ever really sat down and thought about 1) what time is and 2) how our understanding and measurement of time controls our individual mental state (i'll start on monday) and financial system (fiscal year)? why do we measure months on four sets of seven day intervals? gregorian calendar system aside, our ability to agree on a system and measure time precisely has so much impact on financial instruments (trade execution and settlement, algo trading but more broadly stuff like insurance and futures). when i worked out in rural thailand, no one had a timekeeping piece on them--friends would tell me to meet by the road 'a little after lunch' and what that translated to was maybe one hour, maybe two hours, maybe we'll do it tomorrow. i loved watching old folks just sit by the road and plastic chairs talking to each other or playing chess for hours, not an anxious care for what time it was in their hearts. in places like that, time drips by like molasses.

five moments from the history of timekeeping

• 3500 BC: egyptian obelisks used to cast a shadow to indicate time

• 1000s: hourglasses for a greater precision in time measurement used at sea (you can imagine how soulless it must be to drift in open ocean for months losing all sense of time) and large town clocks begin to appear (communal agreement of time)

• 1884: adoption of the greenwich mean time, global standardization of timekeeping

• 1949: first atomic clock is created and measures time using ammonia molecule vibrations

• 1999: the cesium fountain clock uses laser cooling techniques to slow cesium atoms, which allows for a longer observation time

ok so what does this have to do with the efficient market hypothesis (EMH)? EMH states that asset prices reflect all public and private information so consistent alpha generation is impossible, hence investors must take on outsized risk in order to outperform the overall market.

this is obviously not true ubiquitously, and especially so in tech. folks that are acclimated to value investing in traditional finance simply do not stand a chance in traditional tech (where narrative bubbles occur but much more slowly than they do in crypto) and especially so in crypto, where an outsized amount of token demand comes from an extremely fickle and easily fascinated pool of capital.

what i find most fascinating about crypto is you can actually see the market learning and adjusting to information in real time and the collective hivemind slowly inches forward. commerce, markets, our global interconnected financial system depend so much on these critical moments (consensus on timekeeping systems, adoption of double entry bookkeeping) for facilitation of the dissemination of information.

i think EMH will always fail to apply to crypto because there is no single town timekeeping device much less a greenwhich mean time that we all agree on (ive met the china cabal, the SE Asia cabal, the europe cabal, middle eastern cabal, american cabal, and the nyc cabal which is different from american cabal). cabals maintain their own timekeeping systems. repricing of assets occurs largely because of social signals that fit some broadly accepted narrative. there are some crypto investors that describe themselves as fundamentals based investors but i largely think this is marketing for exit liquidity (everything is marketing) and their frameworks on how to appropriately value ethereum and solana are relics of their time slavishly building discounted cash flows in tradfi.

anyways, here's a little takeaway. its important to stay situationally aware of trad tech developments but even more so in crypto. some folks last year swore to me their [good narrative predatory] coins would hit top ten market cap but pump.fun and daos.fun completely changed the game (for better, in my opinion).

the current town i reside in is me and the boys are gonna make [good products people actually use] flip [shit that no one uses] this cycle. top 100 market cap crypto coins will no longer embarrass me to outsiders looking in, wondering why a ghost asset is worth x billions of dollars.