AGI, or Artificial General Intelligence, refers to AI that can understand, learn, and apply knowledge across a wide range of tasks at a level equal to or beyond human capabilities. Unlike narrow AI, which is limited to specific tasks, AGI can reason, solve problems, and adapt to new situations. AGI is a theoretical concept that has not yet been achieved.

This is the condensed definition of AGI by Le Chat from Mistral.

Hard to achieve, and yet, closer than ever. We’re reaching a state where the line between what a machine can or cannot do is blurrier than ever.

I have to admit I’ve been “tunneling” on the AGI topic recently. The reason is that it will have many implications that are discussed at length in the Situation Awareness website written by Leopold Aschenbrenner, with whom I share many thoughts. I don’t think we’ve seen anything like this before in human history.

Intelligence explosion

Agents are made of code and agents CAN code. So they'll soon enhance their code by themselves, each time increasing their capacity to... enhance themselves :). The effect will compound until we reach a point where we can’t even access and comprehend what they're doing, thinking, or discovering.

Think like an ant near a human. It has no clue about humans, what they do, what they have discovered, why we find this funny, etc.

Humans will be left static on the left side of the curve while AI agents will get exponentially cleverer and more capable.

This is what is called the intelligence explosion: we’ll get more and more intelligent agents at an increasing speed.

Some argue that we'll face challenges that could block or significantly delay AGI development. For instance, LLMs heavily rely on data—having the capacity to process vast amounts of data was one of the most significant breakthroughs in their development. Critics suggest we'll hit a "data wall" since most available internet data is already being used. However, I believe this is merely a solvable engineering and scientific problem. Ironically, AI might be able to help scientists overcome it: imagine agents are good enough to speed up research on the data wall topic by 20%. It means that researchers will be able to make discoveries faster that can feed the models that make agents even better, that speed up the research process, and… here we go, we’re compounding again.

For the purpose of this essay, let's assume that the data wall challenge will be overcome.

Power

Whoever controls this tech will access a power that was not accessible before, whether it’s a human, a country, or… nobody.

Having access to AGI gives the capacity to solve extremely complex challenges and do it exponentially faster. One could effectively create a swarm of millions of Einstein-like researchers working together to solve it.

This can give the power to solve fascinating engineering and research problems. There’s all the good that these researchers could do: build fusion reactor providing safe, cheap, and abundant energy, curing all types of diseases, … but, there’s also all the bad ones: non-detectable biological weapons or swarms of micro-robots killing only short people like me, psychological influence forcing all humanity to play Candy Crush, etc. And in both cases, we actually lack the brainpower to comprehend the magnitude of their abilities.

I can’t even mention the things that I’m too “ant” to comprehend. Have you seen an ant enjoying Mozart’s Requiem or coming up with the idea of wheels for luggage?

As progress will be exponential a slight advance is enough to make a drastic difference between the first and second. The one not having this technology will be left powerless against it.

Alignment

The above is in the case where humans are still in control, but there’s also a very high likelihood of it getting out of hands: if we don’t understand it, how could we even control it?

In the example below, you might think it’s ok to save the child and kill the car occupants, the AI might think otherwise. We don’t care about which answer is right, we only care that the AI behavior is aligned with what a human would do. Hence, the vast topic of alignment.

Alignment raises deep philosophical questions to answer: Are there common and global human values that could/should be enshrined in a model? How can we dissect between bias and truth? What decision should be made during an ethical dilemma? If an AI were to achieve consciousness or sentience, what ethical considerations would arise? Would such an AI have rights, and how would we balance those against human interests?

Right now, to assess whether a model is safe, we use many different methods, among them Reinforcement Learning from Human Feedback (RLHF). RLHF works by having humans evaluate and rate the AI's outputs - good responses are rewarded while wrong ones are discouraged. Through this feedback loop, the AI gradually learns to act in accordance with human values and preferences.

This method has obvious limits when the answers become too complicated for a human to even understand as AI progresses goes further.

Some argue that AI is closer to oil than a sort of god in the sense that it’s an enabler for humans to create new things, but it lacks the motivation, curiosity, and independent agency that are proper for humans. I agree with it in the short term: AI and humans are already complementary in many domains like medicine, and it will create new opportunities. However, there’s no known physical constraint that prevents an AI from showing curiosity or developing ulterior motives.

Whatever is right, given the pace of AI advancement, it’s critical to dedicate resources NOW before it's too late and AI is so smart that we can't counterattack. We wouldn’t like to have the bots proposing the eradication of the human race to solve climate change, transforming the whole universe into paperclips, or suggesting we shoot Fast and Furious XI (we really wouldn't).

Time is our enemy

As you can see the consequences are existential. But how far are we from this?

Well, AI models are already outperforming humans on many tasks, and this number is increasing very fast.

But the crux of the problem for me is not about whether the model is better than us on everything. It’s not really AGI. It just needs to be very good at one single specific thing: improving itself.

Once we reach this state, we could “just” ask the model to improve itself and give it the necessary resources to do it. It requires the capacity to conduct research and code and the access to infrastructure to run the models.

For the coding part, models are almost on par with humans. For the research side, we’ve already reached a state where LLMs are on par with what a research intern can do (see OpenAI deep research).

For the resource, it seems that trillions might be invested in the coming years into AI and its infrastructure: OpenAI announced the $500B Stargate Project, Meta announced a $200B data center, Amazon plans to invest $100B in AI this year, etc. We’re not even in “war-mode”, and yet, investments are already staggering.

Once this self-improvement point is reached, we’ll have a very, very short window to solve the alignment problem while the race is raging. Philosophical or ethical questions that could hinder progress might be easily put aside.

The race has already started. It’s a race for who will decide the future of humanity. It’s not a hyperbole. I truly think this is what is at stake. The first one to reach the self-improvement zone will get access to unlimited power very rapidly, which will make nuclear weapons irrelevant. If a country gets superintelligence, it could turn off the arsenal of its opponents at a distance without them noticing. I hope that those building it will have good intentions toward humanity.

Hence, we need to put the highest security into the making of those models. It’s a topic of security defense. Anyone having access to the weights of a given model can reuse it to do any action. Industrial espionage has been there for centuries, there’s no reason why AI labs would be exempt especially when governments realize the strategic reach of AGI. So, AI labs will face highly sophisticated state-sponsored attacks. Some spying agents may have already infiltrated them (I’m talking about real humans this time). This means that training centers, scientists, and researchers willing to work in this field would need to comply

If we’re serious about it, AGI should have its own Manhattan project, i.e., a consortium of trusted parties (with stable people at their head —not hinting on anything here ofc—) focusing resources to solve the alignment problem and get AGI as fast as possible before any authoritarian country or malicious group gets it. It seems monstrously hard, but the atomic bomb was conceived in just 3 years, with 130,000 people working on it at its peak. Problems are solvable.

Who’s assembling this group?