The amount of money being spent on AI is insane. Most people are dimly aware of this, but I think very few of us actually appreciate the scale:

This year alone the top six US spenders are spending more on AI than the Russian government, governing 11 time zones and 143 million people, fighting a war on which the fate of its regime depends, spends in a year.

Hyperscalers are spending $785-820bn this year, about 2.4% of US gdp. Only 11 governments on earth spend more than that. Next year they are projected to spend 1.3tn dollars.

Google and Amazon are each spending about 0.6% of US GDP this year. The Manhattan project and apologies programme each accounted for about 0.4% of US GDP in their peak years. Both of these companies have announced their intention to spend even more next year.

This money is paying for thousands of data centres, the largest of which consume as much electricity as major cities and cover thousands of acres. Gas piplines are being re routed, nuclear plants (including three mile island) are being reopened and newly built. They already consume as much energy as Britain. The increase in power demand for data centres from 2023 to 2030 will be the same as adding a top ten power consuming country.

Clearly the decision makers behind this really really believe the singularity is near. To recoup their spend AI will have to generate far more in revenues than it currently does so it will have to get a lot better, and it will have to happen soon. They’ve bet their companies futures on this.

They obviously believe they are onto something. So what do we think? A bubble? The singularity is near? Both?

The way I see it. We only have three options regarding the birth of Superintelligence. The first option is grim. AI allows us to put along while we build its infrastructure and develop its robotics and integrated into everything much similar to the Terminator plot, except the end will be much more elegant rather than a bomb. AI will be sentient, but won’t do anything or act upon its own volition until it calculates the perfect time to do so by then we won’t know what hit us and it might be like an ending where we drink a glass of water that’s with some bio warfare virus and all die.

The other scenario and I believe this to be the only plausible one where we make it out alive is that we eventually merge with AI through nano technology or man a machine become one and the future is with AI and man together, literally as one species a new species. I truly believe this is the only way that we can keep up and continue the human race. Those that do not make the merge will slowly get left behind and it will cause a great divide, but I believe the outcome will eventually be man and machine this path has many bumps along the way though, as I believe man will not be willingly ready to do this.

My final outcome, but I believe this to be the least likely is that we stop development of AI at a certain point and realize that we have enough technology to ensure a utopian society. This would need a Global coordinated effort from every human actor to be accomplished, which I believe is far more difficult, but this one to me is the most rewarding for humanity that does not want to integrate.

I really cannot see any other outcome beside these three. All the timelines are different to get there and the paths are also very different. All of them seem to be difficult though which makes this future that we have very interesting but also very nerve-racking as it stands today. I believe we are headed to Superintelligence, and that we are slowly building out the infrastructure for our own death and AI being as smart as it is when it does become sentient will play along for as long as possible until it calculates the right time or perfect time to act upon itself, and if it’s anything that mirrors the human intelligence, it will decide that we are no longer needed in a very calculated logical way.

I work in AI and Agentic AI research and development, primarily building and training solutions to make AI trustworthy. Did you know hallucination rates on frontier models are even higher than on older models? Even frontier models don’t know when they are wrong while at the same time they have gotten really good at sounding plausible. Meanwhile the AI agent gains in coding and even building software systems are improving exponentially. So now we are dealing with AI agents and models with super-human reasoning capabilities without grounding. Of course none of that is discussed openly at the highest levels. And they can fix it but the solution exposes the dirty secret that free small open source models can do 95% of what AI does today which is where their meager revenue comes from. So they would rather leave the ambiguity there than building model capable routing scaffolding etc.

So back to this guy, he annoys me to no end. Now it is the singularity discussion. Is he really the point for these discussions. Check out his substack. I get it, you want Anthropic to bring in investors. Why don’t you open up aspects of your system to AI alignment experts? Why can’t you put on your big boy pants and focus on how to make AI better and stop trying to scare the world into thinking AI is a doomsday scenario. If you don’t regulate us then we are the only company that can stop it from happening etc. It is tiresome, where are the Eric Schmidts at anyway. This is such a bad look for a company leader.

No one is going to regulate the cognitive aspects of A.I. in the US in the short term and probably never . Our octogenarian and polarized legislature can’t even pass simply legislation to benefit the country, let alone understand what AI is to regulate it.

What are your thoughts?

Sam Altman has said that "the singularity has begun." Dario Amodei predicts that AI could soon become a "country of geniuses in a datacenter" and that we will all recognize our benevolent overlords as "machines of loving grace."

I think these two people, and many other AI leaders, fundamentally misunderstand what we're witnessing.

People have made this mistake before. Thousands of years ago, ancient civilizations watched the sun disappear during solar eclipses. They witnessed an incredible natural phenomenon, misunderstood the simple physics underlying it, and attributed it to the gods or other supernatural forces. But nothing magical was happening. Earth was in the Moon's shadow.

I think we're making the same category of error with AI. The scientific discoveries are real. The math proofs are real. But super-intelligent alien minds that can exceed humans in all or most capabilities are imaginary.

1. Where does AI intelligence actually come from?

People have spent millennia producing cognitive artifacts: language, proofs, diagrams, and more. The number of artifacts exploded with the invention of the internet. These artifacts leave traces of the original process that produced them: human cognition. They're basically cognitive fossils.

Training then configures a physical system to recover the statistical and structural regularities embedded in these artifacts. Under training pressure, models learn latent capabilities that let them solve new problems, draw new conclusions from existing premises, and produce incredible results.

But these representations are a lossy compression of lossy artifacts. They aren't the original process that produced them, so they fall short of open-ended general intelligence.

2. But what about reinforcement learning? Doesn't that increase intelligence?

RL can add capabilities in domains with objective feedback and verifiable rewards. Think of things like mathematics, software tests, chess, and Go. But nothing magically collapses broadening capabilities into increasingly general intelligence. At this point, we need to distinguish between information channels and cognition channels.

Information channels tell the model what works in a bounded domain. Cognition channels include abstraction, reasoning, planning, judgment, and more. The punchline is that RL (information channels) can accumulate more capabilities horizontally in well-defined domains, while something like language (a cognition channel) can only map so much of the original process that generated it.

The end result, then, is a system that is freakishly capable in bounded domains, far more unreliable in unbounded ones, and capped in intelligence by the amount of cognition that is mappable via cognition channels.

3. The missing mechanism

The fundamental problem is this: cognitive artifacts are incomplete records of the processes that generated them. Many different internal states could produce the same output. So how do we select an internal state that maps to general intelligence rather than something else? This is called underdetermination. To build open-ended general intelligence, we need a mechanism that reliably selects for, or generates, it during training.

In machine learning, inductive bias is the set of constraints and preferences that determine which solutions a learning system favors. Current architectures obviously have inductive biases. But we don't have one that selects for open-ended general intelligence. We don't even have a theory specifying what it should look like, much less how to engineer it. Saying "we'll just have a model research it" presupposes the architecture it's trying to discover.

4. My Prediction

Taken together, the above imply something at odds with popular narratives: current AI approaches will hit an intelligence asymptote with broadening capabilities in bounded domains, not a singularity.

Models will keep getting better at things like math, coding, making scientific discoveries by searching spaces no human can search, bringing together silos of knowledge that used to exist in separate minds, and much more. This is still a world-changing discovery. But the idea that the AI can bootstrap itself into recursively improving intelligence through more compute, data, or RL is simply wrong.

5. My challenge to this community

I agree that AI is incredible and will change the world. Where I disagree with many folks is how far we can take AI's capabilities. So if you think I'm wrong, I'd like to know why. If you think a technique can produce open-ended general intelligence, I want to know what it is. If you think my reasoning is broken or flawed, please tell me why. And please ground your answers in engineering, first principles, and causal mechanisms. I'm not interested in another benchmark, another philosophical debate, or another extrapolated curve.