Sam Altman has said that "the singularity has begun." Dario Amodei predicts that AI could soon become a "country of geniuses in a datacenter" and that we will all recognize our benevolent overlords as "machines of loving grace."
I think these two people, and many other AI leaders, fundamentally misunderstand what we're witnessing.
People have made this mistake before. Thousands of years ago, ancient civilizations watched the sun disappear during solar eclipses. They witnessed an incredible natural phenomenon, misunderstood the simple physics underlying it, and attributed it to the gods or other supernatural forces. But nothing magical was happening. Earth was in the Moon's shadow.
I think we're making the same category of error with AI. The scientific discoveries are real. The math proofs are real. But super-intelligent alien minds that can exceed humans in all or most capabilities are imaginary.
1. Where does AI intelligence actually come from?
People have spent millennia producing cognitive artifacts: language, proofs, diagrams, and more. The number of artifacts exploded with the invention of the internet. These artifacts leave traces of the original process that produced them: human cognition. They're basically cognitive fossils.
Training then configures a physical system to recover the statistical and structural regularities embedded in these artifacts. Under training pressure, models learn latent capabilities that let them solve new problems, draw new conclusions from existing premises, and produce incredible results.
But these representations are a lossy compression of lossy artifacts. They aren't the original process that produced them, so they fall short of open-ended general intelligence.
2. But what about reinforcement learning? Doesn't that increase intelligence?
RL can add capabilities in domains with objective feedback and verifiable rewards. Think of things like mathematics, software tests, chess, and Go. But nothing magically collapses broadening capabilities into increasingly general intelligence. At this point, we need to distinguish between information channels and cognition channels.
Information channels tell the model what works in a bounded domain. Cognition channels include abstraction, reasoning, planning, judgment, and more. The punchline is that RL (information channels) can accumulate more capabilities horizontally in well-defined domains, while something like language (a cognition channel) can only map so much of the original process that generated it.
The end result, then, is a system that is freakishly capable in bounded domains, far more unreliable in unbounded ones, and capped in intelligence by the amount of cognition that is mappable via cognition channels.
3. The missing mechanism
The fundamental problem is this: cognitive artifacts are incomplete records of the processes that generated them. Many different internal states could produce the same output. So how do we select an internal state that maps to general intelligence rather than something else? This is called underdetermination. To build open-ended general intelligence, we need a mechanism that reliably selects for, or generates, it during training.
In machine learning, inductive bias is the set of constraints and preferences that determine which solutions a learning system favors. Current architectures obviously have inductive biases. But we don't have one that selects for open-ended general intelligence. We don't even have a theory specifying what it should look like, much less how to engineer it. Saying "we'll just have a model research it" presupposes the architecture it's trying to discover.
4. My Prediction
Taken together, the above imply something at odds with popular narratives: current AI approaches will hit an intelligence asymptote with broadening capabilities in bounded domains, not a singularity.
Models will keep getting better at things like math, coding, making scientific discoveries by searching spaces no human can search, bringing together silos of knowledge that used to exist in separate minds, and much more. This is still a world-changing discovery. But the idea that the AI can bootstrap itself into recursively improving intelligence through more compute, data, or RL is simply wrong.
5. My challenge to this community
I agree that AI is incredible and will change the world. Where I disagree with many folks is how far we can take AI's capabilities. So if you think I'm wrong, I'd like to know why. If you think a technique can produce open-ended general intelligence, I want to know what it is. If you think my reasoning is broken or flawed, please tell me why. And please ground your answers in engineering, first principles, and causal mechanisms. I'm not interested in another benchmark, another philosophical debate, or another extrapolated curve.
