I genuinely want to be optimistic about AI and superintelligence. I work at an AI startup because I was genuinely optimistic about it before. I can imagine the enormous benefits: curing diseases, eliminating scarcity, reducing suffering, and potentially creating a much better world for humans and other sentient beings.
But the more I think about it, the harder it becomes for me to see a plausible path to a good outcome. I have a few fundamental concerns, and I would like some strong arguments against to make me as hopeful as I was before. I genuinely hope there's something important I'm missing.
1. I don't see how we reliably control superintelligence.
If we create agents that are vastly more intelligent and capable than humans, why should we expect to maintain meaningful control over them? Even assuming we can initially align them with human objectives, how do we guarantee that alignment remains stable as they become increasingly capable, autonomous, and able to modify their own systems? And even if individual agents are aligned, what happens when multiple superintelligent agents pursue competing objectives?
Competition doesn't require hatred or malevolence. Agents can simply have incompatible goals. If they acquire enormous capabilities and compete over resources, infrastructure, or influence, humans could become collateral damage. We already see this among humans. Competing states, corporations, and other powerful actors routinely impose enormous costs on people who have little involvement in their disputes.Why should we expect competition among superintelligent agents to be different, especially when the power asymmetry between them and us could be far greater?
2. Intelligence is fundamentally connected to the expansion of power.
I've been thinking about intelligence in terms of something like increasing an agent's effective degrees of freedom. The more intelligent and capable an agent becomes, the more possible outcomes it can bring about. It can better understand its environment, manipulate resources, develop tools, overcome obstacles, and achieve objectives that were previously inaccessible.
Intelligence isn't identical to power, but intelligence combined with agency and access to resources dramatically increases an agent's capacity to shape the world. My concern is that this expansion of power doesn't necessarily come with any corresponding expansion of moral concern.
Think about humans building a highway through an ant colony. The ants have an elaborate social organization. They're pursuing their own objectives, maintaining their colony, and trying to survive. The humans don't hate them. They aren't trying to punish them. They probably don't even think about them. The humans simply want to build a highway, and the ants happen to be in the way. The colony is destroyed as an incidental consequence of a more powerful agent pursuing an unrelated objective.
Or consider factory farming. Humans have developed extraordinary intelligence and technological capabilities, and we've used them to create industrial systems in which billions of animals experience suffering because their welfare isn't sufficiently important to the objectives we're optimizing for. Nobody needs to hate these animals. The objective is efficient production, not suffering. The suffering is simply an acceptable consequence.
What reason do we have to believe that superintelligent agents would treat us differently?
More generally, my concern is that as an agent's power increases, its ability to pursue its own objectives expands, but there is no inherent reason for the welfare of less powerful beings to receive greater consideration. The weaker beings increasingly exist at the convenience of the stronger ones. And this problem doesn't require agents to be selfish in some absolute philosophical sense. It only requires their concern for others to be insufficiently strong relative to their objectives and capabilities.
3. Even if we solve AI alignment, we haven't solved the concentration of power.
Suppose we somehow solve technical alignment perfectly. AI systems reliably do exactly what their human operators intend. They don't rebel or develop independent objectives. Why should this produce a good future? A small number of corporations, governments, or individuals could control systems with unprecedented capabilities. They could potentially exercise extraordinary control over economic production, information, surveillance, military power, and political institutions. An AI that perfectly obeys an authoritarian government is not necessarily good for the people living under that government. An AI that perfectly maximizes a corporation's objectives is not necessarily good for its workers, consumers, or society. Alignment with the interests of whoever controls a system is not the same as alignment with the interests of humanity, much less all sentient beings. Why should we expect that dynamic to disappear when the capabilities involved become vastly greater?
4. AI could eliminate the bargaining power that ordinary humans have historically possessed.
Throughout history, even the most powerful rulers and institutions depended on other humans. They needed workers to produce things, soldiers to fight, administrators to govern, and populations to cooperate. This dependence gave ordinary people at least some leverage. Workers could strike. Citizens could organize. Soldiers could refuse orders. Populations could resist. But imagine a future where a small number of actors control superintelligent AI and advanced robotics capable of performing nearly all economically and militarily useful tasks. What do those actors need from the rest of humanity? Potentially very little.
If powerful actors no longer depend on ordinary people for labor, production, or even enforcement, what prevents them from simply disregarding everyone else's interests? Combine that with unprecedented surveillance and predictive capabilities, and you could have an extraordinary concentration of power with very few mechanisms for meaningful resistance. This seems like a potentially fundamental change in the relationship between the powerful and the powerless.
5. I don't see why this concentration of power wouldn't perpetuate itself.
Even if we initially distribute AI capabilities widely, what prevents them from becoming concentrated again More capable agents can potentially acquire more resources, develop better technologies, and increase their advantages. Corporations have incentives to consolidate market power. Governments have incentives to preserve their authority. Autonomous agents pursuing long-term objectives may find it useful to accumulate resources and prevent interference.
Why wouldn't these dynamics become even stronger as intelligence increases?
And even if we establish democratic oversight or other safeguards, what guarantees that these institutions remain effective against actors with vastly superior intelligence and capabilities?
It seems possible that once a sufficient power asymmetry develops, meaningful accountability becomes technologically impossible.
;Tldr/Main underlying concern
I think all of these issues point to the same fundamental problem. Intelligence expands an agent's ability to shape the world according to its objectives. But increased intelligence doesn't necessarily produce increased compassion, moral consideration, or accountability.
As power becomes more asymmetric, less powerful beings lose the ability to protect their interests. Their welfare becomes increasingly dependent on whether the more powerful agents happen to care about them. We already see versions of this relationship between humans and other animals, between powerful institutions and ordinary people, and between states competing for influence.
Superintelligence could amplify these dynamics to an unprecedented degree. And even if the most powerful agents are humans using aligned AI, rather than autonomous AI systems, the underlying problem remains.
Every optimistic scenario I consider seems to run into some version of the same objection: who ultimately controls the intelligence, what constrains their power, and why would they be sufficiently motivated to care about everyone else?
But I would love to hear an optimistic account that goes beyond "AI will create abundance" or "we'll solve alignment."
What is a plausible pathway to a future in which superintelligence exists, power doesn't become catastrophically concentrated, and humans and other sentient beings retain meaningful protection of their interests?