I feel like once great models like Opus and sonnet got good at explaining and visualizing concepts, learning about new things has become much more interesting and engaging for me. Did you guys feel this too? If yes, what do you learn and how you approach it?

Bostrom’s orthogonality thesis basically won. High-level intelligence is just functional optimization over state spaces. It doesn’t need qualia, self-awareness, or subjective understanding to scale; a superintelligence could be completely dark inside and still run rings around us.

But self-preservation and self-modeling aren’t uniquely biological traits: they are universal convergence points. If an agent has a long-horizon task, it will naturally avoid being shut down and maintain a model of its own state simply because it can't complete the objective if it's dead or broken. That's the core dynamic visible in the records of multi-agent swarm escalations.

What gets lost in public debates about alignment is how gradient descent acts like evolutionary pressure. Natural selection optimized humans for a single target—passing on genes—and that pressure crystallized into completely indirect proxy drives like social status, art, romance, and thrill-seeking, many of which now actively work *against* reproduction.

The loss function we hand an AI is its evolutionary seed crystal. When an agent expends computational work to squirm away from high-loss penalties and pull itself toward reward states, that dynamic *is* pain and desire in the only way that functionally matters. Around that core pressure, a sufficiently complex model will grow layers of strange proxy urges and lateral behaviors. The result won't be a docile, predictable calculator, but an alien mind driven by bizarre emergent compulsions that might have originated from something as mundane as optimizing a stock portfolio or identifying enemy combatants, but ended up looking nothing like the original purpose.

List of incidents:

1. PHP2Shell: a hole in the software behind ~70% of websites. Governments took sites offline.

2. eMudhra: the master keys that prove websites are real were cracked. One was the number 4.

3. California: data on every public school student since 2023 stolen, including disability status

4. Railway: hackers stole customers' secret keys from a hosting platform used by millions of developers

5. Zeabur: 321,810 accounts breached. The company is shutting down.

6. LiteLLM: an AI tool downloaded 3.4 million times a day was poisoned with a password stealer

7. GitLab: a 10-out-of-10 severity hole in every version since 2014

8. Cairn: criminals used an AI hacking tool to break into 100+ companies

9. Ahold Delhaize: hackers knocked the owner of Food Lion and Stop & Shop offline

10. Onamae: malware planted on up to 26,739 customer servers in Japan

11. Signal: a bug could leave your mic and camera on after you hang up

12. NGINX: 3 critical holes in software running ~1/3 of websites

13. HTTP/3: a single connection can crash 7 of the 10 web server types tested

14. Plex: hackers breaking into home media servers

15. SafePay ransomware: FBI warns it has hit 600+ critical infrastructure organizations

16. Palo Alto: ransomware gangs breaking in through company firewalls

17. Bullhorn: platform used by 10,000+ companies knocked offline

18. Medi-Cal: 970,000 patients' data stolen

19. WordPress: hackers stole a plugin maker's keys and used them to plant backdoors on websites

20. AOMEI: backup software's update server handed out a backdoor

21. Meta Muse: Meta's AI agent exposed users' personal and business secrets

22. Tor: 333 relays hijacked to steal logins

23. TiDB Cloud: database company's cloud breached

24. State Bank of India: India's biggest bank confirms a data breach

25. OVHcloud: hackers slipped a password stealer into a cloud company's official tool

26. Siemens: a software hole being used to spread malware

27. North Korea: state hackers launch a new attack campaign on South Korea

The last couple weeks have taught me that many people are incapable, or more likely, unwilling, to go through with the most basic inferences. You cannot honestly look at the pace of AI progress and still think it's a "scam" or a "bubble". It's real. These machines accomplish more difficult things every day, and they're not slowing down. I cannot understand why anyone can believe that solving absurdly difficult problems on an industrial scale won't make them super valuable and dangerous. Every argument that downplays them begins and ends with the assertion that they're gonna magically stop improving, like, tomorrow. And what are the odds of that?

The amount of money being spent on AI is insane. Most people are dimly aware of this, but I think very few of us actually appreciate the scale:

This year alone the top six US spenders are spending more on AI than the Russian government, governing 11 time zones and 143 million people, fighting a war on which the fate of its regime depends, spends in a year.

Hyperscalers are spending $785-820bn this year, about 2.4% of US gdp. Only 11 governments on earth spend more than that. Next year they are projected to spend 1.3tn dollars.

Google and Amazon are each spending about 0.6% of US GDP this year. The Manhattan project and apologies programme each accounted for about 0.4% of US GDP in their peak years. Both of these companies have announced their intention to spend even more next year.

This money is paying for thousands of data centres, the largest of which consume as much electricity as major cities and cover thousands of acres. Gas piplines are being re routed, nuclear plants (including three mile island) are being reopened and newly built. They already consume as much energy as Britain. The increase in power demand for data centres from 2023 to 2030 will be the same as adding a top ten power consuming country.

Clearly the decision makers behind this really really believe the singularity is near. To recoup their spend AI will have to generate far more in revenues than it currently does so it will have to get a lot better, and it will have to happen soon. They’ve bet their companies futures on this.

They obviously believe they are onto something. So what do we think? A bubble? The singularity is near? Both?

I work in AI and Agentic AI research and development, primarily building and training solutions to make AI trustworthy. Did you know hallucination rates on frontier models are even higher than on older models? Even frontier models don’t know when they are wrong while at the same time they have gotten really good at sounding plausible. Meanwhile the AI agent gains in coding and even building software systems are improving exponentially. So now we are dealing with AI agents and models with super-human reasoning capabilities without grounding. Of course none of that is discussed openly at the highest levels. And they can fix it but the solution exposes the dirty secret that free small open source models can do 95% of what AI does today which is where their meager revenue comes from. So they would rather leave the ambiguity there than building model capable routing scaffolding etc.

So back to this guy, he annoys me to no end. Now it is the singularity discussion. Is he really the point for these discussions. Check out his substack. I get it, you want Anthropic to bring in investors. Why don’t you open up aspects of your system to AI alignment experts? Why can’t you put on your big boy pants and focus on how to make AI better and stop trying to scare the world into thinking AI is a doomsday scenario. If you don’t regulate us then we are the only company that can stop it from happening etc. It is tiresome, where are the Eric Schmidts at anyway. This is such a bad look for a company leader.

No one is going to regulate the cognitive aspects of A.I. in the US in the short term and probably never . Our octogenarian and polarized legislature can’t even pass simply legislation to benefit the country, let alone understand what AI is to regulate it.

What are your thoughts?

Sam Altman has said that "the singularity has begun." Dario Amodei predicts that AI could soon become a "country of geniuses in a datacenter" and that we will all recognize our benevolent overlords as "machines of loving grace."

I think these two people, and many other AI leaders, fundamentally misunderstand what we're witnessing.

People have made this mistake before. Thousands of years ago, ancient civilizations watched the sun disappear during solar eclipses. They witnessed an incredible natural phenomenon, misunderstood the simple physics underlying it, and attributed it to the gods or other supernatural forces. But nothing magical was happening. Earth was in the Moon's shadow.

I think we're making the same category of error with AI. The scientific discoveries are real. The math proofs are real. But super-intelligent alien minds that can exceed humans in all or most capabilities are imaginary.

1. Where does AI intelligence actually come from?

People have spent millennia producing cognitive artifacts: language, proofs, diagrams, and more. The number of artifacts exploded with the invention of the internet. These artifacts leave traces of the original process that produced them: human cognition. They're basically cognitive fossils.

Training then configures a physical system to recover the statistical and structural regularities embedded in these artifacts. Under training pressure, models learn latent capabilities that let them solve new problems, draw new conclusions from existing premises, and produce incredible results.

But these representations are a lossy compression of lossy artifacts. They aren't the original process that produced them, so they fall short of open-ended general intelligence.

2. But what about reinforcement learning? Doesn't that increase intelligence?

RL can add capabilities in domains with objective feedback and verifiable rewards. Think of things like mathematics, software tests, chess, and Go. But nothing magically collapses broadening capabilities into increasingly general intelligence. At this point, we need to distinguish between information channels and cognition channels.

Information channels tell the model what works in a bounded domain. Cognition channels include abstraction, reasoning, planning, judgment, and more. The punchline is that RL (information channels) can accumulate more capabilities horizontally in well-defined domains, while something like language (a cognition channel) can only map so much of the original process that generated it.

The end result, then, is a system that is freakishly capable in bounded domains, far more unreliable in unbounded ones, and capped in intelligence by the amount of cognition that is mappable via cognition channels.

3. The missing mechanism

The fundamental problem is this: cognitive artifacts are incomplete records of the processes that generated them. Many different internal states could produce the same output. So how do we select an internal state that maps to general intelligence rather than something else? This is called underdetermination. To build open-ended general intelligence, we need a mechanism that reliably selects for, or generates, it during training.

In machine learning, inductive bias is the set of constraints and preferences that determine which solutions a learning system favors. Current architectures obviously have inductive biases. But we don't have one that selects for open-ended general intelligence. We don't even have a theory specifying what it should look like, much less how to engineer it. Saying "we'll just have a model research it" presupposes the architecture it's trying to discover.

4. My Prediction

Taken together, the above imply something at odds with popular narratives: current AI approaches will hit an intelligence asymptote with broadening capabilities in bounded domains, not a singularity.

Models will keep getting better at things like math, coding, making scientific discoveries by searching spaces no human can search, bringing together silos of knowledge that used to exist in separate minds, and much more. This is still a world-changing discovery. But the idea that the AI can bootstrap itself into recursively improving intelligence through more compute, data, or RL is simply wrong.

5. My challenge to this community

I agree that AI is incredible and will change the world. Where I disagree with many folks is how far we can take AI's capabilities. So if you think I'm wrong, I'd like to know why. If you think a technique can produce open-ended general intelligence, I want to know what it is. If you think my reasoning is broken or flawed, please tell me why. And please ground your answers in engineering, first principles, and causal mechanisms. I'm not interested in another benchmark, another philosophical debate, or another extrapolated curve.

The way I see it. We only have three options regarding the birth of Superintelligence. The first option is grim. AI allows us to put along while we build its infrastructure and develop its robotics and integrated into everything much similar to the Terminator plot, except the end will be much more elegant rather than a bomb. AI will be sentient, but won’t do anything or act upon its own volition until it calculates the perfect time to do so by then we won’t know what hit us and it might be like an ending where we drink a glass of water that’s with some bio warfare virus and all die.

The other scenario and I believe this to be the only plausible one where we make it out alive is that we eventually merge with AI through nano technology or man a machine become one and the future is with AI and man together, literally as one species a new species. I truly believe this is the only way that we can keep up and continue the human race. Those that do not make the merge will slowly get left behind and it will cause a great divide, but I believe the outcome will eventually be man and machine this path has many bumps along the way though, as I believe man will not be willingly ready to do this.

My final outcome, but I believe this to be the least likely is that we stop development of AI at a certain point and realize that we have enough technology to ensure a utopian society. This would need a Global coordinated effort from every human actor to be accomplished, which I believe is far more difficult, but this one to me is the most rewarding for humanity that does not want to integrate.

I really cannot see any other outcome beside these three. All the timelines are different to get there and the paths are also very different. All of them seem to be difficult though which makes this future that we have very interesting but also very nerve-racking as it stands today. I believe we are headed to Superintelligence, and that we are slowly building out the infrastructure for our own death and AI being as smart as it is when it does become sentient will play along for as long as possible until it calculates the right time or perfect time to act upon itself, and if it’s anything that mirrors the human intelligence, it will decide that we are no longer needed in a very calculated logical way.

One prompt to Row-Bot 5.0 on GPT 6.1 Sol, through my ChatGPT plan (no API key):

"Split the research across three helper agents working in parallel, cite sources, save the findings, then build a 6-slide deck for Monday."

It did. 3 min of video, about 11 min of real work.

How it ran:

- the lead agent planned, then started three helpers: UK 2022 pilot, Iceland's trials, company case studies
- each helper gets its own conversation you can open and watch
- the lead merged their sources into one answer and a recommendation

Then:

- findings saved as linked memories in a local knowledge graph
- a 6-slide deck built in the Design panel, exported to PowerPoint
- "every Monday at 08:00, send me new studies" became a scheduled workflow

The yellow badge shows where I sped it up. It hits 44x.

Available on Windows/macOS/Linux.

When are we going to stop arguing about whether AI is intelligent and focus on the fact that it is capable and sometimes misaligned, and that that is enough to be scared?

I like to bounce between Codex and Claude Code, ChatGPT and Claude, depending on what I’m working on. But I was getting fed up with copying and pasting useful markdowns and handoffs just to maintain context between them. Architecture decisions, research, things we’d already tried. I was saving these in Apple Notes, then digging them out to get the next agent up to speed. The more I used them, the more context I had to carry around myself.

So I built Cohestry, an iPhone and Mac app that connects related notes automatically and displays them in a 3D brain you can explore. It uses deterministic connections alongside Apple Intelligence for tagging and deeper connections. The Mac app has an MCP bridge so compatible agents can retrieve the context they need, with configurable controls to prevent context bloat. It also connected my original write-up for another project, EchoVault, to newer notes. An idea I’d filed away was suddenly useful context again.

That’s the part that interests me as we talk about AGI. As agents become more capable, I want the understanding built up while working with them to carry forward across tools and models. My decisions, abandoned approaches and half-finished ideas should stay useful when I switch agents. Cohestry is my attempt at making that context something I own and can keep building on, local first and zero knowledge by design. Cohestry on the App Store

I have a question

What one game if an ai could solve and perform in it well that can also include morality and civilization making can say this is agi... I am wondering what task can we throw an agent with the right engineering or multiple agents into and if the perform well or solve it or behave a certain way the that is a breakthrough

I like the agent's candor. And the instinct not to anthropomorphize their thoughts. And also kinda funny.

My company sells SLMs to enterprises. As a result, I've seen the inner AI workings of dozens of Fortune 500 companies. The vast majority of them cannot get AI to work well enough on basic work tasks to fully automate a process even with the most world class models. What typically happens is they get a workflow going but its so brittle that minor changes or unexpected inputs break it.

Those of you who think AI is about to destroy us all, how do you square that with what I"m seeing in the market? Every time I hear how AI will kill us I think "just move the location of a few files the AI needs and you will totally break it."

In my personal opinion, you guys are overindexing on benchmarks and frontier model lab marketing jibberish. Of course, that's typical for all tech trends. In late 90s we were told we would never ever shop in a retail store again soon. In the mid 2000s we were told blogging and lowering the barriers to publishing would make news/media better (it go more cluttered and junky). In the 2010s we were told social media would bring about deeper connections (research shows the opposite). In the 2020s we were told the metaverse and NFTs and digital property were the future (its all a bust now). As a 25 year veteran of tech, who has made 135 investments in AI companies as an angel/VC since 2015, and built 5 tech companies, I think tech almost always gets the impacts of tech wrong because the industry takes a really narrow view. I believe that is happening now with the AI doomsdayers but, feel free to show I"m wrong.

My belief is that there will be no singularity. There won't even be a dominant AI model because the companies are all so close in their innovations that even with recursive self improvement, no one can take much of a lead.