AI’s Biggest Warning is Coming from the People Building It

AI safety used to feel like one of those topics that lived comfortably in the background of the internet. You had researchers talking about alignment, tech people arguing about AGI, and the occasional Reddit post predicting the end of civilisation before Tuesday. Most of us could scroll past it and get back to watching someone make a three-minute pasta recipe.

Then something got weird. Some of the people actually building advanced AI started leaving their jobs and publicly warning about where the technology could be heading. That is a very different kind of warning.

When someone who has never worked in AI says, “I think this stuff could get dangerous,” it is easy to file it under general internet anxiety. When someone who has spent years working on the technology itself leaves and says they are worried about what it could become, you at least have to stop scrolling for a second.

Former Anthropic researcher Jacob Coxon did exactly that in September 2026. After working in AI research, including at OpenAI and Anthropic, he announced that he was leaving, arguing that AI companies were “racing straight to self-improving superintelligence and gambling with our lives.”

That is a slightly more alarming resignation letter than “Thanks for the memories, everyone.”

Marc Fasel; Medium

Former Google DeepMind research engineer Bilal Chughtai made a similar warning after leaving the company. Chughtai had worked on AGI safety and alignment, the area of research concerned with making increasingly capable AI systems behave in ways that remain compatible with human intentions. In his statement about leaving, he wrote, “I earnestly believe that AI has the potential to kill us all, and that we might be running out of time to avoid this outcome.”

Now, obviously, this does not mean AI is definitely going to kill everyone. It does not mean every AI researcher secretly knows something catastrophic is about to happen. Predictions about technology decades into the future are, by definition, uncertain, and plenty of researchers disagree with the most extreme scenarios.

Still, there is something deeply strange about the people closest to a technology becoming some of the people most concerned about its direction.

We usually imagine technological fear happening the other way around. Scientists build something new, the public panics, and the scientists come out to explain that everyone needs to calm down.

With AI, some of the people building the thing are the ones saying, “Actually, can we calm down a little?” And that raises a much more interesting question than whether robots are going to become evil. What are the people building AI seeing that the rest of us aren’t?

The problem isn’t that AI is evil

The easiest way to talk about dangerous AI is to turn it into a Hollywood plot. The machine becomes self-aware, develops a superiority complex, looks at humanity and decides we have had a good run. Cue dramatic music. Real AI safety concerns are much less entertaining, which is probably why they are harder to pay attention to.

A major concern is what happens as AI systems become more capable and more autonomous. Today’s AI can already write code, browse information, use tools, analyse documents and carry out multi-step tasks. As systems become more capable, researchers have to think about what happens when they are given more independence and more access to the systems around them.

Igor Omilaev; Unsplash

The uncomfortable part is that a system does not need to hate humans for things to go badly. It does not even need to be conscious.

A sufficiently capable system could simply pursue a goal in a way its creators did not anticipate, especially if that system has access to tools, information or resources that allow it to act independently.

That is where words like alignment and interpretability enter the conversation. They sound like terms invented specifically to make normal people leave a technology conference early, but the underlying questions are fairly straightforward.

Can we understand why an advanced AI is doing what it is doing? Can we predict how it will behave in situations it has never encountered? Can we stop it if it starts doing something we do not want? And, perhaps most importantly, can we answer those questions before the systems become significantly more powerful?

That last part is where the anxiety around AI starts to make a little more sense. We are extremely good at asking whether we can make AI more capable. We are much less certain about whether our ability to control and understand it is developing at the same speed. That gap is the thing worth watching.

So why are they still building it?

This is where the whole conversation becomes almost annoyingly complicated. If people inside the industry are worried about advanced AI, why does everyone keep making it more powerful? Why not just stop? Because “just stop” sounds considerably easier when there aren’t billions of dollars, national security concerns, investors, competitors and entire industries sitting behind the decision.

Imagine two companies building increasingly powerful AI systems. One decides to slow down because it believes the risks have become too serious. The other keeps developing and releases a more capable system.

Michelle Greenwald; Forbes

The first company has suddenly made itself less competitive. Now add countries to the equation. If one country slows down while another believes advanced AI could give it a huge economic, military or geopolitical advantage, the incentive to keep moving becomes even stronger. This is part of why discussions around AI increasingly sound less like ordinary product development and more like an arms race.

Everyone can believe something is dangerous while simultaneously believing that someone else will build it if they don’t. That is the trap. It also explains why calls for slowing down are usually about coordination rather than simply switching everything off. Anthropic CEO Dario Amodei has publicly argued for slowing the pace of frontier AI development and for stronger safety measures, including more independent evaluation and international coordination.

There is a big difference between saying, “We should never build this,” and saying, “Maybe we should stop accelerating while we’re still working out how to control it.”

It is basically the technological equivalent of discovering that your car has no brakes halfway through a road trip and deciding that perhaps 140 kilometres an hour was an unnecessarily ambitious cruising speed. The uncomfortable part is that nobody has to be acting maliciously for this situation to happen.

Companies can genuinely believe AI will be enormously beneficial. Researchers can genuinely want to make it safer. Governments can genuinely want responsible development. Investors can genuinely want technological progress.

Put all those incentives together, and you can still end up with a race. That is what makes the situation interesting. The problem may not be that everyone is reckless. It may be that everyone has a reason to keep moving.

The future isn’t decided yet, which is exactly the problem

The people leaving AI companies are not necessarily saying that disaster is guaranteed. That distinction matters.

Chughtai’s warning is a concern about a possible future, not proof that an AI catastrophe is inevitable. Coxon’s resignation reflects his own judgement about the direction of the industry, rather than some secret industry-wide admission that the machines are about to escape.

There are also researchers who believe many of the more extreme AI risks are overstated, or that technical safety work can keep pace with capability improvements. There is no scientific consensus that humanity is doomed and pretending there is would make the conversation less serious, not more. What does seem difficult to argue with is that AI capabilities are developing incredibly quickly, while governments, companies and researchers are still working out how much oversight these systems should have and what meaningful safeguards should look like.

That leaves us in a strange position. We are building systems whose future capabilities we cannot confidently predict, while simultaneously creating enormous economic incentives to make them more capable. The people building them can see the upside. They can also see the risks.

Some are staying inside the industry because they think the best place to solve those risks is from within. Others are leaving because they no longer trust the direction of the race. Some executives are asking for slower development and stronger safeguards, while their companies continue competing in one of the most aggressive technological races in history.

None of this proves that AI will destroy humanity. It does prove that the conversation has moved beyond, “Haha, ChatGPT sometimes makes things up.”

The bigger question now is what kind of relationship we want with systems that are becoming increasingly capable, increasingly autonomous and increasingly woven into everyday life. Maybe the most unsettling part of AI is not that its creators are afraid of it.

It is that some of them are afraid of what it could become, and they are still struggling to figure out how a race like this can actually be slowed down. That is the part worth paying attention to. Because the future of AI is not written yet. And for once, that uncertainty is not a comforting thought. It is the reason we still have a say in what comes next.