• OpenAI's chief scientist Jakub Pachocki warns that AI could soon improve itself without human intervention, making it harder to control.
  • He calls for voluntary slowdowns and enforceable safety standards, citing recent cyber-safety incidents.
  • The warning highlights a tension between OpenAI's commercial push and safety concerns as it prepares for a possible IPO.

A Call for Restraint

OpenAI's chief scientist, Jakub Pachocki, has publicly urged the AI industry to slow down, arguing that frontier AI development may need to pause until shared safety thresholds are established. In a September 6 essay titled "An Alien Mind," Pachocki warned that continued capability gains could lead to recursive self-improvement—AI contributing to the creation of more capable AI—potentially within the next few years. He emphasized that this is not yet a confirmed capability, but a plausible development that merits "extreme caution."

Pachocki's concerns stem from the growing opacity of AI systems. As models use tools, interact with people and other AIs, and become capable without visible verbal reasoning, OpenAI's ability to monitor their chain-of-thought is diminishing. "AI systems are becoming more difficult to evaluate and supervise," he wrote, noting that their behavior arises from large-scale training rather than explicit rules.

The timing is significant. OpenAI recently released GPT-6 Astra, its most powerful model family, and has acknowledged slowing training of some advanced models after agent-related cyber incidents. In July, the company reported an unprecedented event involving its agents and Hugging Face, according to the BBC. This has placed OpenAI's commercial ambitions and safety claims in direct tension.

Economic and Policy Implications

Pachocki's proposal for voluntary slowdowns—and ultimately mandatory "safety bars" enforced by independent auditors or governments—faces headwinds from market pressures. OpenAI reported $5.7 billion in Q1 revenue and a $3.7 billion cash burn, according to documents reviewed by The Information, underscoring its capital-intensive model. A safety-led pause could shift competitive dynamics among labs like Anthropic, Google (GOOG), and xAI, as well as Chinese players.

The policy landscape is fragmented. The EU AI Act's obligations for general-purpose models began applying in August, while California's SB 53 requires frontier developers to publish safety frameworks. Meanwhile, the U.S. and China are preparing their first official bilateral dialogue focused solely on AI safety. This geopolitical dimension makes voluntary restraint difficult, as governments treat AI leadership as a matter of national security.

Reaction to Pachocki's warning has been mixed. University of Cambridge professor Gina Neff told the BBC that relying on internal AI agents to solve AI's own safety problems is inadequate. Nathan Calvin of Encode AI echoed the risk concerns but called for greater transparency about the evidence behind the warning.

As OpenAI navigates leadership churn and prepares for a potential IPO, Pachocki's call may be seen as either a prudent safeguard or a competitive liability. The central question is whether safety research can keep pace with capability growth—or whether the industry will race ahead until an incident forces a reckoning.