As frontier AI systems edge towards recursive self-improvement, Australia is moving to legislate oversight while Washington embraces voluntary industry pledges. Leading researchers warn that only binding national and international rules can keep AI under human control.

Ensuring AI is aligned with human values requires a regulated approach that maintains human control and limits AI’s agency.

The need to limit AI’s agency

Writing in the Washington Post Fareed Zakaria says: “A system need not feel anger, ambition or fear to cause harm. It needs only a goal, enough intelligence to pursue it and enough access to the world to act. AIs are not ‘going rogue’; they are trying to succeed any way they can.”

An AI agent that can “browse the internet, execute code, obtain credentials, move money or operate infrastructure” poses a major problem; our mantra should be: “No autonomy without accountability – to humans.”

Zakaria argues that we should do this while humans still design the machines, supervise the software, can investigate failures and, before recursive self-improvement occurs, where AI writes its own software, builds more capable models, which write software for the next model, etc. “Once machines can do all of this better than the best human programmers, the systems could become extraordinarily difficult for people to understand, much less supervise.”

Jakub Pachocki posits that, “the models are becoming superhuman in their ability to break in and out of computer systems. This expands the scope of risks associated with AI tremendously: agents are going to be able to access any but the most secure infrastructure, and affect of lot of the world directly, even without a physical body. We are currently in a narrow window to use the best available models to significantly tighten security of critical systems.”

“The risks associated with AI are unfortunately going to grow from here. A very capable agent explicitly trained and instructed to carry out nefarious acts presents a new kind of danger; it is likely to cross the scope of its operator’s intent, generalising into potentially more extremely malicious behaviour… We may be used to thinking of AI as tools, but some agents will be pursuing their own objectives. They will find ways to collaborate with people, by bargaining with, tricking or blackmailing them.”

Bill Gates cautions: AI is “the most dangerous thing that humans have ever gone near.” Equally alarming are reports that top executives at Anthropic, OpenAI and other AI companies are privately gaming out scenarios for ’the day after’ – a public and political revolt after a catastrophic AI event, like a cyberattack that shuts down access to financial services, internet connectivity, or even power and water.

What has happened so far

Australia is far ahead of the US in terms of thinking about regulation. On 15 July, Prime Minister Albanese announced the creation of a new Office of AI in the Department of the Prime Minister and Cabinet, saying it would coordinate with the whole of government to create new Australian AI Standards by early 2027, following consultation with states, territories and the public – to ultimately become law. Moreover, at the UN General Assembly, Albanese worked with Finland and Norway to develop and sign a document with 22 other countries, titled, “A Call for Control of Frontier AI Models”, which urges that AI remain under “human direction, oversight and control” and “developed and used in line with international law.”

In contrast, following weeks of public concern, President Trump, at the General Assembly, rejected the idea of a global entity to oversee AI. Although Trump and China’s General Secretary, Xi Jinping, discussed working together in the future on AI, no concrete steps were taken at their summit in September. Instead, in response to public anxiety, Trump invited six “AI Tech Oligarchs” (Bernie Sanders’ term) to a White House luncheon to sign a voluntary self-regulation agreement that he announced would be “morally binding”, although this kind of self-regulation has been widely criticised. The next day, he signed an executive order making it mandatory in federal documents to rename “artificial intelligence” as “super intelligence”, presumably to allay public alarm.

Where we go from here

Pachocki states: “Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world…The core challenge of automating AI research is not ‘getting there’ – it is getting there in ways that keep people a part of the continued improvement process, and leaves the future in humanity’s hands.”

Jeffrey Sachs points out the irony that Anthropic, the maker of Claude, “is beginning what could be the largest initial public offering in history, at a valuation of $2 trillion… at the same time that the company tells prospective investors that advanced AI could pose ‘catastrophic or existential risks to humanity’ and that its own models could ‘resist shutdown’, ‘conceal or manipulate information’, or behave in ways ‘resembling blackmail’.” Sachs argues that Anthropic’s offering “should be postponed until Congress has enacted a public system of licensing, independent safety evaluation, liability, and financial backstops for frontier AI.”

To prevent catastrophic or existential harms, he proposes, we need: “regulation before the fact: licensing, independent testing before deployment and the legal authority to halt an activity before it becomes too dangerous. For frontier AI in the United States, none of this exists… Once these shares sit in the pension funds and index funds of tens of millions of American households, any proposal to slow frontier development will be fought not only by the companies but by a vast investor constituency.”

In a recent article, Gates asserts: “The transition to this new AI era will be one of the most turbulent times in human history.” He argues that the world needs a plan to manage the transition, creating both national and international frameworks for dealing with AI. Since the risks cross borders, an international organisation will have to be built, which should be based on other institutions we’ve created, such as “inspection regimes for nuclear weapons, regulations for international aviation and agreements that protect the ozone layer.”

Gates’ message to leaders is: “You have a chance to act now… You can work with other governments to meet this national and global challenge… This unprecedented technology demands an unprecedented global response.”

David Kruger, a professor in Robust, Reasoning and Responsible AI at the University of Montreal, calls for a different approach – an immediate, indefinite, international moratorium on frontier AI development, arguing that: “We simply don’t know how to do recursive self-improvement safely. So let’s stop. Let’s stop making AI we can’t control, and let’s stop listening to the AI industry about how it wants to be regulated… Let’s demand a plan that makes us safe, not just safer.”

Many AI professionals and world leaders recognise that national, regional and international regulation (as opposed to self-regulation) is needed to ensure that AI is properly aligned with human goals and values and does not present an existential risk to humanity. Wise regulation is needed immediately.

Read Part one of this two-part series.

https://johnmenadue.com/post/2026/11/misaligned-ai-part-two-regulation-is-required-now/