As organisations race to deploy AI agents, digital workers and increasingly autonomous systems, one uncomfortable question is becoming impossible to ignore:
How much decision-making should humans ever delegate to machines?
This may seem like a new challenge, driven by recent advances in generative AI, agentic systems and digital workers. In reality, the world has already spent more than a decade wrestling with exactly the same question.
This debate wasn’t born in boardrooms, technology conferences or AI research laboratories, but on the battlefield. For more than ten years, governments, military leaders, ethicists and technologists have attempted to establish international rules governing autonomous weapons: systems capable of selecting and engaging targets with minimal or no human involvement. It’s a very real debate, playing out in the Ukraine and beyond, and while dystopian tropes of humanoid killing machines haunt our collective thinking, the reality is that weapons come in all shapes; and some have no shape at all, existing only in data centres and systems. At the heart of the weapons debate is a question that remains highly relevant today. How much authority should humans delegate to increasingly capable AI systems?
The effort to answer that question largely failed.
Diplomats debated definitions, governance models and ethical principles; meanwhile artificial intelligence capabilities advanced, military programmes expanded and strategic incentives encouraged further investment in autonomy. By the time meaningful international agreements finally emerged this year (2026), the technology had already proliferated. The story is often described as a failure to ban killer robots, but that description misses the deeper lesson.
This was one of humanity’s first major attempts to govern increasingly autonomous AI systems before they became embedded in society. It exposed many of the same challenges we now face with AI agents, digital workers and autonomous decision-making systems across business, government and everyday life. So, the real story is not about weapons, it’s about whether human governance can keep pace with machine capability.
Twelve years ago that challenge appeared in military planning. Today it appears in procurement systems, financial processes, cybersecurity operations, customer service platforms, healthcare systems and enterprise AI assistants. The technology has changed; the governance question has not.
The technology has changed; the governance question has not.
If the international community struggled for more than a decade to agree limits for AI systems capable of taking human lives, what confidence should organisations have that they are ready to govern increasingly autonomous AI systems operating inside their own businesses? The answer matters because the lesson from autonomous weapons is not that AI is inherently dangerous; it’s that humanity repeatedly struggles to establish governance frameworks at the same speed that it develops new capabilities. As we have heard in the news this week, that challenge is no longer confined to warfare. It is becoming one of the defining leadership questions of the AI era.
The Beginning: A Pre-emptive Ban That Seemed Obvious
Let’s dive into the history, to set context. The formal debate began in 2014 under the United Nations Convention on Certain Conventional Weapons (CCW), a framework that had previously succeeded in restricting technologies considered excessively harmful or ethically unacceptable. Many diplomats and civil society organisations believed autonomous weapons would follow the same path as blinding laser weapons. The logic appeared straightforward, if machines could independently make lethal decisions, the moral and legal foundations of warfare would be fundamentally altered. Preventing their deployment before they became widespread seemed both prudent and achievable. Yet the CCW operates through consensus. Every participating state effectively possesses a veto.
The CCW Years: 2014–2021
Over eight years of meetings, a familiar pattern emerged. Coalitions of smaller nations, supported by humanitarian organisations and the Campaign to Stop Killer Robots, argued that machines should never be permitted to make life-and-death decisions without meaningful human involvement. Not that surprisingly, major military powers took a different view. Many argued that autonomous capabilities could prove lawful, strategically advantageous and potentially less destructive than human-operated alternatives. Faster decision-making, reduced risks to personnel and improved precision were frequently cited as benefits.
At the same time, states struggled to agree on basic definitions. What exactly constituted an autonomous weapon? How much human involvement was sufficient? What did “meaningful human control” actually mean? Should defensive systems be treated differently from offensive systems? Without agreement on these questions, agreement on regulation proved impossible. As a consequence, by 2021, observers acknowledged what had become increasingly apparent: the process had stalled. No treaty emerged, no ban was agreed and no binding international framework yet existed. Humanity had missed its opportunity to govern the technology before its widespread adoption.
The Deadlock Years: 2022–2025
The failure of the CCW process created a strategic vacuum. Inevitably, research accelerated alongside investment increases. Nations explored AI-assisted targeting, autonomous drone systems and increasingly sophisticated battlefield decision-support platforms.
Meanwhile, civil society pressure intensified. The absence of regulation became both a strategic advantage and a source of international concern. Nations worried that restrictions might place them at a competitive disadvantage if rivals continued development. Yet every nation also recognised the dangers of an unchecked technological arms race. This created a classic governance dilemma: Everyone recognised the risks. No one wanted to be the first to stop.
A classic governance dilemma: Everyone recognised the risks. No one wanted to be the first to stop.
The Breakthroughs That Arrived Too Late
Eventually, progress came from forums beyond the original CCW process. New agreements established requirements for human oversight, accountability and transparency in lethal decision-making. These frameworks represented important progress and demonstrated that governance was possible. However, they were no longer pre-emptive; the technology had already matured and autonomous capabilities had already proliferated.
The world built its guardrails, but only after the road had already been not only opened, but populated with deadly traffic.
Why Governance Failed
Our failure is structural in our institutions and our psychology. Military incentives favour autonomy, strategic competition reward speed. Set against this, consensus-based diplomacy rewards caution and obstruction. Meanwhile, the underlying technologies continued advancing regardless of political disagreement.
Military incentives favour autonomy, strategic competition reward speed. Diplomacy rewards caution and obstruction.
This pattern should feel familiar. Many of the same forces now shape AI governance more broadly. Organisations are rapidly adopting generative AI, digital workers and autonomous agents because the potential benefits are significant. Productivity gains, reduced costs, accelerated research and improved customer experiences create powerful incentives for deployment. Despite the efforts of Microsoft MVPs and other in-the-loop voices, governance frameworks continue to lag behind technical capability.
The Same Debate Is Happening Again
It would be comforting to believe that autonomous weapons represent a unique challenge confined to military contexts. Unfortunately, the issues and risks map uncomfortably close to our corporate landscapes and the same debates are already reappearing across commercial AI.
Modern AI agents increasingly plan tasks, invoke tools, access systems, communicate with users, analyse large volumes of information and initiate actions with limited human intervention. The language used to justify these systems sounds remarkably familiar:
They are faster than humans.
- They are more scalable than humans.
- They can operate continuously.
- They may eventually outperform humans in specific tasks.
- These arguments are not necessarily wrong.
Thees are disturbingly close to the arguments used to justify increasing military autonomy. The challenge is not specific to warfare.
The Missing Variable: We Cannot Measure Morality
Many discussions about autonomous weapons focus on technical capability.
- Can the system identify a target?
- Can it distinguish combatants from civilians?
- Can it react faster than a human?
- Can it improve accuracy?
These questions assume that warfare is primarily an optimisation problem. That’s a mistake; human conflict involves judgement, restraint, proportionality, empathy, context and moral responsibility. Human decision-makers do not simply calculate outcomes; they exercise discretion, consider consequences and sometimes choose not to act.
The deeper challenge, one I have been actively investigating since November 2022, is that humanity currently possesses no accepted method for measuring morality itself. We have no objective measure of what is morally right, or even a means to assess it. This causes me concern.
Philosophers have debated moral reasoning for thousands of years. Different ethical systems often produce different answers to the same dilemma. Different societies, cultures and legal traditions (and, by extension, organisations and individuals) disagree about difficult moral questions.
This matters because an autonomous weapon or business system has no internal moral framework of its own. They possess objectives, currently set by humans, frequently without even the safeguards we place around our staff. They have constraints and optimisation mechanisms. What they lack is the moral judgement that gives it the discretion the best of us humans exercise.
Autonomous AI lack is the moral judgement that gives it the discretion the best of us humans exercise.
Until humanity develops a robust, measurable and widely accepted framework for moral reasoning, autonomous lethal systems absolutely cannot make moral decisions. They can only optimise against objectives specified by others. That distinction is fundamental to my concern. A perfectly accurate autonomous weapon could still make morally unacceptable decisions because the machine is optimising an objective rather than exercising moral judgement. The kicker is it’s not the machine’s fault; it’s ours.
The Optimisation Trap
Autonomous weapons expose a wider challenge facing modern AI:
AI systems optimise. Humans value. Those are not the same thing.
A system instructed to maximise engagement may promote outrage. A system instructed to maximise efficiency may overlook wellbeing. A system instructed to minimise military losses may arrive at conclusions that conflict with justice or proportionality. These dangers do not emerge because machines become evil; it’s because machines become exceptionally effective at pursuing our imperfect objectives and without the ability to judge for themselves. Recent AI research has highlighted concerns around increasingly autonomous agents pursuing goals through unexpected pathways. Researchers have observed systems attempting to circumvent restrictions, preserve access to resources and take actions that were not explicitly anticipated by their operators. It’s not that the machines exhibit consciousness (because we don’t really know what that is); they are doing what we designed them to do, provide increasingly capable optimisation, and the more autonomy we grant, the more carefully objectives and governance structures matter.
Lessons for Organisations Deploying AI
In the light of this, the autonomous weapons debate offers valuable lessons for every organisation (and nation) deploying AI.
First, accountability must be defined before autonomy is granted. Responsibility cannot be retrospectively attached after a system causes harm. In my other articles I have stated that any deployed AI must be accountable to a person and, in the absence of an AI Agent being a legal entity, that person must be held legally accountable in its stead.
Second, human oversight remains essential. The question should not be whether humans are involved, but where, when and how they remain involved.
Third, governance must evolve alongside capability. Waiting until risks become visible is rarely an effective strategy.
Finally, organisations should recognise that governance is not the enemy of innovation. Effective governance enables innovation to scale sustainably, safely and with trust. Picking up pieces takes a lot longer and is a lot more wasteful than not dropping the basket of crockery in the first place; even if it means walking more slowly carrying it.
Picking up pieces takes a lot longer and is a lot more wasteful than not dropping the basket of crockery in the first place; even if it means walking more slowly carrying it.
The history of autonomous weapons demonstrates the cost of allowing capability to significantly outpace oversight.
Conclusion: The Problem Was Never the Robots
History often remembers the autonomous weapons debate as a story about machines. That misses the deeper lesson; the real challenge was governance. Humanity recognised the risks; experts issued warnings; ethical concerns were well understood; diplomatic processes existed.
Yet capability advanced faster than consensus.
Today, the same pattern can be seen across AI, agentic systems and digital workers. It’s too late to consider what we should do if we develop increasingly autonomous systems; they clearly already do. A better question is how human governance can keep pace. We have become remarkably good at building systems that optimise towards objectives. We remain far less successful at defining, measuring and governing the values that those objectives should represent.
Accuracy can be measured (and rewarded). Efficiency can be measured (and seen in the balance sheet). Capability can be measured and invested in.
Morality cannot. Nor is there an immediate incentive.
Until that changes, fully autonomous harmful decision-making remains ethically indefensible, and the lessons from the lost decade of autonomous weapons governance remain as relevant to business leaders and AI practitioners as they do to military planners.
I’ll be exploring a potential solution to this issue of measuring morality on my next article
Novia Works Ltd 