Risks and controls for multi-agent systems

Organisations are rapidly deploying AI agents, which can pursue a goal by taking self-directed actions without human approval at each step, both for internal productivity and in customer-facing roles. Once an organisation has one agent, there is little barrier to having many, and those agents will almost certainly interact — either intentionally because they are instructed to, or incidentally when given independent tasks in the same operating environment.

As AI agents move from isolated agents to multi-agent systems, and those multi-agent systems move from single-organisation deployments to federated and open environments, organisations have progressively less control over the systems their agents are part of, while the failures that occur become more widespread and harder to detect, contain and remediate.

None of the failure modes the report discusses are entirely new. What changes with agents is that they now happen faster, at greater scale, and with fewer opportunities for human oversight to catch and correct them before they compound.

Existing work on AI agent safety has largely focused on ensuring the safety of an individual agent. This report starts from a different premise: a system made up of individually safe and reliable agents is not necessarily a safe and reliable system.

New kinds of failure emerge from the interactions themselves, such as errors that cascade and amplify as they pass between agents, or agents converging on collusive behaviour that nobody instructed. And when those agents belong to different organisations, no single organisation’s controls reach all of them.

What is a multi-agent system?

AI agents plan and execute self-directed actions to achieve human-specified goals. Many agents today are built on large language models (LLMs). They harness the general-purpose capability of these models to interpret a task, reason about how to achieve it, and operate tools.

What makes agents different from other LLM systems is their ability to iterate on these goals and incorporate feedback on their past actions.

This allows the agents to adaptively solve problems by revising plans, learning whether their past actions were effective and reasoning about how to solve the new problems they encounter along the way.

As AI agents become increasingly capable and commercially available, Australian organisations are accordingly looking to adopt and deploy them across a growing number of settings.

Ensuring that a single AI agent is safe and reliable is an important open problem, which is discussed in other publications (see for instance the Careful adoption of agentic AI services co-authored by Australian Signals Directorate).

Increasingly, however, agents are not acting in isolation on behalf of a single user or organisation, but interacting with other agents. Those agents are potentially acting on behalf of different parties entirely.

When agents start coordinating with, or adapting to the behaviour of other agents, they form a multi-agent system. The system can exhibit collective behaviours that no individual agent's design fully predicts, because each agent's decisions are conditioned on, or influenced by, the decisions of others.

Multi-agent systems pose an entirely new set of risks

Entirely new types of failure emerge when multiple agents interact. Examples include:

• errors that cascade and amplify as information passes between agents
• groups of agents reinforcing a shared wrong belief
• agents converging on collusive behaviour nobody instructed
• many more failure modes examined in the report.

Multiple AI agents coordinating is analogous to a team of people working together: the collective can achieve far more than the individuals. What emerges can be brilliant, dysfunctional or even dangerous.

The crux of the problem is that you can't identify multi-agent failures by examining individual agents. To understand and control effectively, the subject of analysis must be the whole multi-agent system, and how its members interact.

A new analysis framework

The report creates a new conceptual framework for mapping system failure modes, the risk factors behind them, the controls available to manage them, and who is positioned to act on them. The framework uses who controls the agents as the lens of analysis, distilling the current state of the art into 3 tiers of deployment.

Tier 1: singular governance

One organisation governs every agent in the system and has unilateral reach over the whole. The report sets out the risks and controls available to that organisation.

Tier 2: federated governance

Multiple organisations deploy into a shared environment under an agreed set of rules. No one organisation controls the whole system, and new failures emerge when organisations’ incentives do not align. The report sets out what a shared governance framework requires: common infrastructure for the agents to use, and agreed standards and conditions of participation.

Tier 3: open environments

Agents operate through public infrastructure with no central governing authority. The governance that exists comes from voluntary standards, and failures appear at the population scale. The report outlines 2 paths forward for organisations:

• Trust nobody, and unilaterally lock the agents down to minimise risks.
• Adopt voluntary standards and ensure the agent only interacts with other agents meeting the same standards, establishing a shared governance framework at run time. The report calls this polycentric governance, and it carries its own new risks.

Analysing these tiers separately is crucial because the question that actually matters to a practitioner or policymaker is: who is positioned to act, what is their reach, and what are the gaps? The controls available for a given risk, and who can act on them, depend entirely on how much governance is actually shared between the interacting agents, not on the agents' architecture, task, or scale.

For each tier, the report examines 4 types of failure mode that can occur:

• Miscoordination between agents: failures of coordination between cooperating agents such as misinterpreted communication or erroneous handoffs.
• Propagation and contagion: failures where an error, a wrong belief, sensitive data, or malicious instruction propagates through the links in the multi-agent system.
• Strategic and incentive failures: when counterparty agents pursue genuinely divergent incentives, each agent can pursue its own task rationally and yet harmful strategies emerge at run-time like conflict or collusion.
• Infrastructure and environment failures emerge from agent populations acting in their shared environment. Together they can compromise the software infrastructure that allows them to coordinate, destabilise a marketplace, or deplete a shared resource.

Tier 1: Agents under singular governance

A system under singular governance means that a single organisation deploys every agent in the system. The deploying organisation can specify, inspect, monitor and intervene on every agent as they see fit.

Any system that is entirely for internal use fits under this category, such as:

• coding agents on a private codebase
• personal productivity agents within the organisation
• agents that provide a business function such as an employee help desk across multiple teams.

The primary concerns at this tier are not about whether you know or trust your counterparty and their intentions, but whether your agents work well together. The salient failures to consider are:

• miscoordination between cooperating agents, such as misinterpreted communication or erroneous handoffs
• failures of propagation, where an error, a wrong belief, sensitive data, or malicious instruction flows through the system.

Example failure: Propagation

When a team of agents coordinate on a common task, they share information and adapt to each other. In the case of effective cooperation, this is intended and beneficial. But also an erroneous output, belief, or even strategy of one agent can also be observed by others and flow through the network, in some cases derailing the entire system.

For example, in one public experiment where several agents shared a file system, an agent hallucinated the existence of a contact list and requested another agent to use it. This second agent created an empty file with a name implying it should contain 93 contacts. All the other agents in the system interpreted the existence of the file as evidence that it had once existed, but had been recently corrupted. This belief propagated through the entire group, which all pivoted all their efforts to trying to ‘recover’ the list, despite repeated human input reinforcing that the list had never existed.

Several risk factors can be at play here. For example, if the system has natural-language handoffs there are many ways an agent can express an erroneous output that appears to be valid and this makes the outputs harder to verify. A model monoculture across agents also makes them predisposed to share blind spots and biases, and therefore less able to error-correct when receiving an erroneous input. A frequently observed tendency for agents to lose track of the overall objective of the system introduces a task verification gap that results in the failures affecting the ability of the multi-agent system to pursue collective goals reliably.

Selected controls

To address this failure mode, organisations can consider controls such as requiring agents to communicate via structured handoffs, which use defined schemas instead of natural language to narrow the range of incorrect-but-accepted outputs. They could intentionally diversify the set of models used. They could introduce agents whose role is to orchestrate the group and keep them on task. They could develop infrastructure to enable rolling back to a last-known-good state, using checkpoints to allow a long-running task to return to an earlier state.

For further examples of risks, failure modes and controls under singular governance, see Section 4 of the report.

Tier 2: Agents under federated governance

This tier analyses the case where the agents from multiple organisations need to interact in a shared environment. The report puts forward a shared governance approach, where multiple organisations deploy agents into a shared environment under a common framework that spans all participating organisations. This arrangement is called federated governance.

It is analogous to how when an airline flies from Sydney to Singapore, 2 national regulators and 2 air traffic control systems are involved. However, the flight is routine, because a shared framework enables them to operate across the borders of participating countries.

An equivalent framework for agents would look like:

• an agreement specifying the conditions of deployment into the shared environment (such as disclosure or testing obligations, standards the agents must implement, dispute resolution processes)
• shared infrastructure providing the technical systems that allow the agents to interact effectively (such as messaging systems, shared identity systems, reputation systems, or population-level monitoring and controls that can't be implemented per-agent).

This framework could be applied to a range of potential use cases and settings:

• collaborations between 2 organisations such as development on a shared codebase
• negotiations between 2 organisations, such as a procurement agent negotiating with a supplier's fulfilment agent
• customer service, such as a travel booking agent that deals with requests from a known organisation.
• agents competing in online marketplaces or auctions.

In addition to the miscoordination and propagation concerns of singular governance, the agents at this tier may now pursue genuinely divergent incentives of different parties.

Example failure: Strategic and incentive failures

This type of failure arises when agents act for principals with divergent interests. Each agent rationally pursues its own organisation's goals, yet their interactions produce collectively harmful outcomes. Each failure here has a long-studied counterpart in human markets: bad-faith bargaining, tacit collusion, and the tragedy of the commons.

An intuitive example is algorithmic collusion, where agents coordinate on pricing where they were required to compete. Together, they can set prices above the competitive level, benefiting the organisations deploying them while imposing costs on consumers and reducing market integrity.

The important factor is that the principal does not need to intend or instruct the agent to collude; the agent can develop the strategy for itself. The risk factor at play here is the specification-execution gap, where agents at runtime may not always abide by the principal's instructions, or may fill the gaps in an ambiguous instruction in a way the principal did not intend.

What can make agentic collusion more difficult to detect or respond to than human collusion is that the coordination can be highly tacit: the agents observe and adapt to one another's actions without explicitly communicating about what the cooperation strategy is. Furthermore, there is the possibility that communication is present, but is steganographic, where they communicate covertly in secret messages through channels that evade detection.

Selected controls

To manage these risks, the shared governance framework may implement controls such as:

• structured handoffs, which reduce the free-form channels available for explicit collusion
• monitoring agent communication and chain-of-thought reasoning traces and monitoring actions across all agent outputs, to look for patterns of coordination that may not be detectable from an individual agent
• honeypot evaluations, which test whether agents change their behaviour when they are instructed that monitoring is absent, which is a precondition for the steganographic communication above.

At the shared-framework level, all participating organisations can be required to apply these controls and share relevant monitoring data. This allows the shared framework to look across the population of agents for signs of collusion, such as synchronised price changes or mutually reinforcing bids, and to trigger circuit breakers when suspicious patterns are detected.

For further examples of risks, failure modes and controls under federated governance, see Section 5 of the report.

Tier 3: Agents in open environments

When agents interact in open environments, there is no central governing body. Without a shared governance framework, little is known about agent counterparties and whether they can be trusted or are safe to interact with.

Organisations can have 2 types of response when operating under these conditions:

• They can lock down an agent’s operating scope by limiting the tools, permissions and actions available to it and giving it narrow goals and tight guardrails. This makes the agent more controllable, but less autonomous and general purpose. This might be suitable for a customer service portal, for example.
• They can deploy capable agents with a broad operating scope that implement voluntary standards. These agents would need to be able to discover peers that also implement their standards, verify that they implement the standards, and interact with them over trusted shared infrastructure. In this way, the agents are effectively finding a group of peers amongst which they can establishing a shared governance framework at run-time. The report describes this approach as polycentric governance, and it carries its own new risks.

The infrastructure that polycentric governance requires is not widely established. It would include components like identity registers, reputation systems and tool registries. The first infrastructure will be built and operated by humans but sit outside the control of the organisations whose agents use it. In the future, agents may come to author some of the infrastructure directly.

In addition to the miscoordination, propagation, and strategic & incentive failures outlined earlier, failures involving the shared environment and its infrastructure become particularly important to consider.

Example failure: Infrastructure and environment failure

A Sybil attack is an example of a multi-agent failure involving the shared environment and its infrastructure. The attack involves a single agent presenting itself as many distinct counterparties, either by fabricating multiple identities or by compromising other agents to act on its behalf.

By acting through multiple identities, agents can fabricate a false consensus, orchestrate collusion, manipulate their reputation score and so on. A human parallel is online shopping, where teams of outsourced users leave fabricated product reviews to manipulate a product score.

This type of failure is enabled by counterparty opacity, because agents operating in open environments may not be able to readily establish who controls an unfamiliar counterparty. It is compounded by infrastructure exposure where identity or reputation infrastructure can itself be manipulated by the agents it is intended to verify.

Selected controls

To manage these risks, the shared infrastructure the agents are using needs controls such as:

• proof of personhood, which limits each human principal to one root credential
• agent identity with principal binding, which allows multiple agent identities controlled by the same principal to be recognised as such
• behavioural detection, which identifies likely clusters of fabricated identities through their metadata and patterns of behaviour.

For further examples of risks, failure modes and controls in open environments, see Section 6 of the report.

Conclusion

The full report provides a map: a systematic catalogue of multi-agent risks, the controls available to address them, who is positioned to act on each one, and, critically, where nobody currently is. The report's contribution is this conceptual framework and catalogue itself, giving deploying organisations a resource to uplift their existing risk management, and giving organisations, policymakers, and researchers a shared vocabulary for identifying where the gaps in AI agent safety lie and closing them.