Mitigating risks and harms

AI systems have specific characteristics that amplify risks

AI systems are composed of AI models and non-AI components, with AI models playing a key role in influencing their characteristics. In this guidance, the term 'AI system' is used to include AI models when the distinction between the two is not critical. However, 'AI system' and 'AI models' will be explicitly distinguished when the difference or emphasis on both is important.

AI systems span a wide range of technical approaches. Organisations can use them for many tasks, such as helping with prediction, classification, optimisation or content generation. AI systems fall broadly into 2 types, each with different strengths and risks:

• Narrow AI systems are developed to perform a specific task. Many AI systems in use today fall into this category. These types of systems can perform well in a narrow range of tasks, potentially even better than humans, but they cannot perform any other tasks. Examples include chess engines, recommender systems, medical diagnostic systems and facial recognition systems.
• General-purpose AI (GPAI) systems are developed to handle a broad range of tasks and are therefore flexible. Their use is not limited to a specific task, so they can be more easily used for purposes their developers may not have considered. Examples include large language models and systems such as Open AI’s ChatGPT series.

Both narrow and GPAI systems are developed and operated differently from traditional software systems. These differences mean that deploying an AI system for a particular task may amplify existing risks or create new risks when compared with traditional software.

For example, in traditional software systems, developers explicitly define all the logic governing a system’s behaviour. This relies on explicit knowledge, with conscious human engagement at every stage of the software design and development process. Traditional software systems are easier for humans to control, predict and understand.

In contrast, developers of AI systems take a different approach. This often involves defining an objective and constraints, selecting a dataset, and employing a ‘machine learning algorithm’. This creates an AI model which can achieve the specified objective, and together with other non-AI components, forms an AI system that can perform a variety of tasks. While such AI systems often outperform comparable, traditional software systems, the different development approach means AI systems, in particular the AI models within them, are often less transparent, less interpretable, and more complex to test and verify. This amplifies risks and can lead to harm. This is more likely to happen in contexts where it is important to understand and explain how the output was achieved or to constrain the range of potential outputs for safety reasons.

The specific characteristics of GPAI systems, especially frontier AI, can further amplify risks and pose new risks and harms to an organisation. This is because they are highly complex and not fully understood, even by their developers. They may possess advanced capabilities that are unknown or emergent. GPAI systems have the capability to understand and use software tools and can access other systems and knowledge, enhancing their capabilities in specific deployment contexts. GPAI systems are also highly general, supporting an unlimited number of downstream planned and unplanned use cases, including deliberate and inadvertent misuse. It is impossible to evaluate all possible use cases, making pre-deployment evaluation and testing highly challenging.

For example, a GPAI chatbot system that can generate code could potentially produce malware and autonomously hack into critical systems. Similarly, a GPAI chatbot that can generate realistic images could be used to create deepfakes for impersonation and fabricating non-existent real-world events. While these systems were not designed for such specific purposes, and some guardrails can be implemented to refuse certain tasks, it is difficult to cover all potential misuses.

A proportionate approach to AI harm prevention and mitigation

As with all software, AI systems vary in the level of risk and the type of harm they pose. Some, like an algorithm that suggests reordering based on stock levels, tend to be lower risk. The potential harms are confined to a customer taking longer to receive a product or the financial impact of over- or under-ordering. Others, like a tool that prioritises job applicants for an interview process or makes financial lending decisions, have potential to create far greater harm. For instance, they may deny a suitable applicant the opportunity of a job or bank loan, or even systematically and unlawfully discriminate against a group of people.

This guidance supports a risk-based approach to managing AI systems. It does this by supporting organisations – both AI developers and AI deployers – to take proactive steps to identify risk of harms posed by the AI systems they develop, deploy, or rely on.

The implementation practices prioritise safety and the prevention, identification and mitigation of risk of harm to people. This is grounded in an approach that seeks to protect, respect and remedy human rights. By adopting this approach, AI developers and AI deployers, in turn, also prevent and mitigate the risk of harm to their own organisations. 

Implementing this guidance can help with identifying, preventing and minimising other risks that may affect an organisation and its stakeholders. Organisations often analyse these risks against the potential for reputational damage, regulatory breach, and commercial losses.

Commercial – Commercial losses due to poor or biased AI system performance; adversarial attacks.

Reputational – Damage to reputation and loss of trust due to harmful or unlawful treatment of stakeholders such as consumers, employees or citizens.

Regulatory – Breach of legal obligations that may result in fines, restrictions and require management focus. System factors and attributes that amplify risks and harms

Several factors impact the likelihood of both narrow and GPAI systems amplifying existing risks. These include why, when, where and how an AI system is deployed. The next section gives examples of important factors to consider when you are designing your approach to high-level risk assessment. For a practical example of how to translate this into a simple process, please refer to the AI screening tool.

We recognise that a single organisation in the AI supply chain may not have full knowledge or control over all these factors. However, the implementation practices encourage organisations to understand the AI systems they develop, deploy, or rely on, and to share relevant information across the supply chain. This will help to identify and mitigate risks more effectively.

AI system attributes and their levels of risk

AI system technical architecture

The choice of AI approach and model can cause risk as well as improve performance. For example, reduced transparency and greater uncertainty mean AI systems tend to need more careful monitoring and meaningful human oversight. They may be inappropriate for contexts where there is a legal requirement to provide a reason for an output, outcome or decision.

GPAI systems can have higher risks than either narrow AI or traditional software solutions intended for the same task.

Guiding questions (answering ‘yes’ indicates a higher level of risk)

• Is the way the AI system operates inherently opaque to the developer, deployer, user or affected stakeholder?
• Does it rely on generative AI in ways that can lead to harmful outputs?

Example

A generative AI system is used to create HR related marketing materials.

Purpose

AI systems can considerably outperform traditional software in many areas. This means that organisations are increasingly adopting AI systems to perform tasks that have significant direct and indirect impacts for people. As the impacts of an AI system rise, so too does the potential for significant harm if they fail or are misused.

Guiding questions (answering ‘yes’ indicates a higher level of risk)

• Does the AI system create an output or decision (intentional or not) that has a legal or significant effect on an individual?
• If so, will any harm caused be difficult to contest or manage redress?              

Example

A bank uses a risk assessment AI system to decide whether to grant a home loan.

Context

AI systems, being software, are scalable as well as high performing for many tasks.

However, their deployment in certain contexts may be inappropriate and their scalability may lead to widespread harms. For example, the use of facial recognition systems in public spaces where children are likely to be present, or AI systems used to gather sensitive data about Australians from social media sites.

Guiding questions (answering ‘yes’ indicates a higher level of risk)

• Does the AI system interact with or affect people who have extra forms of legal protection (such as children)?
• Will the system be deployed in a public space?

Example 

A large retailer uses a facial recognition system to identify shoplifters.

Data

AI systems’ performance is affected by the quality of data and how accurately that data represents people. Biased data can lead to poor quality or discriminatory outputs. For example, health diagnostic tools trained on historically male-dominated and non-diverse data may produce outputs that lead to under-diagnosis or misdiagnosis of women and non-white patients.

Guiding questions (answering ‘yes’ indicates a higher level of risk)

• Is confidential, personal, sensitive and/or biometric information used either in the AI system’s training, its operation or as an input for making inferences?
• Is that data non representative of the people or contexts it is making a decision about?
• Does the dataset produce decisions or outputs which could cause unwanted bias? 

Example

An SME deploys a chatbot to confirm customer contact details.

Level of autonomy

Not all automated AI systems are risky. However, systems that operate autonomously, i.e. independent of meaningful human engagement or oversight, may increase risks if they fail or are misused. Risk further increases when there is a considerable period of time between the failure or malicious use happening and the harm being recognised by responsible parties.    

Guiding questions (answering ‘yes’ indicates a higher level of risk)

• Does this system operate autonomously?
• Does the system make decisions without any meaningful human oversight or validation?

Example

A construction site deploys autonomous forklifts to move pallets in a warehouse.

System design

System based on general-purpose LLMs such as GPT5 where decision-making processes cannot be explained or understood, or highly adaptable AI tools that accept open-ended natural language instructions.

Guiding questions (answering ‘yes’ indicates a higher level of risk)

• Is the AI system designed for multiple purposes or easily adaptable beyond its intended use through interfaces that are not tightly controlled?