Applying and adopting the standard through examples

The range of applications of AI is effectively infinite. While we can’t give guidance on how the standard might apply to every use case, we can use examples to illustrate how you can use the guardrails to manage the risks and benefits of a specific AI system.

We’ve chosen 4 examples to show how individual guardrails might be apply in different use cases. The examples explore how organisations may use particular guardrails as part of their overall approach to deploying AI systems. The examples show that the guardrails can be applied in different situational contexts, for different technologies.

These examples are not intended to represent a comprehensive application of all relevant guardrails, responsibilities or other legal obligations that may be relevant for the specified use cases. They are to provide examples of how the guardrails can be applied in a selection of fictional examples.

Example 1: General-purpose AI chatbot

A detailed example representing a common use case for organisations of all sizes, across all sectors. Due to the growing ubiquity of this technology, we’ve provided extra detail on how an organisation could adopt a range of guardrails. As a point of contrast, this example includes potential outcomes where safe and responsible AI methodologies are not followed.

Example 2: Facial recognition technology

A simplified example on the use of facial recognition technology. It illustrates the use of the guardrails to decide that non-AI-based solutions will better achieve strategic and operational goals.

Example 3: Recommender engine

A simplified example of a common use case in which a recommender engine is used to improve customer experience and meet organisational goals. It includes reference to a court case in which a business using this kind of technology was ordered to pay a substantial financial penalty for not meeting legal obligations.

Example 4: Warehouse accident detection

A detailed example to outline obligations for testing of AI systems. In this example, we offer guidance on linking areas of concern with acceptance criteria. It covers testing at different stages during the AI system and governance lifecycle, due to the specific and technical nature of meeting relevant guardrails.

Example 1: General-purpose AI Chatbot

NewCo background

NewCo is a fast-growing B2C company with 50 employees, selling a range of products in a niche market. It has an annual turnover of $3.5 million.

The company is approaching a major product launch that they expect will create a significant increase in demand. NewCo’s head of sales proposes to use the latest advances in AI and procure a new chatbot for their website. The chatbot would engage with customers to answer the most commonly asked questions. The company expects the new product to sell over 10,000 units in the first month because of an aggressive social media strategy featuring early-bird discounts.

The new chatbot is meant to reduce the amount of time customers wait for a phone operator by shifting those with routine queries to the online chatbot. This should reduce the need to expand phone support and allow employees to spend more time on complex tasks. The most common customer queries include delivery times, returns and the application of time-limited discount codes.

The head of sales suggests that a chatbot based on general-purpose AI would help the company respond to and resolve customer queries faster, leading to improved customer satisfaction scores (CSAT). CSAT scores are considered lead indicators for revenue growth goals, so NewCo hopes that a suitable customer query chatbot would also support growth in sales.

Case study: Moffatt v Air Canada 2024 BCCRT 149

Air Canada deployed a chatbot on its website which made statements to a customer about the airline’s bereavement fares. These statements were inconsistent with Air Canada’s policy, to which the chatbot had provided a link.

The customer sought a refund through legal proceedings. Air Canada claimed that the chatbot was a 'separate legal entity that is responsible for its own actions' and the customer was not entitled to a refund according to its bereavement policy.

The tribunal rejected these arguments and found Air Canada responsible for all information provided on its website, whether from a static page or chatbot. Air Canada was found to have had a duty of care to take reasonable steps to ensure that information was accurate.

There are similar protections in Australia for interactions with chatbots as part of an organisation’s customer service offering (Lifshitz and Hung, 2024).

NewCo’s use of the standard: a comparison 

NewCo wants to procure a generative AI chatbot with the promise of:

  • reduced customer wait time
  • reduced customer service phone support time for staff.

The table below compares what happens when NewCo follows the Voluntary AI Safety Standard, and what happens if it chooses not to follow the standard. 

Actions and outcomes Does not follow the standard Does follow the standard
Organisational-level actions

Head of sales (HOS) conducts online research into potential developers – decides an off-the-shelf solution will allow NewCo to quickly launch and use the AI system.

Developer selected and ‘NewChat’ launched within a week in parallel with the new product launch.

Standard identified as basis for effective governance of the new chatbot.

NewCo commits to organisational-level safe and responsible AI use that: 
System-level actions None.

HOS takes overall responsibility for developer selection, contract negotiation, implementation and monitoring. She has recently undertaken training on deploying responsible and safe AI systems (guardrail 1).

HOS engages with internal and external stakeholders to understand potential impacts and harms (guardrail 10).

HOS tests the system with a planned promotional discount. The test detects unwanted bias in the outputs and the agreed fairness metric in the testing criteria is not met (guardrail 4).

HOS conducts a risk assessment. Some risks and mitigating actions are identified (including NewCo modifying the system to minimise bias). Based on the risks HOS decides that only internal use of AI system as appropriate at this stage (guardrail 2).
Outcomes

System behaviour and impacts

NewChat holds convincing conversations with users and asks them for personal information, including gender.

To maximise sales, NewChat offers customers discounts above agreed promotional rates.

Customer Service team is unaware that NewChat is offering customers discounts and refuses to apply them to purchases at checkout. NewChat is only offering these discounts to people who report their gender as ‘male’. It does not otherwise offer any discounts.

Because of a viral Reddit thread, thousands of customer complaints accuse NewCo of discrimination. They demand NewCo extend the chatbot-generated rate to all purchasers.

Customer Service team overwhelmed with level of complaints from people whose discounts have been refused as well as those claiming they have been discriminated against.

Harm to people and organisation

Personal information is collected without being reasonably necessary for its functions.

People who don’t report their gender as ‘male’ miss out on the discount.

Financial, legal and reputational risks

Customer satisfaction score drops significantly.

Negative global media news coverage of incident.

Potential breach of consumer laws for misleading or deceptive conduct in not honouring the offered discount.

Potential breach of privacy laws for the collection of personal information that was not necessary for its functions.

Potential complaints made to relevant regulatory bodies for unlawful discrimination based on a protected attribute (gender).

Successful product launch

Customer Service teams use general-purpose AI as an internal resource to find relevant company documentation to answer customer queries more quickly.

Customer satisfaction scores increase.

Employee productivity increases.

Example 2: Facial recognition technology

EcoRetail background

EcoRetail has 20 permanent employees and over 100 casual workers across its nationwide chain of 15 stores.

Its brand is heavily tied to advancing social good, including diversity and inclusion.

Their customer base includes people from many different demographic groups. 

EcoRetail’s AI system vendor, FRTCo Ltd, suggests installing facial recognition technology, which it states can: 

  • identify known shoplifters and limit losses from shoplifting 
  • identify other criminal activities (such as physical violence) to support staff safety. 

Facial recognition technology (FRT) is a type of AI that remotely captures sensitive biometric data to verify, identify or analyse people. This functionality poses heightened privacy and discrimination risks to human rights. While there is currently no specific Australian law governing the use of this technology, the Australian Government is considering the need for new guardrails for FRT as part of its broader Privacy Act reform process. 

How EcoRetail uses the standard 

EcoRetail wants to procure FRT to:

  • accurately identify and deter shoplifters
  • prevent violence, protecting customers, staff and assets.

They use the guardrails to inform their actions.

 

Guardrails Actions
Guardrail 1: Establish, implement and publish an accountability process including governance, internal capability and a strategy for regulatory compliance.

EcoRetail holds discussions with FRTCo Ltd (AI system vendor) to ensure that FRT aligns with business objectives (minimising loss from shoplifting) and strategic goals (act in accordance with Australia’s AI Ethics Principles and Australian legislation). 

To understand how the use of FRT aligns with EcoRetail’s organisational strategy and risk appetite, EcoRetail evaluates the following characteristics of the technology and how it will be deployed: 

  • Spatial context of deployment: commercial, publicly accessible space.
  • Functionality of the FRT: facial identification – comparing a single face in the store to a large database of many faces to find a match. FRTCo Ltd is unable to provide detail as to where they have obtained the dataset, how representative it is or whether they followed privacy guardrails. 
  • Performance: 99% performance accuracy applied to the estimated 300 people per day (foot traffic across all EcoRetail stores) equates to the potential for 3 people per day to be incorrectly identified. 
  • Outcomes: the FRT would impact people’s rights (including privacy of sensitive information and the potential for arbitrary detainment) and people’s ability to access goods and services. 
  • Free and informed consent: signs posted at store entry may not be sufficient for express and sufficiently informed consent.
Guardrail 10: Engage your stakeholders and evaluate their needs and circumstances, with a focus on safety, diversity, inclusion and fairness. 

Senior leaders at EcoRetail held consultations with permanent and casual staff to understand how the use of FRTCo Ltd’s FRT system might impact them and their customers.

During the consultation, staff received FRTCo Ltd’s reports on the accuracy of its product.

The staff asked if the accuracy rate applied equally across different demographic groups and discovered that the accuracy rate reduces to 95% for particular racial groups. FRTCo Ltd was unable to give any detail of methodologies used to reduce outcomes based on unwanted bias or show the representation of its dataset.

Although the staff indicated that they were sometimes concerned for their safety, they did not feel that the potential benefit from the AI system outweighed the level of surveillance.

Outcomes

EcoRetail decided that using FRT would not align with its strategic goals, risk appetite and legal obligations.

Collecting sensitive biometric information posed too great a risk to the organisation from a legal perspective. EcoRetail also recognised that the scale and impact of potential harm to customers, particularly to those incorrectly identified as shoplifters, was too great. 

The possibility of reputational damage, exacerbated by potential regulatory activity for discrimination, was likely to have negative commercial outcomes.

Example 3: Recommender engine

TravelCo.com background

TravelCo.com is a global hotel booking app that is paid by commission. Hotels will pay TravelCo.com a fee every time a user clicks on the offer for their hotel.

Hotels are also able to pay a fee so their hotel appears higher up in search results. 

To meet shareholder expectations, TravelCo.com wants to increase market share by telling customers that they can get the cheapest possible price for the same hotel using the TravelCo.com app. 

Search results rely on recommender engines as an underlying technology. These use AI to analyse an individual’s web browsing activities to give content suggestions based on inferences made about their demographic characteristics, behaviours and interests. 

TravelCo.com has engaged a company called XYZ to supply their recommender engine.

Case study: Australian Competition and Consumer Commission v Trivago N.V. (No 2) [2022] FCA 417

Trivago stated it could help consumers find the ‘best deal’ or cheapest price by comparing hotel rates on different websites. 

The algorithm driving Trivago’s recommender engine did not use the price of the room as the sole factor in ranking search results. Consumers were not aware that another significant factor was the value of the fee paid by the third-party booking site to have its search result ranking improved.

Consumers were frequently not shown the cheapest price for a hotel in their top search result. In some cases, they were overpaying for the hotel listed as compared to other booking sites.

Trivago was ordered to pay $44.7 million in penalties because of the Federal Court finding it had misled consumers (Federal Court of Australia 2022).

 

How TravelCo.com uses the standard 

TravelCo.com wants to procure a recommender engine to:

  • meet shareholder expectations of increasing market share
  • improve capabilities with AI and data analytics.

They use the guardrails to inform their actions.

Guardrails Actions
Guardrail 2: Establish and implement a risk management process to identify and mitigate risks.

XYZ notifies TravelCo.com of the challenge in providing a real-time ‘cheapest price’ because of the large and dynamic dataset of hotel pricing.

It would take at least 10 seconds to return a search result, which is not in line with customer expectations for instant information.

To minimise lag time for the customer, XYZ suggests updating a static version of the data every 3 hours. 

As a B2C organisation, TravelCo.com identifies the regulatory risk related to consumer law – that advertising cannot be misleading or deceptive. The pricing at the time the customer searches may no longer be the cheapest option, because of changes since the last update.
Guardrail 6: Inform end-users regarding AI-enabled decisions, interactions with AI and AI-generated content.

The recommender engine uses several factors to create rankings of search results, including alignment to TravelCo.com’s business model.

Another risk identified during the assessment is that the website does not clearly state that ranking of results is influenced by the commercial arrangements TravelCo.com has with the hotels.

Customers could assume that the highest ranked result is the cheapest and therefore overpay.

Outcome

TravelCo.com decided to change its advertising materials from ‘cheapest’ or ‘best’ price to stating that it provides comparisons only.

TravelCo.com also decided to include a clear and prominent notice with every search that reflects its commercial arrangements with hotels. 

Example 4: Warehouse accident detection

ManufaxCo background

ManufaxCo is a manufacturing company that has built an AI system in house called Safe Zone. SafeZone monitors high-risk factory environments for potential safety hazards and alerts staff to hazards in real-time to prevent accidents and keep workers and assets safe.

SafeZone combines computer vision and Natural Language Processing (NLP) technologies. Cameras installed throughout the factory capture real-time video feeds, which AI analyses to detect safety hazards like spills, obstructions, or people entering unsafe zones. The NLP component allows the system to understand and process verbal commands or alerts from workers, creating a more interactive and complete safety monitoring approach.

How ManufaxCo uses the standard

ManufaxCo uses the guardrails to inform their actions.

Guardrails Actions
Guardrail 2: Establish and implement a risk management process to identify and mitigate risks. ManufaxCo has carried out a risk assessment and found a set of concerns. The concerns (effectiveness and reliability, fairness, and privacy) are not an exhaustive list for this AI system. For example, they do not cover concerns about accountability or potential misuse. 
Guardrail 4.2: Commit to specifying, justifying and documenting acceptance criteria needed for the potential harms to be adequately controlled. 

For each concern, the accountable owner in ManufaxCo sets acceptance criteria to control for the anticipated harms.

1. Effectiveness and reliability: system errors are highly impactful – both false positives (which stop work) and false negatives (where an accident may occur).

Set appropriate thresholds such as: 

  • fraction of hazards detected (recall) must be greater than 0.9
  • frequency of unnecessary stop-works (false discovery rate) must be less than 0.3
  • raise an alarm if a camera view is significantly obstructed for more than 20 seconds.

The system must fully integrate with existing safety guardrails and communication systems, with no reported compatibility issues during a 2-week trial period.

The system must have an uptime of at least 99.5%, as measured over a 3-month period.

At least 80% of staff must rate the system's user interface as ‘easy to use’ by in a user satisfaction survey.

2. Fairness: concerns the safety benefits offered by the system may not apply equally to all workers in the environment. For example, if there is a representation problem in the data.

The NLP component must correctly understand and process commands or alerts from workers with at least a 90% accuracy rate across different accents and dialects.

3. Privacy: worker stakeholders raise concerns about their privacy at work.

The system must pass a privacy compliance audit, ensuring adherence to relevant privacy regulations for handling video feeds and worker data.

The system is designed and built to meet these criteria. A third-party vendor supplies voice recognition models for controlling the system. The hazard detection model is trained on historical data. Under normal operating conditions, occurrences of hazards may be rare, so controlled simulations of hazards augment the data.
Guardrail 4.3: Testing of AI systems or models to determine performance and mitigate any risks

ManufaxCo develops a test plan to evaluate the system.

They acquire the testing data to evaluate against the acceptance criteria under controlled conditions. This includes evaluations specifically for the acceptance criteria:

  • hazard detection rates – tested using performed simulations for different types of hazard
  • false positive count – tested on operational data collected during a small pilot under full human oversight
  • functionality of failure alert system – inducing camera failures or placing obstructions.

They design tests to identify implementation errors and system problems:

  • a team is assigned to design edge cases such as placing equipment to obscure potential hazards
  • tests are performed to ensure voice control is performing well enough in various working conditions of machinery
  • interactions with employees are observed to find out whether they are interacting correctly with the system and as it was intended in the initial design and tests.

The tests find the system is functioning as intended, with the exception that initial testing reveals a problem with the false positive rate. The system has many false alarms during normal safe operation. The findings are reported, summarising the objectives, methods and metrics used.

The accountable owners assign the development team to investigate, and they determine that the problem is because of differences in the environment between the training data and the pilot plant (such as lighting, camera angles, wall colours, specific equipment models). They acquire an updated dataset and re-test the system. Over this period, workers using the system report feedback about voice recognition issues, particularly for workers from multicultural backgrounds. The owners address this by acquiring and swapping in a voice recognition model from a different vendor with models that perform well across a more diverse set of accents. The accountable owners review the reporting to confirm the mitigations have been effective, and they approve the system for deployment.
Guardrail 4.4: Commit to implementing robust AI system performance monitoring and evaluation, and to ensuring each system remains fit for purpose.

A month into deployment, ManufaxCo's monitoring indicates a reliability problem with the system. Timely investigation reveals a camera calibration issue that hardware configuration and updating the computer vision pipeline’s preprocessing stage fixes.

ManufaxCo then rolls the system out across multiple warehouses. Initially, the system proves effective in identifying common safety hazards, leading to a noticeable reduction in accidents and meeting all its acceptance criteria. 

However, as the warehouse operations expand to include new types of machinery and materials, the system experiences a dataset shift. It fails to recognise new hazards that were not present in its training data, resulting in several near-miss incidents that are reported through the feedback channels. 

The accountable owners examining the monitoring recognise this problem, and they assign the development team to address. The development team updates the training dataset again to include the new hazards. The model is updated and re-tested. 
Guardrail 4.5: Commit to regular system audits for ongoing compliance with the acceptance criteria (or justify why audits aren’t needed).

Considering the serious safety impacts of the systems, the accountable owner requests another independent internal technical team do an assessment before the final roll out across all warehouses. 

During the assessment of the design documentation and pilot monitoring logs, the independent assessors identify and recommend better camera placement to minimise chances of blind spots caused by machines and their operators. ManufaxCo applies this recommendation as an update to the existing installed systems and records it as a consideration for any future deployment in other warehouses. 
Guardrail 4.1: Commit to a robust process for timely and regular monitoring, evaluation and reporting of AI system performance. Given this is a complex new system that could have significant safety impacts, accountable owners decide to audit the system and its governance in 6 months. At this stage there will be an existing operational track record.