Skip to content

Independent standards discovery and compliance platform

How we work
Compliance practice

AI Impact Assessment Example: What to Record Before a Pilot

See a worked AI impact assessment for support-ticket triage, with practical evidence, launch criteria and a review checklist based on ISO guidance.

AI Impact Assessment Example: What to Record Before a Pilot

Suppose your team wants AI to sort incoming support tickets by urgency. The prototype saves time in a demo, but a ticket marked “routine” can wait hours before anyone reads it. An impact assessment makes the team decide who could be affected by that delay, which evidence is worth collecting and what would stop the pilot.

Disclosure: Some catalogue guides linked here offer an ANSI Webstore purchase option. European Standards may earn a commission on qualifying purchases.

ISO/IEC 42005:2025 provides guidance on AI system impact assessment. This worked example is a starting point for a fictional pilot. Tailor the scope, controls and approval criteria to the system you actually plan to use.

Describe the decision the AI will influence

Write down the input, output, intended use and decision owner. Here, the AI reads the ticket subject and message and suggests a queue. An agent can override the suggestion. The support manager owns the pilot. State whether messages are sent to an external provider and how much ticket content it receives.

A support analyst compares ticket categories and urgency flags across two screens.
Review missed urgent tickets by category instead of relying on one overall accuracy score.

Map who could be affected and how

List the people affected by each plausible failure. A customer whose urgent report is downgraded may wait too long. A customer writing in a language poorly represented in your test data may see inconsistent routing. Support agents may over-rely on a confident-looking label. For each case, record the impact, what could cause it and how you would detect it.

Define what evidence would justify going live

Build a test set with historical tickets that the team can use appropriately. Have agents label urgency independently and resolve disagreements before comparing the AI output. Include rare urgent tickets, ambiguous wording and each language the pilot will support. Examine missed urgent tickets by category, alongside queue time and reviewer overrides. A high overall accuracy score can hide the failure that matters most.

Write a release decision before testing. For example, restrict the pilot to one queue, require human review of uncertain cases and keep the normal urgent-report route available. Set a stop rule for a serious missed escalation, identify who can pause the pilot and document how the team returns to manual routing.

A support team reviews evidence and decides whether to launch the pilot.
Record who can stop the pilot and how the team will return to manual routing.

Set the conditions that trigger reassessment

Review the decision after a model or prompt change, a new language or queue, or an incident that changes your understanding of the risk. ISO/IEC 23894:2023 gives broader AI risk-management guidance. ISO/IEC 5338:2023 is relevant when assigning ownership across the AI system life cycle.

Keep the test set version, results, sign-off and pause criteria together. The audit evidence checklist shows how to make that record retrievable. The internal audit checklist shows how an auditor can sample decisions after launch. See the official ISO/IEC 42005 record for the source standard.

Continue reading

Related standards and compliance insights