A policy needs an operating home

Many organizations have an AI policy before they have a dependable way to apply it. The policy may say that systems must be tested, reviewed and monitored. A delivery team still needs to know who performs those checks, where the results live and what happens when a check fails.

The checklist below is a starting point for an internal engineering conversation. It is not a substitute for legal, regulatory or sector-specific requirements.

1. Inventory the use case and its owner

Record what the system does, whose decisions it affects, what data it uses, which model or provider is involved and who is accountable for the deployed service.

  • Name a business owner and a technical owner.
  • Describe intended users and foreseeable misuse.
  • List connected data sources, external services and automated actions.

2. Classify risk before choosing controls

A writing assistant for internal notes and an agent that can change customer records deserve different scrutiny. Document the likely impact of an error, the reversibility of actions and whether a person can detect harm in time.

  • Identify privacy, security, safety, legal and operational concerns relevant to the use case.
  • Record the reason for the risk classification.
  • Reassess after material changes to data, models, tools or autonomy.

3. Make evaluation a release requirement

Define examples of acceptable and unacceptable behavior before launch. Include ordinary tasks, edge cases, misleading inputs and situations where the system should decline or escalate.

A single accuracy percentage rarely tells the whole story. Review failure severity and whether the evaluation set resembles the work people will actually ask the system to do.

  • Version the evaluation cases and record the results.
  • Define a decision owner for release exceptions.
  • Retest after meaningful model, prompt or tool changes.

4. Enforce access and human oversight

A governance control must exist where an action happens. Apply identity checks, tool permissions, approval gates and separation of duties in the surrounding application and services.

Human review only helps if the reviewer receives enough context, has time to intervene and can actually stop the action.

5. Keep evidence that someone can use

Capture what was approved, what version was released, which tests ran, who accepted an exception and how incidents were handled. Avoid collecting sensitive material merely because it might be useful someday.

Evidence should answer a concrete question. Could an engineer explain why a tool action was permitted? Could a risk owner identify the currently deployed model and its known limitations?

6. Set a review and incident rhythm

Decide how teams will spot drift, report harmful outcomes, pause a workflow and revise controls. A checklist completed once and forgotten has limited value.

Start with one AI use case and walk it through these six areas. Where the team cannot point to an owner, a decision or a record, you have found a practical governance gap worth closing.