Agentic AI · 18 Sep 2026
Agentic AI Risk Review: Governance of Tools, Human Approvals, Memory and Audit Trails
A plain research outline for reviewing tool-using AI workflows before deployment.
AI-based agents do more than produce text; they understand objectives, plan tasks, use tools to find and retrieve data, manage context, and interact with outside systems to complete these objectives. This enhanced level of self-governance provides many benefits - however, this also alters the form of risk.
When performing a standard review of an AI model, the questions typically include if the model is accurate, fair, safe and relevant to a specific application. An agentic AI review needs to include additional questions such as: What capabilities does the system have, who gives it permission to act, what information does it rely upon, how is it monitored while operating and what record exists of its activity?
The purpose of this article is to provide a practical framework to conduct a review of four key areas of control that require specific focus: the ability to access tools, the need for human approval, the use of memory and the creation of audit trails.
An agent is a whole system; not just a model. The model's behaviour will be influenced by how it interacts with the instruction set given to it, retrieved content, tools used by the model, the model's identity, the model's memory, the orchestration layer that manages the model, and the user who interacts with the model.
While an agent would likely have low risk while interacting in a chat window, the same agent could potentially cause significant damage if it had the capability to send out emails, update customer records, process payments, write code, or retrieve sensitive data.
Therefore, the primary focus of the review should be on whether the agent has produced an incorrect answer, but also on whether it has taken incorrect actions too quickly, chained multiple correct yet individually plausible actions together to produce a negative result, or created an action that cannot easily be recreated due to the transient nature of the relevant context and/or poor logging.
goal → plan → retrieve → decide → call tool → observe result → update context → act again
The more consequential the action and the longer the loop, the stronger the controls should be.
A risk-based review framework Prior to reviewing individual controls, establish the agent's designed operating envelope. Establish documentation regarding: its purpose, intended users, those who will be impacted, and the business owner; the decisions and actions it will potentially take; the sources of data and tools it can interact with; the environments in which it can run; the potential financial, operational, legal, privacy, and safety impacts of a failure of the agent; and the conditions under which a human decision is required or the agent must automatically cease operation. Establish a basic classification system for risk (e.g., low, moderate, high, prohibited). The level of classification should be determined by the level of impact, not by the sophistication of the agent's design. An agent that is small and has access to payroll or production infrastructure may need more review than a much larger agent that simply creates internal notes. The NIST AI Risk Management Framework provides an effective lifecycle framework using Govern, Map, Measure, and Manage.
Additionally, the Generative AI Profile within the NIST AI RMF includes additional risks and recommendations for action related to foundation-model applications. The MITRE ATLAS (Adversarial Tactics, Techniques & Common Knowledge) framework is a continuously evolving knowledge base of adversarial tactics and techniques for attacking AI systems.
1. Tool access: constrain capability before improving intelligence
Tools are the agent’s effective authority. A language model may be probabilistic, but the tool it invokes can have deterministic and consequential effects. Tool access should therefore be designed around least privilege and explicit purpose.
Review questions
Is every tool necessary for the stated use case?
Is access granted to a narrow function rather than a broad shell, database, or API credential?
Are read and write operations separated?
Can the tool access only the tenant, records, repositories, or environments required?
Are high-risk operations idempotent, reversible, rate-limited, or subject to transaction limits?
Does the tool validate inputs independently of the model’s instructions?
Can retrieved content or a malicious prompt cause the agent to invoke a tool outside its intended purpose?
Are secrets kept outside prompts, memory, and model-visible logs?
Avoid giving an agent a general-purpose capability when a constrained business function will do. For example, expose create_refund(order_id, amount) with server-side policy checks instead of unrestricted access to a payments database. Use separate identities for separate agents and environments, and make permissions expire when possible.
Tool descriptions also need security review. They are part of the agent’s instruction surface and can be manipulated through configuration drift, supply-chain compromise, or ambiguous wording. Each tool should have a clear owner, version, input schema, output schema, failure behaviour, and risk rating.
2. Human approval: design meaningful control, not ceremonial clicks
Human-in-the-loop controls are valuable only when the person has enough information, authority, time, and practical ability to reject an action. An approval button that appears after the agent has already committed an irreversible change is not meaningful oversight.
Approval thresholds should be tied to impact. Examples include:
Action | Typical control expectation |
|---|---|
Draft an internal summary | No approval, with normal quality checks |
Send an external message | Review or policy-based approval, depending on audience and sensitivity |
Change a customer, financial, or legal record | Explicit approval and strong identity verification |
Transfer funds, publish production code, or delete data | Dual approval, transaction limits, and an emergency stop |
Make a decision affecting legal rights, employment, health, access, or safety | Qualified human decision-maker; agent output should remain advisory unless applicable requirements permit otherwise |
The approval interface should display the proposed action, target of that action, all associated parameters (if any), supporting evidence (if available), any uncertainties or policy flags related to this action and potential consequences/side effects. The reviewer should be able to view these elements as part of their approval process without needing to infer them from a very general description of how the agent arrived at its recommendation.
Additionally, consider how to avoid reviewer "approval fatigue." If an agent generates numerous low-value approval requests each day, then reviewers may become conditioned to automatically approve each request.
Therefore, use policy-based automatic approvals where appropriate for low-risk actions and allow human review only for those actions that are exceptional and have significant implications. Establish timeout periods for responses to approval requests, escalation procedures for non-responsive agents, roles/responsibilities for review processes and a 'kill switch' prior to implementing the agent.
3. Memory: treat retained context as data, not personality
Memory can improve continuity, but it introduces persistence, privacy, and integrity risks. Distinguish clearly between:
working memory: temporary context for the current task;
episodic memory: records of past interactions or actions;
semantic memory: facts, documents, or preferences retained for future use; and
system state: authoritative records in business systems.
An agent's memory must not be a silent authority on truth. Instead, agents must read their important facts from authoritative sources and check them when they are used. Users' statements, documents that have been retrieved, and signed records of business must not be given the same amount of credibility. When reviewing memory, look at data minimization, retention period, deletion, correction, access control, tenant isolation, encryption, and export.
Ask if sensitive information could accidentally enter memory via conversation, tool outputs or retrieved documents. Consider if users can review and correct retained facts, and if deleting something will cause it to be deleted from vector indexes, caches, backup systems, evaluation datasets and summary versions. Memory can also be poisoned.
An instruction that was malicious or wrong when it was initially saved in an interaction may influence subsequent decisions. Store information about where the memory came from (provenance), how confident you are in the memory (confidence), when the memory was created (timestamp), where the memory was obtained (source), and who authorized the memory to be saved (authorization) along with any durable memories. Before a memory can be used to perform a high-risk action, it must be validated. Untrusted content must be kept separate from system instructions. If personal data is involved, then review must comply with relevant privacy regulations and guidelines, including those found in the UK ICO AI and Data Protection Risk Toolkit. The EU AI Act may also apply based upon the purpose of the system, its use cases, its provider and deployer, and the individual(s) whose rights are impacted by the system.
4. Audit trails: make the agent’s behaviour reconstructable
An audit trail should answer five questions:
What did the agent receive?
What context, policies, tools, and model version were available?
What did it propose and why was the action allowed?
What actually happened in the external system?
Who or what approved, overrode, or stopped it?
At minimum, record a correlation ID, timestamp, user and service identity, agent and workflow version, model identifier, prompt or instruction version, retrieved sources, tool calls and parameters, policy decisions, approval events, outputs, errors, retries, and final external-system results. Capture enough information to investigate incidents without unnecessarily storing full sensitive content.
Logs must be tamper-evident, access-controlled, time-synchronised, retained according to policy, and searchable across the entire action chain. A model-provider log alone is insufficient: it may not show local routing, retrieved documents, tool authorisation, approval decisions, or the final state change.
Observability should support both real-time intervention and retrospective assurance. Alert on unusual tool sequences, repeated failures, access to sensitive records, excessive spend, policy bypass attempts, unexpected destinations, and deviations from normal task duration. Test that alerts reach a person who can act.
Testing the controls
Review documents are not enough. Test the deployed system with normal scenarios, edge cases, and adversarial scenarios. Useful tests include:
prompt injection in retrieved documents, email, web pages, and tool responses;
an attempt to escalate from read access to write access;
conflicting instructions from the user, system policy, and external content;
stale or poisoned memory affecting a later task;
a tool returning malformed, misleading, or excessively large data;
approval timeout, reviewer absence, duplicate approval, and emergency-stop paths;
partial failure after one step in a multi-step workflow;
replay and reconstruction of an incident from the audit trail; and
tenant-isolation and data-deletion tests.
Define measurable release criteria. Examples include zero unauthorised high-impact tool calls in a test set, complete traceability for every state-changing action, successful revocation of credentials, and demonstrated recovery within a specified time. Re-test when the model, prompt, tool, memory store, policy, or workflow changes not only when application code changes.
A practical go-live checklist
An agent should not go live until the owner can demonstrate that:
the use case and risk classification are approved;
tools are minimal, scoped, authenticated, authorised, and independently validated;
high-impact actions have meaningful human approval or a documented reason why they do not;
memory has clear provenance, retention, deletion, correction, and isolation controls;
state-changing actions are logged end to end and logs are protected from tampering;
monitoring, alerting, rollback, credential revocation, and shutdown procedures work;
security, privacy, legal, compliance, and domain specialists have reviewed the design where relevant; and
a named owner accepts residual risk and schedules a post-deployment review.
Conclusion
Agentic AI risk review is fundamentally an authority and accountability exercise. The key question is not whether the agent sounds intelligent. It is whether its capabilities are bounded, its decisions are supervised in proportion to impact, its memory is trustworthy and governable, and its actions can be reconstructed when something goes wrong.
The strongest programmes treat agents as continuously changing socio-technical systems. They combine least-privilege engineering, human-centred approval, privacy-aware memory design, adversarial testing, and durable audit evidence. That combination allows organisations to use autonomy where it creates value while keeping responsibility visible and control recoverable.
References
1. National Institute of Standards and Technology, AI Risk Management Framework.
2. National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1.
3. OWASP GenAI Security Project, Top 10 for Agentic Applications.
4. MITRE, Adversarial Threat Landscape for AI Systems (ATLAS).
5. European Union, Regulation (EU) 2024/1689 - Artificial Intelligence Act.
6. Information Commissioner’s Office, AI and data protection risk toolkit.
7. International Organization for Standardization, ISO/IEC 42001:2023 - Artificial intelligence management system.