An AI agent is not a web request. It plans, calls tools, retries, reads untrusted content and changes state. When it fails, a stack trace is not enough: you need to reconstruct what it saw, what it decided and what it did. That is agent observability.

This guide covers the three layers that matter — logs, traces and audit trails — and the fields to capture so an incident review is possible. For the security-monitoring angle, see LLM Security Monitoring & Observability.

Why traditional APM is not enough

APM tools answer "which service is slow?" Agent failures are different: the service is fine, but the model chose the wrong tool, trusted a poisoned document, or looped on a failed call. You need semantic context around each step, not just latency and status codes.

Layer 1 — Logs

Log one structured event per step, not free-text lines. A useful schema:

  • trace_id / span_id — correlates the whole task and each step.
  • agent_id and caller_id — who acted, and on whose behalf.
  • model and version — provider, model name, temperature, max tokens.
  • prompt hash — not the raw prompt if it may contain secrets or PII.
  • tool name — and a redacted parameter summary.
  • policy decision — allowed, denied, needs approval, and why.
  • result status, latency, token counts and cost.
  • output hash — to detect changes without storing sensitive text.

Keep raw prompts in a separate, access-controlled store with short retention. Logs are for operations; raw content is for investigations.

Layer 2 — Traces

A trace represents one user task. Each model call, retrieval, tool call and policy check is a span. That structure lets you see the actual path, including retries and branches.

Minimum spans for an agent task:

  1. input validation and policy check;
  2. model planning call;
  3. retrieval or document read, with source provenance;
  4. each tool call, including arguments and outcome;
  5. final answer generation and output filtering;
  6. post-task audit write.

Store the trace ID in the user-visible response or support ticket. It turns "the agent did something weird" into a reproducible case.

Layer 3 — Audit trail

An audit trail is not a log with a longer retention. It is an append-only record of security-relevant decisions: who granted a permission, which policy blocked a call, which human approved an irreversible action. Requirements:

  • append-only and tamper-evident (hash chain or WORM storage);
  • least-privilege access — the agent cannot edit its own audit records;
  • defined retention — often 90 days hot, 1 year cold;
  • time synchronization and a stable schema;
  • export to the SIEM or data lake.

What to alert on

Alerts should be about behaviour, not just errors:

  • spike in denied tool calls for one caller;
  • repeated retries on the same tool or document;
  • prompt-injection indicators from retrieved content;
  • cost or token usage far above the task baseline;
  • agent attempting a tool outside its role;
  • audit write failure — treat it as a security incident.

Common mistakes

1. Logging raw prompts with secrets. Redact and hash; keep sensitive content in a separate store.

2. No correlation ID. Without a trace ID, multi-step failures are impossible to reconstruct.

3. Sampling away errors. Sample successful traffic; keep all failures and all policy denials.

4. No policy context. A denied call without the role, mask and rule is not actionable.

5. Letting the agent write its own audit. The audit path must be outside the agent's authority.

FAQ

What is AI agent observability? The practice of recording and correlating an agent's prompts, model calls, retrieval, tool calls, policy decisions and outputs so behaviour can be monitored and reconstructed.

What is the difference between logging and tracing? Logs are events; traces connect events into a causal path. You need both.

What should an agent audit trail contain? Security-relevant decisions and actions: identity, permission, tool call, approval, outcome and timestamp, in an append-only store.

How long should we retain agent logs? Define it by risk and regulation. A common starting point is 90 days hot for traces, longer cold for audit records.

Next step

Start with one trace per agent task and the fields above. Then add policy decisions and an append-only audit path. ReskPoints implements agent action logging with multiple destinations; the AI agent security hub covers identity and least privilege.