AI agent observability: how to monitor agents

Testing tells you an agent works before you deploy it. Observability tells you whether it's still working after you have. Those are different problems.

A test passes on static inputs. Production brings variable inputs, changing upstream data, API failures, model drift, and edge cases no test anticipated. An agent that scored well in pre-deploy validation can start producing worse outputs a week later — and without observability, you won't know until someone complains.

Observability for AI agents means three things: tracing what happened, measuring whether outputs are good, and alerting when something degrades. This post covers each.

---

Quick answer

AI agent observability has three layers: tracing (what steps the agent took, what tools it called, what inputs and outputs moved through each step), measurement (whether outputs meet quality thresholds over time), and alerting (triggers when latency, cost, or quality cross a threshold). Most teams implement tracing first and skip measurement — that's the gap that lets degradation go unnoticed.

---

Why agent observability is harder than service observability

A traditional web service is deterministic. Given the same input, you get the same output. Observability is straightforward: log the request and response, measure latency, alert on errors.

Agents are different in three ways.

Non-determinism. The same input can produce different outputs. A latency spike doesn't mean something is broken — it might mean the model took a different reasoning path. A response that looks right might still be wrong. You can't verify outputs by comparing them to a fixed expected value.

Natural language outputs. You can't parse "was this a good summary?" with a status code check. Output quality requires either a human review process or a secondary evaluation model — neither is as simple as checking a response status.

Multi-step traces. A single agent run might involve a dozen tool calls, two sub-agent invocations, and a human gate. When something goes wrong, you need to know exactly where in that chain it happened — not just that the final output was bad.

These aren't unsolvable problems, but they mean you need more than a standard monitoring tool. You need structured traces, not just logs.

---

What to trace

A useful agent trace captures everything that happened in a single run, in enough detail to replay it if you need to debug.

The fields that matter:

• Run ID and timestamp — every run needs a unique identifier so you can correlate logs, costs, and outcomes
• Inputs — the full input the agent received, including any retrieved context
• Tool calls — which tools were called, in what order, with what arguments, and what they returned
• Sub-agent calls — if the run involved other agents, what was passed to them and what came back
• Outputs — the full output before any post-processing
• Latency — total run time, and latency per step (tool calls are usually the bottleneck)
• Token usage and cost — per-call and per-run totals
• Gate outcomes — if there was a human review step, what was the decision and was there a correction

That last field is often missing. Human gate rejections and corrections are the most valuable signal you have — they are ground truth about where the agent failed. Capturing them in structured form turns them into a feedback dataset.

---

What to measure

Tracing tells you what happened. Measurement tells you whether what happened was good — and whether it is getting better or worse over time.

Three metrics worth tracking for most production agents:

Task completion rate. What percentage of runs reach a successful output without human intervention or retry? A declining completion rate is usually the earliest signal of degradation.

Human rejection rate at gates. If an agent's outputs go through a human review step, track how often reviewers reject or significantly edit them. A rising rejection rate means outputs are deteriorating. A falling rate means the agent is improving.

Output quality score. For agents with consistent output formats — reports, drafts, structured data — you can run a secondary evaluation pass: either a lightweight model that checks against a rubric, or a human spot-check sample. This takes more setup but catches degradation that completion rates miss.

You don't need all three from day one. Start with rejection rate if you have human gates. Add task completion rate once you have enough runs to make it meaningful.

---

What to alert on

Alerts should fire before a user notices something is wrong.

Four conditions worth alerting on:

Latency spikes. If average run time increases significantly, a tool or model call is probably degrading. Alert on p95 latency, not average — averages hide tail behaviour.

Cost anomalies. A sudden cost increase usually means a tool is being called more often than expected, or token usage has ballooned. Sometimes it is a prompt change that increased context size without intent.

Rising rejection rates. If your human gate rejection rate climbs above a threshold, outputs are getting worse. Alert before it becomes a user complaint.

Tool failure rates. External tool calls fail. When failure rates for a specific tool exceed a threshold, the agent is either retrying excessively or silently failing. Alert per-tool, not just overall.

Keep alerts actionable. An alert that fires every time there is a transient API timeout trains people to ignore it. Alert on trends and thresholds, not individual events.

---

How to structure logs for debugging

A good trace should make it possible to replay a failure. That means logs need to be structured, not free text.

Store each step as a record: run ID, step type (tool call, model call, gate, sub-agent), step inputs, step outputs, latency, and any error. Link records by run ID so you can reconstruct the full trace in order.

When a run fails, you want to be able to answer: what input triggered it, which step produced the bad output, what the model was given at that step, and what it returned. Free-text logs rarely let you do that. Structured records do.

For agents handling sensitive data, also log what was and was not passed to the model — this matters both for debugging and for compliance audits.

---

The human feedback loop

Human gate rejections are often treated as friction — something to reduce. They are actually your best observability data.

When a reviewer rejects or edits an agent output, that is a labelled failure: you know exactly what the agent produced, what was wrong with it, and (from the edit) what would have been right. Systematically capturing those corrections gives you a dataset that can be used to improve prompts, update evaluation rubrics, and identify recurring failure patterns.

Set up a lightweight process: when a reviewer makes a significant correction, flag the run for review. Review flagged runs weekly. Look for patterns — same tool failing, same input type producing bad outputs, same section of the output consistently edited. Fix the pattern, not the individual instance.

This is the difference between an agent that stays at the same quality after deployment and one that improves.

---

Worked example: a finance report agent

A finance report agent runs weekly, pulls data from a spreadsheet tool and an accounting API, and produces a variance commentary for review. It has a human gate where an analyst reviews and approves each report before it is sent.

A minimal observability setup:

Traces: run ID, inputs (reporting period, data source), tool calls (spreadsheet pull, API call), output (draft report), latency per step, token cost.

Measurement: analyst rejection rate (target: under 10%), tracked weekly. Monthly spot-check on output quality against a rubric — is the variance analysis correct, is the commentary accurate.

Alerts: latency alert if any run exceeds 3× baseline. Cost alert if token usage increases by 50%+. Rejection rate alert if the weekly rate exceeds 15%.

Feedback loop: rejected reports are flagged. Monthly review of flagged reports looks for patterns — most often, the agent misreads a specific data format from the accounting API. Each pattern gets a prompt update.

That is it. Nothing exotic: structured traces, three metrics, four alert conditions, a monthly review cycle.

---

Where Envelope fits

Envelope's Runs view gives you a structured trace for every agent execution — inputs, tool calls, step latency, and token usage, all in one place. Human gate decisions are logged alongside the run, so rejection rates are visible without a separate tracking system. The design captures what each agent can access before you build it, which makes trace analysis cleaner — you already know what tools were in scope.

→ How to test an AI agent before deploying it → Human-in-the-loop: how to design approval gates into a multi-agent workflow → AI agent security: what to lock down before you deploy → How to design AI agents: a practical guide

---

Frequently asked questions

What is the difference between AI agent testing and observability?

Testing is pre-deploy validation — you run the agent against known inputs and check it produces acceptable outputs before going live. Observability is post-deploy monitoring — you track what actually happens in production, including inputs you did not anticipate, and detect when quality degrades over time. Testing is a gate you pass once. Observability is an ongoing practice.

Do I need a specialised tool for AI agent observability?

Not necessarily. Structured logging to any queryable store — a database, a data warehouse, a log aggregator — covers the basics. Purpose-built LLM observability tools add visualisation and evaluation features that save time at scale. If you are running a small number of agents at moderate volume, structured logs and a simple dashboard are often enough to start.

How do I measure output quality if outputs are natural language?

The most practical approach is to use human gates as your quality signal — track how often reviewers accept, edit, or reject outputs. For agents without human gates, a secondary evaluation model (a cheaper LLM that checks outputs against a rubric) can provide a quality score. Human spot-check sampling is also useful for catching edge cases that automated evaluation misses.

What is the first thing to instrument when going to production?

Structured traces for every run, with inputs, outputs, tool calls, and latency. Even if you do not analyse them immediately, having the records means you can debug failures retrospectively. The second thing is human gate rejection rates if you have gates — that is the fastest available signal on output quality. Add alerting once you have a baseline to alert against.

How do I know if my agent is getting worse over time?

Track key metrics — task completion rate, rejection rate, output quality score — on a consistent cadence, weekly or monthly. A rising rejection rate or falling completion rate signals degradation. Also watch for latency and cost anomalies: they often indicate something upstream has changed (a tool API, a data format, a model update) before it shows up in output quality.

What should I log from human gate decisions?

At minimum: the original agent output, the reviewer's decision (approved, edited, rejected), and the edited version if they made changes. Also useful: how long the reviewer spent, and any free-text notes if your gate interface supports them. The edited versions are the most valuable — they are ground truth for what a correct output looks like, and the basis for improving prompts over time.