In this article (8 sections)
In short: an AI agent is working if it resolves the task without errors, without help and at a lower cost than doing it by hand. You measure it with four numbers: resolution rate, handoff rate to a person, time saved and errors. Set the starting point before the agent goes live: without the "before" figure, the "after" figure tells you nothing.
Measuring isn't a formality. The NIST AI Risk Management Framework includes measurement as one of its four functions, alongside govern, map and manage (NIST, AI Risk Management Framework). The practical idea for a business is simple: what you don't measure, you can't correct.
The four metrics
Metric | What it measures | How it's calculated | What to look for |
|---|---|---|---|
Resolution | Cases the agent closes on its own | Cases closed without help over total cases | That it rises week over week |
Handoff | Cases passed to a person | Handed-off cases over total cases | Why they're handed off: missing information or permission |
Time saved | Hours your team no longer spends | Minutes per task before and after, times number of tasks | That it is worth more than the agent's monthly cost |
Errors | Wrong answers or actions | Reviewed cases with a failure over reviewed cases | That it falls and there are no serious errors |
How to measure without overcomplicating it
- Measure the before. For two weeks, note how many inquiries or tasks there are, how long each takes and how many are resolved well.
- Review a sample. Each week, read a few dozen conversations or actions at random and mark which were right. It is the most direct way to see errors.
- Log handoffs with their reason. If the same one repeats, there is something to teach the agent.
- Compare with cost. Add up the agent's monthly cost and compare it with the hours saved. More on costs in how much an AI agent costs.
What not to measure alone
The number of messages handled is misleading: an agent can answer a lot and resolve little. The same goes for speed without quality. Always look at resolution together with errors, and be wary of an agent that never hands off: it is probably improvising.
When to adjust or stop
If resolution stalls, there is usually missing information or a business rule. If there are serious errors, restrict the agent to actions with approval while you fix it. Permissions and the action log help here: we cover them in is it safe to give an AI agent access to your systems.
Atomix builds AI agents for business with a dashboard to review conversations and actions, and builds a free working demo in 24 to 48 hours.
Frequently asked questions
What is a good resolution rate?
There is no universal number: it depends on the task. For simple inquiries it can be high, and for complex cases you expect more handoffs. Compare against your own starting point and look for an improving trend.
How often should I review?
Every week during the first month, then monthly. If you change prices, policies or products, review again.
Who should review the results?
Someone who knows the work, such as the area lead. No technical profile is needed: it's about reading cases and marking whether they were right.
If you want to define the metrics for your case, book a call.