Measuring it

Is your AI chatbot actually helping?

The number most vendors lead with is deflection rate - the share of conversations that never reached a human. It's the easiest metric to move and the easiest to fool yourself with, because a customer who gave up looks exactly like a customer who was helped. Here's what to watch instead.

Download Free When Not to Use AI

The problem with deflection rate

Deflection counts what didn't happen. A conversation that ended without a human could mean the customer got a clear answer in twenty seconds - or that they typed the same question three times, gave up, and bought from someone else. Those are opposite outcomes and the metric scores them identically. Worse, it scores the second one as a success while your revenue quietly leaves. Any measure you can improve by making it harder to reach a person is not measuring help.

Four measures that tell you the truth

Measure	What it tells you	Where to find it
Satisfaction on AI-handled chats	Whether people felt helped, not merely intercepted. Compare it against human-handled chats.	Ratings and comments in the dashboard
Handoff rate, and its reasons	Where the AI runs out of road. Rising handoffs on documented topics means a documentation gap.	Conversation history and tags
Repeat contacts	The honest counterweight to deflection - if people come straight back, the first answer failed.	Session history by visitor
First response time	Whether the AI freed your agents up, or just added a step before the same wait.	Dashboard response-time figure

Automatic tags on finished conversations make most of these answerable by filtering rather than reading.

Take a baseline first

Record satisfaction, response times and your topic mix before switching the AI on. Teams that skip this end up arguing from memory, and memory is generous about how things used to be.

Read the comments

A rating tells you the score; the comment tells you why. Ten free-text complaints about the bot are worth more than a dashboard trend, because they name the specific failure you can fix.

Read the bad ones yourself

Once a week, open the lowest-rated AI conversations and read them end to end. Nothing else surfaces the confidently-wrong answer that a metric will happily average away.

The decision the numbers serve

Be willing to turn it off

Measurement only means something if a bad result can change what you do. If satisfaction drops on AI-handled chats and repeat contacts rise, the honest response is to narrow where the bot runs - or switch it off for that audience and keep the parts that clearly work, like drafts for your agents. A metric nobody would act on isn't a metric, it's reassurance.

Included

Satisfaction ratings with comments, first-response times, filterable history and missed chats - part of the product, not an analytics add-on.

Your data

It's your database, so you can query it directly for anything the dashboard doesn't show.

Why is deflection rate a misleading measure?

Deflection counts conversations that did not reach a human. It cannot tell the difference between a customer who got their answer and one who gave up in frustration and went to a competitor. Both look identical - and the second is a worse outcome than a ticket.

What should we measure instead?

Satisfaction on conversations the AI handled, how often it hands off, whether customers come back with the same question, and whether your agents' response times improved. Together those tell you whether people were helped, rather than merely intercepted.

Is a high handoff rate a failure?

Not necessarily. Handing off quickly on questions it should not attempt is the AI working correctly. What matters is whether handoffs happen for the right reasons - a rising handoff rate on questions your documentation covers is the signal worth chasing.

How long before we can judge it?

Take a baseline before switching the AI on - satisfaction, response times, the mix of topics you get. Without that you are comparing against memory. A few weeks of traffic afterwards is usually enough to see a direction.

What does ZChat report out of the box?

Customer satisfaction ratings with the comments people left, average first-response time, conversation history with automatic tags you can filter by, and missed chats. That is enough to answer most of the questions on this page without extra tooling.

Self-hosted customer support software

Deploy live chat on your own terms, not on someone else's pricing model.

ZChat gives you the installable server, web dashboard, website widget, and desktop agent tools in one self-hosted product, with live chat free to run. Run it on infrastructure you trust and connect AI only if and how you want it.

Download ZChat See Product Demo

Deployment

Install on Windows or Linux, behind IIS or Nginx, in a VM, or in Docker if that fits your stack.

Commercial model

Live chat is free with unlimited agents. Only the optional AI assistant is billed, per agent per month.

AI Flexibility

Use Ollama locally or connect OpenAI and Anthropic with your own provider accounts.