Judge the Chatbot by Problems Solved Not Customers Kept Away

Judge the Chatbot by Problems Solved Not Customers Kept Away

Judge a chatbot by the problems it solved and how cleanly it handed over the rest, never by how many customers it kept away from your staff. Written for owners of small restaurants, clinics, gyms and shops who are being sold a chatbot on its containment rate.

The news

Judge the Chatbot by Problems Solved Not Customers Kept Away

Verint is a large American company that sells customer service software to the big call centres run by banks, airlines and phone companies. It has just published its State of Contact Center AI 2026 report, built on a survey of 602 organisations about what they most want their customer-facing AI to do this year. Trade coverage picked the ranking apart this week, most recently on 10 September.

The list is blunt. First place went to improving containment, the share of chats the bot closes without a person ever getting involved. Handing the customer over to a human when the bot cannot help came ninth, last of nine.

In between sat driving revenue, better customer outcomes, lower costs, supporting staff, widening what the bot attempts, availability, and accuracy in eighth. Almost 90 per cent of the leaders surveyed plan to spend more on customer-facing AI in 2026. Fewer than half say AI has significantly reduced routine work for their staff, and separate consumer research cited with the report found fewer than 60 per cent of customers who noticed AI in a service conversation thought it made things better.

What this actually is

Containment is a call centre word for a simple count. It is the percentage of chats that ended without a human stepping in. It treats the customer who got their answer and the customer who gave up and closed the window as the same result.

The handover is the moment the bot passes the conversation to a person. In a good system the person sees the whole chat and picks up mid-sentence. In a bad one the customer types "speak to someone" three times and then starts again from scratch with a member of staff who has no idea what was said.

The direct answer

Should I judge my chatbot by its containment rate? No. Ask instead what share of conversations ended with the customer's problem solved, and how quickly the customers the bot could not help reached a person. A high containment rate with no handover number beside it tells you how many people went away, not how many got what they came for.

The misreading

The comfortable reading is that this is a big company problem. Banks and airlines have thousands of staff to cut, so of course they count the chats that never reach one. A clinic with two receptionists is a different world.

It is not. The companies that sell chatbots to a five-person clinic learnt their trade selling to these 602 organisations. The sales deck, the dashboard and the monthly report all lead with the same number, because containment is the one that, as the trade coverage put it, "most directly offsets headcount".

For a bank with thousands of staff, that sum is real. For a restaurant with a manager and two front of house staff, there is no headcount to offset. There is only a customer who either booked a table or did not.

The position

Judge a chatbot by the problems it solved and by how cleanly it hands over the ones it could not, and never by how many customers it kept away from your staff. A bot that cannot hand over is not saving you staff time. It is quietly losing you customers, one closed window at a time, and the report it sends you each month will call that a win.

Why it works this way

Take a dental clinic in Bangkok that gets 200 chats a month through its website and messaging apps. The bot answers opening hours, prices and parking, and it books routine cleanings. Say 140 of those 200 chats close without a person, so the vendor reports 70 per cent containment and everyone is pleased.

Look inside the 140. Around 100 got exactly what they wanted. The other 40 asked something the bot could not handle: a child with a chipped tooth, a question about an invoice, someone in pain who needs to be seen today. The bot offered a link to the help page, and they closed the window.

Those 40 count as contained. They are also the 40 most valuable conversations of the month, the emergencies and the people ready to pay for treatment now. Meanwhile the 60 chats that did reach reception arrived with no history attached, so the receptionist asked every question again and the customer wondered why they bothered with the bot at all.

That is the cost in all three places. Money, because the emergency appointments are the highest value bookings and they walked. Staff, because the receptionist spent the morning redoing work the bot had already done. Customers, because the ones who were in pain now think you are hard to reach.

Verint's own consumer research is the tell. Its State of Customer Experience 2026 report found that 69 per cent of people who currently prefer a human would switch to automated service if it could fully resolve their issue. Customers do not hate bots. They hate bots that cannot finish and will not let go.

The concession

The strongest argument for containment is that it is honest about cost and easy to count. Every chat that reaches a person costs staff time, and a bot that closes 70 per cent of chats has, mechanically, taken 70 per cent of the load. For simple, high volume questions like opening hours, a closed chat is exactly what solved looks like.

Fair, as far as it goes. But containment can be raised by making the bot harder to escape, and that is precisely the incentive a ninth place ranking for handover creates. It is cheaper to hide the exit than to build a good one, and any measure you can improve by making customers give up is not a measure of success.

What to do on Monday

First, open your own chatbot as a customer with a problem it cannot solve, ask about a refund on a specific invoice or say you need to be seen today, and count how many messages it takes to reach a person. If it takes more than two, or the person has to ask you to repeat yourself, that is the fix to make before you add a single new feature.

Second, ask the company that makes your bot for two numbers side by side: the containment rate and the resolved rate, meaning the share of chats where the customer confirmed the answer helped or completed a booking or payment. If they can only give you the first, you have learnt what they built the product to do.

Third, pull 20 chats at random from the ones the bot closed on its own last month and mark each one solved, gave up, or unclear. It takes half an hour and no technician. Those 20 transcripts will tell you more about your bot than any dashboard it produces.

The number on the sales deck measures how many customers went away. Buy the bot that measures how many came back.

If this touches how you run operations, talk to us.