The Two-Week Chatbot Probation Review: What to Read, Count and Change

Treat the first fourteen days as a probation period

A newly launched customer chatbot needs supervision, just like a new front-desk employee. The first two weeks are an opportunity to discover which enquiries it handles well, which instructions are unclear and where a human must step in. The purpose is not to produce a flattering performance score. It is to decide whether the assistant is helping customers without creating extra work or making unauthorized promises.

Set a clear review period of 14 calendar days. During that time, keep its responsibilities narrow: published information, basic service descriptions, routine enquiries and requests that can be safely passed to staff. Avoid giving it authority over refunds, unusual bookings, individual assessments or disputed payments while its instructions are still being refined.

• Name one staff member responsible for reading conversations.
• Keep a record of changes to the briefing.
• Record the date of each policy or price update.
• Keep a clear human contact route available throughout probation.

Days one to three: Read the conversations before counting

Start by reading a small, consistent sample of conversations from each day. If there are fewer than 20 conversations, read all of them. If there are more, choose 20 spread across different days and enquiry types. Include short exchanges, conversations that ended without a booking and conversations that required staff attention. A collection of easy opening questions will give an incomplete picture.

For every conversation, ask four things: Was the answer factually correct? Did it address the actual question? Was the tone appropriate? Did it stay within the assistant's authority? Mark each problem with a short note. A tutoring studio, for example, may find that the assistant explains lesson formats accurately but gives an overly definite answer about whether a particular student is suitable for a class.

• Read the customer's original question, not just the assistant's final reply.
• Check important facts against the current business source.
• Mark incorrect, incomplete and unnecessarily complicated answers separately.
• Record any customer who appeared to need a person but received no handoff.

Days four to seven: Count the work the assistant creates

Once the conversation sample is understood, count a few practical categories by hand. Keep the categories simple enough for an owner to maintain in a spreadsheet or notebook. Useful measures include total conversations reviewed, correct direct answers, human handoffs, unanswered questions, inaccurate answers and conversations that ended without a clear next step. Record the counts as numbers, not impressions.

For example, imagine a bike repair shop reviews 40 conversations. The assistant answers 24 routine questions correctly, sends 10 to a mechanic and gives six incomplete responses. Those figures reveal where attention is needed. The ten handoffs are not automatically failures: some repair questions should go to a mechanic. The six incomplete responses deserve investigation because they may indicate missing information or unclear instructions.

• Count conversations, not individual message bubbles.
• Record how many handoffs reached the intended staff member.
• Separate appropriate handoffs from unnecessary ones.
• Count incorrect answers even when the customer does not complain.

Days eight to ten: Look for patterns across different businesses' enquiries

The most useful review question is not whether the chatbot has made a mistake. It is whether the same type of mistake keeps appearing. Group similar problems together: missing opening hours, unclear service descriptions, outdated prices, misunderstood booking requests or answers that sound too formal. Three conversations with the same problem usually deserve a closer look at the underlying instruction.

Different businesses will have different patterns. A customer chatbot for a bike shop may struggle to distinguish a general repair enquiry from a request for a diagnosis. A chatbot for guesthouse enquiries may need clearer instructions about availability, arrival times and changes to existing bookings. A tutoring business may need a sharper boundary between explaining lesson options and making an individual educational recommendation.

• Group repeated errors by subject.
• Identify whether the problem comes from missing facts or unclear instructions.
• Check whether staff receive the information needed to continue a handoff.
• Note questions that customers repeatedly rephrase.

Days eleven to fourteen: Make a small number of controlled changes

Choose the three most important problems and correct those first. If the assistant repeatedly gives an outdated opening time, update the source-of-truth information. If it promises an individual repair estimate, tighten the boundary around technical diagnosis. If customers cannot reach a person, repair the handoff route before changing the chatbot's personality. Every correction should have an owner and a clear reason.

Avoid changing several unrelated instructions at once. Otherwise, it becomes difficult to understand which change improved the conversation. Keep a short change log with the original problem, the updated instruction, the person responsible and the date. Repeat the affected questions using the business's own test script. If the correction introduces another problem, revise it before widening the assistant's role.

• Correct inaccurate business facts first.
• Repair missing or unreliable handoff routes.
• Rewrite confusing answers in the business's normal language.
• Remove responsibilities that repeatedly produce unsafe or unauthorized answers.

The probation decision: Keep, narrow or expand the role

At the end of day 14, choose one of three decisions. Keep the current role if routine answers are accurate and staff can manage the handoffs. Narrow the role if important questions still produce guesses, misleading promises or missing next steps. Expand it only when the existing responsibilities are dependable and the next task has clear instructions, reliable business information and a human fallback.

A useful review summary is short: what the assistant handles well, what it must pass to a person, which three changes were made and who will maintain its information. This gives the owner a practical record for the next review. Set the next review date before closing the probation file, and keep collecting customer questions that reveal gaps in the briefing.

Questions readers ask