Test your AI before you go live

Docs · Quality

Run answer tests and multi-turn scenarios on the Test AI answers page before the widget goes live. Each test states what the AI must do, and each run shows whether it did.

Before you start

  • Difficulty: Easy
  • Time: 15 minutes
  • You need: An account that can open Test AI answers
  • You need: Knowledge sources or articles for the AI to answer from

Steps

  1. Open Test AI answers

    In the dashboard, open Test AI answers under AI & Answers. The page has two lists: Answer tests, one question each, and Scenarios, a conversation of up to 8 visitor turns.

  2. Add an answer test

    Type the visitor's question and choose the expected behaviour: Must answer, Should hand off to a teammate, or Should decline. Optionally type an expected phrase in the box marked Answer must contain. The phrase is checked for Must answer and Should hand off to a teammate; the match is case-insensitive. The box is disabled for Should decline.

  3. Run the answer tests

    Run next batch runs up to 50 enabled answer tests, the ones that were run longest ago first. Each result shows whether the AI would hand off or decline, the knowledge sources it used and when it last ran, and lets you review the last answer.

  4. Build a scenario

    Give the scenario a name and add up to 8 turns. Each turn is a visitor message with optional checks, listed under What each check means. A check left at Not checked is not tested. Each turn sees the turns before it.

  5. Save a conversation as a test draft

    Open the conversation in the inbox and choose Save as AI test. ZChat copies the visitor's messages, up to 8 turns, into a draft scenario, and personal details are scrubbed from the copied text. For each turn it records only what happened: a handoff check if the chat was handed to a teammate on that turn, and an action check if an action ran. The AI's replies are not copied.

  6. Review the draft and save it

    A draft never runs by itself. Open it, write the reply expectations yourself, check the other expectations, and save it. It does not run until you save it and run it.

  7. Optional: run the answer tests every day

    Under AI settings, Scheduled Answer tests runs the saved answer tests once a day in the background. It is off by default and uses AI quota for each run. When a test that passed starts failing, ZChat sends an alert to your webhooks. Scenarios are not part of the daily run.

What each check means

  • Must answer: The AI gives a reply, does not hand off to a teammate, and the reply contains the answer phrase if one is set.
  • Should hand off to a teammate: The AI hands the chat to a teammate, and the reply contains the answer phrase if one is set.
  • Should decline: The guardrails decline the question. The phrase is not checked.
  • Reply must contain, Reply must not contain: The reply contains the phrase, or does not. The match is case-insensitive.
  • Should hand off, Should not hand off: The AI's handoff decision on that turn is yes, or no.
  • Should ask a question back, Should not ask a question back: The reply contains a question mark, or does not. The check looks only for the character.
  • Action it should call, Action it must not call: The AI calls that action on the turn, or does not. An action that is not turned on for the workspace fails the check, and the test says so.
  • Not checked: The check is skipped.

Good to know

  • A scenario passes only when every turn passes. A run stops and reports the reason if the AI cannot answer a turn, for example during a provider outage.
  • A scenario can be run only after it is saved. Up to 25 scenarios per workspace.
  • Each failed turn reports only the first check that does not hold.

What you will see

  • Each answer test shows pass or fail, its last run time, the AI's last answer, and whether the AI would hand off or decline.
  • Each scenario turn shows the reply, the handoff and decline results, the actions the AI called, and the first check that failed, in plain words.

Common mistakes

  • Reading a phrase match as a meaning match. The phrase is a substring, so refund passes on no refunds.
  • Expecting a Must answer test to pass when the chat was handed to a teammate. A handoff fails that test even when the phrase is in the reply.
  • Writing an action check for an action that is not turned on. The check fails, because the AI was never offered that action.
  • Saving a conversation as a test and keeping its handoff and action checks without asking whether they are right. A draft records what the AI did, not what it should have done.
  • Expecting scenarios to run every day. Only answer tests run on the daily schedule.

Answer tests check what the AI says. Approvals check what it does: see Approvals for AI actions before you turn on an action that writes data.