The Answers Quality menu

The Answers Quality menu helps business teams measure and improve the quality of the bot answers, in particular the answers generated by the RAG. It provides two tools:

• Evaluations: humans review and grade a sample of real (or test) conversations.
• Datasets: a fixed list of questions is replayed against the bot, so that runs can be compared after a change of settings, prompt or documents.

To access this menu, you need the botUser role (more details on roles in security).

Evaluations

An evaluation sample is a set of dialogues whose bot answers are reviewed one by one.

Creating a sample

Click on NEW SAMPLE and fill in:

• the Sample name and an optional Sample description,
• the Number of dialogues to randomly select,
• the period covered, with a Start date and an End date: only dialogues created during this period are selected,
• Allow test dialogues: whether dialogues created from Tock Studio tests can be selected.

A sample can also be created from a dataset run.

The list of samples shows, for each sample, its type (random or generated from a dataset run), its state (In Progress or Validated), its positive score, its number of dialogues and answers, and who created and validated it.

Evaluating the answers

Open a sample to review its dialogues. For each bot answer:

• evaluate it positively (thumb up),
• or evaluate it negatively (thumb down) and choose a reason:
  • Question not/misunderstood
  • Inaccurate answer
  • Incomplete answer
  • Incomplete sources / documents
  • Obsolete sources / documents
  • Business lexicon problem
  • Wrong answer format
  • Hallucination
  • Other

The progress bar shows the number of evaluated answers, and the positive / negative evaluations. The sample details give the period covered, the number of requested and actual dialogues, the total number of dialogues recorded for the bot during the period, and the resulting coverage rate.

Validating a sample

Once all the answers are evaluated, click on Validate and add an optional comment. A validated sample can no longer be modified. It can be exported as a PDF report with Export.

Dialogues are subject to the data retention policy of the platform. If some dialogues of a sample have been purged, a warning is displayed and the sample can no longer be validated.

Datasets

A dataset is a named list of questions, each with an optional ground truth (the expected answer). Running a dataset sends every question to the bot and records the answers, so that two runs can be compared question by question: typically before and after a change of the RAG settings, of the prompt or of the indexed documents.

Creating a dataset

Click on NEW DATASET, then enter a name, an optional description and the questions, with an optional ground truth for each of them.

A dataset can also be imported (button Import a dataset):

• from a JSON file containing a name, a description and an array of questions, each with a question and a groundTruth field,
• or from a CSV file with a question column and an optional groundTruth column: download the CSV template from the import dialog to get the expected columns. The name and description are then entered in the dialog.

From the edition screen, EXPORT DATASET exports the full dataset as JSON, or its questions only as CSV.

Run results are matched by question. Rephrasing a question slightly is fine, but changing its meaning makes runs compare different things: in that case, delete the question and add a new one.

Running a dataset

Click on Run dataset. Each question is sent to the bot with the test connector, as if typed in the Test menu, with the debug mode and the source contents enabled. Questions are processed one by one, in the background: the list shows the progress of the active run, which can be cancelled.

Running a dataset requires a test configuration for the bot. Only one run can be active for a dataset, and a dataset cannot be edited or deleted while a run is active.

For each run, the list of runs shows its dates, duration and state, and counts the questions:

• Completed and Failed (execution failures or technical errors),
• Intent: answered by intent detection,
• Found in context: answered by the RAG with relevant documents,
• Not found in context: handled by the RAG without relevant documents.

A run that ends with some failed questions is still completed: failed questions are reported in the run statistics.

Comparing runs

Open a run to display its answers. The screen compares a Run A with a Run B (by default, the latest run), question by question: answer text, answer type (RAG or intent) and sources. Sources that were added, modified or removed between the two runs are highlighted.

The Show only diffs filters display only the questions whose status, sources, answer type or wording changed.

The settings and prompts links show the RAG settings and prompts used by each run, and compare them between the two runs.

Creating an evaluation sample from a run

From the list of runs, Create an evaluation sample based on the results of this run creates an evaluation sample containing the answers of the run, so that they can be graded by humans.

Configuration

The following properties of the Tock Studio backend (tock/bot_admin) configure datasets:

Environment variable	Default	Description
tock_datasets_max_questions	200	Maximum number of questions in a dataset
tock_dataset_run_worker_poll_interval_ms	2000	Interval between two checks for new runs to process (ms)
tock_dataset_run_worker_max_retries	0	Number of retries of a question after a technical error

Chat with Tock

×