Improving the generated answers

A RAG bot is never finished on the first try: documents evolve, users ask unexpected questions, and a change of model or prompt can improve some answers and degrade others. This page describes the improvement loop, and the Tock Studio tools used at each step.

flowchart LR
    O[Observe<br/>real conversations] --> D[Diagnose<br/>a wrong answer]
    D --> F[Fix<br/>documents, settings, prompt, stories]
    F --> M[Measure<br/>replay a dataset]
    M --> O

1. Observe

The Dashboard summarizes the RAG activity: share of the answers grounded in the retrieved documents, topics answered, state of the knowledge base, checks of the Gen AI configuration.
Analytics > Dialogs shows the conversations, with the RAG answer and its sources. They can be filtered on the RAG answer status, for example to list the questions answered without relevant documents. Enable Dialogs debug in the RAG settings to also record the condensed question and the documents retrieved.
The RAG answer status, the topic and the suggested topics returned by the LLM are also recorded as indicators (RAG Status, RAG Topics, RAG Suggested Topics), available in the Metrics menu.
With an observability provider such as Langfuse, each answer links to the trace of its LLM calls: exact prompts, documents, token usage and latency.
Evaluations let business experts review and grade a sample of real conversations.

2. Diagnose

From a bot answer in Analytics > Dialogs, shortcuts open the question in the retrieval diagnostic or in the playground, with the values recorded for this exchange. First find out where the chain went wrong:

Symptom	Likely cause	Tool
The right document is not among the sources	Retrieval: the document is missing from the index, badly chunked, or ranked too low	Retrieval diagnostic, vector store exploration
The right document is among the sources, but the answer is wrong or incomplete	Generation: prompt, model or context size	Playground, observability traces
The bot answered a question it should not have	Scope: topic not excluded, or a validated answer is missing	RAG exclusions, prompt context, FAQ
A question was answered by the wrong story	NLU: the sentence was classified in an intent that has a story	Language understanding, model quality

3. Fix

Depending on the cause:

Documents: complete or clean the sources, then re-index them and switch the bot to the new indexing session. The vector store exploration reports nearly empty chunks, sources without URL and duplicate titles.
Retrieval: change the search type (hybrid search with PGVector), the number of documents retrieved in the RAG settings, or the compressor. Compare the searches in the retrieval diagnostic before and after.
Generation: adjust the answering prompt or the model, after trying them in the playground.
Scope: add covered or excluded topics and business terms, exclude sentences from the RAG, or answer with an FAQ or a story (see How the bot answers).

4. Measure

Before deploying a change, check that it does not degrade other answers:

Build a dataset of representative questions, including those that were answered badly.
Run it before and after the change, and compare the runs.
If needed, create an evaluation sample from a run so that business experts grade the new answers.

The Dashboard history keeps track of the changes of the Gen AI settings and of the corpus, to relate a change in the metrics to a change in the configuration.

Chat with Tock

×