Vector store inspection
The vector store inspection tools show what the vector store of the bot actually contains, and why a given chunk is (or is not) handed to the answering model for a question. They are available in the Gen AI menu:
- Vector store exploration: what does the index contain?
- Retrieval diagnostic: why this answer? The whole retrieval chain for a question, from the store to the chunks sent to the model.
To access these pages, you need the admin role (more details on roles in security).
The inspection is available for the PGVector provider. With OpenSearch, only the vector search mode is available: full text and hybrid modes are not implemented for this provider, and the diagnostic uses vector search whatever mode the bot is configured with (as the runtime does).
Vector store exploration
Select an index (one index per indexing session) to display its ingestion report:
- statistics: number of documents and chunks, average number of chunks per document, median chunk length,
- anomalies, usable as filters:
- nearly empty chunks: chunks whose content is too short to be usable,
- non-URL sources: documents whose source is not a URL, and cannot be used as a link in answers,
- duplicate titles: identical titles carried by different documents,
- the paginated list of documents, which can be filtered by title, identifier or content, and expanded to show their chunks.
A chunk can be pinned: pinned chunks are tracked in the retrieval diagnostic, where they always appear in the results, even when no search channel returns them. The Diagnose button opens the diagnostic with the pinned chunks.
Retrieval diagnostic
The diagnostic screen follows the runtime retrieval chain, from top to bottom:
- Question: the question as a user would ask it. It can be condensed (the condensed question and the keywords are filled by the condensing LLM, and can be edited). Condensation is not deterministic: the search uses the displayed values as they are, so a search can be replayed identically.
- Retrieval: the index, the search mode (Vector, Full text or Hybrid; full text and hybrid modes require
keywords) and Max documents retrieved (
k): the number of chunks each channel returns, kept by the RRF fusion in hybrid mode, and forming the pool handed to the compressor. The search mode is taken from the bot configuration and shown read-only. - Compression: whether to apply the document compressor, with its minimum score and maximum number of documents. That maximum is what sets how many chunks reach the answering model.
- Run: the warnings and the divergence banner, then the search.
The form starts from the bot's actual retrieval settings, and the search waits until they are loaded; a banner lists any setting you changed and offers to reset it to the bot's. A message warns when the bot has no RAG configuration, when its RAG is disabled (the runtime then does not apply these settings), or when the settings could not be loaded (the form then starts from defaults). The compressor runs on the retrieved documents, so its maximum number of documents cannot exceed Max documents retrieved. Changing the minimum score or the maximum number of documents is recomputed instantly in the browser, without a new search: every candidate's reranker score is shown with the threshold marker, so the minimum can be chosen by eye (the score scale depends on the reranker model).
With the maximum number of documents equal to Max documents retrieved, no minimum score and no padding, the compressor keeps every chunk and only reorders them: a message points it out.
The reranker is called with the same timeout as in production: retrieving more documents increases the reranking time and may trigger a fallback, as it would in production.
Funnel and results
The funnel shows each stage with its number of chunks: vector and full text channels (in hybrid mode), RRF fusion, the top-k cut, then compression (with its duration). When the compressor fails, it falls back as in production: it passes on the first documents in retrieval order, unscored, up to its maximum number of documents. The stage is then shown as failed, fell back with the reason (reranker error, failed call, or question too long for the reranker); the error message is available as a tooltip. If compression fails outside that fault tolerance (an invalid compressor configuration, for instance), the stage is shown as failed and no chunk is passed on: in production, the request would fail. The thresholds cannot be retuned until a new search returns scores.
The results table shows one chunk per line, with its rank and score for each channel (vector, full text, RRF, rerank), the number of windows an oversized chunk was scored across, and its outcome:
| Outcome | Meaning |
|---|---|
| kept | Present in the context handed to the answering model |
| cut | Returned by the search, but dropped by the top-k cut |
| below threshold | Reached the compressor but scored below the minimum |
| ranked out | Above the threshold, but ranked past the maximum number of documents |
| padded in | Scored below the threshold, but pulled back to fill the document count |
| cut by fallback | Compressor fell back: past the first documents passed on unscored |
| not passed on | Compression failed: in production, no chunk would be passed on |
| absent | Returned by no search channel at all |
Comparing searches
Set as reference freezes a search as the baseline for the next ones. The following searches are then compared with it: the parameters that changed (index, search mode, question, keywords, max documents retrieved, compression — including the minimum score and maximum number of documents), the chunks that were lost, gained or moved, and those now passed on to the model or no longer passed on. A client-side threshold change produces a new comparable search without a backend call: it moves no rank, but changes which chunks are passed on. Two kinds of absence are distinguished:
- absent from index: the two searches target different indexes and the chunk does not exist in the current one (an ingestion problem),
- outside top k: the chunk still exists, but no longer surfaces within the retrieved window (a ranking problem).
Comparisons are computed in the browser and are not saved.
Chat with Tock