Skip to content

Gen AI Orchestrator API

The Gen AI orchestrator is a Python (FastAPI) service called by Tock (the Tock Studio backend and the bots) for all the generative AI features. Its sources are in gen-ai/orchestrator-server.

Tock reaches it with the following properties:

Property Default Description
tock_gen_ai_orchestrator_server_url http://localhost:8000 URL of the orchestrator
tock_gen_ai_orchestrator_client_request_timeout_ms 55000 Timeout of the requests to the orchestrator (ms)

The complete, up-to-date reference of the API (request and response schemas, examples) is generated by FastAPI: open /docs (Swagger UI) or /redoc on the orchestrator, for instance http://localhost:8000/docs. The OpenAPI descriptor is available on /openapi.json.

The environment variables of the orchestrator itself are listed in the configuration reference.

Endpoints

Generation

Method Path Description
POST /rag Answers a question with a RAG chain: question condensing, document retrieval, answer generation. debug=true adds debug information
POST /qa Returns the documents of the knowledge base matching a question, without generating an answer
POST /completion/ Sends a prompt to an LLM (used by the playground)
POST /completion/sentences Generates sentences (used by sentence generation)

Some settings of the /rag request can only be set through the API, not in Tock Studio:

  • guardrail_setting: checks the toxicity of the generated answer with a self-hosted Bloomz classifier ({"provider": "BloomzGuardrail", "api_base": "...", "max_score": 0.3}). If one of the scores returned by the classifier exceeds max_score, the answer is rejected with the error 1004.
  • compressor_setting with the AwsBedrockRerank provider (see Compressor settings).

Providers

For each kind of provider, the same four endpoints are available:

Method Path Description
GET /<providers> Lists the supported providers
GET /<providers>/{provider_id} Returns a provider
GET /<providers>/{provider_id}/setting/example Returns an example of setting for the provider
POST /<providers>/{provider_id}/setting/status Checks a setting (for instance that the API key and the model are valid)

with <providers>:

Vector store inspection

Used by the vector store inspection screens:

Method Path Description
POST /vector-store-inspection/capabilities Capabilities of the vector store provider (search modes, etc.)
POST /vector-store-inspection/indexes Lists the indexes
POST /vector-store-inspection/documents Documents and chunks of an index, with statistics and anomalies
POST /vector-store-inspection/condense Condenses a question and extracts its keywords
POST /vector-store-inspection/search Runs a search and returns the detail of each stage

Monitoring

Method Path Description
GET /health-check Health check
GET /liveness-check Liveness check

Errors

Business errors are returned with the HTTP status 400 and a JSON body with a code, a message, a detail and, when the error comes from a provider, an info object (provider, error, cause, request). The codes are defined in ErrorCode (gen_ai_orchestrator/models/errors/errors_models.py):

Codes Category
1000–1005 Orchestrator: unknown error, connection, authentication, unknown provider setting, guardrail check, prompt template
2000–2007 AI provider (LLM / embeddings): unknown provider, bad request, API error, resource / model / deployment not found, context length exceeded
3000–3003 Vector store: unknown provider or setting, no document retrieved, data error
4000–4003 OpenSearch: settings, transport, resource or index not found
5000–5002 Observability: unknown provider or setting, API error
6000–6003 Document compressor: unknown provider, setting or label, API error
Chat with Tock
×