Supervision

This chapter presents some supervision and monitoring aspects of the functioning of the platform and Tock bots.

Healthchecks

The APIs of the different Tock components include different probes or lifelines (healthchecks) to check if everything is working properly.

These lifelines can be used by automated monitoring systems.

Probes & paths

Each WebVerticle exposes 3 probes returning the HTTP 200 code if everything is working.

Default path	Description	Property to modify the path
<root path>/healthcheck	The component is working properly. (Detailed mode: see below)	<verticle>_tock_vertx_healthcheck_path
/health/readiness	The component is ready to process requests.	<verticle>_tock_vertx_readinesscheck_path
/health/liveness	The component is started.	<verticle>_tock_vertx_livenesscheck_path

For some components and images, the lifeline cannot be exposed directly at the root, the path is modified. In particular:

• For tock/bot_admin, the lifeline is located by default in /rest/admin/healthcheck
• For tock/nlp_api, the lifeline is /rest/nlp/healthcheck

Each default path can be modified with a dedicated property (see table above), prefixed by the name of the verticle, for instance nlpverticle_tock_vertx_healthcheck_path (see Configuration).

The Gen AI orchestrator exposes two probes, on its own port (8000 by default):

Path	Description
/health-check	The orchestrator is working properly
/liveness-check	The orchestrator is started

Detailed mode

The main lifeline /healthcheck can perform a more detailed inspection (ie. usually check the connection to other components) if the tock_detailed_healthcheck_enabled property is enabled.

The lifeline response then specifies the components checked.

Here is an example of activation in the bot_admin Docker image:

docker-compose.ymldockerrun.aws.json

services:
  admin_web:
    image: tock/bot_admin:$TAG
    environment:
    - tock_detailed_healthcheck_enabled=true

{
  "AWSEBDockerrunVersion": 2,
  "containerDefinitions": [
    {
      "name": "admin_web",
      "image": "tock/bot_admin:${TAG}",
      "environment": [
        { "name": "tock_detailed_healthcheck_enabled", "value": "true" }
      ]
    }
  ]
}

Example of a lifeline response in detailed mode:

  {
    "results": [
      {
        "id": "duckling_service",
        "status": "OK"
      },
      {
        "id": "tock_front_database",
        "status": "OK"
      },
      {
        "id": "tock_model_database",
        "status": "OK"
      },
      {
        "id": "tock_bot_database",
        "status": "OK"
      }
    ]
  }

See below for a summary of the lifelines component by component.

Details by component

The table below details the path and checks performed by the main lifeline component by component, in normal and detailed mode:

Component / Image	Default Path	Default Check	Detailed Check (tock_detailed_healthcheck_enabled=true)
NLP	/rest/nlp/healthcheck	Duckling / entity providers OK.	front and model databases OK.
Duckling / RestEntityProvider	/healthcheck	Duckling bridge initialized.	Idem
Build Worker	/healthcheck	Worker ready to analyze the model.	front and model databases OK.
Bot / Bot Api	/healthcheck	Bot installed, connected to the database, NLP OK.	Idem
WebHook (Bot Api)	/healthcheck	OK	Idem
Kotlin Compile	/healthcheck	OK	Idem
NLP / Bot Admin	/rest/admin/healthcheck	OK	Duckling / entity providers OK, front model and bot databases OK.

Below are examples of responses from different components in verbose mode:

NLPDucklingBuild WorkerBot ApiKotlin CompilerBot Admin

{
  "results": [
    {
      "id": "duckling_service",
      "status": "OK"
    },
    {
      "id": "tock_front_database",
      "status": "OK"
    },
    {
      "id": "tock_model_database",
      "status": "OK"
    }
  ]
}

{
  "results": [
    {
      "id": "duckling_bridge",
      "status": "OK"
    }
  ]
}

{
  "results": [
    {
      "id": "tock_front_database",
      "status": "OK"
    },
    {
      "id": "tock_model_database",
      "status": "OK"
    }
  ]
}

{
  "results": [
    {
      "id": "nlp_client",
      "status": "OK"
    }
  ]
}

{
 "results": []
}

{
  "results": [
    {
      "id": "duckling_service",
      "status": "OK"
    },
    {
      "id": "tock_front_database",
      "status": "OK"
    },
    {
      "id": "tock_model_database",
      "status": "OK"
    },
    {
      "id": "tock_bot_database",
      "status": "OK"
    }
  ]
}

Lifeline Monitoring

The different probes and lifelines can be used to configure monitoring systems and other container orchestrators, for example, to measure service availability, trigger alerts or dynamically remediate problems.

Depending on the tools and technologies used, lifeline configuration can be done in different ways. Here are some examples for reference:

• At the level of the Cloud or on-premise load balancers (load balancers), for example:
• in the Health checks section at the ELB level in AWS
• with the health_check directive in NGINX
• with the httpchk option in HAProxy
• In Docker (Dockerfile descriptors):
• with the health_check instruction HEALTHCHECK to check the component status
• with ENTRYPOINT or CMD to wait for the component to be ready at startup
• In Docker Compose (docker-compose.yml descriptors):
• with healthcheck
• In Kubernetes:
• with livenessProbe and readinessProbe
• Etc.

Logging

Application logs

Tock uses SLF4J and Logback to generate its server-side application logs.

By default, Tock automatically configures its logs and a few properties allow you to modify the configuration.

It is possible to finely configure the logs according to your needs, in particular with Logback configuration files or with Docker Compose.

Auto-configuration

By default, in the absence of specific configuration, Tock configures its logs automatically using the class LogbackConfigurator. This configures Logback programmatically, with the following behavior:

• General log level DEBUG if tock_env=dev (default), or INFO for other environments and production
• Exception for org.mongodb.driver logs always at INFO
• Exception for io.netty logs always at INFO
• Exception for okhttp3 logs always at INFO
• Exception for io.mockk logs always at INFO
• Logs directed to the console (standard output) by default, but possibility to write to files
• In file log mode, writing to log/logFile.log, archiving a file every day (log/logFile.%d{yyyy-MM-dd}.log), maximum 30 days or 3GB of logs kept

To learn more about automatic configuration of Tock logs, refer to the implementation LogbackConfigurator.

Environment variables allow to configure these different logging modes. They can be defined independently on each component producing logs.

Environment variable	Default value	Description
tock_env	dev	Environment (warning: controls other mechanisms than logs).
tock_logback_enabled	true	Enable logs.
tock_default_log_level	DEBUG if tock_env=dev (otherwise INFO)	General log level (except exceptions, see above).
tock_retrofit_log_level	BASIC if tock_env=dev (otherwise NONE)	Log level (requests and responses) for application client services using Retrofit (TockNlpClient, BotApiClient, and many connectors...)
tock_logback_file_appender	false	Logs files (see details above) instead of console logs (standard output).

Depending on the deployment mode used, these environment variables can be added either directly on the command line, or in a descriptor like docker-compose.yml, dockerrun.aws.json or other.

Here is an example configuring Tock Studio logs (admin_web) in Docker:

docker-compose.ymldockerrun.aws.json

services:
  admin_web:
    image: tock/bot_admin:$TAG
    environment:
    - tock_default_log_level=WARN
    - tock_logback_file_appender=true

{
  "AWSEBDockerrunVersion": 2,
  "containerDefinitions": [
    {
      "name": "admin_web",
      "image": "tock/bot_admin:${TAG}",
      "environment": [
        { "name": "tock_default_log_level", "value": "WARN" },
        { "name": "tock_logback_file_appender", "value": "true" }
      ]
    }
  ]
}

Logback Files

It is possible to finely configure the Tock logs by directly configuring Logback. For this, different possibilities exist, in particular configuration files in XML or Groovy format. Below is an example configuration:

logback.xmllogback.groovy

<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="false">

    <logger name="ai.tock" level="DEBUG" />

    <logger name="org.mongodb" level="WARN" />

    <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>
                %d %-5level [%thread] %logger{1}: %msg%n
            </pattern>
        </encoder>
    </appender>

    <root level="INFO">
        <appender-ref ref="console" />
    </root>

</configuration>

import ch.qos.logback.classic.encoder.PatternLayoutEncoder
import ch.qos.logback.core.ConsoleAppender

import static ch.qos.logback.classic.Level.DEBUG
import static ch.qos.logback.classic.Level.INFO
import static ch.qos.logback.classic.Level.WARN

appender("console", ConsoleAppender) {
  encoder(PatternLayoutEncoder) {
    pattern = "%d %-5level [%thread] %logger{1}: %msg%n"
  }
}
logger("ai.tock", DEBUG)
logger("org.mongodb", WARN)
root(INFO, ["console"])

In this example:

• The general log level is INFO, with exceptions:
• Exception for ai.tock logs always at DEBUG
• Exception for org.mongodb logs always at WARN
• Logs are directed to the console (standard output)

Many possibilities are offered to configure the logs thanks to these configuration files: log levels adjusted according to the embedded packages/frameworks, modification of the default pattern, logging of log files and automatic archiving with a RollingFileAppender, redirection of logs to a database or email server, etc. For more information, refer to the Logback documentation.

Once the Logback configuration file has been created, you must ensure that it is in the classpath of the Tock component and that Logback identifies it as the configuration to follow (for this, you generally use the -Dlogback.configurationFile property when starting the Java component).

Below is a complete example in Docker Compose with:

• A logback.xml file embedded by Maven (pom.xml) in the Docker image
• Some environment variables defined to be able to quickly adjust the main log levels directly in Docker-Compose (without having to modify the XML file or the Docker image)'

^

logback.xmlpom.xmldocker-compose.yml

<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="false">

    <!-- Leverage env variables with defaults, for more flexibility -->
    <variable name="tock_default_log_level" value="${tock_default_log_level:-WARN}" />
    <variable name="tock_service_log_level" value="${tock_service_log_level:-INFO}" />
    <variable name="tock_database_log_level" value="${tock_database_log_level:-WARN}" />

    <logger name="ai.tock" level="${tock_service_log_level}" />
    <logger name="org.mongodb" level="${tock_database_log_level}" />

    <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>
                %d %-5level [%thread] %logger{1}: %msg%n
            </pattern>
        </encoder>
    </appender>

    <root level="${tock_default_log_level}">
        <appender-ref ref="console" />
    </root>

</configuration>

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <artifactId>tock-docker-bot-admin</artifactId>
    <build>
        <plugins>
            <plugin>
                <groupId>io.fabric8</groupId>
                <artifactId>docker-maven-plugin</artifactId>
                <configuration>
                    <images>
                        <image>
                            <name>tock/bot_admin:${project.version}</name>
                            <build>
                                <assembly>
                                    <inline>
                                        <dependencySets>
                                            ...
                                        </dependencySets>
                                        <files>
                                            <file>
                                                <source>logback.xml</source>
                                                <outputDirectory>.</outputDirectory>
                                            </file>
                                        </files>
                                    </inline>
                                </assembly>
                                <cmd>
                                    <shell>java $JAVA_ARGS -Dlogback.configurationFile='file:///maven/logback.xml' -Dfile.encoding=UTF-8 -Dtock_nlp_model_refresh_rate=10 -cp '/maven/*' ai.tock.bot.admin.StartBotAdminServerKt</shell>
                                </cmd>
                                ...

services:
  admin_web:
    image: tock/bot_admin:$TAG
    environment:
    - tock_default_log_level=WARN # Default, see logback.xml
    - tock_service_log_level=INFO # Default, see logback.xml
    - tock_database_log_level=WARN # Default, see logback.xml

The full code for this example applied to all Tock modules is available in the tock-docker repository on the logbackxml branch.

Docker Compose

For Docker Compose users, a mechanism allows to configure logs directly in YAML descriptors. See the tock-docker repository for an example of Tock implementation in Docker Compose.

Each application component can have its own configuration:

docker-compose.yml

logging:
  driver: "json-file"
  options:
    max-size: "10m"
    max-file: "5"

The example above configures an automatic rotation of log files, so as to have a maximum of 5 log files of maximum 10 MB each (the oldest being deleted to create a new one if necessary).

For more information, see the Docker Compose documentation for this mechanism.

Encryption and anonymization

See the security page for log encryption and anonymization options.

Chat with Tock

×