Observability

Export Orchestrator traces and metrics with OpenTelemetry, follow a request with correlation ids, and know which signals to watch.

The Orchestrator emits OpenTelemetry traces and metrics over OTLP, writes structured JSON logs, and tags every request with a correlation id. This page shows how to turn the export on and which signals are worth a dashboard or an alert.

Turn on OpenTelemetry export#

Export is off until you give the Orchestrator an OTLP endpoint, such as an OpenTelemetry Collector.

Prerequisites#

  • An OTLP receiver reachable from the Orchestrator host (for example a Collector listening on port 4317).
  • Access to the deployment's .env (Docker Compose).

Steps#

  1. Set VELOPHEX_OTLP_ENDPOINT in .env. The compose file passes it to the API as OpenTelemetry__OtlpEndpoint.

    .envINICopy
    VELOPHEX_OTLP_ENDPOINT=http://otel-collector:4317
  2. Recreate the API container:

    BashCopy
    docker compose -f docker-compose.core.yml up -d api

The value must be an absolute URI, or the process refuses to start with a message naming OpenTelemetry:OtlpEndpoint.

Verify#

Start a job, then look in your tracing back end. You should see traces for the HTTP requests and database calls, and the VeloPhex.Orchestrator meter's metrics arriving at your collector.

What is exported#

Traces cover incoming ASP.NET Core requests (except /health), outgoing HTTP calls and PostgreSQL commands (Npgsql).

Metrics include ASP.NET Core, HTTP client and Npgsql instruments, plus the Orchestrator's own meter, VeloPhex.Orchestrator:

Instrument Type What it tells you
velophex.jobs.claimed Counter Job attempts handed to Robots, by runtime.
velophex.jobs.results Counter Attempt results applied, by reported outcome and resulting job status.
velophex.jobs.recovered Counter Attempts the recovery sweep acted on, by reason (for example an expired lease or a timeout).
velophex.jobs.queue_depth Gauge Pending jobs waiting to be claimed.
velophex.runner.heartbeats Counter Robot heartbeats recorded.
velophex.gateway.closed Counter Robot command-channel connections closed, by close code.

Names that say runner

Some instrument names use runner, the Robot's former name. The names are part of the telemetry contract and do not change.

There is no Prometheus scrape endpoint. Send OTLP to a Collector and export from there to the back end you use.

Logs#

  • The API writes one JSON object per line with Category, Level and TraceId. Collect container logs with your usual log shipper.
  • The bundled Caddy writes JSON access logs.
  • Every response carries an X-Correlation-ID header, and error responses include correlationId in their body. Search your logs for that value to find everything that happened in one request.

Useful alerts#

Alert Signal
Orchestrator not ready /health/ready not returning 200
Worker failing Log lines with Level Error from a worker category, for example RetentionWorker or JobRecoveryWorker
Backlog growing velophex.jobs.queue_depth rising for longer than your normal peak
Robots dropping off A drop in velophex.runner.heartbeats, or com.velophex.robot.offline webhook events
Jobs dying com.velophex.job.dead_lettered webhook events
Command channel churn A rising velophex.gateway.closed rate, especially codes 4001 (token lapsed) and 4002 (Robot stopped reading)
Recovery activity A spike in velophex.jobs.recovered with reason lease_expired, which means machines or networks are failing mid-job

Gateway close codes#

When you use the WebSocket command channel, the close code on velophex.gateway.closed tells you what to do:

Code Meaning Action
4000 No protocol version in common Upgrade the Robot
4001 The access token lapsed Check the Robot can reach the token endpoint
4002 The Robot stopped reading Look at the machine
4003 No Robot for this machine The Robot was deleted or revoked; re-create it
4004 The gateway is going away Normal during a deployment
4005 A protocol violation Capture the Robot log and contact support
4006 The tenant uses the HTTPS channel Expected after switching a tenant back

The Robot side#

The Robot records its own activity and metrics internally, but does not yet export them over OpenTelemetry. On a Robot machine, use its log files, the Windows Event Log and VeloPhex.Robot.Service.exe status. See Robot logs and troubleshooting.

Next steps#

Turn on OpenTelemetry export

Prerequisites

Steps

Verify

What is exported

Logs

Useful alerts

Gateway close codes

The Robot side

Next steps