Observability
Export Orchestrator traces and metrics with OpenTelemetry, follow a request with correlation ids, and know which signals to watch.
The Orchestrator emits OpenTelemetry traces and metrics over OTLP, writes structured JSON logs, and tags every request with a correlation id. This page shows how to turn the export on and which signals are worth a dashboard or an alert.
Turn on OpenTelemetry export#
Export is off until you give the Orchestrator an OTLP endpoint, such as an OpenTelemetry Collector.
Prerequisites#
- An OTLP receiver reachable from the Orchestrator host (for example a Collector listening on port 4317).
- Access to the deployment's
.env(Docker Compose).
Steps#
-
Set
.envINICopyVELOPHEX_OTLP_ENDPOINTin.env. The compose file passes it to the API asOpenTelemetry__OtlpEndpoint.VELOPHEX_OTLP_ENDPOINT=http://otel-collector:4317 -
Recreate the API container:
BashCopydocker compose -f docker-compose.core.yml up -d api
The value must be an absolute URI, or the process refuses to start with a message naming OpenTelemetry:OtlpEndpoint.
Verify#
Start a job, then look in your tracing back end. You should see traces for the HTTP requests and database calls, and the VeloPhex.Orchestrator meter's metrics arriving at your collector.
What is exported#
Traces cover incoming ASP.NET Core requests (except /health), outgoing HTTP calls and PostgreSQL commands (Npgsql).
Metrics include ASP.NET Core, HTTP client and Npgsql instruments, plus the Orchestrator's own meter, VeloPhex.Orchestrator:
| Instrument | Type | What it tells you |
|---|---|---|
velophex.jobs.claimed |
Counter | Job attempts handed to Robots, by runtime. |
velophex.jobs.results |
Counter | Attempt results applied, by reported outcome and resulting job status. |
velophex.jobs.recovered |
Counter | Attempts the recovery sweep acted on, by reason (for example an expired lease or a timeout). |
velophex.jobs.queue_depth |
Gauge | Pending jobs waiting to be claimed. |
velophex.runner.heartbeats |
Counter | Robot heartbeats recorded. |
velophex.gateway.closed |
Counter | Robot command-channel connections closed, by close code. |
Names that say runner
Some instrument names use runner, the Robot's former name. The names are part of the telemetry contract and do not change.
There is no Prometheus scrape endpoint. Send OTLP to a Collector and export from there to the back end you use.
Logs#
- The API writes one JSON object per line with
Category,LevelandTraceId. Collect container logs with your usual log shipper. - The bundled Caddy writes JSON access logs.
- Every response carries an
X-Correlation-IDheader, and error responses includecorrelationIdin their body. Search your logs for that value to find everything that happened in one request.
Useful alerts#
| Alert | Signal |
|---|---|
| Orchestrator not ready | /health/ready not returning 200 |
| Worker failing | Log lines with Level Error from a worker category, for example RetentionWorker or JobRecoveryWorker |
| Backlog growing | velophex.jobs.queue_depth rising for longer than your normal peak |
| Robots dropping off | A drop in velophex.runner.heartbeats, or com.velophex.robot.offline webhook events |
| Jobs dying | com.velophex.job.dead_lettered webhook events |
| Command channel churn | A rising velophex.gateway.closed rate, especially codes 4001 (token lapsed) and 4002 (Robot stopped reading) |
| Recovery activity | A spike in velophex.jobs.recovered with reason lease_expired, which means machines or networks are failing mid-job |
Gateway close codes#
When you use the WebSocket command channel, the close code on velophex.gateway.closed tells you what to do:
| Code | Meaning | Action |
|---|---|---|
| 4000 | No protocol version in common | Upgrade the Robot |
| 4001 | The access token lapsed | Check the Robot can reach the token endpoint |
| 4002 | The Robot stopped reading | Look at the machine |
| 4003 | No Robot for this machine | The Robot was deleted or revoked; re-create it |
| 4004 | The gateway is going away | Normal during a deployment |
| 4005 | A protocol violation | Capture the Robot log and contact support |
| 4006 | The tenant uses the HTTPS channel | Expected after switching a tenant back |
The Robot side#
The Robot records its own activity and metrics internally, but does not yet export them over OpenTelemetry. On a Robot machine, use its log files, the Windows Event Log and VeloPhex.Robot.Service.exe status. See Robot logs and troubleshooting.
Next steps#
Turn on OpenTelemetry export
Prerequisites
Steps
Verify
What is exported
Logs
Useful alerts
Gateway close codes
The Robot side
Next steps