Monitoring and health

Watch jobs, machines and robots on the Monitoring dashboard, and check a self-hosted Orchestrator's health endpoints, logs and background workers.

This page has two parts: the dashboards every tenant administrator uses, and the checks an operator of a self-hosted Orchestrator runs. For metrics and traces, see Observability.

Watch the tenant on the Monitoring dashboard#

  1. In the sidebar, switch to Tenant and select Monitoring.
  2. Pick the time range at the top right (Last hour, Last day, Last week or Last 30 days).
  3. Read the cards: counts of processes, assets, queues, triggers, access entries and machines; Machines and robots (online, offline, busy and idle); Job status (pending and running now); and Job history (completed, failed and canceled).

Each workspace has the same view, plus jobs over time, transactions, top processes, robot utilization and queue health, on its Home page. The dashboards update live.

Other places to look:

  • Machines shows which machines are connected and warns about machines with no robot. See Machines and robots.
  • Settings ▸ Alerts notifies people about failed jobs, long jobs, SLA risk and offline robots. See Alerts.
  • Webhooks sends the same events to your monitoring tools. See Audit log and webhooks.

A robot that has not sent a heartbeat for 90 seconds (the default) is shown as offline.

Health endpoints (self-hosted)#

The rest of this page is for operators of a self-hosted Orchestrator. The commands assume you are in the deployment directory (for example /srv/velophex) and use this alias:

BashCopy
alias dc='docker compose -f docker-compose.core.yml'
Endpoint Answers Use it for
/health/live 200 while the process runs. Checks no dependencies. Liveness probes.
/health/ready 200 Healthy when PostgreSQL is reachable; 200 Degraded when only Redis is down; 503 when PostgreSQL is unreachable. Readiness probes and load balancer checks.

Both are anonymous. From the Docker host:

BashCopy
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/health/live
curl -s http://127.0.0.1:8080/health/ready

The Docker Compose stack wires these endpoints into its container health checks.

Redis in a single-host stack

In the Docker Compose stack, a Redis outage only turns readiness to Degraded (still 200). Start it again with dc up -d redis.

Logs#

The API writes one JSON object per line, with Category, Level and TraceId fields. Every HTTP response carries an X-Correlation-ID header, and error responses include the same correlationId, so you can find the log lines for a failed request.

Question Command
What is it doing now dc logs -f --since 10m api
What happened at start dc logs api and read the first lines: the environment, the built-in VeloPhex package signer, the bootstrap administrator and the background services starting
Errors in the last hour dc logs --since 1h api | grep '"Level":"Error"'
Database reachable dc exec postgres pg_isready -U "$VELOPHEX_POSTGRES_USER"
Disk use docker system df -v | grep velophex_orchestrator

The bundled Caddy front door writes JSON access logs for every request that reached the stack.

A start-up warning about the signer

If the start-up log says the build carries no VeloPhex production signer, you are not running a release image. Such a build refuses every VeloPhex.* package.

Background workers#

In the Docker Compose stack, background work runs inside the single api process. Each worker runs on an interval, logs "<Worker> run failed" at Error level when a run fails, and carries on. A failing worker never stops the process, so alert on these errors.

Worker Default interval If it keeps failing
JobRecoveryWorker 15 s Jobs on lost robots stay Running.
RunnerResultInboxWorker 5 s Jobs finish on the robot but not in the Orchestrator.
TriggerSchedulerWorker 10 s Scheduled jobs are not created (they fire late once it recovers).
QueueItemRecoveryWorker 30 s Items whose lease expired stay In progress.
RunnerPresenceMonitorWorker 60 s Nobody is told a robot went offline.
RetentionWorker 60 min The database grows without bound.
PartitionMaintenanceWorker 60 min, and at start Robot commands, domain events and webhook deliveries stop being recorded once the pre-created days run out.

Set up an alert on log lines whose Category names one of these workers and whose Level is Error.

Never switch the workers off

Workers__Enabled must stay true on the process that runs background work. With it off, jobs are still claimed and completed, but nothing recovers, schedules, sweeps or notifies. To pause the workers for maintenance, stop the API container (dc stop api) instead.

Commands waiting for robots#

If you suspect a robot is not receiving commands (such as a stop), this query lists commands issued but not yet acknowledged:

Unacknowledged Robot commandsSQLCopy
SELECT machine_id, count(*) AS owed, min(issued_at) AS oldest
FROM robot_commands
WHERE acknowledged_at IS NULL AND expires_at > now()
GROUP BY machine_id
ORDER BY oldest;

Run it with dc exec postgres psql -U "$VELOPHEX_POSTGRES_USER" -d "$VELOPHEX_POSTGRES_DB". A machine with commands older than a couple of minutes is not receiving them, whichever command channel its tenant uses.

A daily check#

  1. /health/ready returns 200 Healthy.
  2. No worker Error lines in the last 24 hours.
  3. Monitoring shows the robots you expect online.
  4. Failed jobs (including dead-lettered ones) have not grown unexpectedly.
  5. Disk use on the data and database volumes is within budget.

Next steps#

Watch the tenant on the Monitoring dashboard

Health endpoints (self-hosted)

Logs

Background workers

Commands waiting for robots

A daily check

Next steps