Monitoring and health
Watch jobs, machines and robots on the Monitoring dashboard, and check a self-hosted Orchestrator's health endpoints, logs and background workers.
This page has two parts: the dashboards every tenant administrator uses, and the checks an operator of a self-hosted Orchestrator runs. For metrics and traces, see Observability.
Watch the tenant on the Monitoring dashboard#
- In the sidebar, switch to Tenant and select Monitoring.
- Pick the time range at the top right (Last hour, Last day, Last week or Last 30 days).
- Read the cards: counts of processes, assets, queues, triggers, access entries and machines; Machines and robots (online, offline, busy and idle); Job status (pending and running now); and Job history (completed, failed and canceled).
Each workspace has the same view, plus jobs over time, transactions, top processes, robot utilization and queue health, on its Home page. The dashboards update live.
Other places to look:
- Machines shows which machines are connected and warns about machines with no robot. See Machines and robots.
- Settings ▸ Alerts notifies people about failed jobs, long jobs, SLA risk and offline robots. See Alerts.
- Webhooks sends the same events to your monitoring tools. See Audit log and webhooks.
A robot that has not sent a heartbeat for 90 seconds (the default) is shown as offline.
Health endpoints (self-hosted)#
The rest of this page is for operators of a self-hosted Orchestrator. The commands assume you are in the deployment directory (for example /srv/velophex) and use this alias:
alias dc='docker compose -f docker-compose.core.yml'
| Endpoint | Answers | Use it for |
|---|---|---|
/health/live |
200 while the process runs. Checks no dependencies. | Liveness probes. |
/health/ready |
200 Healthy when PostgreSQL is reachable; 200 Degraded when only Redis is down; 503 when PostgreSQL is unreachable. |
Readiness probes and load balancer checks. |
Both are anonymous. From the Docker host:
BashCopycurl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/health/live
curl -s http://127.0.0.1:8080/health/ready
The Docker Compose stack wires these endpoints into its container health checks.
Redis in a single-host stack
In the Docker Compose stack, a Redis outage only turns readiness to Degraded (still 200). Start it again with dc up -d redis.
Logs#
The API writes one JSON object per line, with Category, Level and TraceId fields. Every HTTP response carries an X-Correlation-ID header, and error responses include the same correlationId, so you can find the log lines for a failed request.
| Question | Command |
|---|---|
| What is it doing now | dc logs -f --since 10m api |
| What happened at start | dc logs api and read the first lines: the environment, the built-in VeloPhex package signer, the bootstrap administrator and the background services starting |
| Errors in the last hour | dc logs --since 1h api | grep '"Level":"Error"' |
| Database reachable | dc exec postgres pg_isready -U "$VELOPHEX_POSTGRES_USER" |
| Disk use | docker system df -v | grep velophex_orchestrator |
The bundled Caddy front door writes JSON access logs for every request that reached the stack.
A start-up warning about the signer
If the start-up log says the build carries no VeloPhex production signer, you are not running a release image. Such a build refuses every VeloPhex.* package.
Background workers#
In the Docker Compose stack, background work runs inside the single api process. Each worker runs on an interval, logs "<Worker> run failed" at Error level when a run fails, and carries on. A failing worker never stops the process, so alert on these errors.
| Worker | Default interval | If it keeps failing |
|---|---|---|
JobRecoveryWorker |
15 s | Jobs on lost robots stay Running. |
RunnerResultInboxWorker |
5 s | Jobs finish on the robot but not in the Orchestrator. |
TriggerSchedulerWorker |
10 s | Scheduled jobs are not created (they fire late once it recovers). |
QueueItemRecoveryWorker |
30 s | Items whose lease expired stay In progress. |
RunnerPresenceMonitorWorker |
60 s | Nobody is told a robot went offline. |
RetentionWorker |
60 min | The database grows without bound. |
PartitionMaintenanceWorker |
60 min, and at start | Robot commands, domain events and webhook deliveries stop being recorded once the pre-created days run out. |
Set up an alert on log lines whose Category names one of these workers and whose Level is Error.
Never switch the workers off
Workers__Enabled must stay true on the process that runs background work. With it off, jobs are still claimed and completed, but nothing recovers, schedules, sweeps or notifies. To pause the workers for maintenance, stop the API container (dc stop api) instead.
Commands waiting for robots#
If you suspect a robot is not receiving commands (such as a stop), this query lists commands issued but not yet acknowledged:
Unacknowledged Robot commandsSQLCopySELECT machine_id, count(*) AS owed, min(issued_at) AS oldest
FROM robot_commands
WHERE acknowledged_at IS NULL AND expires_at > now()
GROUP BY machine_id
ORDER BY oldest;
Run it with dc exec postgres psql -U "$VELOPHEX_POSTGRES_USER" -d "$VELOPHEX_POSTGRES_DB". A machine with commands older than a couple of minutes is not receiving them, whichever command channel its tenant uses.
A daily check#
/health/readyreturns 200Healthy.- No worker
Errorlines in the last 24 hours. - Monitoring shows the robots you expect online.
- Failed jobs (including dead-lettered ones) have not grown unexpectedly.
- Disk use on the data and database volumes is within budget.
Next steps#
Watch the tenant on the Monitoring dashboard
Health endpoints (self-hosted)
Logs
Background workers
Commands waiting for robots
A daily check
Next steps