Common issues
Symptoms, causes and fixes for the problems operators and developers meet most often.
Each entry gives the symptom, the cause and the fix. For where each log lives, see Log locations.
Most checks below use two commands. On the Orchestrator host, from the deployment directory:
BashCopyalias dc='docker compose -f docker-compose.core.yml'
On a Robot machine (status works unelevated; the other commands need an elevated prompt):
& "$env:ProgramFiles\VeloPhex\Robot\VeloPhex.Robot.Service.exe" status
Orchestrator#
The web app does not load, or readiness fails#
Check liveness and readiness on the host:
BashCopycurl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/health/live
curl -s http://127.0.0.1:8080/health/ready
| Result | Cause | Fix |
|---|---|---|
/health/live does not answer |
The api container is restarting | dc ps and dc logs --since 10m api |
/health/ready is 503 |
PostgreSQL is unreachable | See the next entry |
/health/ready is 200 with Degraded |
Only Redis is down | dc up -d redis; the core platform keeps working meanwhile |
503 straight after an upgrade, and the log says a relation or column does not exist |
The release's core-schema.sql was not applied. The Orchestrator never migrates on start |
Apply it as described in Upgrade |
PostgreSQL is down or slow#
FATAL: password authentication failedindc logs --since 5m postgres:.envwas changed after the volume was created. The password in.envis only applied when the volume is first created; change it inside the database withALTER USERinstead.data directory ... has wrong ownershiporcould not open file: a volume problem.- Start it with
dc up -d postgres, wait untildc psshows it healthy, then recheck/health/ready. The api reconnects by itself. - Slow rather than down: list long-running statements in
pg_stat_activity. A longDELETEis usually the retention sweep.
The api refuses to start#
- No trusted proxy network. The api needs
VELOPHEX_TRUSTED_PROXY_NETWORK. With the bundled Caddy it defaults to the stack's own network. Behind your own proxy, set it to that proxy's network in CIDR form (a single address as/32). - Missing secrets.
VELOPHEX_KEKandVELOPHEX_TOKEN_PEPPERare required (base64-encoded 32 bytes each). Compose refuses to start without them.
Scheduled jobs do not start, or jobs stay Running forever#
Background work (schedulers, lease recovery, retention, partition maintenance, email) runs inside the api process in the Compose deployment.
- Check it is on:
dc exec api env | grep Workers__must showWorkers__Enabled=true. - Search the log for
run failedlines from worker categories such asTriggerSchedulerWorker,JobRecoveryWorkerorRetentionWorker. A failing worker logs an error on every run but does not stop the process.
Robots stop receiving commands and events stop being recorded#
Symptom: the log shows no partition of relation … found for row.
Cause: the command, event and webhook delivery tables are partitioned by UTC day, and the worker that creates partitions ahead of time has not run.
Fix: restart the api; the worker runs at start. Never drop a partition by hand, or create one with bounds that are not UTC midnight.
The database keeps growing#
Symptom: Retention run failed every hour.
Cause: usually a statement timeout on a large backlog.
Fix: set Retention__BatchSize=1000 in velophex.prod.env and restart; the sweep catches up over several runs. Do not delete rows by hand: the sweep's order keeps foreign keys and blobs consistent. See Retention.
A request is refused with 403#
Every 403 for a missing permission has code: permission_required, names the permission in requiredPermission, and gives its scope, for example "You need Start Jobs in workspace Finance." Ask an administrator for a role that grants it. See RBAC and permissions.
Robots#
A Robot shows offline#
A Robot sends a heartbeat every 30 seconds and is shown offline after 90 seconds without one.
- Run
statuson the machine and readconnectivity:Connected: the machine is fine; check the Orchestrator side.Disconnectedwith a connection error: DNS, firewall, proxy or TLS trust. If the machine reaches the internet through a proxy, set it withVeloPhex.Robot.Service.exe configure --proxy <url>, thenRestart-Service VeloPhexRobot.EnrollmentFailed: the machine was refused.statusprints the reason.NoRuntimeInstalled: see The Robot reports no runtime.
- If the Orchestrator's certificate comes from an internal CA or Caddy's internal CA, trust that root on the machine.
A job stays Pending#
The job's row on the Jobs grid and the job page show why it is waiting, with the fix. The reasons, in the order they are checked:
| Reason | Fix |
|---|---|
| The robot the job is pinned to was revoked | Start the job again without a robot, or enrol the machine with a new machine key |
| No robot serves the workspace | Add a machine to the workspace (workspace Settings > Machines) |
| No robot has the runtime | Install the runtime on a machine that serves the workspace |
| Robot disabled | Enable it on the Robots page |
| In maintenance | Resume the machine on its Health tab |
| Only personal robots | A personal robot runs only its owner's Assistant starts. Connect a machine for unattended jobs |
| No robot online | Start the Robot service, or check the machine's network |
| Session unavailable | The robot account's Windows session could not be opened. Sign the account in at the console, or allow RDP sessions in the machine's Robot settings |
| Waiting for a robot account | Every robot account of the workspace is busy; the job starts when one is free |
| All robots busy | Wait, or raise the machine's concurrency |
Starting a job is refused#
| Response | Cause | Fix |
|---|---|---|
409 no_capable_runner (VEL-JOB-409-001) |
No active robot that serves the workspace reports the runtime the package needs | Install the runtime, or assign a machine that has it. The identifier keeps an older name for compatibility |
409 no_run_as_account |
An unattended job needs a robot account and the workspace has none | Create one under Manage access > Robot accounts and assign it to the workspace |
422 (VEL-JOB-422-001) |
The inputs do not match the entry point's input schema | Correct the inputs. Re-publish the package if the workflow's arguments changed |
409 package_not_runnable |
The package is a library | Libraries cannot run as jobs |
A job fails with VXR-RUNAS-REQUIRED, VXR-RUNAS-LOGON-FAILED or VXR-RUNAS-SESSION-FAILED#
VXR-RUNAS-REQUIRED: the Robot got an unattended job with no robot account. Assign a robot account to the job's workspace.VXR-RUNAS-LOGON-FAILED: Windows refused the robot account's sign-in (wrong password, locked, disabled or expired account, or no "Allow log on locally" right). Update the password under Manage access > Robot accounts, then use Verify on machine.VXR-RUNAS-SESSION-FAILED: the job needs a desktop and the robot account's session (Console or RDP) could not be created. Sign the account in at the console, or set up RDP sessions for it.
A job failed with VXR-OUTCOME-UNKNOWN#
Cause: the attempt ended after the automation may have started but before it reported a result: the executor crashed, the Robot service stopped, the Robot lost its lease, or the run overran its time limit and did not stop. Fix: check the systems the automation writes to (the job's logs show how far it got), finish or undo its work by hand, then use Re-run on the job. Such jobs are never retried automatically, and their queue items end Failed with "Outcome unknown"; requeue them once reconciled. If the Robot's own result arrives later, the job is reconciled and shows the real outcome.
The Robot reports no runtime#
Symptom: the log says every installed runtime failed its self-check, and the Robots page shows no runtimes.
Cause: the executor files do not match their recorded hashes or VeloPhex signature, or the runtime was not installed.
Fix: repair the installation (run Setup again and choose Repair). If %ProgramData%\VeloPhex\Robot\runtimes.d is empty, the runtime component was not installed; modify the installation to add it.
Jobs fail with DISK_FULL#
The Robot keeps at least Execution:MinimumFreeDiskMiB (1,024 MiB by default) free beyond twice the size of the package it prepares. Below that it refuses the job, reports its runtimes as Degraded and stops claiming until space is freed. Free disk space on the volume that holds %ProgramData%\VeloPhex\Robot.
Enrolment fails#
enroll exits with code 3 when the Orchestrator refuses the enrolment (2 when the command could not run, for example because it was not elevated).
| Cause | Code | Fix |
|---|---|---|
Key already used (enrollment_key_spent) |
VEL-AUTH-403-001 |
Regenerate the machine key and enrol again. A key is single-use |
| Key expired | VEL-AUTH-403-002 |
Regenerate the machine key |
| Key issued to a different host | VEL-AUTH-403-003 |
Use a key for this host |
| Machine disabled or revoked | VEL-AUTH-403-004 |
Re-enable the machine, or enrol with a new key if it was revoked on purpose |
| Clock more than two minutes off | VEL-AUTH-409-001 |
Synchronize the machine's clock |
An Orchestrator that is only unreachable is not a failure: enroll exits 0, saves the connection and the service keeps trying. Setup checks the key's shape only, so a refused key leaves the machine installed but not connected; status then exits with 4.
The Robot is refused after it was connected#
| Response | Cause | Fix |
|---|---|---|
403 robot_revoked (VEL-ROBOT-403-001) |
An administrator revoked the machine | Regenerate the machine key and enrol again |
401 with VEL-AUTH-401-001 |
The machine signs with a key the Orchestrator does not hold, for example after a reimage | Enrol again with a new machine key. Enrolling on the same host recovers its identity and history |
426 robot_protocol_too_old (VEL-ROBOT-426-001) |
The Robot is older than the Orchestrator accepts | Update it from Machines > Updates, or install the current Setup |
403 personal_connection_ended (VEL-AUTH-403-008) |
The owner of a personal machine signed out or lost access | The owner signs in to the Assistant again |
If an administrator deleted the robot but not the machine, the Robot registers itself again with the identity it already holds; nothing needs to be done on the machine.
Robots disconnect repeatedly on the WebSocket channel#
This applies only to tenants that use the WebSocket command channel. The gateway closes with a code that tells you why:
| Code | Meaning | Action |
|---|---|---|
| 4000 | Protocol versions do not overlap | Upgrade the Robot |
| 4001 | The access token lapsed | Check the Robot can reach /api/v1/oauth/token |
| 4002 | The Robot stopped reading | Investigate the machine |
| 4003 | No robot for this machine | The robot was deleted or revoked; re-create or re-enrol it |
| 4004 | Gateway going away | Normal during a deployment |
| 4005 | Protocol violation | Collect the Robot log and report it |
| 4006 | Tenant is on the HTTPS channel | Expected after switching back |
| 4007 | A newer connection from the same machine replaced this one | Normal once; repeated, two Robots share one identity (for example an unsysprepped clone) |
With your own load balancer, check that it forwards WebSocket upgrades on /api/v1/robot/gateway and /hubs/*, has an idle timeout far above the 15-second heartbeat, does not buffer those routes, passes the original Host and allows request bodies as large as the Orchestrator's upload limit (100 MB by default, configurable up to 512 MB). See Network and firewall.
After an Orchestrator restart, everything is slow for a minute#
Every Robot reconnects and claims at once. Expect a short burst of latency, then normal. If it does not settle, a proxy is buffering long-poll or WebSocket routes.
Studio#
Packages show "Not resolved"#
Open Output, choose the Packages channel and read the diagnostic. It names the package, what requires it and the sources that were searched. Enable the source it names, or restore again. See Manage packages.
Publishing to the Orchestrator fails#
| Response | Cause | Fix |
|---|---|---|
401 |
The sign-in expired | Select Sign in to Orchestrator again |
409 package_version_exists |
This version already exists with different content | Increase the project version |
409 package_binary_duplicate |
These exact bytes are already published | Nothing to do |
400 |
The package failed inspection | Fix what the response lists |
422 (VEL-PKG-422-002) |
The package was built under a looser expression policy than the tenant's | Set expressions in project.json to the tenant's policy, or loosen the tenant's under Settings > Expressions |
Studio refuses to send a token over plain http. Use the Orchestrator's https address. See Publish to Orchestrator.
Next steps#
Orchestrator
The web app does not load, or readiness fails
PostgreSQL is down or slow
The api refuses to start
Scheduled jobs do not start, or jobs stay Running forever
Robots stop receiving commands and events stop being recorded
The database keeps growing
A request is refused with 403
Robots
A Robot shows offline
A job stays Pending
Starting a job is refused
A job fails with VXR-RUNAS-REQUIRED, VXR-RUNAS-LOGON-FAILED or VXR-RUNAS-SESSION-FAILED
A job failed with VXR-OUTCOME-UNKNOWN
The Robot reports no runtime
Jobs fail with DISK_FULL
Enrolment fails
The Robot is refused after it was connected
Robots disconnect repeatedly on the WebSocket channel
After an Orchestrator restart, everything is slow for a minute
Studio
Packages show "Not resolved"
Publishing to the Orchestrator fails
Next steps