Common issues

Symptoms, causes and fixes for the problems operators and developers meet most often.

Each entry gives the symptom, the cause and the fix. For where each log lives, see Log locations.

Most checks below use two commands. On the Orchestrator host, from the deployment directory:

BashCopy
alias dc='docker compose -f docker-compose.core.yml'

On a Robot machine (status works unelevated; the other commands need an elevated prompt):

PowerShellCopy
& "$env:ProgramFiles\VeloPhex\Robot\VeloPhex.Robot.Service.exe" status

Orchestrator#

The web app does not load, or readiness fails#

Check liveness and readiness on the host:

BashCopy
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/health/live
curl -s http://127.0.0.1:8080/health/ready
Result Cause Fix
/health/live does not answer The api container is restarting dc ps and dc logs --since 10m api
/health/ready is 503 PostgreSQL is unreachable See the next entry
/health/ready is 200 with Degraded Only Redis is down dc up -d redis; the core platform keeps working meanwhile
503 straight after an upgrade, and the log says a relation or column does not exist The release's core-schema.sql was not applied. The Orchestrator never migrates on start Apply it as described in Upgrade

PostgreSQL is down or slow#

  • FATAL: password authentication failed in dc logs --since 5m postgres: .env was changed after the volume was created. The password in .env is only applied when the volume is first created; change it inside the database with ALTER USER instead.
  • data directory ... has wrong ownership or could not open file: a volume problem.
  • Start it with dc up -d postgres, wait until dc ps shows it healthy, then recheck /health/ready. The api reconnects by itself.
  • Slow rather than down: list long-running statements in pg_stat_activity. A long DELETE is usually the retention sweep.

The api refuses to start#

  • No trusted proxy network. The api needs VELOPHEX_TRUSTED_PROXY_NETWORK. With the bundled Caddy it defaults to the stack's own network. Behind your own proxy, set it to that proxy's network in CIDR form (a single address as /32).
  • Missing secrets. VELOPHEX_KEK and VELOPHEX_TOKEN_PEPPER are required (base64-encoded 32 bytes each). Compose refuses to start without them.

Scheduled jobs do not start, or jobs stay Running forever#

Background work (schedulers, lease recovery, retention, partition maintenance, email) runs inside the api process in the Compose deployment.

  1. Check it is on: dc exec api env | grep Workers__ must show Workers__Enabled=true.
  2. Search the log for run failed lines from worker categories such as TriggerSchedulerWorker, JobRecoveryWorker or RetentionWorker. A failing worker logs an error on every run but does not stop the process.

Robots stop receiving commands and events stop being recorded#

Symptom: the log shows no partition of relation … found for row. Cause: the command, event and webhook delivery tables are partitioned by UTC day, and the worker that creates partitions ahead of time has not run. Fix: restart the api; the worker runs at start. Never drop a partition by hand, or create one with bounds that are not UTC midnight.

The database keeps growing#

Symptom: Retention run failed every hour. Cause: usually a statement timeout on a large backlog. Fix: set Retention__BatchSize=1000 in velophex.prod.env and restart; the sweep catches up over several runs. Do not delete rows by hand: the sweep's order keeps foreign keys and blobs consistent. See Retention.

A request is refused with 403#

Every 403 for a missing permission has code: permission_required, names the permission in requiredPermission, and gives its scope, for example "You need Start Jobs in workspace Finance." Ask an administrator for a role that grants it. See RBAC and permissions.

Robots#

A Robot shows offline#

A Robot sends a heartbeat every 30 seconds and is shown offline after 90 seconds without one.

  1. Run status on the machine and read connectivity:
    • Connected: the machine is fine; check the Orchestrator side.
    • Disconnected with a connection error: DNS, firewall, proxy or TLS trust. If the machine reaches the internet through a proxy, set it with VeloPhex.Robot.Service.exe configure --proxy <url>, then Restart-Service VeloPhexRobot.
    • EnrollmentFailed: the machine was refused. status prints the reason.
    • NoRuntimeInstalled: see The Robot reports no runtime.
  2. If the Orchestrator's certificate comes from an internal CA or Caddy's internal CA, trust that root on the machine.

A job stays Pending#

The job's row on the Jobs grid and the job page show why it is waiting, with the fix. The reasons, in the order they are checked:

Reason Fix
The robot the job is pinned to was revoked Start the job again without a robot, or enrol the machine with a new machine key
No robot serves the workspace Add a machine to the workspace (workspace Settings > Machines)
No robot has the runtime Install the runtime on a machine that serves the workspace
Robot disabled Enable it on the Robots page
In maintenance Resume the machine on its Health tab
Only personal robots A personal robot runs only its owner's Assistant starts. Connect a machine for unattended jobs
No robot online Start the Robot service, or check the machine's network
Session unavailable The robot account's Windows session could not be opened. Sign the account in at the console, or allow RDP sessions in the machine's Robot settings
Waiting for a robot account Every robot account of the workspace is busy; the job starts when one is free
All robots busy Wait, or raise the machine's concurrency

Starting a job is refused#

Response Cause Fix
409 no_capable_runner (VEL-JOB-409-001) No active robot that serves the workspace reports the runtime the package needs Install the runtime, or assign a machine that has it. The identifier keeps an older name for compatibility
409 no_run_as_account An unattended job needs a robot account and the workspace has none Create one under Manage access > Robot accounts and assign it to the workspace
422 (VEL-JOB-422-001) The inputs do not match the entry point's input schema Correct the inputs. Re-publish the package if the workflow's arguments changed
409 package_not_runnable The package is a library Libraries cannot run as jobs

A job fails with VXR-RUNAS-REQUIRED, VXR-RUNAS-LOGON-FAILED or VXR-RUNAS-SESSION-FAILED#

  • VXR-RUNAS-REQUIRED: the Robot got an unattended job with no robot account. Assign a robot account to the job's workspace.
  • VXR-RUNAS-LOGON-FAILED: Windows refused the robot account's sign-in (wrong password, locked, disabled or expired account, or no "Allow log on locally" right). Update the password under Manage access > Robot accounts, then use Verify on machine.
  • VXR-RUNAS-SESSION-FAILED: the job needs a desktop and the robot account's session (Console or RDP) could not be created. Sign the account in at the console, or set up RDP sessions for it.

A job failed with VXR-OUTCOME-UNKNOWN#

Cause: the attempt ended after the automation may have started but before it reported a result: the executor crashed, the Robot service stopped, the Robot lost its lease, or the run overran its time limit and did not stop. Fix: check the systems the automation writes to (the job's logs show how far it got), finish or undo its work by hand, then use Re-run on the job. Such jobs are never retried automatically, and their queue items end Failed with "Outcome unknown"; requeue them once reconciled. If the Robot's own result arrives later, the job is reconciled and shows the real outcome.

The Robot reports no runtime#

Symptom: the log says every installed runtime failed its self-check, and the Robots page shows no runtimes. Cause: the executor files do not match their recorded hashes or VeloPhex signature, or the runtime was not installed. Fix: repair the installation (run Setup again and choose Repair). If %ProgramData%\VeloPhex\Robot\runtimes.d is empty, the runtime component was not installed; modify the installation to add it.

Jobs fail with DISK_FULL#

The Robot keeps at least Execution:MinimumFreeDiskMiB (1,024 MiB by default) free beyond twice the size of the package it prepares. Below that it refuses the job, reports its runtimes as Degraded and stops claiming until space is freed. Free disk space on the volume that holds %ProgramData%\VeloPhex\Robot.

Enrolment fails#

enroll exits with code 3 when the Orchestrator refuses the enrolment (2 when the command could not run, for example because it was not elevated).

Cause Code Fix
Key already used (enrollment_key_spent) VEL-AUTH-403-001 Regenerate the machine key and enrol again. A key is single-use
Key expired VEL-AUTH-403-002 Regenerate the machine key
Key issued to a different host VEL-AUTH-403-003 Use a key for this host
Machine disabled or revoked VEL-AUTH-403-004 Re-enable the machine, or enrol with a new key if it was revoked on purpose
Clock more than two minutes off VEL-AUTH-409-001 Synchronize the machine's clock

An Orchestrator that is only unreachable is not a failure: enroll exits 0, saves the connection and the service keeps trying. Setup checks the key's shape only, so a refused key leaves the machine installed but not connected; status then exits with 4.

The Robot is refused after it was connected#

Response Cause Fix
403 robot_revoked (VEL-ROBOT-403-001) An administrator revoked the machine Regenerate the machine key and enrol again
401 with VEL-AUTH-401-001 The machine signs with a key the Orchestrator does not hold, for example after a reimage Enrol again with a new machine key. Enrolling on the same host recovers its identity and history
426 robot_protocol_too_old (VEL-ROBOT-426-001) The Robot is older than the Orchestrator accepts Update it from Machines > Updates, or install the current Setup
403 personal_connection_ended (VEL-AUTH-403-008) The owner of a personal machine signed out or lost access The owner signs in to the Assistant again

If an administrator deleted the robot but not the machine, the Robot registers itself again with the identity it already holds; nothing needs to be done on the machine.

Robots disconnect repeatedly on the WebSocket channel#

This applies only to tenants that use the WebSocket command channel. The gateway closes with a code that tells you why:

Code Meaning Action
4000 Protocol versions do not overlap Upgrade the Robot
4001 The access token lapsed Check the Robot can reach /api/v1/oauth/token
4002 The Robot stopped reading Investigate the machine
4003 No robot for this machine The robot was deleted or revoked; re-create or re-enrol it
4004 Gateway going away Normal during a deployment
4005 Protocol violation Collect the Robot log and report it
4006 Tenant is on the HTTPS channel Expected after switching back
4007 A newer connection from the same machine replaced this one Normal once; repeated, two Robots share one identity (for example an unsysprepped clone)

With your own load balancer, check that it forwards WebSocket upgrades on /api/v1/robot/gateway and /hubs/*, has an idle timeout far above the 15-second heartbeat, does not buffer those routes, passes the original Host and allows request bodies as large as the Orchestrator's upload limit (100 MB by default, configurable up to 512 MB). See Network and firewall.

After an Orchestrator restart, everything is slow for a minute#

Every Robot reconnects and claims at once. Expect a short burst of latency, then normal. If it does not settle, a proxy is buffering long-poll or WebSocket routes.

Studio#

Packages show "Not resolved"#

Open Output, choose the Packages channel and read the diagnostic. It names the package, what requires it and the sources that were searched. Enable the source it names, or restore again. See Manage packages.

Publishing to the Orchestrator fails#

Response Cause Fix
401 The sign-in expired Select Sign in to Orchestrator again
409 package_version_exists This version already exists with different content Increase the project version
409 package_binary_duplicate These exact bytes are already published Nothing to do
400 The package failed inspection Fix what the response lists
422 (VEL-PKG-422-002) The package was built under a looser expression policy than the tenant's Set expressions in project.json to the tenant's policy, or loosen the tenant's under Settings > Expressions

Studio refuses to send a token over plain http. Use the Orchestrator's https address. See Publish to Orchestrator.

Next steps#

Orchestrator

The web app does not load, or readiness fails

PostgreSQL is down or slow

The api refuses to start

Scheduled jobs do not start, or jobs stay Running forever

Robots stop receiving commands and events stop being recorded

The database keeps growing

A request is refused with 403

Robots

A Robot shows offline

A job stays Pending

Starting a job is refused

A job fails with VXR-RUNAS-REQUIRED, VXR-RUNAS-LOGON-FAILED or VXR-RUNAS-SESSION-FAILED

A job failed with VXR-OUTCOME-UNKNOWN

The Robot reports no runtime

Jobs fail with DISK_FULL

Enrolment fails

The Robot is refused after it was connected

Robots disconnect repeatedly on the WebSocket channel

After an Orchestrator restart, everything is slow for a minute

Studio

Packages show "Not resolved"

Publishing to the Orchestrator fails

Next steps