Performance tuning
Adjust connection limits, database pooling, retention batches and proxy settings when an Orchestrator cell needs more headroom.
Most cells need no tuning: the defaults are sized for a large fleet. Tune only when a measurement points at a specific limit. This page lists the settings that matter, where to set them, and the symptoms that tell you to change them.
Where settings go#
The Orchestrator reads configuration from environment variables named Section__Key. In the Docker Compose stack:
- Values with a
VELOPHEX_*variable (listed in.env.example) go in.env. - Any other
Section__Key=valuegoes invelophex.prod.envnext to the compose file, which theapiservice reads.
After a change, recreate the API container:
BashCopydocker compose -f docker-compose.core.yml up -d api
Database#
| Setting | Default | Change when |
|---|---|---|
VELOPHEX_DB_POOL_SIZE |
100 | Logs show connection pool has been exhausted, or statements time out waiting for a connection. |
VELOPHEX_POSTGRES_MAX_CONNECTIONS |
200 | You raise the pool or add Orchestrator processes. The sum of all pools must fit under it. |
Database__CommandTimeoutSeconds |
30 | Only temporarily, for a known bulk operation. Put it back afterwards. |
VELOPHEX_POSTGRES_MEMORY_LIMIT |
2g |
You tune PostgreSQL's own memory settings upward. |
A pool that is too small shows up as slow requests and retries rather than clear errors. If several processes share one database, divide the pool between them so the total stays under max_connections.
To see what the database is busy with:
Active statements, oldest firstSQLCopySELECT pid, now() - query_start AS age, state, left(query, 80)
FROM pg_stat_activity
WHERE state <> 'idle'
ORDER BY 2 DESC;
A long DELETE is usually the retention sweep. Many short-lived locks on job or Robot rows right after a restart are a reconnect storm, which settles on its own.
Connection ceilings#
Each process limits its connections so a misbehaving fleet cannot exhaust it:
| Setting | Default |
|---|---|
Kestrel__Limits__MaxConcurrentConnections |
16,000 |
Kestrel__Limits__MaxConcurrentUpgradedConnections |
12,000 |
When the WebSocket ceiling is reached, the gateway answers 503 with Retry-After: 30 and logs a warning naming Kestrel:Limits:MaxConcurrentUpgradedConnections. Those Robots keep working over HTTPS. Raise the ceiling if you see these warnings regularly, and do not set it below the number of Robots you expect on the WebSocket channel.
Retention sweep#
Retention deletes old jobs, logs, queue items, audit events and notifications in batches.
| Setting | Default | Notes |
|---|---|---|
Retention__IntervalMinutes |
60 | How often the sweep runs. |
Retention__BatchSize |
5000 | Rows per batch. Lower it to 1000 if batches time out. |
Retention__MaxBatchesPerRun |
20 | Batches per category and tenant per run. |
If a large backlog built up (for example after raising retention windows and lowering them again), let the sweep catch up over several runs. For a very large backlog, run it off-hours with Retention__IntervalMinutes=1 and Retention__MaxBatchesPerRun=200, then restore the defaults. Do not delete rows by hand: the sweep's order keeps foreign keys and blobs consistent.
Shorter retention windows are the most effective way to keep the database small and fast.
Command channel#
By default, Robots poll an HTTPS command cursor for commands. Switching a tenant to the WebSocket gateway delivers commands as soon as they are written, instead of on the next poll:
HTTPCopyPUT /api/v1/tenants/{tenantId}/settings/robots
Content-Type: application/json
{ "commandChannel": "websocket" }
Before you switch, make sure the front door forwards WebSocket upgrades on /api/v1/robot/gateway, has an idle timeout far above the 15 second heartbeat, and does not buffer responses on that route. Switch one tenant at a time and watch it for a day. To go back, set commandChannel to https and restart the gateway processes.
Your own proxy or load balancer#
If you replace the bundled Caddy, most performance problems come from the proxy. Check:
- WebSocket upgrades are forwarded on
/api/v1/robot/gatewayand/hubs/*. - The idle timeout is far above 15 seconds. An hour is a sensible value.
- Response buffering is off on those routes.
- The original
Hostheader is passed through, withX-Forwarded-ForandX-Forwarded-Proto. VELOPHEX_TRUSTED_PROXY_NETWORKis set to the proxy's network. Without it, rate limits collapse into one bucket for all clients.- The request body limit is at least your package upload limit (
Packages:MaxPackageSizeBytes, 100 MB by default) plus 1 MB. - Client affinity is enabled for the gateway where you can.
Robots#
- Match each machine's concurrency to its CPU, memory and the applications it drives. Desktop automation usually needs one job per interactive session.
- Keep enough free disk: at least 1 GiB plus twice the largest package being prepared.
- Place Robots close, in network terms, to the systems they automate. Most job time is spent in those systems, not in the Orchestrator.
Tune in this order#
- Confirm the symptom with metrics and logs (Observability).
- Rule out the proxy and placement (workspace assignment, runtimes).
- Shorten retention windows.
- Adjust the database pool and
max_connectionstogether. - Move PostgreSQL to dedicated resources.
- Only then consider splitting roles (see Availability and recovery), keeping one worker.
Next steps#
Where settings go
Database
Connection ceilings
Retention sweep
Command channel
Your own proxy or load balancer
Robots
Tune in this order
Next steps