Performance tuning

Adjust connection limits, database pooling, retention batches and proxy settings when an Orchestrator cell needs more headroom.

Most cells need no tuning: the defaults are sized for a large fleet. Tune only when a measurement points at a specific limit. This page lists the settings that matter, where to set them, and the symptoms that tell you to change them.

Where settings go#

The Orchestrator reads configuration from environment variables named Section__Key. In the Docker Compose stack:

  • Values with a VELOPHEX_* variable (listed in .env.example) go in .env.
  • Any other Section__Key=value goes in velophex.prod.env next to the compose file, which the api service reads.

After a change, recreate the API container:

BashCopy
docker compose -f docker-compose.core.yml up -d api

Database#

Setting Default Change when
VELOPHEX_DB_POOL_SIZE 100 Logs show connection pool has been exhausted, or statements time out waiting for a connection.
VELOPHEX_POSTGRES_MAX_CONNECTIONS 200 You raise the pool or add Orchestrator processes. The sum of all pools must fit under it.
Database__CommandTimeoutSeconds 30 Only temporarily, for a known bulk operation. Put it back afterwards.
VELOPHEX_POSTGRES_MEMORY_LIMIT 2g You tune PostgreSQL's own memory settings upward.

A pool that is too small shows up as slow requests and retries rather than clear errors. If several processes share one database, divide the pool between them so the total stays under max_connections.

To see what the database is busy with:

Active statements, oldest firstSQLCopy
SELECT pid, now() - query_start AS age, state, left(query, 80)
FROM pg_stat_activity
WHERE state <> 'idle'
ORDER BY 2 DESC;

A long DELETE is usually the retention sweep. Many short-lived locks on job or Robot rows right after a restart are a reconnect storm, which settles on its own.

Connection ceilings#

Each process limits its connections so a misbehaving fleet cannot exhaust it:

Setting Default
Kestrel__Limits__MaxConcurrentConnections 16,000
Kestrel__Limits__MaxConcurrentUpgradedConnections 12,000

When the WebSocket ceiling is reached, the gateway answers 503 with Retry-After: 30 and logs a warning naming Kestrel:Limits:MaxConcurrentUpgradedConnections. Those Robots keep working over HTTPS. Raise the ceiling if you see these warnings regularly, and do not set it below the number of Robots you expect on the WebSocket channel.

Retention sweep#

Retention deletes old jobs, logs, queue items, audit events and notifications in batches.

Setting Default Notes
Retention__IntervalMinutes 60 How often the sweep runs.
Retention__BatchSize 5000 Rows per batch. Lower it to 1000 if batches time out.
Retention__MaxBatchesPerRun 20 Batches per category and tenant per run.

If a large backlog built up (for example after raising retention windows and lowering them again), let the sweep catch up over several runs. For a very large backlog, run it off-hours with Retention__IntervalMinutes=1 and Retention__MaxBatchesPerRun=200, then restore the defaults. Do not delete rows by hand: the sweep's order keeps foreign keys and blobs consistent.

Shorter retention windows are the most effective way to keep the database small and fast.

Command channel#

By default, Robots poll an HTTPS command cursor for commands. Switching a tenant to the WebSocket gateway delivers commands as soon as they are written, instead of on the next poll:

HTTPCopy
PUT /api/v1/tenants/{tenantId}/settings/robots
Content-Type: application/json

{ "commandChannel": "websocket" }

Before you switch, make sure the front door forwards WebSocket upgrades on /api/v1/robot/gateway, has an idle timeout far above the 15 second heartbeat, and does not buffer responses on that route. Switch one tenant at a time and watch it for a day. To go back, set commandChannel to https and restart the gateway processes.

Your own proxy or load balancer#

If you replace the bundled Caddy, most performance problems come from the proxy. Check:

  1. WebSocket upgrades are forwarded on /api/v1/robot/gateway and /hubs/*.
  2. The idle timeout is far above 15 seconds. An hour is a sensible value.
  3. Response buffering is off on those routes.
  4. The original Host header is passed through, with X-Forwarded-For and X-Forwarded-Proto.
  5. VELOPHEX_TRUSTED_PROXY_NETWORK is set to the proxy's network. Without it, rate limits collapse into one bucket for all clients.
  6. The request body limit is at least your package upload limit (Packages:MaxPackageSizeBytes, 100 MB by default) plus 1 MB.
  7. Client affinity is enabled for the gateway where you can.

Robots#

  • Match each machine's concurrency to its CPU, memory and the applications it drives. Desktop automation usually needs one job per interactive session.
  • Keep enough free disk: at least 1 GiB plus twice the largest package being prepared.
  • Place Robots close, in network terms, to the systems they automate. Most job time is spent in those systems, not in the Orchestrator.

Tune in this order#

  1. Confirm the symptom with metrics and logs (Observability).
  2. Rule out the proxy and placement (workspace assignment, runtimes).
  3. Shorten retention windows.
  4. Adjust the database pool and max_connections together.
  5. Move PostgreSQL to dedicated resources.
  6. Only then consider splitting roles (see Availability and recovery), keeping one worker.

Next steps#

Where settings go

Database

Connection ceilings

Retention sweep

Command channel

Your own proxy or load balancer

Robots

Tune in this order

Next steps