Upgrade the Orchestrator
Upgrade a self-hosted Orchestrator to a new release, and roll back if you need to.
Upgrading a self-hosted Orchestrator means taking a backup, switching the image tag, applying the new schema script and restarting. The database schema only ever changes additively, which is what makes a rollback to the previous image possible.
Before you start#
- Read the release notes for the new version. They say when a release needs anything beyond these steps.
- Plan a short maintenance window. The API restarts, and during the restart users cannot sign in and Robots cannot claim new work. Robots keep running the jobs they already have and deliver results when the Orchestrator is back.
- Upgrade the Orchestrator before the Robots. See Updating the Robot.
Never run two versions against one database
Stop the old API container before the new one starts. docker compose up -d does this for you. Do not start a second stack against the same database.
Upgrade with Docker Compose#
Prerequisites#
- Shell access to the Docker host and the deployment directory (for example
/srv/velophex). - The new release's files: at least
core-schema.sql, and the newdocker-compose.core.ymlanddeploy/caddy/Caddyfileif they changed. - Your escrowed
.envvalues.
Steps#
-
Take a cold backup. This is your restore point if the upgrade has to be undone.
PowerShellCopypwsh /path/to/release/orchestrator/backup.ps1 -ProjectDirectory /srv/velophex -Destination /backup/velophex -
Set the new image tag in
.envINICopy.env:VELOPHEX_IMAGE=velophex/orchestrator:<version> -
Pull the image:
BashCopycd /srv/velophex docker compose -f docker-compose.core.yml pullIf you received the image as a file, load it with
docker load -i velophex-orchestrator-<version>.tarinstead. -
Apply the schema. The API never migrates the database on start. The release's
BashCopycore-schema.sqlis idempotent: it applies only the migrations the database does not have yet.set -a; . ./.env; set +a docker compose -f docker-compose.core.yml exec -T postgres \ psql -U "$VELOPHEX_POSTGRES_USER" -d "$VELOPHEX_POSTGRES_DB" -v < core-schema.sql -
Restart the stack:
BashCopydocker compose -f docker-compose.core.yml up -d
Verify#
-
Readiness returns 200:
BashCopycurl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/health/readyYou should see
200. -
Sign in to the web app. You should see your tenants and workspaces as before.
-
Start one job. You should see it claimed by a Robot and succeed.
If readiness returns 503 after an upgrade
If the log says a table or column does not exist, the schema script was not applied. Run step 4 and restart the API.
Roll back#
Because migrations are additive, the previous image normally runs on the newer schema:
- Set
VELOPHEX_IMAGEback to the previous tag in.env. - Run
docker compose -f docker-compose.core.yml up -d. - Check
/health/ready, sign in and run a job.
Only if the release notes say the new migrations changed something the old version cannot handle, restore the backup you took in step 1 (see Backup and restore). Never try to roll the schema itself back by hand.
Upgrading a stack from before the bundled Caddy#
Newer compose files give the stack's network a fixed subnet, so the API can trust forwarded headers from the bundled Caddy. Docker cannot change the subnet of a network that containers are attached to, so for this one upgrade run:
BashCopydocker compose -f docker-compose.core.yml down
docker compose -f docker-compose.core.yml up -d
down keeps your volumes.
After the upgrade#
- Upgrade your Robots and Studio installations to the matching release. See Updating the Robot.
- Run a restore drill against the new image when convenient, as recommended in Backup and restore.
Next steps#
Before you start
Upgrade with Docker Compose
Prerequisites
Steps
Verify
Roll back
Upgrading a stack from before the bundled Caddy
After the upgrade
Next steps