Availability and recovery
What happens when parts of a cell fail, which parts you can run more than once, and how you recover.
This page explains how an Orchestrator cell behaves when something fails, what the platform allows you to run as several replicas, and how you recover from the failures that replicas do not cover. Read it together with Backup and restore.
What keeps working during an outage#
The platform is built so that a short outage delays work instead of losing it:
| If this fails | What happens |
|---|---|
| The Orchestrator process restarts | Robots keep running the jobs they already have. Results wait in each Robot's outbox and are delivered when the Orchestrator answers again. Robots reconnect with backoff and jitter. |
| PostgreSQL is unreachable | /health/ready returns 503 and requests fail. A result a Robot posted during the outage is refused, kept by the Robot, and posted again later. Every multi-statement write is transactional, so nothing is half applied. |
| Redis is down | In a single-node stack, nothing changes except readiness reporting Degraded. With several API processes, live browser updates across processes pause until Redis returns. |
| A Robot machine dies mid-job | Its lease expires. The recovery sweep requeues or fails the job according to the process's retry policy (retryOnLeaseExpiry). Queue items it held return to the queue. |
| The background worker stops | Jobs are still claimed and completed, but schedules, recovery, presence alerts and retention pause until it runs again. |
Jobs restart from the beginning
A job interrupted by a lost machine is retried as a new attempt. It does not resume from the middle, so design automations that can be safely run again (for example, check whether an invoice is already posted before posting it).
Which roles you can run more than once#
A cell has three roles: api, gateway and worker. See Architecture for scale.
| Role | Replicas | Notes |
|---|---|---|
api |
One or more | Stateless. Several replicas need a shared data-protection key ring, shared blob storage and the Redis backplane for live updates. |
gateway |
One or more | Holds Robot WebSockets. A Robot whose gateway restarts reconnects and replays from its last acknowledged command. Give Robots client affinity to one gateway replica where your load balancer allows it. |
worker |
Exactly one | Not leader-elected. A second replica would run every schedule and sweep twice. |
The Docker Compose stack runs all roles in one process, which is the right choice for a single host and the only topology covered by the published scale certification.
Requirements for more than one API process#
- Shared data-protection keys. Every replica must read the same key ring, or users are signed out on restart and one replica rejects another's cookies.
- Shared blob storage. Use S3-compatible storage, Azure Blob, or a shared volume.
- Redis backplane. Turn on
Realtime__UseRedisBackplaneand setRedis__ConnectionString, so live updates reach browsers connected to any replica. The process refuses to start with the backplane on and no Redis connection string. - Divide the database pool. Replicas share PostgreSQL's
max_connections. - Rate limits are per process. Each replica keeps its own counters, so the effective limit grows with the replica count.
What is not covered by replicas#
- The worker is a single process. If it stops, restart it; schedules fire late according to each trigger's misfire policy, and nothing is lost.
- The database is a single PostgreSQL instance in the Docker Compose stack. For database-level redundancy, use a managed PostgreSQL 18 service or your own replication, and point
Database__ConnectionStringat it. - The site. A cell runs in one location. Recovery from the loss of a host or site is a restore from backup onto new infrastructure.
Recovery objectives#
The documented backup schedule gives these recovery points on a single host:
| Backup | Recovery point |
|---|---|
| Nightly cold backup | Up to 24 hours |
| Hot backup every 1 to 4 hours | Up to 4 hours |
Recovery time is dominated by restoring the database and the data volume. Measure it in a quarterly restore drill and record your own number. See Backup and restore.
Checklist#
- Escrow the key-encryption key and token pepper.
- Schedule cold and, if needed, hot backups, and copy them off the host.
- Run a restore drill every quarter and after every upgrade.
- Alert on
/health/ready, on worker errors and on Robots going offline. - If you run several API or gateway replicas, meet the requirements above, and keep exactly one worker.
Next steps#
What keeps working during an outage
Which roles you can run more than once
Requirements for more than one API process
What is not covered by replicas
Recovery objectives
Checklist
Next steps