Webhooks

Subscribe an HTTPS endpoint to job, queue, Robot and package events and verify each signed delivery.

Webhooks let the Orchestrator tell your systems when something happens, instead of your systems polling the API. A tenant subscribes an HTTPS endpoint to the event types it cares about, and the Orchestrator POSTs each event as a signed CloudEvent.

Every event is written in the same database transaction as the change it describes. If the change rolls back, the event disappears with it, and if the process stops after the commit, the event is delivered on the next sweep. You are never told about a job that does not exist, and you are not silently skipped for one that does.

Event types#

Type When it fires
com.velophex.job.queued A job was accepted and is waiting to be claimed
com.velophex.job.started A Robot claimed the job and an attempt is running
com.velophex.job.succeeded The job finished successfully
com.velophex.job.faulted The job failed and will not be retried
com.velophex.job.stopped The job was canceled
com.velophex.job.dead_lettered The job's retries ran out. This is terminal and the best one to alert on
com.velophex.job.outcome_reconciled A result that arrived after the lease expired replaced the job's unknown outcome
com.velophex.attempt.succeeded One attempt of a job succeeded
com.velophex.attempt.faulted One attempt failed; a later attempt may still succeed
com.velophex.queue_item.added An item was added to a queue
com.velophex.queue_item.started A job took the item
com.velophex.queue_item.completed The item was processed successfully
com.velophex.queue_item.failed Processing the item failed
com.velophex.queue_item.abandoned The item was abandoned
com.velophex.queue_item.retried The item was replaced by a retry
com.velophex.machine.registered A host enrolled
com.velophex.machine.deleted A host was removed
com.velophex.robot.online The Robot on a machine reported in
com.velophex.robot.offline The Robot went quiet past the tenant's threshold
com.velophex.package.published A package version reached the tenant feed and can be run
com.velophex.alert.raised An alert rule with the webhook channel fired
com.velophex.alert.resolved The condition behind an alert cleared
com.velophex.storage.file.created A file was uploaded, restored or moved into a storage bucket
com.velophex.storage.file.deleted A file or folder was deleted or moved away
com.velophex.storage.bucket.changed A storage bucket was created, changed or deleted

The catalog also contains event types for optional products, such as the QA Hub and Launchpad, which fire only when those products are enabled for the tenant. The live list is served by GET /api/v1/tenants/{tenantId}/webhooks/event-types. A subscription that names an unknown type is refused with 400, so a typo cannot create a subscription that never fires.

Create a subscription#

Prerequisites#

  • The settings.edit permission in the tenant (settings.view to read subscriptions, settings.delete to delete them).
  • A reachable https endpoint that answers with a 2xx status. Plain http and addresses in loopback, link-local or private ranges are refused, both when you save the URL and when the delivery connects.

Steps#

  1. In the web app, open Webhooks in the tenant menu and add a subscription, or call the API:

    HTTPCopy
    POST /api/v1/tenants/{tenantId}/webhooks HTTP/1.1
    Authorization: Bearer vpx_pat_...
    Content-Type: application/json
    
    {
      "url": "https://hooks.example.com/velophex",
      "eventTypes": ["com.velophex.job.dead_lettered", "com.velophex.robot.offline"],
      "description": "Page the on-call team",
      "isActive": true
    }
  2. Copy the secret from the response. It is shown only in this response and cannot be read back. If you lose it, roll it with POST /webhooks/{id}/secret; the old secret stops verifying immediately.

  3. Store the secret with your receiver and implement signature verification as shown below.

Verify#

Trigger an event, for example by starting a job that fails. Then open the subscription's deliveries, or call GET /api/v1/tenants/{tenantId}/webhooks/{id}/deliveries. You should see a delivery with status Delivered. A delivery that failed shows the last status code and the start of what your endpoint answered.

The request your endpoint receives#

The body is a CloudEvents 1.0 event in structured JSON mode, with Content-Type: application/cloudevents+json:

JSONCopy
{
  "id": "0199a4c1-7b2e-7c31-9f40-2a1d4c6e8b90",
  "source": "/velophex/orchestrator/tenants/7f9c2b18-...",
  "type": "com.velophex.job.faulted",
  "subject": "3d1f8a44-...",
  "time": "2026-09-22T17:51:14.4821330+00:00",
  "specversion": "1.0",
  "datacontenttype": "application/json",
  "data": { "jobId": "3d1f8a44-...", "workspaceId": "...", "status": "Failed", "attemptCount": 3, "faultMessage": "..." }
}

subject is the id of the thing the event is about: a job id for job.*, an attempt id for attempt.*, a queue item id for queue_item.*, a machine or Robot id for presence events, a package version id for package.published, the alert rule id for alert.*, and the bucket id for storage.*. You can route on it without parsing data. Events never copy secrets or queue item content into data.

id and time are fixed when the change is committed. Every retry of a delivery carries the same id and the same body, so use id to deduplicate.

Three headers accompany each request:

TextCopy
VeloPhex-Signature: t=1790104274,v1=9f8c1a...
VeloPhex-Delivery:  0199a4c1-7b2e-7c31-9f40-2a1d4c6e8b91
VeloPhex-Event:     com.velophex.job.faulted

Verify the signature#

The signed material is the timestamp t (Unix seconds, in decimal), one full stop, and the raw request body exactly as received:

TextCopy
ASCII(t) || "." || body

v1 is the HMAC-SHA-256 of that material, keyed with the subscription secret as UTF-8 bytes, encoded as lowercase hex. To verify a request:

  1. Read the raw body before you parse it as JSON.
  2. Recompute the HMAC and compare it with v1 in constant time.
  3. Reject the request if t is further from your clock than you tolerate. Five minutes is a common choice.

The timestamp check is required: without it, a captured request could be replayed forever. The timestamp is taken per attempt, so a retry has a different signature even though its body is identical.

PythonNode.jsverify.pyPythonCopy
import hashlib
import hmac
import time


def is_valid(raw_body: bytes, signature_header: str, secret: str, tolerance: int = 300) -> bool:
    fields = dict(item.split("=", 1) for item in signature_header.split(","))
    timestamp, received = fields.get("t"), fields.get("v1")
    if not timestamp or not received:
        return False
    if abs(time.time() - int(timestamp)) > tolerance:
        return False
    message = timestamp.encode("ascii") + b"." + raw_body
    expected = hmac.new(secret.encode("utf-8"), message, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, received)
verify.tsTypeScriptCopy
import { createHmac, timingSafeEqual } from "node:crypto";

export function isValid(rawBody: Buffer, header: string, secret: string, tolerance = 300): boolean {
  const fields = Object.fromEntries(header.split(",").map((part) => part.split("=", 2)));
  const t = fields["t"];
  const v1 = fields["v1"];
  if (!t || !v1) return false;
  if (Math.abs(Date.now() / 1000 - Number(t)) > tolerance) return false;

  const expected = createHmac("sha256", Buffer.from(secret, "utf8"))
    .update(Buffer.concat([Buffer.from(`${t}.`, "ascii"), rawBody]))
    .digest("hex");
  const a = Buffer.from(expected, "utf8");
  const b = Buffer.from(v1, "utf8");
  return a.length === b.length && timingSafeEqual(a, b);
}

If your receiver must act exactly once, also remember each CloudEvent id you have processed and ignore repeats.

Delivery, retry and dead letter#

  • Any 2xx answer completes the delivery.
  • Anything else, or no answer within 10 seconds, is a failed attempt. It is retried after 1, 2, 4 and 8 minutes.
  • After five attempts the delivery is dead-lettered and not retried automatically. The record keeps the last status code and the first 500 characters of the response. You can put a dead-lettered delivery back in the queue with POST /webhooks/{id}/deliveries/{deliveryId}/retry.
  • Redirects are not followed. If your endpoint moves, update the subscription URL.
  • After 20 dead-lettered deliveries in a row (Webhooks:DeactivateAfterConsecutiveFailures), the subscription is switched off and shows when and why. A successful delivery resets the count. To turn it back on, set isActive to true, which also clears the count.

Answer quickly and do slow work asynchronously, so that deliveries do not time out.

Manage subscriptions#

All routes are under /api/v1/tenants/{tenantId}.

Method Route Purpose
GET /webhooks List subscriptions (paged; q filters by URL and description)
GET /webhooks/event-types The event type catalog
POST /webhooks Create; the only response that contains the secret
GET /webhooks/{id} Read one
PUT /webhooks/{id} Change URL, event types, description or isActive
POST /webhooks/{id}/secret Roll the secret
DELETE /webhooks/{id} Delete, with anything still queued for it (settings.delete)
GET /webhooks/{id}/deliveries Recent deliveries, newest first; status (Pending, Delivered or DeadLettered) and limit (default 20) narrow the list
POST /webhooks/{id}/deliveries/{deliveryId}/retry Retry a dead-lettered delivery with a fresh set of attempts

Events and deliveries are kept for 7 days by default (Webhooks:RetentionDays), which is also how far back the deliveries list reaches.

Next steps#

Event types

Create a subscription

Prerequisites

Steps

Verify

The request your endpoint receives

Verify the signature

Delivery, retry and dead letter

Manage subscriptions

Next steps