Run tests

Start executions, let Robots run the automated test cases as jobs, record manual results step by step, rerun failures, and report results from CI or Studio.

An execution is one run of a test set (or of chosen test cases). It gets a key (EX-1) and one result per test case, pinned to the test case version that ran. How each result is produced depends on the test case: Test case Result source What happens Manual, or Automated/Hybrid without a link Manual A tester records it in the execution assistant. Linked to an Orchestrator test Automated QA Hub starts a job for the test in the project's default workspace. A Robot runs it and the job's outcome becomes the result. Linked to another framework (pytest, Playwright, JUnit…) External The result arrives in a report from your CI pipeline or from Studio. An execution that has both manual and automated results is Mixed. Statuses# Execution Meaning Pending Started; nothing recorded yet. Running Results are being recorded or jobs are running. Completed Every result is settled. Cancelled Someone cancelled it. Pending results are cancelled and running jobs are stopped. Timed out It ran past its deadline (12 hours after the start by default). A result is Pending, Running, Passed, Failed, Blocked, Skipped, Error (the test could not run to the end) or Cancelled. Once an execution is completed, cancelled or timed out, its results are read-only. Before you run automated test cases# The project has a default workspace and a resynced catalog. See Projects and settings. The automated test cases are linked to tests in the catalog. See Test cases. A machine assigned to that workspace has a connected Robot with the package's runtime, and a robot account is assigned to the workspace. See Machines and robots. Start an execution# Open the test set and select Execute (or select Run now in the Test sets list). The dialog shows how many test cases will run, and how many are manual and automated. Optionally change the Name, and pick the Release and the Environment (from the project's environments list). Select Start execution. The execution opens. To run one test case on its own, open it and select ⋮ ▸ Run. Follow an execution# The execution page is Live: counters and results update as Robots finish. For automated results, Robot / tester shows the Robot that ran the job. Filter the results with the status chips (All, Pending, Passed, Failed…) or Search. Affected requirements lists the requirements whose test cases failed. Details shows the test set, who started it and the deadline. Cancel stops the execution: pending results are cancelled, running jobs are stopped, recorded results stay. Export CSV downloads the results. Each automated test runs as an Orchestrator job with source Test. Test jobs are never retried automatically, so a flaky test shows up as a failure. You also see them under Automations ▸ Jobs in the workspace. See Jobs. When the execution finishes you get a notification with its counts. Record a manual result# In the execution, select ⋮ on a pending manual result ▸ Open the assistant. Follow each step and mark it Pass, Fail, Block or Skip. For Fail and Block, describe the actual result. Each step is saved as you go. Optionally select Attach evidence on a step, or in the Evidence panel, to add screenshots or files. The Progress panel shows the timer, the counts and the Derived outcome: failed if any step failed, else blocked if any is blocked, else passed. Skipped steps do not count. Select Finish, check the summary and confirm with Finish. After finishing, choose Create defect, Next pending or Back to EX-…. Keyboard shortcutsIn the assistant, P, F, B and S mark the current step, N or J go to the next step, K to the previous one, and ? shows the list. Read a result# Open a result to see its outcome, the failure reason, the assertions the test reported (expected, actual, message), the version executed, package version, runner, the job and its attempts. The Logs tab shows the job's log lines (filter them by level), and Open in Jobs opens the job in Orchestrator. Evidence holds screenshots and files. While the execution is still open, Override replaces a settled result with Passed, Failed, Blocked or Skipped. A comment is required, and the original status is kept. Rerun the failures# When an execution has ended, select Rerun failed. QA Hub starts a new execution of the Failed, Error and Blocked results, named after the original with "(rerun)". Find executions# Testing ▸ Executions lists every execution with its test set and environment, status, kind, results and who started it. Report results from CI or Studio# From a CI pipeline# Tests that run outside Orchestrator (pytest, Playwright, JUnit and others) report their results to QA Hub. Link each test case to the framework and its Test id first (see Test cases). Create a personal access token with permission to create executions (the QA Hub CI set of permissions is enough). See Authentication. Send the results: Report test resultsHTTPCopyPOST /api/v1/tenants/{tenantId}/qa-hub/projects/{projectId}/executions/report Authorization: Bearer <your access token> Idempotency-Key: nightly-2026-10-06 Content-Type: application/json { "name": "Nightly", "provider": "pytest", "environmentKey": "staging", "results": [ { "testId": "tests/test_refund.py::test_ok", "outcome": "Failed", "durationMilliseconds": 1200, "message": "Refund not posted" } ] } outcome is Passed, Failed, Skipped or Error. QA Hub creates a completed execution from the matched test cases and answers with the execution and the matched, unmatched and rejected test ids. Test ids that match no test case are listed under unmatched and never create test cases. When nothing matches, no execution is created. To send a JUnit XML file instead, post it to the same address with Content-Type: application/xml and put the other values in the query string, for example ?provider=junit&environmentKey=staging. From Studio# Studio's ribbon has a QA Hub menu for the open project: Connect project… picks the QA Hub project the Studio project's test cases belong to. Link test case… links the active test case to a QA Hub test case, or creates one there. Unlink test case removes the link. Publish test cases creates a QA Hub test case for every test case without one and links each to its published automation. Run and report runs the linked test cases and reports their results to QA Hub as one execution. Upload test results sends the Test Explorer's last results without running them again. QA Hub opens the project, or the active test case, in the browser. Disconnect stops reporting; test cases in QA Hub are kept. Results from Studio show runner Studio and package version local. See Run and debug for Studio's Test Explorer. Next steps# Defects Reports

Statuses

Before you run automated test cases

Start an execution

Follow an execution

Record a manual result

Read a result

Rerun the failures

Find executions

Report results from CI or Studio

From a CI pipeline

From Studio

Next steps