Execution Statuses

3 min

Overview

Scenario Statuses represent the lifecycle and final outcome of each scenario execution. They help you understand whether a scenario is in progress, completed successfully, failed, or affected by system or timing constraints.

Core execution result statuses such as Success, Failure, Error, and Skipped are primarily determined according to the XML report generated by the underlying test framework. The system reads and interprets these framework-level results to determine the final scenario outcome.

Additional statuses such as Waiting, Processing, Reporting, Terminating, Stopped, Timeout, Unexecuted, and Warning are system-level statuses defined by the system to provide better execution tracking and operational visibility.

Execution Flow

A scenario moves through several states during its execution lifecycle. Under normal conditions, the standard execution flow is:

Waiting → Processing → Running → Reporting → Final State

Depending on the execution conditions or user actions, a scenario may follow a different path.

In-Progress States

Waiting Scenario is queued and waiting for available resources (parallel slot, environment, device, etc.)

Processing Pre-execution checks are performed before the scenario starts (for example, environment readiness, configuration validation). If these checks fail, the scenario may return to Waiting

Running Scenario is actively being executed

Reporting Execution is completed, and results are being finalized

Terminating A stop action has been initiated for the plan containing the scenario. The scenario remains in this state while the system performs the required termination operations. After these operations are completed, the scenario transitions to Stopped.

Final States

Success Scenario completed successfully according to the framework XML report

Failure Scenario failed according to the framework XML report. This usually represents assertion failures, validation mismatches, or intentionally failed test flows defined within the automation code

Error Scenario could not be executed successfully due to system-related issues, or system could not find the XML report generated by the test framework. In both cases, the scenario is marked as Error. In some cases, the framework itself may also report system-level problems as Error instead of Failure

Skipped The scenario was marked as skipped by the test framework. This status is generally controlled by the automation code and may occur intentionally because of test conditions, configurations, dependencies, or runtime decisions

Warning A custom status defined by our system to indicate that the scenario completed with non-critical issues that may require attention, while not being treated as a direct failure or error

Timed Out Scenario exceeded the maximum execution time set by a user and was automatically terminated

Stopped The plan that the scenario belongs to is purposefully stopped by the user. After the termination flow is completed, the scenario transitions to Stopped. If a Plan execution is stopped, all scenarios that are still in progress (Running, Processing, or Waiting) first move to Terminating, and then to Stopped

Unexecuted Scenario could not reach the execution (build) phase due to a system-level issue and therefore was never executed

Notes

These statuses are used across Dashboard, Reports, and execution views for clear tracking and analysis

Framework XML reports are the primary source for execution result interpretation

Stopping a Plan affects all ongoing scenarios by transitioning them through Terminating → Stopped

Unexecuted indicates scenarios that never became runnable due to system issues before execution started

Processing acts as a validation gate before execution and may return to Waiting if checks fail