Runtime
Runtime values explain how long a test run took and how execution time is represented across different levels of reporting. These metrics help users understand execution behavior and compare test runs consistently.
The system provides two runtime metrics, each serving a different analytical purpose.
1 - Total Execution Runtime
Total Execution Runtime represents the end-to-end duration of a test execution, measured from the moment the user triggers the Run Test action until all execution-related processes are fully completed.
This value reflects the actual real-world time experienced by the user.
- It includes waiting time before execution begins due to environment availability (queue time).
- It includes all execution operations.
- It can be reduced by the amount of parallel executions. More parallel means faster execution.
The purpose of Total Execution Runtime is to show the real-world elapsed time of a test execution from the moment the user triggers the run action until completion.
2 - Total Scenario Runtime
Total Scenario Runtime represents the cumulative execution time of all test scenarios, calculated by summing the execution durations of each scenario individually.
This metric represents a theoretical execution time and does not reflect real-world elapsed duration.
- It excludes waiting time before execution begins due to environment availability (queue time).
- It includes code building, scenario-level execution, and time spent running test logic and producing results for each scenario, individually.
- The value does not change based on the amount of parallel executions.
- It always reflects the sum of individual scenario execution times.
The purpose of Total Scenario Runtime is to show the total amount of test execution work performed, independent of real-world time and parallel execution.