Overview

Builds on Common Methodology

Measures each provider’s execution times and sandbox costs across coding and CI workloads, combined into one composite time and one composite cost.

View the Procedures (3 Scripts)

• src/sandbox_benchmarks/procedures/deep_swe.py
• src/sandbox_benchmarks/procedures/terminal_bench.py
• src/sandbox_benchmarks/procedures/github_actions.py

View Overview Results

Metrics

Composite Workload Time

Composite Workload Time is the unweighted geometric mean of a provider's workload times across the workloads in the selected scope. Because the composite is reported as a concrete time rather than normalized to a 0–100 score or indexed to a reference provider, adding or removing another provider does not alter any provider's composite time when all units are completed.

Composite Workload Cost

Composite Workload Cost is the unweighted geometric mean of a provider's workload costs across the workloads in the selected scope.

• Requires a provider-reported or list-rate-calculated cost for every counted trial; if any counted trial lacks a provider-sourced cost or has a zero unit cost, the provider has no composite cost.

Composite Workload Time vs Cost

Composite Workload Time vs Cost is the point plotted for each provider, with Time Over the Costed Units on the vertical axis and Composite Cost on the horizontal axis.

• Displays only providers that have both a valid Composite Workload Cost and a Composite Workload Time over the costed unit set; unpriced providers are listed under Not Displayed.

T = \left( \prod_{k=1}^{n} t_k \right)^{1/n}

• T Composite workload time.
• t_k Time for workload k, with the missing-result policy applied.
• n Number of workloads in the selected scope.

Workload	Quantity	Execution	Coding	CI
DeepSWE	Time per trace	Yes	Yes	No
Terminal-Bench 4.0	Time per trace	Yes	Yes	No
CI Actions	Time per job	Yes	No	Yes

Aggregation proceeds in two geometric-mean levels so neither long-running units nor multi-unit workloads dominate the index:

• First, within each workload, the provider's workload time is the geometric mean across all units completed by at least one provider, using the selected statistic (median by default, or mean or p95); if a provider completed at least one unit in the workload but has no successful trials on a unit, that unit's value is replaced by the slowest value observed for that unit across all providers that completed it.
• Second, the provider's Composite Workload Time is the unweighted geometric mean of its workload times across the workloads in the selected scope, giving each workload equal proportional weight; if a provider has no completed units in an entire workload, its workload time is replaced by the slowest workload time observed across providers that completed that workload.

Like Composite Workload Time, Composite Workload Cost aggregates in two geometric-mean steps:

• Within each workload, a provider's workload cost is the geometric mean of its per-unit costs across all units completed by at least one provider, substituting the highest observed unit cost when a provider failed a unit.
• Across workloads in the selected scope, Composite Workload Cost is the unweighted geometric mean of those workload costs, substituting the highest observed workload cost if a provider failed an entire workload.

In Composite Workload Time vs Cost, each provider is plotted by two coordinates: Time Over the Costed Units (vertical axis, lower is better) and Composite Cost (horizontal axis, in USD, lower is better). Providers down and to the left deliver both faster execution and lower billed cost, forming the Pareto frontier.

For every provider with a valid Composite Workload Cost, the horizontal axis coordinate is its Composite Workload Cost value and the vertical axis coordinate is the geometric mean of its workload times restricted to the costed units, using the same equal workload weights and missing-unit substitution rules as Composite Workload Time and Composite Workload Cost.

To keep both axes directly comparable when a provider has cost data for only a subset of units, Composite Workload Time vs Cost computes each workload's time over the same set of costed units that enter that provider's Composite Workload Cost calculation.

Composite Workload Time runs no separate trials of its own; it aggregates the completed trial records of the constituent workloads in the selected scope. Every underlying trial follows the sandbox sizing, image pinning, guest-clock timing, and post-run verification rules defined on the DeepSWE, Terminal-Bench 4.0, and CI Actions methodology pages.

Composite Workload Cost uses the same underlying trials as Composite Workload Time. For each successful trial, the harness sums the provider-sourced cost of the sandbox from create to destroy and scales per-unit costs to cost per 1,000 successes where displayed on individual workload pages.

Scope Limits and Caveats

• Applies no extra ranking floor beyond the underlying workloads, and substitutes the slowest observed unit or workload time when a provider has partial results so failing a slow task cannot improve a provider’s composite.
• All three input workloads measure in-sandbox step execution on the guest clock, while sandbox creation, image preparation and per-call API time are measured separately in Sandbox Operations.

Failures and Exclusions

A workload is included in a scope only if every provider offers the capabilities that workload requires; if any provider's result on a workload is marked Not Offered or Account Limit, that workload is omitted from the composite scope for all providers. Providers with zero completed units across all workloads in the selected scope are not displayed.

A provider is omitted from Composite Workload Cost and Composite Workload Time vs Cost and listed under Not Displayed if any of its counted trials in the scope lacks a provider-sourced cost, if any unit's cost on the selected statistic is zero (so the geometric mean is undefined), or if the provider completed no units in the scope.

At a Glance

Short Name

Composite time and cost

Metrics

• Composite Workload Time Time · Milliseconds (ms)
• Composite Workload Cost Cost · US Dollars (USD)
• Composite Workload Time vs Cost Time and cost

Direction

Lower Is Better

Statistic

Geometric Mean

Scopes

• Execution
• Coding
• CI