Sandbox Benchmarks
More providers coming soon.
Composite Workload Time
Composite time. Lower is better.
Composite Workload Cost
Composite cost. Lower is better.
Cold Startup Time
Create to first command. Lower is better.
All Methods
Overview
Sandboxes are evaluated on coding and CI workloads to determine a composite time and cost.
Composite Workload Time
Measures each provider’s execution times across coding and CI workloads combined into one composite time. Lower is better.
† Provider Failures: (CI Actions), (CI Actions), (Terminal-Bench 4.0). Click provider to see details.
Details
Workloads
Replayed workloads compare coding agent, CI, and RL work.
Coding Workloads
Coding agents' shell commands and file edits replay in each task's own image. No model is called in the replay.
DeepSWE Composite Time · All Traces
Aggregates a time statistic over each trace, a replay of commands from an agent's attempt to solve a repository task (selected from the DeepSWE v1.1 raw data). Time, lower is better.
Details
CI Workloads
Public repository jobs compare install, lint, test, and build steps.
CI Actions Composite Time · All Jobs
Aggregates a time statistic over each job, a replay of one repository's CI steps from a cold start (selected from open-source repositories' GitHub Actions workflows). Time per job, lower is better.
Not Ranked: , . Click provider to see details.
† Provider Failures: (gin and gson), (gson). Click provider to see details.
Details
RL Workloads
Recorded reinforcement-learning rollouts compare environment setup, tool calls, and reward checks.
R2E-Gym Composite Time · All Tasks
Aggregates a time statistic over each task, a replay of four recorded rollouts from an agent's attempts at a repository task (selected from the R2E-Gym raw data). Time per task, lower is better.
Not Displayed: Blaxel (not supported)
† Provider Failures: (numpy-strided-custom-dtype). Click provider to see details.
Details
Sandbox Operations
Basic sandbox calls are measured under a variety of configurations.
Sequential TTI
Times each sandbox from its create request until its first command returns. Lower is better.
Details
Machine Performance
Machine specifications are benchmarked by standard tests.
CPU Speed (7-Zip)
Measures single-core integer compute speed with 7-Zip's built-in compression and decompression benchmark on one thread. Rating, higher is better.
Details
ReliabilityComing Soon
Failure rates only count the provider-caused errors.