Methodology

Read what each benchmark measures, how it is calculated, and how to interpret its results.

Shared Methodology

• Common Methodology Defines the test environment, trial lifecycle, timing clocks, failure classifications, statistical estimators, and independence checks shared across all BoxRouter benchmarks.
• Pricing Methodology Defines how sandbox costs are obtained from provider billing records, aggregated across benchmark trials, and verified against published list rates.

• Test Environment
• Trial Lifecycle
• Timing and Clocks
• Failures, Exclusions, and Top-Ups
• Statistics
• Provider Scores
• Independence

Overview

• Overview Measures each provider’s execution times across coding and CI workloads combined into one composite time.

Coding Workloads

• DeepSWE Times the sum of each command's runtime in a replayed DeepSWE trace, on the sandbox's clock, leaving out model speed and agent quality.
• Terminal-Bench 4.0 Times the total guest-clock runtime required to execute recorded Claude Code shell commands and file edits across ten terminal tasks, with model inference excluded.

CI Workloads

• CI Actions Times each job as the sum of its step runtimes on the sandbox's clock; All Jobs is the geometric mean of the selected jobs.

RL Workloads

• R2E-Gym Times the total time required to execute one complete RL training group: environment setup, snapshotting, four sequential rollout restores, tool execution, test grading, and sandbox cleanup.

Sandbox Operations

• Create Times each sandbox from its create request until its first command returns.
• Exec Times 100 calls in a row to a command with almost no runtime and adds up their times.
• Files Times the job-host round trip for BoxRouter to read a 4 KiB incompressible file from the sandbox and return its contents.

• Pause and Resume

  Interactive coding agents, chat workspaces, and notebook sessions sit idle between user turns.

  • Pause and Resume Times a sandbox's pause, its resume and a file read afterwards, under different disk loads.
  • Delayed Resume Times reading data from a sandbox that was paused or idle for 0 s, 60 s or 15 min.

• Snapshots

  Agent harnesses and reinforcement learning loops use snapshots to save a configured sandbox once and boot clean copies from that state without repeating installation or repository setup.

  • Snapshot and Restore Times how long a provider takes to save a sandbox snapshot, create a new sandbox from it, and run the first Python, Node.js, and git command.
  • Fork Times how long a provider takes to create 4 sequential sandboxes from a single snapshot and run the first Python, Node.js, and git command in each fork.
• Previews Times from launching a Vite development server inside a sandbox until its public preview URL returns the HTTP 200 response to the job host, split into Server and URL Reachable.
• Reconnect Times a fresh client process attaching to an existing sandbox by its ID and executing its first true command.

Machine Performance

• CPU Measures single-core integer compute speed with 7-Zip's built-in compression and decompression benchmark on one thread.
• Memory Measures sustained main-memory bandwidth with the STREAM Triad benchmark across three 610.4 MiB arrays using one OpenMP thread per requested vCPU.
• Disk Measures uncached 1 MiB sequential read throughput from a 1 GiB file in /workspace/fio using direct I/O (O_DIRECT) over a 60 s measured window.
• Filesystem Measures the rate of hard links created per second in /workspace/hardlink over a 30-second stress-ng run, with link removal time included.
• Network Times the in-sandbox command runtime to clone a pinned GitHub repository or install pinned npm or PyPI dependencies from a cold cache, by the sandbox's own clock.

Reliability Coming Soon

Failure rates only count the provider-caused errors.