R2E-Gym

Times the total time required to execute one complete RL training group: environment setup, snapshotting, four sequential rollout restores, tool execution, test grading, and sandbox cleanup.

Reinforcement learning trains a coding agent by running the same repository task repeatedly. A training loop sets up the repository once in a base sandbox, snapshots it, starts a group of rollout sandboxes from the snapshot, lets the policy execute tool calls in each one, and grades each rollout by running the task's test suite.

Because RL rollouts are short, sandbox lifecycle operations (create, snapshot, restore, and destroy) account for a large share of total training time and billed cost.

R2E-Gym measures that complete multi-sandbox cycle across ten tasks. In each trial, one sandbox is set up and snapshotted, and four recorded rollouts run one after another in their own sandboxes restored from that snapshot.

View the Procedure View R2E-Gym Results Download Raw Data View the Parameters

Metrics

R2E-Gym is the sum of every timed span across a trial's 5 sandboxes: one base sandbox and four rollout sandboxes.

Timed Part	Starts	Ends
Create	The create request	The create call returns
Reset	Each reset command starting, by the sandbox's own clock	it exits, by the sandbox's own clock
Snapshot	The snapshot request	The snapshot call returns
Create from Snapshot	The restore request	The restore call returns
Tool Calls	Each call starting, by the sandbox's own clock	it exits, by the sandbox's own clock
Reward	The reward command	The call returns, both by the sandbox's own clock
Destroy	The destroy request	The destroy call returns

T = c + r + s + \sum_{k=1}^{4} \left( p_k + u_k + w_k \right) + \sum_{j=1}^{5} d_j

T Group time: the sum of every timed span across all five sandboxes of a trial.
c Create time of sandbox 1, on the job host's clock.
r Reset time of sandbox 1, the in-sandbox runtime of the environment reset.
s Snapshot time of sandbox 1, on the job host's clock.
p_k Create-from-snapshot time of rollout k, on the job host's clock.
u_k Tool-call time of rollout k, the in-sandbox runtime of its recorded tool calls.
w_k Reward time of rollout k, the in-sandbox runtime of the task's test suite.
d_j Destroy time of sandbox j, on the job host's clock. The job host's times include network transport; the in-sandbox runtimes are measured by each sandbox's own clock and validated against /proc/uptime.

Measurement

Each trial uses 5 sandboxes created from the task's pinned R2E-Gym Docker image (namanjain12/numpy_final:<commit> or namanjain12/coveragepy_final:<commit>, with the R2E-Gym Python package and tools pre-installed) on providers that support snapshot restore through BoxRouter Core.

Tool Time Create sandbox 1. Timed The harness starts the loopback HTTP deny proxy.
Run R2E-Gym's 17-command reset. Timed
Snapshot and destroy sandbox 1 (Setup). Timed
For each of four rollouts, restore a sandbox from the snapshot (Create from Snapshot). Timed The harness starts the loopback deny proxy if the snapshot did not preserve running processes.
Replay recorded tool calls (Tool Calls). Timed The harness replays the rollout's recorded tool calls in /testbed as one background script.
Verify the git patch.
Tool Time Run the test suite (Reward). Timed The harness runs the task's reward evaluation script, checking that the parsed reward (0 or 1) matches the rollout's recorded outcome.
Destroy the sandbox (Destroy). Timed
Delete the snapshot.

The workload replays 10 tasks from R2E-Gym/R2E-Gym, each paired with 4 recorded rollout trajectories (run1 through run4, 40 rollouts total). Trajectories span 7 to 44 tool calls per rollout (828 tool calls total) and include both passing (reward = 1) and failing (reward = 0) runs, mirroring a GRPO training batch where candidate rollouts vary in length and outcome.

Task	Repository	Commit
numpy-masked-method-lookup	`numpy/numpy`	`0a476002`
numpy-issubdtype-abstract-types	`numpy/numpy`	`dda20349`
numpy-apply-axis-empty	`numpy/numpy`	`61640a84`
numpy-unstructured-single-field	`numpy/numpy`	`171a3e58`
numpy-masked-median-scalar	`numpy/numpy`	`bb46a495`
numpy-indices-sparse-option	`numpy/numpy`	`504b287b`
numpy-strided-custom-dtype	`numpy/numpy`	`3c0b4db0`
coveragepy-behavior-flag-rename	`nedbat/coveragepy`	`89f44c12`
numpy-masked-median-multiple	`numpy/numpy`	`6b8d55e6`
numpy-histogram-edge-rounding	`numpy/numpy`	`ad48eaa7`

Scope Limits and Caveats

The four rollouts in a group run sequentially rather than in a parallel burst, and tool calls run back to back without policy inference pauses.
A production RL training loop restores all sandboxes of a group at once, while a trial restores the four one after another, so four simultaneous restores may run slower or faster than four in sequence.
Because rollouts spend little time inside tool calls, group time is dominated by sandbox lifecycle overhead and test-suite execution (reward), which is the bottleneck for RL training.
Some providers snapshot full VM memory and running processes while others snapshot only the filesystem, and with no background process running at the snapshot the two kinds produce identical state here.
Nine of the ten qualifying tasks come from NumPy and one comes from coverage.py.

Failures and Exclusions

Outcome classifications, top-up rounds, and outlier rules follow Common Methodology; the first failed step ends the trial without in-trial retries.

A trial succeeds only when all 5 sandboxes complete their steps within timeout, every guest-clock span passes the twenty-millisecond /proc/uptime gate, all four rollouts produce their expected reward value (0 or 1), and all 5 sandboxes and the snapshot are confirmed deleted.

If a rollout's tool call exits with an unexpected status or its reward output disagrees with the recorded trajectory after clean execution, the trial is excluded (does not replay here); if any create, snapshot, restore, or destroy call fails, the trial is a provider error.

Step	Time Limit
Create	10 min
Reset	150 s
Snapshot	10 min
Create from Snapshot	10 min
Tool Calls	2 min
Reward	330 s
Destroy	5 min

Statistics

Based On: Common Methodology

Per-unit percentiles, bootstrap confidence intervals, and tail rules follow Common Methodology.

The All view aggregates each unit's selected statistic across the workload using the unweighted geometric mean and missing-unit substitution penalty defined in Common Methodology. A provider is ranked once it has at least 90% of the successes requested before any top-up: 180 across all units. It also needs a provider-sourced cost. One task on its own needs 90% of its 20 trials.

Cost

Aggregates a cost statistic over each task, a replay of four recorded rollouts from an agent's attempts at a repository task (selected from the R2E-Gym raw data).

Only successful trials contribute to the cost value. Each stage's cost is the shown cost times that stage's share of the Time bar under the same statistic.

Sandbox and Trials

Reference

Each task's own image, pinned by digest (docker.io/namanjain12/<repository>_final@sha256:<digest>); the digests are in the parameters' workloads.json.

Lifetime

15.0 min

Idle Connection Time

5.0 min

From Common Methodology

Idle Connection Exceptions

Provider	Rule	File Transfer Idle Time
Daytona	The SDK has no idle-timeout setting. Its command channel keeps its connection open, and its file-transfer channel reconnects after 5 s of idle time.	5.0 s

From Common Methodology

Network Path Attempts

From Common Methodology

Mode: Sequential
Sandboxes per Trial: 5
Parallel Trials: 10
Concurrent Sandboxes: 1

Provider Requirements

Required: Snapshots

At a Glance

Metrics

Tool Time Tool-call runtime
Reward Time Reward-command runtime

Unit

Milliseconds (ms)

Direction

Lower Is Better

Statistic

Median

Clock

Job-host clock and Sandbox clock

From Common Methodology

Machine Size

2 vCPU / 4 GiB

Trials

Variants

All Methods

R2E-Gym

Builds on Common Methodology

Times the total time required to execute one complete RL training group: environment setup, snapshotting, four sequential rollout restores, tool execution, test grading, and sandbox cleanup.

Because RL rollouts are short, sandbox lifecycle operations (create, snapshot, restore, and destroy) account for a large share of total training time and billed cost.