DeepSWE

Builds on Common Methodology

Times the sum of each command's runtime in a replayed DeepSWE trace, on the sandbox's clock, leaving out model speed and agent quality.

DeepSWE measures how long each sandbox provider takes to execute the shell commands of a realistic software engineering session. We take ten sessions recorded by Claude Opus 5 (high reasoning) on real repository tasks in DeepSWE v1.1 and replay their shell commands in order, without the model.

It shows which providers offer the fastest tool use for a continuously running agent. Replaying removes the model, whose inference time would otherwise dominate the measurement, and a live model takes a different path on every attempt. Every provider runs the same commands.

View the Procedure View DeepSWE Results Download Raw Data View the Parameters

Metrics

DeepSWE is the sum of command runtimes across a replayed trace. Each command is assigned by a deterministic command-string parser in priority order to one of five mutually exclusive stages: Test -> Install -> Build -> Write -> Read.

Timed Part	Starts	Ends
Command replay	The command starts, by the sandbox's own clock.	The command exits, by the sandbox's own clock; its runtime is the difference between two consecutive readings.

Stage	Matches
Test	Test runners (pytest, tox, unittest, nosetests, cargo test, go test, npm test, jest, vitest, ctest) and inline verification commands when they are not combined with file-mutation operators
Install	Package managers (pip, conda, poetry, uv, npm, yarn, pnpm, cargo install, apt-get, apk)
Build	Compilers, build tools, linters (ruff, flake8, black, isort, mypy, pylint, eslint, tsc), and git commit/merge/rebase/cherry-pick
Write	File-creation and mutation commands (apply_patch, sed -i, patch, git apply/checkout/reset/restore/clean, mkdir, mv, cp, rm, chmod, touch, heredocs, redirects, and inline python -c scripts that open files for writing or call os.remove/os.rename/shutil)
Read	The default fallback for repository navigation, search (grep, rg, find, cat, head, tail, git status/diff/log/show), and read-only inspection

T = \sum_{k=1}^{K} \left( e_k - s_k \right)

• T Tool time of a replay, or job time of a CI job: the sum of its command or step runtimes on the sandbox's clock.
• K Number of commands or steps in the trace, session or job.
• s_k The script's $EPOCHREALTIME reading immediately before command or step k.
• e_k The script's $EPOCHREALTIME reading immediately after command or step k.

Measurement

Each trial uses a sandbox created from the task's pinned DeepSWE Docker image (public.ecr.aws/d3j8x8q7/swe-bench-202605:<tag>-v1.1, pinned by SHA-256 digest) with outbound network traffic blocked.

1. Create sandbox.
2. Run readiness check.
3. Start loopback deny proxy. Before replay begins, an untimed setup step launches a local Python HTTP proxy on localhost and sets HTTP_PROXY, HTTPS_PROXY, http_proxy, https_proxy, and PIP_INDEX_URL so any accidental external network request fails immediately with HTTP 403, matching upstream DeepSWE's offline evaluation posture.
4. Tool Time Replay all trace commands in /app as one background script (Read, Write, Install, Build, Test). Timed Each command is wrapped in Bash $EPOCHREALTIME and /proc/uptime readings and guarded by timeout --kill-after=5s, with a per-command limit of at least three minutes or three times the original recorded step duration.
5. Verify final git patch SHA-256.
6. Destroy sandbox.

The workload replays 10 single-agent traces from datacurve-ai/deep-swe, spanning 25 to 112 sequential bash tool calls per trace (599 commands total) and 8 s to 199 s of tool time.

Candidate traces exclude tasks requiring GPUs or custom kernel modules, tasks exceeding 4 GiB peak RAM, tasks whose upstream evaluation failed or hit step limits, and tasks with non-deterministic test suites (network calls, wall-clock sleeps, or unseeded random tests).

The remaining pool is stratified across heavy C/C++/Rust compilation, large Python test suites, and rapid file-editing and navigation loops into 10 traces: abs-stepped-slices, actionlint-action-pinning-lint, adaptix-name-mapping-aliases, aiomonitor-webui-asyncio-console, alembic-autogenerate-Variant-types, anyio-CancelScope-lifecycle-bug, arviz-plot-lm-dims, astroid-six-moves-brain, atrx-vep-crispr, and authlib-flask-oauth-client.

Scope Limits and Caveats

• Commands run back to back without LLM thinking pauses (keeping the OS page cache warm and preventing idle auto-pause), and tool time measures in-sandbox execution rather than per-command API round-trip time.
• Where a provider's container reports the host's CPU count instead of the requested vCPUs (as Daytona does), tools that size worker pools from the visible count start more threads, and the result counts that as the provider's behavior.
• Blaxel offers no provider-level network lockdown, so only the in-sandbox loopback proxy blocks outbound traffic there, and tools that ignore HTTP_PROXY (such as Node's built-in fetch) can still reach the internet.
• Providers that lazily fetch container image blocks on first read run their first repository commands slower, which tool time captures because users see the same delay.
• All ten traces come from Claude Opus 5 (high), exclude sessions that walk / with find, background jobs and 30 s timeouts, and include no Rust repository.
• Running on 4 vCPUs rather than DeepSWE's 2 CPUs speeds up multi-threaded builds and starts more parallel test workers than during the original recording.
• Replaying each trace 20× within one batch measures run-to-run spread within that batch, not day-to-day provider drift.

Failures and Exclusions

Outcome classifications, top-up rounds, and outlier rules follow Common Methodology; the first failed step ends the trial without in-trial retries.

A trial succeeds only when all commands return their expected exit codes, every command's guest clock reading agrees with /proc/uptime within twenty milliseconds, the final git diff HEAD and git status --porcelain SHA-256 digest matches the trace's expected patch hash, and the sandbox is confirmed destroyed.

Step	Time Limit
Create	10 min
Command replay	10 min
Destroy	5 min

Outcome	When
Excluded (does not replay here)	A command exits with a different status code, times out (exit 124) while the sandbox remains responsive, or produces a mismatched final git patch hash
Provider error	The runner script terminates early without writing a mismatch record or the sandbox stops answering

Statistics

Based On

Common Methodology

Per-unit percentiles, bootstrap confidence intervals, and tail rules follow Common Methodology.

The All view aggregates each unit's selected statistic across the workload using the unweighted geometric mean and missing-unit substitution penalty defined in Common Methodology. A provider is ranked once it has at least 90% of the successes requested before any top-up: 180 across all units. It also needs a provider-sourced cost.

Cost

Aggregates a cost statistic over each trace, a replay of commands from an agent's attempt to solve a repository task (selected from the DeepSWE v1.1 raw data).

Only successful trials contribute to the cost value. Each stage's cost is the shown cost times that stage's share of the Time bar under the same statistic.

Sandbox and Trials

Reference

Each trace's task image, pinned by digest (public.ecr.aws/d3j8x8q7/swe-bench-202605:<task tag>-v1.1@sha256:<digest>); the digests are in the parameters' workloads.json.

Lifetime

25.0 min

Idle Connection Time

5.0 min

From Common Methodology

Idle Connection Exceptions

Provider	Rule	File Transfer Idle Time
Daytona	The SDK has no idle-timeout setting. Its command channel keeps its connection open, and its file-transfer channel reconnects after 5 s of idle time.	5.0 s

From Common Methodology

Network Path Attempts

10

From Common Methodology

Mode

Single

Sandboxes per Trial

1

Parallel Trials

10

Concurrent Sandboxes

1

Provider Requirements

Policy

All network access is blocked.

At a Glance

Metrics

• Tool Time Command runtime
• Time per Trace Set Total command runtime

Unit

Milliseconds (ms)

Direction

Lower Is Better

Statistic

Median

Clock

Sandbox clock

From Common Methodology

Machine Size

4 vCPU / 8 GiB

Trials

20

Variants

• Claude Opus 5 high