Terminal-Bench 4.0

Builds on Common Methodology

Times the total guest-clock runtime required to execute recorded Claude Code shell commands and file edits across ten terminal tasks, with model inference excluded.

Terminal-Bench 4.0 evaluates agentic terminal work across software engineering, scientific computing, machine learning, hardware CAD, formal theorem proving, and operations. We take ten Claude Code sessions recorded by Claude Opus 5.5 (max reasoning) from its public leaderboard and replay their shell commands and file edits in order, without the model.

It shows which providers execute terminal tool calls and file edits fastest across scientific and systems workloads. Replaying removes the model, whose inference time would otherwise dominate the measurement, and a live model takes a different path on every attempt.

View the Procedure View Terminal-Bench 4.0 Results Download Raw Data View the Parameters

Metrics

Terminal-Bench 4.0 is the sum of step runtimes across a replayed session of Claude Code tool calls and Bash commands. Each step is assigned to one of four stages: Write for Write and Edit calls (replayed via base64 and perl), Read for Read, Glob, and Grep calls, and Test -> Build -> Write -> Read in priority order for Bash commands.

Timed Part	Starts	Ends
Session replay	The step starts, by the sandbox's own clock.	The step exits, by the sandbox's own clock; its runtime is the difference between its two readings.

T = \sum_{k=1}^{K} \left( e_k - s_k \right)

• T Tool time of a replay, or job time of a CI job: the sum of its command or step runtimes on the sandbox's clock.
• K Number of commands or steps in the trace, session or job.
• s_k The script's $EPOCHREALTIME reading immediately before command or step k.
• e_k The script's $EPOCHREALTIME reading immediately after command or step k.

Measurement

Each trial uses a sandbox created from the task's pinned Terminal-Bench 4.0 image plus Harbor's system dependencies, running as root with default open networking.

1. Create sandbox.
2. Run readiness check.
3. Tool Time Replay all recorded Bash, Edit (perl), and Write (base64) steps (Read, Write, Build, Test). Timed The harness uploads the trace's step payloads and launches a single background replay script (polled every ten seconds) that executes every recorded Bash command and file edit in sequence inside the task's working directory, timing each step on the sandbox's own clock via $EPOCHREALTIME and /proc/uptime readings.
4. Destroy sandbox. After the last replayed step completes, the harness runs the task's upstream verification script (tests/test.sh) untimed and checks that /logs/verifier/reward.txt contains 1 (pass).

The workload replays 10 Harbor sessions (6 to 71 steps per session, 268 steps total) across parser translation, C-to-Rust migration, compiler builds, binary debugging, Git recovery, image processing, and Awk/regex scripting: bnf-to-ebnf, c-to-rust-stats, compile-compcert, cron-cheater, debug-long-program, git-repo-recovery, image-stitching, linux-kernel-module, log-summary-awk, and regex-golf.

Each task image builds on the upstream environment/ directory of harbor-framework/terminal-bench at the pinned commit, pre-installing curl and procps where missing, pre-populating Harbor's test-dir dependencies, and stripping /root/.local/bin and /root/.nvm so commands resolve the task image's own language runtimes rather than Harbor's Node and Python wrappers.

Scope Limits and Caveats

• Steps run back to back without LLM pauses, file edits spawn a subprocess (perl or base64) rather than writing in-process from Node.js, and selection filters out sessions with network installs or background jobs.
• Tool time measures in-sandbox execution (runtime(cmd)), which agents running inside the sandbox pay directly, while external harnesses that send one API request per command also pay per-call transport time.
• Filtering out background jobs, 10 min timeouts, subagents and network installs selects shorter, cleaner trajectories than average.
• Claude Code edits files from its own Node process, while the replay starts a perl or base64 process for each of the 66 Edit and Write steps.
• The task images were built after the leaderboard runs, and several upstream Dockerfiles install unpinned apt or micromamba packages, so minor tool versions can differ from the original recording.
• Where a provider's container reports the host's CPU count instead of the requested vCPUs (as Daytona does), tools that size worker pools from the visible count start more threads, and the result counts that as the provider's behavior.

Failures and Exclusions

Outcome classifications, top-up rounds, and outlier rules follow Common Methodology; the first failed step ends the trial without in-trial retries.

A trial succeeds only when every replayed step exits with its expected code, every step's guest clock reading passes the twenty-millisecond /proc/uptime gate, the upstream verifier (tests/test.sh) writes reward 1, and the sandbox is confirmed destroyed.

Trials where a deterministic trace command diverges on a provider or where the verifier does not write 1 after clean command execution are classified as excluded (does not replay here), whereas transport errors, unresponsive sandboxes, or early script termination are classified as provider errors.

Step	Time Limit
Create	10 min
Session replay	10 min
Destroy	5 min

Statistics

Based On

Common Methodology

Per-unit percentiles, bootstrap confidence intervals, and tail rules follow Common Methodology.

The All view aggregates each unit's selected statistic across the workload using the unweighted geometric mean and missing-unit substitution penalty defined in Common Methodology. A provider is ranked once it has at least 90% of the successes requested before any top-up: 180 across all units. It also needs a provider-sourced cost.

Cost

Aggregates a cost statistic over each trace, a replay of an agent's commands and file edits on a terminal task (selected from the Terminal-Bench 4.0 public leaderboard).

Only successful trials contribute to the cost value. Each stage's cost is the shown cost times that stage's share of the Time bar under the same statistic.

Sandbox and Trials

Reference

Each task's own image, pinned by digest and built from the task's environment/ at the pinned commit for linux/amd64 with a thin agent layer on top (for example ghcr.io/sandbox-test-images/base-image@sha256 for the first task).

Lifetime

30 min

Idle Connection Time

5.0 min

From Common Methodology

Idle Connection Exceptions

Provider	Rule	File Transfer Idle Time
Daytona	The SDK has no idle-timeout setting. Its command channel keeps its connection open, and its file-transfer channel reconnects after 5 s of idle time.	5.0 s

From Common Methodology

Network Path Attempts

10

From Common Methodology

Mode

Single

Sandboxes per Trial

1

Parallel Trials

10

Concurrent Sandboxes

1

At a Glance

Quantity

Command and file-edit runtime

Per

Session

Unit

Milliseconds (ms)

Direction

Lower Is Better

Statistic

Median

Clock

Sandbox clock

From Common Methodology

Machine Size

2 vCPU / 4 GiB

Trials

20

Variants

• Claude Opus 5.5 max