CI Actions

Builds on Common Methodology

Times each job as the sum of its step runtimes on the sandbox's clock; All Jobs is the geometric mean of the selected jobs.

CI Actions measures how fast a sandbox executes an end-to-end continuous integration pipeline from a cold start, the batch workload engineering teams offload from hosted CI runners and the verification sequence coding agents run after modifying a repository. We replay six real GitHub Actions jobs from open-source repositories, one per language across Node.js, Python, Go, Rust, C, and Java.

Every trial downloads its repository checkout, compiler toolchain, system packages, and language dependencies from scratch over the provider's network, so the result captures both the provider's network path to public package registries and its CPU and disk speed during linting, compilation, and testing.

View the Procedure View CI Actions Results Download Raw Data View the Parameters

Metrics

CI Actions is the sum of step runtimes across a job. Each step is assigned by the parameter file to one of four stages: Install (git clone, git checkout, apt-get, toolchains, and pnpm install, uv sync, go mod download, cargo fetch, autoreconf/./configure, or Maven) and Lint, Test, or Build (pnpm test, pytest, go test, cargo test, make check, or mvn test).

Timed Part	Starts	Ends
Job step	The step starts, by the sandbox's own clock.	The step exits, by the sandbox's own clock; its runtime is the difference between two consecutive readings.

T = \sum_{k=1}^{K} \left( e_k - s_k \right)

• T Tool time of a replay, or job time of a CI job: the sum of its command or step runtimes on the sandbox's clock.
• K Number of commands or steps in the trace, session or job.
• s_k The script's $EPOCHREALTIME reading immediately before command or step k.
• e_k The script's $EPOCHREALTIME reading immediately after command or step k.

Measurement

Each trial uses a sandbox created from BoxRouter's pinned standard image (box user with passwordless sudo, matching GitHub's ubuntu-latest runner conventions) with open networking and no cached toolchains, packages, builds, or Git state.

1. Create sandbox.
2. Run readiness check.
3. Time per Job Execute all workflow steps across four stages (Install, Lint, Test, Build). Timed The job's steps are executed sequentially inside a single background script (polled every ten seconds) under bash as the non-root box user, with each step's start and stop timestamps recorded via $EPOCHREALTIME and validated against /proc/uptime.
4. Destroy sandbox.

The workload translates 6 .github/workflows jobs from public open-source repositories into self-contained bash step sequences pinned to a repository commit. GitHub Actions workflow expressions, matrix variables, actions/checkout, and actions/setup-* steps become explicit shell commands with caches disabled so each trial clones from GitHub and fetches dependencies from live upstream registries.

Job	Repository	Commit	Steps	Toolchain
Node (zustand)	pmndrs/zustand	d7a5583c	9	Node.js, pnpm, TypeScript typecheck, Vitest, and build
Python (fastapi)	fastapi/fastapi	69ddc2ba	5	Python 3.13, uv, and pytest with coverage
Go (gin)	gin-gonic/gin	43fe48e8	4	Go 1.27 module download, vet, and race-detector test suite
Rust (spotify-player)	aome510/spotify-player	7dc9d17a	7	system audio headers, Rust cargo clippy, cargo fmt, and cargo build/test
C (jq)	jqlang/jq	49fbca12	5	autoreconf, ./configure, make, and make check
Java (gson)	google/gson	881d2b55	3	JDK 21 and mvn test --batch-mode

Scope Limits and Caveats

• Every trial starts cold without dependency caches or pre-baked SDK images, and live downloads from GitHub, npm, PyPI, crates.io, and Maven Central include upstream registry variance.
• Teams that keep warm dependency caches on network volumes, snapshots or prebuilt custom images wait much less on Install than the cold-start times measured here.
• Downloading Go, rustup, uv, Temurin JDK, Maven and apt build packages that GitHub's hosted ubuntu-latest image pre-installs makes Install longer here than on a hosted GitHub runner.
• The suite covers one representative workflow per language on Linux x86-64 (linux/amd64) only.
• The gin test suite binds port 8080, which collides with the port Blaxel reserves for its internal sandbox API.
• Blaxel stores the sandbox filesystem entirely in RAM, and BoxRouter does not yet mount external volumes on Blaxel, which limits disk-heavy Rust builds such as spotify-player.

Failures and Exclusions

Outcome classifications, top-up rounds, and outlier rules follow Common Methodology; the first failed step ends the trial without in-trial retries.

Every step in the job must exit 0 and pass the twenty-millisecond /proc/uptime clock gate.

Because steps fetch from live external registries (github.com, registry.npmjs.org, pypi.org, proxy.golang.org, crates.io, repo.maven.apache.org) and run without a pre-recorded trace, a step that exits non-zero or times out is classified as excluded (does not replay here) when caused by an upstream registry outage or deterministic environment incompatibility, and as a provider error when the sandbox fails to create, loses network connectivity, or stops answering.

Step	Time Limit
Create	5 min
Job step	10 min
Destroy	5 min

Statistics

Based On

Common Methodology

Per-unit percentiles, bootstrap confidence intervals, and tail rules follow Common Methodology.

The All view aggregates each unit's selected statistic across the workload using the unweighted geometric mean and missing-unit substitution penalty defined in Common Methodology. A provider is ranked once it has at least 90% of the successes requested before any top-up: 108 across all units. It also needs a provider-sourced cost. One job on its own needs 90% of its 20 trials.

Cost

Aggregates a cost statistic over each job, a replay of one repository's CI steps from a cold start (selected from open-source repositories' GitHub Actions workflows).

Only successful trials contribute to the cost value. Each stage's cost is the shown cost times that stage's share of the Time bar under the same statistic.

Sandbox and Trials

Reference

A prepared Linux image for the selected job.

Lifetime

15.0 min

Idle Connection Time

5.0 min

From Common Methodology

Idle Connection Exceptions

Provider	Rule	File Transfer Idle Time
Daytona	The SDK has no idle-timeout setting. Its command channel keeps its connection open, and its file-transfer channel reconnects after 5 s of idle time.	5.0 s

From Common Methodology

Network Path Attempts

10

From Common Methodology

Concurrent Sandboxes

1

Parallel Trials

10

Mode

Single

Sandboxes per Trial

1

At a Glance

Quantity

Job-step runtime

Per

Job

Unit

Milliseconds (ms)

Direction

Lower Is Better

Statistic

Median

Clock

Sandbox clock

From Common Methodology

Machine Size

4 vCPU / 8 GiB

Trials

20

Variants

• All Jobs
• Node (zustand)
• Python (fastapi)
• Go (gin)
• Rust (spotify-player)
• C (jq)
• Java (gson)