Installation

SSEBench runs from a clone of its repository, or, for the ssebench command alone, from the package on PyPI. This page sets up the tools it needs; the Quickstart then runs your first task.

Without a clone

With Docker and uv installed, uvx ssebench init in an empty directory sets up a workspace, and uvx ssebench run --task <id> --agent reference runs a pilot task. The package carries what the CLI needs, and the task images are pulled from the registry. Skip to Without a clone.

Prerequisites

Tool	Needed for
Docker Engine, with the buildx and Compose plugins	Building task images, running every task, and the LiteLLM proxy, which is a Compose stack
uv	Running the ssebench CLI; it installs Python for you
just	The recipes in the Justfile; only in a clone
git	Cloning the repository
make (optional)	just base-images, and just demo when the registry does not have the base image of the task, as before a release; the image is then built from the checkout
fzf (optional)	The interactive task, model and agent pickers in just recipes
jq and Typst (optional)	Reading results and building a report from them
Bun (optional)	The web UI and these docs; just demo does not need it
Rust and Go (optional)	just test and just lint for the daemon and the Go components; rust-toolchain.toml pins the Rust version

You also need network access: images are pulled from the registry, and building one downloads packages.

Platform. SSEBench is developed and tested on Linux. An x86-64 host is recommended: every pilot task builds an amd64 image, and many C tasks compile with AddressSanitizer for x86-64 only. On another architecture Docker must run them under emulation; see Architectures. The demo needs a Linux Docker engine.

Disk space. Images are large, and they live where Docker keeps its data (/var/lib/docker by default). Docker Engine 29 stores images with containerd by default, and docker images reports them about 45% larger than the classic overlay2 store of older installs does. Sizes as docker images reports them:

Image	Docker 29, containerd store	Classic overlay2 store
Base images generic-c, generic-go, generic-rust	1.6 GB, 1.0 GB and 2.3 GB	1.1 GB, 0.7 GB and 1.6 GB
LiteLLM proxy	1.7 GB	1.2 GB
Postgres, the proxy's database	0.6 GB	0.5 GB
Web UI and task catalog, for the demo	0.3 GB and 25 MB	0.2 GB and 17 MB
A task's case image	about 45% more than in the next column	0.7 to 3.4 GB, half of them under 1.3 GB
A task's tool and agent layers, on top of its case image	1.5 GB and 1.7 GB in total for gjson-196-bf4efcb, case image included	about 0.4 to 2 GB, depending on the task and the agent

Layers are shared between images, so one task needs a few GB, and the demo about 4 GB of images (docker system df counted 4.4 GB after it on Docker 29). Leave at least 10 GB free to try SSEBench (ssebench doctor fails below that, and warns below 50 GB), and expect all 55 pilot tasks, built for one agent, to need roughly 100 GB. Downloads are smaller than these sizes, because images are compressed on the wire.

Build cache. Building an image also fills Docker's build cache. A demo built from a checkout, which is what happens before a release, leaves about 11 GB of it on top of the 4 GB of images, so plan for 15 GB free; building the three base images alone leaves about 5.6 GB. The cache only speeds up later builds. To get the space back without touching images, containers or volumes:

docker system df          # the Build Cache row shows what it holds
docker builder prune --all

The next build then starts cold and takes longer.

Install the tools

Ubuntu/Debian

# Docker Engine with the buildx and Compose plugins, from Docker's apt repository:
# https://docs.docker.com/engine/install/ubuntu/ (Debian: .../debian/). Install
#   docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
# Not Ubuntu's `docker.io` package: it has neither plugin.
# Let your user run docker: sudo usermod -aG docker "$USER", then log in again

# uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# just
curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to ~/.local/bin
# Open a new shell afterwards, so that ~/.local/bin is on your PATH

# Optional tools
sudo apt-get install -y fzf jq make

macOS

# Docker: Docker Desktop, or another engine such as colima

brew install uv just

# Optional tools
brew install fzf jq typst

Arch Linux

sudo pacman -S docker docker-buildx docker-compose uv just
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"   # then log in again

# Optional tools
sudo pacman -S fzf jq make typst

Check that Docker works with docker info, docker buildx version and docker compose version; ssebench doctor runs these checks for you once you have the code. It reports fail buildx or fail Compose when a plugin is missing, as with Ubuntu's docker.io package.

With Nix

Nix is optional. The flake gives you every toolchain the repository uses (Python, uv, Rust, Go, Bun, just, Typst, the Docker CLI with buildx and Compose, fzf and jq) in one shell, without installing them one by one:

git clone https://github.com/42-b3yond-6ug/ssebench.git
cd ssebench
nix develop

The Docker engine still comes from your system. On NixOS, enable it in your configuration:

virtualisation.docker.enable = true;
users.users.<you>.extraGroups = [ "docker" ];

The shell's uv uses its Python and never downloads one, which is what you want on NixOS. Outside the shell on NixOS, the prebuilt tools that uv installs (ruff and the Node.js that basedpyright needs) run only with nix-ld. nix flake check runs the linters and tests in the Nix sandbox; see CONTRIBUTING.md.

Without Nix

Install the tools in the table above with your package manager, as shown in Install the tools. just setup then installs the Python dependencies with uv, which downloads a matching Python if you have none, and the web UI and docs dependencies when Bun is installed.

Get the code

git clone https://github.com/42-b3yond-6ug/ssebench.git
cd ssebench

Run every command in these docs from the repository root unless it says otherwise.

Set up

just setup

This installs the Python dependencies (and, if Bun is installed, those of the web UI and the docs), then writes .env from .env.example. .env holds a generated master key for the LiteLLM proxy and a generated password for its database; just setup leaves an existing .env unchanged.

Then add the key of each model provider you want to use to .env:

ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
GOOGLE_API_KEY=...

Leave the others empty. The dummy and reference agents make no model calls and need no key.

WARNING

Never commit .env. It is listed in .gitignore.

Check your setup

just doctor

just doctor (uv run ssebench doctor) checks Docker, buildx and Compose, free disk space, the CPU architecture, .env, the LiteLLM proxy and the provider keys, and prints a fix for each problem. It exits non-zero when a required check fails. Troubleshooting explains each check. Run just to list the other recipes.

Without a clone

The ssebench CLI is on PyPI and runs from the package alone. You need Docker with buildx and Compose, and uv; you do not need just, git or a checkout.

mkdir ssebench-work && cd ssebench-work
uvx ssebench init
uvx ssebench doctor

uvx runs the command in a temporary environment. Install the command with uv tool install ssebench (or pip install ssebench) to keep it on your PATH as ssebench. The package carries the agent definitions, the model list, the Compose file of the LiteLLM proxy and the pilot task list; the task images come from the registry, and the layers on top of them are built on your machine. ssebench init writes .env, models/ and results/ into the current directory, which is your workspace: run ssebench from there. See Without a clone for what the package carries and what it pulls.

Architectures

SSEBench's own images (runtime, litellm, catalog and webui) and its tool and agent layers build for linux/amd64 and linux/arm64, and the published images cover both. The case images do not: a task lists the platforms it supports in the arch field of its manifest, and every pilot task is amd64 only, because many C tasks build with AddressSanitizer for x86-64.

Host	pilot tasks
x86-64 (amd64)	Run natively. This is the recommended host.
arm64 (Apple silicon, AWS Graviton and similar)	Run under amd64 emulation.

A run uses one platform, chosen from the task: the host's own architecture if the task's arch lists it, otherwise the first architecture that it lists. The tool layer is added on top of the case image, and the agent on top of the tool layer, so all of them must have the case image's architecture. ssebench run therefore builds the case, tool and agent images with --platform and starts the container with it. On an arm64 host with an amd64-only task, that puts the whole task container under emulation: the project build, its tests, the grader and the agent. The LiteLLM proxy and the web UI keep running natively.

Emulation is slow, and AddressSanitizer may misbehave under QEMU, so a task that passes on an x86-64 host can fail or time out on an arm64 one. ssebench doctor warns when the host is not amd64, and ssebench run prints the same warning once at the start of a run.

Docker needs an amd64 emulator on such a host. Docker Desktop includes one. On Linux, install QEMU's handlers once, or use your distribution's qemu-user-static package:

docker run --privileged --rm tonistiigi/binfmt --install amd64
docker run --rm --platform linux/amd64 alpine uname -m   # prints x86_64

The base images are pulled for the platform of the run. If you build them from your checkout instead, build the amd64 variant, because the case images build on it:

make -C images/base-images all BUILD_FLAGS="--platform linux/amd64"

Next steps

• Try the demo: a graded run in the web UI, with no key
• Quickstart: run an agent on a pilot task
• Troubleshooting: when ssebench doctor fails
• Add a model: use a provider or model that isn't listed in models/