Local stack

The local stack is a Docker Compose project, deploy/compose/docker-compose.yaml, that runs the LiteLLM proxy and its Postgres database. Every model call of every run goes through it, so it has to be up before a run starts. ssebench run starts it when it is not running, and just launch starts it on its own.

The stack holds only the proxy. The run containers are started by docker run, not by Compose, and the web UI and the catalog service run separately, except in the demo, which adds them.

                 host                                      Compose project
 ssebench CLI --- http://localhost:$LITELLM_PORT ------>  litellm  (port 4000)
                                                          litellm_db  (Postgres)
 run containers --- http://litellm:4000 (<project>_agents network)

Services

Service	Image	Role
litellm	$SSEBENCH_REGISTRY/litellm:<version>, built from images/litellm/	The proxy. It publishes container port 4000 on the host at LITELLM_BIND:LITELLM_PORT, and is healthy once /health/liveliness answers
litellm_db	postgres:16	LiteLLM's database: the per-run keys, their budgets and spend. It has no published port, and restarts with the Docker daemon

The proxy image contains the model list from models/*.yaml, so a change to the models takes effect once the image is rebuilt; ssebench proxy up and ssebench run do that when the files changed. Provider keys are not in the image: the proxy gets them from .env when its container starts.

docker compose names things after the project, ssebench by default:

What	Name
Containers	<project>-litellm-1, <project>-litellm_db-1
Networks	<project>_default, and the internal <project>_agents
Volume	<project>_postgres_data, mounted at /var/lib/postgresql/data

<project>_default is an ordinary bridge, through which the proxy reaches the model providers. <project>_agents is internal, so it has no route out of the host: run containers join it by default and reach the proxy and nothing else. Integrity and egress explains the two networks and the --egress option that chooses between them.

WARNING

The proxy holds the master key and the provider keys, so its port is published on 127.0.0.1 only. Set LITELLM_BIND=0.0.0.0 to publish it on every interface, and then the master key is the only thing that guards it: firewall LITELLM_PORT on a host that other machines can reach. Run containers reach the proxy over the Docker network, not through this port.

Settings

Compose reads the stack's settings from the environment and from .env in the workspace, which is the repository root in a checkout. just setup writes .env from .env.example, with a generated master key and database password, and leaves an existing file alone. From a package install, ssebench init does the same in the current directory:

just setup          # in a checkout
uvx ssebench init   # without a clone

Setting	Default	Meaning
LITELLM_MASTER_KEY	generated by just setup or ssebench init	The proxy's admin key. The CLI uses it to create a key for each run. Required
POSTGRES_PASSWORD	generated by just setup or ssebench init	The database password: letters and digits only, since it becomes part of a URL. Required
POSTGRES_USER	litellm	The database user
ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_API_KEY	unset	Provider keys that the models in models/ refer to. The proxy reads them from .env only, when its container starts
LITELLM_PORT	4000	The host port of the proxy
LITELLM_BIND	127.0.0.1	The host address the proxy's port is published on
COMPOSE_PROJECT_NAME	ssebench	The project, which names the containers, networks and volume
SSEBENCH_REGISTRY	ghcr.io/42-b3yond-6ug/ssebench	The registry prefix of the proxy image
SSEBENCH_ENV_FILE	.env in the workspace	The file the proxy container reads the provider keys from. The CLI sets it; set it yourself only when you run docker compose directly

Compose refuses to start the stack without the master key and the password, and says run just setup to write .env. See Environment variables for the settings and Configuration files for .env.

WARNING

Postgres applies POSTGRES_PASSWORD only when it creates the volume. If you change the password later, the database keeps the old one and the proxy cannot log in; remove the volume with the stack, as below.

Running the stack

Command	Does
just launch, uv run ssebench proxy up	Builds the proxy image if it is missing or out of date, starts the stack, and waits up to three minutes for the proxy to answer
uv run ssebench proxy build	Builds the image, if it is missing or out of date, and does not start anything
uv run ssebench proxy up --rebuild	Builds the image even when it is current
just stop, uv run ssebench proxy down	Removes the containers and networks and keeps the database volume

The CLI passes its settings to Compose, so COMPOSE_PROJECT_NAME, LITELLM_PORT and SSEBENCH_REGISTRY from the environment or from .env decide which stack these commands act on. ssebench proxy is described in CLI.

To check that the proxy answers, ask for its health, or run ssebench doctor, which also checks Docker, the disk, .env and which provider keys are set:

curl http://localhost:4000/health/liveliness
uv run ssebench doctor

The proxy's logs are those of its container:

docker logs ssebench-litellm-1

Changing something

You changed	Do
models/*.yaml	just launch, which rebuilds the image and recreates the proxy
A provider key in .env	just launch, so that the proxy restarts with it
LITELLM_PORT	just launch, which recreates the proxy container with the new port
The SSEBench version, by pulling a new checkout	just launch. The image tag carries the version, so the new proxy image is built

Stopping while runs are kept

proxy down cannot remove <project>_agents while a container is still attached to it, and prints Resource is still in use for the network. That happens when you stop the stack while --keep-container runs are still around. Those containers have lost their proxy. Once they are gone, the next down removes the network.

Two stacks side by side

The names above all carry the project name, so a second stack needs only its own project name and its own port:

COMPOSE_PROJECT_NAME=exp-a LITELLM_PORT=4001 uv run ssebench run ...
COMPOSE_PROJECT_NAME=exp-b LITELLM_PORT=4002 uv run ssebench run ...

ssebench run starts each stack when it is not running. The two stacks have their own containers, their own database volume, and networks that keep the run containers of one from reaching those of the other. A key and its spend exist in one stack's database only. The stacks share the image, since its tag is the same, and .env, so both proxies get the same provider keys and master key unless you override them in the environment.

Set the same two variables in the environment of the web UI if it should launch runs on a particular stack. The server passes its environment on to the runs it starts. The web UI lists the containers of every stack, though.

Results go to results/ in the working directory, so start the experiments from different working directories if they might run the same task, model and agent. See also Running experiments side by side.

Removing a stack

proxy down keeps the database, so the run keys and spend survive it. To delete a stack and its data, stop it and remove its volume:

COMPOSE_PROJECT_NAME=exp-a LITELLM_PORT=4001 uv run ssebench proxy down
docker volume rm exp-a_postgres_data

Removing the volume erases the stack's keys and spend history, and it is the way to start over after changing POSTGRES_PASSWORD.

The demo stack

deploy/compose/demo.yaml adds two services to the stack for the demo, and only adds: the proxy and its database stay as described above.

Service	Image	Role
catalog	$SSEBENCH_REGISTRY/catalog:<version>	Serves the pilot manifest on 127.0.0.1:$SSEBENCH_DEMO_CATALOG_PORT (8090)
webui	$SSEBENCH_REGISTRY/webui:<version>	The web UI on 127.0.0.1:$SSEBENCH_DEMO_WEBUI_PORT (3001), with its terminal off. It shares the host's network, to reach each run container by its IP address, and mounts the Docker socket

just demo starts the layered stack as the project ssebench-demo (see SSEBENCH_DEMO_PROJECT), and just demo-down removes it with its volume. Both services carry the label ssebench.demo, which the demo uses to tell that a project is its own. To start it by hand, list both files:

set -a; . ./.env; set +a
docker compose --project-name ssebench-demo \
  --file deploy/compose/docker-compose.yaml --file deploy/compose/demo.yaml \
  up --detach

Without the CLI

ssebench proxy runs docker compose with the project name and the file. If you run Compose yourself in a checkout, do the same and export the settings, since the file needs LITELLM_MASTER_KEY and POSTGRES_PASSWORD even for down:

set -a; . ./.env; set +a
docker compose --project-name ssebench --file deploy/compose/docker-compose.yaml up --detach

This starts whatever proxy image is present and does not check the model list, so prefer ssebench proxy up.

Kubernetes

The stack is for one host. The Kubernetes deployment is not available yet.

Next steps

• LiteLLM proxy: keys, budgets and the model list
• Integrity and egress: the networks and what a run container can reach
• Add a model
• Environment variables