CosmicAC Logo

Deploy CosmicAC

Deploy the CosmicAC Docker Compose stack on your host machine.

Deploy the CosmicAC stack on your host machine with Docker Compose. For how CosmicAC connects to your cluster and runs jobs, see Deployment architecture.

Stack

The stack runs seven services, plus Redis and Caddy:

The CosmicAC services use private images published under ghcr.io/tetherto/, in the form ghcr.io/tetherto/<repo>:<tag>. Redis and Caddy use their official public images.

Prerequisites

The recommended host is Ubuntu 22.04 or 24.04 on x86_64. On that host, you need the following:

  • Docker Engine and Docker Compose v2.
  • Task.
  • The jq, node, and kubectl command-line tools.
  • Access to the private CosmicAC deployment repository, which holds the Compose files and deploy scripts. Request access from the CosmicAC team.
  • GitHub Container Registry (GHCR) credentials for private images.
    • Your GitHub username.
    • A classic GitHub personal access token with the read:packages scope so it can pull the private ghcr.io/tetherto images. If the tetherto org enforces single sign-on, approve the token for the org. See Managing your personal access tokens.
  • A GPU Kubernetes cluster that already meets the Requirements. CosmicAC connects to this cluster but doesn't set it up.
  • A valid, readable kubeconfig for the GPU Kubernetes cluster.

Steps

Verify the prerequisites

Enable Docker, then confirm you have everything listed in Prerequisites:

sudo systemctl enable --now docker
docker compose version
task --version
node --version
kubectl version --client

Set up the environment

Clone the deployment repository and change into it:

git clone <deployment-repo-url>
cd <deployment-repo>

Create the .env file from the example:

cp .env.example .env

In .env, set the variables your deployment needs. See Required before the first deployment for the ones you must set, and the deployment configuration for the rest.

If GITHUB_PAT and GITHUB_USER aren't set in .env, bootstrap prompts for them so it can log in to GHCR and pull the private images.

Add the kubeconfig

Get the kubeconfig from your cluster administrator. It must meet the Kubeconfig requirements.

Open a file on the host, paste the kubeconfig, and save it:

nano ~/kubeconfig

Print the file's absolute path:

realpath ~/kubeconfig

Set KUBECONFIG_SRC in .env to that path:

KUBECONFIG_SRC=/home/<user>/kubeconfig

Confirm the kubeconfig is readable and reaches your cluster:

test -s ~/kubeconfig && echo ok
kubectl --kubeconfig ~/kubeconfig config current-context
kubectl --kubeconfig ~/kubeconfig cluster-info

Run the first-time bootstrap

task bootstrap deploys the whole stack with one command, using TAG from .env. For what it runs and the other deployment commands, see Task deployment commands.

Print the deployment directory and copy the path:

pwd

Open a root shell:

sudo -i

A root shell starts in root's home directory, so return to the copied path:

cd <copied-path>

Run the bootstrap:

task bootstrap

Verify the deployment

Check the services and the API:

task ps
curl -s4 -o /dev/null -w '%{http_code}\n' http://127.0.0.1:5173/
curl -s4 "http://127.0.0.1:5173/api/auth/servers?overwrite_cache=true"
curl -s4 "http://127.0.0.1:5173/api/auth/pricing?location=IN&type=gpu&hrs=2"
curl -s4 "http://127.0.0.1:5173/api/auth/jobs?page=1&pageSize=10"

The calls carry no token because a default deployment runs with authentication turned off.

The deployment is healthy when task ps shows every service Up and http://127.0.0.1:5173/ returns 200. The servers call lists your GPUs, and the pricing call returns a total cost rather than an error.

Open the web interface

Open http://<server-ip>:5173 in your browser. If you changed UI_PORT, use that port instead.

A default deployment runs with authentication turned off, so anyone who reaches this port has full access without signing in. Restrict network access to the deployment. See Deployment configuration for the authentication defaults.

Help and troubleshooting

Bootstrap fails with rm: cannot remove ...: Permission denied

Bootstrap resets the HyperMQ stores under ./services/*/store before it deploys. The stack's containers create those files as root, so a non-root user can't delete them.

Print the deployment directory and copy the path:

pwd

Open a root shell:

sudo -i

Return to the copied path:

cd <copied-path>

Rerun the bootstrap:

task bootstrap
Bootstrap fails with error from registry: denied

Error response from daemon: error from registry: denied

[ERROR] Required image is unavailable: ghcr.io/tetherto/cosmicac-wrk-ork:release-1.1.0

Docker stores GHCR credentials separately for each user, in that user's ~/.docker/config.json. A user who hasn't logged in to GHCR can't pull the private ghcr.io/tetherto images, even when another user on the same host can.

Print the deployment directory and copy the path:

pwd

Open a root shell:

sudo -i

Return to the copied path:

cd <copied-path>

Rerun the bootstrap:

task bootstrap

Next steps

With CosmicAC running, create your first job or learn how to operate the deployment:

On this page