Developer guide

This guide is for people changing Armada itself. It builds every Armada component from source, runs them as programs on your machine, and connects them to a small Kubernetes cluster, so you can change code, restart one component and test the result in seconds.

If you only want to use Armada, follow Getting started instead.

For the full local-development reference, including profiles, service ports, tests and editor configurations, see docs/developer_guide.md. This published entrypoint is maintained in website/content/docs/developer-guide.mdx; follow website/README.md to edit and check website content.

Time needed: about 30 minutes the first time, most of it downloading and compiling.

How the setup works

• Dependencies (PostgreSQL, Redis, Pulsar, Jaeger, an OpenTelemetry collector, and Keycloak when you test logins) run as Docker containers.
• Armada's components (server, scheduler, executor, ingesters, Lookout, Binoculars and the Lookout web UI) run as programs on your machine, started together by Goreman from a Procfile in _local/procfiles/.
• Jobs run in a Kind cluster called armada-test, or, with the fake executor, nowhere at all.
• Mage is the build tool that ties it together. It sets up, starts and stops the stack.

Before you start

Install the tools

Mage doesn't need installing: the commands below run a pinned version with go run.

Give Docker enough memory

The dependencies and the three-node armada-test cluster need several GB. If you also keep a laptop install running, it needs about 6 GB more. This guide was tested with Docker Desktop set to 16 GB (Settings → Resources → Memory) with a laptop install alongside.

Changing Docker's memory restarts Docker, which breaks a running laptop install; reinstall it afterwards (see Troubleshooting).

Ready when:

go version
docker info > /dev/null && echo "Docker is running"
yarn --version

shows Go 1.27.1 or later (or Go 1.21 or later, if you're online), Docker is running, and a Yarn version.

Step 1. Get the code

git clone https://github.com/armadaproject/armada.git
cd armada

Run every command in this guide from this folder.

Step 2. Set up Mage and the build tools

Make mage run a pinned, known-good version, and put Go's tool folder first on your PATH (if you set GOBIN, use that folder instead). These last until you close the terminal, so run them again in each new terminal:

alias mage='go run github.com/magefile/mage@v1.17.2'
export PATH="$(go env GOPATH)/bin:$PATH"

Then install the build tools (kind, sqlc, golangci-lint, goreleaser and others, listed in tools.yaml) into Go's tool folder:

mage BootstrapTools

This took about 2 minutes.

Success:

mage checkDeps

shows PASSED for docker, go, mage, kind, kubectl, protoc, sqlc and docker compose. On Docker Desktop, docker buildx fails with found docker-buildx version 0.33.0-desktop.1 but it failed constraint >=0.6.3, so the command ends with Error: check(s) failed.; that's a problem with the check, not your setup, and you can ignore it.

The PATH line matters: without it, Mage uses whatever kind you installed yourself, and older versions fail in the next step with ERROR: unknown containerd config version: 4 (supported versions: 2 and 3).

Step 3. Create the Kubernetes cluster

You need this once. It creates a three-node Kind cluster called armada-test, separate from a laptop install's armada cluster.

On Docker Desktop, first download every platform of the images Mage loads into the cluster. Recent Docker versions keep only your own platform, and loading then fails with ctr: content digest sha256:...: not found:

for image in nginx:1.27.0 alpine:3.20.0 bitnamilegacy/kubectl:1.33.4 \
    registry.k8s.io/ingress-nginx/controller:v1.4.0 \
    registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343; do
  for platform in $(docker buildx imagetools inspect "$image" --raw \
      | python3 -c 'import json,sys; print(" ".join(m["platform"]["os"]+"/"+m["platform"]["architecture"]+("/"+m["platform"]["variant"] if m["platform"].get("variant") else "") for m in json.load(sys.stdin)["manifests"] if m["platform"]["os"]!="unknown"))'); do
    docker pull -q --platform "$platform" "$image" > /dev/null
  done
done

If a single pull fails with a network error, run the loop again. Then:

mage kind
export KUBECONFIG=$PWD/.kube/external/config

Run the export line again in each new terminal.

Success: mage kind ends with Time to setup kind: ... (about a minute), and

kubectl get nodes

lists armada-test-control-plane, armada-test-worker and armada-test-worker2, all Ready.

Step 4. Start Armada

mage dev:up no-auth

This starts the dependencies in Docker, creates the databases and Kubernetes priority classes, then compiles and starts every component. It keeps running in this terminal and prints every component's log, each line prefixed with its name. Leave it running and open a second terminal for the next steps (run the Step 2 and Step 3 alias and export lines there too).

The first start compiles everything and takes a few minutes. Later starts take seconds.

Success: a scheduler line containing Retrieved 1 executors (about a minute after starting, on a clean start). Before that line appears, Armada isn't ready: jobs submitted too early are rejected with Number of nodes in cluster: 0.

no-auth is the profile. The argument is required: plain mage dev:up fails with not enough arguments for target "Dev:Up", expected 1, got 0. Other profiles are in Other ways to run Armada.

Step 5. Check every component is running

In your second terminal:

./bin/goreman run status

Mage installs Goreman into the repository's bin folder, so use ./bin/goreman (plain goreman gives command not found unless you've installed it yourself).

Success: every line starts with *:

*server
*scheduler
*scheduleringester
*eventingester
*executor
*lookout
*lookoutingester
*binoculars
*lookoutui

Step 6. Run a job

Use armadactl from your source tree, with the repository's own config file. Without --config, armadactl uses ~/.armadactl.yaml, which points at a laptop install if you have one.

go build -o ./bin/armadactl ./cmd/armadactl
./bin/armadactl --config _local/.armadactl.yaml create queue dev-test

Wait 15 seconds (Armada checks for new queues every 10 seconds), then submit a job:

cat > /tmp/hello.yaml <<'EOF'
queue: dev-test
jobSetId: hello
jobs:
  - namespace: default
    priority: 0
    podSpec:
      terminationGracePeriodSeconds: 0
      restartPolicy: Never
      containers:
        - name: hello
          image: alpine:3.21
          command: ['sh', '-c', 'echo hello']
          resources:
            requests: { cpu: 100m, memory: 64Mi }
            limits: { cpu: 100m, memory: 64Mi }
EOF
./bin/armadactl --config _local/.armadactl.yaml submit /tmp/hello.yaml
./bin/armadactl --config _local/.armadactl.yaml watch dev-test hello --exit-if-inactive

Success: the watch ends with JobSucceededEvent, about 15 seconds after submitting.

The Lookout web UI from your source tree is at http://localhost:3000, and shows the job.

Working on a component

After changing code, rebuild and restart just that component:

./bin/goreman run restart server

Use the names from goreman run status. Goreman rebuilds the component from source before starting it. (Plain goreman restart server, without run, only prints Goreman's help.)

To stop everything: press Ctrl-C in the terminal running mage dev:up, then remove the dependency containers:

mage dev:down

This also wipes the dev databases, so your next start is clean. The armada-test cluster keeps running; remove it with mage kindTeardown when you no longer need it.

Other ways to run Armada

mage dev:up takes one of these profiles. To also start an extra Docker Compose profile, add it after a comma, for example mage dev:up auth,prometheus.

Run mage dev:down before switching profiles. Otherwise the scheduler still remembers the executor from your last run. For up to 10 minutes (scheduling.executorTimeout) it can hand jobs to that executor, which no longer exists, and those jobs stay in Leased until the scheduler gives up on them, which can take up to an hour (executorTimeout).

Test with logins

mage dev:up auth

This also starts Keycloak, with a test realm, and waits for it before starting Armada. The server accepts Keycloak (OpenID) logins and a username and password. Without credentials, requests fail with Request could not be authenticated with any of the supported schemes.

_local/.armadactl.yaml has a context for each login method. Pick one with --context:

./bin/armadactl --config _local/.armadactl.yaml --context auth-password get queues

Success: no rpc error line. The admin user is in the admins group, which can submit, cancel, reprioritize, preempt and watch any job, create and delete queues, and manage retry policies. It can't cordon queues or nodes (see _local/server/config-auth.yaml).

The Keycloak admin console is at http://localhost:8180, with user admin and password admin (set in _local/compose/stack.yaml). The realm has only the admin user; add others there to test permissions. One limitation: with the realm as shipped, tokens have no sub claim, so Armada sees every Keycloak user with an empty name. Permissions granted to groups work; permissions for individual users and queue owners don't.

To run the steps by hand instead (for example to restart only Armada and keep Keycloak):

docker compose -f _local/compose/stack.yaml --profile auth up -d
_local/scripts/init.sh
./bin/goreman -f _local/procfiles/auth.Procfile start

Wait until http://localhost:8180/realms/armada answers before the last command, or components can fail to start.

Run the tests

Run the end-to-end test cases against your running Armada. Like armadactl, the test runner reads ~/.armadactl.yaml unless you pass --config:

export ARMADA_EXECUTOR_INGRESS_URL="http://localhost"
export ARMADA_EXECUTOR_INGRESS_PORT=5001
go run cmd/armadactl/main.go --config _local/.armadactl.yaml create queue e2e-test-queue
go run cmd/testsuite/main.go test --config _local/.armadactl.yaml \
  --tests "testsuite/testcases/basic/*" --junit junit.xml

Success: after about 2 minutes:

Ran 17 test(s) in 1m48s
Success: 17
Failure: 0

mage testsuite runs the test groups CI runs (basic, categorization, retry, preemption, reprioritization, queue and two node tests), but only against the containerised stack from mage dev:full. Against mage dev:up, it waits a minute and fails with Error: timed out waiting for scheduler to start.

Run the whole stack in containers

mage dev:full builds container images of every component and runs the whole stack in Docker against the Kind cluster, the same way CI does. Use it to check a change the way CI will, and to run mage testsuite. For day-to-day work, mage dev:up is faster.

Stop mage dev:up and run mage dev:down first, because both start the same services.

On Docker Desktop, the image build needs Docker's default context. Set it for these commands only, rather than switching your whole machine:

export DOCKER_CONTEXT=default
mage dev:full

Without it, the build fails with docker buildx is not set to default context - please switch with 'docker context use default'. mage dev:full also runs mage kind, so do the image download loop from Step 3 first if you haven't.

The first run compiles for about 5 minutes before building the images; later runs took about 1.5 minutes.

Success: it ends with each component's container Started and the migration containers Exited.

Then run the same tests as CI:

mage testsuite

Success: after about 5.5 minutes, each group reports Failure: 0, and the last line is (Real) Total time to run all tests: .... (Tested: 34 tests in 8 groups, all passed.)

To stop it, and delete the armada-test cluster:

mage dev:fullDown

Profile a component with pprof

Add a profiling block to the component's config file, then restart it. The file is the one after --config on the component's line in _local/procfiles/<profile>.Procfile: with auth, for example, the server reads config-auth.yaml, but the ingesters still read config.yaml. The block needs an auth section; with only a port, the component logs Pprof server auth not configured, will not set up pprof and doesn't open the port.

This example has no login, and the port listens on every network interface, not just localhost. Anyone who can reach it can read the profiles, so use it only on a network you trust, and remove the block when you're done.

profiling:
  port: 6060
  auth:
    anonymousAuth: true
    permissionGroupMapping:
      pprof: ['everyone']

./bin/goreman run restart server
go tool pprof -top -seconds 10 http://localhost:6060/debug/pprof/profile

Success: http://localhost:6060/debug/pprof/ opens, and go tool pprof prints Saved profile in ... and a table of functions. Use a different port for each component you profile.

Debugging

Components run as host processes built with debug flags, so you can attach a debugger directly. To start the standard profile with Delve DAP servers instead (this needs Delve installed):

mage dev:up no-auth -dap

See the detailed local-development reference for debugging, GoLand and VS Code configurations, and current debug ports.

Troubleshooting