Developer Guide

This guide helps you set up a development environment for contributing to Armada or customizing it with new features. For contribution guidelines, see the Contributor Guide.

Prerequisites

Install the following tools before you begin. These are verified requirements from the Armada source code:

• Go (version 1.26 or later) - Required for building Armada
• gcc (for Windows, see tdm-gcc) - Required for CGO compilation
• mage - Build tool used throughout the Armada project (similar to Make, written in Go)
• Docker - Container runtime for running dependencies
• kubectl - Kubernetes command-line tool
• protobuf (version 3.17.3 or later) - Protocol buffer compiler (required if you modify .proto files)
• kind - Kubernetes in Docker (bootstrapped via mage BootstrapTools)

Note: Additional tools are automatically installed via mage BootstrapTools from tools.yaml, including golangci-lint, sqlc, go-swagger, and others.

Development Environment Setup

Armada provides two main ways to run components locally for development. Choose the method that best fits your workflow.

Using mage dev (Recommended)

The mage dev targets automate the setup process and are the recommended way to get started:

• Bootstraps required tools from tools.yaml
• Starts dependencies (Pulsar, Redis, PostgreSQL) in containers
• Builds and starts Armada components (mage dev:up runs them via goreman; mage dev:full runs them in containers against a kind cluster)

Note: If you edit a proto file, run mage proto to regenerate the Go code.

The mage dev targets:

# Run dependencies in containers and Armada components via goreman (fast iteration)
mage dev:up

# Run the entire stack in containers against a Kind cluster (what CI uses)
mage dev:full

We use mage dev:full to test the CI pipeline. Use it to test changes to core components.

To stop the local development environment:

mage dev:down       # stop the dependency containers (after `mage dev:up`)
mage dev:fullDown   # stop the containerized stack and tear down Kind (after `mage dev:full`)

Using Goreman

Goreman is a Go-based clone of Foreman that manages Procfile-based applications, allowing you to run multiple processes with a single command. Goreman will build the components from source and run them locally, making it easy to test changes quickly.

1. Install goreman:

   go install github.com/mattn/goreman@latest

2. Start dependencies:

   docker compose -f _local/compose/stack.yaml up -d

   Note: Images can be overridden using environment variables: REDIS_IMAGE, POSTGRES_IMAGE, PULSAR_IMAGE, KEYCLOAK_IMAGE

3. Initialize databases and Kubernetes resources:

   _local/scripts/init.sh

4. Start Armada components:

   goreman -f _local/procfiles/no-auth.Procfile start

Local Development with Authentication

To run Armada with OIDC authentication enabled using Keycloak:

1. Start dependencies with the auth profile:

   docker compose -f _local/compose/stack.yaml --profile auth up -d

   This starts Redis, PostgreSQL, Pulsar, and Keycloak with a pre-configured realm.

2. Initialize databases and Kubernetes resources:

   _local/scripts/init.sh

3. Start Armada components with auth configuration:

   goreman -f _local/procfiles/auth.Procfile start

4. Use armadactl with OIDC authentication:

   armadactl --config _local/.armadactl.yaml --context auth-oidc get queues

Local Development with Fake Executor

For testing Armada without a real Kubernetes cluster, you can use the fake executor that simulates a Kubernetes environment:

goreman -f _local/procfiles/fake-executor.Procfile start

The fake executor simulates:

• 2 virtual nodes with 8 CPUs and 32Gi memory each
• Pod lifecycle management without actual container execution
• Resource allocation and job state transitions

This is useful for:

• Testing Armada's scheduling logic
• Development when Kubernetes is not available
• Integration testing of job flows

Configuration Options

You can set the ARMADA_COMPONENTS environment variable to choose which components to run:

export ARMADA_COMPONENTS="server,executor"

Testing Your Setup

Verify that your development environment is working:

# Run the test suite
mage testsuite

Or manually:

go run cmd/armadactl/main.go create queue e2e-test-queue
export ARMADA_EXECUTOR_INGRESS_URL="http://localhost"
export ARMADA_EXECUTOR_INGRESS_PORT=5001
go run cmd/testsuite/main.go test --tests "testsuite/testcases/basic/*" --junit junit.xml

Code Structure

Understanding Armada's codebase structure will help you navigate and contribute effectively.

Directory Layout

armada/
├── cmd/              # Main entry points for all components
│   ├── server/       # Armada server (API server)
│   ├── executor/     # Executor (runs in each K8s cluster)
│   ├── scheduler/    # Scheduler (job scheduling logic)
│   ├── lookout/      # Lookout (job monitoring/UI backend)
│   └── armadactl/    # Command-line interface
├── internal/         # Internal packages (not for external use)
│   ├── server/       # Server implementation
│   ├── executor/     # Executor implementation
│   ├── scheduler/    # Scheduler implementation
│   ├── lookout/      # Lookout implementation
│   └── common/       # Shared utilities
├── pkg/              # Public packages (for external use)
│   ├── api/          # gRPC API definitions
│   └── client/       # Client libraries
├── config/           # Configuration files for components
├── deployment/       # Helm charts and deployment configs
├── magefiles/        # Build automation (mage targets)
└── testsuite/        # Integration test cases

Key Components

• Server (cmd/server/, internal/server/): The main API server that accepts job submissions and manages queues
• Executor (cmd/executor/, internal/executor/): Runs in each Kubernetes cluster and executes jobs
• Scheduler (cmd/scheduler/, internal/scheduler/): Determines when and where jobs should run
• Lookout (cmd/lookout/, internal/lookout/): Provides job monitoring and UI backend
• armadactl (cmd/armadactl/): Command-line interface for interacting with Armada

Using mage

mage is the build tool used throughout the Armada project. To see all available commands:

mage -l

Common mage targets:

• mage dev:up - Run dependencies in containers and Armada components via goreman
• mage dev:full - Run the entire stack in containers against a Kind cluster (what CI uses)
• mage buildDockers - Build Docker images
• mage proto - Generate Go code from proto files
• mage testsuite - Run the test suite
• mage ui - Build and run the Lookout UI

Debugging and Profiling

Profiling with pprof

Go provides a profiling tool called pprof. To use pprof with Armada, enable the profiling socket in your config.

profiling:
  port: 6060
  hostnames:
    - 'armada-scheduler-profiling.armada.my-k8s-cluster.com'
  clusterIssuer: 'k8s-cluster-issuer'
  auth:
    anonymousAuth: true
    permissionGroupMapping:
      pprof: ['everyone']

Debugging components

mage dev:up builds each component with debug flags (-gcflags="all=-N -l") and runs them as host processes, so you can attach a debugger (Delve, VS Code, or GoLand) to any running process directly. Each component reads _local/<component>/config.yaml.

For VS Code, use the launch configurations in .vscode/launch.json. See the VS Code Debugging Guide for details.

Debug Port Mappings

GoLand Run Configurations

Run configurations are available in the .run directory. When opening the project in GoLand, you can run Armada in both standard and debug mode.

Note: The executor requires a Kubernetes config in $PROJECT_DIR$/.kube/internal/config.

Other Debugging Methods

Run mage dev:deps to only spin up the dependencies (redis, postgres, pulsar), then run individual components yourself. Each component reads its config from _local/<component>/config.yaml.

Extending Armada

Armada can be extended and customized in several ways:

Custom Schedulers

The scheduler is designed to be extensible. You can implement custom scheduling algorithms by modifying the scheduler logic in internal/scheduler/. The scheduler handles:

• Job queuing and prioritization
• Resource allocation
• Gang scheduling
• Preemption logic

The scheduler code is located in internal/scheduler/ and can be customized to implement different scheduling strategies.

Custom Executors

While the standard executor works with Kubernetes, you could create custom executors for other platforms. The executor interface is defined in pkg/executorapi/ and communicates with the scheduler via gRPC.

Client Libraries

Armada provides client libraries for multiple languages that you can extend or use as reference:

• Python: client/python/
• Java: client/java/
• Scala: client/scala/
• .NET: client/DotNet/

These libraries provide programmatic access to Armada's APIs and can be used as reference for building custom clients.

Integration Examples

Armada has been integrated with various systems. These can serve as examples for creating your own integrations:

• Airflow: third_party/airflow/ - Airflow operator for Armada
• Metaflow: armada-metaflow repository - Metaflow decorator for Armada
• Jenkins: jenkins-plugin repository - Jenkins plugin for Armada
• Spark: armada-spark repository - Spark cluster manager for Armada

UI Development

To develop the Lookout UI locally, the UI code is located in internal/lookoutui/. The UI is built with React and TypeScript.

When using Goreman, the UI runs automatically on http://localhost:3000 (frontend dev server).

To run the UI separately for development:

cd internal/lookoutui
yarn
yarn openapi
PROXY_TARGET=http://localhost:8089 yarn dev

This starts a development server on http://localhost:3000 that proxies API requests to the backend.

Alternatively, build a production version with:

mage ui

This builds the UI and makes it available at http://localhost:8089.

Troubleshooting

Port 6443 Already in Use

If port 6443 is already in use, modify _local/kind/cluster.yaml to use a different port:

- containerPort: 6443
  hostPort: 6444 # Change to an available port
  protocol: TCP

Arm/M1 Mac Issues

On Arm/M1 Macs, you may need to set:

export PULSAR_IMAGE=richgross/pulsar:2.11.0

For more information on known issues:

• Arm issue
• Windows issue

Additional Resources

Website Resources

• API Reference - API reference for REST and gRPC
• Contributor Guide - Contribution guidelines and PR process
• Understanding Armada - Core concepts and architecture
• Community - Get help, connect with the community, and find support resources