Core concepts

This page explains the fundamental concepts you need to effectively use Armada. For quick definitions of any term, see the Glossary.

What Armada does

Armada is a multi-cluster batch job scheduler for Kubernetes. You give it jobs, programs that run and then finish, and it decides when and where each one runs. It places jobs to make good use of your compute resources.

It's built for situations where Kubernetes on its own struggles:

• Many jobs: millions of queued jobs, because Armada queues them in its own control plane instead of in Kubernetes.
• Many clusters: one place to submit work that can run on any of several Kubernetes clusters.
• Many teams: each team gets a fair share of the compute farm resources over time, so one busy team can't crowd everyone else out.

Armada doesn't replace Kubernetes. It sits on top, decides which jobs should run where, and then hands each one to the right cluster to run as a normal pod.

Jobs

A job is one piece of work. It's made of:

• A Kubernetes pod spec: the containers to run, and the CPU and memory they need. Exactly one pod per job.
• Optionally, network objects that live and die with the job: services and ingresses. See Interactive jobs and services.
• Armada details: which queue it goes into, which job set it belongs to, and its priority.

Every job gets a unique job ID when you submit it. You use the ID to look the job up, read its output or cancel it.

A job's life

Every job moves through these states:

Succeeded, Failed and Cancelled are final. Lookout also shows two kinds of failure as states of their own: Preempted (Armada stopped the job for other work) and Rejected (the job was turned down before it ran); armadactl watch shows both as failed events. The API can also report Submitted and Unknown. (See the API reference).

Before a job is final, Armada may run it more than once. Each attempt is a run:

• For recoverable startup failures (for example, a pod can't be placed on its node), Armada can put the job back in the queue, up to the configured attempt limit. These are startup retries. Jobs with the annotation armadaproject.io/failFast: "true" don't get them, and neither do gang jobs, which always fail fast.
• If your platform team has set up retry policies, Armada can also rerun a job that started and then failed, for failures where a second run is likely to work. Without one, a failed job stays failed, and you submit it again if the work still needs doing.

Every state change is published as an event, which you can follow with armadactl watch or in Lookout.

Try it: Submitting jobs.

Queues

A queue is where jobs wait, and the unit Armada uses to share resources fairly. Every job belongs to exactly one queue. Most organisations give each team its own queue.

A queue has:

• A priority factor: its weight when Armada shares resources. Lower means a larger share. It only matters relative to other queues.
• Permissions: who may submit, cancel, preempt, reprioritize and watch jobs in it.
  • Owners may do all five. If no user owners are specified when creating a queue, the creator becomes an owner.
• Optional labels, so you can act on many queues at once.
• Optional retry policies, which decide which failures Armada retries automatically for the queue's jobs.

A queue can be cordoned (paused): no new jobs are leased until it's uncordoned. Jobs already leased or running carry on.

Job sets

A job set is a label you choose to group related jobs within a queue, such as every job from one nightly run or one experiment. You set it with jobSetId in your job file.

Job sets make it easy to:

• Watch all the jobs together (armadactl watch <queue> <job-set>).
• Cancel or reprioritize them together.

Job sets are only for grouping. They don't affect when jobs run.

Fair share

When there aren't enough compute resources for every waiting job, Armada decides which queue's jobs go next by fair share: each queue is entitled to a share of those resources, weighted by its priority factor.

• Only active queues count, meaning queues with jobs waiting or running. If a team has nothing to run, its share goes to the others.
• So a queue's share changes as other queues become busy or quiet.
• A queue using less than its share gets its jobs scheduled first. A queue using more may have its jobs preempted (stopped) to give others their share, if those jobs are preemptible. This is on by default.

How fair share is calculated

Armada uses dominant resource fairness, based on what each queue is using now:

1. For each resource (by default CPU, memory, GPUs and ephemeral storage), Armada works out what fraction of the pool the queue's running jobs request. A pool is a group of nodes the scheduler shares out together; a small installation often has just one.
2. The queue's cost is the largest of those fractions: its dominant resource. A queue using 30% of the pool's CPU and 10% of its memory costs 0.3.
3. Each queue's cost is divided by its weight, which is 1 ÷ priority factor, and Armada balances that cost ÷ weight across active queues. A queue with priority factor 2 is entitled to half as much as one with priority factor 1.

If your platform team sets a pool's shortJobPenaltyCutoff, very short jobs are charged as if they ran for at least that long, so submitting many tiny jobs doesn't take more than its share. It's off by default.

The scheduler balances three goals: fairness (queues below their share go first), throughput (keeping the machines busy) and timeliness (urgent jobs before less urgent ones).

Priority

Two different things are called "priority" in Armada:

Within a queue, jobs are ordered by their priority class's urgency, then by job priority, then by when they were submitted. Jobs that are already running always come before waiting ones.

Priority classes

A priority class has two properties:

• Priority (urgency): jobs in a higher class are scheduled first, and can push out preemptible running jobs from lower classes.
• Preemptible or not: whether Armada may stop the job once it's running.

By default, Armada offers armada-default (not preemptible, and used when you don't choose) and armada-preemptible. Both have the same urgency, so urgency preemption doesn't happen until a platform team adds a more urgent class.

Armada's priority classes look like Kubernetes priority classes, but they're Armada's own. Armada makes every decision about urgency and preemption itself, before pods reach Kubernetes.

Preemption

Preemption means Armada stops a running job to make room for other work. It happens in three ways:

• Urgency preemption: a job in a higher priority class needs resources that preemptible lower-priority jobs are using.
• Fair-share preemption: a queue is using more than its fair share while other queues are waiting. On by default.
• Manual preemption: a person runs armadactl preempt job.

Only preemptible jobs are ever preempted: running jobs in a non-preemptible class such as armada-default are left alone, whatever the reason.

A preempted job gets a short time to shut down (its termination grace period), then ends as failed with the reason (shown as Preempted in Lookout). It isn't restarted, even by a retry policy. Jobs that can be preempted should save their progress and be safe to submit again.

Try it: Priority and preemption.

Gangs

A gang is a group of jobs that Armada schedules onto a set of nodes all together, or not at all. Distributed programs such as MPI, multi-node training and Spark need every worker running at once: half the workers is no use.

You make a gang with annotations on each job:

• armadaproject.io/gangId: the same name on every member
• armadaproject.io/gangCardinality: how many members there are
• armadaproject.io/gangNodeUniformityLabel (optional): keep every member on nodes that share the same value of a label, such as the same zone. The platform team must enable the label: the executor has to report it (kubernetes.trackedNodeLabels) and the scheduler has to index it (indexedNodeLabels). Otherwise the gang never starts.

Armada waits until it can start every member at once. If one member is preempted, the whole gang is stopped. Gangs always fail fast: Armada doesn't retry them, either when a pod can't start or through retry policies. Without a uniformity label, a gang's members can be placed in different worker clusters.

Try it: Gang jobs.

Failures and retries

When a run fails, Armada can label it with a failure category, a short name such as oom (out of memory) or user_error (your program exited with an error), and sometimes a hint with a known fix. Your platform team defines the categories. They show in Lookout, and they're what retry policies use to decide whether to run a job again: for example, retry oom with more memory, but never retry user_error. See Error categories.

How the pieces fit together

An Armada installation has:

• A control plane: the server (accepts jobs), the scheduler (decides what runs where), Lookout (the web UI), and the ingesters (save events to Armada's databases).
• One executor per Kubernetes worker cluster: it asks the scheduler for jobs, creates their pods, and reports back.

Jobs flow like this:

1. You submit a job to the server, which checks it and records it.
2. The scheduler adds it to its queue (Queued).
3. When there's room and it's the job's turn, the scheduler picks a node (Leased).
4. The executor for that cluster creates the pod (Pending), and Kubernetes starts it (Running).
5. When the job finishes, the executor reports the result (Succeeded, Failed or Cancelled).

For the details, including how Armada stays consistent at scale, see Architecture.

Next steps

• Getting started: run Armada on your laptop.
• Submitting jobs: your first job, step by step.
• Glossary: every Armada term in one place.