Submitting jobs

Before you start

You need a working armadactl. On your laptop, follow Getting started first. On your team's Armada, ask your platform team for the server address and how to log in.

mkdir -p ~/armada-guide
cd ~/armada-guide

1. Create a queue

Every job goes into one queue. Queues are how Armada shares resources fairly between teams. These guides use a queue called my-team.

armadactl create queue my-team --priority-factor 1

Success: Created queue my-team.

Creating a queue requires the create_queue permission. On a laptop install you have it. On a team installation you usually don't: ask your platform team to create the queue for you.

2. Write a job file

A job file describes one or more jobs. Create job.yaml:

cat > job.yaml <<'EOF'
queue: my-team
jobSetId: demo-1
jobs:
  - namespace: default
    priority: 0
    podSpec:
      terminationGracePeriodSeconds: 0
      restartPolicy: Never
      containers:
        - name: sleeper
          image: alpine:3.21
          command: ['sh', '-c']
          args: ['echo hello from Armada; sleep 30']
          resources:
            requests:
              cpu: 200m
              memory: 128Mi
            limits:
              cpu: 200m
              memory: 128Mi
EOF

This job starts a small Linux container, prints hello from Armada, waits 30 seconds, and exits.

Armada rejects the whole file if any job breaks one of these rules:

• Each job has exactly one pod. To start several pods together, use a gang.
• namespace is required.
• Every container has resources with both requests and limits, and by default they're equal. Operators can allow limits above requests with a server setting: see Operator setup.
• Node affinity is supported only in its required... form. preferred... node affinity is rejected.
• priorityClassName, if set, must be on the cluster's allowed list. See Priority and preemption.

3. Check the file

Dry-run for a quick check. It only checks, on your computer, that the file is valid YAML and has at least one job; the rules above are checked by the server when you really submit:

armadactl submit job.yaml --dry-run

Success: it prints nothing and returns to the prompt.

4. Submit the job

JOB_ID=$(armadactl submit job.yaml | awk '{print $5}')
echo "Job ID: $JOB_ID"

This submits the job and keeps its ID in JOB_ID for the rest of this terminal session. You'll need the ID to read the job's output and to cancel it.

Success: it prints the job's ID:

Job ID: <job-id>

If you open a new terminal, set it again with JOB_ID=<job-id>, using your own ID. If you lose the ID, it's in every watch line for the job, and in Lookout.

5. Watch it run

armadactl watch my-team demo-1

You'll see the lifecycle as a stream of events. Each line carries a timestamp and running state counts, then the event:

Watching job set demo-1
Oct  2 18:34:13 | Queued:   0, Leased:   0, ... | JobSubmittedEvent, job id: 01m3...
Oct  2 18:34:13 | Queued:   1, Leased:   0, ... | JobQueuedEvent, job id: 01m3...
Oct  2 18:34:17 | Queued:   0, Leased:   1, ... | JobLeasedEvent, job id: 01m3...
Oct  2 18:34:24 | ... | JobPendingEvent, job id: 01m3... pod: 0
Oct  2 18:34:27 | ... | JobRunningEvent, job id: 01m3... pod: 0
Oct  2 18:34:59 | ... | JobSucceededEvent, job id: 01m3... pod: 0

Success: the last event is JobSucceededEvent.

A job moves through this sequence: Queued (waiting for the scheduler), Leased (cluster and node chosen), Pending (pod being created), Running, then Succeeded, Failed, or Cancelled.

watch keeps running until you press Ctrl-C. To stop it when every job in the job set has finished, add --exit-if-inactive. It counts only the jobs it has seen so far, so in a job set with older finished jobs it can stop before a newer job ends. If it's still running after the last job finishes, press Ctrl-C.

If watch says the job set does not exist, wait a few seconds and run it again.

You can also follow jobs in Lookout, Armada's web UI. On a laptop install it's at http://localhost:30000.

6. See the job's output

Read the output soon after the job finishes. Finished pods are cleaned up after a few minutes, and their output goes with them.

On a laptop install, use kubectl. Each job's pod is named armada-<job-id>-0:

kubectl logs -n default armada-$JOB_ID-0

(Or replace $JOB_ID with the ID itself.)

On your team's Armada, open the job in Lookout and choose the Logs tab. (This needs a log service that the laptop install doesn't include.)

Success: you see hello from Armada.

7. Reorder or cancel jobs

Change the priority of one job, or of every job in a job set. Lower numbers run first:

armadactl reprioritize job my-team demo-1 <job-id> 5
armadactl reprioritize job-set my-team demo-1 5

This changes the priority of every unfinished job, including running jobs. It does not interrupt a running job, but the new priority affects its order if it needs to run again.

The job you submitted earlier has already finished, so cancelling it does nothing. Submit a new copy first:

armadactl submit job.yaml

Then, within 30 seconds, before it ends, cancel every job in the job set:

armadactl cancel job-set my-team demo-1

To cancel only one job, use its ID: armadactl cancel job my-team demo-1 <job-id>.

Success: in watch or Lookout, the job moves to Cancelled. If you cancel a job while it's starting, watch may still print JobPendingEvent or JobRunningEvent after JobCancelledEvent; these are late updates, and the job stays cancelled.

Clean up

Nothing to do: finished pods are cleaned up automatically. Keep the my-team queue and the ~/armada-guide folder for the next guides.

Reference

Job file fields

Fields on each entry under jobs::

The pod spec

Apart from the rules in Step 2, the pod spec is plain Kubernetes: imagePullSecrets, secret and config-map volumes, securityContext, init containers, nodeSelector, tolerations, activeDeadlineSeconds and serviceAccountName work as the Kubernetes docs describe.

Init containers have one extra rule: by default, Armada rejects a whole-number CPU value such as 1, so use a fractional value such as 900m.

Armada changes three things from a regular Kubernetes job launch:

• It sets restartPolicy: Never, whatever you wrote. Armada handles retries, not Kubernetes.
• It adds a node selector that pins the pod to the node the scheduler chose.
• It adds the environment variables below.

Fields you leave empty may be filled in from server defaults:

Operators can change these.

Special annotations

Environment variables in every job

Armada adds these to every container, including init containers, unless you set a variable with the same name:

Getting data in and out

Armada runs pods. It doesn't move data.

• Inputs come from your container image, or from volumes in the job's namespace (a PVC, or object storage your code reads).
• Results must be written to storage outside the pod before the container exits. The pod's files are gone when the job ends.
• Logs are normal container logs, readable while the pod exists (see Step 6). Finished pods are cleaned up after a few minutes. Lookout keeps the job's outcome: its state, error message and exit codes. To keep logs, write them to your own storage.

Operator setup

These settings control the rules and defaults above. They're in the Armada server's configuration, under submission:. On an Operator install, that's the ArmadaServer resource's spec.applicationConfig.submission.

Troubleshooting