Monitoring and fleet

This page shows you how to follow what your jobs are doing, find out why jobs are waiting, and cancel or preempt many jobs at once without affecting more than you meant to.

You have two tools:

• Lookout, Armada's web UI, for browsing, filtering and acting on jobs.
• armadactl, Armada's command-line tool, for watching a job set, reading scheduling reports, and cancelling, reprioritizing or preempting many jobs at once. It also works in scripts.

Before you start

Check that armadactl can reach Armada:

armadactl get queues

Ready when: it prints apiVersion: armadaproject.io/v1beta1 and kind: Queue, then your queues, with no line containing error. On a new install, the list is empty.

Find Lookout:

• Laptop install: http://localhost:30000
• Team installation (an Armada that your platform team runs for many users): ask them for the address.

Ready when: Lookout opens in your browser and shows a job list.

The examples use the my-team queue from Submitting jobs. To have something to look at, submit a job first.

Follow jobs in Lookout

Find jobs

Lookout has two views, chosen at the top: Jobs lists individual jobs, and Job Sets lists job sets. In the Jobs view you can:

• Filter by queue, state, job ID, submit time or annotation. To filter by job set, filter by queue first.
• Group results, for example by queue, then job set, then state, to see a whole team's work at once. Lookout only offers grouping by job set after you've grouped by queue.
• Choose the columns with Configure columns. Besides the defaults, you can add namespace, priority, priority class, owner, resource requests, node, cluster, pool, exit code, runtime and failure category.
• Select jobs and cancel, reprioritize or preempt them together.

Look inside a job

Click a job to open its details on the right. The tabs are:

A job that ended without running shows This job did not run. in the Result and Logs tabs. A job that's still waiting shows This job has not run.

Logs are only there while the pod exists. Finished pods are cleaned up a few minutes after the job ends, so check logs soon. The job's result stays in Lookout.

On a laptop install, the Logs tab shows TypeError: Failed to fetch. Read logs with kubectl instead. Each job's pod is named armada-<job-id>-0:

kubectl logs -n default armada-<job-id>-0

The Commands tab's commands, and the kubectl hint that watch prints, include --context <cluster>, using Armada's name for the cluster. On a laptop install, that name (Cluster1) isn't a kubectl context, so leave --context out or use --context kind-armada.

States in Lookout

A short job can skip Pending or Running. armadactl watch counts Preempted and Rejected jobs as failed, so use Lookout to tell them apart. For every transition, see Job lifecycle.

Act on jobs

You can cancel, reprioritize and preempt jobs from Lookout, one at a time or several selected together. You need the same permissions as on the command line.

Follow jobs with armadactl

Watch a job set

armadactl watch my-team demo-1

watch streams every change to the jobs in one job set. See Submitting jobs for what the output means.

watch keeps running until you press Ctrl-C. To stop it when every job in the job set has finished, add --exit-if-inactive. It counts only the jobs it has seen so far, so in a job set with older finished jobs it can stop before a newer job ends. If it's still running after the last job finishes, press Ctrl-C.

watch covers one job set at a time. For everything in a queue, use Lookout.

For jobs with an ingress, the address to reach them appears in Lookout's per-run details once the job is running. From a script, read the JobIngressInfoEvent in the event stream. See Interactive jobs and services.

Find out why jobs are waiting

watch shows what's happening to a job set, and the three reports explain what the scheduler is doing:

scheduling-report also accepts --queue and --job-id, but both are deprecated: use queue-report and job-report instead.

For queue-report and scheduling-report, add -v, -vv or -vvv for more detail. job-report does not accept these flags.

In Lookout, a queued job's Scheduling tab shows the same reports.

These reports only cover scheduling attempts. A job rejected with a message such as exceeds the total limit never reached an attempt, so job-report shows none. Read its reason on the Job failed: line in watch, or in the job's Result tab in Lookout.

From your own code

The Python and Go clients can perform the same requests:

See Client libraries for an example.

Act on many jobs at once

One job set

A job set is the everyday unit for acting on many jobs. These need the cancel or reprioritize permission on the queue:

# Cancel every unfinished job in the job set
armadactl cancel job-set my-team demo-1

# Give every unfinished job in the set priority 0 (lower runs first)
armadactl reprioritize job-set my-team demo-1 0

Reprioritizing changes every unfinished job in the set, including running ones. It doesn't interrupt a running job, but the new priority decides its order if it has to run again.

Whole queues

cancel queues and preempt queues act on every matching job in one or more queues. They are the most powerful commands in armadactl.

To cancel jobs in a queue:

1. Dry run first. Name the queue, the job states and the priority classes:

   armadactl cancel queues my-team \
     --job-states queued,running \
     --priority-classes armada-default \
     --dry-run

   It prints the same line and nothing more when it succeeds.

   The output is one line for each queue that would be affected:

   Requesting cancellation of jobs matching queue: my-team, priorityClasses: armada-default, jobStates: queued,running, pools: all

2. Run it:

   armadactl cancel queues my-team \
     --job-states queued,running \
     --priority-classes armada-default

3. Check in Lookout that only the jobs you meant are cancelled.

The flags:

preempt queues takes the same flags except --job-states. It targets allocated jobs in the leased, pending and running states. It only stops jobs in preemptible priority classes.

See Priority and preemption.

On a laptop install, cancel queues works, but preempt queues is refused: everyone has cancel_any_jobs, but not preempt_any_jobs.

Common tasks

A runaway workload is using up your team's share. In Lookout, filter by your queue and group by job set to find the job set with the most jobs. Cancel it with armadactl cancel job-set. If the whole queue must stop, use cancel queues as above, dry run first.

The nightly run needs to go first. armadactl reprioritize job-set my-team <job-set> 0 gives its waiting jobs the lowest numeric priority. Lower values run first within the same priority class; jobs in higher priority classes still come first. It only reorders your own queue. It doesn't take anything from other teams.

What is my team running right now? In Lookout, group by queue, then job set, then state.

The nightly jobs start late every day. That's a scheduling question. Run armadactl get queue-report my-team while they're waiting. If the queue is always short of resources, its priority factor may need changing: ask your platform team.

Operator setup

With the global cancel_any_jobs or preempt_any_jobs permission, you can also act on everything on one executor or node:

armadactl cancel executor <executor>
armadactl cancel node <node-name> --executor <executor>
armadactl preempt executor <executor>
armadactl preempt node <node-name> --executor <executor>

On a laptop install, the executor is called Cluster1.

Troubleshooting