Error categories

When a job fails, the first question is usually: is the problem in my job, or in the platform? An error category answers it at a glance. Armada gives each failed run a short label, such as oom (ran out of memory) or user_error (your program exited with an error), and can add a hint with a known fix. Lookout, job events and metrics call it the failure category.

Your platform team (the operators) decides the categories. Categories are also what retry policies use to decide whether to run a job again.

How categories work

One category per failed run

Each attempt to run a job is a run. Most jobs have one run, but a job that Armada retries has several, and each can fail for a different reason. So the category belongs to the run, not the job.

Who decides the category

When a run fails, the executor (the Armada component that runs pods on a cluster) looks at what Kubernetes knows about the pod:

• each container's exit code, including init containers
• the Kubernetes reason, such as OOMKilled when a container used more memory than its limit
• the container's termination message (what it wrote to /dev/termination-log)
• the pod's error message, for example an image pull error
• the pod's Kubernetes events

Your operator writes rules that turn these into categories. For example:

A subcategory adds detail inside a category. The names on this page are examples: ask your operator for the list on your clusters.

Categories Armada sets itself

Armada sets two categories of its own.

internal means Armada detected the failure itself, not an operator's rule. It's usually a platform problem, but job-rejected and job-creation-failed can point to the job. The subcategory says what happened:

An internal failure has no hint. For job-rejected, fix the job first. Otherwise submit the job again, and tell your operator if it keeps happening.

preemption means Armada stopped the run to give its resources to other work. The subcategory is the kind of preemption, for example fairshare. Lookout shows the job as Preempted, and Armada doesn't retry it. See Priority and preemption.

What a category changes

On its own, it's a label that retry policies can match. Two settings attached to it have effects:

• a hint, added to the failure message you see
• an action for the failed pod: keep it so you can read its logs (Retain, the default), or delete it (Delete)

See why your job failed

In Lookout

• Open the job and select the Result tab. It lists the job's runs, newest first, each with its failure message, Failure Category and Failure Subcategory.
• In the jobs table, add the Failure Category and Failure Subcategory columns from Configure columns. They show the last run's category, so you can scan many jobs at once.

In armadactl watch

watch prints the failure message, including any hint, after Job failed:. It doesn't print the category itself.

In your own tools

The failed event (JobFailedEvent) has the category and subcategory: failureCategory and failureSubcategory in JSON, failure_category and failure_subcategory in the Python client. An empty subcategory is left out of the JSON, and is an empty string in the Python client.

Read the hint

A hint appears after the failure message, separated by blank lines. For example, a job that ran out of memory, from a tested setup:

Container main failed with exit code 137 because OOMKilled:


The job ran out of memory. Increase the memory request, or use a retry policy that adds memory.

What to do for each category

Set up error categories

This section is for operators. Error categories are executor settings, under application.errorCategories. From Armada 0.22.16 they're on by default, and a failure no rule matches gets category unknown and subcategory unknown. In earlier versions they're off by default, so set enabled: true.

Before you start

You need kubectl access to the executor's cluster, with permission to change the executor. Everything on this page needs Armada 0.22.5 or later (onPodEvents arrived in 0.22.5, action: Delete in 0.22.3; categories are older). Lookout and the Lookout ingester must be 0.20.42 or later, with their database migrations run; otherwise categories work but don't appear in Lookout.

Ready when: kubectl get executor -n armada lists your executor (after Getting started, armada-executor).

Step 1. Choose your categories

Start small, with a default for everything else:

application:
  errorCategories:
    enabled: true
    defaultCategory: uncategorized
    categories:
      - name: oom
        rules:
          - onConditions: ['OOMKilled']
            hint: 'The job ran out of memory. Increase the memory request, or use a retry policy that adds memory.'
      - name: node
        rules:
          - onConditions: ['Evicted']
            subcategory: eviction
      - name: user_error
        rules:
          - onExitCodes:
              operator: In
              values: [1, 2, 126, 127]

Step 2. Apply them to the executor

With the Armada Operator, add the settings to the Executor resource under spec.applicationConfig (with Helm, under applicationConfig). Save Step 1's settings with that wrapper in categories-patch.yaml:

spec:
  applicationConfig:
    application:
      errorCategories:
        enabled: true
        defaultCategory: uncategorized
        categories:
          - name: oom
            rules:
              - onConditions: ['OOMKilled']
                hint: 'The job ran out of memory. Increase the memory request, or use a retry policy that adds memory.'
          - name: node
            rules:
              - onConditions: ['Evicted']
                subcategory: eviction
          - name: user_error
            rules:
              - onExitCodes:
                  operator: In
                  values: [1, 2, 126, 127]

Then patch the executor and wait for it to restart:

kubectl patch executor <executor-name> -n armada --type merge --patch-file categories-patch.yaml
kubectl rollout status deploy/<executor-name> -n armada

Apply the same settings to every executor: a failure gets the categories of the executor that ran it.

Success: the rollout finishes, and the executor log has no Config error in error categories line. If the settings aren't valid, the new executor pod stops at startup (see Check the settings). With a rolling update, the old executor pod keeps running meanwhile, so jobs aren't interrupted.

Step 3. Check a failure gets its category

Submit a job that exits with code 1, then look at its Result tab in Lookout.

Success: the run shows Failure Category user_error.

Step 4. Improve the rules over time

1. Add the Failure Category column in Lookout, and watch the metrics (see Monitor categories).
2. Look at failures in uncategorized, and add rules for the common ones.
3. Add hints to rules for failures with a known fix.

To turn your categories off, set enabled: false (with the Armada Operator: kubectl patch executor <executor-name> -n armada --type merge -p '{"spec":{"applicationConfig":{"application":{"errorCategories":{"enabled":false}}}}}'). Armada's own internal and preemption categories still appear.

Settings reference

Names and subcategories can be at most 63 characters.

Matchers

Each rule uses exactly one matcher:

The container matchers only look at containers that exited with a non-zero code, including init containers. containerName limits them to one container; it applies to onConditions: ["OOMKilled"], onExitCodes and onTerminationMessage, and has no effect on the pod reasons or the pod matchers.

The pod's error message

onPodError matches the message the executor records for the pod, which takes one of these shapes:

• A pod that failed with a status message (an eviction, say): that message.

• A pod that failed without one: one line per failed container, Container <name> failed with exit code <code> because <reason>: <message>.

• A pod that never started: the message of the kubernetes.pendingPodChecks rule that failed it, including the matched event. For example, for an image that doesn't exist:

  Unable to start pod - encountered an unrecoverable problem.
  Matched check: "check-0"
  Failed to pull image "alpine:does-not-exist-9": rpc error: code = NotFound desc = failed to pull and unpack image ...

When the pod matchers see a failure

• A pod that failed: both onPodError and onPodEvents can match, in every category from Armada 0.22.13 (#5250). In earlier versions they only match in a category with action: Delete; in a Retain category, the failure gets the default category instead. If a kubernetes.failedPodChecks rule matches first, the executor returns the job to the queue as a startup retry. onPodEvents can still set that run's category from 0.22.15; in earlier versions it never sees it.

• A pod that never started (for example, an image pull error): a kubernetes.pendingPodChecks rule with action: Fail fails the pod after its grace period, and onPodError matches that rule's message. From 0.22.15, onPodEvents can match too; in earlier versions it doesn't. A rule with action: Retry makes a startup retry instead, which no retry policy sees.

The grace period counts from the pod's last status change, and the executor checks every 10 seconds. With the default image-pull rule, the failure arrives about 90 to 100 seconds after the pod went Pending.

Rule order

The executor checks rules in the order they're written, across all categories. The first match gives the category, subcategory and pod action, so put specific rules before general ones.

Check the settings

When enabled is true, the executor checks the settings at startup. If something is wrong, it stops with a line like:

FATAL application.go:211 Config error in error categories: category "oom" rule 0: unknown condition "OutOfMemory", valid values: OOMKilled, Evicted, DeadlineExceeded

It checks for:

• an unknown condition in onConditions
• a wrong operator, or an empty list of exit codes
• a regular expression that isn't valid
• a rule with no matcher, or more than one
• an onPodEvents type other than Normal or Warning
• two categories with the same name, a category with no rules, or an empty name
• a name, subcategory or default longer than 63 characters
• an action other than Retain or Delete
• from 0.22.13: a category or defaultCategory named internal or preemption (category name "internal" is reserved for categories that Armada sets)

Keep or delete the failed pod

• Retain (default) keeps the failed pod, so people can read its logs, until normal clean-up: after kubernetes.failedPodExpiry (10 minutes by default), or earlier if the cluster holds more than kubernetes.maxTerminatedPods finished pods (1000). Normal clean-up leaves pods younger than kubernetes.minimumPodAge (3 minutes) alone, but the maxTerminatedPods limit can delete a younger pod.
• Delete deletes the pod, and its logs, and reports the failure once the pod is gone, about 10 to 20 seconds later than with Retain. If the pod isn't gone after kubernetes.stuckTerminatingPodExpiry (30 seconds), the failure is reported as internal, stuck-terminating.

The action only applies to pods that failed. A pod that never started is always deleted after its failure is reported.

Retry policies need Delete on every category they retry; otherwise the retry's pod name collides with the kept pod. Keep Retain on categories you don't retry, such as user_error, so people can read their logs.

Write useful hints

People read the hint first, so make it a short instruction ("Increase the memory request"). Don't repeat the failure message. Only add hints for failures with a known fix.

Monitor categories

Watch the share of uncategorized failures. A high share means your rules miss common failures.

Examples

These keep the default action: Retain, except the GPU example. Set action: Delete on any category a retry policy retries.

Separate program errors from platform errors

The configuration in Step 1 gives:

Only look at the main container

A job has a main container and a sidecar that uploads results. A sidecar error isn't a program error, so this rule looks only at main:

- name: user_error
  rules:
    - containerName: main
      onExitCodes:
        operator: NotIn
        values: [137, 143]

NotIn matches every exit code except those listed: here, everything except being stopped by a signal (137 is SIGKILL, 143 is SIGTERM). Running out of memory also gives 137, so keep a separate oom rule.

An image built for the wrong CPU

Someone builds an image on an Arm laptop and runs it on x86 nodes. The image pull fails with a known message:

- name: image
  rules:
    - onPodError:
        pattern: 'no match for platform in manifest'
      subcategory: platform_mismatch
      hint: "Build the image for the cluster's CPU architecture, for example linux/amd64."

The pod never starts, so the failure arrives when the default image-pull check fails it, about 90 seconds after the pod went Pending.

No free GPU on the node at startup

Sometimes a node has no free GPU when the pod starts, even though Armada placed the job there. The kubelet rejects the pod with an UnexpectedAdmissionError event, such as Pod Allocate failed due to requested number of devices unavailable for nvidia.com/gpu. Requested: 1, Available: 0:

- name: gpu
  action: Delete
  rules:
    - onPodEvents:
        type: Warning
        reason: UnexpectedAdmissionError
        regexp: 'nvidia.com/gpu'
      subcategory: unavailable
      hint: 'The node could not give a GPU to the job. Submit the job again.'

The pod fails before its container starts, so before Armada 0.22.13 this rule needs action: Delete (see When the pod matchers see a failure). From 0.22.13 it also works with Retain.

Messages your program writes

A program can write a short message to /dev/termination-log before exiting, and a rule can match it:

- name: data
  rules:
    - onTerminationMessage:
        pattern: '(?i)input file not found'
      subcategory: missing-input
      hint: 'Check that the input path of the job exists.'

Tested: a container that ran echo Input file not found > /dev/termination-log; exit 4 got category data, subcategory missing-input, and the hint.

Troubleshooting