Armada Airflow Operator

This class provides integration with Airflow and Armada

armada.operators.armada module

class armada.operators.armada.ArmadaOperator(name, armada_client, job_service_client, armada_queue, job_request_items, lookout_url_template=None, **kwargs)

Bases: BaseOperator

Implementation of an ArmadaOperator for airflow.

Airflow operators inherit from BaseOperator.

execute(context)

Executes the Armada Operator.

Runs an Armada job and calls the job_service_client for polling.

armada.operators.armada.annotate_job_request_items(context, job_request_items)

Annotates the inbound job request items with Airflow context elements

armada.operators.armada.get_annotation_key_prefix()

Provides the annotation key perfix, which can be specified in env var ANNOTATION_KEY_PREFIX. A default is provided if the env var is not defined

armada.operators.jobservice module

class armada.operators.jobservice.JobServiceClient(channel, max_workers=None)

Bases: object

The JobService Client

Implementation of gRPC stubs from JobService

get_job_status(queue, job_set_id, job_id)

Get job status of a given job in a queue and job_set_id.

Uses the GetJobStatus rpc to get a status of your job

health()

Health Check for GRPC Request

armada.operators.utils module

class armada.operators.utils.JobState(value)

Bases: Enum

An enumeration.

CANCELLED(_ = _ )

CONNECTION_ERR(_ = _ )

DUPLICATE_FOUND(_ = _ )

FAILED(_ = _ )

JOB_ID_NOT_FOUND(_ = _ )

RUNNING(_ = _ )

SUBMITTED(_ = _ )

SUCCEEDED(_ = _ )

armada.operators.utils.airflow_error(job_state, name, job_id)

Throw an error on a terminal event if job errored out

AirflowFailException tells Airflow Schedule to not reschedule the task

armada.operators.utils.default_job_status_callable(armada_queue, job_set_id, job_id, job_service_client)

armada.operators.utils.job_state_from_pb(state)

armada.operators.utils.search_for_job_complete(armada_queue, job_set_id, airflow_task_name, job_id, job_service_client=None, job_status_callable=, time_out_for_failure=7200)

Poll JobService cache until you get a terminated event.

A terminated event is SUCCEEDED, FAILED or CANCELLED