taskmaster.md

Taskmaster

The taskmaster runs one TES task in Kubernetes. The TESK API starts one taskmaster per task, as a Kubernetes job, and passes it the task as JSON: the executors already converted to Kubernetes job specs, and the task's inputs, outputs, volumes and resources.

What it does

1. If the task has inputs, outputs or volumes, the taskmaster creates a persistent volume claim (PVC) of the task's disk size (resources.disk_gb) for them. If it has inputs, it then runs the filer as a job (<task>-inputs-filer) that downloads them onto that volume. Each volume, and the directory of each input and output, is a directory of the PVC, mounted at the task's path in the executors and below /tesk/task in the filer (see Task paths in the filer). An output's directory is the directory of its path, except for an output with wildcards and a path_prefix (TES 1.1): that one is mounted at the pattern's deepest directory without wildcards (/data for /data/*/out.txt), below which every match is (see Outputs with wildcards). A pattern with wildcards in its first component (/*.txt) would need a mount at /, so the taskmaster refuses it: it stops with an error before it creates the PVC or any job.
2. It runs the executors, one after another, each as a job with the task's volume mounted. If an executor has resource requests but no limits, the requests also become its limits.
3. If the task has outputs, it runs the filer again (<task>-outputs-filer) to upload them. It then deletes the PVC, if there is one. Volumes need no filer: their directories on the PVC are created when the executors start, and their contents aren't uploaded.

The filer jobs have the label taskmaster-name: <task>, as the executor jobs have it from the TESK API, so that all jobs of a task can be selected by it (kubectl get jobs -l taskmaster-name=<task>). The executor jobs have job-type: executor; the filer jobs have no job-type (see Upgrading for the TESK API release they need). The label is on the jobs only; the filer pods have their own labels (see Pod security).

The taskmaster polls each job until it completes. If a job doesn't complete, the taskmaster stops without running the remaining jobs, deletes the PVC, and exits with code 0. The TESK API derives the task's state from the jobs, not from the taskmaster's exit code: an executor job that failed gives EXECUTOR_ERROR, a filer job that failed SYSTEM_ERROR. So the taskmaster keeps the jobs that failed, with the logs of their pods that ended.

An executor can ignore its errors (TES 1.1 executors[].ignore_error): the TESK API then gives the executor's job the annotation ignore-error: "true". If such an executor's job fails, the taskmaster logs a warning (Executor job '...' failed; ignoring the error), keeps the job, and goes on with the next executor and, after the last, the outputs filer, as if the executor had completed. Before it goes on, it waits until Kubernetes has marked the job Failed, which Kubernetes 1.31 and later do only once the job's pods have ended (after FailureTarget), so that they don't run next to the next job on the task's volume; if that takes longer than --pod-timeout, the task stops. The TESK API reports the executor's exit code and logs from the kept job, and the task's state from the other jobs (COMPLETE if they complete); this needs a TESK API release that passes ignore_error (see tesk-api#159). Any failure of the job's pods is ignored: a command that exits with a non-zero code, but also a pod that Kubernetes evicts, kills for using too much memory, or loses with its node, as the taskmaster can't tell these apart. A job that the taskmaster ended itself still stops the task, as for any executor, as its container may never have run: one it failed because a pod couldn't start (e.g. an image that can't be pulled) or wasn't scheduled in time, also in an earlier run of the taskmaster (the job's activeDeadlineSeconds is 1, see below), and one it deleted. That holds also when the taskmaster's attempt to fail or delete the job got a server error, even if the job then fails on its own: in doubt, the task stops. The TESK API is expected not to ignore such a failure either: a job that the taskmaster failed has the condition Failed with the reason DeadlineExceeded, and a deleted job is missing, which gives SYSTEM_ERROR. A cancelled task stops as before. Other values of the annotation, or none, stop the task at the executor's failure.

Once the outputs filer has completed, the taskmaster records the uploaded files (TES 1.1 logs[].outputs): it reads the last line that starts with TESK_OUTPUT_FILE_LOGS among the last 1,000 lines of the log of the filer's succeeded pod (see Filer), checks it, adds each file's URL and path in the executors, derived from its output as the upload does, and stores the list, with the keys url, path and size_bytes, as the annotation output-file-logs (compact JSON) of the outputs filer's job, from which the TESK API reports it (tesk-api#160). The annotation holds at most 200,000 bytes, as Kubernetes allows 256 KiB for all of a job's annotations: if the list is longer, the last entries are left out, with a warning. If the log can't be read, has no valid line, or the job can't be patched, or any other error occurs, the taskmaster logs a warning and the task completes without the annotation; the outputs are uploaded either way. The URLs are those of the task, so the annotation holds credentials only where the task's URLs do, like the task stored with the taskmaster's job. This uses the role's existing permissions (list on pods, get on pods/log, patch on jobs).

The PVC is also deleted when the taskmaster stops on a cancelled task, also one cancelled before the taskmaster started. Kubernetes frees the PVC's storage only once no pod refers to it: until the task's jobs, and so their pods, are deleted, the PVC stays Terminating, as after a task that completed. The jobs stay until they are deleted, by hand or after a TTL (see Deleting finished tasks). To keep the PVCs of tasks that don't complete, e.g. to inspect their files, set KEEP_FAILED_PVC to true; then delete them yourself. Tasks that complete delete their PVC either way.

An error deleting the PVC is logged, also a request that times out (after 5 seconds to connect or 30 seconds to answer, per attempt of the Kubernetes client); the taskmaster goes on as it would (e.g. stops, or ends a completed task). An unexpected error of the taskmaster itself fails its pod and keeps the PVC, as a retry of the pod reuses it; if every retry fails, the PVC is left behind. An interrupt (SIGINT or SIGTERM, see below) keeps it too, except a SIGTERM on a cancelled task, which deletes it like any other cancellation.

A job counts as failed when Kubernetes sets its condition Failed, or FailureTarget, which Kubernetes 1.31 and later set first; it counts as complete with Complete or SuccessCriteriaMet. Other conditions, e.g. Suspended, leave the job running.

A pending pod whose containers can't start fails its job after --pod-timeout seconds. That covers an image that can't be pulled (ErrImagePull, ImagePullBackOff, InvalidImageName) and a broken container configuration (CreateContainerConfigError, e.g. a missing secret or ConfigMap), for init containers too. The timeout is measured from the pod's start, or from its creation if it hasn't started. The taskmaster checks the pods only once the job has been active for longer than --pod-timeout, then at most once a minute, and stops checking once none of them is pending. When a pod of the job fails, it checks again, so that a retry or the replacement of an evicted pod that can't start times out too. A pull error that comes after the pod has waited longer than --pod-timeout, e.g. after a long pull of a large image, fails the job at the next check.

To fail the job, the taskmaster sets its activeDeadlineSeconds to 1 second, so that Kubernetes deletes the job's active pods and marks the job Failed with the reason DeadlineExceeded (the pod's events show why it couldn't start). The taskmaster then handles it like any job that failed, also after a restart of the taskmaster. If Kubernetes hasn't marked the job within another --pod-timeout, the taskmaster stops anyway and keeps the job. If the Kubernetes API rejects the change (e.g., the taskmaster's role lacks patch on jobs), the taskmaster deletes the job, as earlier releases did, and stops; after other errors, it tries again at the next poll. Pods that are pending for other reasons, e.g. waiting to be scheduled or pulling a large image, are queued and keep waiting.

By default, a pod that the scheduler can't place waits forever, e.g. one that asks for more CPU or memory than any node has: the task stays QUEUED. With TESK_MAX_QUEUE_SECONDS, such a pod (its condition PodScheduled is False with the reason Unschedulable) ends the task once it has waited that many seconds since its creation. The taskmaster logs an error with the job's name and the scheduler's message, e.g. Pod '...' of job 'task-1234-ex-00' wasn't scheduled within 3600 s: 0/3 nodes are available: 3 Insufficient cpu., and stops. The TESK API then reports SYSTEM_ERROR:

• An executor's job is deleted, as its executor never ran: the task lacks an executor, which gives SYSTEM_ERROR rather than EXECUTOR_ERROR. This needs a TESK API release that derives the task state from all jobs of a task (tesk-api#104); TESK API 1.1 and earlier report a task whose later executor was deleted as COMPLETE, so don't set TESK_MAX_QUEUE_SECONDS with them.
• If a pod of the executor's job ran or failed before (e.g. a retry, with EXECUTOR_BACKOFF_LIMIT, whose new pod can't be scheduled), the job is failed and kept instead, with the logs of the pods that ran: the TESK API reports EXECUTOR_ERROR.
• A filer's job is failed and kept, as for pods that can't start; a failed filer gives SYSTEM_ERROR.

If the Kubernetes API rejects the deletion, the job is failed and kept instead; after other errors, including a request that times out after 30 seconds, the taskmaster tries again at the next poll, before anything else.

The pods are checked once the job has been active for longer than the shorter of the max queue time and --pod-timeout, then at most once a minute: with TESK_MAX_QUEUE_SECONDS=600 and --pod-timeout 240, from 240 seconds on. So a task can end up to about a minute later than the max queue time.

The max queue time covers the pods of the task's jobs only, not the taskmaster's own pod: a task whose taskmaster can't be scheduled waits as before. Neither does it cover pods that are never created, e.g. because a ResourceQuota of the namespace is used up.

Executor jobs aren't retried (backoffLimit: 0) unless EXECUTOR_BACKOFF_LIMIT (or the job) sets a limit, so that the state and logs are those of the executor's only run. The limit also covers pods that Kubernetes evicts, e.g. from a drained node: without retries, an evicted executor fails. TESK's Helm chart sets EXECUTOR_BACKOFF_LIMIT from tesk.executorRetries (2 by default); set it to 0 for single runs. The TESK API creates executor jobs with the restart policy Never; with OnFailure, Kubernetes would also restart failed containers within the pod.

To cancel a task, the TESK API labels the taskmaster's job and, if it runs, its pod with task-status: Cancelled. The taskmaster checks for this label before it creates the PVC, before the inputs filer, each executor and the outputs filer, and while it polls: on its pod, through /podinfo/labels (a Kubernetes downward API volume), and on its job, read from the Kubernetes API (the job's name is in the pod's job-name label). A task that was cancelled while its taskmaster's pod was pending, or before the pod existed, has the label only on its job; its taskmaster stops before it creates any job. When the taskmaster finds the label while a job runs, it deletes that job, also one that it is failing, and stops; when it finds it between jobs or before the first, it starts no other job, and deletes the task's jobs that are still running (by name, also those that an earlier attempt of its pod started), keeping the jobs that ended, with their logs. Each request to read or delete a job times out after 5 seconds to connect and 10 (read) or 30 (delete) seconds to answer, per attempt; the Kubernetes client tries up to 4 times. A job that is already gone counts as deleted, and other errors are logged: the taskmaster then deletes the PVC and stops as cancelled.

Reading the job adds one get request per check, which needs get on jobs for the taskmaster's service account (granted by the role of TESK's Helm chart, charts/tesk/templates/common/taskmaster-rbac.yaml). Each attempt of the request times out after 5 seconds to connect and 10 seconds to answer; the Kubernetes client tries up to 4 times. If the taskmaster can't read its job, also if the job doesn't exist, it logs an error (once; later ones only with --debug) and checks again at the next poll. Without /podinfo/labels (e.g. when run by hand), it doesn't check for cancellation at all.

On SIGINT (e.g. Ctrl-C when run by hand), the taskmaster deletes the jobs that it created and exits with code 130; a SIGTERM then doesn't cut that short. On SIGTERM, which Kubernetes sends when it stops the taskmaster's pod (e.g. when its job is deleted, or its node is drained), it logs an error and exits with code 143, and keeps the jobs (except for a cancelled task, see below): a new attempt of the pod, if the taskmaster's job has retries left, continues with them; otherwise they run until the taskmaster's job is deleted, by hand or after TASK_TTL_SECONDS, which deletes them with it (see Deleting finished tasks). Deleting them on SIGTERM would restart a running executor from scratch after, e.g., a node drain; a run by hand has no new attempt that would continue them.

A cancelled task is the exception: on SIGTERM, once it has read the task, the taskmaster checks for cancellation as above, and if the task was cancelled, it stops the task's running jobs, deletes the PVC (unless KEEP_FAILED_PVC keeps it) and exits with code 0, as when it finds the label while it runs. TESK API 1.2.0 and later also set activeDeadlineSeconds: 1 on the job of a task that they cancel, so that Kubernetes fails the job and stops its pod: a pod that started just before, without the cancel label, would otherwise leave its jobs running. The check and the deletions have to finish within the pod's termination grace period (30 seconds by default); Kubernetes kills the taskmaster after it, and jobs that aren't deleted yet keep running until the taskmaster's job is deleted.

Options

The TESK API sets these in the taskmaster's job; they are listed for development and debugging.

Option	Default	Description
JSON (positional) or -f, --file FILE		The task: as a JSON string, or as a file (gzipped if it ends with .gz; - for stdin). The TESK API passes /jsoninput/JSON_INPUT.gz.
-n, --namespace	default	Kubernetes namespace of the jobs and the PVC.
-fn, --filer-name	docker.io/elixircloud/tesk-core-filer	Image of the filer, without its tag.
-fv, --filer-version	latest	Tag of the filer image.
-p, --poll-interval	5	Seconds between polls of a job, from 0.1 to 3600 (e.g. 0.5).
-pt, --pod-timeout	240	Seconds after which a pending pod that can't start fails its job (see above).
-d, --debug	off	Debug logging, also of the filer; without it, the taskmaster logs info messages, warnings and errors, the filer only errors. The job manifests in the log show URLs without queries and user info, and hide the FTP credentials. The HTTP and S3 libraries log only warnings and errors, also with --debug.
--pull-policy-always	off	Pull the filer image on every start (imagePullPolicy: Always).
--localKubeConfig	off	Use the local kubeconfig instead of the in-cluster configuration (for running the taskmaster outside a cluster).

The TESK API always passes the filer image of its configuration; the defaults of --filer-name and --filer-version are for running the taskmaster by hand. latest is the newest release and changes with each one, so it need not match the taskmaster's version: pass the tag of the taskmaster's release (e.g. -fv 0.11.0) for matching and reproducible runs.

Environment variables

The TESK API passes each property tesk.api.taskmaster.environment.<name> of its configuration to the taskmaster as the environment variable <NAME> (upper case, dots as underscores); TESK's Helm chart sets them. The FTP credentials come from a Kubernetes secret (see Filer).

Variable	Description
EXECUTOR_BACKOFF_LIMIT	Retries of each executor's job (backoffLimit), from 0 to 2147483647; unset or empty, 0, unless the TESK API sets a limit in the job. Other values log an error and are ignored.
FILER_BACKOFF_LIMIT	Retries of each filer job (backoffLimit), from 0 to 2147483647; unset or empty, Kubernetes' default. Other values log an error and are ignored.
FILER_HTTP_TIMEOUT	Timeout of the filer's HTTP(S) requests (not FTP or S3), in seconds, for connecting and each read (see Filer); passed on to the filer; 300 if unset.
STORAGE_CLASS_NAME	Storage class of the task's PVC; the cluster's default if unset.
TESK_MAX_QUEUE_SECONDS	Seconds after which a pod that the scheduler can't place ends the task with SYSTEM_ERROR (see What it does), from 1 to 2147483647; unset or empty waits forever. Other values log an error and wait forever.
TASK_TTL_SECONDS	Seconds after which Kubernetes deletes the taskmaster's finished job, and with it the task's jobs (ttlSecondsAfterFinished; see Deleting finished tasks), from 0 to 2147483647; unset or empty keeps them. Other values log an error and set no TTL.
KEEP_FAILED_PVC	true keeps the PVC of a task that doesn't complete (a job failed, or the task was cancelled); unset, empty or false deletes it (see above). Other values log a warning and delete it.
FILER_POD_SECURITY_CONTEXT, FILER_SECURITY_CONTEXT	Security context of the filer's pod and of its container, as JSON objects; each replaces its whole default (repeat the default fields that you want to keep, see Pod security), and {} leaves it out. Unset or empty values keep the defaults; values that aren't JSON objects, or whose fields or types don't fit Kubernetes' PodSecurityContext or SecurityContext, log an error and keep them too.
FILER_RESOURCES	Resource requests and limits of the filer's container, as a JSON object (e.g. {"requests": {"cpu": "100m", "memory": "256Mi"}, "limits": {"memory": "1Gi"}}), with every quantity as a JSON string ("1", not 1); unset, empty or {} sets none. Values that aren't JSON objects, or whose fields or types don't fit Kubernetes' ResourceRequirements, log an error and set none.
TESK_FTP_USERNAME, TESK_FTP_PASSWORD	FTP credentials, passed on to the filer: as references to the secret TESK_FTP_SECRET_NAME if that is set, otherwise as values in the filer's job, with a warning.
TESK_FTP_SECRET_NAME	Name of the secret with the FTP credentials (keys username and password), which the filer's job then references instead of holding the credentials (see Filer).
TESK_FTP_HOSTS	Hosts that the filer sends the FTP credentials to, passed on to the filer with them (see Filer).
NETRC_SECRET_NAME	Name of a secret with a .netrc file (key .netrc), mounted into the filer for HTTP(S) and FTP credentials.
HOST_BASE_PATH, CONTAINER_BASE_PATH, TRANSFER_PVC_NAME	The shared volume of the file: protocol, passed on to the filer (see Filer).

The taskmaster also mounts the secret aws-secret into the filer, if it exists, for S3 (see Filer).

The taskmaster checks only the fields and types of the JSON settings (with Kubernetes clients from 37, as in the images), not their values: the API server rejects other invalid values (e.g. a CPU of lots, or a limit below its request) when the taskmaster creates a filer job, which fails every task with such a setting. Check a new setting with a test task.

Numbers may have spaces around them; an empty value counts as unset, so that a chart value left empty keeps the default. The names are the ones that TESK's Helm chart (0.3.0 and later) and the TESK API set; they are kept as they are, also the ones without the TESK_ prefix, as renaming them would break existing deployments.

Deleting finished tasks

The filer and executor jobs that the taskmaster creates are owned by the taskmaster's job (ownerReferences): when that job is deleted, Kubernetes deletes them and their pods too. The taskmaster takes the job's name and UID from its pod's labels (batch.kubernetes.io/job-name and batch.kubernetes.io/controller-uid, or job-name and controller-uid before Kubernetes 1.27); without them (e.g. outside a pod), the jobs get no owner. The references don't block the owner's deletion (blockOwnerDeletion isn't set), so they need no permission beyond creating jobs. To delete a task by hand, delete its taskmaster's job with kubectl delete job <task>, or through the API with the propagation policy Background or Foreground: a batch/v1 job deleted through the API without a policy leaves the jobs it owns behind.

By default, nothing deletes the jobs. With TASK_TTL_SECONDS set to a whole number of seconds, the taskmaster sets ttlSecondsAfterFinished on its own job when it starts, also for a task that is cancelled before it runs. Kubernetes then deletes the job that many seconds after it finished (completed or failed), and with it the task's jobs and pods. This needs patch on jobs; an error is logged, and the task runs without a TTL.

Some objects of a task aren't owned by its job, and stay:

• the ConfigMap with the task's JSON that the TESK API creates for the taskmaster (named after the task);
• a PVC that KEEP_FAILED_PVC keeps;
• the PVC of a task whose taskmaster failed unexpectedly (see What it does).

TESK keeps no database: the TESK API reads tasks from their jobs. A task whose jobs are deleted is gone: it no longer shows in ListTasks, and GetTask doesn't find it. Pick a TTL long enough for the clients of the TESK API (e.g. workflow engines) to read the final state and logs of their tasks: hours rather than seconds. With 0, a task disappears as soon as it finishes. While Kubernetes deletes a task's jobs, which can take a moment, the TESK API can report a task that completed as SYSTEM_ERROR, once some of its executor jobs are gone; a long TTL makes it unlikely that clients still read the task then.