diff --git a/docs/operators/services/funnel.md b/docs/operators/services/funnel.md index e09db69..fecc517 100644 --- a/docs/operators/services/funnel.md +++ b/docs/operators/services/funnel.md @@ -1,7 +1,8 @@ # Funnel [Funnel](https://github.com/calypr/funnel) is a [GA4GH TES][ga4gh-tes] -service. It runs tasks on a cluster through a backend, such as Slurm. +service. It runs tasks on a cluster through a backend, such as Slurm or +PBS. ## Role in the deployment @@ -16,14 +17,88 @@ deploy it, register it in the [Cloud Registry][elixir-cloud-registry] (see ## Notes for the deployment -- The ELIXIR On-Cloud documents Funnel with Slurm only. For other - backends, see the Funnel docs. -- Place the `funnel` binary on a server that has access to Slurm. -- Put the config file on the same server. [This file][funnel-config-slurm] - is a starting point. +- The ELIXIR On-Cloud documents Funnel with Slurm and with OpenPBS (below). + For other backends, such as HTCondor or Grid Engine, see the Funnel docs. +- Place the `funnel` binary on a server that can submit jobs to the + cluster: a login or submission node of Slurm or PBS. +- Put the config file on the same server. For Slurm, [this + file][funnel-config-slurm] is a starting point. - If everything works, Funnel listens on port `8080`. - Funnel's service info can't be set yet ([calypr/funnel#1493](https://github.com/calypr/funnel/issues/1493)), and this deployment refuses its environment, `development`: register it by hand (see [Manual entry, the fallback](../service-info.md#manual-entry-the-fallback)). + +## Funnel with OpenPBS + +Funnel's `pbs` backend submits each task as a job with `qsub` and cancels it +with `qdel` (see [PBS/Torque][funnel-pbs] in the Funnel docs). It renders +the job script from a template, with the variables `TaskId`, `WorkDir`, +`Cpus`, `RamGb`, `DiskGb` and `Zone` of the task, and `Args`, which points +the worker to a copy of Funnel's config in the work directory. The worker +in the job reports the state of its task to the Funnel server. + +1. Write the job script template, e.g., to `/etc/funnel/pbs-template.txt`. + The [template of the Funnel docs][funnel-pbs-template] uses the resource + syntax of PBS/Torque (`-l nodes=1:ppn=…`). OpenPBS requests resources by + chunks (`-l select=…`), so adapt it, e.g.: + + ```text + #!/bin/bash + #PBS -N {{.TaskId}} + #PBS -o {{.WorkDir}}/funnel-stdout + #PBS -e {{.WorkDir}}/funnel-stderr + #PBS -l select=1:ncpus={{if ne .Cpus 0}}{{.Cpus}}{{else}}1{{end}}{{if ne .RamGb 0.0}}{{printf ":mem=%.0fgb" .RamGb}}{{end}} + + funnel worker run {{.Args}} --taskID {{.TaskId}} + ``` + + Add the queue (`#PBS -q …`) and the account or project options that + your cluster requires. Mind what the template leaves out or rounds: + + - **Disk:** OpenPBS has no standard resource for the free disk space of + a job, so the template ignores `DiskGb`. + - **Zone:** it ignores `Zone`, the zone that a task asks for, too. The + Funnel docs suggest it as a queue name (e.g., `{{if .Zone}}#PBS -q + {{.Zone}}{{end}}`); add that only if your queues should be chosen by + the users of your service. + - **Memory:** OpenPBS takes whole units, and templates can't round up: + `%.0f` rounds `RamGb` to the nearest GB, so a task that asks for + 1.4 GB gets 1 GB, and one that asks for less than 0.5 GB gets + `mem=0gb`. If tasks of your users ask for fractions, set a default + memory per job in the queue, or add a margin in the template. + + The `funnel` binary must be on the `PATH` of the jobs on the execution + hosts too, e.g., on a shared file system. + +2. Point Funnel's config file at it, and keep the reconciler off, as it is + by default (see the warning below): + + ```yaml + Compute: pbs + PBS: + TemplateFile: /etc/funnel/pbs-template.txt + DisableReconciler: true + ``` + +3. Put Funnel's work directory (`WorkDir` in the config file), which also + holds the job's `funnel-stdout` and `funnel-stderr` and the config for + the worker, on a file system that the submission host and the execution + hosts share: the worker of a task runs on an execution host and needs + the task's files there. + +!!! warning "Not tested here" + These steps follow the Funnel docs, Funnel's code and the OpenPBS + resource syntax; they haven't been tested on an OpenPBS cluster for + these docs. Funnel's backend was written for PBS/Torque. Its reconciler, + which marks a task as failed when its job fails without the worker + reporting it, reads the XML output of `qstat -x` of Torque; in OpenPBS, + `qstat -x` lists finished jobs instead, in plain text. With the + reconciler off, a task whose job fails that way (e.g., killed for + exceeding its limits) may keep its last state: check such jobs with the + tools of OpenPBS. Please report what works on your cluster to the + [admins][elixir-on-cloud-email]. + +[funnel-pbs]: +[funnel-pbs-template]: diff --git a/docs/operators/services/index.md b/docs/operators/services/index.md index 8f28783..6e83046 100644 --- a/docs/operators/services/index.md +++ b/docs/operators/services/index.md @@ -14,7 +14,7 @@ Every Node runs one of these in front of its cluster (see |---|---|---| | [TESK](tesk.md) | [GA4GH TES][ga4gh-tes] | Kubernetes, OpenShift | | [Poiesis](poiesis.md) | [GA4GH TES][ga4gh-tes] | Kubernetes | -| [Funnel](funnel.md) | [GA4GH TES][ga4gh-tes] | HPC or HTC (Slurm) | +| [Funnel](funnel.md) | [GA4GH TES][ga4gh-tes] | HPC or HTC (Slurm, OpenPBS) | ## Other services diff --git a/docs/operators/services/poiesis.md b/docs/operators/services/poiesis.md index 8240638..1e62078 100644 --- a/docs/operators/services/poiesis.md +++ b/docs/operators/services/poiesis.md @@ -22,6 +22,19 @@ Registry][elixir-cloud-registry] (see [Onboarding](../onboarding.md)). it describes the project, not your Node. Register it by hand (see [Manual entry, the fallback](../service-info.md#manual-entry-the-fallback)). +- Poiesis reads the inputs of tasks from, and writes their outputs to, S3 + and HTTP(S) URLs, but not FTP (see [Usage][poiesis-usage] in its docs). A + Node with Poiesis needs an S3 server, e.g., MinIO (see [Deploy + storage](../onboarding.md#2-deploy-storage)). +- Poiesis validates access tokens of one OIDC issuer, but doesn't check the + groups of a user yet: the check for the LS Login group `ECP_CLN`, which the + policy of the federation requires, is tracked in + [generic-issues#21][gi-21]. + !!! warning "Under construction" - It is not documented yet how Poiesis is used in the ELIXIR On-Cloud - (for example, which Nodes run it, and with which storage). + Which Nodes of the ELIXIR On-Cloud run Poiesis is recorded with the + configuration of the gateway, in [generic-issues#31][gi-31]. + +[poiesis-usage]: +[gi-21]: +[gi-31]: diff --git a/docs/operators/services/protes.md b/docs/operators/services/protes.md index 7fa2f8f..41f4295 100644 --- a/docs/operators/services/protes.md +++ b/docs/operators/services/protes.md @@ -21,5 +21,9 @@ that a new Node is used once it is registered and verified. - [Plugins](https://protes.readthedocs.io/en/latest/plugins/) !!! warning "Under construction" - The address of the ELIXIR On-Cloud's proTES, its plugins and the - registry that it reads are not documented yet. + The address of the ELIXIR On-Cloud's proTES, its plugins and whether it + reads its TES services from the Cloud Registry are not documented yet. + Its configuration (client, TES services and how it authenticates at + them) is tracked in [generic-issues#31][gi-31]. + +[gi-31]: diff --git a/docs/operators/using-cloud-registry.md b/docs/operators/using-cloud-registry.md index ca4e5fe..aab7e15 100644 --- a/docs/operators/using-cloud-registry.md +++ b/docs/operators/using-cloud-registry.md @@ -76,8 +76,44 @@ of the service.** It says nothing about the service as it is now. here,** as drift can't be avoided. Admins verify services; you can't verify your own. Changing the URL or the API -type of a service removes its verification. See [Verified -services][cr-verified]. +type of a service removes its verification: ask again after such a change. See +[Verified services][cr-verified]. + +### What the admins check + +The registry leaves the checks to each deployment. Before they verify an entry +of the ELIXIR On-Cloud, the admins check with you, its owner, that: + +- **They know you:** you run the service, or act for those who do. +- **The values are right:** the URL, the API type and the organization of the + entry are those of your service, and the URL starts with `https`. +- **The service is live:** it answers at its URL, and its service info is + public, with sync on (see above). +- **The service isn't open to anyone:** it accepts requests only from the + users that the policy of the federation allows, e.g., for a TES service, + members of the LS Login group `ECP_CLN` (see [Enable LS Login and ECP_CLN + access](node-onboarding-tutorial.md#3-enable-ls-login-and-ecp_cln-access)), + and from the gateway. A service that can't check the group itself meets + this only if it accepts nothing but the credentials of the gateway, which + checks the group: e.g., Poiesis accepts any valid token of its issuer, as + its group check is still missing ([generic-issues#21][gi-21]). + Verification doesn't prove that a service enforces the policy itself. + +To ask for verification, write to +[cloud-service@elixir-europe.org][elixir-on-cloud-email] with the address of +your entry. + +### Who uses the verified entries + +- **proTES**, the gateway, when it discovers its TES services in a Cloud + Registry: it uses verified entries only (see [Get added behind the + gateway](node-onboarding-tutorial.md#7-get-added-behind-the-gateway)). + Whether the ELIXIR On-Cloud's proTES discovers its TES services this way, + or lists them in its configuration, is tracked in + [generic-issues#31][gi-31]. +- **Users**, who check the label before they trust a service. Krini and the + CLI don't show it: they list verified and unverified services alike (see + [Krini](../researchers/krini.md) and the [CLI](../researchers/cli.md)). ## Labels @@ -158,3 +194,5 @@ UI](using-cloud-registry-ui.md) adds what applies to this deployment. [cr-ui-browsing]: [cr-ui-user-guide]: [cr-verified]: +[gi-21]: +[gi-31]: