From 6988850c6541b1c55ac01d19e0a7a9ba2bfb8085 Mon Sep 17 00:00:00 2001 From: Alex Kanitz Date: Fri, 13 Nov 2026 14:10:45 +0100 Subject: [PATCH 1/2] docs: fill gaps of the Node Operators pages Document what the registry admins check before they verify a service and which services use the verified entries, Funnel with OpenPBS, the storage and token support of Poiesis, and point the proTES note to the issue that tracks its configuration. Part of #81. Co-Authored-By: Claude Opus 5.5 --- docs/operators/services/funnel.md | 69 +++++++++++++++++++++++--- docs/operators/services/index.md | 2 +- docs/operators/services/poiesis.md | 17 ++++++- docs/operators/services/protes.md | 8 ++- docs/operators/using-cloud-registry.md | 37 +++++++++++++- 5 files changed, 120 insertions(+), 13 deletions(-) diff --git a/docs/operators/services/funnel.md b/docs/operators/services/funnel.md index e09db69..e3e86f0 100644 --- a/docs/operators/services/funnel.md +++ b/docs/operators/services/funnel.md @@ -1,7 +1,8 @@ # Funnel [Funnel](https://github.com/calypr/funnel) is a [GA4GH TES][ga4gh-tes] -service. It runs tasks on a cluster through a backend, such as Slurm. +service. It runs tasks on a cluster through a backend, such as Slurm or +PBS. ## Role in the deployment @@ -16,14 +17,70 @@ deploy it, register it in the [Cloud Registry][elixir-cloud-registry] (see ## Notes for the deployment -- The ELIXIR On-Cloud documents Funnel with Slurm only. For other - backends, see the Funnel docs. -- Place the `funnel` binary on a server that has access to Slurm. -- Put the config file on the same server. [This file][funnel-config-slurm] - is a starting point. +- The ELIXIR On-Cloud documents Funnel with Slurm and with OpenPBS (below). + For other backends, such as HTCondor or Grid Engine, see the Funnel docs. +- Place the `funnel` binary on a server that can submit jobs to the + cluster: a login or submission node of Slurm or PBS. +- Put the config file on the same server. For Slurm, [this + file][funnel-config-slurm] is a starting point. - If everything works, Funnel listens on port `8080`. - Funnel's service info can't be set yet ([calypr/funnel#1493](https://github.com/calypr/funnel/issues/1493)), and this deployment refuses its environment, `development`: register it by hand (see [Manual entry, the fallback](../service-info.md#manual-entry-the-fallback)). + +## Funnel with OpenPBS + +Funnel's `pbs` backend submits each task as a job with `qsub` and cancels it +with `qdel` (see [PBS/Torque][funnel-pbs] in the Funnel docs). It renders +the job script from a template, with the variables `TaskId`, `WorkDir`, +`Cpus`, `RamGb`, `DiskGb` and `Zone` of the task. The worker in the job +reports the state of its task to the Funnel server. + +1. Write the job script template, e.g., to `/etc/funnel/pbs-template.txt`. + The [template of the Funnel docs][funnel-pbs-template] uses the resource + syntax of PBS/Torque (`-l nodes=1:ppn=…`). OpenPBS requests resources by + chunks (`-l select=…`), so adapt it, e.g.: + + ```text + #!/bin/bash + #PBS -N {{.TaskId}} + #PBS -o {{.WorkDir}}/funnel-stdout + #PBS -e {{.WorkDir}}/funnel-stderr + #PBS -l select=1:ncpus={{if ne .Cpus 0}}{{.Cpus}}{{else}}1{{end}}{{if ne .RamGb 0.0}}{{printf ":mem=%.0fgb" .RamGb}}{{end}} + + funnel worker run --config /etc/funnel/funnel-config.yml --taskID {{.TaskId}} + ``` + + Add the queue (`#PBS -q …`) and the account or project options that + your cluster requires. OpenPBS has no standard resource for the free + disk space of a job, so the template above ignores `DiskGb`. + +2. Point Funnel's config file at it, and keep the reconciler off, as it is + by default (see the warning below): + + ```yaml + Compute: pbs + PBS: + TemplateFile: /etc/funnel/pbs-template.txt + DisableReconciler: true + ``` + +3. Pass the same config file to the workers, as in the template above, so + that they reach the Funnel server and the storage of the Node. + +!!! warning "Not tested here" + These steps follow the Funnel docs, Funnel's code and the OpenPBS + resource syntax; they haven't been tested on an OpenPBS cluster for + these docs. Funnel's backend was written for PBS/Torque. Its reconciler, + which marks a task as failed when its job fails without the worker + reporting it, reads the XML output of `qstat -x` of Torque; in OpenPBS, + `qstat -x` lists finished jobs instead, in plain text. With the + reconciler off, a task whose job fails that way (e.g., killed for + exceeding its limits) may keep its last state: check such jobs with the + tools of OpenPBS. Please report what works on your cluster to the + [admins][elixir-on-cloud-email]. + +[funnel-pbs]: +[funnel-pbs-template]: diff --git a/docs/operators/services/index.md b/docs/operators/services/index.md index 8f28783..6e83046 100644 --- a/docs/operators/services/index.md +++ b/docs/operators/services/index.md @@ -14,7 +14,7 @@ Every Node runs one of these in front of its cluster (see |---|---|---| | [TESK](tesk.md) | [GA4GH TES][ga4gh-tes] | Kubernetes, OpenShift | | [Poiesis](poiesis.md) | [GA4GH TES][ga4gh-tes] | Kubernetes | -| [Funnel](funnel.md) | [GA4GH TES][ga4gh-tes] | HPC or HTC (Slurm) | +| [Funnel](funnel.md) | [GA4GH TES][ga4gh-tes] | HPC or HTC (Slurm, OpenPBS) | ## Other services diff --git a/docs/operators/services/poiesis.md b/docs/operators/services/poiesis.md index 8240638..1e62078 100644 --- a/docs/operators/services/poiesis.md +++ b/docs/operators/services/poiesis.md @@ -22,6 +22,19 @@ Registry][elixir-cloud-registry] (see [Onboarding](../onboarding.md)). it describes the project, not your Node. Register it by hand (see [Manual entry, the fallback](../service-info.md#manual-entry-the-fallback)). +- Poiesis reads the inputs of tasks from, and writes their outputs to, S3 + and HTTP(S) URLs, but not FTP (see [Usage][poiesis-usage] in its docs). A + Node with Poiesis needs an S3 server, e.g., MinIO (see [Deploy + storage](../onboarding.md#2-deploy-storage)). +- Poiesis validates access tokens of one OIDC issuer, but doesn't check the + groups of a user yet: the check for the LS Login group `ECP_CLN`, which the + policy of the federation requires, is tracked in + [generic-issues#21][gi-21]. + !!! warning "Under construction" - It is not documented yet how Poiesis is used in the ELIXIR On-Cloud - (for example, which Nodes run it, and with which storage). + Which Nodes of the ELIXIR On-Cloud run Poiesis is recorded with the + configuration of the gateway, in [generic-issues#31][gi-31]. + +[poiesis-usage]: +[gi-21]: +[gi-31]: diff --git a/docs/operators/services/protes.md b/docs/operators/services/protes.md index 7fa2f8f..41f4295 100644 --- a/docs/operators/services/protes.md +++ b/docs/operators/services/protes.md @@ -21,5 +21,9 @@ that a new Node is used once it is registered and verified. - [Plugins](https://protes.readthedocs.io/en/latest/plugins/) !!! warning "Under construction" - The address of the ELIXIR On-Cloud's proTES, its plugins and the - registry that it reads are not documented yet. + The address of the ELIXIR On-Cloud's proTES, its plugins and whether it + reads its TES services from the Cloud Registry are not documented yet. + Its configuration (client, TES services and how it authenticates at + them) is tracked in [generic-issues#31][gi-31]. + +[gi-31]: diff --git a/docs/operators/using-cloud-registry.md b/docs/operators/using-cloud-registry.md index ca4e5fe..adf0e6e 100644 --- a/docs/operators/using-cloud-registry.md +++ b/docs/operators/using-cloud-registry.md @@ -76,8 +76,40 @@ of the service.** It says nothing about the service as it is now. here,** as drift can't be avoided. Admins verify services; you can't verify your own. Changing the URL or the API -type of a service removes its verification. See [Verified -services][cr-verified]. +type of a service removes its verification: ask again after such a change. See +[Verified services][cr-verified]. + +### What the admins check + +The registry leaves the checks to each deployment. Before they verify an entry +of the ELIXIR On-Cloud, the admins check with you, its owner, that: + +- **They know you:** you run the service, or act for those who do. +- **The values are right:** the URL, the API type and the organization of the + entry are those of your service, and the URL starts with `https`. +- **The service is live:** it answers at its URL, and its service info is + public, with sync on (see above). +- **The service isn't open to anyone:** it accepts requests only from the + users that the policy of the federation allows, e.g., for a TES service, + members of the LS Login group `ECP_CLN` (see [Enable LS Login and ECP_CLN + access](node-onboarding-tutorial.md#3-enable-ls-login-and-ecp_cln-access)), + and from the gateway. + +To ask for verification, write to +[cloud-service@elixir-europe.org][elixir-on-cloud-email] with the address of +your entry. + +### Who uses the verified entries + +- **proTES**, the gateway, when it discovers its TES services in a Cloud + Registry: it uses verified entries only (see [Get added behind the + gateway](node-onboarding-tutorial.md#7-get-added-behind-the-gateway)). + Whether the ELIXIR On-Cloud's proTES discovers its TES services this way, + or lists them in its configuration, is tracked in + [generic-issues#31][gi-31]. +- **Users**, who check the label before they trust a service. Krini and the + CLI don't show it: they list verified and unverified services alike (see + [Krini](../researchers/krini.md) and the [CLI](../researchers/cli.md)). ## Labels @@ -158,3 +190,4 @@ UI](using-cloud-registry-ui.md) adds what applies to this deployment. [cr-ui-browsing]: [cr-ui-user-guide]: [cr-verified]: +[gi-31]: From 367c9be84505580275a5b307fba75d98feb140b3 Mon Sep 17 00:00:00 2001 From: Alex Kanitz Date: Fri, 13 Nov 2026 14:51:14 +0100 Subject: [PATCH 2/2] docs: address review of the OpenPBS and checks Use Funnel's Args for the worker config, require a shared work directory and the binary on the execution hosts, say what the PBS template ignores or rounds, and say that a service without its own group check meets the access check only behind the gateway. Part of #81. Co-Authored-By: Claude Opus 5.5 --- docs/operators/services/funnel.md | 32 ++++++++++++++++++++------ docs/operators/using-cloud-registry.md | 7 +++++- 2 files changed, 31 insertions(+), 8 deletions(-) diff --git a/docs/operators/services/funnel.md b/docs/operators/services/funnel.md index e3e86f0..fecc517 100644 --- a/docs/operators/services/funnel.md +++ b/docs/operators/services/funnel.md @@ -35,8 +35,9 @@ deploy it, register it in the [Cloud Registry][elixir-cloud-registry] (see Funnel's `pbs` backend submits each task as a job with `qsub` and cancels it with `qdel` (see [PBS/Torque][funnel-pbs] in the Funnel docs). It renders the job script from a template, with the variables `TaskId`, `WorkDir`, -`Cpus`, `RamGb`, `DiskGb` and `Zone` of the task. The worker in the job -reports the state of its task to the Funnel server. +`Cpus`, `RamGb`, `DiskGb` and `Zone` of the task, and `Args`, which points +the worker to a copy of Funnel's config in the work directory. The worker +in the job reports the state of its task to the Funnel server. 1. Write the job script template, e.g., to `/etc/funnel/pbs-template.txt`. The [template of the Funnel docs][funnel-pbs-template] uses the resource @@ -50,12 +51,26 @@ reports the state of its task to the Funnel server. #PBS -e {{.WorkDir}}/funnel-stderr #PBS -l select=1:ncpus={{if ne .Cpus 0}}{{.Cpus}}{{else}}1{{end}}{{if ne .RamGb 0.0}}{{printf ":mem=%.0fgb" .RamGb}}{{end}} - funnel worker run --config /etc/funnel/funnel-config.yml --taskID {{.TaskId}} + funnel worker run {{.Args}} --taskID {{.TaskId}} ``` Add the queue (`#PBS -q …`) and the account or project options that - your cluster requires. OpenPBS has no standard resource for the free - disk space of a job, so the template above ignores `DiskGb`. + your cluster requires. Mind what the template leaves out or rounds: + + - **Disk:** OpenPBS has no standard resource for the free disk space of + a job, so the template ignores `DiskGb`. + - **Zone:** it ignores `Zone`, the zone that a task asks for, too. The + Funnel docs suggest it as a queue name (e.g., `{{if .Zone}}#PBS -q + {{.Zone}}{{end}}`); add that only if your queues should be chosen by + the users of your service. + - **Memory:** OpenPBS takes whole units, and templates can't round up: + `%.0f` rounds `RamGb` to the nearest GB, so a task that asks for + 1.4 GB gets 1 GB, and one that asks for less than 0.5 GB gets + `mem=0gb`. If tasks of your users ask for fractions, set a default + memory per job in the queue, or add a margin in the template. + + The `funnel` binary must be on the `PATH` of the jobs on the execution + hosts too, e.g., on a shared file system. 2. Point Funnel's config file at it, and keep the reconciler off, as it is by default (see the warning below): @@ -67,8 +82,11 @@ reports the state of its task to the Funnel server. DisableReconciler: true ``` -3. Pass the same config file to the workers, as in the template above, so - that they reach the Funnel server and the storage of the Node. +3. Put Funnel's work directory (`WorkDir` in the config file), which also + holds the job's `funnel-stdout` and `funnel-stderr` and the config for + the worker, on a file system that the submission host and the execution + hosts share: the worker of a task runs on an execution host and needs + the task's files there. !!! warning "Not tested here" These steps follow the Funnel docs, Funnel's code and the OpenPBS diff --git a/docs/operators/using-cloud-registry.md b/docs/operators/using-cloud-registry.md index adf0e6e..aab7e15 100644 --- a/docs/operators/using-cloud-registry.md +++ b/docs/operators/using-cloud-registry.md @@ -93,7 +93,11 @@ of the ELIXIR On-Cloud, the admins check with you, its owner, that: users that the policy of the federation allows, e.g., for a TES service, members of the LS Login group `ECP_CLN` (see [Enable LS Login and ECP_CLN access](node-onboarding-tutorial.md#3-enable-ls-login-and-ecp_cln-access)), - and from the gateway. + and from the gateway. A service that can't check the group itself meets + this only if it accepts nothing but the credentials of the gateway, which + checks the group: e.g., Poiesis accepts any valid token of its issuer, as + its group check is still missing ([generic-issues#21][gi-21]). + Verification doesn't prove that a service enforces the policy itself. To ask for verification, write to [cloud-service@elixir-europe.org][elixir-on-cloud-email] with the address of @@ -190,4 +194,5 @@ UI](using-cloud-registry-ui.md) adds what applies to this deployment. [cr-ui-browsing]: [cr-ui-user-guide]: [cr-verified]: +[gi-21]: [gi-31]: