docs: fill gaps of the Node Operators pages

docs/operators/services/funnel.md

@@ -1,7 +1,8 @@
 # Funnel
 
 [Funnel](https://github.com/calypr/funnel) is a [GA4GH TES][ga4gh-tes]
-service. It runs tasks on a cluster through a backend, such as Slurm.
+service. It runs tasks on a cluster through a backend, such as Slurm or
+PBS.
 
 ## Role in the deployment
 
@@ -16,14 +17,88 @@ deploy it, register it in the [Cloud Registry][elixir-cloud-registry] (see
 
 ## Notes for the deployment
 
-- The ELIXIR On-Cloud documents Funnel with Slurm only. For other
-  backends, see the Funnel docs.
-- Place the `funnel` binary on a server that has access to Slurm.
-- Put the config file on the same server. [This file][funnel-config-slurm]
-  is a starting point.
+- The ELIXIR On-Cloud documents Funnel with Slurm and with OpenPBS (below).
+  For other backends, such as HTCondor or Grid Engine, see the Funnel docs.
+- Place the `funnel` binary on a server that can submit jobs to the
+  cluster: a login or submission node of Slurm or PBS.
+- Put the config file on the same server. For Slurm, [this
+  file][funnel-config-slurm] is a starting point.
 - If everything works, Funnel listens on port `8080`.
 - Funnel's service info can't be set yet
   ([calypr/funnel#1493](https://github.com/calypr/funnel/issues/1493)), and
   this deployment refuses its environment, `development`: register it by
   hand (see [Manual entry, the
   fallback](../service-info.md#manual-entry-the-fallback)).
+
+## Funnel with OpenPBS
+
+Funnel's `pbs` backend submits each task as a job with `qsub` and cancels it
+with `qdel` (see [PBS/Torque][funnel-pbs] in the Funnel docs). It renders
+the job script from a template, with the variables `TaskId`, `WorkDir`,
+`Cpus`, `RamGb`, `DiskGb` and `Zone` of the task, and `Args`, which points
+the worker to a copy of Funnel's config in the work directory. The worker
+in the job reports the state of its task to the Funnel server.
+
+1. Write the job script template, e.g., to `/etc/funnel/pbs-template.txt`.
+   The [template of the Funnel docs][funnel-pbs-template] uses the resource
+   syntax of PBS/Torque (`-l nodes=1:ppn=…`). OpenPBS requests resources by
+   chunks (`-l select=…`), so adapt it, e.g.:
+
+    ```text
+    #!/bin/bash
+    #PBS -N {{.TaskId}}
+    #PBS -o {{.WorkDir}}/funnel-stdout
+    #PBS -e {{.WorkDir}}/funnel-stderr
+    #PBS -l select=1:ncpus={{if ne .Cpus 0}}{{.Cpus}}{{else}}1{{end}}{{if ne .RamGb 0.0}}{{printf ":mem=%.0fgb" .RamGb}}{{end}}
+
+    funnel worker run {{.Args}} --taskID {{.TaskId}}
+    ```
+
+    Add the queue (`#PBS -q …`) and the account or project options that
+    your cluster requires. Mind what the template leaves out or rounds:
+
+    - **Disk:** OpenPBS has no standard resource for the free disk space of
+      a job, so the template ignores `DiskGb`.
+    - **Zone:** it ignores `Zone`, the zone that a task asks for, too. The
+      Funnel docs suggest it as a queue name (e.g., `{{if .Zone}}#PBS -q
+      {{.Zone}}{{end}}`); add that only if your queues should be chosen by
+      the users of your service.
+    - **Memory:** OpenPBS takes whole units, and templates can't round up:
+      `%.0f` rounds `RamGb` to the nearest GB, so a task that asks for
+      1.4 GB gets 1 GB, and one that asks for less than 0.5 GB gets
+      `mem=0gb`. If tasks of your users ask for fractions, set a default
+      memory per job in the queue, or add a margin in the template.
+
+    The `funnel` binary must be on the `PATH` of the jobs on the execution
+    hosts too, e.g., on a shared file system.
+
+2. Point Funnel's config file at it, and keep the reconciler off, as it is
+   by default (see the warning below):
+
+    ```yaml
+    Compute: pbs
+    PBS:
+      TemplateFile: /etc/funnel/pbs-template.txt
+      DisableReconciler: true
+    ```
+
+3. Put Funnel's work directory (`WorkDir` in the config file), which also
+   holds the job's `funnel-stdout` and `funnel-stderr` and the config for
+   the worker, on a file system that the submission host and the execution
+   hosts share: the worker of a task runs on an execution host and needs
+   the task's files there.
+
+!!! warning "Not tested here"
+    These steps follow the Funnel docs, Funnel's code and the OpenPBS
+    resource syntax; they haven't been tested on an OpenPBS cluster for
+    these docs. Funnel's backend was written for PBS/Torque. Its reconciler,
+    which marks a task as failed when its job fails without the worker
+    reporting it, reads the XML output of `qstat -x` of Torque; in OpenPBS,
+    `qstat -x` lists finished jobs instead, in plain text. With the
+    reconciler off, a task whose job fails that way (e.g., killed for
+    exceeding its limits) may keep its last state: check such jobs with the
+    tools of OpenPBS. Please report what works on your cluster to the
+    [admins][elixir-on-cloud-email].
+
+[funnel-pbs]: <https://github.com/calypr/funnel/blob/v0.12.3/docs/compute/pbs-torque.md>
+[funnel-pbs-template]: <https://github.com/calypr/funnel/blob/v0.12.3/website/static/funnel-config-examples/pbs-template.txt>

docs/operators/services/index.md

@@ -14,7 +14,7 @@ Every Node runs one of these in front of its cluster (see
 |---|---|---|
 | [TESK](tesk.md) | [GA4GH TES][ga4gh-tes] | Kubernetes, OpenShift |
 | [Poiesis](poiesis.md) | [GA4GH TES][ga4gh-tes] | Kubernetes |
-| [Funnel](funnel.md) | [GA4GH TES][ga4gh-tes] | HPC or HTC (Slurm) |
+| [Funnel](funnel.md) | [GA4GH TES][ga4gh-tes] | HPC or HTC (Slurm, OpenPBS) |
 
 ## Other services
 

docs/operators/services/poiesis.md

@@ -22,6 +22,19 @@ Registry][elixir-cloud-registry] (see [Onboarding](../onboarding.md)).
   it describes the project, not your Node. Register it by hand (see [Manual
   entry, the fallback](../service-info.md#manual-entry-the-fallback)).
 
+- Poiesis reads the inputs of tasks from, and writes their outputs to, S3
+  and HTTP(S) URLs, but not FTP (see [Usage][poiesis-usage] in its docs). A
+  Node with Poiesis needs an S3 server, e.g., MinIO (see [Deploy
+  storage](../onboarding.md#2-deploy-storage)).
+- Poiesis validates access tokens of one OIDC issuer, but doesn't check the
+  groups of a user yet: the check for the LS Login group `ECP_CLN`, which the
+  policy of the federation requires, is tracked in
+  [generic-issues#21][gi-21].
+
 !!! warning "Under construction"
-    It is not documented yet how Poiesis is used in the ELIXIR On-Cloud
-    (for example, which Nodes run it, and with which storage).
+    Which Nodes of the ELIXIR On-Cloud run Poiesis is recorded with the
+    configuration of the gateway, in [generic-issues#31][gi-31].
+
+[poiesis-usage]: <https://poiesis.jaeaeich.com/docs/usage/usage.html>
+[gi-21]: <https://github.com/elixir-cloud-aai/generic-issues/issues/21>
+[gi-31]: <https://github.com/elixir-cloud-aai/generic-issues/issues/31>

docs/operators/services/protes.md

@@ -21,5 +21,9 @@ that a new Node is used once it is registered and verified.
 - [Plugins](https://protes.readthedocs.io/en/latest/plugins/)
 
 !!! warning "Under construction"
-    The address of the ELIXIR On-Cloud's proTES, its plugins and the
-    registry that it reads are not documented yet.
+    The address of the ELIXIR On-Cloud's proTES, its plugins and whether it
+    reads its TES services from the Cloud Registry are not documented yet.
+    Its configuration (client, TES services and how it authenticates at
+    them) is tracked in [generic-issues#31][gi-31].
+
+[gi-31]: <https://github.com/elixir-cloud-aai/generic-issues/issues/31>

docs/operators/using-cloud-registry.md

@@ -76,8 +76,44 @@ of the service.** It says nothing about the service as it is now.
 here,** as drift can't be avoided.
 
 Admins verify services; you can't verify your own. Changing the URL or the API
-type of a service removes its verification. See [Verified
-services][cr-verified].
+type of a service removes its verification: ask again after such a change. See
+[Verified services][cr-verified].
+
+### What the admins check
+
+The registry leaves the checks to each deployment. Before they verify an entry
+of the ELIXIR On-Cloud, the admins check with you, its owner, that:
+
+- **They know you:** you run the service, or act for those who do.
+- **The values are right:** the URL, the API type and the organization of the
+  entry are those of your service, and the URL starts with `https`.
+- **The service is live:** it answers at its URL, and its service info is
+  public, with sync on (see above).
+- **The service isn't open to anyone:** it accepts requests only from the
+  users that the policy of the federation allows, e.g., for a TES service,
+  members of the LS Login group `ECP_CLN` (see [Enable LS Login and ECP_CLN
+  access](node-onboarding-tutorial.md#3-enable-ls-login-and-ecp_cln-access)),
+  and from the gateway. A service that can't check the group itself meets
+  this only if it accepts nothing but the credentials of the gateway, which
+  checks the group: e.g., Poiesis accepts any valid token of its issuer, as
+  its group check is still missing ([generic-issues#21][gi-21]).
+  Verification doesn't prove that a service enforces the policy itself.
+
+To ask for verification, write to
+[cloud-service@elixir-europe.org][elixir-on-cloud-email] with the address of
+your entry.
+
+### Who uses the verified entries
+
+- **proTES**, the gateway, when it discovers its TES services in a Cloud
+  Registry: it uses verified entries only (see [Get added behind the
+  gateway](node-onboarding-tutorial.md#7-get-added-behind-the-gateway)).
+  Whether the ELIXIR On-Cloud's proTES discovers its TES services this way,
+  or lists them in its configuration, is tracked in
+  [generic-issues#31][gi-31].
+- **Users**, who check the label before they trust a service. Krini and the
+  CLI don't show it: they list verified and unverified services alike (see
+  [Krini](../researchers/krini.md) and the [CLI](../researchers/cli.md)).
 
 ## Labels
 
@@ -158,3 +194,5 @@ UI](using-cloud-registry-ui.md) adds what applies to this deployment.
 [cr-ui-browsing]: <https://cloud-registry-ui.readthedocs.io/en/v1.0.0-rc2/user-guide/browsing/>
 [cr-ui-user-guide]: <https://cloud-registry-ui.readthedocs.io/en/v1.0.0-rc2/user-guide/>
 [cr-verified]: <https://cloud-registry.readthedocs.io/en/latest/api/#verified-services>
+[gi-21]: <https://github.com/elixir-cloud-aai/generic-issues/issues/21>
+[gi-31]: <https://github.com/elixir-cloud-aai/generic-issues/issues/31>