diff --git a/docs/tutorials/tes-services/interactive_services_tes.md b/docs/tutorials/tes-services/interactive_services_tes.md index 99ca766..2000f45 100644 --- a/docs/tutorials/tes-services/interactive_services_tes.md +++ b/docs/tutorials/tes-services/interactive_services_tes.md @@ -17,10 +17,11 @@ explanations. [cloud-service@elixir-europe.org][elixir-on-cloud-email]. !!! danger "Use synthetic data only" - The TESK of ELIXIR-FI has no authentication. Anyone can read a task with - `?view=FULL`, including the passwords of the hub and of the database that - the scripts put into it. RStudio, JupyterLab and Atlas have no login. - Don't use real or sensitive data with this setup. + Anyone who may read your tasks on the TES service sees, with + `?view=FULL`, the passwords of the hub and of the database that the + scripts put into them. On a service without tokens, that is everyone. + RStudio, JupyterLab and Atlas have no login. Don't use real or sensitive + data with this setup. ## How it works @@ -60,6 +61,7 @@ In the TES task: | |---|---| | [`tes-hub/tes-hub.sh`](tes-hub/tes-hub.sh) | Runs the hub on the virtual machine. | | [`tes-hub/common.sh`](tes-hub/common.sh) | Runs first in every task. Downloads chisel and opens the tunnel. | +| [`tes-hub/tes-api.sh`](tes-hub/tes-api.sh) | Sends the requests of the launchers to the TES service, with your access token. | | [`tes-hub/tes-service.sh`](tes-hub/tes-service.sh) | Runs a service that fits into one container image, from a directory with a `service.conf` and a `pod.sh`. | | [`rstudio-tesk/service.conf`](rstudio-tesk/service.conf), [`rstudio-tesk/pod.sh`](rstudio-tesk/pod.sh) | RStudio Server, with the image `rocker/rstudio`. | | [`omop-tesk/omop-tesk-fi.sh`](omop-tesk/omop-tesk-fi.sh) | Runs the Atlas stack as three tasks, which reach each other through the hub. | @@ -72,8 +74,8 @@ passwords (readable only by you), the task IDs and the logs. ## What you need -- A Linux virtual machine (x86-64 or ARM64) with `bash`, `curl`, `python3`, - `ss`, `setsid` and `flock`, that you can reach with SSH. The services run on the TES service, not on +- A Linux virtual machine (x86-64 or ARM64) with `bash`, `curl` (7.55 or + newer), `python3`, `ss`, `setsid` and `flock`, that you can reach with SSH. The services run on the TES service, not on the virtual machine. - A public host name for the virtual machine (`` below), and port 443 open to the internet. The tasks must reach this port: the TESK of @@ -81,9 +83,17 @@ passwords (readable only by you), the task IDs and the logs. - nginx and a TLS certificate for `` on the virtual machine, for example from [Let's Encrypt](https://letsencrypt.org/) with [Certbot](https://certbot.eff.org/). The hub itself speaks plain HTTP. -- A TES service whose tasks can make outbound HTTPS connections. The scripts - use the TESK of ELIXIR-FI (`https://csc-tesk-noauth.rahtiapp.fi`). To use - another service, set `TES_URL`. +- A TES service whose tasks can make outbound HTTPS connections, and an + access token for it if it needs one (see [Get an access + token](#get-an-access-token)). The scripts use the TESK of ELIXIR-FI + without tokens (`https://csc-tesk-noauth.rahtiapp.fi`). To use another + service, set `TES_URL`. + +!!! note "The default address will change" + The TES services of ELIXIR On-Cloud move to access with tokens. The + default `TES_URL` will change to the address of the TESK of ELIXIR-FI + with tokens once that is published. Until then, set `TES_URL` to the + address that you were given. ## Set up the hub @@ -148,6 +158,51 @@ The output must contain `proxy route OK`. The hub answers requests without a tunnel with `Not found`, so `curl https:///tes-tunnel/` must return exactly this text. +## Get an access token + +The TES services of ELIXIR On-Cloud accept only requests with an access +token of [LS Login](https://www.lifescience-ri.eu/life-science-login/). To +run tasks, your LS Login account must be a member of the ELIXIR On-Cloud +compute group `ECP_CLN:EOC-Compute` (members of `ECP_CLN:ADMIN` may run tasks +too). To join the group, write to +[cloud-service@elixir-europe.org][elixir-on-cloud-email]. + +1. Get an access token of your LS Login account from the web interface of + the deployment that you use, or as [Using the + APIs](../../partners-and-developers/apis.md#2-get-an-access-token) + describes. A token lives for about 4 hours. +2. Set it in the shell (bash) in which you run the scripts. With + `read -s`, the token is neither shown nor saved in the shell history: + + ```bash + read -rsp 'TES token: ' TES_TOKEN && echo && export TES_TOKEN + ``` + +If `TES_TOKEN` is set, the scripts send it with every request to the TES +service, in the header `Authorization: Bearer`, and only over HTTPS: set +`TES_URL` only to a TES service that you trust with the token. They pass it to `curl` +through a pipe, so that it doesn't appear on a command line, and they don't +write it to files or logs. They give it to their keep-alive loop only, not +to other programs such as the hub. Without `TES_TOKEN`, they send no token, +for TES services that don't need one. + +Each command that uses the TES service first checks the token, and ends +with an error if the service refuses it. Only `down` goes on: it stops the +keep-alive loop and locks the old tasks out of the hub, then warns that it +couldn't cancel them. Run `cancel` once you have a new token, before `up`: +otherwise `up` waits for the old task, which can't connect any more. + +| Answer | Meaning | What to do | +|---|---|---| +| `401 Unauthorized` | No token, or the token is invalid or has expired. | Get a new token and set `TES_TOKEN` again. | +| `403 Forbidden` | The token is valid, but your account may not run tasks. | Ask for membership in `ECP_CLN:EOC-Compute`. | + +The keep-alive loop keeps the token that it started with, in its +environment. When the token has expired, the loop writes the error to its +log and sends no new tasks, but running tasks go on. Get a new token, set +`TES_TOKEN` and run `up` again: with `TES_TOKEN` set, `up` replaces the +loop, so that it uses the new token. + ## Start RStudio Server ```bash @@ -240,9 +295,9 @@ For example: `./tes-hub/tes-service.sh rstudio-tesk status` or `./omop-tesk/omop-tesk-fi.sh logs webapi`. `./tes-hub/tes-hub.sh status` shows the hub, the path in nginx and all tunnels. -After a reboot of the virtual machine, start the hub, then run `up` for each -service. Running tasks reconnect by themselves, and `up` starts the -keep-alive loop again. +After a reboot of the virtual machine, start the hub, set `TES_TOKEN`, then +run `up` for each service. Running tasks reconnect by themselves, and `up` +starts the keep-alive loop again. ## Limits @@ -268,7 +323,8 @@ To change a default, set the environment variable before you run a script: | Variable | Default | Used by | |---|---|---| -| `TES_URL` | `https://csc-tesk-noauth.rahtiapp.fi` | all scripts | +| `TES_URL` | `https://csc-tesk-noauth.rahtiapp.fi` (will change, see [What you need](#what-you-need)) | all scripts | +| `TES_TOKEN` | none: no token is sent (see [Get an access token](#get-an-access-token)) | all scripts | | `HUB_LISTEN_PORT` | `8090` (`setup` saves it) | `tes-hub.sh` | | `CHISEL_VERSION` | `1.12.0` | `tes-hub.sh` | | `WATCH_INTERVAL` | `60` (seconds) | the keep-alive loops | @@ -279,6 +335,11 @@ To change a default, set the environment variable before you run a script: ## Troubleshooting +- **`401 Unauthorized` or `403 Forbidden`:** the TES service refused the + token, see [Get an access token](#get-an-access-token). If the keep-alive + loop stops replacing a service after a few hours, its token has expired: + its log `.state/watch.log` shows the error. Set a new token and run `up` + again. - **`proxy route not working`:** the hub doesn't run (start it with `./tes-hub/tes-hub.sh start`), or the path is missing in nginx. If `curl https:///tes-tunnel/` returns HTML or a redirect, the request @@ -331,6 +392,18 @@ nginx: We did not run the changed scripts on the TESK of ELIXIR-FI, and did not test nginx with TLS or JupyterLab. +We tested the access tokens against a small mock of a TES service that +required tokens, with a stand-in for the hub: without a token, with an +invalid one and with one outside the compute group, every command ended +after its first request with the matching error. With a valid token, +`selftest`, `up`, `status`, `logs`, `print` and `down` worked for both +launchers, `up` replaced the keep-alive loop, and the loop logged the error +when the token expired. With the expired token, `down` still stopped the +loop and locked the tasks out. The token appeared in no command line, file, +log or environment of the hub. + +We did not test the tokens against a real TES service. + ## Where to go next - [Interactive Services on TES: Commands](interactive_services_tes_commands.md) diff --git a/docs/tutorials/tes-services/interactive_services_tes_commands.md b/docs/tutorials/tes-services/interactive_services_tes_commands.md index b6ca783..9c9e4a9 100644 --- a/docs/tutorials/tes-services/interactive_services_tes_commands.md +++ b/docs/tutorials/tes-services/interactive_services_tes_commands.md @@ -18,6 +18,10 @@ mkdir -p ~/eoc_services/{tes-hub,omop-tesk/pod,rstudio-tesk} && cd ~/eoc_service cat > tes-hub/tes-hub.sh <<'__TES_SERVICES_EOF__' --8<-- "docs/tutorials/tes-services/tes-hub/tes-hub.sh" __TES_SERVICES_EOF__ +# requests to the TES service, with the access token in TES_TOKEN +cat > tes-hub/tes-api.sh <<'__TES_SERVICES_EOF__' +--8<-- "docs/tutorials/tes-services/tes-hub/tes-api.sh" +__TES_SERVICES_EOF__ # code sent into every task: downloads chisel, opens the tunnel cat > tes-hub/common.sh <<'__TES_SERVICES_EOF__' --8<-- "docs/tutorials/tes-services/tes-hub/common.sh" @@ -85,6 +89,19 @@ location /tes-tunnel/ { sudo systemctl reload nginx ``` +## Set the access token + +An LS Login access token, of an account in the compute group, lives for +about 4 hours: see [Get an access +token](interactive_services_tes.md#get-an-access-token). + +```bash +# paste an LS Login access token (account in ECP_CLN:EOC-Compute); not shown, not in the history +read -rsp 'TES token: ' TES_TOKEN && echo && export TES_TOKEN +# check it: lists one task, or curl fails (error 22) with 401 (invalid or expired token) or 403 (not in the compute group) +printf 'Authorization: Bearer %s\n' "$TES_TOKEN" | curl -q -fsS -H @- "${TES_URL:-https://csc-tesk-noauth.rahtiapp.fi}/ga4gh/tes/v1/tasks?view=MINIMAL&page_size=1" +``` + ## Start Atlas with OMOP demo data ```bash @@ -185,14 +202,19 @@ cd ~/eoc_services # RStudio: show the TES request ./tes-hub/tes-service.sh rstudio-tesk print +# token expired (after about 4 hours, 401 in .state/watch.log): set a new token as in +# "Set the access token", then run up again, which gives the keep-alive loop the new token +./omop-tesk/omop-tesk-fi.sh up +./tes-hub/tes-service.sh rstudio-tesk up + # JupyterLab: status / logs / restart / down ./tes-hub/tes-service.sh jupyter-tesk status ./tes-hub/tes-service.sh jupyter-tesk logs ./tes-hub/tes-service.sh jupyter-tesk restart ./tes-hub/tes-service.sh jupyter-tesk down -# list all tasks on TESK @ ELIXIR-FI -curl -fsS "https://csc-tesk-noauth.rahtiapp.fi/ga4gh/tes/v1/tasks?view=MINIMAL" +# list all tasks on the TES service (the token goes through stdin, not the command line) +printf 'Authorization: Bearer %s\n' "$TES_TOKEN" | curl -q -fsS -H @- "${TES_URL:-https://csc-tesk-noauth.rahtiapp.fi}/ga4gh/tes/v1/tasks?view=MINIMAL" ``` ## After a reboot @@ -200,7 +222,7 @@ curl -fsS "https://csc-tesk-noauth.rahtiapp.fi/ga4gh/tes/v1/tasks?view=MINIMAL" ```bash # start the hub again; running tasks reconnect within ~30 seconds ./tes-hub/tes-hub.sh start -# restart the keep-alive loops of the single-container services +# restart the keep-alive loops of the single-container services (set the access token first) ./tes-hub/tes-service.sh rstudio-tesk up ./tes-hub/tes-service.sh jupyter-tesk up ``` @@ -208,8 +230,9 @@ curl -fsS "https://csc-tesk-noauth.rahtiapp.fi/ga4gh/tes/v1/tasks?view=MINIMAL" ## Settings ```bash -# TES instance: TESK @ ELIXIR-FI (default) +# TES instance: TESK @ ELIXIR-FI (default; will change to the address with tokens) export TES_URL=https://csc-tesk-noauth.rahtiapp.fi +# access token: see "Set the access token" (never as TES_TOKEN=... on the command line) # hub port and chisel version (defaults) export HUB_LISTEN_PORT=8090 CHISEL_VERSION=1.12.0 # Atlas stack images (defaults) diff --git a/docs/tutorials/tes-services/omop-tesk/omop-tesk-fi.sh b/docs/tutorials/tes-services/omop-tesk/omop-tesk-fi.sh index 2c16dc7..968ce13 100755 --- a/docs/tutorials/tes-services/omop-tesk/omop-tesk-fi.sh +++ b/docs/tutorials/tes-services/omop-tesk/omop-tesk-fi.sh @@ -25,11 +25,17 @@ # ./omop-tesk-fi.sh selftest db|atlas # short task without tunnel; fails unless SELFTEST_OK # ./omop-tesk-fi.sh print # print the TES request JSON # -# SECURITY: csc-tesk-noauth is unauthenticated. Anyone can read a task with ?view=FULL, -# including the hub credentials and DB password in it. The hub only lets those credentials -# open the three ports above, and the DB holds synthetic demo data only. Don't use real data. +# TES_TOKEN: if set, every request to the TES carries it as a bearer token (see +# ../tes-hub/tes-api.sh). The keep-alive loop keeps the token it started with: run up +# again with a new token to replace the loop. +# +# SECURITY: whoever can read a task with ?view=FULL (on a TES without tokens: anyone) sees +# the hub credentials and DB password in it. The hub only lets those credentials open the +# three ports above, and the DB holds synthetic demo data only. Don't use real data. set -euo pipefail +# Keep the access token from the programs that this script runs (see tes-api.sh). +if [[ -n "${TES_TOKEN:-}" ]]; then export -n TES_TOKEN; fi HERE="$(cd "$(dirname "$0")" && pwd)" STATE_DIR="${STATE_DIR:-$HERE/.state}" @@ -38,7 +44,8 @@ mkdir -p "$STATE_DIR"; chmod 700 "$STATE_DIR" [[ -f "$STATE_DIR/config" ]] && source "$STATE_DIR/config" # ---- configuration ----------------------------------------------------------------------- -TES_URL="${TES_URL:-https://csc-tesk-noauth.rahtiapp.fi}" # TESK @ ELIXIR-FI +# TESK @ ELIXIR-FI. To be updated: the address of the instance with token access. +TES_URL="${TES_URL:-https://csc-tesk-noauth.rahtiapp.fi}" TES_API="${TES_URL%/}/ga4gh/tes/v1" HUB="${HUB:-$HERE/../tes-hub/tes-hub.sh}" # shared tunnel hub HUB_ATLAS_PORT="${HUB_ATLAS_PORT:-8080}" # -> http://localhost:8080 @@ -65,10 +72,12 @@ ROLES=(db webapi atlas) die() { echo "ERROR: $*" >&2; exit 1; } need() { command -v "$1" >/dev/null || die "'$1' is required"; } need curl; need python3 +# shellcheck source=SCRIPTDIR/../tes-hub/tes-api.sh +source "$(dirname "$HUB")/tes-api.sh" json_get() { python3 -c 'import json,sys; d=json.load(sys.stdin); print(eval(sys.argv[1], {"d": d}))' "$1"; } id_file() { echo "$STATE_DIR/task_$1"; } task_id() { [[ -s "$(id_file "$1")" ]] && cat "$(id_file "$1")" || true; } -task_state() { curl -fsS -m 30 "$TES_API/tasks/$1?view=MINIMAL" | json_get 'd["state"]'; } +task_state() { tes_curl -m 30 "$TES_API/tasks/$1?view=MINIMAL" | json_get 'd["state"]'; } WATCH_PID="$STATE_DIR/watch.pid" # The pid must still be this stack's loop: after a reboot, another process may have it. watching() { @@ -138,7 +147,7 @@ PY submit_role() { local role="$1" resp id [[ "$("$HUB" status)" == *"hub running"* ]] || die "hub not running (run: setsid -f $HUB start)" - resp="$(build_task "$role" | curl -fsS --connect-timeout 10 -m 60 -X POST -H 'Content-Type: application/json' \ + resp="$(build_task "$role" | tes_curl --connect-timeout 10 -m 60 -X POST -H 'Content-Type: application/json' \ --data-binary @- "$TES_API/tasks")" || die "submitting $role to $TES_API failed" id="$(json_get 'd["id"]' <<<"$resp")" echo "$id" > "$(id_file "$role")" @@ -147,7 +156,7 @@ submit_role() { cancel_role() { local id; id="$(task_id "$1")" - [[ -n "$id" ]] && curl -fsS -m 30 -X POST "$TES_API/tasks/$id:cancel" >/dev/null && echo "$1: canceled $id" || true + [[ -n "$id" ]] && tes_curl -m 30 -X POST "$TES_API/tasks/$id:cancel" >/dev/null && echo "$1: canceled $id" || true } roles_arg() { if [[ -z "${1:-}" || "$1" == all ]]; then echo "${ROLES[@]}"; else check_role "$1"; echo "$1"; fi; } @@ -202,8 +211,10 @@ cmd_up() { else submit_role "$r"; fi done unlock + # The loop keeps the token it started with: replace it, so that it gets the current one. + if watching && [[ -n "${TES_TOKEN:-}" ]]; then lock; stop_watch; unlock; fi if ! watching; then - setsid -f "$HERE/omop-tesk-fi.sh" watch >> "$STATE_DIR/watch.log" 2>&1 < /dev/null + TES_TOKEN="${TES_TOKEN:-}" setsid -f "$HERE/omop-tesk-fi.sh" watch >> "$STATE_DIR/watch.log" 2>&1 < /dev/null echo "keep-alive loop started (log: $STATE_DIR/watch.log)" fi echo "waiting for Atlas, WebAPI and the data source (first start 10-15 minutes; Ctrl-C is safe, the tasks keep running)" @@ -225,19 +236,31 @@ cmd_up() { # a hub restart lock them out. All three roles share the hub user, so this replaces the stack. lock_out() { rm -f "$STATE_DIR/config"; HUB_AUTH="" DB_PASSWORD=""; cmd_setup; "$HUB" restart; } +# Stops the keep-alive loop and waits until it has ended, so that it can't remove the pid +# file of a new loop. +stop_watch() { + local pid i; pid="$(cat "$WATCH_PID")" + kill "$pid" || return 0 + for ((i = 0; i < 1500; i++)); do kill -0 "$pid" 2>/dev/null || break; sleep 0.1; done + ! kill -0 "$pid" 2>/dev/null || die "keep-alive loop $pid still running, try again later" + rm -f "$WATCH_PID"; echo "keep-alive loop stopped" +} + # down: stop the loop first (it would resubmit), lock the pods out before cancelling, so that -# an interrupted down never leaves them connected. +# an interrupted down never leaves them connected. Cancelling needs the TES. cmd_down() { - if watching; then kill "$(cat "$WATCH_PID")" && rm -f "$WATCH_PID" && echo "keep-alive loop stopped"; fi lock + if watching; then stop_watch; fi lock_out - local r; for r in "${ROLES[@]}"; do cancel_role "$r"; done + local r + if tes_check; then for r in "${ROLES[@]}"; do cancel_role "$r"; done + else echo "WARNING: tasks not canceled; run: $0 cancel" >&2; fi unlock } cmd_logs() { local id; check_role "${1:-}"; id="$(task_id "$1")"; [[ -n "$id" ]] || die "$1 not submitted" - curl -fsS -m 60 "$TES_API/tasks/$id?view=FULL" | json_get \ + tes_curl -m 60 "$TES_API/tasks/$id?view=FULL" | json_get \ '"state: " + d["state"] + "\n" + "\n".join((l.get("stdout") or "") + (l.get("stderr") or "") for t in d.get("logs",[]) for l in t.get("logs",[]))' } @@ -245,7 +268,7 @@ cmd_selftest() { local role="${1:-}" resp id state i out [[ "$role" == db || "$role" == atlas ]] || die "usage: $0 selftest db|atlas" [[ -n "$HUB_AUTH" && -n "$DB_PASSWORD" ]] || cmd_setup - resp="$(build_task "$role" selftest | curl -fsS --connect-timeout 10 -m 60 -X POST \ + resp="$(build_task "$role" selftest | tes_curl --connect-timeout 10 -m 60 -X POST \ -H 'Content-Type: application/json' --data-binary @- "$TES_API/tasks")" || die "submitting to $TES_API failed" id="$(json_get 'd["id"]' <<<"$resp")"; echo "selftest $role: $id" for ((i = 0; i < 90; i++)); do @@ -253,7 +276,7 @@ cmd_selftest() { case "$state" in COMPLETE|EXECUTOR_ERROR|SYSTEM_ERROR|CANCELED) break ;; esac sleep 10 done - out="$(curl -fsS -m 60 "$TES_API/tasks/$id?view=FULL" | json_get \ + out="$(tes_curl -m 60 "$TES_API/tasks/$id?view=FULL" | json_get \ '"state: " + d["state"] + "\n" + "\n".join((l.get("stdout") or "") + (l.get("stderr") or "") for t in d.get("logs",[]) for l in t.get("logs",[]))')" echo "$out" [[ "$out" == "state: COMPLETE"* && "$out" == *SELFTEST_OK* ]] || die "selftest $role failed" @@ -271,10 +294,12 @@ replace_stack() { cmd_watch() { echo $$ > "$WATCH_PID" - trap 'rm -f "$WATCH_PID"' EXIT + trap '[[ "$(cat "$WATCH_PID" 2>/dev/null)" != "$$" ]] || rm -f "$WATCH_PID"' EXIT echo "$(date -Is) keeping OMOP tasks alive on $TES_URL (every ${WATCH_INTERVAL}s)" local r id state while true; do + # A rejected or expired token: log why (tes_check), don't resubmit. + if ! tes_check; then sleep "$WATCH_INTERVAL"; continue; fi lock for r in "${ROLES[@]}"; do id="$(task_id "$r")"; state=NONE @@ -296,13 +321,20 @@ cmd_watch() { done } +# Commands that use the TES fail early if it rejects the token (or its absence). down +# locks the pods out first, and needs the TES only to cancel. +case "${1:-}" in + up|submit|logs|cancel|selftest) tes_check || die "cannot use the TES at $TES_URL" ;; + status) tes_check || TES_REFUSED=1 ;; +esac + case "${1:-}" in up) cmd_up ;; down) cmd_down ;; - restart) cmd_down; cmd_up ;; + restart) tes_check || die "cannot use the TES at $TES_URL"; cmd_down; cmd_up ;; setup) cmd_setup ;; submit) lock; for r in $(roles_arg "${2:-}"); do submit_role "$r"; done; unlock ;; - status) cmd_status ;; + status) cmd_status; [[ -z "${TES_REFUSED:-}" ]] ;; logs) cmd_logs "${2:-}" ;; cancel) for r in $(roles_arg "${2:-}"); do cancel_role "$r"; done ;; watch) cmd_watch ;; diff --git a/docs/tutorials/tes-services/tes-hub/tes-api.sh b/docs/tutorials/tes-services/tes-hub/tes-api.sh new file mode 100644 index 0000000..9ce6b9a --- /dev/null +++ b/docs/tutorials/tes-services/tes-hub/tes-api.sh @@ -0,0 +1,46 @@ +# shellcheck shell=bash +# Requests to the GA4GH TES API, sourced by the launchers on this VM (tes-service.sh, +# ../omop-tesk/omop-tesk-fi.sh). They set TES_URL and TES_API first. +# +# If TES_TOKEN is set, every request carries "Authorization: Bearer $TES_TOKEN". The +# header reaches curl through a pipe that the builtin printf writes (process +# substitution), so the token never appears on a command line, in a file or in a log. + +# The token stays a shell variable: not exported, so that no child process (such as the +# hub, which runs for days) gets it. The launchers pass it to their keep-alive loop only. +if [[ -n "${TES_TOKEN:-}" ]]; then export -n TES_TOKEN; fi + +# tes_curl : prints the body of a 2xx answer. Otherwise prints why to +# stderr and returns 1. +tes_curl() { + local out code + if [[ -n "${TES_TOKEN:-}" ]]; then + # A token goes over HTTPS only (plain HTTP only to this VM, e.g. for tests). + [[ "$TES_URL" == https://* || "$TES_URL" =~ ^http://(127\.0\.0\.1|localhost)(:[0-9]+)?(/|$) ]] \ + || { echo "ERROR: TES_URL must start with https:// when TES_TOKEN is set" >&2; return 1; } + # -q first: ignore ~/.curlrc, which could turn on verbose output with the header. + out="$(curl -q -sS -w '\n%{http_code}' -H @<(printf 'Authorization: Bearer %s\n' "$TES_TOKEN") "$@")" || return 1 + else + out="$(curl -q -sS -w '\n%{http_code}' "$@")" || return 1 + fi + code="${out##*$'\n'}"; out="${out%$'\n'*}" + case "$code" in + 2??) printf '%s\n' "$out" ;; + 401) + if [[ -n "${TES_TOKEN:-}" ]]; then + echo "ERROR: $TES_URL answered 401 Unauthorized: the token in TES_TOKEN is invalid or has expired (after about 4 hours). Get a new LS Login access token, set TES_TOKEN again and rerun the command" >&2 + else + echo "ERROR: $TES_URL answered 401 Unauthorized: it needs a token. Get an LS Login access token and set TES_TOKEN (see the tutorial)" >&2 + fi + return 1 ;; + 403) + echo "ERROR: $TES_URL answered 403 Forbidden: the token is valid, but your account may not run tasks there. On the TES services of ELIXIR On-Cloud, you need membership in the compute group ECP_CLN:EOC-Compute of LS Login" >&2 + return 1 ;; + *) + echo "ERROR: $TES_URL answered HTTP ${code:-without status}: ${out:0:300}" >&2 + return 1 ;; + esac +} + +# tes_check: fails early, with the reason, if the TES doesn't let us list tasks. +tes_check() { tes_curl -m 30 "$TES_API/tasks?view=MINIMAL&page_size=1" >/dev/null; } diff --git a/docs/tutorials/tes-services/tes-hub/tes-hub.sh b/docs/tutorials/tes-services/tes-hub/tes-hub.sh index 012f8ad..8308188 100755 --- a/docs/tutorials/tes-services/tes-hub/tes-hub.sh +++ b/docs/tutorials/tes-services/tes-hub/tes-hub.sh @@ -22,6 +22,8 @@ # survive a VM reboot: run start again. set -euo pipefail +# The hub runs for days and needs no access token of the TES: don't pass one on to it. +unset TES_TOKEN HERE="$(cd "$(dirname "$0")" && pwd)" STATE_DIR="${HUB_STATE_DIR:-$HERE/.state}" diff --git a/docs/tutorials/tes-services/tes-hub/tes-service.sh b/docs/tutorials/tes-services/tes-hub/tes-service.sh index fc72b52..e69d94e 100755 --- a/docs/tutorials/tes-services/tes-hub/tes-service.sh +++ b/docs/tutorials/tes-services/tes-hub/tes-service.sh @@ -20,8 +20,14 @@ # selftest short task without tunnel (pod.sh sees SELFTEST=1); fails unless SELFTEST_OK # print the TES request JSON # setup | submit | cancel | watch single steps, used by up/down +# +# TES_TOKEN: if set, every request to the TES carries it as a bearer token (see +# tes-api.sh). The keep-alive loop keeps the token it started with: run up again with a +# new token to replace the loop. set -euo pipefail +# Keep the access token from the programs that this script runs (see tes-api.sh). +if [[ -n "${TES_TOKEN:-}" ]]; then export -n TES_TOKEN; fi HERE="$(cd "$(dirname "$0")" && pwd)" [[ $# -ge 2 ]] || { sed -n '2,/^set -euo/p' "$0" | sed '$d; s/^# \{0,1\}//'; exit 1; } @@ -37,7 +43,8 @@ source "$DIR/service.conf" CONF_KEYS="$(sed -n 's/^[[:space:]]*\([A-Za-z_][A-Za-z0-9_]*\)=.*/\1/p' "$DIR/service.conf")" # ---- configuration ----------------------------------------------------------------------- -TES_URL="${TES_URL:-https://csc-tesk-noauth.rahtiapp.fi}" # TESK @ ELIXIR-FI +# TESK @ ELIXIR-FI. To be updated: the address of the instance with token access. +TES_URL="${TES_URL:-https://csc-tesk-noauth.rahtiapp.fi}" TES_API="${TES_URL%/}/ga4gh/tes/v1" HUB="${HUB:-$HERE/tes-hub.sh}" # shared tunnel hub : "${NAME:?NAME missing in service.conf}" "${IMAGE:?IMAGE missing in service.conf}" "${PORT:?PORT missing in service.conf}" @@ -51,12 +58,14 @@ WATCH_INTERVAL="${WATCH_INTERVAL:-60}" # ------------------------------------------------------------------------------------------ die() { echo "ERROR: $*" >&2; exit 1; } +# shellcheck source=SCRIPTDIR/tes-api.sh +source "$HERE/tes-api.sh" json_get() { python3 -c 'import json,sys; d=json.load(sys.stdin); print(eval(sys.argv[1], {"d": d}))' "$1"; } ID_FILE="$STATE_DIR/task" WATCH_PID="$STATE_DIR/watch.pid" task_id() { [[ -s "$ID_FILE" ]] && cat "$ID_FILE" || true; } need_id() { [[ -s "$ID_FILE" ]] || die "not submitted"; cat "$ID_FILE"; } -task_state() { curl -fsS -m 30 "$TES_API/tasks/$1?view=MINIMAL" | json_get 'd["state"]'; } +task_state() { tes_curl -m 30 "$TES_API/tasks/$1?view=MINIMAL" | json_get 'd["state"]'; } current_state() { local id; id="$(task_id)"; [[ -n "$id" ]] && { task_state "$id" 2>/dev/null || echo UNREACHABLE; } || echo NONE; } is_done() { [[ "$1" =~ ^(NONE|COMPLETE|EXECUTOR_ERROR|SYSTEM_ERROR|CANCELED|PREEMPTED)$ ]]; } tunnel_up() { [[ "$(ss -tln 2>/dev/null)" == *"127.0.0.1:$HUB_PORT "* ]]; } @@ -76,7 +85,7 @@ build_task() { local HUB_URL CHISEL_URL HUB_FINGERPRINT; eval "$("$HUB" env)" # service.conf values (minus the launcher's own keys) become the pod's environment. for k in $CONF_KEYS PORT HUB_PORT; do - [[ $k =~ ^(NAME|IMAGE|POD_SHELL|RAM_GB|CPU_CORES|TES_URL)$ ]] || printf '%s\0%s\0' "$k" "${!k}" + [[ $k =~ ^(NAME|IMAGE|POD_SHELL|RAM_GB|CPU_CORES|TES_URL|TES_TOKEN)$ ]] || printf '%s\0%s\0' "$k" "${!k}" done > "$STATE_DIR/pod.env" export HUB_AUTH HUB_URL CHISEL_URL HUB_FINGERPRINT NAME IMAGE POD_SHELL RAM_GB CPU_CORES SELFTEST="${selftest:+1}" ENV_FILE="$STATE_DIR/pod.env" \ @@ -100,7 +109,7 @@ PY } post_task() { - build_task "$@" | curl -fsS --connect-timeout 10 -m 60 -X POST -H 'Content-Type: application/json' \ + build_task "$@" | tes_curl --connect-timeout 10 -m 60 -X POST -H 'Content-Type: application/json' \ --data-binary @- "$TES_API/tasks" | json_get 'd["id"]' } @@ -131,7 +140,7 @@ cmd_submit() { cmd_cancel() { local id; id="$(task_id)" - [[ -n "$id" ]] && curl -fsS -m 30 -X POST "$TES_API/tasks/$id:cancel" >/dev/null && echo "canceled $id" || true + [[ -n "$id" ]] && tes_curl -m 30 -X POST "$TES_API/tasks/$id:cancel" >/dev/null && echo "canceled $id" || true } cmd_status() { @@ -148,16 +157,20 @@ cmd_selftest() { is_done "$(task_state "$id" 2>/dev/null || echo UNKNOWN)" && break sleep 10 done - out="$(curl -fsS -m 60 "$TES_API/tasks/$id?view=FULL" | print_logs)"; echo "$out" + out="$(tes_curl -m 60 "$TES_API/tasks/$id?view=FULL" | print_logs)"; echo "$out" [[ "$out" == "state: COMPLETE"* && "$out" == *SELFTEST_OK* ]] || die "selftest failed" } cmd_watch() { echo $$ > "$WATCH_PID" - trap 'rm -f "$WATCH_PID"' EXIT + trap '[[ "$(cat "$WATCH_PID" 2>/dev/null)" != "$$" ]] || rm -f "$WATCH_PID"' EXIT + # SIGTERM (stop_watch) ends the loop between two rounds only, never inside lock_out. + local stop=""; trap 'stop=1' TERM echo "$(date -Is) keeping $NAME alive on $TES_URL (every ${WATCH_INTERVAL}s)" local state while true; do + # A rejected or expired token: log why (tes_check), don't resubmit. + if ! tes_check; then sleep "$WATCH_INTERVAL" & wait $! || true; [[ -z "$stop" ]] || exit 0; continue; fi state="$(current_state)" if is_done "$state"; then echo "$(date -Is) task $(task_id) is $state, resubmitting" @@ -165,7 +178,9 @@ cmd_watch() { if tunnel_up; then echo "$(date -Is) port $HUB_PORT still taken, locking the old pod out"; lock_out || true; fi (cmd_submit) || true fi - sleep "$WATCH_INTERVAL" + [[ -z "$stop" ]] || exit 0 + sleep "$WATCH_INTERVAL" & wait $! || true + [[ -z "$stop" ]] || exit 0 done } @@ -173,8 +188,10 @@ cmd_up() { [[ -n "$HUB_AUTH" ]] || cmd_setup local state i; state="$(current_state)" if is_done "$state"; then cmd_submit; else echo "task $(task_id) is $state, not resubmitting"; fi + # The loop keeps the token it started with: replace it, so that it gets the current one. + if watching && [[ -n "${TES_TOKEN:-}" ]]; then stop_watch; fi if ! watching; then - setsid -f "$HERE/tes-service.sh" "$DIR" watch >> "$STATE_DIR/watch.log" 2>&1 < /dev/null + TES_TOKEN="${TES_TOKEN:-}" setsid -f "$HERE/tes-service.sh" "$DIR" watch >> "$STATE_DIR/watch.log" 2>&1 < /dev/null echo "keep-alive loop started (log: $STATE_DIR/watch.log)" fi echo "waiting for the tunnel (usually 1-3 minutes; Ctrl-C is safe, the task keeps running)" @@ -185,22 +202,41 @@ cmd_up() { echo "and open http://localhost:$HUB_PORT" } +# Stops the keep-alive loop and waits until it has ended (it finishes a round it has +# begun), so that it can't remove the pid file of a new loop. +stop_watch() { + local pid i; pid="$(cat "$WATCH_PID")" + kill "$pid" || return 0 + for ((i = 0; i < 1500; i++)); do kill -0 "$pid" 2>/dev/null || break; sleep 0.1; done + ! kill -0 "$pid" 2>/dev/null || die "keep-alive loop $pid still running, try again later" + rm -f "$WATCH_PID"; echo "keep-alive loop stopped" +} + cmd_down() { - if watching; then kill "$(cat "$WATCH_PID")" && rm -f "$WATCH_PID" && echo "keep-alive loop stopped"; fi + if watching; then stop_watch; fi # Cancel marks the task CANCELED but the pod keeps running: lock it out of the hub first, # so that an interrupted down never leaves the old pod connected. lock_out - cmd_cancel + # The pods are locked out already; cancelling needs the TES. + if tes_check; then cmd_cancel + elif [[ -n "$(task_id)" ]]; then echo "WARNING: task $(task_id) not canceled; run: $SELF cancel" >&2; fi } +# Commands that use the TES fail early if it rejects the token (or its absence). down +# locks the pods out first, and needs the TES only to cancel. +case "$CMD" in + up|submit|logs|cancel|selftest) tes_check || die "cannot use the TES at $TES_URL" ;; + status) tes_check || TES_REFUSED=1 ;; +esac + case "$CMD" in up) cmd_up ;; down) cmd_down ;; - restart) cmd_down; cmd_up ;; + restart) tes_check || die "cannot use the TES at $TES_URL"; cmd_down; cmd_up ;; setup) cmd_setup ;; submit) cmd_submit ;; - status) cmd_status ;; - logs) id="$(need_id)"; curl -fsS -m 60 "$TES_API/tasks/$id?view=FULL" | print_logs ;; + status) cmd_status; [[ -z "${TES_REFUSED:-}" ]] ;; + logs) id="$(need_id)"; tes_curl -m 60 "$TES_API/tasks/$id?view=FULL" | print_logs ;; cancel) cmd_cancel ;; watch) cmd_watch ;; selftest) cmd_selftest ;;