Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 86 additions & 13 deletions docs/tutorials/tes-services/interactive_services_tes.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,11 @@ explanations.
[cloud-service@elixir-europe.org][elixir-on-cloud-email].

!!! danger "Use synthetic data only"
The TESK of ELIXIR-FI has no authentication. Anyone can read a task with
`?view=FULL`, including the passwords of the hub and of the database that
the scripts put into it. RStudio, JupyterLab and Atlas have no login.
Don't use real or sensitive data with this setup.
Anyone who may read your tasks on the TES service sees, with
`?view=FULL`, the passwords of the hub and of the database that the
scripts put into them. On a service without tokens, that is everyone.
RStudio, JupyterLab and Atlas have no login. Don't use real or sensitive
data with this setup.

## How it works

Expand Down Expand Up @@ -60,6 +61,7 @@ In the TES task: |
|---|---|
| [`tes-hub/tes-hub.sh`](tes-hub/tes-hub.sh) | Runs the hub on the virtual machine. |
| [`tes-hub/common.sh`](tes-hub/common.sh) | Runs first in every task. Downloads chisel and opens the tunnel. |
| [`tes-hub/tes-api.sh`](tes-hub/tes-api.sh) | Sends the requests of the launchers to the TES service, with your access token. |
| [`tes-hub/tes-service.sh`](tes-hub/tes-service.sh) | Runs a service that fits into one container image, from a directory with a `service.conf` and a `pod.sh`. |
| [`rstudio-tesk/service.conf`](rstudio-tesk/service.conf), [`rstudio-tesk/pod.sh`](rstudio-tesk/pod.sh) | RStudio Server, with the image `rocker/rstudio`. |
| [`omop-tesk/omop-tesk-fi.sh`](omop-tesk/omop-tesk-fi.sh) | Runs the Atlas stack as three tasks, which reach each other through the hub. |
Expand All @@ -72,18 +74,26 @@ passwords (readable only by you), the task IDs and the logs.

## What you need

- A Linux virtual machine (x86-64 or ARM64) with `bash`, `curl`, `python3`,
`ss`, `setsid` and `flock`, that you can reach with SSH. The services run on the TES service, not on
- A Linux virtual machine (x86-64 or ARM64) with `bash`, `curl` (7.55 or
newer), `python3`, `ss`, `setsid` and `flock`, that you can reach with SSH. The services run on the TES service, not on
the virtual machine.
- A public host name for the virtual machine (`<HOST>` below), and port 443
open to the internet. The tasks must reach this port: the TESK of
ELIXIR-FI allows outbound HTTPS, and other ports were not tested.
- nginx and a TLS certificate for `<HOST>` on the virtual machine, for
example from [Let's Encrypt](https://letsencrypt.org/) with
[Certbot](https://certbot.eff.org/). The hub itself speaks plain HTTP.
- A TES service whose tasks can make outbound HTTPS connections. The scripts
use the TESK of ELIXIR-FI (`https://csc-tesk-noauth.rahtiapp.fi`). To use
another service, set `TES_URL`.
- A TES service whose tasks can make outbound HTTPS connections, and an
access token for it if it needs one (see [Get an access
token](#get-an-access-token)). The scripts use the TESK of ELIXIR-FI
without tokens (`https://csc-tesk-noauth.rahtiapp.fi`). To use another
Comment thread
uniqueg marked this conversation as resolved.
Resolved
service, set `TES_URL`.

!!! note "The default address will change"
The TES services of ELIXIR On-Cloud move to access with tokens. The
default `TES_URL` will change to the address of the TESK of ELIXIR-FI
with tokens once that is published. Until then, set `TES_URL` to the
address that you were given.

## Set up the hub

Expand Down Expand Up @@ -148,6 +158,51 @@ The output must contain `proxy route OK`. The hub answers requests without
a tunnel with `Not found`, so `curl https://<HOST>/tes-tunnel/` must return
exactly this text.

## Get an access token

The TES services of ELIXIR On-Cloud accept only requests with an access
token of [LS Login](https://www.lifescience-ri.eu/life-science-login/). To
run tasks, your LS Login account must be a member of the ELIXIR On-Cloud
compute group `ECP_CLN:EOC-Compute` (members of `ECP_CLN:ADMIN` may run tasks
too). To join the group, write to
[cloud-service@elixir-europe.org][elixir-on-cloud-email].

1. Get an access token of your LS Login account from the web interface of
the deployment that you use, or as [Using the
APIs](../../partners-and-developers/apis.md#2-get-an-access-token)
describes. A token lives for about 4 hours.
2. Set it in the shell (bash) in which you run the scripts. With
`read -s`, the token is neither shown nor saved in the shell history:

```bash
read -rsp 'TES token: ' TES_TOKEN && echo && export TES_TOKEN
```

If `TES_TOKEN` is set, the scripts send it with every request to the TES
service, in the header `Authorization: Bearer`, and only over HTTPS: set
`TES_URL` only to a TES service that you trust with the token. They pass it to `curl`
through a pipe, so that it doesn't appear on a command line, and they don't
write it to files or logs. They give it to their keep-alive loop only, not
to other programs such as the hub. Without `TES_TOKEN`, they send no token,
for TES services that don't need one.

Each command that uses the TES service first checks the token, and ends
with an error if the service refuses it. Only `down` goes on: it stops the
keep-alive loop and locks the old tasks out of the hub, then warns that it
couldn't cancel them. Run `cancel` once you have a new token, before `up`:
otherwise `up` waits for the old task, which can't connect any more.

| Answer | Meaning | What to do |
|---|---|---|
| `401 Unauthorized` | No token, or the token is invalid or has expired. | Get a new token and set `TES_TOKEN` again. |
| `403 Forbidden` | The token is valid, but your account may not run tasks. | Ask for membership in `ECP_CLN:EOC-Compute`. |

The keep-alive loop keeps the token that it started with, in its
environment. When the token has expired, the loop writes the error to its
log and sends no new tasks, but running tasks go on. Get a new token, set
`TES_TOKEN` and run `up` again: with `TES_TOKEN` set, `up` replaces the
loop, so that it uses the new token.

## Start RStudio Server

```bash
Expand Down Expand Up @@ -240,9 +295,9 @@ For example: `./tes-hub/tes-service.sh rstudio-tesk status` or
`./omop-tesk/omop-tesk-fi.sh logs webapi`. `./tes-hub/tes-hub.sh status`
shows the hub, the path in nginx and all tunnels.

After a reboot of the virtual machine, start the hub, then run `up` for each
service. Running tasks reconnect by themselves, and `up` starts the
keep-alive loop again.
After a reboot of the virtual machine, start the hub, set `TES_TOKEN`, then
run `up` for each service. Running tasks reconnect by themselves, and `up`
starts the keep-alive loop again.

## Limits

Expand All @@ -268,7 +323,8 @@ To change a default, set the environment variable before you run a script:

| Variable | Default | Used by |
|---|---|---|
| `TES_URL` | `https://csc-tesk-noauth.rahtiapp.fi` | all scripts |
| `TES_URL` | `https://csc-tesk-noauth.rahtiapp.fi` (will change, see [What you need](#what-you-need)) | all scripts |
| `TES_TOKEN` | none: no token is sent (see [Get an access token](#get-an-access-token)) | all scripts |
| `HUB_LISTEN_PORT` | `8090` (`setup` saves it) | `tes-hub.sh` |
| `CHISEL_VERSION` | `1.12.0` | `tes-hub.sh` |
| `WATCH_INTERVAL` | `60` (seconds) | the keep-alive loops |
Expand All @@ -279,6 +335,11 @@ To change a default, set the environment variable before you run a script:

## Troubleshooting

- **`401 Unauthorized` or `403 Forbidden`:** the TES service refused the
token, see [Get an access token](#get-an-access-token). If the keep-alive
loop stops replacing a service after a few hours, its token has expired:
its log `.state/watch.log` shows the error. Set a new token and run `up`
again.
- **`proxy route not working`:** the hub doesn't run (start it with
`./tes-hub/tes-hub.sh start`), or the path is missing in nginx. If
`curl https://<HOST>/tes-tunnel/` returns HTML or a redirect, the request
Expand Down Expand Up @@ -331,6 +392,18 @@ nginx:
We did not run the changed scripts on the TESK of ELIXIR-FI, and did not
test nginx with TLS or JupyterLab.

We tested the access tokens against a small mock of a TES service that
required tokens, with a stand-in for the hub: without a token, with an
invalid one and with one outside the compute group, every command ended
after its first request with the matching error. With a valid token,
`selftest`, `up`, `status`, `logs`, `print` and `down` worked for both
launchers, `up` replaced the keep-alive loop, and the loop logged the error
when the token expired. With the expired token, `down` still stopped the
loop and locked the tasks out. The token appeared in no command line, file,
log or environment of the hub.

We did not test the tokens against a real TES service.

## Where to go next

- [Interactive Services on TES: Commands](interactive_services_tes_commands.md)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,10 @@ mkdir -p ~/eoc_services/{tes-hub,omop-tesk/pod,rstudio-tesk} && cd ~/eoc_service
cat > tes-hub/tes-hub.sh <<'__TES_SERVICES_EOF__'
--8<-- "docs/tutorials/tes-services/tes-hub/tes-hub.sh"
__TES_SERVICES_EOF__
# requests to the TES service, with the access token in TES_TOKEN
cat > tes-hub/tes-api.sh <<'__TES_SERVICES_EOF__'
--8<-- "docs/tutorials/tes-services/tes-hub/tes-api.sh"
__TES_SERVICES_EOF__
# code sent into every task: downloads chisel, opens the tunnel
cat > tes-hub/common.sh <<'__TES_SERVICES_EOF__'
--8<-- "docs/tutorials/tes-services/tes-hub/common.sh"
Expand Down Expand Up @@ -85,6 +89,19 @@ location /tes-tunnel/ {
sudo systemctl reload nginx
```

## Set the access token

An LS Login access token, of an account in the compute group, lives for
about 4 hours: see [Get an access
token](interactive_services_tes.md#get-an-access-token).

```bash
# paste an LS Login access token (account in ECP_CLN:EOC-Compute); not shown, not in the history
read -rsp 'TES token: ' TES_TOKEN && echo && export TES_TOKEN
# check it: lists one task, or curl fails (error 22) with 401 (invalid or expired token) or 403 (not in the compute group)
printf 'Authorization: Bearer %s\n' "$TES_TOKEN" | curl -q -fsS -H @- "${TES_URL:-https://csc-tesk-noauth.rahtiapp.fi}/ga4gh/tes/v1/tasks?view=MINIMAL&page_size=1"
```

## Start Atlas with OMOP demo data

```bash
Expand Down Expand Up @@ -185,31 +202,37 @@ cd ~/eoc_services
# RStudio: show the TES request
./tes-hub/tes-service.sh rstudio-tesk print

# token expired (after about 4 hours, 401 in .state/watch.log): set a new token as in
# "Set the access token", then run up again, which gives the keep-alive loop the new token
./omop-tesk/omop-tesk-fi.sh up
./tes-hub/tes-service.sh rstudio-tesk up

# JupyterLab: status / logs / restart / down
./tes-hub/tes-service.sh jupyter-tesk status
./tes-hub/tes-service.sh jupyter-tesk logs
./tes-hub/tes-service.sh jupyter-tesk restart
./tes-hub/tes-service.sh jupyter-tesk down

# list all tasks on TESK @ ELIXIR-FI
curl -fsS "https://csc-tesk-noauth.rahtiapp.fi/ga4gh/tes/v1/tasks?view=MINIMAL"
# list all tasks on the TES service (the token goes through stdin, not the command line)
printf 'Authorization: Bearer %s\n' "$TES_TOKEN" | curl -q -fsS -H @- "${TES_URL:-https://csc-tesk-noauth.rahtiapp.fi}/ga4gh/tes/v1/tasks?view=MINIMAL"
```

## After a reboot

```bash
# start the hub again; running tasks reconnect within ~30 seconds
./tes-hub/tes-hub.sh start
# restart the keep-alive loops of the single-container services
# restart the keep-alive loops of the single-container services (set the access token first)
./tes-hub/tes-service.sh rstudio-tesk up
./tes-hub/tes-service.sh jupyter-tesk up
```

## Settings

```bash
# TES instance: TESK @ ELIXIR-FI (default)
# TES instance: TESK @ ELIXIR-FI (default; will change to the address with tokens)
export TES_URL=https://csc-tesk-noauth.rahtiapp.fi
# access token: see "Set the access token" (never as TES_TOKEN=... on the command line)
# hub port and chisel version (defaults)
export HUB_LISTEN_PORT=8090 CHISEL_VERSION=1.12.0
# Atlas stack images (defaults)
Expand Down
Loading
Loading