diff --git a/docs/researchers/index.md b/docs/researchers/index.md index a8644fd..e7490d6 100644 --- a/docs/researchers/index.md +++ b/docs/researchers/index.md @@ -12,6 +12,8 @@ ELIXIR Nodes, and keep sensitive data under the control of its owner. |---|---|---| | [Use Krini](krini.md) | Run TES tasks and WES workflows in your browser, on services from the Cloud Registry, and follow your runs. Describes the new version of the portal, which is being rewritten. | A web browser and an LS Login account. | | [Use the Command Line](cli.md) | Find services in the Cloud Registry, send tasks to a TES service such as proTES, run workflows on a WES service, find workflows in WorkflowHub and look up DRS objects from your terminal with `ga4gh-cli`. | Git and Rust on your computer, and an LS Login access token for services that need a login. | +| [Tutorial: Your First Run in Krini](tutorial-krini.md) | Log in to Krini, run a task and a workflow on demo data, and read the results, step by step. | A web browser, an LS Login account, and access to a TES and a WES service. | +| [Tutorial: Your First Run from the Command Line](tutorial-cli.md) | Install `ga4gh-cli`, find services, log in with a token, run a task and a workflow on demo data, and read the results, step by step. | Git and Rust on your computer, an LS Login account, and access to a TES and a WES service. | | [Run Workflows Across Nodes](federated-workflows.md) | Run a Snakemake workflow on the TES services of several Nodes, through the proTES gateway. | A Python environment on your computer, and the addresses of TES services. | | [Sensitive Data in a Secure Processing Environment](spe-with-weskit.md) | Run workflows on sensitive data that you never see, and read the results. | An LS Login account and permission from the operator of the environment. | | [Crypt4GH with proTES](crypt4gh-with-protes.md) | Process Crypt4GH-encrypted files in a secure container. Needs proTES 0.3.0 or earlier. | Docker and three virtual machines. | diff --git a/docs/researchers/tutorial-cli.md b/docs/researchers/tutorial-cli.md new file mode 100644 index 0000000..b909297 --- /dev/null +++ b/docs/researchers/tutorial-cli.md @@ -0,0 +1,350 @@ +# Tutorial: Your First Run from the Command Line + +## Synopsis + +This tutorial takes you from installing `ga4gh-cli`, the command-line tool of +the [GA4GH SDK](https://github.com/ga4gh/ga4gh-sdk), to the results of your +first analysis on ELIXIR On-Cloud. You find services in the Cloud Registry, +log in with an LS Login token, and count the lines, words and characters of +a small text file twice: once as a [GA4GH TES][ga4gh-tes] task, and once as +a [GA4GH WES][ga4gh-wes] workflow. It takes about an hour, most of it for +building the tool. + +The tutorial uses the demo files in this documentation. Each step does one +thing and links to [Use the Command Line](cli.md), the reference of the +tool, which explains all commands and options; this page doesn't repeat it. +The commands are for Bash or Zsh on Linux or macOS. + +!!! warning "What the CLI can't do yet" + - **No ready-made binaries:** you build the CLI from its source code + ([ga4gh-sdk#91](https://github.com/ga4gh/ga4gh-sdk/issues/91)). + +The CLI doesn't log in itself: it sends an LS Login access token that you +put into a variable. Step 5 shows how to get one. + +!!! warning "Synthetic or public data only" + The deployment is a sandbox for demos and tests. The demo file is public; + don't use the services with personal or other sensitive data. + +## 1. Before you start + +You need: + +- **Git and [Rust](https://www.rust-lang.org/tools/install)**, installed + with `rustup`, to build the CLI, and `curl` and + [`jq`](https://jqlang.org/) for the demo files and results. +- **An LS Login account.** See the [LS Login and registration + guide](https://www.lifescience-ri.eu/ls-login-and-registration-guide/). +- **Access to a TES service,** for part 1, and **to a WES service that runs + CWL `v1.0`,** for part 2, and the information which LS Login tokens they accept. + +The deployment doesn't publish the addresses of all its services for +researchers yet. To ask for services that you can use, write to +[cloud-service@elixir-europe.org][elixir-on-cloud-email] and say that you +want to follow this tutorial. + +The demo files: + +| File | What it is | +|---|---| +| [`LICENSE`](https://raw.githubusercontent.com/elixir-cloud-aai/docs/main/LICENSE) of this documentation | The text file that you count. | +| [`count-words.cwl`](tutorial-files/count-words.cwl) | A workflow in CWL that counts the lines, words and characters of a text file with `wc`, in the container image `alpine:3.20`. | +| [`count-words.json`](tutorial-files/count-words.json) | The parameters of the workflow: the address of the text file. | + +## 2. Install the CLI + +Build and install the CLI at the commit that [Use the Command +Line](cli.md#install) describes: + +```bash +git clone https://github.com/ga4gh/ga4gh-sdk.git +cd ga4gh-sdk +git checkout af0e021 +cargo install --path cli --locked +cd .. +``` + +**Check:** + +```bash +ga4gh-cli --version +``` + +The command prints the version of the CLI. If the shell doesn't find +`ga4gh-cli`, open a new terminal, so that it reads the `PATH` that `rustup` +set. + +## 3. Point the CLI at the Cloud Registry + +The CLI reads the addresses of the services from `~/.ga4gh-cli/config.json`. +Create it with the Cloud Registry of ELIXIR On-Cloud, which anyone can read +without a token. The command doesn't change a file that already exists; in +that case, add the `REGISTRY` section in a text editor (see +[Configure](cli.md#configure)). + +```bash +mkdir -p ~/.ga4gh-cli +[ -e ~/.ga4gh-cli/config.json ] || printf '%s\n' '{ + "REGISTRY": { + "base_path": "https://elixir-on.cloud/ga4gh/registry/v1" + } +}' > ~/.ga4gh-cli/config.json +chmod 600 ~/.ga4gh-cli/config.json +``` + +## 4. Find your services + +List the TES and the WES services in the Cloud Registry (see [Find services +in the Cloud Registry](cli.md#find-services-in-the-cloud-registry)): + +```bash +ga4gh-cli registry list --type tes +ga4gh-cli registry list --type wes +``` + +**Check:** the TES and WES services that you got access to are in the lists, +with their `URL`s. If a service isn't listed, use the address that its +operators gave you. + +!!! warning "Send your token only to services that you trust" + The CLI sends your token to every service that you configure with it, + and anyone who runs that service can act as you until the token + expires. The CLI doesn't show whether a service is verified. Use only + the services that you got access to, check that their addresses match + the ones that their operators gave you, and check them in the [front + end of the Cloud Registry][elixir-cloud-registry]: on the safe side, use + only services that are both verified and synced (see [Using the Cloud + Registry](../operators/using-cloud-registry.md#looking-up-services)). A + label in the registry doesn't replace your own check. + +Then add a `TES` and a `WES` section to `~/.ga4gh-cli/config.json` in a text +editor. `base_path` is the root of the API of the service (see +[Configure](cli.md#configure)): copy the `URL` from the list as it is, and +add the path of the API only if the `URL` ends before it, e.g., +`/ga4gh/tes/v1` for proTES. Your file then looks like this, with these API +roots instead of `` and ``: + +```json +{ + "REGISTRY": { + "base_path": "https://elixir-on.cloud/ga4gh/registry/v1" + }, + "TES": { + "base_path": "", + "oauth_access_token_env": "LS_LOGIN_TOKEN" + }, + "WES": { + "base_path": "", + "oauth_access_token_env": "LS_LOGIN_TOKEN" + } +} +``` + +For example, a proTES at `tes.example.org` has the API root +`https://tes.example.org/ga4gh/tes/v1`. + +`oauth_access_token_env` tells the CLI to send the token in the variable +`LS_LOGIN_TOKEN`. Leave it out for a service that doesn't need a token. + +## 5. Log in + +The CLI sends a token that you put into a variable; it doesn't get one +itself. Get an LS Login access token in one of these ways, and ask the +operators of your services which of them they accept: + +- **With your own LS Login client,** as [Get an access + token](../partners-and-developers/apis.md#2-get-an-access-token) + describes. +- **From Krini,** the web portal: log in, then copy your token, as [Copy your + access token](krini.md#copy-your-access-token) describes. + +Put the token into the variable in the terminal where you use the CLI (see +[Log in with LS Login](cli.md#log-in-with-ls-login)). The token isn't shown +and isn't saved in the history of the shell: + +```bash +read -rs LS_LOGIN_TOKEN # paste the token, then press Enter +export LS_LOGIN_TOKEN +``` + +**Check:** both services answer: + +```bash +ga4gh-cli tes list +ga4gh-cli wes list +``` + +The commands list your tasks and runs, or none. `401 Unauthorized` means +that the token expired or that the service doesn't accept it: get a new one, +or ask the operators. + +## Part 1: Run a task + +A TES task runs a command in a container. Here, the command is `wc`, and the +service copies the text file into the container before the command starts. + +### 6. Create the task + +Create an empty folder for the demo files, so that they don't overwrite +files of yours, and work in it until the end of the tutorial: + +```bash +mkdir ga4gh-cli-tutorial +cd ga4gh-cli-tutorial +``` + +`mkdir` stops with `File exists` if the folder is already there: then +choose another name. + +Write the task into a file and send it (see [Create a +task](cli.md#create-a-task)): + +```bash +printf '%s\n' '{ + "name": "count-words", + "inputs": [ + { + "url": "https://raw.githubusercontent.com/elixir-cloud-aai/docs/main/LICENSE", + "path": "/data/LICENSE", + "type": "FILE" + } + ], + "executors": [ + { + "image": "alpine:3.20", + "command": ["wc", "/data/LICENSE"] + } + ] +}' > count-words-task.json +TASK_ID=$(ga4gh-cli tes create count-words-task.json) +echo "$TASK_ID" +``` + +**Check:** the CLI prints the ID of the new task. If it ends with an error, +see [Check for errors](cli.md#check-for-errors) before you send the task +again. + +### 7. Follow the task + +Ask for its state until it is `COMPLETE` (see [Get the state of a +task](cli.md#get-the-state-of-a-task)). Tasks first wait in `QUEUED` until +the service has resources; this can take a few minutes. + +```bash +ga4gh-cli tes status "$TASK_ID" +``` + +If the state is `EXECUTOR_ERROR` or `SYSTEM_ERROR`, read the logs as in the +next step. Not every TES service can fetch files from an `https` address: if +the input is the problem, remove `inputs` from the task and use the command +`["echo", "Hello"]` instead, which checks everything else. + +### 8. Read the result + +Get the task with its logs, and pick the standard output of its command (see +[Get a task](cli.md#get-a-task)): + +```bash +ga4gh-cli tes get "$TASK_ID" FULL | jq -r '.logs[-1].logs[0].stdout' +``` + +**Result:** the numbers of lines, words and characters of the file: + +```text + 121 1066 7048 /data/LICENSE +``` + +The numbers change if the file changes. Some services return no standard +output in the logs; then the result is `null`. + +## Part 2: Run a workflow + +A WES service runs a whole workflow. Here, the workflow is +[`count-words.cwl`](tutorial-files/count-words.cwl), which does the same as +the task, and writes the result to the file `counts.txt`. Check with the +operators of your WES service that it runs CWL `v1.0`. If no WES service +that you can use runs it, stop after part 1 and ask the operators for +access to one that does: the steps below work only with CWL `v1.0`. + +### 9. Start the workflow + +Download the parameters of the workflow, then start a run with the address +of the workflow (see [Start a run](cli.md#start-a-run)): + +```bash +curl -fsSLO https://raw.githubusercontent.com/elixir-cloud-aai/docs/main/docs/researchers/tutorial-files/count-words.json +RUN_ID=$(ga4gh-cli wes run \ + https://raw.githubusercontent.com/elixir-cloud-aai/docs/main/docs/researchers/tutorial-files/count-words.cwl \ + --type CWL --type_version v1.0 --params count-words.json) +echo "$RUN_ID" +``` + +**Check:** the CLI prints the ID of the new run. + +### 10. Follow the run + +Ask for its state until it is `COMPLETE` (see [Follow a +run](cli.md#follow-a-run)): + +```bash +ga4gh-cli wes status "$RUN_ID" +``` + +### 11. Read the result + +Get the run, and pick its outputs: + +```bash +ga4gh-cli wes get "$RUN_ID" | jq '.outputs' +``` + +**Result:** `outputs` lists the file `counts.txt` with its address. Where +the file is, and how you download it, depends on the service, e.g., in a +bucket of its object storage. Its content is the same as in step 8, without +the file name: + +```text + 121 1066 7048 +``` + +If the run failed, `jq '.run_log'` instead of `jq '.outputs'` shows the log +of the workflow engine. + +## 12. Clean up + +Remove the token from the terminal, and the demo files and their folder: + +```bash +unset LS_LOGIN_TOKEN +rm count-words-task.json count-words.json +cd .. +rmdir ga4gh-cli-tutorial +``` + +`rmdir` removes only an empty folder, so it keeps any other file that you +saved there. + +The configuration file holds no token, so you can keep it for next time. + +## Where to go next + +- All commands and options, e.g., to find workflows in WorkflowHub or to + attach files to a run: [Use the Command Line](cli.md). +- Do the same in your browser: [Tutorial: Your First Run in + Krini](tutorial-krini.md). +- Run a workflow on the TES services of several Nodes: [Run Workflows + Across Nodes](federated-workflows.md). + +## How this tutorial was tested + +The workflow [`count-words.cwl`](tutorial-files/count-words.cwl) ran with +`cwltool` and Docker on one computer, with +[`count-words.json`](tutorial-files/count-words.json), and gave the result +shown in step 11; the command of the task gave the result shown in step 8 in +the image `alpine:3.20`. The commands and options of the CLI follow [Use +the Command Line](cli.md), which was tested at commit `af0e021` of the SDK. +The steps have not yet been run against deployed TES and WES services of +ELIXIR On-Cloud, and not yet by someone who didn't write them. If you follow +this tutorial, please tell us what didn't work: in the +[Discussions][elixir-on-cloud-forum] or in an issue in the [docs +repository](https://github.com/elixir-cloud-aai/docs/issues). diff --git a/docs/researchers/tutorial-files/count-words.cwl b/docs/researchers/tutorial-files/count-words.cwl new file mode 100644 index 0000000..47215b4 --- /dev/null +++ b/docs/researchers/tutorial-files/count-words.cwl @@ -0,0 +1,19 @@ +# Demo workflow of the researcher tutorials of ELIXIR On-Cloud: counts the +# lines, words and characters of a text file, like `wc` +cwlVersion: v1.0 +class: CommandLineTool +doc: Counts the lines, words and characters of a text file. +requirements: + DockerRequirement: + dockerPull: alpine:3.20 +baseCommand: wc +stdin: $(inputs.text.path) +inputs: + text: + type: File + doc: The text file to count, e.g., at an https URL. +outputs: + counts: + type: stdout + doc: The numbers of lines, words and characters. +stdout: counts.txt diff --git a/docs/researchers/tutorial-files/count-words.json b/docs/researchers/tutorial-files/count-words.json new file mode 100644 index 0000000..497b09f --- /dev/null +++ b/docs/researchers/tutorial-files/count-words.json @@ -0,0 +1,6 @@ +{ + "text": { + "class": "File", + "location": "https://raw.githubusercontent.com/elixir-cloud-aai/docs/main/LICENSE" + } +} diff --git a/docs/researchers/tutorial-krini.md b/docs/researchers/tutorial-krini.md new file mode 100644 index 0000000..06ea978 --- /dev/null +++ b/docs/researchers/tutorial-krini.md @@ -0,0 +1,230 @@ +# Tutorial: Your First Run in Krini + +## Synopsis + +This tutorial takes you from your first login to the results of your first +analysis in [Krini][krini-repo], the web portal of ELIXIR On-Cloud. You count +the lines, words and characters of a small text file twice: once as a +[GA4GH TES][ga4gh-tes] task, and once as a [GA4GH WES][ga4gh-wes] workflow. +It takes about 30 minutes, and you need nothing but a web browser and, for +the last part, a terminal. + +The tutorial uses the demo files in this documentation. Each step does one +thing and links to the [Krini user guide](krini.md), which explains every +field, button and message; this page doesn't repeat it. + +!!! note "The new Krini" + This tutorial describes the new Krini (branch `staging` of the [Krini + repository][krini-repo], commit + [`c81a259`](https://github.com/elixir-cloud-aai/krini/commit/c81a259)), + like the [Krini user guide](krini.md). Parts of it are still in progress + (see [What's coming](krini.md#whats-coming)), so what you see can + differ. + +!!! warning "Synthetic or public data only" + The deployment is a sandbox for demos and tests. The demo file is public; + don't use Krini with personal or other sensitive data. + +## 1. Before you start + +You need: + +- **An LS Login account.** See [Before you start](krini.md#before-you-start) + in the user guide. +- **Access to a TES service that accepts requests without a token,** for + part 1. Krini doesn't send your token to TES services yet (see [Run a + task](krini.md#run-a-task)). +- **Access to a WES service that runs CWL `v1.0`,** for part 2. + +Krini shows the services of the Cloud Registry, but doesn't give you access +to them. To ask for services that you can use, write to +[cloud-service@elixir-europe.org][elixir-on-cloud-email] and say that you +want to follow this tutorial. + +The demo files: + +| File | What it is | +|---|---| +| [`LICENSE`](https://raw.githubusercontent.com/elixir-cloud-aai/docs/main/LICENSE) of this documentation | The text file that you count. | +| [`count-words.cwl`](tutorial-files/count-words.cwl) | A workflow in CWL that counts the lines, words and characters of a text file with `wc`, in the container image `alpine:3.20`. | +| [`count-words.json`](tutorial-files/count-words.json) | The parameters of the workflow: the address of the text file. | + +## 2. Log in + +1. Open Krini in your browser. +2. Log in with LS Login, as [Log in](krini.md#log-in) describes. + +**Check:** the sidebar shows your name and email address. + +## Part 1: Run a task + +A TES task runs a command in a container. Here, the command is `wc`, and the +service copies the text file into the container before the command starts. + +### 3. Choose a TES service + +1. Open **Tasks** in the sidebar. +2. Choose the TES service that you got access to in **Select TES Instance** + (see [Choose a service](krini.md#choose-a-service)). + +**Check:** the list below shows the tasks of the service, or none. + +!!! warning "Don't choose TEST SERVER" + The **TEST SERVER** entry in the list of TES services doesn't work yet + ([krini#67](https://github.com/elixir-cloud-aai/krini/issues/67)). + Choose the service that you got access to. + +### 4. Create the task + +1. Select **Create Task**. +2. Fill in the form with these values, and leave the other fields empty + (see [Run a task](krini.md#run-a-task) for all fields): + + | Field | Value | + |---|---| + | **Name** | `count-words` | + | **Executors** › **Image** | `alpine:3.20` | + | **Executors** › **Commands** | `wc` in the first field, `/data/LICENSE` in the second | + | **Inputs** › **URL** | `https://raw.githubusercontent.com/elixir-cloud-aai/docs/main/LICENSE` | + | **Inputs** › **Path** | `/data/LICENSE` | + +3. Submit the form. + +**Check:** Krini shows the ID of the new task. Note it. + +!!! tip "If the service can't fetch the input" + Not every TES service can fetch files from an `https` address. If the + task ends in `SYSTEM_ERROR`, create it again without the input, with the + commands `echo` and `Hello`. That checks everything else. + +### 5. Follow the task and read the result + +1. Open **Tasks** again. Your task is in the list with its state (see + [Follow your tasks](krini.md#follow-your-tasks) for what the states + mean). +2. Wait until the state is `COMPLETE`. Tasks first wait in `QUEUED` until the + service has resources; this can take a few minutes. +3. Expand the task and read the standard output in its logs. + +**Result:** the standard output of `wc`, the numbers of lines, words and +characters of the file: + +```text + 121 1066 7048 /data/LICENSE +``` + +The numbers change if the file changes. If the state is `EXECUTOR_ERROR` or +`SYSTEM_ERROR`, read the logs and see +[Troubleshooting](krini.md#troubleshooting). + +!!! warning "No result files from tasks yet" + This task writes its result to the standard output only. Tasks that + write result files need **Outputs**, which the task form of Krini + doesn't pass on to the service yet + ([cloud-components#574](https://github.com/elixir-cloud-aai/cloud-components/issues/574); + see [Run a task](krini.md#run-a-task)). Part 2 gets a result file + through WES instead. + +## Part 2: Run a workflow + +A WES service runs a whole workflow. Here, the workflow is +[`count-words.cwl`](tutorial-files/count-words.cwl), which does the same as +the task, and writes the result to the file `counts.txt`. + +### 6. Choose a WES service + +1. Open **Workflows** in the sidebar. +2. Choose the WES service that you got access to in **Select WES Instance** + (see [Choose a service](krini.md#choose-a-service)). Choose only a + service that you trust: Krini sends it your token. +3. Point at the info icon next to the service and check that it supports + the workflow type `CWL` in the version `v1.0`. + +If no WES service that you can use runs CWL `v1.0`, stop after part 1 and +ask the operators for access to one that does: the steps below work only +with CWL `v1.0`. + +### 7. Start the workflow + +1. Select **Create Workflow**. +2. Fill in the form (see [Run a workflow](krini.md#run-a-workflow) for all + fields): + + | Field | Value | + |---|---| + | **Workflow URL** | `https://raw.githubusercontent.com/elixir-cloud-aai/docs/main/docs/researchers/tutorial-files/count-words.cwl` | + | **Workflow Type** | `CWL` | + | **Workflow Type Version** | `v1.0` | + | **Workflow Parameters (JSON)** | The content of [`count-words.json`](tutorial-files/count-words.json), shown below | + + ```json + { + "text": { + "class": "File", + "location": "https://raw.githubusercontent.com/elixir-cloud-aai/docs/main/LICENSE" + } + } + ``` + +3. Select **Submit Run**. + +**Check:** Krini shows the run ID. Note it. + +### 8. Follow the run + +1. Open **Workflows** again. Your run is in the list with its state (see + [Follow your workflow runs](krini.md#follow-your-workflow-runs)). +2. Wait until the state is `COMPLETE`. + +### 9. Read the result + +!!! warning "No run details in Krini yet" + **View Details** doesn't open anything in Krini yet + ([krini#66](https://github.com/elixir-cloud-aai/krini/issues/66)), so + this step needs a terminal. + +Read the details and outputs of the run from the WES service, with your +access token, as [Read the details and outputs of a +run](krini.md#read-the-details-and-outputs-of-a-run) describes: copy your +token in Krini, look up the API endpoint of the service, and call it with +`curl` and your run ID. + +**Result:** in the answer, `outputs` lists the file `counts.txt` with its +address. Where the file is, and how you download it, depends on the service, +e.g., in a bucket of its object storage. Its content is the same as in step +5, without the file name: + +```text + 121 1066 7048 +``` + +## 10. Clean up + +- Log out of Krini (see [Log in](krini.md#log-in)). +- In the terminal, end the session in which you pasted your token, or remove + it with `unset TOKEN`. + +## Where to go next + +- Run your own task or workflow: the [Krini user guide](krini.md) explains + all fields, and [GA4GH TES][ga4gh-tes] and [GA4GH WES][ga4gh-wes] the + APIs. +- Do the same from your terminal: [Tutorial: Your First Run from the Command + Line](tutorial-cli.md). +- Run a workflow on the TES services of several Nodes: [Run Workflows + Across Nodes](federated-workflows.md). + +## How this tutorial was tested + +The workflow [`count-words.cwl`](tutorial-files/count-words.cwl) ran with +`cwltool` and Docker on one computer, with +[`count-words.json`](tutorial-files/count-words.json), and gave the result +shown in step 9. The values of the forms follow the [Krini user +guide](krini.md) and the source code of Krini at commit `c81a259`. The steps +have not yet been run against a deployed Krini and deployed TES and WES +services, and not yet by someone who didn't write them. If you follow this +tutorial, please tell us what didn't work: in the +[Discussions][elixir-on-cloud-forum] or in an issue in the [docs +repository](https://github.com/elixir-cloud-aai/docs/issues). + +[krini-repo]: diff --git a/mkdocs.yml b/mkdocs.yml index 976db03..cc71586 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -117,6 +117,8 @@ nav: - "researchers/index.md" - "Use Krini": "researchers/krini.md" - "Use the Command Line": "researchers/cli.md" + - "Tutorial: First Run in Krini": "researchers/tutorial-krini.md" + - "Tutorial: First Run from the CLI": "researchers/tutorial-cli.md" - "Run Workflows Across Nodes": "researchers/federated-workflows.md" - "SPE with WESkit": "researchers/spe-with-weskit.md" - "Crypt4GH with proTES": "researchers/crypt4gh-with-protes.md"