Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ version: 2
# whatever the default branch is (`main` only moves forward to releases);
# security updates always target the default branch, though
updates:
# Dependencies in `pom.xml`. Java 8 and Spring Boot 2 stay until the move to
# Spring Boot 3, which needs code changes
# Dependencies in `pom.xml`. Spring Boot 2 stays until the move to Spring
# Boot 3, which needs code changes
- package-ecosystem: maven
directory: /
schedule:
Expand Down Expand Up @@ -38,8 +38,9 @@ updates:
groups:
github-actions:
patterns: ["*"]
# Base images in the `Dockerfile` (tag and digest). Java 8 stays until the
# move to Spring Boot 3
# Base images in the `Dockerfile` (tag and digest). Java 21 (LTS), as in
# `pom.xml` and the CI job `Test`: a new major version of Java is a
# decision of its own
- package-ecosystem: docker
directory: /
schedule:
Expand Down
9 changes: 5 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -91,9 +91,10 @@ jobs:
docker run --rm --volume "${PWD}:/repo" --workdir /repo \
rhysd/actionlint:1.7.7@sha256:887a259a5a534f3c4f36cb02dca341673c6089431057242cdc931e9f133147e9 \
-color
# Java 8, as in `test`: the plugin versions are the last that run on it.
# No Maven cache: it would share the key of `test` and could be saved
# with only the lint plugins in it
# Java 8, unlike `test`: google-java-format 1.7 doesn't run on Java 21,
# and newer versions format the code differently (to be updated with the
# move to Spring Boot 3, #66). No Maven cache: it would share the key of
# `test` and could be saved with only the lint plugins in it
- name: Set up Java
uses: actions/setup-java@77dd5b2ce225ea11469928ac317850245a74ae78 # v6.2.0
with:
Expand Down Expand Up @@ -129,7 +130,7 @@ jobs:
uses: actions/setup-java@77dd5b2ce225ea11469928ac317850245a74ae78 # v6.2.0
with:
distribution: temurin
java-version: "8"
java-version: "21"
cache: maven
- name: Build and run tests
run: ./mvnw -B clean verify
Expand Down
12 changes: 3 additions & 9 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Builds the application from source, so that a plain `docker build .` is all
# that is needed. Java 8, as in `pom.xml`; the base images are pinned by digest
# that is needed. Java 21, as in `pom.xml`; the base images are pinned by digest
# and updated by Dependabot
FROM eclipse-temurin:8-jdk@sha256:e9255b50040337b0dec91027868dca1952ef5441bf0c8f878c5f790eaedbfea2 AS build
FROM eclipse-temurin:21-jdk@sha256:0a3f329821dcadf54b2a3033a15e2e9f007069a9db43d7f51897e6b80e726b07 AS build
# The Maven wrapper downloads the .zip of Maven and checks its SHA-256 sum
# (`.mvn/wrapper/maven-wrapper.properties`); without `unzip` it would fetch the
# .tar.gz instead, which the sum does not match
Expand All @@ -15,13 +15,7 @@ COPY src src
RUN ./mvnw -B -Dmaven.test.skip=true package \
&& cp target/tesk-*.jar /app.jar

FROM eclipse-temurin:8-jre@sha256:377fe8e36a4b1530d9d52020069367ef44313fa891a82ae102b1016a6a3a72f5
# The pinned base image has a vulnerable apt (CVE-2026-109518; fixed by
# 3.2.1ubuntu0.1 in the Ubuntu security updates); can go once Dependabot moves
# the base image to a build with the fix
RUN apt-get update && \
apt-get install -y --no-install-recommends --only-upgrade apt libapt-pkg7.0 && \
rm -rf /var/lib/apt/lists/*
FROM eclipse-temurin:21-jre@sha256:18bb4a91a3dea3f03ee31d3755af34dfb40f904fc77ab56cdc2f092e32fff9c2
# Run as non-root user; group 0 owns the home directory, so that arbitrary
# user IDs (e.g., on OpenShift, always in group 0) can use it too. The app
# writes only to `/tmp`; the Kubernetes client reads `~/.kube/config` (or
Expand Down
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,9 @@ To deploy TESK to a cluster, use the Helm chart of [TESK][tesk].

## Development

TESK API builds and runs on Java 8. To build it and run the unit and
integration tests (no cluster needed):
TESK API builds and runs on Java 21 (the formatter and linter still need
Java 8; see [Development][docs-development]). To build it and run the unit
and integration tests (no cluster needed):

```bash
./mvnw clean verify
Expand Down
4 changes: 2 additions & 2 deletions docs/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ cosign verify elixircloud/tesk-api:1.2.0 \
For `dev` images, the identity is
`https://github.com/elixir-cloud-aai/tesk-api/.github/workflows/ci.yml@refs/heads/dev`.

The image runs the application with Java 8 on [Eclipse
The image runs the application with Java 21 on [Eclipse
Temurin](https://adoptium.net/) and listens on port 8080. `JAVA_OPTS` passes
options to the JVM. It runs as the non-root user `tesk` (UID 1000, group 0,
home directory `/home/tesk`); any other UID in group 0 works too, as on
Expand Down Expand Up @@ -66,7 +66,7 @@ needs to manage jobs, pods and persistent volume claims.
## Running locally

To run TESK API outside of Kubernetes, e.g. against a local cluster such as
[kind](https://kind.sigs.k8s.io/) or [k3d](https://k3d.io/), with Java 8 and
[kind](https://kind.sigs.k8s.io/) or [k3d](https://k3d.io/), with Java 21 and
a `KUBECONFIG` file for that cluster:

```bash
Expand Down
25 changes: 16 additions & 9 deletions docs/development.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
# Development

TESK API is a [Spring Boot](https://spring.io/projects/spring-boot) 2.7
application on Java 8, built with Maven 3.9 (through the wrapper `./mvnw`,
which downloads the version pinned in `.mvn/wrapper/maven-wrapper.properties`;
Maven 3.9 still runs on Java 8). The server stub is generated from the TES
specification with the [Swagger Codegen Maven
application on Java 21, built with Maven 3.9 (through the wrapper `./mvnw`,
which downloads the version pinned in `.mvn/wrapper/maven-wrapper.properties`).
The server stub is generated from the TES specification with the [Swagger Codegen Maven
plugin](https://github.com/swagger-api/swagger-codegen/tree/master/modules/swagger-codegen-maven-plugin),
and the [Kubernetes Java client](https://github.com/kubernetes-client/java)
talks to the Kubernetes API.
Expand Down Expand Up @@ -66,10 +65,15 @@ neither):
./mvnw spotless:check checkstyle:check
```

The plugin versions (spotless-maven-plugin 2.30.0 with google-java-format 1.7,
Checkstyle 9.3) are the last that run on Java 8; newer ones need a newer JDK
(the move to Spring Boot 3,
[#66](https://github.com/elixir-cloud-aai/tesk-api/issues/66)).
These two commands need Java 8 (e.g. `JAVA_HOME` pointing to a Java 8 JDK),
as the CI job `Lint` runs them: google-java-format 1.7 (with
spotless-maven-plugin 2.30.0) doesn't run on Java 21, and the newer versions
that do format the code differently. The update, with the reformat it brings,
is part of the move to Spring Boot 3
([#66](https://github.com/elixir-cloud-aai/tesk-api/issues/66)). Building and
testing need Java 21. Until then, write code that Java 8 can parse (no
records, text blocks, `switch` expressions or `instanceof` patterns), or the
formatter fails.

The commits that only reformatted the code are listed in
`.git-blame-ignore-revs`. GitHub skips them in its blame view; for `git blame`,
Expand Down Expand Up @@ -218,7 +222,10 @@ Validation](https://beanvalidation.org/) annotations beyond those derived from
the specification (e.g. `@Digits` and `@DecimalMin` on the resources of
`TesResources`); restore them by hand in the generated code, and reconcile
any other manual changes, e.g. the internal field `userIdFromLegacyLabel` of
`TesTaskLog` (not serialized), which the access check of a task needs. Until
`TesTaskLog` (not serialized), which the access check of a task needs, and
the check `isUrlOrContentGiven` of `TesInput` (a generator that adds
`@ScriptAssert(lang = "javascript")` instead brings back a check that fails
on Java 15 and later, which have no JavaScript engine). Until
the experimental private image credential workaround remains supported, also
restore the `docker_registry`, `docker_login` and `docker_pass` fields,
validation, write-only handling, and experimental API descriptions in
Expand Down
15 changes: 14 additions & 1 deletion docs/upgrading.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,21 @@

## To the next release

From 1.2.0, clients may notice changed answers:
From 1.2.0, deployments may have to check their JVM options, and clients may
notice changed answers:

- **Java 21.** The image runs TESK API on Java 21 instead of Java 8 (see
[Deployment](deployment.md)). If you pass options to the JVM in
`JAVA_OPTS`, check that Java 21 still accepts them: options that it
removed (e.g. `-XX:MaxPermSize`, `-XX:+UseConcMarkSweepGC`,
`-XX:+UseCGroupMemoryLimitForHeap`) stop the JVM
at startup. Without `JAVA_OPTS`, nothing changes.
- **Inputs without a source.** An input with neither a `url` (that isn't
blank) nor a `content` (that isn't empty) is still refused with `400 Bad
Request`, but the message now names the check:
`Field 'inputs[0].urlOrContentGiven' URL or content required` instead of
`Field 'inputs[0]' URL or content required`. The messages are meant for
people; clients should rely on the status code, not on their text.
- **Wildcards without `path_prefix`.** An output whose `path` has wildcards
(`*`, `?`, `[`) but no `path_prefix` is refused with `400 Bad Request`,
unless its `url` is a `file:` URL or has no scheme, as TES 1.1 requires
Expand Down
9 changes: 7 additions & 2 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<log4j2.version>2.17.1</log4j2.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
<java.version>21</java.version>
<!-- Dependency versions -->
<springfox-swagger.version>3.0.0</springfox-swagger.version>
<kubernetes.client-java.version>11.0.4</kubernetes.client-java.version>
Expand Down Expand Up @@ -293,6 +293,10 @@
<version>${maven-surefire-plugin.version}</version>
<configuration>
<useSystemClassLoader>false</useSystemClassLoader>
<!-- JaCoCo's agent (`argLine`, set by prepare-agent), and access to the JDK
internals that system-rules' EnvironmentVariables changes, which Java 17
and later close (TesKubernetesConverterTest) -->
<argLine>@{argLine} --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED</argLine>
</configuration>
</plugin>
<plugin>
Expand Down Expand Up @@ -396,7 +400,8 @@
</plugin>
<!-- Formatting (google-java-format, AOSP style) and lint (checkstyle.xml):
./mvnw spotless:apply to format, ./mvnw spotless:check checkstyle:check to check
(CI job Lint). These versions are the last that run on Java 8. -->
(CI job Lint), on Java 8: google-java-format 1.7 does not run on Java 21, and
newer versions format differently (docs/development.md). -->
<plugin>
<groupId>com.diffplug.spotless</groupId>
<artifactId>spotless-maven-plugin</artifactId>
Expand Down
27 changes: 25 additions & 2 deletions src/main/java/uk/ac/ebi/tsc/tesk/tes/model/TesInput.java
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
package uk.ac.ebi.tsc.tesk.tes.model;

import java.util.Objects;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.annotation.JsonSetter;
import io.swagger.annotations.ApiModel;
import io.swagger.annotations.ApiModelProperty;
import org.hibernate.validator.constraints.ScriptAssert;

import javax.validation.Valid;
import javax.validation.constraints.*;
Expand All @@ -18,7 +18,6 @@
*/
@ApiModel(description = "Input describes Task input files.")
@javax.annotation.Generated(value = "org.openapitools.codegen.languages.SpringCodegen", date = "2021-03-24T17:10:08.716Z[Europe/London]")
@ScriptAssert(lang = "javascript", script = "(_this.url != null && (_this.url.trim().length) > 0) || (_this.content != null && _this.content.length > 0)", message = "URL or content required")
public class TesInput {
@JsonProperty("name")
private String name;
Expand Down Expand Up @@ -191,6 +190,30 @@ public void setContent(String content) {
this.content = content;
}

/**
* An input needs a source: a `url` that isn't blank (white space as JavaScript's `trim` saw it
* before), or a `content` that isn't empty. A plain method rather than a script
* (`@ScriptAssert`), as Java 15 and later have no JavaScript engine.
*
* @return whether `url` or `content` is given
*/
@JsonIgnore
@AssertTrue(message = "URL or content required")
public boolean isUrlOrContentGiven() {
return (url != null && !isJavaScriptBlank(url)) || (content != null && !content.isEmpty());
}

/**
* Whether a string holds only white space and line terminators as JavaScript's `trim` sees
* them (ECMAScript WhiteSpace and LineTerminator). Unlike {@link String#isBlank()}, this
* includes U+00A0, U+2007, U+202F and U+FEFF, and excludes U+001C to U+001F.
*/
private static boolean isJavaScriptBlank(String s) {
return s.codePoints().allMatch(c -> c == '\t' || c == '\n' || c == 0x0B || c == '\f'
|| c == '\r' || c == 0xFEFF || c == 0x2028 || c == 0x2029
|| Character.getType(c) == Character.SPACE_SEPARATOR);
}


@Override
public boolean equals(Object o) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -243,7 +243,15 @@ public void createTask_invalid_inputs() throws Exception {
String[][] files =
new String[][] {
new String[] {"no_path.json", "inputs[0].path"},
new String[] {"no_source.json", "URL or content required"},
new String[] {
"no_source.json", "inputs[0].urlOrContentGiven' URL or content required"
},
new String[] {
"blank_url.json", "inputs[0].urlOrContentGiven' URL or content required"
},
new String[] {
"empty_content.json", "inputs[0].urlOrContentGiven' URL or content required"
},
new String[] {"wrong_type.json", "inputs[0].type"},
new String[] {"empty_type.json", "inputs[0].type"}
};
Expand Down
32 changes: 32 additions & 0 deletions src/test/java/uk/ac/ebi/tsc/tesk/tes/model/TesInputTest.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
package uk.ac.ebi.tsc.tesk.tes.model;

import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;

import org.junit.Test;

public class TesInputTest {

@Test
public void urlOrContentGiven() {
assertTrue(new TesInput().url("s3://bucket/file").isUrlOrContentGiven());
assertTrue(new TesInput().content("data").isUrlOrContentGiven());
assertTrue(new TesInput().url("s3://bucket/file").content("data").isUrlOrContentGiven());
assertTrue(new TesInput().url(" ").content(" ").isUrlOrContentGiven());
// JavaScript's trim keeps the information separators U+001C to U+001F.
assertTrue(new TesInput().url("\u001c").isUrlOrContentGiven());
}

@Test
public void neitherUrlNorContentGiven() {
assertFalse(new TesInput().isUrlOrContentGiven());
assertFalse(new TesInput().url(" ").isUrlOrContentGiven());
assertFalse(new TesInput().url("\u3000\t").isUrlOrContentGiven());
assertFalse(
new TesInput()
.url("\t\n\u000b\f\r\u00a0\ufeff\u2007\u202f\u2028\u2029")
.isUrlOrContentGiven());
assertFalse(new TesInput().content("").isUrlOrContentGiven());
assertFalse(new TesInput().url("").content("").isUrlOrContentGiven());
}
}
20 changes: 20 additions & 0 deletions src/test/resources/invalid_tasks/inputs/blank_url.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"executors": [
{
"image": "ubuntu",
"command": [
"echo",
"hello world"
]
}
],
"inputs": [
{
"name": "input1",
"description": "Example",
"path": "/sth/sth.json",
"type": "FILE",
"url": " "
}
]
}
20 changes: 20 additions & 0 deletions src/test/resources/invalid_tasks/inputs/empty_content.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"executors": [
{
"image": "ubuntu",
"command": [
"echo",
"hello world"
]
}
],
"inputs": [
{
"name": "input1",
"description": "Example",
"path": "/sth/sth.json",
"type": "FILE",
"content": ""
}
]
}
Loading