Translator SRI Automated Test Harness

Automated Test Harness that downloads Translator Tests and executes them via Translator Runners

Overview

The Test Harness is a wrapper around the Test Runners. Its job is to retrieve automated tests, run the given queries, and then pass the responses along to test runners, and then send the report to a test dashboard, all while being easily and automatically instantiated.

Test Runners

The Test Harness incorporates Test Runners that run analyses on the responses of the automated tests. These Runners must be pip installable and take a test asset input and response as arguments. An example Test Runner function can be found here. The list of current Test Runners can be found in requirements-runners.txt.

Test Schema

WARNING: This schema is likely to change as the Test Cases are finalized

• env: the environment to run the queries against. (dev, ci, test, prod)
• query_type: type of query to test. (treats(creative), upregulates, downregulates)
• expected_output: whether the output curie is good or bad. (TopAnswer, Acceptable, BadButForgivable, NeverShow)
• input_curie: curie used in the initial query.
• output_curie: curie checked for in the results

How to use:

The Test Harness is a CLI that you need to install:

• pip install -r requirements.txt to install normal dependencies
• pip install -r requirements-runners.txt to install the Test Runners
• pip install . to install the Test Harness CLI

Once everything is installed, you can call

• test-harness -h to see available options

Running locally

By default the Test Harness reports results to an Information Radiator and posts them to Slack. To run everything locally without those services (for example while developing), pass --local:

• test-harness --local download <suite>

In local mode the harness makes no network calls to the Information Radiator or Slack. The test results (CSV and JSON) and any performance artifacts are saved to a local directory instead (test_results/ by default, configurable with --output_dir).

You don't have to use --local to get local files: if Slack isn't configured (no SLACK_WEBHOOK_URL / SLACK_TOKEN / SLACK_CHANNEL), the results are saved to --output_dir automatically. Likewise, if the Information Radiator isn't configured (no ZE_BASE_URL / ZE_REFRESH_TOKEN), the harness falls back to a local reporter.

The new Information Radiator

A replacement for the Zebrunner-based Information Radiator lives in this repo, in radiator/ (full details in docs/information-radiator.md). It gets structured results instead of labels and log lines: for each test asset, every agent's status, whether the expected answer was found and its rank/score, the number of results, the HTTP status, and (for direct queries) the response time. Its dashboard shows each run, what changed since the previous one, each asset's history, agent pass-rate trends, and performance trends.

Trying it out with Docker Compose

With Docker installed, from the repo root:

• docker compose up --build

That starts Postgres, sets up its tables, starts the radiator, and then runs a small test suite (local_acceptance) that reports to it. When the suite finishes, the harness container exits; the radiator keeps running at http://localhost:8000. Log in with translator / radiator.

The suite queries the real Translator services (the ARS on ci), so the run needs internet access and takes a few minutes. It reports to the new radiator only, unless you put ZE_BASE_URL and ZE_REFRESH_TOKEN in .env to report to Zebrunner as well (see A Slack channel for the new radiator for where Slack posts go then).

Everything is configurable from a .env file next to compose.yml; copy .env.example to .env to start from the Slack settings:

Setting	Default
HARNESS_ARGS	load local_acceptance	What the harness runs, eg download sprint_4_tests, or --query_type MVP1 download sprint_4_tests
RADIATOR_USERNAME / RADIATOR_PASSWORD	translator / radiator	The login
RADIATOR_TOKEN	local-dev-token	The API token the harness uploads with
RADIATOR_PORT	8000	Where the radiator is published on your machine
QUERY_PARAMETERS	unset	Parameters sent with every query, eg {"timeout": 300} (see Sending parameters with every query)
SLACK_WEBHOOK_URL, SLACK_TOKEN, SLACK_CHANNEL	unset	Set all three to also post the report, with the radiator's charts, to Slack
ZE_BASE_URL, ZE_REFRESH_TOKEN	unset	Also report to Zebrunner; SLACK_CHANNEL then gets only Zebrunner's report, and the radiator's go to RADIATOR_SLACK_* (or nowhere, if that isn't set)
RADIATOR_SLACK_WEBHOOK_URL, RADIATOR_SLACK_CHANNEL (RADIATOR_SLACK_TOKEN)	unset	A channel of the radiator's own (see A Slack channel for the new radiator)

Then:

• Run another suite against the same radiator, without restarting it: docker compose run --rm harness test-harness download sprint_4_tests
• Run a cycle (acceptance, then pathfinder, then performance) and get its board: docker compose run --rm harness test-harness-cycle --download --acceptance sprint_4_tests --pathfinder pathfinder_tests (see Run cycles).
• See the dashboard with made-up history instead (eg to look around without waiting for real runs): docker compose run --rm radiator python -m radiator.demo. Don't do this in a radiator you'll keep: the demo runs mix in with real ones.
• Stop it: docker compose down keeps the results (in the radiator-db volume) for next time; docker compose down -v wipes them.

If the harness exits with Permission denied: '/app/logs/harness.log' (a Linux host where ./logs isn't writable by the container's user), run chmod a+w logs (and remove logs/harness.log if it exists and isn't yours), then docker compose up again.

Reporting to a radiator from the harness

Point the harness at a radiator with RADIATOR_URL and RADIATOR_TOKEN (or --radiator_url / --radiator_token). If people open the radiator at a different address than the harness uploads to (eg an in-cluster service name), set RADIATOR_PUBLIC_URL for the links in the logs and Slack. Until the new radiator takes over, the harness reports to it alongside Zebrunner, if that's configured too.

Results are posted in batches; an unreachable radiator never fails a run. Whenever a run isn't uploaded (--local, the radiator isn't configured, or an upload failed) it's saved to --output_dir as radiator_<run id>.json instead, which can be uploaded later:

• test-harness-radiator push test_results/radiator_<run id>.json

Uploads are idempotent, so pushing a run that partly made it is safe. The radiator never depends on Zebrunner or Slack: if either fails, the run still reaches the radiator (or is saved for it).

When the radiator has the run, the Slack report also gets a headline against the previous run and a chart of the last 30 runs: for acceptance runs, the pass rate, regressions, and fixes, charted per agent; for performance runs, each service's max sustainable concurrency and checkpoint verdict. Runs of a run cycle also get the cycle's board when the cycle finishes, and the radiator's Weekly page and Monday digest cover every environment.

The dashboard's Run grid page shows a suite's runs in every environment over time, colored by whether the pass rate went up, down, or held. The dev environment is shown greyed out as excluded there; set RADIATOR_EXCLUDED_ENVS on the radiator (comma-separated) to change which.

A Slack channel for the new radiator

Which channel gets what depends on whether Zebrunner is still configured (ZE_BASE_URL and ZE_REFRESH_TOKEN):

• Zebrunner configured: SLACK_CHANNEL is still Zebrunner's. It gets the report it got before the new radiator (the Zebrunner link, the result summary and the result files) and nothing of the new radiator's. The new radiator's report goes to a channel of its own, if you give it one (below).
• No Zebrunner (after the cutover, or in Docker Compose without ZE_* in .env): SLACK_CHANNEL is the new radiator's and gets its report.

This keys off whether Zebrunner is configured, not whether it answered, so a Zebrunner outage never switches the old channel over to the new dashboard.

To give the new radiator its own channel while both run, set:

Setting
RADIATOR_SLACK_WEBHOOK_URL	An incoming webhook for the new channel (a webhook posts to one channel only)
RADIATOR_SLACK_CHANNEL	The new channel's ID, eg C0456EFGH
RADIATOR_SLACK_TOKEN	Optional: defaults to SLACK_TOKEN. Invite that bot to the new channel too

That channel then gets the new radiator's report for each run (its link, the headline against the previous run, the history charts and the result files), the run cycle boards and the weekly digest. Each run still runs once and reports to both dashboards. A failure posting to the new channel (eg the bot isn't in it) is logged and never costs the old channel its report. Without it, the new radiator still records every run; it just posts nothing to Slack while Zebrunner is configured.

When you retire Zebrunner, drop the ZE_* settings, point SLACK_* at the new channel, and drop RADIATOR_SLACK_*.

Deploying it

To run it for real, in the same Kubernetes namespace as the harness, use the Helm charts in deploy/helm: one for the radiator (and, if you need it, a small Postgres), one for the harness's scheduled run cycles, digest and sweeps. Copy each chart's values.example.yaml to a git-ignored values.<name>.yaml and install. The plain manifests in deploy/ show the same setup without Helm; see Deploying. To bring over the history from Zebrunner, see Importing from Zebrunner.

Running local test files

test-harness download <suite> fetches the suites from NCATSTranslator/Tests every run, which is no good when you want to change a CURIE and see what happens. A few small, editable suites ship in test_harness/test_suites/; run one with load instead of download:

• test-harness --local load local_acceptance

Suite	What it is
local_acceptance	Two acceptance cases — one MVP1 (drug treats disease), one MVP2 (chemical affects gene) — so --query_type has something to filter.
local_performance	One QuantitativeTest case, run by HelmsDeep.
local_pathfinder	One Pathfinder case: two pinned endpoints, expecting a named node on a path between them.

The suite name is the file's name without .json, the same way the downloaded suites are named, so the same name works with either subcommand.

The harness reads test_suites/ in the working directory when that exists — so a suite you're editing in your checkout is found without a flag — and otherwise the copy shipped with the package. --tests_dir overrides both:

• test-harness --local load my_suite --tests_dir ~/scratch/suites

These are ordinary Translator Testing Model TestSuite documents, identical in shape to the downloaded ones. Copy one, change the CURIEs, run it. See test_harness/test_suites/README.md for which fields actually drive behavior — in particular, a performance case's test_run_time/spawn_rate and its asset's query are inert, because HelmsDeep owns the ramp and the corpus (see Performance tests).

Running one type of query

A suite normally mixes the MVP query types. To run just one of them — handy for a local evaluation run of a change that only affects one — pass --query_type:

• test-harness --local --query_type MVP1 download <suite>

• MVP1 runs the drug treats disease queries (test assets with a biolink:treats predicate).

• MVP2 runs the chemical affects gene queries (biolink:affects).

• Without the flag, every test in the suite is run, as before.

Filtering happens per test asset, so a test case that mixes both types contributes only its matching assets, and a test case left with no assets is dropped. Pathfinder test cases don't generate an MVP query, so they're skipped whenever --query_type is given. Saved result filenames are prefixed with the query type (eg MVP1_test_report.json) so runs of different slices stay distinguishable.

TRAPI version

The harness speaks TRAPI 2.0 and 1.6. Queries are built with translator_tom (TOM), the Translator-wide TRAPI object model, so each query is validated against the spec before it is sent. --trapi_version picks the version the queries are written in and which services are taken from the SmartAPI registry (only those registered with the same x-trapi major.minor):

• auto (the default): the version the environment's ARS is registered with in SmartAPI (or, with --target, that service's), the newest the harness speaks if it's registered with more than one. While environments move to 2.0 one at a time, each run follows its environment with no configuration. The log says what it chose and why; if nothing is registered, or SmartAPI can't be read, it falls back to 2.0 with a warning.
• --trapi_version 1.6.0 or --trapi_version 2.0.0 to choose one, eg if an environment's SmartAPI registration and its deployment disagree for a while. In a run cycle, pass it after the -- (or, with the Helm chart, in that environment's harnessArgs).

Each run's TRAPI version is recorded in the Information Radiator, shown on the run and in the runs list, and the start of the Slack report says it. When a run's version differs from the previous run's (an environment that just moved to 2.0), the run page, What changed, the cycle board's flags and the Slack headline all say so: what changed may be the upgrade, not the services.

What 2.0 changed in the queries the harness sends: an MVP2 query's qualifiers go in the query edge's constraints.qualifiers as one {qualifier_type_id: qualifier_value} mapping (1.6's qualifier_constraints is rejected by 2.0 services), and a Pathfinder query is an ordinary query graph with paths and no edges. Responses are read in either shape — a 2.0 binding is one object with an ids list, a 1.x binding a list of objects each with an id — since the ARS can return either.

Sending parameters with every query

To send the same parameters with every query of a run, such as a longer timeout or a request to skip caches, pass them as a JSON object:

• test-harness --query_parameters '{"timeout": 300, "bypass_cache": true}' download <suite>

or keep them in a file and pass --query_parameters @params.json. The QUERY_PARAMETERS environment variable does the same, which is handier in .env for Docker Compose or in a CronJob; the flag wins if both are set.

Where they go depends on the TRAPI version:

• TRAPI 2.0: in the query's parameters object. The standard ones are timeout (seconds), log_level (ERROR, WARNING, INFO or DEBUG) and bypass_cache; a service can define its own as well, and those are passed through as given.
• TRAPI 1.6: as top-level fields of the query. 1.6 only has log_level and bypass_cache, so anything else is refused.

They're checked before the run starts: a value of the wrong type, an unknown log level, or a parameter the TRAPI version can't carry stops the run with a message saying which, instead of failing every query. The run's parameters are shown in its Slack message and recorded in the Information Radiator, on the run page.

They go to whatever the harness queries (the ARS, or the --target_url service). Performance tests don't use them: HelmsDeep sends its own queries.

Result files

The acceptance results CSV has a row per test asset, with the asset's name, its Information Radiator URL, the ARS pk, and the test case/asset ids. Every agent then contributes four columns:

• <agent> - the agent's status for the asset (PASSED, FAILED, NO_RESULTS, SKIPPED, ERROR).
• <agent>_found - true/false for whether the expected answer was anywhere in that agent's response, regardless of whether that made the test pass. It is blank when the agent never got far enough for the question to have an answer (it was skipped, or errored before returning results).
• <agent>_rank - the rank the expected answer came back at (the ARS's own rank for ARS results, the result's position for an ARA's results). Blank when the expected answer wasn't found.
• <agent>_score - the score the expected answer came back with (the ARS sugeno score for ARS results, the analysis score for an ARA's). Blank when the expected answer wasn't found.

These are the same numbers as the actual_output in the per-test report JSON uploaded to the Information Radiator.

Performance tests

Performance tests (QuantitativeTest cases) are run by HelmsDeep, installed from requirements-runners.txt. The harness does not measure anything itself: it picks the HelmsDeep run type for the test case, runs its CLI, and reports what came back.

HelmsDeep owns two things the harness used to own, and this is deliberate:

• The ramp. HelmsDeep's LoadTestShape drives users, spawn rate, and duration from its own per-layer stages table, so a test case's test_run_time and spawn_rate are ignored. A run takes as long as the run type's ramp says (tens of minutes to over an hour). HelmsDeep can compress a run to a wall-clock budget, but its own docs are explicit that a compressed run is indicative rather than a measurement, so the harness never asks for it.
• The queries. HelmsDeep sends its own corpus and varies the pinned entity per request, so the numbers cover a real cost surface instead of warming one cache. The test asset's TRAPI query is not what goes under load; the Information Radiator log for the test records the run plan (layer, ramp, SLO, checkpoints) instead, which is what you need to read the result.

Which run type a test gets

The layer follows the component: ars uses HelmsDeep's async submit/poll/merge targets, and every other component is queried as an ARA (a blocking POST /query). The stack cascades ARS → ARAs → KPs, so exactly one layer is loaded per run.

The query profile within that layer defaults to default — the layer's own single-class corpus, which answers the open question how far can we go? and reports a knee (max sustainable concurrency). Two heavier profiles are available:

• mixed — the 2:1 inferred/Pathfinder acceptance profile. This is the only profile that carries pass/fail checkpoints (does 30 concurrent hold? 45? does 60 avoid substantial failures?).
• pathfinder — the two-pinned-endpoint path queries, the heaviest class.

Select one with --performance_profile, or per test case with a mixed / pathfinder entry in its test_runner_settings; the flag wins over the test case.

• test-harness --local --performance_profile mixed download <suite>

What gets reported

Two artifacts per run are uploaded to Slack (or saved to --output_dir):

• <host>_<run_type>_summary.json — HelmsDeep's summary.json verbatim. This is the authoritative result: the knee, every stage's numbers, the checkpoint verdicts, measurement-quality warnings, and (for ARS runs) health and red flags.
• <host>_<run_type>_report.html — locust's own HTML report: charts and request/failure tables.

Both are uploaded with a comment carrying the run's checkpoint verdict, and the Slack summary message leads with the overall pass/fail. A run with no checkpoints reports its knee and says so explicitly rather than claiming a pass it never tested for. A missed checkpoint, or a HelmsDeep run that never produced a summary at all, is reported as FAILED in the Information Radiator too, so the dashboard agrees with Slack.

The rest of HelmsDeep's output (_stages.csv, _by_qtype.csv, _checkpoints.csv, and the ARS debug logs) is left on disk under --output_dir for digging in.

Sweeping several services in one scheduled job

HelmsDeep loads exactly one layer per run, and the harness resolves one target per invocation, so measuring the ARS and each ARA means several runs. They must not overlap: the stack cascades ARS → ARAs → KPs, so loading two layers at once double-loads whatever sits underneath and corrupts both measurements.

test-harness-sweep runs them sequentially in one process tree, which is what lets a single Kubernetes Job (or docker run, or a local shell) do the whole sweep:

test-harness-sweep --suite performance_tests --download \
    ars=https://ars.ci.transltr.io \
    aragorn=https://aragorn.ci.transltr.io \
    arax=https://arax.ci.transltr.io \
    bte=https://bte.ci.transltr.io \
    -- --performance_profile mixed

Each NAME=URL pair becomes one test-harness --target_url URL --target NAME invocation, in the order given, and the sweep waits for each to finish before starting the next. Anything after -- is passed through to every invocation.

Pass --tests_url (and --tests_dir) to the sweep, not through the --. They belong to the harness's download/load subcommands, and passthrough arguments go in front of the subcommand, where argparse rejects them; the sweep catches that and says so rather than letting the run die on "unrecognized arguments". Targets can come from the PERFORMANCE_TARGETS environment variable instead, as a comma-separated list of the same pairs — usually the easier knob in a CronJob, since changing the sweep is then a manifest edit rather than an image rebuild.

Two behaviors worth knowing:

• Every target runs even if an earlier one fails. A broken ARS shouldn't cost you the ARA numbers. The sweep exits non-zero if any target failed, and prints a per-target summary with exit codes and durations.
• Each target gets its own --output_dir subdirectory. HelmsDeep names its raw files after the run type and test case id, neither of which mentions the host, so every ARA in a sweep writes the same helmsdeep_aras_case_<id>_* names and would otherwise overwrite the previous target's files.

Exit status

The harness's exit status answers "did the run happen?", never "did the tests pass?":

• Exit 0 — the run happened, whatever it found. Failed acceptance tests, a missed performance checkpoint, a service that was down and produced no summary: those are results. They go to Slack and the Information Radiator, and a scheduled job stays green, because a slow ARA is news for the channel rather than a broken cron.
• Exit 1 — the harness couldn't carry out the run: --target_url without --target, a suite that isn't there, a suite that --query_type filtered down to nothing, or a crash. That job produced no results at all, which is exactly the failure that otherwise sits green and silent for months.

The sweep inherits this: it exits non-zero only if a harness invocation couldn't carry out its run, so a red Job always means something operational.

deploy/cronjob.example.yaml is a worked CronJob for this, with the fields a multi-hour job actually needs — concurrencyPolicy: Forbid so the next schedule can't start a second overlapping sweep, backoffLimit: 0 so a failure doesn't re-run hours of load, an activeDeadlineSeconds ceiling, PYTHONUNBUFFERED so kubectl logs shows progress, and CPU requests that keep the load generator from measuring its own scheduling delay.

Budget the time before you schedule it: with --performance_profile mixed a four-service sweep is roughly 70 min for the ARS plus 61 min per ARA (~4.2 hours); the single-class default profile is ~58 + ~35 × 3 (~2.7 hours). Nothing overlaps, by design.

Run cycles and the weekly digest

A run cycle is a weekly look at one environment: acceptance, then pathfinder, then a HelmsDeep performance run against that environment's ARS, one after another so they never load the services at the same time (about 2 hours with the mixed profile). The intended schedule is one environment a day: dev on Monday, ci on Tuesday, test on Wednesday, prod on Thursday.

test-harness-cycle --download \
    --acceptance sprint_4_tests --pathfinder pathfinder_tests \
    --performance performance_tests \
    --performance_target ars=https://ars.ci.transltr.io \
    --performance_profile mixed \
    -- --log_level INFO

Each step is an ordinary harness run with its usual Slack report. Leave out any step you don't want; anything after -- goes to every step (eg --query_parameters). When the last step finishes, the cycle posts the Information Radiator's board: each test against its previous run in that environment (pass rate or max concurrency, the change, regressions and fixes), with a verdict of Regressed, Improved, Mixed or Steady, the regressions and fixes by agent, and flags for oddities a pass rate hides:

• an agent returned no results on 3+ assets that had results last run;
• an agent errored on 3+ assets that didn't error last run;
• an agent's median response time at least doubled, and by 5s or more;
• the expected answer fell 30+ places while still passing;
• the ARS's max sustainable concurrency fell by 25% or more, its run failed, or it missed checkpoints it passed last run.

The cycle needs RADIATOR_URL/RADIATOR_TOKEN for the board (without them, the steps still run and report). Its exit status works like the sweep's. The test-harness Helm chart schedules a cycle per environment; deploy/cycle.example.yaml is the same CronJob without Helm.

Every Monday morning, test-harness-radiator digest posts the weekly digest: each environment's latest runs of acceptance, pathfinder and performance over the past week, against the run before, with their flags. It reads from the radiator only, so it's quick; the Helm chart schedules it too (or deploy/digest.example.yaml without Helm). The same view is the radiator's Weekly page, with a page per cycle.

Overriding the target service

Tests specify which component to run against (ars, ara, ...), and the harness normally resolves those components to deployed services through the SmartAPI registry. To run the tests against a service that isn't deployed yet — for example a locally running ARA you want to check before releasing — you can override the target with --target_url and --target:

• test-harness --local --target_url http://localhost:8080 --target aragorn download <suite>

With an override in place:

• Every query is sent directly to --target_url, regardless of the components specified in the tests, and the SmartAPI registry is not consulted.
• --target is the infores identifier of the service (with or without the infores: prefix). Any target other than ars is treated as a single service and queried with a POST to <target_url>/query; use --target ars to run a local ARS with the usual submit/poll flow.
• Pass/fail results are reported for the override target (instead of being driven by the ARS), and performance tests are pointed at the override URL.
• Saved result filenames (CSV/JSON results, performance artifacts, and the --json_output report) are prefixed with the target, e.g. aragorn_test_report.json, so runs against different services stay distinguishable.