Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 6 additions & 8 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -148,9 +148,9 @@ jobs:
# 3.15, its `requires-python` and classifiers), and the default (3.13, the
# newest in its security phase), which reports coverage with the entry
# marked `coverage`; all gate, as in FOCA. FOCA: the lowest supported
# version (0.18.0, the first with the plugin API), the other releases
# up to the newest (0.19.0, 0.20.0, 0.21.0, 0.22.0) and FOCA's `dev` branch, so that changes of the plugin API show
# before FOCA releases them (raise the lowest version with the plugins'
# version (0.19.0, the first with the plugin store), the other releases
# up to the newest (0.20.0, 0.21.0, 0.22.0) and FOCA's `dev` branch, so
# that changes of the plugin API show before FOCA releases them (raise the lowest version with the plugins'
# lower bounds, and the newest with their upper bounds)
test:
name: Test (${{ matrix.package.name }}, Python ${{ matrix.python-version }}, FOCA ${{ matrix.foca }})
Expand All @@ -163,14 +163,12 @@ jobs:
matrix:
package: ${{ fromJSON(needs.changes.outputs.packages) }}
python-version: ["3.11", "3.13", "3.15"]
foca: ["0.18.0", "0.19.0", "0.20.0", "0.21.0", "0.22.0", dev]
foca: ["0.19.0", "0.20.0", "0.21.0", "0.22.0", dev]
include:
- foca: "0.18.0"
foca-spec: "foca==0.18.0"
# The entry that reports coverage (with the default Python)
coverage: true
- foca: "0.19.0"
foca-spec: "foca==0.19.0"
# The entry that reports coverage (with the default Python)
coverage: true
- foca: "0.20.0"
foca-spec: "foca==0.20.0"
- foca: "0.21.0"
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,9 @@ repositories work the same way.

<!-- --8<-- [start:status] -->
**Status:** the plugin API is new in FOCA 0.18.0; the plugins support
FOCA 0.18 to 0.22. Plugins so far, both experimental and released as
0.1.x: `generic.routing.random` and
`tes.routing.distance`, plus the example `generic.example.noop`, the
FOCA 0.19 (the first release with the plugin store) to 0.22. Plugins so
far, both experimental and released as 0.1.x: `generic.routing.random`
and `tes.routing.distance`, plus the example `generic.example.noop`, the
template of new plugins (never published). The latest release train is
`foca-plugins` 0.3.0.
<!-- --8<-- [end:status] -->
Expand Down
10 changes: 5 additions & 5 deletions docs/writing-plugins.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,11 +138,11 @@ page](https://foca.readthedocs.io/en/latest/plugins/#testing-plugins).

CI tests each plugin with the lowest and highest Python versions that FOCA
supports and the default one (3.11, 3.13, 3.15), and with the lowest
supported FOCA version, 0.18.0, the first release with the plugin API (the
lower bound of every plugin, `foca>=0.18.0`), the later FOCA releases up
to the newest, 0.19.0, 0.20.0, 0.21.0 and 0.22.0 (the upper bounds are `<0.23`), and
FOCA's `dev` branch, from Git,
so that changes of the plugin API show before FOCA releases them. The
supported FOCA version, 0.19.0, the first release with the plugin store
(`ctx.store`; the lower bound of every plugin, `foca>=0.19.0`; 0.18.0, the
first release with the plugin API, is no longer supported), the later FOCA
releases up to the newest, 0.20.0, 0.21.0 and 0.22.0 (the upper bounds are
`<0.23`), and FOCA's `dev` branch, from Git, so that changes of the plugin API show before FOCA releases them. The
development environment has the newest FOCA release (`uv.lock`).

## Changes and releases
Expand Down
2 changes: 1 addition & 1 deletion meta/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ classifiers = [
]
# The FOCA versions that the train was tested with; set in the release pull
# request
dependencies = ["foca>=0.18.0,<0.23"]
dependencies = ["foca>=0.19.0,<0.23"]

# Extras per group (`<group>`) and subgroup (`<group>-<subgroup>`), plus
# `all` (stable plugins only) and `experimental` (experimental ones). Each
Expand Down
2 changes: 1 addition & 1 deletion plugins/generic/example/noop/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ and is **not published**.
| Maturity | experimental |
| Data transfers | none |
| Data stored | none (no plugin store) |
| FOCA | 0.18, 0.19, 0.20, 0.21, 0.22 (`>=0.18.0,<0.23`) |
| FOCA | 0.19, 0.20, 0.21, 0.22 (`>=0.19.0,<0.23`) |

## What it does

Expand Down
4 changes: 2 additions & 2 deletions plugins/generic/example/noop/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,9 @@ classifiers = [
# <<< template only
]
dependencies = [
# The plugin API of FOCA 0.18 to 0.22 (`foca.plugins`, API version 1);
# The plugin API of FOCA 0.19 to 0.22 (`foca.plugins`, API version 1);
# raise the upper bound once CI has tested the next minor version
"foca>=0.18.0,<0.23",
"foca>=0.19.0,<0.23",
"pydantic>=2.7,<3",
]

Expand Down
7 changes: 7 additions & 0 deletions plugins/generic/routing/random/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@ Features, Security, Fixes, Documentation, Maintenance; empty ones left
out), each entry with its issue and pull request numbers (see "Releasing"
in the docs).

## Unreleased

### Breaking changes

- Drop FOCA 0.18 (`foca>=0.19.0,<0.23`), as all plugins of the repository
do: `tes.routing.distance` needs FOCA 0.19's plugin store (#21).

## 0.1.3

Support for FOCA 0.21 and 0.22: the plugin now installs alongside FOCA
Expand Down
2 changes: 1 addition & 1 deletion plugins/generic/routing/random/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ proTES's random task distribution.
| Maturity | experimental |
| Data transfers | none |
| Data stored | none (no plugin store) |
| FOCA | 0.18, 0.19, 0.20, 0.21, 0.22 (`>=0.18.0,<0.23`) |
| FOCA | 0.19, 0.20, 0.21, 0.22 (`>=0.19.0,<0.23`) |

## What it does

Expand Down
4 changes: 2 additions & 2 deletions plugins/generic/routing/random/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,9 @@ classifiers = [
"Typing :: Typed",
]
dependencies = [
# The plugin API of FOCA 0.18 to 0.22 (`foca.plugins`, API version 1);
# The plugin API of FOCA 0.19 to 0.22 (`foca.plugins`, API version 1);
# raise the upper bound once CI has tested the next minor version
"foca>=0.18.0,<0.23",
"foca>=0.19.0,<0.23",
"pydantic>=2.7,<3",
]

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,7 @@ class Random(Plugin[RandomConfig, RandomParams]):
" the order depends only on the seed and the order of the candidates"
" (on the same Python version)."
)
# Nothing in the plugin store (listed by FOCA 0.19; declared here, as
# FOCA 0.18 has no such attribute)
# Nothing in the plugin store (listed in FOCA's plugin listing)
data_stored = ()

# Set by `setup()`, from the admin settings
Expand Down
2 changes: 1 addition & 1 deletion plugins/tes/data/crypt4gh/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ middleware of proTES.
| Maturity | experimental |
| Data transfers | none |
| Data stored | none (no plugin store) |
| FOCA | 0.18, 0.19, 0.20, 0.21, 0.22 (`>=0.18.0,<0.23`) |
| FOCA | 0.19, 0.20, 0.21, 0.22 (`>=0.19.0,<0.23`) |

## What it does

Expand Down
4 changes: 2 additions & 2 deletions plugins/tes/data/crypt4gh/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,9 @@ classifiers = [
"Typing :: Typed",
]
dependencies = [
# The plugin API of FOCA 0.18 to 0.22 (`foca.plugins`, API version 1);
# The plugin API of FOCA 0.19 to 0.22 (`foca.plugins`, API version 1);
# raise the upper bound once CI has tested the next minor version
"foca>=0.18.0,<0.23",
"foca>=0.19.0,<0.23",
"pydantic>=2.7,<3",
]

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -155,8 +155,7 @@ class Crypt4gh(Plugin[Crypt4ghConfig, EmptyModel]):
" Crypt4GH inputs as they are."
)
data_transfers = ()
# Nothing in the plugin store (listed by FOCA 0.19; declared here, as
# FOCA 0.18 has no such attribute)
# Nothing in the plugin store (listed in FOCA's plugin listing)
data_stored = ()

settings: Crypt4ghConfig
Expand Down
24 changes: 24 additions & 0 deletions plugins/tes/routing/distance/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,30 @@ Features, Security, Fixes, Documentation, Maintenance; empty ones left
out), each entry with its issue and pull request numbers (see "Releasing"
in the docs).

## Unreleased

### Breaking changes

- Require FOCA 0.19 (`foca>=0.19.0,<0.23`), for its plugin store (#21).
- With `provider: dbip`, each caller now gets at most 100 uncached
lookups per day by default (`dbip_user_daily_lookups`; set `null` for
the earlier behaviour), and the plugin keeps locations and lookup counts
in FOCA's plugin store: with a database, in its collection
`plugin_store`, as personal data (hashed keys) for up to `cache_ttl` or a
day; set FOCA's `plugins.defaults.store.key_secret` and review
`max_ttl` before upgrading (#21).

### Features

- With `provider: dbip`, keep the locations of addresses (also misses and
failures) in FOCA's plugin store (`ctx.store`), shared by all processes
and replicas, instead of per process (#21).
- With `provider: dbip`, limit the lookups of each caller to
`dbip_user_daily_lookups` (default 100) per day, counted in the plugin
store by a hash of the caller's identity; `null` turns the limit off
(#21).
- Declare the plugin's entries in the plugin store (`data_stored`) (#21).

## 0.1.3

Support for FOCA 0.21 and 0.22: the plugin now installs alongside FOCA
Expand Down
69 changes: 54 additions & 15 deletions plugins/tes/routing/distance/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ successor of proTES's distance-based task distribution.
| Capabilities | `routing-by-data-locality` |
| Maturity | experimental |
| Data transfers | none by default; with `provider: dbip`, db-ip.com and Nominatim (see [Data transfers](#data-transfers)) |
| Data stored | none (no plugin store) |
| FOCA | 0.18, 0.19, 0.20, 0.21, 0.22 (`>=0.18.0,<0.23`) |
| Data stored | none by default; with `provider: dbip`, located addresses and daily lookup counts per caller, by hashes (see [Data stored](#data-stored)) |
| FOCA | 0.19, 0.20, 0.21, 0.22 (`>=0.19.0,<0.23`) |

## What it does

Expand Down Expand Up @@ -57,19 +57,28 @@ Annotations, counts only (hosts, addresses and places of inputs could be
personal data): `inputs` (input URLs), `input_hosts`, `located_inputs`
(input URLs with a location), `located_backends`, `lookups` (by the
provider), `cache_hits`, `failed_lookups` and `skipped_lookups`
(addresses not looked up for lack of time or beyond `dbip_max_lookups`).
(addresses not looked up for lack of time, beyond `dbip_max_lookups` or
beyond the caller's `dbip_user_daily_lookups`).
The order is in the provenance of each request (`route.candidates_out`).

### Cache and limits

Each process caches the lookups of addresses, also misses (addresses
without location), so that backends are looked up once per process until
the entry expires, not per task: `cache_ttl` (1 day) for locations,
The plugin caches the lookups of addresses, also misses (addresses
without location), so that backends are looked up once until the entry
expires, not per task: `cache_ttl` (1 day) for locations,
`cache_miss_ttl` (1 hour) for misses and `cache_failure_ttl` (1 minute)
for failed lookups (e.g., a service is down; not lookups that only ran
out of time), at most `cache_size` addresses (the least recently used go
first). The cache doesn't use FOCA 0.19's plugin store yet (#21), so it
is not shared between processes.
out of time).

- With the `offline` provider, each process caches at most `cache_size`
addresses (the least recently used go first).
- With `dbip`, the cache is in FOCA's plugin store (`ctx.store`), shared
by all processes and replicas if the gateway has a database (otherwise
per process, see FOCA's [plugin
store](https://foca.readthedocs.io/en/latest/configuration/plugins/#plugin-store)).
Its entries are limited by the TTLs above, capped by the operator's
`max_ttl` of the store, not by `cache_size` (0 still turns it off). If
the store fails or is full, the call goes on without the cache.

Name resolution can't be interrupted, and hosts of inputs come from
clients, so slow or hostile name servers must not tie up the gateway's
Expand Down Expand Up @@ -118,12 +127,13 @@ know them: they are exact, and need no lookup.
| `config.nominatim_interval` | seconds, 0 to 60 | `1.0` | time between requests to Nominatim, per process (its policy: at most 1 per second) |
| `config.dbip_backoff` | seconds, 0 to 86400 | `600` | time to ask DB-IP nothing after it refused a lookup, e.g., over its daily quota (`dbip`) |
| `config.dbip_max_lookups` | 1 to 1000 | `5` | uncached lookups per call, for inputs and for backends each (`dbip`), so that no single task uses up DB-IP's quota |
| `config.dbip_user_daily_lookups` | 1 to 1000000, or `null` | `100` | uncached lookups per caller and day, counted in FOCA's plugin store (`dbip`), so that no single user uses up DB-IP's quota; `null` for no limit |
| `config.locate_backends` | boolean | `true` | whether backends without configured location are located by their address |
| `config.max_hosts` | 1 to 1000 | `20` | input hosts to locate per task |
| `config.resolve_timeout` | seconds, up to 60 | `2.0` | time to resolve the host names of a call, in total: half for inputs, the rest for backends (and never beyond the step's deadline) |
| `config.resolve_threads` | 1 to 256 | `16` | host names resolved at a time, per process, for inputs and for backends each (a call: at most a quarter for its inputs) |
| `config.resolve_cache_ttl` | seconds, ≥ 0 | `300` | time to keep the addresses of host names, also of names that couldn't be resolved |
| `config.cache_size` | ≥ 0 | `10000` | entries of each cache (addresses; host names of inputs and of backends; with `dbip`, places and coordinates); 0 turns the caches off |
| `config.cache_size` | ≥ 0 | `10000` | entries of each cache of a process (addresses with `offline`; host names of inputs and of backends; with `dbip`, places and coordinates); 0 turns the caches off, also that of addresses in the plugin store (`dbip`) |
| `config.cache_ttl` | seconds, ≥ 0 | `86400` | time to keep located addresses |
| `config.cache_miss_ttl` | seconds, ≥ 0 | `3600` | time to keep addresses without location |
| `config.cache_failure_ttl` | seconds, ≥ 0 | `60` | time to keep addresses whose lookup failed as unlocated |
Expand Down Expand Up @@ -188,11 +198,21 @@ public instance one per second. The plugin caches places and coordinates,
looks up at most `dbip_max_lookups` uncached addresses per call (for
inputs and for backends each), asks DB-IP only if Nominatim has a turn
before the deadline, and backs off for `dbip_backoff` when DB-IP refuses
lookups. These limits can't keep users who submit many tasks with fresh
public addresses from using up DB-IP's daily quota; distance routing then
doesn't apply (the next alternative routes) until the quota resets. Use
`offline` where that matters, and your own Nominatim instance
(`nominatim_url`) if you can.
lookups. Each caller may make at most `dbip_user_daily_lookups` (100)
uncached lookups per day (a fixed window from the caller's first lookup),
counted in FOCA's plugin store by a hash of the caller's identity
(`subject_type` and `subject`; all anonymous callers share one count, and
a service client has one, whichever users it acts for; lookups that sent
no request, e.g., while DB-IP is backed off, don't count); if the store
fails, the call makes no lookups (the budget fails closed). Without a
database, the counts are per process, so the limit is only approximate.
The window is capped by the operator's `max_ttl` of the store: keep it at
a day or more, or the limit holds per `max_ttl` instead of per day.
These limits make it harder, but can't keep users from using up DB-IP's
daily quota (e.g., several users together); distance routing then doesn't
apply (the next alternative routes) until the quota resets. Use `offline`
where that matters, and your own Nominatim instance (`nominatim_url`) if
you can.

DB-IP's page of the free API asks for no attribution; check DB-IP's terms
of service for your use. Nominatim's data is © OpenStreetMap contributors
Expand All @@ -213,6 +233,25 @@ of service for your use. Nominatim's data is © OpenStreetMap contributors
e.g., in your gateway's privacy notice. Clients see these transfers in
the plugin listing (`data_transfers`), whatever the provider.

## Data stored

- **Default (`offline`):** nothing in FOCA's plugin store.
- **`provider: dbip`**, in FOCA's plugin store (`ctx.store`):
- the locations (latitude and longitude) of the public IP addresses of
the hosts of task inputs and of backends, also misses, keyed by a hash
of the address, for up to `cache_ttl`;
- daily lookup counts per caller, keyed by a hash of the caller's
identity, for a day (only with `dbip_user_daily_lookups`).

Keys are hashes from `ctx.store.key_for()` (HMACs, if the operator set
FOCA's key secret, `plugins.defaults.store.key_secret`), never the
addresses or identifiers as they are. They pseudonymize, but don't
anonymize: the entries remain personal data, kept at most for the
operator's `max_ttl` of the store. Clients see them in the plugin
listing (`data_stored`). Set the key secret with `provider: dbip`:
without it, anyone who can read the database (or a backup) can recover
the addresses by hashing all IPv4 addresses.

## Installation

```bash
Expand Down
4 changes: 2 additions & 2 deletions plugins/tes/routing/distance/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,9 @@ classifiers = [
"Typing :: Typed",
]
dependencies = [
# The plugin API of FOCA 0.18 to 0.22 (`foca.plugins`, API version 1);
# The plugin API of FOCA 0.19 to 0.22 (`foca.plugins`, API version 1);
# raise the upper bound once CI has tested the next minor version
"foca>=0.18.0,<0.23",
"foca>=0.19.0,<0.23",
"pydantic>=2.7,<3",
# Reads the offline GeoIP database (MaxMind DB format, e.g., DB-IP Lite)
"maxminddb>=2.6,<4",
Expand Down
Loading
Loading