diff --git a/RETENTION_PLAN.md b/RETENTION_PLAN.md new file mode 100644 index 0000000000..74b5f6359e --- /dev/null +++ b/RETENTION_PLAN.md @@ -0,0 +1,148 @@ +# Data retention — policies, holds, runs, archive, disposal register + +Written 2026-11-03. Phases 1 and 2 are built; phase 3 is described at the end and not built. +Working rules as elsewhere: the user commits, the assistant never does. + +## Why + +Records must be kept for as long as a records authority, a regulator or the deployment's own rules say, +and no longer (data minimisation). Those periods differ per deployment and change over time, so they are +configuration held per space, never code. Every disposal must be accounted for, nothing may be disposed of +while it is on legal hold, and the destructive steps must be impossible to switch on by accident. + +## The model + +A **retention policy** belongs to one space: a bank, or `SYS` for the system space (the same meaning of +`SYS` as in `DYNAMIC_ENTITY_SPACE_MODEL_PLAN.md`). It governs one resource type of that space: + +| resource_type | live tables | date_basis | notes | +|---|---|---|---| +| `dynamic_entity_record` | `DynamicData` (+ `DynamicDataAccess` rows of the record, the SQL projection row) | `created`, `updated`, `field:` (ISO 8601 date in the record) | needs `entity_name`; condition on the record's own fields | +| `file` | `File`, `FileContent`, `FileAttachment`, `FileAccess` | `created` | | +| `chat_message` | `ChatMessage`, `Reaction` | `created`, `updated` | the room's last-message preview is cleared if it came from a disposed message | +| `chat_room` | `ChatRoom`, `Participant`, and all the room's messages and reactions | `created`, `updated`, `last_activity` | a room holding a message on hold stays whole | +| `customer` | `MappedCustomer`, attributes, addresses, dependants, tax residences, `MappedUserCustomerLink`, `CustomerAccountLink` | `created`, `updated` | bank only, never `SYS` | +| `consent` | `MappedConsent`, `consent_item`, consent auth context | `created`, `updated`, `expires` (default: the token's expiry) | a bank's policy governs Consents whose items name only that bank; `SYS` governs Consents naming no bank. Identified by `consent_reference_id`; the consent id, secret and JWT are never copied | + +Fields of a policy: `retain_days` (how long the resource stays live after its trigger date, at least +`retention.minimum_retain_days`), `action`, +`archive_days`, an optional `condition` (`field` = `value`, compared as text; for dynamic entity records on +the record's own fields, for the other types on the fields of the resource's JSON copy), `cascade`, +`requires_approval`, `enabled`, `name`, `description`. + +**Actions.** + +* **archive** — the resource leaves the live tables and is written to the **retention archive** + (`RetentionArchive`, a JSON copy; a file's bytes, Base64 in a text column, in `RetentionArchiveContent`). The archive is + read-only, reachable only through the retention archive endpoints with `CanGetRetentionArchive`, and + is **not an undo**: there is no restore. After `archive_days` the entry is deleted. `archive_days` is + required for archive and not allowed for the other actions. +* **delete** — final. +* **anonymise** — customers only. The personal fields (legal name becomes `ANONYMISED`, email, phone, + date of birth, face image, title, name suffix, relationship status, education, employment, credit) are + cleared and attributes, addresses, dependants, tax residences and links to Users are deleted; the + customer id, bank, customer number and links to accounts stay, so references stay valid. An anonymised + customer is not matched again. + +**Cascade.** A disposed record's file attachments always go. With `cascade`, each attached file goes +too, with the record's action (archive stays archive; delete and anonymise delete the file), unless it is +still attached to another record or is on hold. A dynamic entity record's files are attached with the +entity name as `record_type`; a customer's with `customer`. (No endpoint writes `FileAttachment` yet, so +this is the convention the file endpoints must follow.) + +**Legal holds** (`RetentionHold`) name one resource (`resource_type` + `resource_id`, optionally +`entity_name`), or every record of one dynamic entity (`entity_name` alone). Every run skips what is held +and counts it; an archived resource on hold is not deleted from the archive. Releasing a hold keeps the +row with who released it and when. + +**Runs** (`RetentionRun`) have a mode, `preview` or `execute`, and a status: `pending` (waiting for a +second user), `running`, `done`, `refused` (props off, rejected by the approver, or policy deleted) or +`failed` (some resources failed and were left in place; see `message`). One execution disposes of at +most `retention.run.max_items` resources, oldest row first, then deletes from the archive what that +policy archived whose archive period has ended. The counts include cascaded files and a room's messages. + +**Four eyes.** With `requires_approval`, an execution is created as pending; a different user holding +`CanApproveRetentionRun` approves (it executes then) or rejects it. The requester can never decide their +own run. A run the scheduler requested may be approved by anyone holding the Role. + +**Disposal register** (`RetentionRunItem`): one row per resource archived, anonymised or deleted, and per +archive entry deleted (`delete_from_archive`): ids, owner, action, time, parent, archive id — never +content, so it outlives the resources. + +No webhook is sent for a disposal: a retention run is not a write through the endpoints. + +## Safety switches (props, documented in `sample.props.template`) + +| prop | default | effect | +|---|---|---| +| `retention.archive.enabled` | false | an execution of an `archive` policy is refused (400, recorded as a `refused` run) | +| `retention.delete.enabled` | false | an execution of a `delete` or `anonymise` policy is refused; archived entries past their period are kept | +| `retention.minimum_retain_days` | 1825 | a policy with a shorter `retain_days` is refused; an existing one runs as though its `retain_days` were this (five years by default) | +| `retention.scheduler.enabled` | false | start `RetentionScheduler` | +| `retention.scheduler.interval_in_seconds` | 3593 | seconds between scheduler runs | +| `retention.run.max_items` | 1000 | most resources one run disposes of | + +A preview always works. The scheduler, modelled on `MetricsArchiveScheduler` (a `JobScheduler` lock row), +executes the enabled policies that do not require approval and have something due (skipping, with a log +line, those the props switch off), creates one pending run for each enabled policy that requires approval +and has something due, and, with `retention.delete.enabled`, deletes every archive entry whose period has +ended — whichever policy archived it, so deleting a policy never strands its archive. + +## Endpoints (v7.0.0; BANK_ID is a bank id or `SYS`) + +| verb | path | Role(s), held at BANK_ID | +|---|---|---| +| POST | `/management/banks/BANK_ID/retention-policies` | CanManageRetentionPolicy | +| GET | `/management/banks/BANK_ID/retention-policies` | CanGetRetentionPolicies or CanManageRetentionPolicy | +| GET | `/management/banks/BANK_ID/retention-policies/RETENTION_POLICY_ID` | CanGetRetentionPolicies or CanManageRetentionPolicy | +| PUT | `/management/banks/BANK_ID/retention-policies/RETENTION_POLICY_ID` | CanManageRetentionPolicy | +| DELETE | `/management/banks/BANK_ID/retention-policies/RETENTION_POLICY_ID` | CanManageRetentionPolicy | +| GET | `/management/banks/BANK_ID/retention-policies/RETENTION_POLICY_ID/preview` | CanGetRetentionPolicies, CanRunRetentionPolicy or CanManageRetentionPolicy | +| POST | `/management/banks/BANK_ID/retention-policies/RETENTION_POLICY_ID/runs` | CanRunRetentionPolicy | +| GET | `/management/banks/BANK_ID/retention-runs` | CanGetRetentionRuns | +| GET | `/management/banks/BANK_ID/retention-runs/RETENTION_RUN_ID` | CanGetRetentionRuns | +| PUT | `/management/banks/BANK_ID/retention-runs/RETENTION_RUN_ID/approval` | CanApproveRetentionRun | +| GET | `/management/banks/BANK_ID/retention-runs/RETENTION_RUN_ID/items` | CanGetRetentionRuns | +| POST | `/management/banks/BANK_ID/retention-holds` | CanManageRetentionHold | +| GET | `/management/banks/BANK_ID/retention-holds` | CanManageRetentionHold or CanGetRetentionPolicies | +| DELETE | `/management/banks/BANK_ID/retention-holds/RETENTION_HOLD_ID` | CanManageRetentionHold (release) | +| GET | `/management/banks/BANK_ID/retention-archive` | CanGetRetentionArchive | +| GET | `/management/banks/BANK_ID/retention-archive/RETENTION_ARCHIVE_ID` | CanGetRetentionArchive | + +All seven Roles are single names with `requiresBankId = true`, granted at a bank or at `SYS`, per +`ANY_BANK_ROLE_REMOVAL_PLAN.md`: a Role targets one space. Everything is looked up within the space of the +URL. Code: `code.retention` (tables, engine, one handler per resource type), `code.scheduler.RetentionScheduler`, +`code.api.v7_0_0.Http4s700Retention` and `JSONFactory700Retention`; errors OBP-30609 to OBP-30623. +Tests: `code.api.v7_0_0.RetentionTest`. + +## Tables (all new; no column added to an existing table) + +`RetentionPolicy`, `RetentionArchive`, `RetentionArchiveContent`, `RetentionHold`, `RetentionRun`, +`RetentionRunItem`. Created by Schemifier from `ToSchemify.models`; no migration is needed. Their user id +columns are in `UserReference` as audit columns (`UseAuthenticatedUserId`); the owner columns are copies +of the disposed resource's owner, never the caller. + +## Known limits + +* An HTTP-started run executes inside the request's transaction, so a large run holds one transaction; + keep `retention.run.max_items` moderate. Scheduler runs use auto-commit statements, like the metrics + archive. +* `FileContent` keeps bytes in a `MappedBinary`, which Lift maps to a one-byte `BINARY` on H2, so on H2 (the + test databases) a file of more than one byte cannot be stored at all; the retention archive avoids the + problem by storing Base64 text. Postgres is unaffected. +* A preview counts at most 100000 resources (`count_is_capped`). +* The `field:` date basis and conditions scan the entity's records in pages; there is no index on + JSON fields. +* Consents are matched in memory against their items; a Consent naming several banks is governed by no + bank's policy (and not by `SYS`). + +## Phase 3 — Users (not built) + +Users are the identity other records point at, so they are **anonymised only**, never archived or deleted: +`resource_type: user`, action `anonymise` only, at `SYS` (a User is not owned by a bank). It would clear the +personal fields of `ResourceUser` and `AuthUser` (name, email, username replaced by a generated +placeholder, phone), revoke tokens and Consents, delete the User's attributes, auth contexts and +user-customer links, lock the login, and keep the `user_id` so every reference stays valid. Trigger dates: +`created`, `last_login`, or `field:` equivalents on user attributes. Holds and the register work unchanged. +It needs its own decisions first: what counts as inactivity, how it interacts with consent users +(`ON_BEHALF_OF_USER_ID_PLAN.md`), and whether metrics rows keep the user id. diff --git a/ideas/ASSET_REGISTRY.md b/ideas/ASSET_REGISTRY.md index 35b8b0a7ae..44b338ef59 100644 --- a/ideas/ASSET_REGISTRY.md +++ b/ideas/ASSET_REGISTRY.md @@ -1,6 +1,6 @@ # Draft v4: Asset Registry -**Status**: partly implemented. The registry exists, is seeded, and decides which currency codes are accepted and their decimal places, giving the same answers as the built-in list it replaced. The only changes callers can see so far are two attribute types (`DECIMAL`, `BOOLEAN`) and an amount too large to store being refused instead of wrapped. The progress section below lists what is built and what is next; section 12 lists the open questions. +**Status**: partly implemented. The registry exists, is seeded, and decides which currency codes are accepted and their decimal places, giving the same answers as the built-in list it replaced. The only changes callers can see so far are two attribute types (`DECIMAL`, `BOOLEAN`) and an amount too large to store being refused instead of wrapped. The progress section below lists what is built and what is next; section 12 records how the questions it raised were settled. **Changes in v4 (2026-10-05)**: adds the progress section. `chain_scheme` names the network as well as the chain (§2); questions 10 and 11 are new. Records what the implementation settled or exposed: the seed has no switch (question 6), Mapper cannot declare the partial unique indexes (§2), the 4–10 character rule for non-seeded codes is not enforced yet (§3), the test setup empties the registry between tests (§11), and the unused `MappedCurrency` table (§11). Code references are brought up to date. @@ -54,14 +54,15 @@ The work is ordered so that each step changes nothing a caller can see until the | Step | What | State | |---|---|---| | 1 | Tests that describe today's currency behaviour: `CurrencyHandlingTest` (`code.util`) | Done (`26e9f9b39`) | -| 2 | Read-only report of stored amounts per currency code: `scripts/asset_registry_currency_report.sql` | Done, not committed | +| 2 | Read-only report of stored amounts per currency code: `scripts/asset_registry_currency_report.sql` | Done (`b4ac8c374`) | | 3 | `DECIMAL` and `BOOLEAN` attribute types (§9) | Done (`26e9f9b39`) | -| 4 | `Asset` and `AssetStatusHistory` tables, provider and boot seed (§2, §6, §7) | Done, not committed | -| 5 | Read-only endpoints `GET /assets`, `GET /assets/ASSET_CODE`, reverse chain lookup (§8), and the Glossary entry (§11) | Done, not committed | -| 6 | `isValidCurrencyISOCode` and `currencyDecimalPlaces` read from the registry (§4) | Done, not committed | -| 7 | Case-insensitive codes (§4): codes upper-cased in requests by `ResourceDocMiddleware`, compared ignoring case | Done, not committed | -| 8 | Rejecting excess decimals (§4): 400 (OBP-10068) from `ResourceDocMiddleware`, crypto assets exempt until §7 | Done, not committed | -| Later | write endpoints (§8), amount storage (§5, §7 Part A), precision corrections (§7 Part B), folding `lovelace` and `wei` (§7 Part C, §10) | Each changes behaviour or waits on an open question | +| 4 | `Asset` and `AssetStatusHistory` tables, provider and boot seed (§2, §6, §7) | Done (`b4ac8c374`) | +| 5 | Read-only endpoints `GET /assets`, `GET /assets/ASSET_CODE`, reverse chain lookup (§8), and the Glossary entry (§11) | Done (`b4ac8c374`) | +| 6 | `isValidCurrencyISOCode` and `currencyDecimalPlaces` read from the registry (§4) | Done (`b4ac8c374`) | +| 7 | Case-insensitive codes (§4): codes upper-cased in requests by `ResourceDocMiddleware`, compared ignoring case | Done (`8763728f6`) | +| 8 | Rejecting excess decimals (§4): 400 (OBP-10068) from `ResourceDocMiddleware`, crypto assets exempt until §7 | Done (`91289dbf0`) | +| 9 | Create Asset, `POST /banks/BANK_ID/assets` (§8) | Done, not committed | +| Later | the other write endpoints (§8), amount storage (§5, §7 Part A), precision corrections (§7 Part B), folding `lovelace` and `wei` (§7 Part C, §10) | Each changes behaviour; the open questions they waited on are settled (§12) | **Step 1.** `CurrencyHandlingTest` is a pure unit test (no server, no database). It asserts correct behaviour only. The three known defects are written as the correct expectation inside `pendingUntilFixed`, so they report as pending now and fail, demanding the wrapper's removal, once fixed: - codes are not accepted in every letter case; @@ -118,6 +119,16 @@ The registry is read once into memory and read again after any write through `As `CurrencyCodesTest` (unit) covers the rule. The two letter-case defects in `CurrencyHandlingTest` are no longer pending. `AssetLookupTest`, `FundsAvailableTest` (lower-case `eur` now gives 200) and `ExchangeRateTest` (an FX rate created with `eur`/`usd` is stored as `EUR`/`USD` and read back with lower-case path segments) were updated. +**Step 9.** `Http4s700Assets.createAsset` serves `POST /obp/v7.0.0/banks/BANK_ID/assets` with the new bank-scoped Role `CanCreateAsset`, checked at BANK_ID; the ResourceDoc declares `allowSystemSpace()`, so `SYS` reaches it with no bank looked up: +- **At a bank** only the issued types (§3) are accepted, and the bank is stored as `issuer_bank_id`. **At `SYS`** only `CRYPTO`, with no issuer and no `issuer_product_code`. +- **Rules in `Assets.createAsset`**, so no other writer can skip them: an issued type's code is 4 to 10 characters (§3); the name is 1 to 125 characters; `chain_scheme` and `chain_asset_id` come together or not at all; `chain_scheme` is one of `ChainSchemes.all` (`CARDANO_MAINNET`, `CARDANO_PREPROD`, `CARDANO_PREVIEW`, `ETHEREUM_MAINNET`, `ETHEREUM_SEPOLIA`); `dti` is 9 letters or digits. A broken rule is 400 (OBP-30582, `InvalidAsset`) with the rule in the message. +- **Conflicts** are 409: a taken code (OBP-30583), chain identity (OBP-30584) or DTI (OBP-30585). The endpoint checks the last two itself, since the partial unique indexes are not declared (§2). +- **`issuer_product_code`** must name a product at BANK_ID, else 404 (OBP-30011). +- **The creator** is stored through `attributedUserId` with `UserReference.Asset_CreatedByUserId`, so a consent user's asset is stored against the person it acts for; the seed keeps `system:asset-seed`. +- **Decimal places** are capped at 18 until §5 is built, because amounts are still stored as `Long` minor units. + +A created code is accepted in `currency` fields at once (`AssetLookup.invalidate`). A 3-character `CRYPTO` code such as `SOL` is allowed at `SYS`; nothing yet stops a future ISO 4217 code from colliding with one. `AssetsEndpointTest` (CI shard 6) covers 401, 403, the Role at another bank, 201 at a bank and at `SYS`, the 409s, the 400 rules and the product check. The Glossary "Asset" entry describes adding an Asset. + **Step 8.** An amount with more decimal places than its currency allows is refused with 400 (OBP-10068, `InvalidAmountPrecision`) instead of being cut off when stored. `code.asset.AmountPrecision` finds the amounts; `ResourceDocMiddleware.validateAmountPrecision` runs it as the last validation step, after authentication and Roles, so it applies to every static endpoint in every version: - **Where an amount is found.** In a JSON body, an object with an `amount` field (any letter case) next to its currency: the field named `currency`, or else the object's only currency-named field. In a query string, an `amount` parameter next to a currency parameter. Numbers are parsed exactly (`useBigDecimalForDouble`). Trailing zeros do not count. - **Not checked.** Unknown codes and non-numbers (left to the endpoint), and crypto assets: `AssetLookup.enforcedDecimalPlaces` is None for a registry `CRYPTO` asset and for `lovelace` and `wei`. Their registered decimal places are still the built-in 2, so enforcing them would refuse 0.001 ETH. They are checked once §7 records their real precision. @@ -168,7 +179,7 @@ A holder at bank B holding units issued by bank A has an account at B with `curr Indexes: unique `(asset_code)`; unique `(chain_scheme, chain_asset_id)` where not null; unique `(dti)` where not null; index `(issuer_bank_id)`. -*As built (step 4)*: the table is `Asset`, with Mapper column names (`AssetCode`, `DecimalPlaces`, `CreationDate`, `LastUpdate` and so on) and a unique index on `AssetId` as well. Optional columns hold an empty string rather than null when unset. The two partial unique indexes are not declared: they must skip unset values, and Mapper can only declare a plain unique index, which would treat every unset (empty) value as a duplicate of every other. Nothing writes these columns yet; the write endpoints (§8) must check uniqueness themselves, or a migration must create the partial indexes in SQL for each database. +*As built (step 4)*: the table is `Asset`, with Mapper column names (`AssetCode`, `DecimalPlaces`, `CreationDate`, `LastUpdate` and so on) and a unique index on `AssetId` as well. Optional columns hold an empty string rather than null when unset. The two partial unique indexes are not declared: they must skip unset values, and Mapper can only declare a plain unique index, which would treat every unset (empty) value as a duplicate of every other. Create Asset (step 9) checks uniqueness itself before it writes; a migration could still create the partial indexes in SQL for each database, which would also close the race between two simultaneous creates. **`chain_scheme` names the network as well as the chain.** The same contract address can exist on Ethereum mainnet, on a test network such as Sepolia, and on networks built on Ethereum such as Polygon or Arbitrum, each time as a different token; Cardano has the same split between mainnet and its preprod and preview test networks. So `(chain_scheme, chain_asset_id)` is only a unique key if the scheme says which network. The form is `_`, upper case, and the network is always written, even for mainnet: OBP does not know which network it is on (it is whatever node `ethereum.rpc.url` or the Cardano wallet API points at), so a bare `CARDANO` would be mainnet on one deployment and preprod on another. A network built on Ethereum is its own chain: `POLYGON_MAINNET`, `ARBITRUM_MAINNET`. The accepted values are a fixed list in code, extended when OBP supports another network. @@ -191,7 +202,7 @@ This departs from the account routing vocabulary, where a Cardano address uses t | `FUND_SHARE` | bank | issuer | `CanCreateAsset` at the issuer | | `OTHER` | bank | issuer | `CanCreateAsset` at the issuer | -Codes for non-seeded types must be 4–10 characters, so they cannot collide with a current or future ISO 4217 code. This is not enforced yet: `Assets.createAsset` accepts 3 to 10 characters for every type, because the seed uses it for 3-letter ISO codes. The create endpoint (§8) must enforce it. +Codes for non-seeded types must be 4–10 characters, so they cannot collide with a current or future ISO 4217 code. `Assets.createAsset` enforces it for the issued types since step 9; `CRYPTO` keeps 3 to 10, because the seed and crypto codes such as `SOL` use 3. ## 4. Validation changes @@ -210,7 +221,7 @@ Codes for non-seeded types must be 4–10 characters, so they cannot collide wit Today amounts are stored as `Long` minor units (`(amount * 10^dp).toLong`). A `Long` holds at most about 9.22 × 10^18, which cannot hold exact amounts for high-precision assets. At 18 decimal places (ETH, and many Ethereum tokens such as DAI) the largest balance a `Long` can hold is about 9.2 ETH, while total ETH supply is about 1.2 × 10^26 wei. Any cap on `decimal_places` small enough to fit a `Long` means incoming on-chain amounts below the cap cannot be credited exactly, and OBP's ledger stops matching the chain. Rounding hides that; rejecting does not help either, because the funds are already in the bank's wallet. -So the amount columns change from `Long` minor units to **`DECIMAL(38, 18)` holding the amount in the asset's main unit** (`12.45` EUR, not `1245` cents): +So the amount columns change from `Long` minor units to **an exact decimal holding the amount in the asset's main unit** (`12.45` EUR, not `1245` cents): `DECIMAL(60, 24)` on PostgreSQL and H2, and `DECIMAL(38, 18)` on SQL Server, Oracle and MySQL (see "Width per database" below). The table writes `DECIMAL(38, 18)` for short: | Column | Today | After | |---|---|---| @@ -221,7 +232,9 @@ So the amount columns change from `Long` minor units to **`DECIMAL(38, 18)` hold | `ProductFee.Amount` | `MappedDecimal(DECIMAL128, 2)` | same | - **Why main units.** The stored value no longer depends on `decimal_places`, so correcting an asset's precision (JOD 2 → 3) changes no stored data (§7). With minor units, the same stored number means a different amount at each precision. -- **Why `DECIMAL(38, 18)`.** 38 digits is the largest precision every supported database accepts (SQL Server's maximum). It leaves 20 digits before the decimal point (up to 10^20 − 1 main units) and 18 after, which covers ETH at full precision with total supply to spare. `decimal_places` is therefore capped at 18, the storage scale, and the asset's `decimal_places` (not the column) decides how many of those 18 places an amount may use. +- **Why `DECIMAL(38, 18)`.** 38 digits is the largest precision SQL Server and Oracle accept, and OBP must keep supporting both. It leaves 20 digits before the decimal point (up to 10^20 − 1 main units) and 18 after, which covers ETH at full precision with total supply to spare. A larger scale in 38 digits does not work: `DECIMAL(38, 24)` leaves 14 digits before the point, too few for large VND balances or the SHIB supply. +- **Width per database (settled 2026-10-20).** On PostgreSQL the columns are `DECIMAL(60, 24)`: 36 digits before the point and 24 after, so an asset at 24 decimal places (NEAR) is held exactly and every amount `DECIMAL(38, 18)` holds still fits. H2, the test database, takes the same width, so the tests cover 24 places. SQL Server, Oracle and MySQL keep `DECIMAL(38, 18)`. The cap on `decimal_places` follows the database: 24 on PostgreSQL and H2, 18 elsewhere. `Assets` refuses an asset above the cap with an error saying that more than 18 decimal places needs PostgreSQL; the Glossary entry and the asset endpoints' ResourceDocs say so too. A deployment that holds such an asset cannot move to another database. This was chosen over storing whole numbers of each asset's smallest unit, which would allow 24 places everywhere, because amounts stay in the main unit on every database and precision corrections stay metadata changes. +- **Decimal places per amount.** The asset's `decimal_places`, not the column, decides how many of the column's places an amount may use. - **The public model does not change.** `BankAccount.balance` and the transaction amount are already `BigDecimal` in `obp-commons` (`BankingModel.scala:213`); only the mapped storage and the two conversion helpers (`convertToSmallestCurrencyUnits`, `smallestCurrencyUnitToBigDecimal`) change, and the helpers can be removed. - **Connector-sourced accounts** (non-mapped connectors) already report amounts as decimals and are unaffected. @@ -253,7 +266,7 @@ With amounts stored in main units (§5), the migration has two parts: convert st 1. Drop any database view that depends on the table's amount columns (see §11). Recreate a view afterwards only if something still needs it. 2. Rename the existing `Long` column to `_legacy_minor_units` (for example `accountbalance_legacy_minor_units`). This is the backup: the original stored values, untouched. -3. Add the new `DECIMAL(38, 18)` column under the original name, and fill it with `legacy_minor_units / 10^legacy_dp`, where `legacy_dp` is what `currencyDecimalPlaces` returns for the row's currency today. This is exact: the stored value was written at that precision, and dividing by a power of ten in decimal arithmetic loses nothing. +3. Add the new decimal column (§5, width per database) under the original name, and fill it with `legacy_minor_units / 10^legacy_dp`, where `legacy_dp` is what `currencyDecimalPlaces` returns for the row's currency today. This is exact: the stored value was written at that precision, and dividing by a power of ten in decimal arithmetic loses nothing. 4. Verify every row before committing: `new_value * 10^legacy_dp = legacy_minor_units`, and no new value is null where the legacy one was not. Any mismatch rolls the whole table back, and the migration reports it. 5. Write the `legacy_dp` used for each currency into the migration log entry, so the backup can be read back without depending on the old hard-coded table. @@ -266,7 +279,7 @@ After the migration, application code neither reads nor writes the `_legacy_mino 3. Where the target precision is **lower** (the ISO 0 codes in §Background): update `decimal_places` only if no stored amount in that code has non-zero digits beyond the new precision. Otherwise the code keeps its legacy precision and the migration reports the code and the number of offending rows. It never rounds. 4. `PRECIOUS_METAL` and `ACCOUNTING_UNIT` stay at 2 unless a deployment chooses otherwise, since ISO defines no minor unit. -**Part C: fold `lovelace` and `wei` into `ADA` and `ETH`.** Rows with currency `lovelace` get currency `ADA` and amount `amount / 10^6`; rows with `wei` get `ETH` and `amount / 10^18`. Both are exact in `DECIMAL(38, 18)`. A `lovelace` or `wei` amount with a fractional part (possible today, because both got 2 decimal places) is not a real on-chain amount: those rows are reported and left unconverted, never rounded. Rows with `ada` become `ADA`. +**Part C: fold `lovelace` and `wei` into `ADA` and `ETH`.** Rows with currency `lovelace` get currency `ADA` and amount `amount / 10^6`; rows with `wei` get `ETH` and `amount / 10^18`. Both are exact at either column width (§5). A `lovelace` or `wei` amount with a fractional part (possible today, because both got 2 decimal places) is not a real on-chain amount: those rows are reported and left unconverted, never rounded. Rows with `ada` become `ADA`. Each code and each table is migrated in its own transaction, so one failing check does not hold back the others. @@ -366,18 +379,20 @@ The existing `CARDANO` transaction request body carries `assets: [{policy_id, as - **Views over amount columns.** PostgreSQL refuses to change or rename a column a view depends on, so each Part A migration drops the dependent views first, in the same transaction, and recreates one only if something still needs it. Never drop all views on every boot. Today no OBP view depends on these columns: the only ones that did, `v_fast_firehose_accounts` and `mv_fast_firehose_accounts` (both selecting `mappedbankaccount.accountbalance`), are already dropped by `MigrationOfDropFastFireHoseViews`, and nothing replaces them. The migration should still look up dependent views in the database catalog rather than assume none exist, because a deployment may have created its own; their names belong in the migration log. - **Tests empty the registry.** The test setup empties every table in `ToSchemify.models` after each test, so the rows the boot seed inserts do not survive into later tests. That is harmless while nothing reads the registry. Step 6 must either exclude `Asset` from that wipe (as `Consumer` and `AuthUser` are) or seed it in test setup, or every currency check in the test suite will fail. - **`MappedCurrency`.** The existing `code.fx.MappedCurrency` table (code, name, symbol) is never written or read; it is only the foreign-key target of `MappedFXRate`. It is a candidate for removal once FX rates refer to the registry. -- **No silent rounding in Scala.** Scala's `BigDecimal` uses `MathContext.DECIMAL128` by default, which keeps 34 significant digits and rounds arithmetic results beyond that. A `DECIMAL(38, 18)` value can have 38. Amount arithmetic (balance updates, sums, comparisons) must use `MathContext.UNLIMITED`, or check the result is exact; Lift's `MappedDecimal(DECIMAL128, …)` has the same limit and is not suitable for the new columns as is. Add tests at 38 digits. +- **No silent rounding in Scala.** Scala's `BigDecimal` uses `MathContext.DECIMAL128` by default, which keeps 34 significant digits and rounds arithmetic results beyond that. A stored value can have 38 digits, or 60 on PostgreSQL. Amount arithmetic (balance updates, sums, comparisons) must use `MathContext.UNLIMITED`, or check the result is exact; Lift's `MappedDecimal(DECIMAL128, …)` has the same limit and is not suitable for the new columns as is. The column type must also be chosen per database, which `MappedDecimal` does not do. Add tests at 38 and 60 digits. ## 12. Open questions -1. **Retirement precondition.** Should `RETIRED` require zero outstanding balance across all banks in this OBP instance? The check is cheap for mapped accounts, impossible for connector-sourced ones. -2. **Multi-instance scope.** `asset_code` is unique per OBP instance. Two OBP instances could register the same code for different assets. Options: prefix issued codes with an issuer identifier, or rely on `chain_asset_id` / ISIN / ISO 24165 DTI as the cross-instance identity and treat `asset_code` as local. *Leaning*: treat `asset_code` as local, and use the chain identity, ISIN or DTI for cross-instance identity. -3. **DTI.** *Leaning, adopted in v2*: the nullable, set-once `dti` column is in §2 now, since it costs little and avoids a later migration. Still open: whether any endpoint should require it. -4. **Required attributes per asset type.** Enforce the §9 attribute set for issued types at product-link time, or leave as convention? -5. **Decimal cap.** *Settled in v3*: amounts are stored exactly as `DECIMAL(38, 18)` (§5), so `decimal_places` is capped at 18 by the storage scale, not by `Long`. Still open: is any planned asset above 18 decimals? NEAR, for example, has 24. Such an asset would need a wider scale, which reduces the digits left before the decimal point. +All the questions below are settled. Questions 6, 7 and 8 were settled in earlier drafts. The others were settled on 2026-10-20 by taking the default each one leaned towards or proposed. + +1. **Retirement precondition.** Should `RETIRED` require zero outstanding balance across all banks in this OBP instance? The check is cheap for mapped accounts, impossible for connector-sourced ones. *Settled (2026-10-20)*: retiring is refused while any mapped account holds a non-zero balance in the asset. For connector-sourced accounts OBP cannot see the balance, so it does not check them; the retirement is recorded in `AssetStatusHistory` with the user who made it. +2. **Multi-instance scope.** `asset_code` is unique per OBP instance. Two OBP instances could register the same code for different assets. Options: prefix issued codes with an issuer identifier, or rely on `chain_asset_id` / ISIN / ISO 24165 DTI as the cross-instance identity and treat `asset_code` as local. *Settled (2026-10-20)*: `asset_code` is local to the instance and carries no issuer prefix. The chain identity, ISIN or DTI identify an asset across instances, where one exists. +3. **DTI.** The nullable, set-once `dti` column is in §2 (adopted in v2), since it costs little and avoids a later migration. *Settled (2026-10-20)*: it is stored when given and no endpoint requires it. +4. **Required attributes per asset type.** Enforce the §9 attribute set for issued types at product-link time, or leave as convention? *Settled (2026-10-20)*: left as convention; linking a product does not check its attributes. +5. **Decimal cap.** *Settled in v3*: amounts are stored exactly as decimals in the main unit (§5), so `decimal_places` is capped by the storage scale, not by `Long`. *Settled (2026-10-20)*: the cap depends on the database. PostgreSQL (and H2, for tests) stores amounts as `DECIMAL(60, 24)` and allows up to 24 decimal places, which covers NEAR. SQL Server, Oracle and MySQL keep `DECIMAL(38, 18)` and allow up to 18, because SQL Server and Oracle accept at most 38 digits and must stay supported. So a platform holding assets above 18 decimal places must run on PostgreSQL. Storing whole numbers of each asset's smallest unit would allow 24 places everywhere, but was not chosen: it makes the stored amount depend on `decimal_places`, so a precision correction (§7 Part B) would rewrite stored amounts. 6. **Seeding vs. config.** Should deployments be able to disable seeded fiat codes (e.g. a node that only handles TZS and one token)? *Adopted in step 4*: the seed has no property; a deployment suspends codes it does not handle via `status = SUSPENDED` at `SYS`, so there is one model and one audit trail. Nothing can suspend a code until the status endpoint (§8) exists. 7. **`ada` and `lovelace`.** *Settled in v3*: one code, `ADA`, at 6 decimal places; `lovelace` remains only as the Cardano protocol unit in the request body (§7 Part C, §10). 8. **`ETH` precision.** *Settled in v3*: `ETH` at its full 18 decimal places, stored exactly (§5); `wei` is folded into `ETH` (§7 Part C). -9. **Codes that fail the downward check.** When a code like ISK keeps legacy precision because some stored amounts have a fractional part, what then? Leave it at 2 permanently, or produce a report so the bank can correct the data and rerun? The step 2 report counts the affected rows; a local sandbox has none, so this needs a run against a copy of real data. -10. **Account routing schemes and networks.** Asset `chain_scheme` values name the network (§2), but a Cardano address on an account still uses the bare routing scheme `CARDANO`, which is mainnet on one deployment and preprod on another. Should account routings move to `CARDANO_MAINNET` and so on? That would change existing routing rows and the Open Corridor settlement account's routing, so it needs its own migration. -11. **Chain identity of native coins.** `ADA` and `ETH` are the chains' own currencies, not tokens, so they have no policy id or contract address. *Leaning*: they have no chain identity; the reverse lookup is for tokens only, and a plain ADA or ETH amount is recognised by the transaction request type. +9. **Codes that fail the downward check.** When a code like ISK keeps legacy precision because some stored amounts have a fractional part, what then? *Settled (2026-10-20)*: a code with such rows keeps its legacy precision, and the migration lists the affected rows (table, row id, amount) in its log so the bank can correct the data and the correction can be run again. The step 2 report counts the affected rows; a local sandbox has none, so it should still be run against a copy of real data before §7 Part B is written. +10. **Account routing schemes and networks.** Asset `chain_scheme` values name the network (§2), but a Cardano address on an account still uses the bare routing scheme `CARDANO`, which is mainnet on one deployment and preprod on another. Should account routings move to `CARDANO_MAINNET` and so on? *Settled (2026-10-20)*: not now. Account routings keep the bare scheme; moving them would change existing routing rows and the Open Corridor settlement account's routing, and is taken up only if the ambiguity causes a real problem. +11. **Chain identity of native coins.** `ADA` and `ETH` are the chains' own currencies, not tokens, so they have no policy id or contract address. *Settled (2026-10-20)*: they have no chain identity; the reverse lookup is for tokens only, and a plain ADA or ETH amount is recognised by the transaction request type. diff --git a/obp-api/src/main/resources/docs/introductory_system_documentation.md b/obp-api/src/main/resources/docs/introductory_system_documentation.md index 1e60c2f2ba..f947e2b551 100644 --- a/obp-api/src/main/resources/docs/introductory_system_documentation.md +++ b/obp-api/src/main/resources/docs/introductory_system_documentation.md @@ -139,7 +139,7 @@ The Open Bank Project (OBP) is an open-source RESTful API platform for banks tha - **Views System**: Account data visibility control (owner, public, accountant, auditor, custom) - **Scope Management**: OAuth 2.0 scope definitions - **Authentication Type Validation**: Enforce authentication requirements per endpoint -- **Endpoint Auth Modes**: Per-endpoint control over user vs application authentication (UserOnly, ApplicationOnly, UserOrApplication, UserAndApplication) +- **Endpoint Auth Modes**: Per-endpoint control over user vs application authentication (UserOnly, ApplicationOnly, UserXORApplication, UserAndApplication) - **Mandates**: Corporate account governance with provisions, signatory panels, and multi-signature approval #### 1.2.9 Extensibility & Customization @@ -2699,7 +2699,7 @@ OBP can bootstrap a Consumer (Application) for OBP-OIDC at startup. This allows The bootstrap consumer is granted the following Scopes: `CanGetConsumers`, `CanCreateConsumer`, `CanVerifyOidcClient`, `CanGetOidcClient`, `CanGetOidcConsent`. -These endpoints use `authMode = UserOrApplication`, meaning they can be accessed either by a logged-in User with Entitlements, or by an Application using a Consumer Key with Scopes. +These endpoints use `authMode = UserXORApplication`, meaning they can be accessed either by a logged-in User with Entitlements, or by the Application acting as itself (e.g. with a client-credentials token) with its Consumer's Scopes. A User calling through the Consumer does not get its Scopes. ```properties # Bootstrap OIDC Operator Consumer @@ -3022,18 +3022,18 @@ Previously, a global `allow_entitlements_or_scopes` config flag allowed consumer |------|-------------| | `UserOnly` | (Default) Requires user authentication and user entitlements | | `ApplicationOnly` | Requires consumer scope only, no user authentication needed | -| `UserOrApplication` | Either consumer scope OR user entitlement is sufficient | +| `UserXORApplication` | Consumer scope when the application acts as itself (no user, or its own client-credentials user, provider `obp-oidc#client`); user entitlement when a person acts. A person never uses the consumer's scopes. Formerly `UserOrApplication` (either was sufficient) | | `UserAndApplication` | Both consumer scope AND user entitlement are required | **Migration from `allow_entitlements_or_scopes`:** -The global `allow_entitlements_or_scopes` config flag has been removed. Deployments that previously used `allow_entitlements_or_scopes=true` must now set `authMode = UserOrApplication` on individual endpoint `ResourceDoc` instances. +The global `allow_entitlements_or_scopes` config flag has been removed. Deployments that previously used `allow_entitlements_or_scopes=true` must now set `authMode = UserXORApplication` on individual endpoint `ResourceDoc` instances. -The global overrides `require_scopes_for_all_roles` and `require_scopes_for_listed_roles` continue to work and force `UserAndApplication` behavior. +The global overrides `require_scopes_for_all_roles` and `require_scopes_for_listed_roles` continue to work and force `UserAndApplication` behavior, except for an application acting as itself on an `ApplicationOnly` or `UserXORApplication` endpoint, which still needs only its scopes. **Example:** -The `verifyUserCredentials` endpoint uses `UserOrApplication` mode, so it can be called either by a logged-in user with the appropriate entitlement, or by an application with the appropriate consumer scope. +The `verifyUserCredentials` endpoint uses `UserXORApplication` mode, so it can be called either by a logged-in user with the appropriate entitlement, or by an application acting as itself with the appropriate consumer scope. ### 8.8 Maker/Checker for Transaction Requests @@ -4298,7 +4298,7 @@ docker run -p 8080:8080 \ **Dynamic Endpoint:** User-defined API endpoint -**Endpoint Auth Mode:** Per-endpoint setting controlling whether user authentication, application scope, or both are required (UserOnly, ApplicationOnly, UserOrApplication, UserAndApplication) +**Endpoint Auth Mode:** Per-endpoint setting controlling whether user authentication, application scope, or both are required (UserOnly, ApplicationOnly, UserXORApplication, UserAndApplication) **Entitlement:** Permission to perform specific operation (same as Role) diff --git a/obp-api/src/main/resources/props/sample.props.template b/obp-api/src/main/resources/props/sample.props.template index 19e30fd05a..cd4ea1034b 100644 --- a/obp-api/src/main/resources/props/sample.props.template +++ b/obp-api/src/main/resources/props/sample.props.template @@ -271,6 +271,13 @@ jwt.use.ssl=false ## Unit is a second # obp_expired_consents_interval_in_seconds = +## Reject OBP consents still "INITIATED" (their challenge was never answered) after a defined time. +## They never granted access; rejecting them stops an app that reuses a pending consent from being stuck on one. +## Both have defaults: the task runs every 593 seconds, and rejects consents unchanged for 21600 seconds (6 hours). +## Set the interval to 0 to stop the task. +# obp_unfinished_consents_time_in_seconds = 21600 +# obp_unfinished_consents_interval_in_seconds = 593 + ## Expire UK Open Banking consents with status "Authorised" ## Unlike the three keys above this one HAS a default (601 seconds, a prime near 10 minutes, so the ## sweep does not settle into step with other periodic schedulers). Set it to 0 to stop the task. @@ -422,6 +429,7 @@ public_obp_api_url=http://localhost:8080 public_obp_portal_url=http://localhost:5174 public_obp_api_explorer_url=http://localhost:5173 public_obp_api_manager_url=http://localhost:3003 +public_obp_cohort_url=http://localhost:5183 public_obp_sandbox_populator_url=http://localhost:5178 public_obp_oidc_url=http://localhost:9000 public_keycloak_url=http://localhost:7787 @@ -914,6 +922,8 @@ defaultBank.bank_id=OBP ## CanCreateEntitlementAtAnyBank ## CanCreateEntitlementAtOneBank ## CanGetAnyUser +## CanGetEntitlementRequestsAtAnyBank +## CanDeleteEntitlementRequestsAtAnyBank ## After they have granted these roles, they can grant further roles and remove their # user_id from the super_admin_user_ids list because its redundant. ## Once you have the roles above you can grant any other system or bank related roles to yourself. @@ -1436,7 +1446,8 @@ database_messages_scheduler_interval=3600 # --------------------------------------------------- # -- Consents --------------------------------------------- -# In case isn't defined default value is "false" +# In case isn't defined default value is "false". While false, no consent is accepted as a login (OBP-35007) +# and none is created either: every endpoint that creates a consent or a consent request answers OBP-35007. # consents.allowed=true # # In order to pin a consent to a consumer we can use the property consumer_validation_method_for_consent @@ -2003,6 +2014,31 @@ user_account_is_validated = false # aggregate_metrics_max_days = 31 +# Data retention: how long the resources of a space (a bank, or SYS) stay live, and what happens to +# them then, is set per space with the retention policy endpoints +# (/obp/v7.0.0/management/banks/BANK_ID/retention-policies). Periods are set there, not here, except +# for the minimum below; the other props only switch the destructive steps on. Defaults live in +# code.retention.RetentionProps; the values shown here MUST match. See RETENTION_PLAN.md. +# The shortest retention period allowed, in days: no resource is archived, deleted or anonymised sooner +# than this after its trigger date. A policy with a shorter retain_days is refused, and an existing one +# runs as though its retain_days were this. 1825 days is five years. +# retention.minimum_retain_days = 1825 +# Allow runs that move resources out of the live tables into the retention archive. While false, +# such a run is refused with an error. A preview always works. +# retention.archive.enabled = false +# Allow runs that delete or anonymise resources, and the deletion of archived resources whose archive +# period has ended. While false, such a run is refused with an error. A preview always works. +# retention.delete.enabled = false +# Run the enabled retention policies on a timer. A policy that requires approval gets a pending run +# for a second user to approve; the others are executed, as far as the two props above allow. +# retention.scheduler.enabled = false +# Seconds between two scheduler runs. 3593, a prime just under an hour, so the runs drift across the +# clock rather than lining up with other schedulers. +# retention.scheduler.interval_in_seconds = 3593 +# The most resources one run disposes of; what is left is disposed of by the next run. +# retention.run.max_items = 1000 + + # Defines endpoints we want to store responses at Metric table metrics_store_response_body_for_operation_ids= diff --git a/obp-api/src/main/scala/bootstrap/liftweb/Boot.scala b/obp-api/src/main/scala/bootstrap/liftweb/Boot.scala index 13f4aafc88..f16cae5f9e 100644 --- a/obp-api/src/main/scala/bootstrap/liftweb/Boot.scala +++ b/obp-api/src/main/scala/bootstrap/liftweb/Boot.scala @@ -90,7 +90,7 @@ import code.etag.MappedETag import code.featuredapicollection.FeaturedApiCollection import code.fx.{MappedCurrency, MappedFXRate} import code.group.Group -import code.organisation.Organisation +import code.tenant.Tenant import code.routingscheme.{RoutingScheme, BankSupportedRoutingScheme} import code.payeelookup.PayeeLookup import code.utilitypayment.UtilityPaymentCallback @@ -659,6 +659,9 @@ class Boot extends MdcLoggable { case false => // Do not start it } + // Data retention: runs the enabled retention policies (see RETENTION_PLAN.md). Off by default. + code.scheduler.RetentionScheduler.startIfEnabled() + // ConnectorEndpoints (connector.name.export.as.endpoints) registered via Lift statelessDispatch // which is no longer reachable (Lift bridge removed in Phase B). Disabled until migrated to http4s. @@ -1159,7 +1162,8 @@ object ToSchemify extends MdcLoggable { BankAccountBalance, Group, code.group.GroupMembership, - Organisation, + Tenant, + code.model.dataAccess.SelfServiceBank, RoutingScheme, BankSupportedRoutingScheme, code.asset.Asset, @@ -1169,6 +1173,13 @@ object ToSchemify extends MdcLoggable { code.platformapp.PlatformAppRequiredScope, code.domainapi.DomainApi, code.dynamicentitywebhook.DynamicEntityWebhook, + code.accountcreation.AccountCreation, + code.retention.RetentionPolicy, + code.retention.RetentionArchive, + code.retention.RetentionArchiveContent, + code.retention.RetentionHold, + code.retention.RetentionRun, + code.retention.RetentionRunItem, PayeeLookup, UtilityPaymentCallback, BulkPayment, diff --git a/obp-api/src/main/scala/code/accountcreation/AccountCreation.scala b/obp-api/src/main/scala/code/accountcreation/AccountCreation.scala new file mode 100644 index 0000000000..4930ef0004 --- /dev/null +++ b/obp-api/src/main/scala/code/accountcreation/AccountCreation.scala @@ -0,0 +1,134 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ + +package code.accountcreation + +import java.util.Date + +import code.consent.Consents +import code.users.{UserReference, Users} +import net.liftweb.common.{Box, Full} +import net.liftweb.mapper._ +import net.liftweb.util.Helpers.tryo +import net.liftweb.util.SimpleInjector + +/** + * Who created a bank account, and for whom. + * + * The account itself is created through the Connector, which may keep it outside OBP's database, so + * the creator cannot be a column of the account. This table holds one row per account created + * through v7.0.0 Create Account. It records both ids, like a Counterparty or a Dynamic Entity + * webhook: the caller, which is an agent's own id when an agent made the call, and the person the + * call was made for. See UserReference.AccountCreation_CreatedByUserId. When an agent made the call, + * it also records the consent_reference_id of the Consent it acted under, the id the metrics record + * for the same call. + */ +object AccountCreations extends SimpleInjector { + val accountCreationProvider = new Inject(() => buildOne) {} + def buildOne: AccountCreationProvider = AccountCreationDbProvider +} + +trait AccountCreationTrait { + def bankId: String + def accountId: String + def createdByUserId: String + def createdByOnBehalfOfUserId: String + /** The Consent the agent acted under; None when the caller acted for themselves. */ + def consentReferenceId: Option[String] + def createdAt: Date +} + +trait AccountCreationProvider { + /** Records that `createdByUserId`, the authenticated caller, created the account. */ + def record(bankId: String, accountId: String, createdByUserId: String): Box[AccountCreationTrait] + def get(bankId: String, accountId: String): Box[AccountCreationTrait] + /** The records of those of `accountIds` at `bankId` that have one, in one query. */ + def getForAccounts(bankId: String, accountIds: List[String]): List[AccountCreationTrait] +} + +object AccountCreationDbProvider extends AccountCreationProvider { + + override def record(bankId: String, accountId: String, createdByUserId: String): Box[AccountCreationTrait] = + for { + attribution <- Users.users.vend.attributionOf(createdByUserId, UserReference.AccountCreation_CreatedByUserId) + consentReferenceId = attribution.consentId + .flatMap(id => Consents.consentProvider.vend.getConsentByConsentId(id).toOption) + .map(_.consentReferenceId) + row <- tryo { + AccountCreation.create + .BankId(bankId) + .AccountId(accountId) + .CreatedByUserId(attribution.userId) + .CreatedByOnBehalfOfUserId(attribution.onBehalfOfUserId) + .ConsentReferenceId(consentReferenceId.orNull) + .saveMe() + } + } yield row + + override def get(bankId: String, accountId: String): Box[AccountCreationTrait] = + AccountCreation.find(By(AccountCreation.BankId, bankId), By(AccountCreation.AccountId, accountId)) + + override def getForAccounts(bankId: String, accountIds: List[String]): List[AccountCreationTrait] = + if (accountIds.isEmpty) Nil + else AccountCreation.findAll(By(AccountCreation.BankId, bankId), ByList(AccountCreation.AccountId, accountIds)) +} + +class AccountCreation extends AccountCreationTrait with LongKeyedMapper[AccountCreation] with IdPK { + def getSingleton = AccountCreation + + object BankId extends MappedString(this, 255) { + override def dbColumnName = "bank_id" + } + object AccountId extends MappedString(this, 255) { + override def dbColumnName = "account_id" + } + object CreatedByUserId extends MappedString(this, 255) { + override def dbColumnName = "created_by_user_id" + } + object CreatedByOnBehalfOfUserId extends MappedString(this, 255) { + override def dbColumnName = "created_by_on_behalf_of_user_id" + } + object ConsentReferenceId extends MappedString(this, 255) { + override def dbColumnName = "consent_reference_id" + } + object CreatedAt extends MappedDateTime(this) { + override def dbColumnName = "created_at" + override def defaultValue = new Date() + } + + override def bankId: String = BankId.get + override def accountId: String = AccountId.get + override def createdByUserId: String = CreatedByUserId.get + override def createdByOnBehalfOfUserId: String = CreatedByOnBehalfOfUserId.get + override def consentReferenceId: Option[String] = Option(ConsentReferenceId.get).filter(_.nonEmpty) + override def createdAt: Date = CreatedAt.get +} + +object AccountCreation extends AccountCreation with LongKeyedMetaMapper[AccountCreation] { + override def dbTableName = "account_creation" + override def dbIndexes = UniqueIndex(BankId, AccountId) :: super.dbIndexes +} diff --git a/obp-api/src/main/scala/code/api/ResourceDocs1_4_0/OpenAPI31JSONFactory.scala b/obp-api/src/main/scala/code/api/ResourceDocs1_4_0/OpenAPI31JSONFactory.scala index 76f5b2e117..5146eb8c3e 100644 --- a/obp-api/src/main/scala/code/api/ResourceDocs1_4_0/OpenAPI31JSONFactory.scala +++ b/obp-api/src/main/scala/code/api/ResourceDocs1_4_0/OpenAPI31JSONFactory.scala @@ -168,7 +168,7 @@ object OpenAPI31JSONFactory extends MdcLoggable { | |### Client Credentials Flow (2-legged / Application Only) | - |Use this flow for endpoints with authMode ApplicationOnly or UserOrApplication. + |Use this flow for endpoints with authMode ApplicationOnly or UserXORApplication. |The application authenticates itself using its client_id and client_secret without a user login. | | POST $oidcTokenEndpoint diff --git a/obp-api/src/main/scala/code/api/ResourceDocs1_4_0/SwaggerDefinitionsJSON.scala b/obp-api/src/main/scala/code/api/ResourceDocs1_4_0/SwaggerDefinitionsJSON.scala index d30800a8ab..7d4109f5ec 100644 --- a/obp-api/src/main/scala/code/api/ResourceDocs1_4_0/SwaggerDefinitionsJSON.scala +++ b/obp-api/src/main/scala/code/api/ResourceDocs1_4_0/SwaggerDefinitionsJSON.scala @@ -6388,6 +6388,7 @@ object SwaggerDefinitionsJSON { ConfigPropJsonV600("public_obp_portal_url", "http://localhost:5174"), ConfigPropJsonV600("public_obp_api_explorer_url", "http://localhost:5173"), ConfigPropJsonV600("public_obp_api_manager_url", "http://localhost:3003"), + ConfigPropJsonV600("public_obp_cohort_url", "http://localhost:5183"), ConfigPropJsonV600("public_obp_sandbox_populator_url", "http://localhost:5178"), ConfigPropJsonV600("public_obp_oidc_url", "http://localhost:9000"), ConfigPropJsonV600("public_keycloak_url", "http://localhost:7787"), diff --git a/obp-api/src/main/scala/code/api/UKOpenBanking/v3_1_0/Http4sUKOBv310AccountAccess.scala b/obp-api/src/main/scala/code/api/UKOpenBanking/v3_1_0/Http4sUKOBv310AccountAccess.scala index 092ba151d5..d39b8c0a82 100644 --- a/obp-api/src/main/scala/code/api/UKOpenBanking/v3_1_0/Http4sUKOBv310AccountAccess.scala +++ b/obp-api/src/main/scala/code/api/UKOpenBanking/v3_1_0/Http4sUKOBv310AccountAccess.scala @@ -32,7 +32,7 @@ import cats.data.{Kleisli, OptionT} import cats.effect.IO import code.api.Constant import code.api.UKOpenBanking.v3_1_0.JSONFactory_UKOpenBanking_310.ConsentPostBodyUKV310 -import code.api.util.APIUtil.{Http4sHandler, Http4sRoute, EmptyBody, ResourceDoc, UserOrApplication, connectorEmptyResponse, mockedDataText, passesPsd2Aisp, unboxFullOrFail, parseIso8601OrDayDate} +import code.api.util.APIUtil.{Http4sHandler, Http4sRoute, EmptyBody, ResourceDoc, UserXORApplication, connectorEmptyResponse, mockedDataText, passesPsd2Aisp, unboxFullOrFail, parseIso8601OrDayDate} import code.api.util.ApiTag import code.api.util.CustomJsonFormats import code.api.util.ErrorMessages.{AuthenticatedUserIsRequired, ConsentNotFound, ConsentViewNotFund, InvalidJsonFormat, InvalidUKConsentPermissions, UnknownError} @@ -75,6 +75,7 @@ object Http4sUKOBv310AccountAccess extends MdcLoggable { EndpointHelpers.executeFutureCreated(req) { implicit val cc: CallContext = req.callContext for { + _ <- code.api.util.Consent.checkConsentsAllowed(Some(cc)) // Client-credentials lodging: require some authentication (consumer or user) but not a // PSU specifically; reject only a fully anonymous request. The PSU is bound later at // authorise time. Mirrors the Berlin Group native consent flow and the v4.0.1 handler. @@ -201,8 +202,8 @@ object Http4sUKOBv310AccountAccess extends MdcLoggable { // Without it the doc defaults to UserOnly, which sends the middleware down anonymousAccess and // 401s any request that carries no user -- so the endpoint only works today because OAuth2 // token parsing auto-vivifies a user for a client-credentials token. Matches the Berlin Group - // twin (Http4sBGv13AIS.createConsent), which has always been UserOrApplication. - authMode = UserOrApplication, + // twin (Http4sBGv13AIS.createConsent), which has always been UserXORApplication. + authMode = UserXORApplication, http4sPartialFunction = Some(createAccountAccessConsents) ) @@ -239,7 +240,7 @@ object Http4sUKOBv310AccountAccess extends MdcLoggable { ApiTag("Account Access") :: Nil, // As with the POST that lodges the consent: revoking is a client-credentials call in the // standard's AISP flow, so a PSU cannot be required. See Consent.checkUKConsentAccess. - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(deleteAccountAccessConsentsConsentId) ) @@ -317,7 +318,7 @@ object Http4sUKOBv310AccountAccess extends MdcLoggable { List(AuthenticatedUserIsRequired, UnknownError), ApiTag("Account Access") :: Nil, // As above: the AISP polls its own consent with a client-credentials token. - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(getAccountAccessConsentsConsentId) ) diff --git a/obp-api/src/main/scala/code/api/UKOpenBanking/v4_0_1/Http4sUKOBv401AccountInfo.scala b/obp-api/src/main/scala/code/api/UKOpenBanking/v4_0_1/Http4sUKOBv401AccountInfo.scala index 7cb11a3b8d..aa7be7bf8d 100644 --- a/obp-api/src/main/scala/code/api/UKOpenBanking/v4_0_1/Http4sUKOBv401AccountInfo.scala +++ b/obp-api/src/main/scala/code/api/UKOpenBanking/v4_0_1/Http4sUKOBv401AccountInfo.scala @@ -34,7 +34,7 @@ import cats.effect.IO import code.api.APIFailureNewStyle import code.api.Constant import code.api.UKOpenBanking.v3_1_0.JSONFactory_UKOpenBanking_310.ConsentPostBodyUKV310 -import code.api.util.APIUtil.{Http4sHandler, Http4sRoute, EmptyBody, ResourceDoc, HTTPParam, UserOrApplication, connectorEmptyResponse, createQueriesByHttpParams, defaultBankId, fullBoxOrException, passesPsd2Aisp, unboxFull, unboxFullOrFail, parseIso8601OrDayDate} +import code.api.util.APIUtil.{Http4sHandler, Http4sRoute, EmptyBody, ResourceDoc, HTTPParam, UserXORApplication, connectorEmptyResponse, createQueriesByHttpParams, defaultBankId, fullBoxOrException, passesPsd2Aisp, unboxFull, unboxFullOrFail, parseIso8601OrDayDate} import code.api.util.ApiTag import code.api.util.CallContext import code.api.util.CustomJsonFormats @@ -124,6 +124,7 @@ object Http4sUKOBv401AccountInfo extends MdcLoggable { EndpointHelpers.executeFutureCreated(req) { implicit val cc: CallContext = req.callContext for { + _ <- code.api.util.Consent.checkConsentsAllowed(Some(cc)) // Spec Step 2: the TPP lodges the consent via a client-credentials grant -- authenticated // as an app (consumer) but with no PSU yet. Require some authentication (a consumer or a // user) but not a PSU specifically; reject only a fully anonymous request. The PSU is @@ -209,8 +210,8 @@ object Http4sUKOBv401AccountInfo extends MdcLoggable { // Without it the doc defaults to UserOnly, which sends the middleware down anonymousAccess and // 401s any request that carries no user -- so the endpoint only works today because OAuth2 // token parsing auto-vivifies a user for a client-credentials token. Matches the Berlin Group - // twin (Http4sBGv13AIS.createConsent), which has always been UserOrApplication. - authMode = UserOrApplication, + // twin (Http4sBGv13AIS.createConsent), which has always been UserXORApplication. + authMode = UserXORApplication, http4sPartialFunction = Some(createAccountAccessConsents) ) @@ -290,7 +291,7 @@ object Http4sUKOBv401AccountInfo extends MdcLoggable { ApiTag("Account Access Consents") :: Nil, // Same reasoning as the POST that lodges the consent: the AISP polls it with a // client-credentials token, so a PSU cannot be required. See Consent.checkUKConsentAccess. - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(getAccountAccessConsentsConsentId) ) @@ -323,7 +324,7 @@ object Http4sUKOBv401AccountInfo extends MdcLoggable { List(AuthenticatedUserIsRequired, UnknownError), ApiTag("Account Access Consents") :: Nil, // As above: revoking is a client-credentials call in the standard's AISP flow. - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(deleteAccountAccessConsentsConsentId) ) diff --git a/obp-api/src/main/scala/code/api/berlin/group/v1_3/Http4sBGv13AIS.scala b/obp-api/src/main/scala/code/api/berlin/group/v1_3/Http4sBGv13AIS.scala index 91aeaa019e..02add13773 100644 --- a/obp-api/src/main/scala/code/api/berlin/group/v1_3/Http4sBGv13AIS.scala +++ b/obp-api/src/main/scala/code/api/berlin/group/v1_3/Http4sBGv13AIS.scala @@ -38,7 +38,7 @@ import code.api.berlin.group.v1_3.model._ import code.api.berlin.group.v1_3.{BgSpecValidation, JSONFactory_BERLIN_GROUP_1_3, JvalueCaseClass} import code.api.RequestHeader import code.api.util.APIUtil -import code.api.util.APIUtil.{Http4sHandler, Http4sRoute, EmptyBody, ResourceDoc, UserOrApplication, connectorEmptyResponse, createQueriesByHttpParams, fullBoxOrException, getHttpRequestUrlParam, getSuggestedDefaultScaMethod, mockedDataText, passesPsd2Aisp, unboxFull, unboxFullOrFail} +import code.api.util.APIUtil.{Http4sHandler, Http4sRoute, EmptyBody, ResourceDoc, UserXORApplication, connectorEmptyResponse, createQueriesByHttpParams, fullBoxOrException, getHttpRequestUrlParam, getSuggestedDefaultScaMethod, mockedDataText, passesPsd2Aisp, unboxFull, unboxFullOrFail} import code.api.util.CallContext import code.api.util.ApiTag._ import code.api.util.CustomJsonFormats @@ -110,6 +110,7 @@ object Http4sBGv13AIS extends MdcLoggable { val createdByUser: Option[User] = cc.user.toOption .filterNot(u => cc.consumer.map(_.key.get).contains(u.idGivenByProvider)) for { + _ <- code.api.util.Consent.checkConsentsAllowed(callContext) _ <- passesPsd2Aisp(callContext) failMsg = s"$InvalidJsonFormat The Json body should be the $PostConsentJson " consentJson <- NewStyle.function.tryons(failMsg, 400, callContext) { @@ -847,7 +848,7 @@ recurringIndicator: ), List(AuthenticatedUserIsRequired, UnknownError), ApiTag("Account Information Service (AIS)") :: apiTagBerlinGroupM :: Nil, - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(createConsent) ) @@ -863,7 +864,7 @@ recurringIndicator: EmptyBody, List(AuthenticatedUserIsRequired, UnknownError), ApiTag("Account Information Service (AIS)") :: apiTagBerlinGroupM :: Nil, - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(deleteConsent) ) @@ -901,7 +902,7 @@ where the consent was directly managed between ASPSP and PSU e.g. in a re-direct }""")), List(AuthenticatedUserIsRequired, ConsentNotFound, UnknownError), ApiTag("Account Information Service (AIS)") :: apiTagBerlinGroupM :: Nil, - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(getConsentInformation) ) @@ -919,7 +920,7 @@ where the consent was directly managed between ASPSP and PSU e.g. in a re-direct }""")), List(AuthenticatedUserIsRequired, UnknownError), ApiTag("Account Information Service (AIS)") :: apiTagBerlinGroupM :: Nil, - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(getConsentStatus) ) @@ -980,7 +981,7 @@ using the extended forms as indicated above. // so the doc can now say what the call actually is, as its GET siblings already do. Left on // the UserOnly default it would 401 a client-credentials caller the day OAuth2 token parsing // stops auto-vivifying a user. - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(startConsentAuthorisationAll) ) @@ -995,7 +996,7 @@ using the extended forms as indicated above. startConsentAuthorisationResponse, List(AuthenticatedUserIsRequired, UnknownError), ApiTag("Account Information Service (AIS)") :: apiTagBerlinGroupM :: Nil, - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(startConsentAuthorisationAll) ) @@ -1010,7 +1011,7 @@ using the extended forms as indicated above. startConsentAuthorisationResponse, List(AuthenticatedUserIsRequired, UnknownError), ApiTag("Account Information Service (AIS)") :: apiTagBerlinGroupM :: Nil, - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(startConsentAuthorisationAll) ) @@ -1057,7 +1058,7 @@ Maybe in a later version the access path will change. ), List(AuthenticatedUserIsRequired, UnknownError), ApiTag("Account Information Service (AIS)") :: apiTagBerlinGroupM :: Nil, - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(updateConsentsPsuDataAll) ) @@ -1077,7 +1078,7 @@ Maybe in a later version the access path will change. | }""".stripMargin)), List(AuthenticatedUserIsRequired, UnknownError), ApiTag("Account Information Service (AIS)") :: apiTagBerlinGroupM :: Nil, - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(updateConsentsPsuDataAll) ) @@ -1105,7 +1106,7 @@ Maybe in a later version the access path will change. | }""".stripMargin)), List(AuthenticatedUserIsRequired, UnknownError), ApiTag("Account Information Service (AIS)") :: apiTagBerlinGroupM :: Nil, - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(updateConsentsPsuDataAll) ) @@ -1125,7 +1126,7 @@ Maybe in a later version the access path will change. | }""".stripMargin)), List(AuthenticatedUserIsRequired, UnknownError), ApiTag("Account Information Service (AIS)") :: apiTagBerlinGroupM :: Nil, - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(updateConsentsPsuDataAll) ) } @@ -1347,8 +1348,8 @@ This function returns an array of hyperlinks to all generated authorisation sub- // Embedded. The handler already reflects that, taking no user; only the doc disagreed, and a // doc left on the UserOnly default sends the middleware down anonymousAccess, which 401s a // request carrying no user. Brings these into line with the consent endpoints in this same - // file, which have been UserOrApplication all along. - authMode = UserOrApplication, + // file, which have been UserXORApplication all along. + authMode = UserXORApplication, http4sPartialFunction = Some(getConsentAuthorisation) ) @@ -1368,7 +1369,7 @@ This method returns the SCA status of a consent initiation's authorisation sub-r List(AuthenticatedUserIsRequired, UnknownError), ApiTag("Account Information Service (AIS)") :: apiTagBerlinGroupM :: Nil, // As above -- reading the SCA status is the TPP polling its own authorisation sub-resource. - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(getConsentScaStatus) ) diff --git a/obp-api/src/main/scala/code/api/berlin/group/v1_3/Http4sBGv13PIS.scala b/obp-api/src/main/scala/code/api/berlin/group/v1_3/Http4sBGv13PIS.scala index 97cab75128..b84e35d1ff 100644 --- a/obp-api/src/main/scala/code/api/berlin/group/v1_3/Http4sBGv13PIS.scala +++ b/obp-api/src/main/scala/code/api/berlin/group/v1_3/Http4sBGv13PIS.scala @@ -34,7 +34,7 @@ import code.api.berlin.group.ConstantsBG import code.api.berlin.group.v1_3.JSONFactory_BERLIN_GROUP_1_3._ import code.api.berlin.group.v1_3.model.TransactionStatus.mapTransactionStatus import code.api.berlin.group.v1_3.model._ -import code.api.util.APIUtil.{Http4sHandler, Http4sRoute, EmptyBody, ResourceDoc, UserOrApplication, getScaMethodAtInstance, getServerUrl, isValidCurrencyISOCode, mockedDataText, passesPsd2Pisp} +import code.api.util.APIUtil.{Http4sHandler, Http4sRoute, EmptyBody, ResourceDoc, UserXORApplication, getScaMethodAtInstance, getServerUrl, isValidCurrencyISOCode, mockedDataText, passesPsd2Pisp} import code.api.util.ApiTag._ import code.api.util.ErrorMessages._ import code.api.util.CustomJsonFormats @@ -143,7 +143,7 @@ object Http4sBGv13PIS extends MdcLoggable { /** * Shared business logic for all three initiate-payment variants (payments / periodic-payments / * bulk-payments). Mirrors `initiatePaymentImplementation` from the Lift builder; auth is handled - * by middleware (authMode = UserOrApplication), so no inline applicationAccess call. + * by middleware (authMode = UserXORApplication), so no inline applicationAccess call. */ private def initiatePaymentImpl( paymentService: String, @@ -427,7 +427,7 @@ object Http4sBGv13PIS extends MdcLoggable { } // ── POST /payments/{paymentProduct} ────────────────────────────────────────────────────────────── - // Auth: applicationAccess in Lift → authMode = UserOrApplication in ResourceDoc + // Auth: applicationAccess in Lift → authMode = UserXORApplication in ResourceDoc lazy val initiatePayments: Http4sRoute = Http4sRoute { case req @ POST -> `bgV13Prefix` / "payments" / paymentProduct => EndpointHelpers.executeFutureCreated(req) { @@ -1042,7 +1042,7 @@ $generalPaymentSummaryText""", initiatePaymentResponseBody, List(AuthenticatedUserIsRequired, UnknownError), ApiTag("Payment Initiation Service (PIS)") :: apiTagBerlinGroupM :: Nil, - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(initiatePayments) ) @@ -1075,7 +1075,7 @@ $generalPaymentSummaryText""", }""")), List(AuthenticatedUserIsRequired, UnknownError), ApiTag("Payment Initiation Service (PIS)") :: apiTagBerlinGroupM :: Nil, - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(initiatePeriodicPayments) ) @@ -1111,7 +1111,7 @@ $generalPaymentSummaryText""", }""")), List(AuthenticatedUserIsRequired, UnknownError), ApiTag("Payment Initiation Service (PIS)") :: apiTagBerlinGroupM :: Nil, - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(initiateBulkPayments) ) } diff --git a/obp-api/src/main/scala/code/api/dynamic/entity/Http4sDynamicEntity.scala b/obp-api/src/main/scala/code/api/dynamic/entity/Http4sDynamicEntity.scala index b1b815a2e1..1eefbaec02 100644 --- a/obp-api/src/main/scala/code/api/dynamic/entity/Http4sDynamicEntity.scala +++ b/obp-api/src/main/scala/code/api/dynamic/entity/Http4sDynamicEntity.scala @@ -343,7 +343,7 @@ object Http4sDynamicEntity extends MdcLoggable { private def entityAccess(cc: CallContext, bankId: Option[String], entityName: String, isPersonalEntity: Boolean): Future[(Box[User], Option[CallContext])] = if (isPersonalEntity || isRowLevel(bankId, entityName)) authenticatedAccess(cc) else authModeOf(bankId, entityName) match { - case ApplicationOnly | UserOrApplication => applicationAccess(cc) + case ApplicationOnly | UserXORApplication => applicationAccess(cc) case _ => authenticatedAccess(cc) } diff --git a/obp-api/src/main/scala/code/api/dynamic/entity/helper/DynamicEntityHelper.scala b/obp-api/src/main/scala/code/api/dynamic/entity/helper/DynamicEntityHelper.scala index 9043c19a11..ef657f47e0 100644 --- a/obp-api/src/main/scala/code/api/dynamic/entity/helper/DynamicEntityHelper.scala +++ b/obp-api/src/main/scala/code/api/dynamic/entity/helper/DynamicEntityHelper.scala @@ -1081,7 +1081,8 @@ case class DynamicEntityInfo(definition: String, entityName: String, bankId: Opt /** The entity's auth mode as the framework type; unknown or empty values read as UserOnly. */ val endpointAuthMode: code.api.util.APIUtil.EndpointAuthMode = authMode match { case code.dynamicEntity.DynamicEntityAuthMode.ApplicationOnly => code.api.util.APIUtil.ApplicationOnly - case code.dynamicEntity.DynamicEntityAuthMode.UserOrApplication => code.api.util.APIUtil.UserOrApplication + case code.dynamicEntity.DynamicEntityAuthMode.UserXORApplication | code.dynamicEntity.DynamicEntityAuthMode.LegacyUserOrApplication => + code.api.util.APIUtil.UserXORApplication case code.dynamicEntity.DynamicEntityAuthMode.UserAndApplication => code.api.util.APIUtil.UserAndApplication case _ => code.api.util.APIUtil.UserOnly } diff --git a/obp-api/src/main/scala/code/api/util/APIUtil.scala b/obp-api/src/main/scala/code/api/util/APIUtil.scala index 83e2855fd6..f82a6bccc7 100644 --- a/obp-api/src/main/scala/code/api/util/APIUtil.scala +++ b/obp-api/src/main/scala/code/api/util/APIUtil.scala @@ -1587,7 +1587,9 @@ object APIUtil extends MdcLoggable with CustomJsonFormats{ sealed trait EndpointAuthMode case object UserOnly extends EndpointAuthMode case object ApplicationOnly extends EndpointAuthMode - case object UserOrApplication extends EndpointAuthMode + /** Either a person or the application acting as itself, never a mix: when the application acts as itself + * (see isApplicationActingAsItself) only its Consumer's Scopes count; otherwise only the User's Entitlements. */ + case object UserXORApplication extends EndpointAuthMode case object UserAndApplication extends EndpointAuthMode // Used to document the API calls @@ -1622,7 +1624,7 @@ object APIUtil extends MdcLoggable with CustomJsonFormats{ specialInstructions: Option[String] = None, var specifiedUrl: Option[String] = None, // A derived value: Contains the called version (added at run time). See the resource doc for resource doc! createdByBankId: Option[String] = None, //we need to filter the resource Doc by BankId - authMode: EndpointAuthMode = UserOnly, // Per-endpoint auth mode: UserOnly, ApplicationOnly, UserOrApplication, UserAndApplication + authMode: EndpointAuthMode = UserOnly, // Per-endpoint auth mode: UserOnly, ApplicationOnly, UserXORApplication, UserAndApplication http4sPartialFunction: Http4sEndpoint = None, // http4s endpoint handler // Native http4s handler for runtime-compiled dynamic endpoints (Piece C). Defaulted to None so // no existing construction site changes. Set by DynamicResourceDocsEndpointGroup / practise group @@ -1663,16 +1665,16 @@ object APIUtil extends MdcLoggable with CustomJsonFormats{ // Auth-mode-aware error message adjustments authMode match { - case ApplicationOnly | UserOrApplication => + case ApplicationOnly | UserXORApplication => errorResponseBodies ?+= ApplicationNotIdentified if (authMode == ApplicationOnly) { errorResponseBodies ?-= AuthenticatedUserIsRequired } - if (authMode == UserOrApplication) { + if (authMode == UserXORApplication) { description += s""" | - |This endpoint supports **User OR Application** authentication. You can authenticate either as a logged-in User (with Entitlements) or as an Application using a Consumer Key (with Scopes). + |This endpoint supports **User XOR Application** authorisation: either a person acting on their own Entitlements, or an Application acting as itself (e.g. with an OAuth2 client-credentials token) on its Consumer's Scopes. A person calling through a Consumer does not get that Consumer's Scopes. |See ${Glossary.getGlossaryItemLink("API.Endpoint Auth Modes")} for more information. |""" } @@ -1840,7 +1842,7 @@ object APIUtil extends MdcLoggable with CustomJsonFormats{ def checkAuth(cc: CallContext): Future[(Box[User], Option[CallContext])] = authMode match { case UserOnly | UserAndApplication => if (AuthCheckIsRequired) authenticatedAccessFun(cc) else anonymousAccessFun(cc) - case ApplicationOnly | UserOrApplication => applicationAccessFun(cc) + case ApplicationOnly | UserXORApplication => applicationAccessFun(cc) } def checkObpIds(pairs: List[(String, String)], callContext: Option[CallContext]): Future[Option[CallContext]] = Future { val invalid = pairs.filter(p => !checkObpId(p._2).equals(SILENCE_IS_GOLDEN)) @@ -2153,14 +2155,32 @@ object APIUtil extends MdcLoggable with CustomJsonFormats{ user_ids.filter(_ == user_id).length > 0 } - // Virtual roles granted by super_admin_user_ids prop - val superAdminVirtualRoles: List[String] = List("CanCreateEntitlementAtOneBank", "CanCreateEntitlementAtAnyBank", "CanGetAnyUser") + // Virtual roles granted by super_admin_user_ids prop. The entitlement request Roles let a super admin see + // and accept requests (accepting deletes the request) on a fresh instance, where nobody holds them yet; + // they add no power, since a super admin can grant themselves any Role. + val superAdminVirtualRoles: List[String] = List("CanCreateEntitlementAtOneBank", "CanCreateEntitlementAtAnyBank", "CanGetAnyUser", + "CanGetEntitlementRequestsAtAnyBank", "CanDeleteEntitlementRequestsAtAnyBank") // Virtual roles granted by oidc_operator_user_ids prop val oidcOperatorVirtualRoles: List[String] = List("CanGetAnyUser", "CanVerifyUserCredentials", "CanVerifyOidcClient", "CanGetOidcClient", "CanGetOidcConsent", "CanGetConsumers", "CanCreateConsumer") def hasScope(bankId: String, consumerId: String, role: ApiRole): Boolean = { !Scope.scope.vend.getScope(bankId, consumerId, role.toString).isEmpty } + /** True when no person is behind the call: there is no User, or the User is the application's own, the one an + * OBP-OIDC client-credentials token resolves to (provider obp-oidc#client, named after the Consumer's client id). + * consumerId is the Consumer's primary key, as from getConsumerPrimaryKey. */ + def isApplicationActingAsItself(userId: String, consumerId: String): Boolean = + if (userId.isEmpty) consumerId.nonEmpty + else (for { + user <- Users.users.vend.getUserByUserId(userId) + primaryKey <- tryo(consumerId.toLong) + consumer <- Consumers.consumers.vend.getConsumerByPrimaryId(primaryKey) + } yield isApplicationsOwnUser(user, consumer)).openOr(false) + + def isApplicationsOwnUser(user: User, consumer: code.model.Consumer): Boolean = + user.provider == code.api.OAuth2Login.OBPOIDC.clientCredentialsProvider && + Set(Option(consumer.key.get), Option(consumer.azp.get)).flatten.filter(_.nonEmpty).contains(user.idGivenByProvider) + def getConsumerPrimaryKey(callContext: Option[CallContext]): String = { callContext match { case Some(cc) => @@ -2246,7 +2266,17 @@ object APIUtil extends MdcLoggable with CustomJsonFormats{ // when roles is empty, that means no access control, treat as pass auth check @deprecated("Use handleAccessControlRegardingEntitlementsAndScopes instead. It checks virtual roles (super_admin, oidc_operator), Scopes, and just-in-time entitlements in addition to Entitlements.", "OBP v6.0.0") def hasAtLeastOneEntitlement(bankId: String, userId: String, roles: List[ApiRole]): Boolean = - roles.isEmpty || roles.exists(hasEntitlement(bankId, userId, _)) + roles.isEmpty || roles.exists(role => holdsVirtualRole(userId, role) || hasEntitlement(bankId, userId, role)) + + /** True when the User holds `role` virtually, through super_admin_user_ids or oidc_operator_user_ids. The + * inline checks (hasAtLeastOneEntitlement, hasAllEntitlements) honour these as the middleware's check does, + * so an endpoint that checks its Roles again in its handler doesn't refuse what its ResourceDoc allowed. */ + def holdsVirtualRole(userId: String, role: ApiRole): Boolean = { + val virtualRoles = if (isSuperAdmin(userId)) superAdminVirtualRoles + else if (isOidcOperator(userId)) oidcOperatorVirtualRoles + else List.empty + virtualRoles.contains(role.toString) + } /** * Grant a caller the Roles they are missing, at the moment they first need them. @@ -2383,9 +2413,11 @@ object APIUtil extends MdcLoggable with CustomJsonFormats{ * Per-endpoint auth mode access control. Checks virtual roles first, then matches on authMode: * - UserOnly: user entitlements only (includes just-in-time entitlements) * - ApplicationOnly: consumer scopes only - * - UserOrApplication: scopes OR entitlements + * - UserXORApplication: consumer scopes when the application acts as itself, otherwise user entitlements * - UserAndApplication: scopes AND entitlements - * Global overrides (require_scopes_for_all_roles, require_scopes_for_listed_roles) force UserAndApplication behavior. + * Global overrides (require_scopes_for_all_roles, require_scopes_for_listed_roles) force UserAndApplication behavior, + * except for an application acting as itself on an ApplicationOnly or UserXORApplication endpoint: it has no + * Entitlements to add, so it keeps needing only its Scopes. */ def handleAccessControlWithAuthMode(bankId: String, userId: String, consumerId: String, roles: List[ApiRole], authMode: EndpointAuthMode): Boolean = { if (roles.isEmpty) { @@ -2411,14 +2443,19 @@ object APIUtil extends MdcLoggable with CustomJsonFormats{ def consumerHasTheScopes: Boolean = roles.exists(role => hasScope(if (role.requiresBankId) bankId else "", consumerId, role)) + lazy val applicationActsAsItself = isApplicationActingAsItself(userId, consumerId) + if (globalOverrideToUserAndApp) { - // Global config forces both scopes AND entitlements - userHasTheRoles && roles.exists(hasScope(bankId, consumerId, _)) + authMode match { + case ApplicationOnly | UserXORApplication if applicationActsAsItself => consumerHasTheScopes + // Global config forces both scopes AND entitlements + case _ => userHasTheRoles && roles.exists(hasScope(bankId, consumerId, _)) + } } else { authMode match { case UserOnly => userHasTheRoles case ApplicationOnly => consumerHasTheScopes - case UserOrApplication => consumerHasTheScopes || userHasTheRoles + case UserXORApplication => if (applicationActsAsItself) consumerHasTheScopes else userHasTheRoles case UserAndApplication => userHasTheRoles && consumerHasTheScopes } } @@ -2432,7 +2469,7 @@ object APIUtil extends MdcLoggable with CustomJsonFormats{ // TODO Should we accept Option[BankId] for bankId instead of String ? @deprecated("Use handleAccessControlRegardingEntitlementsAndScopes instead. It checks virtual roles (super_admin, oidc_operator), Scopes, and just-in-time entitlements in addition to Entitlements.", "OBP v6.0.0") def hasAllEntitlements(bankId: String, userId: String, roles: List[ApiRole]): Boolean = - roles.forall(hasEntitlement(bankId, userId, _)) + roles.forall(role => holdsVirtualRole(userId, role) || hasEntitlement(bankId, userId, role)) def getCustomers(ids: List[String]): List[Customer] = { val customers = { @@ -3488,6 +3525,18 @@ object APIUtil extends MdcLoggable with CustomJsonFormats{ registerDefault(nameOfProperty, defaultValue.toString) getPropsValue(nameOfProperty) map(toBoolean) openOr(defaultValue) } + /** + * The Transaction Request types this instance advertises: `transactionRequests_supported_types`, with + * OPEN_CORRIDOR_PROMISE present exactly when `open_corridor_enabled` is true. Open Corridor has its own + * switch rather than a place in the list, and this keeps what is advertised in line with what is served. + */ + def advertisedTransactionRequestTypes: List[String] = { + val openCorridor = com.openbankproject.commons.model.enums.TransactionRequestTypes.OPEN_CORRIDOR_PROMISE.toString + val listed = getPropsValue("transactionRequests_supported_types", "") + .split(",").map(_.trim).filter(_.nonEmpty).toList.filterNot(_ == openCorridor) + if (getPropsAsBoolValue("open_corridor_enabled", false)) listed :+ openCorridor else listed + } + def getPropsAsIntValue(nameOfProperty: String): Box[Int] = { getPropsValue(nameOfProperty) map(toInt) } @@ -3853,6 +3902,7 @@ object APIUtil extends MdcLoggable with CustomJsonFormats{ "public_obp_portal_url" -> getPropsValue("public_obp_portal_url").openOr(getPropsValue("portal_external_url").openOr("http://localhost:5174")), "public_obp_api_explorer_url" -> getPropsValue("public_obp_api_explorer_url").openOr("http://localhost:5173"), "public_obp_api_manager_url" -> getPropsValue("public_obp_api_manager_url").openOr("http://localhost:3003"), + "public_obp_cohort_url" -> getPropsValue("public_obp_cohort_url").openOr("http://localhost:5183"), "public_obp_sandbox_populator_url" -> getPropsValue("public_obp_sandbox_populator_url").openOr("http://localhost:5178"), "public_obp_oidc_url" -> getPropsValue("public_obp_oidc_url").openOr("http://localhost:9000"), "public_keycloak_url" -> getPropsValue("public_keycloak_url").openOr("http://localhost:7787"), diff --git a/obp-api/src/main/scala/code/api/util/ApiRole.scala b/obp-api/src/main/scala/code/api/util/ApiRole.scala index e45bd657bc..0e30a86af3 100644 --- a/obp-api/src/main/scala/code/api/util/ApiRole.scala +++ b/obp-api/src/main/scala/code/api/util/ApiRole.scala @@ -485,6 +485,29 @@ object ApiRole extends MdcLoggable{ case class CanUpdateProductTagsAtAnyBank(requiresBankId: Boolean = false) extends ApiRole lazy val canUpdateProductTagsAtAnyBank = CanUpdateProductTagsAtAnyBank() + case class CanCreateAsset(requiresBankId: Boolean = true) extends ApiRole + lazy val canCreateAsset = CanCreateAsset() + + // Files: held at a bank or at SYS (the system space). There are deliberately no AtAnyBank versions. + case class CanCreateFileAtOneBank(requiresBankId: Boolean = true) extends ApiRole + lazy val canCreateFileAtOneBank = CanCreateFileAtOneBank() + + case class CanCreatePublicFileAtOneBank(requiresBankId: Boolean = true) extends ApiRole + lazy val canCreatePublicFileAtOneBank = CanCreatePublicFileAtOneBank() + + case class CanGetFileAtOneBank(requiresBankId: Boolean = true) extends ApiRole + lazy val canGetFileAtOneBank = CanGetFileAtOneBank() + // Sets a file's scan status (pending, clean, rejected), and reads the files of the space that are not clean, + // so a scanner can fetch what it has to scan. Held by a scanner app as a Scope, or by staff who approve files. + case class CanUpdateFileScanStatusAtOneBank(requiresBankId: Boolean = true) extends ApiRole + lazy val canUpdateFileScanStatusAtOneBank = CanUpdateFileScanStatusAtOneBank() + + // Queues a plain-text email to one of the bank's Customers through the message outbox (Send Email to Customer). + // Held by an application, such as a programme website, as a Scope, to confirm what a Customer did or tell them + // about a decision. + case class CanSendEmailToCustomerAtOneBank(requiresBankId: Boolean = true) extends ApiRole + lazy val canSendEmailToCustomerAtOneBank = CanSendEmailToCustomerAtOneBank() + case class CanCreateFxRate(requiresBankId: Boolean = true) extends ApiRole lazy val canCreateFxRate = CanCreateFxRate() @@ -637,6 +660,30 @@ object ApiRole extends MdcLoggable{ case class CanDeleteDynamicEntityWebhook(requiresBankId: Boolean = true) extends ApiRole lazy val canDeleteDynamicEntityWebhook = CanDeleteDynamicEntityWebhook() + // Data retention: how long a space's resources stay live, legal holds, runs and approvals, and the + // retention archive. Each Role targets one space: held at a bank id, or at SYS for the system space. + // See RETENTION_PLAN.md. + case class CanManageRetentionPolicy(requiresBankId: Boolean = true) extends ApiRole + lazy val canManageRetentionPolicy = CanManageRetentionPolicy() + + case class CanGetRetentionPolicies(requiresBankId: Boolean = true) extends ApiRole + lazy val canGetRetentionPolicies = CanGetRetentionPolicies() + + case class CanRunRetentionPolicy(requiresBankId: Boolean = true) extends ApiRole + lazy val canRunRetentionPolicy = CanRunRetentionPolicy() + + case class CanApproveRetentionRun(requiresBankId: Boolean = true) extends ApiRole + lazy val canApproveRetentionRun = CanApproveRetentionRun() + + case class CanGetRetentionRuns(requiresBankId: Boolean = true) extends ApiRole + lazy val canGetRetentionRuns = CanGetRetentionRuns() + + case class CanManageRetentionHold(requiresBankId: Boolean = true) extends ApiRole + lazy val canManageRetentionHold = CanManageRetentionHold() + + case class CanGetRetentionArchive(requiresBankId: Boolean = true) extends ApiRole + lazy val canGetRetentionArchive = CanGetRetentionArchive() + // Shows which Consumers and client IP addresses are sending the most traffic to the instance // (TrafficSources). About the instance, so held at the empty bank id. It names Consumers and IP // addresses, which is why it is a Role of its own and not part of CanGetTelemetry. @@ -1531,15 +1578,15 @@ object ApiRole extends MdcLoggable{ case class CanGetGroupsAtOneBank(requiresBankId: Boolean = true) extends ApiRole lazy val canGetGroupsAtOneBank = CanGetGroupsAtOneBank() - // Organisation management roles - case class CanCreateOrganisation(requiresBankId: Boolean = false) extends ApiRole - lazy val canCreateOrganisation = CanCreateOrganisation() - case class CanGetAnyOrganisation(requiresBankId: Boolean = false) extends ApiRole - lazy val canGetAnyOrganisation = CanGetAnyOrganisation() - case class CanUpdateOrganisation(requiresBankId: Boolean = false) extends ApiRole - lazy val canUpdateOrganisation = CanUpdateOrganisation() - case class CanDeleteOrganisation(requiresBankId: Boolean = false) extends ApiRole - lazy val canDeleteOrganisation = CanDeleteOrganisation() + // Tenant management roles. A Tenant belongs to no bank, so these are system Roles. + case class CanCreateTenant(requiresBankId: Boolean = false) extends ApiRole + lazy val canCreateTenant = CanCreateTenant() + case class CanGetTenant(requiresBankId: Boolean = false) extends ApiRole + lazy val canGetTenant = CanGetTenant() + case class CanUpdateTenant(requiresBankId: Boolean = false) extends ApiRole + lazy val canUpdateTenant = CanUpdateTenant() + case class CanDeleteTenant(requiresBankId: Boolean = false) extends ApiRole + lazy val canDeleteTenant = CanDeleteTenant() // Routing Scheme registry roles (system-scoped: schemes are global infrastructure) case class CanCreateRoutingScheme(requiresBankId: Boolean = false) extends ApiRole diff --git a/obp-api/src/main/scala/code/api/util/ApiSession.scala b/obp-api/src/main/scala/code/api/util/ApiSession.scala index 63ff17004c..8888ceec9e 100644 --- a/obp-api/src/main/scala/code/api/util/ApiSession.scala +++ b/obp-api/src/main/scala/code/api/util/ApiSession.scala @@ -130,7 +130,7 @@ case class CallContext( // request, but only the data-read endpoints run on the principal; the consent-management // endpoints (GET/DELETE account-access-consents) do not, and they are how a TPP inspects // or revokes the very consent that cannot be resolved. Failing at authentication would - // lock that door too -- and, since those endpoints are UserOrApplication, would report it + // lock that door too -- and, since those endpoints are UserXORApplication, would report it // as ApplicationNotIdentified rather than as anything to do with the consent. ukConsentUnresolved: Option[String] = None, // How the caller's certificate was established (PeerTrust.Resolution.mode): diff --git a/obp-api/src/main/scala/code/api/util/ApiTag.scala b/obp-api/src/main/scala/code/api/util/ApiTag.scala index adc2db0e02..084d8d4f8e 100644 --- a/obp-api/src/main/scala/code/api/util/ApiTag.scala +++ b/obp-api/src/main/scala/code/api/util/ApiTag.scala @@ -120,7 +120,7 @@ object ApiTag { val apiTagBalance = ResourceDocTag("Balance") val apiTagChat = ResourceDocTag("Chat") val apiTagGroup = ResourceDocTag("Group") - val apiTagOrganisation = ResourceDocTag("Organisation") + val apiTagTenant = ResourceDocTag("Tenant") val apiTagRoutingScheme = ResourceDocTag("Routing-Scheme") val apiTagPayee = ResourceDocTag("Payee") val apiTagWebhook = ResourceDocTag("Webhook") @@ -201,6 +201,8 @@ object ApiTag { val apiTagSignalChannel = ResourceDocTag("Signal-Channel") val apiTagFinancialCrime = ResourceDocTag("Financial-Crime") val apiTagAsset = ResourceDocTag("Asset") + val apiTagFile = ResourceDocTag("File") + val apiTagDataRetention = ResourceDocTag("Data-Retention") private[this] val tagNameSymbolMapTag: MutableMap[String, ResourceDocTag] = MutableMap() diff --git a/obp-api/src/main/scala/code/api/util/ConsentUtil.scala b/obp-api/src/main/scala/code/api/util/ConsentUtil.scala index 0b196fa554..1151eb49ca 100644 --- a/obp-api/src/main/scala/code/api/util/ConsentUtil.scala +++ b/obp-api/src/main/scala/code/api/util/ConsentUtil.scala @@ -719,6 +719,15 @@ object Consent extends MdcLoggable { } } + /** Whether this instance uses consents at all (consents.allowed, default false): as a login and, so that no consent + * is made that could never be used, when one is created. */ + def consentsAllowed: Boolean = APIUtil.getPropsAsBoolValue(nameOfProperty = "consents.allowed", defaultValue = false) + + /** The first step of every endpoint that creates a consent or a consent request: refused with OBP-35007 while + * consents.allowed is off, before any challenge is sent or anything is stored. */ + def checkConsentsAllowed(callContext: Option[CallContext]): Future[Box[Unit]] = + Helper.booleanToFuture(ErrorMessages.ConsentDisabled, 401, callContext)(consentsAllowed) + def applyRules(consentJwt: Option[String], callContext: CallContext): Future[(Box[User], Option[CallContext])] = { val allowed = APIUtil.getPropsAsBoolValue(nameOfProperty="consents.allowed", defaultValue=false) (consentJwt, allowed) match { diff --git a/obp-api/src/main/scala/code/api/util/ErrorMessages.scala b/obp-api/src/main/scala/code/api/util/ErrorMessages.scala index e2bb032945..7ad767f3a4 100644 --- a/obp-api/src/main/scala/code/api/util/ErrorMessages.scala +++ b/obp-api/src/main/scala/code/api/util/ErrorMessages.scala @@ -589,19 +589,64 @@ object ErrorMessages { val GlossaryItemShadowsStaticItem = "OBP-30577: A static Glossary Item with this title already exists. Creating this item would override it in the Glossary. Set overrides_static_item to true if that is intended, or choose a different title." val InvalidGlossarySource = "OBP-30578: Invalid source. Please specify all, static or dynamic." - // Asset registry (OBP-30579 .. OBP-30581) + // Asset registry (OBP-30579 .. OBP-30585) val AssetNotFound = "OBP-30579: Asset not found. Please specify a valid value for ASSET_CODE." val AssetNotFoundByChainIdentity = "OBP-30580: No asset has this chain identity. Please specify a registered CHAIN_SCHEME and CHAIN_ASSET_ID." val InvalidAssetQueryParameter = "OBP-30581: Invalid query parameter for assets." - - val OrganisationNotFound = "OBP-30506: Organisation not found. Please specify a valid value for ORGANISATION_ID." - val OrganisationAlreadyExists = "OBP-30507: Organisation already exists. Please specify a different value for ORGANISATION_ID." - val InvalidOrganisationIdFormat = "OBP-30508: Invalid Organisation Id. The ORGANISATION_ID should only contain 0-9/a-z/A-Z/'-'/'.'/'_', and be between 2 and 64 characters in length." - val InvalidOrganisationStatus = "OBP-30509: Invalid Organisation status. Allowed values are: active, suspended, archived." - val InvalidOrganisationVisibility = "OBP-30510: Invalid Organisation visibility. Allowed values are: public, unlisted, private." - val CreateOrganisationError = "OBP-30511: Could not create Organisation." - val UpdateOrganisationError = "OBP-30512: Could not update Organisation." - val DeleteOrganisationError = "OBP-30513: Could not delete Organisation." + val InvalidAsset = "OBP-30582: Invalid asset." + val AssetAlreadyExists = "OBP-30583: An asset with this code already exists. Please specify a different value for asset_code." + val AssetChainIdentityAlreadyExists = "OBP-30584: Another asset already has this chain identity. Please specify a different chain_scheme and chain_asset_id." + val AssetDtiAlreadyExists = "OBP-30585: Another asset already has this DTI. Please specify a different value for dti." + + // Files (OBP-30586 .. OBP-30596) + val FileNotFound = "OBP-30586: File not found. Please specify a valid value for FILE_ID." + val InvalidFile = "OBP-30587: Invalid file." + val FileTooLarge = "OBP-30588: The file is larger than this instance allows." + val FileMediaTypeNotAllowed = "OBP-30589: This media type is not allowed for files." + val FileSha256Mismatch = "OBP-30590: The sha256 given does not match the content of the file." + val FileAttachmentAlreadyExists = "OBP-30591: The file is already attached to this record." + val FileAttachmentNotFound = "OBP-30592: File attachment not found. Please specify a valid value for FILE_ATTACHMENT_ID." + val FileAccessAlreadyExists = "OBP-30593: This User already has access to the file." + val FileAccessNotFound = "OBP-30594: This User has not been given access to the file." + val InvalidFileQueryParameter = "OBP-30595: Invalid query parameter for files." + val FileOwnerRequired = "OBP-30596: Only the file's owner may do this." + val PersonalFilesNotEnabled = "OBP-30598: Personal files are not enabled in this space." + val PersonalFileQuotaExceeded = "OBP-30599: This upload would exceed the personal files allowed per User in this space." + val PersonalFileRecordNotOwned = "OBP-30600: A personal file can only be attached to a personal Dynamic Entity record the file's owner owns." + val PersonalFileEmailNotValidated = "OBP-30601: Personal files need the User's email address to be validated first." + val FileNotYetScanned = "OBP-30602: The file has not been scanned yet. Its content is served once its scan status is clean." + val FileScanRejected = "OBP-30603: The file was rejected by its scan. Its content is not served." + val InvalidFileScanStatus = "OBP-30604: Invalid scan_status. Allowed values are: pending, clean, rejected." + val CustomerEmailsNotInSelfServiceBank = "OBP-30605: Customers of a self-service bank cannot be emailed." + val CustomerEmailSenderNotConfigured = "OBP-30606: No From address is set for emails to Customers. Set the prop customer_emails.sender_address." + val CustomerHasNoEmail = "OBP-30607: The Customer has no usable email address." + val CustomerEmailLimitReached = "OBP-30608: Too many emails have been sent to this Customer today." + + val TenantNotFound = "OBP-30506: Tenant not found. Please specify a valid value for TENANT_ID." + val TenantAlreadyExists = "OBP-30507: Tenant already exists. Please specify a different value for TENANT_ID." + val InvalidTenantIdFormat = "OBP-30508: Invalid Tenant Id. The TENANT_ID should only contain 0-9/a-z/A-Z/'-'/'.'/'_', and be between 2 and 64 characters in length." + val InvalidTenantStatus = "OBP-30509: Invalid Tenant status. Allowed values are: active, suspended, archived." + val InvalidTenantVisibility = "OBP-30510: Invalid Tenant visibility. Allowed values are: public, unlisted, private." + val CreateTenantError = "OBP-30511: Could not create Tenant." + val UpdateTenantError = "OBP-30512: Could not update Tenant." + val DeleteTenantError = "OBP-30513: Could not delete Tenant." + + // Data retention (OBP-30582 .. OBP-30596) + val RetentionPolicyNotFound = "OBP-30609: Retention policy not found in this space. Please specify a valid value for RETENTION_POLICY_ID." + val InvalidRetentionPolicy = "OBP-30610: Invalid retention policy. " + val RetentionRunNotFound = "OBP-30611: Retention run not found in this space. Please specify a valid value for RETENTION_RUN_ID." + val RetentionArchiveIsDisabled = "OBP-30612: Archiving is switched off on this instance, so a run that would move resources into the retention archive is refused. Set retention.archive.enabled=true in the props to allow it. A preview always works." + val RetentionDeleteIsDisabled = "OBP-30613: Deleting is switched off on this instance, so a run that would delete or anonymise resources is refused. Set retention.delete.enabled=true in the props to allow it. A preview always works." + val RetentionRunNotPending = "OBP-30614: This retention run is not waiting for approval. Only a pending run can be approved or rejected." + val RetentionRunApproverIsRequester = "OBP-30615: A retention run must be approved or rejected by a different user from the one who requested it." + val RetentionHoldNotFound = "OBP-30616: Retention hold not found in this space. Please specify a valid value for RETENTION_HOLD_ID." + val InvalidRetentionHold = "OBP-30617: Invalid retention hold. " + val RetentionArchiveEntryNotFound = "OBP-30618: Retention archive entry not found in this space. Please specify a valid value for RETENTION_ARCHIVE_ID." + val RetentionRunAlreadyInProgress = "OBP-30619: A run of this retention policy is already in progress. Try again when it has ended." + val InvalidRetentionQueryParameter = "OBP-30620: Invalid query parameter for data retention. " + val RetentionHoldAlreadyReleased = "OBP-30621: This retention hold has already been released." + val InvalidRetentionRunApproval = "OBP-30622: Invalid approval. decision must be approve or reject." + val InvalidRetentionRun = "OBP-30623: Invalid retention run. mode must be preview or execute." // Routing Scheme registry (OBP-30514 .. OBP-30525) val RoutingSchemeNotFound = "OBP-30514: Routing Scheme not found. Please specify a valid value for SCHEME." @@ -754,6 +799,7 @@ object ErrorMessages { val GetConsentImplicitSCAError = "OBP-30229: Could not get the implicit SCA consent." val SelfServiceBankCreationDisabled = "OBP-30230: Self-service bank creation is disabled on this instance." val SelfServiceBankLimitReached = "OBP-30231: Self-service bank creation limit reached. To create more banks you require the role CanCreateBank." + val SelfServiceBankNeedsMappedConnector = "OBP-30232: Self-service banks can only be created where OBP stores banks itself (the mapped connector). A User may not create a bank on a backend system." val CreateSystemViewError = "OBP-30250: Could not create the system view" val DeleteSystemViewError = "OBP-30251: Could not delete the system view" diff --git a/obp-api/src/main/scala/code/api/util/Glossary.scala b/obp-api/src/main/scala/code/api/util/Glossary.scala index 8c59b4f8a7..fea65cb89e 100644 --- a/obp-api/src/main/scala/code/api/util/Glossary.scala +++ b/obp-api/src/main/scala/code/api/util/Glossary.scala @@ -1263,11 +1263,11 @@ object Glossary extends MdcLoggable { | |* **ApplicationOnly**: Only the Consumer's Scopes are checked. No User is required. | -|* **UserOrApplication**: Access is granted if the Consumer has the required Scope **OR** the User has the required Entitlement. This effectively gives the union of both. +|* **UserXORApplication**: Either a person or the Application acting as itself, never a mix. When the Application acts as itself (there is no User, or the User is the Application's own, the one an OBP-OIDC client-credentials token resolves to: provider *obp-oidc#client*, named after the client id), only the Consumer's Scopes are checked. Otherwise a person is acting, and only the User's Entitlements are checked: the Consumer's Scopes are ignored. It was called *UserOrApplication* until October 2026, when it granted access if either held the Role. | |* **UserAndApplication**: Access is granted only if the Consumer has the required Scope **AND** the User has the required Entitlement. Both are required. | -|For example, if a User logs in via DirectLogin with a Consumer that has the Scope *CanGetConsumers*, and the endpoint's authMode is *UserOrApplication*, the User can access the endpoint even without a personal *CanGetConsumers* Entitlement (because the Consumer's Scope is sufficient). +|For example, if a User logs in via DirectLogin with a Consumer that has the Scope *CanGetConsumers*, and the endpoint's authMode is *UserXORApplication*, the User still needs a personal *CanGetConsumers* Entitlement: a person never acts on the Consumer's Scopes. The same Consumer calling with a client-credentials token, as itself, is let in on its Scope. So whoever is let in is also who is recorded as acting. | |The authMode is set in the ResourceDoc definition for each endpoint, for example: | @@ -1281,13 +1281,13 @@ object Glossary extends MdcLoggable { | "Get Consumers", | ..., | Some(List(canGetConsumers)), -| authMode = UserOrApplication +| authMode = UserXORApplication |) |``` | |If authMode is not specified, it defaults to UserOnly. | -|Note: If the property *require_scopes_for_all_roles* is set to true, all endpoints behave as *UserAndApplication* regardless of their configured authMode. +|Note: If the property *require_scopes_for_all_roles* is set to true (or *require_scopes_for_listed_roles* lists the Role), endpoints behave as *UserAndApplication* regardless of their configured authMode, except that an Application acting as itself on an *ApplicationOnly* or *UserXORApplication* endpoint still needs only its Scopes: it has no Entitlements to add. | |See also: [Access Control](/index#API.Access-Control), [Scopes](/index#group-Scope), [Roles](/index#group-Role) | @@ -1843,23 +1843,19 @@ object Glossary extends MdcLoggable { val delegated = UserReference.all.filter(ref => ref.policy != AttributionPolicy.UseAuthenticatedUserId) val applied = delegated.count(UserReference.isWired) - s"""When a User grants a ${getGlossaryItemLink("Consent")} to an application, such as an AI agent, the application calls OBP under an identity of its own: a consent user, which OBP creates for that Consent. A call made that way has two Users behind it. `user_id` is the caller, the consent user that authenticated. `on_behalf_of_user_id` is the User the call was made for, the one who granted the Consent. When a User calls OBP for themselves, the two are the same User. + s"""An application acting for a User, such as an AI agent, calls OBP with a [Consent](/glossary#Consent) and authenticates as a consent user, an identity OBP creates for that Consent. `user_id` is the caller (the consent user); `on_behalf_of_user_id` is the User who granted the Consent. OBP derives the latter from the Consent, never from the request. | - |OBP finds the User acted for itself, from the Consent the caller authenticated with. It never takes it from a request body, header or query parameter, so a caller cannot name somebody else to act for. + |Rows stored against a consent user would end with its Consent, so each column that stores a user id has a policy for calls made by a consent user: | - |This matters whenever OBP stores a user id. A consent user belongs to its Consent, so a row stored against it, such as an account holder, belongs to an identity that ends with the Consent, and the User who asked for it can no longer see it. So every database column that stores a user id has a policy, which says whose id to store when a consent user makes the call: + |* **the caller**: store the consent user (audit records, and the Consent's own permissions). + |* **the User acted for**: store the User who granted the Consent. Some tables keep both ids; see Note. + |* **refused**: a consent user may not do this (create a Consent, a Consumer or a token). | - |* **the caller**: store the consent user's own id. This suits audit records of who acted, and the rows that carry a Consent's own permissions, which are meant to end with it. - |* **the User acted for**: store the id of the User who granted the Consent, so the row outlives the Consent. A few tables keep both ids, for example a counterparty or a transaction request, where who made the call is worth keeping too; the Note column says so. - |* **refused**: a consent user must not do this at all, because what it would create (a Consent, a Consumer or a token) outlives the Consent and would let the delegation extend itself. + |A request naming a consent user as its target User gets 400 `InvalidUserId`. v7.0.0 responses that show both ids use `user_id` and `on_behalf_of_user_id`, e.g. [Dynamic Entity](/glossary#Dynamic-Entities) record metadata. | - |A request that explicitly names a consent user as the target User, for example in a `user_id` field or a `USER_ID` path segment, is answered with 400 `InvalidUserId`. + |**Columns that store a user id** | - |Where a v7.0.0 response publishes both ids, it names them `user_id` and `on_behalf_of_user_id`, as in the record metadata of a ${getGlossaryItemLink("Dynamic-Entities")} record. - | - |**Every column that stores a user id** - | - |Policy is whose id the column should hold when a consent user makes the call. Today is whose id it holds on this instance now. Of the ${delegated.size} columns whose policy is "the User acted for" or "refused", ${applied} apply it today. "Not written by a consent user" means no API call writes that column with a consent user's id, because the endpoint refuses the consent user first or no endpoint writes the table. + |Today is what this instance stores now: ${applied} of the ${delegated.size} columns whose policy is "the User acted for" or "refused" apply it. "Not written by a consent user" means no endpoint writes that column with a consent user's id. | |""".stripMargin + (List("| Record | Column | Policy | Today | Note |", "|---|---|---|---|---|") ++ rows).mkString("\n") + "\n" @@ -4091,7 +4087,7 @@ object Glossary extends MdcLoggable { | |Two settings apply on top of the routes: | -|* `authMode` says which credential satisfies the role checks on the System route: `UserOnly` (Entitlements), `ApplicationOnly` (Consumer Scopes), `UserOrApplication`, `UserAndApplication`. The Personal and Row level routes always need a User; `ApplicationOnly` is refused on an entity with `hasPersonalEntity`. +|* `authMode` says which credential satisfies the role checks on the System route: `UserOnly` (Entitlements), `ApplicationOnly` (Consumer Scopes), `UserXORApplication` (Scopes for the Application acting as itself, Entitlements for a person; `UserOrApplication` is accepted as its former name), `UserAndApplication`. The Personal and Row level routes always need a User; `ApplicationOnly` is refused on an entity with `hasPersonalEntity`. |* Field roles: a field with a `read_role` is omitted from every response unless the caller holds that role; a field with a `write_role` is only changed by PATCH from a holder (POST ignores it, PUT preserves its value). | |**Managing a record's access list (row level)** @@ -7585,6 +7581,10 @@ object Glossary extends MdcLoggable { | |Every Asset is administered at exactly one bank: the issuer for the types a bank issues, and the `SYS` bank for everything else. | + |## Adding an Asset + | + |A bank adds an Asset it issues with Create Asset at that bank, and the bank is recorded as its issuer. A crypto asset with no issuer is added at the `SYS` bank. Either way the caller needs the CanCreateAsset Role at that bank. Currencies, precious metals and accounting units come from ISO 4217 and cannot be added. The code of an Asset a bank issues is 4 to 10 letters or digits, so it can never be mistaken for an ISO 4217 code, which is always 3. Once added, the code is accepted in `currency` fields at once, with the Asset's own decimal places. + | |## Codes OBP accepts | |The registry decides which currency codes OBP accepts and how many decimal places each one has. OBP also accepts `lovelace` and `wei`, the smallest units of ADA and ETH, which are not Assets of their own. @@ -7599,13 +7599,84 @@ object Glossary extends MdcLoggable { | |## Endpoints | - |These need no authentication. + |The three reads need no authentication. | + |- [Create Asset](${apiExplorerUrl}/resource-docs/OBPv7.0.0?operationid=OBPv7.0.0-createAsset): `POST /obp/v7.0.0/banks/BANK_ID/assets` |- [Get Assets](${apiExplorerUrl}/resource-docs/OBPv7.0.0?operationid=OBPv7.0.0-getAssets): `GET /obp/v7.0.0/assets` |- [Get Asset](${apiExplorerUrl}/resource-docs/OBPv7.0.0?operationid=OBPv7.0.0-getAsset): `GET /obp/v7.0.0/assets/ASSET_CODE` |- [Get Asset by Chain Identity](${apiExplorerUrl}/resource-docs/OBPv7.0.0?operationid=OBPv7.0.0-getAssetByChainIdentity): `GET /obp/v7.0.0/assets/chain/CHAIN_SCHEME/CHAIN_ASSET_ID` |""".stripMargin) + glossaryItems += GlossaryItem( + title = "Files", + description = + s""" + |# Files + | + |A **File** is a document such as a PDF or a photograph, stored in a space: a bank, or `SYS` for the system space. A file can be attached to records, such as a Customer or a Dynamic Entity record, and can be shared with Users. + | + |## A file never changes + | + |A file's content never changes once stored, and OBP keeps its SHA-256. A new version of a document is a new file. What can change is which records the file is attached to, which Users have been given access to it, and whether it is public. Files are not deleted. + | + |## Storing a file + | + |Create File takes the file base64 encoded in JSON, with its name and media type. It needs the Role CanCreateFileAtOneBank at the space (granted at `SYS` for the system space; there is no AtAnyBank version). The caller becomes the file's owner: the User an agent acts for, when one does. + | + |At most these media types are ever accepted: image/jpeg, image/png, image/webp and application/pdf. The prop `files.allowed_media_types` can narrow that list but never extend it, and the content must really be of the type it claims. The prop `files.max_size_in_bytes` limits the size (10 MB by default). + | + |## Personal files + | + |A person may also store a file in their own name without a Role, through Create My File, for example a pitch deck for an application or a document for a KYC check. Personal files are off unless the space is listed in the prop `files.personal.enabled_bank_ids`. The owner's email address must be validated (for a User of another identity provider, having one is enough); each User holds a limited number and size of personal files per space (`files.personal.max_files_per_user`, `files.personal.max_total_bytes_per_user`, `files.personal.max_size_in_bytes`); the media types are those of `files.personal.allowed_media_types`, within the ones the space allows; and a personal file is never public. It is attached at once to one of the owner's own personal Dynamic Entity records, and only such records take it later, unless a holder of CanCreateFileAtOneBank attaches it. + | + |## Scan status + | + |Every file has a `scan_status`: pending, clean or rejected. A personal file starts pending; a file stored with the Role starts clean, as do files stored before scanning existed, except in a bank made with Create My Bank (self-service): anyone can grant themselves Roles there, so every file stored there starts pending. Until a file is clean, its details are shown as usual but its content is served only to its owner and to holders of CanUpdateFileScanStatusAtOneBank; a rejected file's content is not served at all, and only clean files are public. A scanner, holding CanUpdateFileScanStatusAtOneBank as a Scope, lists the waiting files with Get Files of a Space and `scan_status=pending`, reads them, and sets the result with Update File Scan Status; a member of staff with the Role does the same by hand, to approve a file when no scanner is configured, to reject a reported one, or to clear a false positive. + | + |## Who may read a file + | + |A logged-in caller may read a file they own, a file its owner has given them access to, or any file of a space where they hold the Role CanGetFileAtOneBank. Only the owner gives and takes access, and only to a person: a consent user is refused, since access given to one would end with its Consent. Get Files of a Space lists every file of a space for holders of CanGetFileAtOneBank. + | + |## Attaching files to records + | + |The owner, or a holder of CanCreateFileAtOneBank, attaches a file to a record and detaches it. A record is named by `record_type` (e.g. `customer`, or a Dynamic Entity's name) and `record_id`, and a file is attached to a record at most once. OBP does not look the record up, so access to a file never follows from access to a record: it comes only from owning the file, being given access to it, or the Role CanGetFileAtOneBank. + | + |These endpoints are UserXORApplication: an application acting as itself, with a client-credentials token, may hold the Roles as Scopes on its Consumer. A person calling through a Consumer still needs their own Entitlements. + | + |## Public files + | + |A file can be public, so anyone can read it without logging in. Storing a public file, or making a file public or not later, also needs the Role CanCreatePublicFileAtOneBank. Public files are served by their own endpoints under `public-files`, which serve nothing that is not public and say nothing of who stored the file. Their content may be cached for an hour, so a withdrawn file may still be served from a cache for that long. + | + |## Content + | + |The content endpoints return the file itself, not JSON, with its own `Content-Type`, its SHA-256 as the `ETag`, and `Content-Disposition` naming it (`?inline=true` to show it in a browser). So that a file can never run as a page of this site, they also send `X-Content-Type-Options: nosniff` and a sandbox `Content-Security-Policy`. + | + |## Storage + | + |The bytes are kept apart from what is known about the file. Today they are stored in the database; they can be moved to object storage without changing the API. + | + |## Endpoints + | + |- [Create File](${apiExplorerUrl}/resource-docs/OBPv7.0.0?operationid=OBPv7.0.0-createFile): `POST /obp/v7.0.0/banks/BANK_ID/files` + |- [Get File](${apiExplorerUrl}/resource-docs/OBPv7.0.0?operationid=OBPv7.0.0-getFile): `GET /obp/v7.0.0/banks/BANK_ID/files/FILE_ID` + |- [Get File Content](${apiExplorerUrl}/resource-docs/OBPv7.0.0?operationid=OBPv7.0.0-getFileContent): `GET /obp/v7.0.0/banks/BANK_ID/files/FILE_ID/content` + |- [Get Files Attached to a Record](${apiExplorerUrl}/resource-docs/OBPv7.0.0?operationid=OBPv7.0.0-getFilesAttachedToRecord): `GET /obp/v7.0.0/banks/BANK_ID/files?record_type=&record_id=` + |- [Get My Files](${apiExplorerUrl}/resource-docs/OBPv7.0.0?operationid=OBPv7.0.0-getMyFiles): `GET /obp/v7.0.0/banks/BANK_ID/my/files` + |- [Create My File](${apiExplorerUrl}/resource-docs/OBPv7.0.0?operationid=OBPv7.0.0-createMyFile): `POST /obp/v7.0.0/banks/BANK_ID/my/files`, a personal file + |- [Update File Scan Status](${apiExplorerUrl}/resource-docs/OBPv7.0.0?operationid=OBPv7.0.0-updateFileScanStatus): `PUT /obp/v7.0.0/banks/BANK_ID/files/FILE_ID/scan-status` + |- [Get Files of a Space](${apiExplorerUrl}/resource-docs/OBPv7.0.0?operationid=OBPv7.0.0-getFilesOfSpace): `GET /obp/v7.0.0/management/banks/BANK_ID/files`, every file of a space, for administrators + |- [Update File Is Public](${apiExplorerUrl}/resource-docs/OBPv7.0.0?operationid=OBPv7.0.0-updateFileIsPublic): `PUT /obp/v7.0.0/banks/BANK_ID/files/FILE_ID/is-public` + |- [Create File Attachment](${apiExplorerUrl}/resource-docs/OBPv7.0.0?operationid=OBPv7.0.0-createFileAttachment): `POST /obp/v7.0.0/banks/BANK_ID/files/FILE_ID/attachments` + |- [Get File Attachments](${apiExplorerUrl}/resource-docs/OBPv7.0.0?operationid=OBPv7.0.0-getFileAttachments): `GET /obp/v7.0.0/banks/BANK_ID/files/FILE_ID/attachments` + |- [Delete File Attachment](${apiExplorerUrl}/resource-docs/OBPv7.0.0?operationid=OBPv7.0.0-deleteFileAttachment): `DELETE /obp/v7.0.0/banks/BANK_ID/files/FILE_ID/attachments/FILE_ATTACHMENT_ID` + |- [Create File Access](${apiExplorerUrl}/resource-docs/OBPv7.0.0?operationid=OBPv7.0.0-createFileAccess): `POST /obp/v7.0.0/banks/BANK_ID/files/FILE_ID/access` + |- [Get File Access](${apiExplorerUrl}/resource-docs/OBPv7.0.0?operationid=OBPv7.0.0-getFileAccess): `GET /obp/v7.0.0/banks/BANK_ID/files/FILE_ID/access` + |- [Delete File Access](${apiExplorerUrl}/resource-docs/OBPv7.0.0?operationid=OBPv7.0.0-deleteFileAccess): `DELETE /obp/v7.0.0/banks/BANK_ID/files/FILE_ID/access/USER_ID` + |- [Get Public Files Attached to a Record](${apiExplorerUrl}/resource-docs/OBPv7.0.0?operationid=OBPv7.0.0-getPublicFilesAttachedToRecord): `GET /obp/v7.0.0/banks/BANK_ID/public-files?record_type=&record_id=` + |- [Get Public File](${apiExplorerUrl}/resource-docs/OBPv7.0.0?operationid=OBPv7.0.0-getPublicFile): `GET /obp/v7.0.0/banks/BANK_ID/public-files/FILE_ID` + |- [Get Public File Content](${apiExplorerUrl}/resource-docs/OBPv7.0.0?operationid=OBPv7.0.0-getPublicFileContent): `GET /obp/v7.0.0/banks/BANK_ID/public-files/FILE_ID/content` + |""".stripMargin) + /////////////////////////////////////////////////////////////////// // NOTE! Some glossary items are generated in ExampleValue.scala ////////////////////////////////////////////////////////////////// diff --git a/obp-api/src/main/scala/code/api/util/NewStyle.scala b/obp-api/src/main/scala/code/api/util/NewStyle.scala index e2194dbdf8..40f7489a6a 100644 --- a/obp-api/src/main/scala/code/api/util/NewStyle.scala +++ b/obp-api/src/main/scala/code/api/util/NewStyle.scala @@ -1052,7 +1052,7 @@ object NewStyle extends MdcLoggable{ else errorMsg Helper.booleanToFuture(errorInfo, cc=callContext) { - APIUtil.hasEntitlement(bankId, userId, role) + APIUtil.holdsVirtualRole(userId, role) || APIUtil.hasEntitlement(bankId, userId, role) } map validateRequestPayload(callContext) } // scala not allow overload method both have default parameter, so this method name is just in order avoid the same name with hasEntitlement diff --git a/obp-api/src/main/scala/code/api/util/http4s/AppsPage.scala b/obp-api/src/main/scala/code/api/util/http4s/AppsPage.scala index c0c5cd7a7f..66ba5d3407 100644 --- a/obp-api/src/main/scala/code/api/util/http4s/AppsPage.scala +++ b/obp-api/src/main/scala/code/api/util/http4s/AppsPage.scala @@ -63,6 +63,7 @@ object AppsPage { private val appProbes: Map[String, Set[String]] = Map( "public_obp_portal_url" -> Set("status", "health"), "public_obp_api_manager_url" -> Set("status", "health"), + "public_obp_cohort_url" -> Set("health"), "public_obp_oidc_url" -> Set("status"), "public_obp_api_url" -> Set("status", "health"), "public_obp_opey_url" -> Set("status", "health"), diff --git a/obp-api/src/main/scala/code/api/util/http4s/Http4sSupport.scala b/obp-api/src/main/scala/code/api/util/http4s/Http4sSupport.scala index 9daa0d3d86..85b3b3929e 100644 --- a/obp-api/src/main/scala/code/api/util/http4s/Http4sSupport.scala +++ b/obp-api/src/main/scala/code/api/util/http4s/Http4sSupport.scala @@ -105,6 +105,13 @@ object Http4sRequestAttributes { val cachedBodyKey: Key[Option[String]] = Key.newKey[IO, Option[String]].unsafeRunSync()(cats.effect.unsafe.IORuntime.global) + /** + * Set on a response whose body is a file's bytes (EndpointHelpers.executeAndRespondBytes), so that + * ResourceDocMiddleware keeps its Content-Type instead of making it application/json. + */ + val fileContentResponseKey: Key[Unit] = + Key.newKey[IO, Unit].unsafeRunSync()(cats.effect.unsafe.IORuntime.global) + /** * Vault key for how the caller's certificate was established — the full PeerTrust.Resolution, * from which the CallContext takes certificateTrust (the mode: "direct" / "forwarded" / "none") @@ -238,6 +245,48 @@ object Http4sRequestAttributes { } } + /** A file's bytes and what to send with them; see [[executeAndRespondBytes]]. */ + case class BytesResponse(bytes: Array[Byte], mediaType: MediaType, fileName: String, inline: Boolean, + etag: String, cacheControl: String, metricSummary: Map[String, Any]) + + /** + * Execute Future-based business logic that returns a file's bytes rather than JSON: 200 with the + * file's own Content-Type. Errors are JSON, converted via ErrorResponseConverter, as everywhere else. + * + * The bytes may come from anyone, so the response stops a browser treating them as anything other + * than the declared type: X-Content-Type-Options nosniff, and a sandbox Content-Security-Policy so a + * document opened from this origin cannot run scripts. The metric records metricSummary, never the bytes. + */ + def executeAndRespondBytes(req: Request[IO])(f: CallContext => Future[BytesResponse]): IO[Response[IO]] = { + implicit val cc: CallContext = req.callContext + RequestScopeConnection.fromFuture(f(cc)).attempt.flatMap { + case Right(file) => + val disposition = (if (file.inline) "inline" else "attachment") + + "; filename=\"" + file.fileName.replaceAll("[\"\\\\\r\n]", "_") + "\"" + val response = Response[IO](Status.Ok) + .withEntity(file.bytes) + .withContentType(`Content-Type`(file.mediaType)) + .putHeaders( + Header.Raw(CIString("Content-Disposition"), disposition), + Header.Raw(CIString("X-Content-Type-Options"), "nosniff"), + Header.Raw(CIString("Content-Security-Policy"), "sandbox; default-src 'none'"), + Header.Raw(CIString("ETag"), "\"" + file.etag + "\""), + Header.Raw(CIString("Cache-Control"), file.cacheControl)) + IO.pure(withCallContextHeaders(response).withAttribute(Http4sRequestAttributes.fileContentResponseKey, ())) + .flatTap(recordMetric(file.metricSummary, _)) + case Left(err) => ErrorResponseConverter.toHttp4sResponse(err, cc).flatTap(recordMetric(err.getMessage, _)) + } + } + + /** [[executeAndRespondBytes]] for a validated User. */ + def withUserBytes(req: Request[IO])(f: (User, CallContext) => Future[BytesResponse]): IO[Response[IO]] = + executeAndRespondBytes(req) { cc => + cc.user.toOption match { + case Some(user) => f(user, cc) + case None => Future.failed(new RuntimeException(AuthenticatedUserIsRequired)) + } + } + /** * Execute business logic requiring validated User. * Returns 200 OK on success, converts errors via ErrorResponseConverter. diff --git a/obp-api/src/main/scala/code/api/util/http4s/ResourceDocMiddleware.scala b/obp-api/src/main/scala/code/api/util/http4s/ResourceDocMiddleware.scala index b72c9e5e06..42dda1bf38 100644 --- a/obp-api/src/main/scala/code/api/util/http4s/ResourceDocMiddleware.scala +++ b/obp-api/src/main/scala/code/api/util/http4s/ResourceDocMiddleware.scala @@ -349,14 +349,14 @@ object ResourceDocMiddleware extends MdcLoggable { logger.debug(s"[ResourceDocMiddleware] needsAuthentication for ${resourceDoc.partialFunctionName}: $needsAuth") // Dispatch on authMode the same way Lift's wrappedWithAuthCheck (APIUtil.scala:1783-1788) does: - // ApplicationOnly | UserOrApplication → applicationAccess (returns ApplicationNotIdentified + // ApplicationOnly | UserXORApplication → applicationAccess (returns ApplicationNotIdentified // when neither user nor consumer credentials are valid; also accepts consumer-only). // UserOnly | UserAndApplication → anonymousAccess (returns AuthenticatedUserIsRequired // when needsAuth is true and user is missing). // Without this dispatch, every endpoint behaved as UserOnly — breaking // ApplicationNotIdentified semantics for v5.1.0 createConsumer / getConsumers. val isAppMode = resourceDoc.authMode match { - case APIUtil.ApplicationOnly | APIUtil.UserOrApplication => true + case APIUtil.ApplicationOnly | APIUtil.UserXORApplication => true case _ => false } val io = IO.fromFuture(IO( @@ -372,7 +372,7 @@ object ResourceDocMiddleware extends MdcLoggable { case Right((Full(user), None)) => IO.pure(Right(ctx.copy(user = Full(user)))) // Empty box — no valid credentials provided, and auth is required. - // For UserOrApplication / ApplicationOnly: applicationAccess already returned + // For UserXORApplication / ApplicationOnly: applicationAccess already returned // successfully because the consumer is valid (just no user). Pass through. case Right((_, optCC)) if needsAuth && !isAppMode => val cc2 = optCC.getOrElse(ctx.callContext) @@ -417,10 +417,10 @@ object ResourceDocMiddleware extends MdcLoggable { case Full(user) => val bankId = pathParams.getOrElse("BANK_ID", "") val consumerId = APIUtil.getConsumerPrimaryKey(Some(ctx.callContext)) - // Use handleAccessControlWithAuthMode so authMode = UserOrApplication - // accepts consumer-scope-only requests (the auth-only handleAccess... + // Use handleAccessControlWithAuthMode so authMode = UserXORApplication + // accepts an application acting as itself on its Scopes (the auth-only handleAccess... // function checks user-entitlements + scopes ANDed and rejects pure - // consumer-scope requests with 403 even under UserOrApplication). + // consumer-scope requests with 403 even under UserXORApplication). val ok = APIUtil.handleAccessControlWithAuthMode(bankId, user.userId, consumerId, roles, resourceDoc.authMode) if (ok) success(ctx) else EitherT[IO, Response[IO], ValidationContext]( @@ -749,9 +749,10 @@ object ResourceDocMiddleware extends MdcLoggable { ) } - /** Ensure the response has JSON content type */ + /** Ensure the response has JSON content type, unless its body is a file's bytes with the file's own type */ private def ensureJsonContentType(response: Response[IO]): Response[IO] = { - response.contentType match { + if (response.attributes.contains(Http4sRequestAttributes.fileContentResponseKey)) response + else response.contentType match { case Some(contentType) if contentType.mediaType == MediaType.application.json => response case _ => response.withContentType(jsonContentType) } diff --git a/obp-api/src/main/scala/code/api/util/migration/Migration.scala b/obp-api/src/main/scala/code/api/util/migration/Migration.scala index 69b7798320..5a7c2bffdd 100644 --- a/obp-api/src/main/scala/code/api/util/migration/Migration.scala +++ b/obp-api/src/main/scala/code/api/util/migration/Migration.scala @@ -194,6 +194,8 @@ object Migration extends MdcLoggable { renameDynamicEntityRoles() renameDynamicEntityDefinitionRoles() upperCaseStoredCurrencyCodes() + renameOrganisationsToTenants(startedBeforeSchemifier) + markSelfServiceBanks(startedBeforeSchemifier) ensureConsentViewExists() } @@ -958,6 +960,40 @@ object Migration extends MdcLoggable { * `eur`, would otherwise be missed by database queries that select by currency. The columns, and * what is deliberately left alone, are in [[MigrationOfCurrencyCodesUpperCase]]. */ + /** + * Copy stored Organisations and their members into the Tenant tables, and make their Roles the Tenant system + * Roles. It must run after Schemifier, which creates the Tenant tables. The details are in + * [[MigrationOfOrganisationToTenant]]. + */ + private def renameOrganisationsToTenants(startedBeforeSchemifier: Boolean): Boolean = { + if (startedBeforeSchemifier) { + logger.warn(s"Migration.database.renameOrganisationsToTenants(true) cannot be run before Schemifier.") + true + } else { + val name = nameOf(renameOrganisationsToTenants(startedBeforeSchemifier)) + runOnce(name) { + MigrationOfOrganisationToTenant.migrate(name) + } + } + } + + /** + * Mark the banks made with Create My Bank, and hold the unscanned files stored in them as pending. It must run + * after Schemifier, which creates SelfServiceBank and adds the ScanStatus column. The details are in + * [[MigrationOfSelfServiceBanks]]. + */ + private def markSelfServiceBanks(startedBeforeSchemifier: Boolean): Boolean = { + if (startedBeforeSchemifier) { + logger.warn(s"Migration.database.markSelfServiceBanks(true) cannot be run before Schemifier.") + true + } else { + val name = nameOf(markSelfServiceBanks(startedBeforeSchemifier)) + runOnce(name) { + MigrationOfSelfServiceBanks.markAndHoldFiles(name) + } + } + } + private def upperCaseStoredCurrencyCodes(): Boolean = { val name = nameOf(upperCaseStoredCurrencyCodes) runOnce(name) { diff --git a/obp-api/src/main/scala/code/api/util/migration/MigrationOfOrganisationToTenant.scala b/obp-api/src/main/scala/code/api/util/migration/MigrationOfOrganisationToTenant.scala new file mode 100644 index 0000000000..e999ea6b00 --- /dev/null +++ b/obp-api/src/main/scala/code/api/util/migration/MigrationOfOrganisationToTenant.scala @@ -0,0 +1,158 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ +package code.api.util.migration + +import code.api.Constant.DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID +import code.api.util.APIUtil +import code.api.util.migration.Migration.{DbFunction, saveLog} +import code.entitlement.MappedEntitlement +import code.entitlementrequest.MappedEntitlementRequest +import code.group.Group +import code.scope.MappedScope +import code.tenant.Tenant +import net.liftweb.db.DB +import net.liftweb.mapper.By +import net.liftweb.util.DefaultConnectionIdentifier + +/** + * Organisation is now Tenant: a customer of the operator of this instance, which belongs to no bank. + * + * 1. Rows of the old table Organisation are copied into Tenant, without their bank id. A row whose id is already in + * Tenant is not copied again. Tenants have no members, so OrganisationMember is not copied. The old tables are + * left as they are; nothing reads them any more. + * 2. The four Roles are renamed (CanCreateOrganisation to CanCreateTenant, CanGetOrganisation and its older name + * CanGetAnyOrganisation to CanGetTenant, and so on) and become system Roles: a grant at SYS or at the empty bank + * id moves to the empty bank id. A grant at another bank is left as it was, under its old name, which nothing + * checks: a system Role is never widened from one bank to all of them. Such grants are counted in the log. The + * four Organisation member Roles have no successor and are left as they are. + * + * Four stores hold Role names: Entitlements, Entitlement Requests, Consumer Scopes, and the comma-joined list on a + * Group. A Group at SYS holding only these Roles moves to the empty bank id with them; a mixed one stays, renamed, + * and is named in the log. + */ +object MigrationOfOrganisationToTenant { + + val roleRenames: Map[String, String] = Map( + "CanCreateOrganisation" -> "CanCreateTenant", + "CanGetOrganisation" -> "CanGetTenant", + "CanGetAnyOrganisation" -> "CanGetTenant", + "CanUpdateOrganisation" -> "CanUpdateTenant", + "CanDeleteOrganisation" -> "CanDeleteTenant" + ) + + private def systemLevel(bankId: String): Boolean = + bankId == null || bankId.isEmpty || bankId == DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID + + def migrate(name: String): Boolean = { + val startDate = System.currentTimeMillis() + val report = scala.collection.mutable.ListBuffer[String]() + + // SQL for the old tables: their Mapper classes are gone. + def copyRows(oldTable: String, label: String)(copy: java.sql.ResultSet => Boolean): Unit = + if (DbFunction.tableExistsByName(oldTable)) { + var copied = 0 + DB.use(DefaultConnectionIdentifier) { connection => + val statement = connection.prepareStatement(s"SELECT * FROM $oldTable") + try { + val rows = statement.executeQuery() + while (rows.next()) if (copy(rows)) copied += 1 + } finally statement.close() + } + report += s"$label: copied $copied row(s)" + } + + copyRows("organisation", "Tenants") { row => + val tenantId = row.getString("organisationid") + Tenant.find(By(Tenant.TenantId, tenantId)).isEmpty && { + Tenant.create.TenantId(tenantId).Name(row.getString("name")).Website(row.getString("website")) + .LogoUrl(row.getString("logourl")).Status(row.getString("status")).Visibility(row.getString("visibility")) + .CreatedByUserId(row.getString("createdbyuserid")) + .CreationDate(row.getTimestamp("creationdate")).LastUpdate(row.getTimestamp("lastupdate")) + .save + } + } + + /** Rename each of these Roles held at SYS or the empty bank id, and make it a system Role. */ + def moveRows(label: String, rows: List[(String, String, (String, String) => Unit)]): Unit = { + var changed = 0 + var failed = 0 + var leftAtBanks = 0 + rows.foreach { case (roleName, bankId, save) => + roleRenames.get(roleName).foreach { newName => + if (!systemLevel(bankId)) leftAtBanks += 1 + // A row the same holder already has would break a unique index; it is counted, not fatal. + else if (scala.util.Try(save(newName, "")).isSuccess) changed += 1 else failed += 1 + } + } + val notes = List( + if (failed == 0) "" else s"; $failed could not be saved, most likely already held", + if (leftAtBanks == 0) "" else s"; $leftAtBanks granted at a bank left unchanged, as nothing checks them" + ).mkString + report += s"$label: changed $changed row(s)$notes" + } + + if (DbFunction.tableExists(MappedEntitlement)) { + moveRows("Entitlements", MappedEntitlement.findAll().map(row => + (row.mRoleName.get, row.mBankId.get, (n: String, b: String) => { row.mRoleName(n).mBankId(b).save; () }))) + } + if (DbFunction.tableExists(MappedEntitlementRequest)) { + moveRows("Entitlement Requests", MappedEntitlementRequest.findAll().map(row => + (row.mRoleName.get, row.mBankId.get, (n: String, b: String) => { row.mRoleName(n).mBankId(b).save; () }))) + } + if (DbFunction.tableExists(MappedScope)) { + moveRows("Consumer Scopes", MappedScope.findAll().map(row => + (row.mRoleName.get, row.mBankId.get, (n: String, b: String) => { row.mRoleName(n).mBankId(b).save; () }))) + } + if (DbFunction.tableExists(Group)) { + var renamedGroups = 0 + var movedGroups = 0 + val mixedGroups = scala.collection.mutable.ListBuffer[String]() + Group.findAll().foreach { group => + val storedRoles = group.ListOfRoles.get.split(",").toList.map(_.trim).filter(_.nonEmpty) + if (storedRoles.exists(roleRenames.contains) && systemLevel(group.BankId.get)) { + group.ListOfRoles(storedRoles.map(r => roleRenames.getOrElse(r, r)).mkString(",")) + renamedGroups += 1 + if (group.BankId.get == DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID) { + if (storedRoles.forall(roleRenames.contains)) { + group.BankId("") + movedGroups += 1 + } else mixedGroups += group.GroupName.get + } + group.save + } + } + val mixedNote = + if (mixedGroups.isEmpty) "" + else s"; left at $DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID because they also hold Roles that belong there, so their " + + s"Tenant Roles need granting as system Roles another way: ${mixedGroups.mkString(", ")}" + report += s"Groups: renamed Roles on $renamedGroups group(s), moved $movedGroups to the system level$mixedNote" + } + + saveLog(name, APIUtil.gitCommit, isSuccessful = true, startDate, System.currentTimeMillis(), report.mkString("; ")) + true + } +} diff --git a/obp-api/src/main/scala/code/api/util/migration/MigrationOfSelfServiceBanks.scala b/obp-api/src/main/scala/code/api/util/migration/MigrationOfSelfServiceBanks.scala new file mode 100644 index 0000000000..63b0c61d2b --- /dev/null +++ b/obp-api/src/main/scala/code/api/util/migration/MigrationOfSelfServiceBanks.scala @@ -0,0 +1,82 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ +package code.api.util.migration + +import code.api.util.APIUtil +import code.api.util.migration.Migration.{DbFunction, saveLog} +import code.files.{File, ScanStatus} +import code.model.dataAccess.{MappedBank, SelfServiceBank} +import code.util.BankNameGenerator +import net.liftweb.db.DB +import net.liftweb.util.DefaultConnectionIdentifier + +/** + * OBP now records the banks made with Create My Bank in SelfServiceBank, because anyone may make one and grant + * themselves its Roles, so those Roles vouch for nothing. + * + * 1. A bank in MappedBank made before the record existed is recorded when its id has exactly the shape Create My Bank + * gives (BankNameGenerator.isGenerated: three words from its lists and a four-character hex suffix), with its + * CreatedByUserId. A bank named by hand practically never has that shape. Create My Bank stored its banks through + * the connector, so one made on a backend connector is not in MappedBank and is not found here. + * 2. In a self-service bank, a file written before scanning existed (scan status NULL or empty, which elsewhere counts + * as clean) is set to pending, so its content waits for a scan like a new upload there. + */ +object MigrationOfSelfServiceBanks { + + def markAndHoldFiles(name: String): Boolean = { + val startDate = System.currentTimeMillis() + val report = scala.collection.mutable.ListBuffer[String]() + + if (DbFunction.tableExists(MappedBank)) { + val generated = MappedBank.findAll().filter(bank => BankNameGenerator.isGenerated(bank.permalink.get)) + .filterNot(bank => SelfServiceBank.isSelfService(bank.permalink.get)) + generated.foreach(bank => SelfServiceBank.record(bank.permalink.get, bank.CreatedByUserId.get)) + report += s"Banks: recorded ${generated.size} as self-service" + } + + if (DbFunction.tableExists(File) && DbFunction.tableExists(SelfServiceBank)) { + val selfServiceBankIds = SelfServiceBank.findAll().map(_.BankId.get) + // SQL rather than Mapper: Mapper would read a NULL status back as its default. + val held = DB.use(DefaultConnectionIdentifier) { connection => + selfServiceBankIds.map { bankId => + val statement = connection.prepareStatement( + s"UPDATE ${File._dbTableNameLC} SET ${File.ScanStatus.dbColumnName} = ? " + + s"WHERE ${File.BankId.dbColumnName} = ? AND (${File.ScanStatus.dbColumnName} IS NULL OR ${File.ScanStatus.dbColumnName} = '')") + try { + statement.setString(1, ScanStatus.Pending) + statement.setString(2, bankId) + statement.executeUpdate() + } finally statement.close() + }.sum + } + report += s"Files: set $held file(s) in self-service banks to pending" + } + + saveLog(name, APIUtil.gitCommit, isSuccessful = true, startDate, System.currentTimeMillis(), report.mkString("; ")) + true + } +} diff --git a/obp-api/src/main/scala/code/api/v2_1_0/Http4s210.scala b/obp-api/src/main/scala/code/api/v2_1_0/Http4s210.scala index 4c9fa84db8..8747867d3b 100644 --- a/obp-api/src/main/scala/code/api/v2_1_0/Http4s210.scala +++ b/obp-api/src/main/scala/code/api/v2_1_0/Http4s210.scala @@ -182,9 +182,9 @@ object Http4s210 { code.util.Helper.booleanToFuture(AuthenticatedUserIsRequired, failCode = 401, cc = Some(cc)) { cc.user.isDefined } else Future.unit transactionRequestTypes <- Future { - APIUtil.getPropsValue("transactionRequests_supported_types", "") + APIUtil.advertisedTransactionRequestTypes } - } yield JSONFactory210.createTransactionRequestTypeJSON(transactionRequestTypes.split(",").toList) + } yield JSONFactory210.createTransactionRequestTypeJSON(transactionRequestTypes) } } @@ -193,6 +193,10 @@ object Http4s210 { "/banks/BANK_ID/transaction-request-types", "Get Transaction Request Types at Bank", s"""Get the list of the Transaction Request Types supported by the bank. + | + |These are the types in the `transactionRequests_supported_types` prop. OPEN_CORRIDOR_PROMISE is listed + |exactly when `open_corridor_enabled` is true, so a client can tell whether Open Corridor is on here + |before it sends a promise. | |${userAuthenticationMessage(!getTransactionRequestTypesIsPublic)}""", EmptyBody, transactionRequestTypesJSON, diff --git a/obp-api/src/main/scala/code/api/v3_1_0/Http4s310.scala b/obp-api/src/main/scala/code/api/v3_1_0/Http4s310.scala index 43b5f639ba..1047b39693 100644 --- a/obp-api/src/main/scala/code/api/v3_1_0/Http4s310.scala +++ b/obp-api/src/main/scala/code/api/v3_1_0/Http4s310.scala @@ -4515,6 +4515,7 @@ object Http4s310 { EndpointHelpers.withUserAndBodyCreated[PostConsentBodyCommonJson, Any](req) { (user, consentJson, cc) => val raw = cc.httpBody.getOrElse("") for { + _ <- code.api.util.Consent.checkConsentsAllowed(Some(cc)) _ <- code.util.Helper.booleanToFuture(ConsentAllowedScaMethods, cc = Some(cc)) { List(StrongCustomerAuthentication.SMS.toString, StrongCustomerAuthentication.EMAIL.toString, diff --git a/obp-api/src/main/scala/code/api/v4_0_0/Http4s400.scala b/obp-api/src/main/scala/code/api/v4_0_0/Http4s400.scala index f32ccf454d..ea25ba9bc8 100644 --- a/obp-api/src/main/scala/code/api/v4_0_0/Http4s400.scala +++ b/obp-api/src/main/scala/code/api/v4_0_0/Http4s400.scala @@ -1535,6 +1535,9 @@ object Http4s400 { "/banks/BANK_ID/user_customer_links", "Create User Customer Link", s"""Link a User to a Customer + | + |**Who may call it.** A User with the Role CanCreateUserCustomerLink at BANK_ID, or an application acting as + |itself whose Consumer holds it as a Scope. | |${userAuthenticationMessage(true)}""", createUserCustomerLinkJson, userCustomerLinkJson, @@ -1552,6 +1555,7 @@ object Http4s400 { ), List(apiTagCustomer, apiTagUser), Some(List(canCreateUserCustomerLinkAtAnyBank, canCreateUserCustomerLink)), + authMode = UserXORApplication, http4sPartialFunction = Some(createUserCustomerLinks)) // ─── getSystemDynamicEntities ───────────────────────────────────────────── diff --git a/obp-api/src/main/scala/code/api/v5_0_0/Http4s500.scala b/obp-api/src/main/scala/code/api/v5_0_0/Http4s500.scala index bf6cd74b0b..74612b799b 100644 --- a/obp-api/src/main/scala/code/api/v5_0_0/Http4s500.scala +++ b/obp-api/src/main/scala/code/api/v5_0_0/Http4s500.scala @@ -874,6 +874,7 @@ object Http4s500 { implicit val cc: code.api.util.CallContext = req.callContext val rawBody = cc.httpBody.getOrElse("") for { + _ <- code.api.util.Consent.checkConsentsAllowed(Some(cc)) (_, callContextOpt) <- APIUtil.applicationAccess(cc) _ <- APIUtil.passesPsd2Aisp(callContextOpt) failMsg = s"$InvalidJsonFormat The Json body should be the $PostConsentBodyCommonJson " @@ -1093,6 +1094,7 @@ object Http4s500 { implicit val cc: code.api.util.CallContext = req.callContext val callContextOpt = Some(cc) for { + _ <- code.api.util.Consent.checkConsentsAllowed(callContextOpt) user <- Future.successful(cc.user.openOrThrowException(AuthenticatedUserIsRequired)) createdConsentRequest <- Future(ConsentRequests.consentRequestProvider.vend.getConsentRequestById(consentRequestId)) .map(i => unboxFullOrFail(i, callContextOpt, ConsentRequestNotFound)) diff --git a/obp-api/src/main/scala/code/api/v5_1_0/Http4s510.scala b/obp-api/src/main/scala/code/api/v5_1_0/Http4s510.scala index 3bcff4c290..835fdfafd8 100644 --- a/obp-api/src/main/scala/code/api/v5_1_0/Http4s510.scala +++ b/obp-api/src/main/scala/code/api/v5_1_0/Http4s510.scala @@ -792,7 +792,7 @@ object Http4s510 { List(AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidJsonFormat, InvalidRedirectUrl, RedirectUrlHostNotAllowed, UnknownError), List(apiTagConsumer), Some(List(canCreateConsumer)), - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(createConsumer) ) @@ -856,7 +856,7 @@ object Http4s510 { ), List(apiTagConsumer), Some(List(canGetConsumers)), - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(getConsumers) ) @@ -1234,7 +1234,7 @@ object Http4s510 { List($AuthenticatedUserIsRequired, UnknownError), apiTagSystem :: apiTagApi :: apiTagLogCache :: Nil, Some(List(canGetSystemLogCacheTrace, canGetSystemLogCacheAll)), - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(logCacheTraceEndpoint) ) @@ -1261,7 +1261,7 @@ object Http4s510 { List($AuthenticatedUserIsRequired, UnknownError), apiTagSystem :: apiTagApi :: apiTagLogCache :: Nil, Some(List(canGetSystemLogCacheDebug, canGetSystemLogCacheAll)), - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(logCacheDebugEndpoint) ) @@ -1288,7 +1288,7 @@ object Http4s510 { List($AuthenticatedUserIsRequired, UnknownError), apiTagSystem :: apiTagApi :: apiTagLogCache :: Nil, Some(List(canGetSystemLogCacheInfo, canGetSystemLogCacheAll)), - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(logCacheInfoEndpoint) ) @@ -1315,7 +1315,7 @@ object Http4s510 { List($AuthenticatedUserIsRequired, UnknownError), apiTagSystem :: apiTagApi :: apiTagLogCache :: Nil, Some(List(canGetSystemLogCacheWarning, canGetSystemLogCacheAll)), - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(logCacheWarningEndpoint) ) @@ -1342,7 +1342,7 @@ object Http4s510 { List($AuthenticatedUserIsRequired, UnknownError), apiTagSystem :: apiTagApi :: apiTagLogCache :: Nil, Some(List(canGetSystemLogCacheError, canGetSystemLogCacheAll)), - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(logCacheErrorEndpoint) ) @@ -1369,7 +1369,7 @@ object Http4s510 { List($AuthenticatedUserIsRequired, UnknownError), apiTagSystem :: apiTagApi :: apiTagLogCache :: Nil, Some(List(canGetSystemLogCacheAll)), - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(logCacheAllEndpoint) ) @@ -5102,6 +5102,7 @@ object Http4s510 { implicit val cc: code.api.util.CallContext = req.callContext val callContextOpt = Some(cc) for { + _ <- code.api.util.Consent.checkConsentsAllowed(callContextOpt) user <- Future.successful(cc.user.openOrThrowException(AuthenticatedUserIsRequired)) _ <- Helper.booleanToFuture(ConsentAllowedScaMethods, cc = callContextOpt) { List(StrongCustomerAuthentication.SMS.toString(), @@ -5294,6 +5295,7 @@ object Http4s510 { val rawBody = cc.httpBody.getOrElse("") val parsedBody = com.openbankproject.commons.util.JsonAliases.parse(rawBody) for { + _ <- code.api.util.Consent.checkConsentsAllowed(Some(cc)) (_, callContextOpt) <- APIUtil.applicationAccess(cc) _ <- APIUtil.passesPsd2Aisp(callContextOpt) postConsentRequestJsonV510 <- NewStyle.function.tryons(s"$InvalidJsonFormat The Json body should be the $PostVRPConsentRequestJsonV510 ", 400, callContextOpt) { diff --git a/obp-api/src/main/scala/code/api/v6_0_0/Http4s600.scala b/obp-api/src/main/scala/code/api/v6_0_0/Http4s600.scala index 59f809a825..6a84330f37 100644 --- a/obp-api/src/main/scala/code/api/v6_0_0/Http4s600.scala +++ b/obp-api/src/main/scala/code/api/v6_0_0/Http4s600.scala @@ -580,7 +580,7 @@ object Http4s600 { com.openbankproject.commons.util.JsonAliases.parse(rawBody).extract[CreateDynamicEntityRequestJsonV600] } _ <- validateEntityNameV600(request.entity_name, cc) - _ <- DynamicEntitySpace.requireRoleAtSystemSpace(canCreateDynamicEntityDefinition, cc, code.api.util.APIUtil.UserOrApplication) + _ <- DynamicEntitySpace.requireRoleAtSystemSpace(canCreateDynamicEntityDefinition, cc, code.api.util.APIUtil.UserXORApplication) dynamicEntity <- NewStyle.function.tryons(InvalidJsonFormat, 400, Some(cc)) { DynamicEntityCommons(JSONFactory600.convertV600RequestToInternal(request), None, cc.userId, None) } @@ -1328,6 +1328,7 @@ object Http4s600 { implicit val cc: code.api.util.CallContext = req.callContext val callContextOpt = Some(cc) for { + _ <- code.api.util.Consent.checkConsentsAllowed(callContextOpt) user <- Future.successful(cc.user.openOrThrowException(AuthenticatedUserIsRequired)) _ <- Helper.booleanToFuture(ConsentAllowedScaMethods, cc = callContextOpt) { List(StrongCustomerAuthentication.SMS.toString(), @@ -2537,7 +2538,7 @@ object Http4s600 { } } - // ─── Phase 2: 3 anonymous/UserOrApplication endpoints ───────────────── + // ─── Phase 2: 3 anonymous/UserXORApplication endpoints ───────────────── // GET /obp/v6.0.0/webui-props/WEBUI_PROP_NAME lazy val getWebUiProp: Http4sRoute = Http4sRoute { @@ -4059,7 +4060,8 @@ object Http4s600 { _ <- Future(unboxFullOrFail(roomBox, Some(cc), ChatRoomNotFound, 404)) partBox <- Future(code.chat.ChatPermissions.isParticipant(chatRoomId, user.userId)) _ <- Future(unboxFullOrFail(partBox, Some(cc), NotChatRoomParticipant, 403)) - msgBox <- Future(code.chat.ChatMessageTrait.chatMessageProvider.vend.getMessage(chatMessageId)) + // the message must be in the room in the path, else a participant of one room reads threads of any room + msgBox <- Future(code.chat.ChatMessageTrait.chatMessageProvider.vend.getMessage(chatMessageId).filter(_.chatRoomId == chatRoomId)) _ <- Future(unboxFullOrFail(msgBox, Some(cc), ChatMessageNotFound, 404)) repliesBox <- Future(code.chat.ChatMessageTrait.chatMessageProvider.vend.getThreadReplies(chatMessageId)) replies <- Future(unboxFullOrFail(repliesBox, Some(cc), @@ -4082,7 +4084,8 @@ object Http4s600 { _ <- Future(unboxFullOrFail(roomBox, Some(cc), ChatRoomNotFound, 404)) partBox <- Future(code.chat.ChatPermissions.isParticipant(chatRoomId, user.userId)) _ <- Future(unboxFullOrFail(partBox, Some(cc), NotChatRoomParticipant, 403)) - msgBox <- Future(code.chat.ChatMessageTrait.chatMessageProvider.vend.getMessage(chatMessageId)) + // the message must be in the room in the path, else a participant of one room reads threads of any room + msgBox <- Future(code.chat.ChatMessageTrait.chatMessageProvider.vend.getMessage(chatMessageId).filter(_.chatRoomId == chatRoomId)) _ <- Future(unboxFullOrFail(msgBox, Some(cc), ChatMessageNotFound, 404)) repliesBox <- Future(code.chat.ChatMessageTrait.chatMessageProvider.vend.getThreadReplies(chatMessageId)) replies <- Future(unboxFullOrFail(repliesBox, Some(cc), @@ -4317,9 +4320,10 @@ object Http4s600 { _ <- Future(unboxFullOrFail(roomBox, Some(cc), ChatRoomNotFound, 404)) partBox <- Future(code.chat.ChatPermissions.isParticipant(chatRoomId, user.userId)) _ <- Future(unboxFullOrFail(partBox, Some(cc), NotChatRoomParticipant, 403)) - msgBox <- Future(code.chat.ChatMessageTrait.chatMessageProvider.vend.getMessage(chatMessageId)) + // the message must be in the room in the path, else a participant of one room reads reactions of any room + msgBox <- Future(code.chat.ChatMessageTrait.chatMessageProvider.vend.getMessage(chatMessageId).filter(_.chatRoomId == chatRoomId)) _ <- Future(unboxFullOrFail(msgBox, Some(cc), ChatMessageNotFound, 404)) - reactionsBox <- Future(code.chat.ReactionTrait.reactionProvider.vend.getReactions(chatMessageId)) + reactionsBox <-Future(code.chat.ReactionTrait.reactionProvider.vend.getReactions(chatMessageId)) reactions <- Future(unboxFullOrFail(reactionsBox, Some(cc), s"$UnknownError Cannot get reactions", 400)) } yield JSONFactory600.createReactionsJson(reactions) @@ -4334,9 +4338,10 @@ object Http4s600 { _ <- Future(unboxFullOrFail(roomBox, Some(cc), ChatRoomNotFound, 404)) partBox <- Future(code.chat.ChatPermissions.isParticipant(chatRoomId, user.userId)) _ <- Future(unboxFullOrFail(partBox, Some(cc), NotChatRoomParticipant, 403)) - msgBox <- Future(code.chat.ChatMessageTrait.chatMessageProvider.vend.getMessage(chatMessageId)) + // the message must be in the room in the path, else a participant of one room reads reactions of any room + msgBox <- Future(code.chat.ChatMessageTrait.chatMessageProvider.vend.getMessage(chatMessageId).filter(_.chatRoomId == chatRoomId)) _ <- Future(unboxFullOrFail(msgBox, Some(cc), ChatMessageNotFound, 404)) - reactionsBox <- Future(code.chat.ReactionTrait.reactionProvider.vend.getReactions(chatMessageId)) + reactionsBox <-Future(code.chat.ReactionTrait.reactionProvider.vend.getReactions(chatMessageId)) reactions <- Future(unboxFullOrFail(reactionsBox, Some(cc), s"$UnknownError Cannot get reactions", 400)) } yield JSONFactory600.createReactionsJson(reactions) @@ -7354,7 +7359,7 @@ object Http4s600 { |* Each property can optionally be marked queryable with `"indexed": true` — only indexed fields may be used in the list endpoint's filter/sort query parameters (and a `reference:` field must be indexed to form a join edge). Add `"index": "spatial"` for a GeoJSON geometry index (only valid on a `json` field); the default when omitted is `"index": "scalar"` (B-tree). |* Each property can optionally declare **field-level access control**: `write_role_required`/`read_role_required` (booleans — auto-generate a per-field role) or `write_role`/`read_role` (name an explicit, shareable role). Write-restricted fields are not set via POST/PUT (their existing value is preserved) and are written only via the role-gated PATCH path; read-restricted fields are omitted from GET for callers lacking the read role. `hide_field_from_public_access` (boolean) hides a field of a public entity from callers whose access comes only from its public access, while callers holding the entity's read role still see it. |* Set `has_public_access` to `true` to generate read-only public endpoints (GET only, no authentication required) under `/public/`. - |* Set `auth_mode` to say who may hold the roles that guard the entity's data endpoints: `UserOnly` (default, the User's Entitlements), `ApplicationOnly` (the Consumer's Scopes), `UserOrApplication` (either) or `UserAndApplication` (both). Personal (`/my/`) endpoints always require a User. An entity with `has_personal_entity` cannot be `ApplicationOnly`. + |* Set `auth_mode` to say who may hold the roles that guard the entity's data endpoints: `UserOnly` (default, the User's Entitlements), `ApplicationOnly` (the Consumer's Scopes), `UserXORApplication` (the Consumer's Scopes when the application acts as itself, the User's Entitlements when a person acts; `UserOrApplication` is accepted as its former name) or `UserAndApplication` (both). Personal (`/my/`) endpoints always require a User. An entity with `has_personal_entity` cannot be `ApplicationOnly`. |* Set `has_community_access` to `true` to generate read-only community endpoints (GET only, authentication required + CanGet role) under `/community/`. Community endpoints return ALL records (personal + non-personal from all users). |* Set `personal_requires_role` to `true` to require the corresponding role (e.g. CanCreateDynamicEntity_, CanGetDynamicEntity_) for `/my/` personal entity endpoints. Default is `false` (any authenticated user can use `/my/` endpoints). |* Set `use_row_level_access` to `true` to decide read, update, delete and grant **per record** with an access list, in place of the entity's Get, Update and Delete roles. The User who creates a record holds all four permissions on it and shares it with `GET/PUT /obp/dynamic-entity/ENTITY_NAME/RECORD_ID/access` (a body of `user_id`, `can_read`, `can_update`, `can_delete`, `can_grant`) and `DELETE /obp/dynamic-entity/ENTITY_NAME/RECORD_ID/access/USER_ID`; revoking cascades to the grants that user passed on. Records the caller may not read are omitted from list responses and return 404 individually. Creating a record still needs the entity's Create role, `CanGrantDynamicEntityRowAccess_` administers the access list of any record, and field-level read and write roles still apply on top. It cannot be combined with `has_public_access` or `has_community_access`, and is only supported for locally-backed entities. @@ -7383,7 +7388,7 @@ object Http4s600 { List($AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidJsonFormat, DynamicPathAmbiguous, UnknownError), apiTagManageDynamicEntity :: apiTagApi :: Nil, Some(canCreateDynamicEntityDefinition :: Nil), - authMode = code.api.util.APIUtil.UserOrApplication, + authMode = code.api.util.APIUtil.UserXORApplication, http4sPartialFunction = Some(createSystemDynamicEntity) ).disableAutoValidateRoles() // checked in the handler at SYS: this URL names no bank resourceDocs += ResourceDoc( @@ -7427,7 +7432,7 @@ object Http4s600 { |* Each property can optionally be marked queryable with `"indexed": true` — only indexed fields may be used in the list endpoint's filter/sort query parameters (and a `reference:` field must be indexed to form a join edge). Add `"index": "spatial"` for a GeoJSON geometry index (only valid on a `json` field); the default when omitted is `"index": "scalar"` (B-tree). |* Each property can optionally declare **field-level access control**: `write_role_required`/`read_role_required` (booleans — auto-generate a per-field role) or `write_role`/`read_role` (name an explicit, shareable role). Write-restricted fields are not set via POST/PUT (their existing value is preserved) and are written only via the role-gated PATCH path; read-restricted fields are omitted from GET for callers lacking the read role. `hide_field_from_public_access` (boolean) hides a field of a public entity from callers whose access comes only from its public access, while callers holding the entity's read role still see it. |* Set `has_public_access` to `true` to generate read-only public endpoints (GET only, no authentication required) under `/public/`. - |* Set `auth_mode` to say who may hold the roles that guard the entity's data endpoints: `UserOnly` (default, the User's Entitlements), `ApplicationOnly` (the Consumer's Scopes), `UserOrApplication` (either) or `UserAndApplication` (both). Personal (`/my/`) endpoints always require a User. An entity with `has_personal_entity` cannot be `ApplicationOnly`. + |* Set `auth_mode` to say who may hold the roles that guard the entity's data endpoints: `UserOnly` (default, the User's Entitlements), `ApplicationOnly` (the Consumer's Scopes), `UserXORApplication` (the Consumer's Scopes when the application acts as itself, the User's Entitlements when a person acts; `UserOrApplication` is accepted as its former name) or `UserAndApplication` (both). Personal (`/my/`) endpoints always require a User. An entity with `has_personal_entity` cannot be `ApplicationOnly`. |* Set `has_community_access` to `true` to generate read-only community endpoints (GET only, authentication required + CanGet role) under `/community/`. Community endpoints return ALL records (personal + non-personal from all users). |* Set `personal_requires_role` to `true` to require the corresponding role (e.g. CanCreateDynamicEntity_, CanGetDynamicEntity_) for `/my/` personal entity endpoints. Default is `false` (any authenticated user can use `/my/` endpoints). |* Set `use_row_level_access` to `true` to decide read, update, delete and grant **per record** with an access list, in place of the entity's Get, Update and Delete roles. The User who creates a record holds all four permissions on it and shares it with `GET/PUT /obp/dynamic-entity/ENTITY_NAME/RECORD_ID/access` (a body of `user_id`, `can_read`, `can_update`, `can_delete`, `can_grant`) and `DELETE /obp/dynamic-entity/ENTITY_NAME/RECORD_ID/access/USER_ID`; revoking cascades to the grants that user passed on. Records the caller may not read are omitted from list responses and return 404 individually. Creating a record still needs the entity's Create role, `CanGrantDynamicEntityRowAccess_` administers the access list of any record, and field-level read and write roles still apply on top. It cannot be combined with `has_public_access` or `has_community_access`, and is only supported for locally-backed entities. @@ -7463,7 +7468,7 @@ object Http4s600 { ), apiTagManageDynamicEntity :: apiTagApi :: Nil, Some(canCreateDynamicEntityDefinition :: Nil), - authMode = code.api.util.APIUtil.UserOrApplication, + authMode = code.api.util.APIUtil.UserXORApplication, http4sPartialFunction = Some(createBankLevelDynamicEntity) ) resourceDocs += ResourceDoc( @@ -7503,7 +7508,7 @@ object Http4s600 { |* Each property can optionally be marked queryable with `"indexed": true` — only indexed fields may be used in the list endpoint's filter/sort query parameters (and a `reference:` field must be indexed to form a join edge). Add `"index": "spatial"` for a GeoJSON geometry index (only valid on a `json` field); the default when omitted is `"index": "scalar"` (B-tree). |* Each property can optionally declare **field-level access control**: `write_role_required`/`read_role_required` (booleans — auto-generate a per-field role) or `write_role`/`read_role` (name an explicit, shareable role). Write-restricted fields are not set via POST/PUT (their existing value is preserved) and are written only via the role-gated PATCH path; read-restricted fields are omitted from GET for callers lacking the read role. `hide_field_from_public_access` (boolean) hides a field of a public entity from callers whose access comes only from its public access, while callers holding the entity's read role still see it. |* Set `has_public_access` to `true` to generate read-only public endpoints (GET only, no authentication required) under `/public/`. - |* Set `auth_mode` to say who may hold the roles that guard the entity's data endpoints: `UserOnly` (default, the User's Entitlements), `ApplicationOnly` (the Consumer's Scopes), `UserOrApplication` (either) or `UserAndApplication` (both). Personal (`/my/`) endpoints always require a User. An entity with `has_personal_entity` cannot be `ApplicationOnly`. + |* Set `auth_mode` to say who may hold the roles that guard the entity's data endpoints: `UserOnly` (default, the User's Entitlements), `ApplicationOnly` (the Consumer's Scopes), `UserXORApplication` (the Consumer's Scopes when the application acts as itself, the User's Entitlements when a person acts; `UserOrApplication` is accepted as its former name) or `UserAndApplication` (both). Personal (`/my/`) endpoints always require a User. An entity with `has_personal_entity` cannot be `ApplicationOnly`. |* Set `has_community_access` to `true` to generate read-only community endpoints (GET only, authentication required + CanGet role) under `/community/`. Community endpoints return ALL records (personal + non-personal from all users). |* Set `personal_requires_role` to `true` to require the corresponding role (e.g. CanCreateDynamicEntity_, CanGetDynamicEntity_) for `/my/` personal entity endpoints. Default is `false` (any authenticated user can use `/my/` endpoints). |* Set `use_row_level_access` to `true` to decide read, update, delete and grant **per record** with an access list, in place of the entity's Get, Update and Delete roles. The User who creates a record holds all four permissions on it and shares it with `GET/PUT /obp/dynamic-entity/ENTITY_NAME/RECORD_ID/access` (a body of `user_id`, `can_read`, `can_update`, `can_delete`, `can_grant`) and `DELETE /obp/dynamic-entity/ENTITY_NAME/RECORD_ID/access/USER_ID`; revoking cascades to the grants that user passed on. Records the caller may not read are omitted from list responses and return 404 individually. Creating a record still needs the entity's Create role, `CanGrantDynamicEntityRowAccess_` administers the access list of any record, and field-level read and write roles still apply on top. It cannot be combined with `has_public_access` or `has_community_access`, and is only supported for locally-backed entities. @@ -7567,7 +7572,7 @@ object Http4s600 { |* Each property can optionally be marked queryable with `"indexed": true` — only indexed fields may be used in the list endpoint's filter/sort query parameters (and a `reference:` field must be indexed to form a join edge). Add `"index": "spatial"` for a GeoJSON geometry index (only valid on a `json` field); the default when omitted is `"index": "scalar"` (B-tree). |* Each property can optionally declare **field-level access control**: `write_role_required`/`read_role_required` (booleans — auto-generate a per-field role) or `write_role`/`read_role` (name an explicit, shareable role). Write-restricted fields are not set via POST/PUT (their existing value is preserved) and are written only via the role-gated PATCH path; read-restricted fields are omitted from GET for callers lacking the read role. `hide_field_from_public_access` (boolean) hides a field of a public entity from callers whose access comes only from its public access, while callers holding the entity's read role still see it. |* Set `has_public_access` to `true` to generate read-only public endpoints (GET only, no authentication required) under `/public/`. - |* Set `auth_mode` to say who may hold the roles that guard the entity's data endpoints: `UserOnly` (default, the User's Entitlements), `ApplicationOnly` (the Consumer's Scopes), `UserOrApplication` (either) or `UserAndApplication` (both). Personal (`/my/`) endpoints always require a User. An entity with `has_personal_entity` cannot be `ApplicationOnly`. + |* Set `auth_mode` to say who may hold the roles that guard the entity's data endpoints: `UserOnly` (default, the User's Entitlements), `ApplicationOnly` (the Consumer's Scopes), `UserXORApplication` (the Consumer's Scopes when the application acts as itself, the User's Entitlements when a person acts; `UserOrApplication` is accepted as its former name) or `UserAndApplication` (both). Personal (`/my/`) endpoints always require a User. An entity with `has_personal_entity` cannot be `ApplicationOnly`. |* Set `has_community_access` to `true` to generate read-only community endpoints (GET only, authentication required + CanGet role) under `/community/`. Community endpoints return ALL records (personal + non-personal from all users). |* Set `personal_requires_role` to `true` to require the corresponding role (e.g. CanCreateDynamicEntity_, CanGetDynamicEntity_) for `/my/` personal entity endpoints. Default is `false` (any authenticated user can use `/my/` endpoints). |* Set `use_row_level_access` to `true` to decide read, update, delete and grant **per record** with an access list, in place of the entity's Get, Update and Delete roles. The User who creates a record holds all four permissions on it and shares it with `GET/PUT /obp/dynamic-entity/ENTITY_NAME/RECORD_ID/access` (a body of `user_id`, `can_read`, `can_update`, `can_delete`, `can_grant`) and `DELETE /obp/dynamic-entity/ENTITY_NAME/RECORD_ID/access/USER_ID`; revoking cascades to the grants that user passed on. Records the caller may not read are omitted from list responses and return 404 individually. Creating a record still needs the entity's Create role, `CanGrantDynamicEntityRowAccess_` administers the access list of any record, and field-level read and write roles still apply on top. It cannot be combined with `has_public_access` or `has_community_access`, and is only supported for locally-backed entities. @@ -7638,7 +7643,7 @@ object Http4s600 { |* Each property can optionally be marked queryable with `"indexed": true` — only indexed fields may be used in the list endpoint's filter/sort query parameters (and a `reference:` field must be indexed to form a join edge). Add `"index": "spatial"` for a GeoJSON geometry index (only valid on a `json` field); the default when omitted is `"index": "scalar"` (B-tree). |* Each property can optionally declare **field-level access control**: `write_role_required`/`read_role_required` (booleans — auto-generate a per-field role) or `write_role`/`read_role` (name an explicit, shareable role). Write-restricted fields are not set via POST/PUT (their existing value is preserved) and are written only via the role-gated PATCH path; read-restricted fields are omitted from GET for callers lacking the read role. `hide_field_from_public_access` (boolean) hides a field of a public entity from callers whose access comes only from its public access, while callers holding the entity's read role still see it. |* Set `has_public_access` to `true` to generate read-only public endpoints (GET only, no authentication required) under `/public/`. - |* Set `auth_mode` to say who may hold the roles that guard the entity's data endpoints: `UserOnly` (default, the User's Entitlements), `ApplicationOnly` (the Consumer's Scopes), `UserOrApplication` (either) or `UserAndApplication` (both). Personal (`/my/`) endpoints always require a User. An entity with `has_personal_entity` cannot be `ApplicationOnly`. + |* Set `auth_mode` to say who may hold the roles that guard the entity's data endpoints: `UserOnly` (default, the User's Entitlements), `ApplicationOnly` (the Consumer's Scopes), `UserXORApplication` (the Consumer's Scopes when the application acts as itself, the User's Entitlements when a person acts; `UserOrApplication` is accepted as its former name) or `UserAndApplication` (both). Personal (`/my/`) endpoints always require a User. An entity with `has_personal_entity` cannot be `ApplicationOnly`. |* Set `has_community_access` to `true` to generate read-only community endpoints (GET only, authentication required + CanGet role) under `/community/`. Community endpoints return ALL records (personal + non-personal from all users). |* Set `personal_requires_role` to `true` to require the corresponding role (e.g. CanCreateDynamicEntity_, CanGetDynamicEntity_) for `/my/` personal entity endpoints. Default is `false` (any authenticated user can use `/my/` endpoints). |* Set `use_row_level_access` to `true` to decide read, update, delete and grant **per record** with an access list, in place of the entity's Get, Update and Delete roles. The User who creates a record holds all four permissions on it and shares it with `GET/PUT /obp/dynamic-entity/ENTITY_NAME/RECORD_ID/access` (a body of `user_id`, `can_read`, `can_update`, `can_delete`, `can_grant`) and `DELETE /obp/dynamic-entity/ENTITY_NAME/RECORD_ID/access/USER_ID`; revoking cascades to the grants that user passed on. Records the caller may not read are omitted from list responses and return 404 individually. Creating a record still needs the entity's Create role, `CanGrantDynamicEntityRowAccess_` administers the access list of any record, and field-level read and write roles still apply on top. It cannot be combined with `has_public_access` or `has_community_access`, and is only supported for locally-backed entities. @@ -7961,7 +7966,7 @@ object Http4s600 { ), List(apiTagMetric, apiTagAggregateMetrics), Some(canReadAggregateMetrics :: Nil), - authMode = code.api.util.APIUtil.UserOrApplication, + authMode = code.api.util.APIUtil.UserXORApplication, http4sPartialFunction = Some(getAggregateMetrics) ) resourceDocs += ResourceDoc( @@ -8335,6 +8340,9 @@ object Http4s600 { | |Note: If you need to set a specific customer number, use the Update Customer Number endpoint after this call. | + |**Who may call it.** A User with the Role CanCreateCustomer at BANK_ID, or an application acting as itself whose + |Consumer holds it as a Scope (for example a website that onboards its applicants as Customers). + | |${userAuthenticationMessage(true)} |""", postCustomerJsonV600, @@ -8355,6 +8363,7 @@ object Http4s600 { ), List(apiTagCustomer, apiTagPerson), Some(canCreateCustomer :: Nil), + authMode = code.api.util.APIUtil.UserXORApplication, http4sPartialFunction = Some(createCustomer) ) resourceDocs += ResourceDoc( @@ -9585,7 +9594,7 @@ object Http4s600 { List($AuthenticatedUserIsRequired, UserHasMissingRoles, UnknownError), apiTagOIDC :: apiTagConsumer :: apiTagOAuth :: Nil, Some(canGetOidcClient :: Nil), - authMode = code.api.util.APIUtil.UserOrApplication, + authMode = code.api.util.APIUtil.UserXORApplication, http4sPartialFunction = Some(getOidcClient) ) resourceDocs += ResourceDoc( @@ -9618,7 +9627,7 @@ object Http4s600 { List($AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidJsonFormat, UnknownError), apiTagOIDC :: apiTagConsumer :: apiTagOAuth :: Nil, Some(canVerifyOidcClient :: Nil), - authMode = code.api.util.APIUtil.UserOrApplication, + authMode = code.api.util.APIUtil.UserXORApplication, http4sPartialFunction = Some(verifyOidcClient) ) resourceDocs += ResourceDoc( @@ -10279,7 +10288,7 @@ object Http4s600 { List(UserHasMissingRoles, InvalidJsonFormat, InvalidLoginCredentials, UsernameHasBeenLocked, UnknownError), apiTagUser :: Nil, Some(canVerifyUserCredentials :: Nil), - authMode = code.api.util.APIUtil.UserOrApplication, + authMode = code.api.util.APIUtil.UserXORApplication, http4sPartialFunction = Some(verifyUserCredentials) ) resourceDocs += ResourceDoc( @@ -13873,7 +13882,7 @@ object Http4s600 { "Get App Directory", s"""Get connectivity information for apps in the OBP ecosystem. | - |Returns config (configuration) properties that apps (Portal, API Explorer, API Manager, + |Returns config (configuration) properties that apps (Portal, API Explorer, API Manager, Cohort, |Sandbox Populator, OIDC, Keycloak, Hola, MCP, Opey) and agents can use to discover |endpoints in the OBP ecosystem. | diff --git a/obp-api/src/main/scala/code/api/v6_0_0/JSONFactory6.0.0.scala b/obp-api/src/main/scala/code/api/v6_0_0/JSONFactory6.0.0.scala index 99f0cc9d8a..98539177ca 100644 --- a/obp-api/src/main/scala/code/api/v6_0_0/JSONFactory6.0.0.scala +++ b/obp-api/src/main/scala/code/api/v6_0_0/JSONFactory6.0.0.scala @@ -1044,7 +1044,7 @@ case class CreateDynamicEntityRequestJsonV600( has_community_access: Option[Boolean] = None, // defaults to false if not provided personal_requires_role: Option[Boolean] = None, // defaults to false if not provided use_row_level_access: Option[Boolean] = None, // defaults to false if not provided - auth_mode: Option[String] = None, // UserOnly | ApplicationOnly | UserOrApplication | UserAndApplication; defaults to UserOnly + auth_mode: Option[String] = None, // UserOnly | ApplicationOnly | UserXORApplication | UserAndApplication; defaults to UserOnly schema: org.json4s.JsonAST.JObject ) diff --git a/obp-api/src/main/scala/code/api/v7_0_0/Http4s700.scala b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700.scala index 941a1ae918..cff7c92f5c 100644 --- a/obp-api/src/main/scala/code/api/v7_0_0/Http4s700.scala +++ b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700.scala @@ -35,10 +35,11 @@ import code.api.Constant._ import code.api.ResourceDocs1_4_0.SwaggerDefinitionsJSON._ import code.api.util.APIUtil.{Http4sRoute, EmptyBody, _} import code.api.util.{APIUtil, ApiRole, CallContext, CustomJsonFormats, Glossary, NewStyle} -import code.api.util.ApiRole.{canAttachOpenCorridorPromise, canConfigureAmqpBankBroker, canGetMessageOutbox, canRetryMessageOutbox, canSettleOpenCorridor, canCreateAccount, canCreateEntitlementAtAnyBank, canCreateEntitlementAtOneBank, canCreateMetricsArchiveRun, canCreateGlossaryItem, canCreateOrganisation, canCreateRoutingScheme, canCreateTestEmail, canCreateUtilityVendResult, canDeleteAccountNotificationWebhookAtOneBank, canDeleteEntitlementAtAnyBank, canDeleteGlossaryItem, canDeleteOrganisation, canDeleteRoutingScheme, canDeleteSchedulerJobLock, canDeleteSystemAccountNotificationWebhook, canGetAccountAccessTrace, canGetAnyOrganisation, canGetAnyUser, canGetCacheConfig, canGetCacheInfo, canGetCacheNamespaces, canGetConfig, canGetConnectorHealth, canGetCustomersAtOneBank, canGetDatabasePoolInfo, canGetMetricsDiagnostics, canGetMigrations, canGetSchedulerJobLocks, canReadMetrics, canUpdateBankSupportedRoutingScheme, canUpdateGlossaryItem, canUpdateOrganisation, canUpdateRoutingScheme, canUpdateSystemView} +import code.api.util.ApiRole.{canAttachOpenCorridorPromise, canConfigureAmqpBankBroker, canGetMessageOutbox, canRetryMessageOutbox, canSettleOpenCorridor, canCreateAccount, canCreateEntitlementAtAnyBank, canCreateEntitlementAtOneBank, canCreateMetricsArchiveRun, canCreateGlossaryItem, canCreateRoutingScheme, canCreateTestEmail, canCreateUtilityVendResult, canDeleteAccountNotificationWebhookAtOneBank, canDeleteEntitlementAtAnyBank, canDeleteGlossaryItem, canDeleteRoutingScheme, canDeleteSchedulerJobLock, canDeleteSystemAccountNotificationWebhook, canGetAccountAccessTrace, canGetAccountDirectoryAtOneBank, canGetAnyUser, canGetCacheConfig, canGetCacheInfo, canGetCacheNamespaces, canGetConfig, canGetConnectorHealth, canGetCustomersAtOneBank, canGetDatabasePoolInfo, canGetMetricsDiagnostics, canGetMigrations, canGetSchedulerJobLocks, canReadMetrics, canUpdateBankSupportedRoutingScheme, canUpdateGlossaryItem, canUpdateRoutingScheme, canUpdateSystemView} import code.api.util.CommonsEmailWrapper -import code.model.dataAccess.{AuthUser, BankAccountCreation, MappedBank, ResourceUser} +import code.model.dataAccess.{AuthUser, BankAccountCreation, MappedBank, ResourceUser, SelfServiceBank} import code.consent.Consents +import code.accountcreation.AccountCreations import code.api.util.ApiTag._ import code.api.util.ErrorMessages._ import code.api.util.http4s.{ErrorResponseConverter, Http4sRequestAttributes, IdempotencyMiddleware, RequestScopeConnection, ResourceDocMiddleware, ResourceDocMatcher} @@ -57,7 +58,6 @@ import code.bankconnectors.storedprocedure.StoredProcedureUtils import code.migration.MigrationScriptLogProvider import code.bankconnectors.{Connector => BankConnector} import code.entitlement.Entitlement -import code.organisation.Organisations import code.glossaryitem.DynamicGlossaryItems import code.routingscheme.{RoutingSchemes, RoutingSchemeValidation} import code.payeelookup.PayeeLookups @@ -372,8 +372,9 @@ object Http4s700 { // Quota binds to the human: banks created by the human directly or by any // of their consent-agents count toward the same limit — otherwise every // new consent would arrive with a fresh quota. + // Counted in OBP's record of self-service banks; banks made with CanCreateBank do not count. val creatorUserIds = humanAndAgentUserIds(cc.onBehalfOfUserId) - MappedBank.count(ByList(MappedBank.CreatedByUserId, creatorUserIds)) + SelfServiceBank.count(ByList(SelfServiceBank.CreatedByUserId, creatorUserIds)) } _ <- Helper.booleanToFuture(SelfServiceBankLimitReached, failCode = 403, cc = Some(cc)) { banksCreatedByUser < selfServiceBankLimit @@ -385,6 +386,12 @@ object Http4s700 { Some(cc), UnknownError, 500 ) } + // A self-service bank gives a developer a space of their own without CanCreateBank. A User must + // never create a bank on a backend system, so it is refused unless banks are stored by OBP itself + // (the mapped connector, or a star connector routing this bank's creation to mapped). + _ <- Helper.booleanToFuture(SelfServiceBankNeedsMappedConnector, cc = Some(cc)) { + APIUtil.isDataFromOBPSide("createOrUpdateBank", Array(("bankId", generatedName.bankId))) + } (bank, _) <- NewStyle.function.createOrUpdateBank( generatedName.bankId, generatedName.fullName, @@ -392,6 +399,9 @@ object Http4s700 { "", "", "", "", "", "", Some(cc) ) + // Recorded so that what Roles normally vouch for (e.g. files stored with CanCreateFileAtOneBank + // starting clean) does not apply in a bank anyone could make and grant themselves Roles in. + _ <- Future(SelfServiceBank.record(bank.bankId.value, cc.onBehalfOfUserId)) // Creator grant targets the HUMAN (see v6.0.0 createBank): under a Consent the // authenticated user is a per-consent shadow, and roles granted to it are stranded. _ <- Future(Entitlement.entitlement.vend.addEntitlement( @@ -410,10 +420,15 @@ object Http4s700 { s"""Create a bank for the current User without requiring the role CanCreateBank. | |This is the self-service flavour of Create Bank, intended for - |sandbox instances. + |sandbox instances: it gives a developer a space of their own without the role CanCreateBank. | |$selfServiceBankCreationStatusText | + |It is refused unless OBP stores banks itself (the mapped connector, or a star connector routing + |createOrUpdateBank to mapped): a User may never create a bank on a backend system. Anyone can grant + |themselves Roles in such a bank, so they vouch for nothing there; for example, files stored in it + |start pending until they are scanned. + | |When the limit is reached, further banks require the role CanCreateBank |(see POST /banks). The limit binds to the human User: banks created by the User |directly and by any agent acting for the User under a Consent count toward the @@ -440,6 +455,7 @@ object Http4s700 { $AuthenticatedUserIsRequired, SelfServiceBankCreationDisabled, SelfServiceBankLimitReached, + SelfServiceBankNeedsMappedConnector, InvalidJsonFormat, UnknownError ), @@ -816,7 +832,7 @@ object Http4s700 { List(ApplicationNotIdentified, UnknownError), apiTagConsumer :: apiTagApi :: Nil, None, - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(getCurrentConsumerIdentity) ) @@ -854,7 +870,7 @@ object Http4s700 { List($AuthenticatedUserIsRequired, UserHasMissingRoles, UnknownError), apiTagConsumer :: Nil, Some(List(ApiRole.canGetConsumers)), - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(getConsumers) ) @@ -1085,7 +1101,7 @@ object Http4s700 { List(ApplicationNotIdentified, UnknownError), apiTagConsumer :: apiTagScope :: apiTagApi :: Nil, None, - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(getCurrentConsumerScopes) ) @@ -3248,259 +3264,6 @@ object Http4s700 { http4sPartialFunction = Some(createValidationEmail) ) - // ── Organisations ───────────────────────────────────────────────────────── - // CRUD for the Organisation resource. Migrated from v6.0.0 (Lift) to v7.0.0 - // (http4s). Path uses ORGANISATION_ID; not resolved by middleware (only BANK_ID - // / ACCOUNT_ID / VIEW_ID / COUNTERPARTY_ID are), so endpoints fetch directly - // via Organisations.organisation.vend. - - private val ValidOrganisationStatuses = Set("active", "suspended", "archived") - private val ValidOrganisationVisibilities = Set("public", "unlisted", "private") - private val OrganisationIdRegex = "^[a-zA-Z0-9._-]{2,64}$".r - - val createOrganisation: Http4sRoute = Http4sRoute { - case req @ POST -> `prefixPath` / "organisations" => - EndpointHelpers.withUserAndBodyCreated[JSONFactory700.PostOrganisationJsonV700, JSONFactory700.OrganisationJsonV700](req) { (user, body, cc) => - for { - _ <- Helper.booleanToFuture(InvalidOrganisationIdFormat, 400, Some(cc)) { - OrganisationIdRegex.findFirstIn(body.organisation_id).isDefined - } - status = body.status.getOrElse("active") - visibility = body.visibility.getOrElse("public") - _ <- Helper.booleanToFuture(InvalidOrganisationStatus, 400, Some(cc)) { - ValidOrganisationStatuses.contains(status) - } - _ <- Helper.booleanToFuture(InvalidOrganisationVisibility, 400, Some(cc)) { - ValidOrganisationVisibilities.contains(visibility) - } - existing <- Future(Organisations.organisation.vend.getOrganisation(body.organisation_id)) - _ <- Helper.booleanToFuture(OrganisationAlreadyExists, 409, Some(cc))(existing.isEmpty) - created <- Future { - Organisations.organisation.vend.createOrganisation( - body.organisation_id, body.name, body.website, body.logo_url, - status, visibility, user.userId - ) - }.map(unboxFullOrFail(_, Some(cc), CreateOrganisationError, 400)) - } yield JSONFactory700.createOrganisationJsonV700(created) - } - } - - resourceDocs += ResourceDoc( - implementedInApiVersion, - nameOf(createOrganisation), - "POST", - "/organisations", - "Create Organisation", - """Create an Organisation. - | - |The organisation_id must be a URL-safe string (a-z, A-Z, 0-9, '-', '.', '_'), between 2 and 64 characters in length, and is immutable. - | - |Optional fields: - |- status: one of active, suspended, archived (defaults to active) - |- visibility: one of public, unlisted, private (defaults to public) - |- website, logo_url - | - |Authentication is Required.""".stripMargin, - JSONFactory700.PostOrganisationJsonV700( - organisation_id = "tesobe", - name = "TESOBE GmbH", - website = Some("https://www.tesobe.com"), - logo_url = None, - status = Some("active"), - visibility = Some("public") - ), - JSONFactory700.OrganisationJsonV700( - organisation_id = "tesobe", - name = "TESOBE GmbH", - website = Some("https://www.tesobe.com"), - logo_url = None, - status = "active", - visibility = "public", - created_by_user_id = "9ca9a7e4-6d02-40e3-a129-0b2bf89de9b1", - created_at = new java.util.Date(), - updated_at = new java.util.Date() - ), - List($AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidJsonFormat, - InvalidOrganisationIdFormat, InvalidOrganisationStatus, - InvalidOrganisationVisibility, OrganisationAlreadyExists, - CreateOrganisationError, UnknownError), - apiTagOrganisation :: Nil, - Some(List(canCreateOrganisation)), - http4sPartialFunction = Some(createOrganisation) - ) - - val getOrganisations: Http4sRoute = Http4sRoute { - case req @ GET -> `prefixPath` / "organisations" => - EndpointHelpers.withUser(req) { (user, cc) => - for { - allOrgs <- Organisations.organisation.vend.getAllOrganisations() - .map(unboxFullOrFail(_, Some(cc), UnknownError, 500)) - hasGetAny = APIUtil.hasEntitlement("", user.userId, canGetAnyOrganisation) - visible = if (hasGetAny) allOrgs else allOrgs.filter(_.visibility == "public") - } yield JSONFactory700.createOrganisationsJsonV700(visible) - } - } - - resourceDocs += ResourceDoc( - implementedInApiVersion, - nameOf(getOrganisations), - "GET", - "/organisations", - "Get Organisations", - """Returns Organisations. - | - |By default returns only Organisations whose visibility is `public`. Users granted CanGetAnyOrganisation see all Organisations including unlisted and private. - | - |Authentication is Required.""".stripMargin, - EmptyBody, - JSONFactory700.OrganisationsJsonV700(organisations = List( - JSONFactory700.OrganisationJsonV700( - organisation_id = "tesobe", - name = "TESOBE GmbH", - website = Some("https://www.tesobe.com"), - logo_url = None, - status = "active", - visibility = "public", - created_by_user_id = "9ca9a7e4-6d02-40e3-a129-0b2bf89de9b1", - created_at = new java.util.Date(), - updated_at = new java.util.Date() - ) - )), - List($AuthenticatedUserIsRequired, UnknownError), - apiTagOrganisation :: Nil, - None, - http4sPartialFunction = Some(getOrganisations) - ) - - val getOrganisation: Http4sRoute = Http4sRoute { - case req @ GET -> `prefixPath` / "organisations" / organisationId => - EndpointHelpers.withUser(req) { (user, cc) => - for { - org <- Future(Organisations.organisation.vend.getOrganisation(organisationId)) - .map(unboxFullOrFail(_, Some(cc), OrganisationNotFound, 404)) - _ <- if (org.visibility == "private") - NewStyle.function.hasEntitlement("", user.userId, canGetAnyOrganisation, Some(cc)) - else Future.successful(()) - } yield JSONFactory700.createOrganisationJsonV700(org) - } - } - - resourceDocs += ResourceDoc( - implementedInApiVersion, - nameOf(getOrganisation), - "GET", - "/organisations/ORGANISATION_ID", - "Get Organisation", - """Returns the Organisation specified by ORGANISATION_ID. - | - |Organisations with visibility `public` or `unlisted` are visible to any authenticated user. Organisations with visibility `private` require CanGetAnyOrganisation. - | - |Authentication is Required.""".stripMargin, - EmptyBody, - JSONFactory700.OrganisationJsonV700( - organisation_id = "tesobe", - name = "TESOBE GmbH", - website = Some("https://www.tesobe.com"), - logo_url = None, - status = "active", - visibility = "public", - created_by_user_id = "9ca9a7e4-6d02-40e3-a129-0b2bf89de9b1", - created_at = new java.util.Date(), - updated_at = new java.util.Date() - ), - List($AuthenticatedUserIsRequired, UserHasMissingRoles, OrganisationNotFound, UnknownError), - apiTagOrganisation :: Nil, - None, - http4sPartialFunction = Some(getOrganisation) - ) - - val updateOrganisation: Http4sRoute = Http4sRoute { - case req @ PUT -> `prefixPath` / "organisations" / organisationId => - EndpointHelpers.withUserAndBody[JSONFactory700.PutOrganisationJsonV700, JSONFactory700.OrganisationJsonV700](req) { (_, body, cc) => - for { - _ <- Future(Organisations.organisation.vend.getOrganisation(organisationId)) - .map(unboxFullOrFail(_, Some(cc), OrganisationNotFound, 404)) - _ <- Helper.booleanToFuture(InvalidOrganisationStatus, 400, Some(cc)) { - body.status.forall(ValidOrganisationStatuses.contains) - } - _ <- Helper.booleanToFuture(InvalidOrganisationVisibility, 400, Some(cc)) { - body.visibility.forall(ValidOrganisationVisibilities.contains) - } - updated <- Future { - Organisations.organisation.vend.updateOrganisation( - organisationId, body.name, body.website, body.logo_url, body.status, body.visibility - ) - }.map(unboxFullOrFail(_, Some(cc), UpdateOrganisationError, 400)) - } yield JSONFactory700.createOrganisationJsonV700(updated) - } - } - - resourceDocs += ResourceDoc( - implementedInApiVersion, - nameOf(updateOrganisation), - "PUT", - "/organisations/ORGANISATION_ID", - "Update Organisation", - """Update an Organisation. All body fields are optional. The organisation_id is immutable and cannot be changed. - | - |Authentication is Required.""".stripMargin, - JSONFactory700.PutOrganisationJsonV700( - name = Some("TESOBE GmbH"), - website = Some("https://www.tesobe.com"), - logo_url = None, - status = Some("active"), - visibility = Some("public") - ), - JSONFactory700.OrganisationJsonV700( - organisation_id = "tesobe", - name = "TESOBE GmbH", - website = Some("https://www.tesobe.com"), - logo_url = None, - status = "active", - visibility = "public", - created_by_user_id = "9ca9a7e4-6d02-40e3-a129-0b2bf89de9b1", - created_at = new java.util.Date(), - updated_at = new java.util.Date() - ), - List($AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidJsonFormat, - OrganisationNotFound, InvalidOrganisationStatus, - InvalidOrganisationVisibility, UpdateOrganisationError, UnknownError), - apiTagOrganisation :: Nil, - Some(List(canUpdateOrganisation)), - http4sPartialFunction = Some(updateOrganisation) - ) - - val deleteOrganisation: Http4sRoute = Http4sRoute { - case req @ DELETE -> `prefixPath` / "organisations" / organisationId => - EndpointHelpers.withUserDelete(req) { (_, cc) => - for { - _ <- Future(Organisations.organisation.vend.getOrganisation(organisationId)) - .map(unboxFullOrFail(_, Some(cc), OrganisationNotFound, 404)) - _ <- Future(Organisations.organisation.vend.deleteOrganisation(organisationId)) - .map(unboxFullOrFail(_, Some(cc), DeleteOrganisationError, 400)) - } yield () - } - } - - resourceDocs += ResourceDoc( - implementedInApiVersion, - nameOf(deleteOrganisation), - "DELETE", - "/organisations/ORGANISATION_ID", - "Delete Organisation", - """Delete the Organisation specified by ORGANISATION_ID. - | - |Authentication is Required.""".stripMargin, - EmptyBody, - EmptyBody, - List($AuthenticatedUserIsRequired, UserHasMissingRoles, OrganisationNotFound, - DeleteOrganisationError, UnknownError), - apiTagOrganisation :: Nil, - Some(List(canDeleteOrganisation)), - http4sPartialFunction = Some(deleteOrganisation) - ) - - // ── End Organisations ───────────────────────────────────────────────────── // ── Routing Schemes ─────────────────────────────────────────────────────── // A registry of country-qualified routing scheme names (e.g. TZ.MSISDN, @@ -4945,6 +4708,10 @@ object Http4s700 { EndpointHelpers.withViewAndBodyCreated[JSONFactory700.TransactionRequestBodyOpenCorridorJsonV700, JSONFactory700.TransactionRequestWithChargeOpenCorridorJsonV700](req) { (user, fromAccount, view, body, cc) => val callCtx = Some(cc) for { + // The same switch as settling: a promise that could never be settled here is refused up front. + _ <- code.util.Helper.booleanToFuture(OpenCorridorDisabled, cc = callCtx) { + APIUtil.getPropsAsBoolValue("open_corridor_enabled", false) + } _ <- NewStyle.function.checkAuthorisationToCreateTransactionRequest( view.viewId, BankIdAccountId(fromAccount.bankId, fromAccount.accountId), user, callCtx ) @@ -4995,6 +4762,8 @@ object Http4s700 { | |An OPEN_CORRIDOR_PROMISE does not post a Transaction at create time: the request is held at status `PENDING` as a promise, accumulating for bilateral netting. The Open Corridor settle step later nets all pending promises between a bank pair and posts one net Transaction, at which point covered promises become `COMPLETED`. | + |Requires `open_corridor_enabled=true` on this instance. Whether it is on shows in GET /banks/BANK_ID/transaction-request-types, which lists OPEN_CORRIDOR_PROMISE exactly when it is. + | |Authentication is Required.""".stripMargin, openCorridorBodyExample, JSONFactory700.TransactionRequestWithChargeOpenCorridorJsonV700( @@ -5026,7 +4795,7 @@ object Http4s700 { source = "explicit" )) ), - List($AuthenticatedUserIsRequired, InvalidJsonFormat, InvalidJsonValue, + List($AuthenticatedUserIsRequired, OpenCorridorDisabled, InvalidJsonFormat, InvalidJsonValue, CounterpartyBeneficiaryPermit, InvalidChargePolicy, InsufficientAuthorisationToCreateTransactionRequest, UnknownError), apiTagTransactionRequest :: Nil, @@ -5075,6 +4844,12 @@ object Http4s700 { | |This call is idempotent: re-posting identical evidence returns the stored record. Posting different evidence for a Transaction Request that already has evidence attached is refused — evidence is append-once and cannot be overwritten. | + |**Who may call it.** A person with the Role CanAttachOpenCorridorPromise at BANK_ID, or an application whose Consumer holds it + |as a Scope at BANK_ID and calls as itself, such as the bank's Bank Node with an OAuth2 client-credentials token. + |Such a token gives the application a User of its own (provider obp-oidc#client), so `reported_by_user_id` is + |the application's own User when it acted, or the person's when a person did. + |A person calling through a Consumer that holds the Scope still needs their own Entitlement (authMode UserXORApplication). + | |Authentication is Required.""".stripMargin, openCorridorPromiseBodyExample, JSONFactory700.OpenCorridorPromiseJsonV700( @@ -5094,6 +4869,7 @@ object Http4s700 { OpenCorridorPromiseEvidenceConflict, TransactionRequestLockFailed, UnknownError), apiTagTransactionRequest :: Nil, Some(List(canAttachOpenCorridorPromise)), + authMode = UserXORApplication, http4sPartialFunction = Some(attachOpenCorridorPromise) ) @@ -5268,7 +5044,12 @@ object Http4s700 { (accountAttributes, _) <- NewStyle.function.createAccountAttributes( bankId, accountId, ProductCode(body.product_code), productAttributes, None, Some(cc)) _ <- BankAccountCreation.setAccountHolderAndRefreshUserAccountAccess(bankId, accountId, owner, Some(cc)) - } yield JSONFactory700.createAccountJsonV700(ownerId, bankAccount, accountAttributes) + // Who made the call (user.userId, the agent under a Consent) and for whom: kept in OBP, because + // the account itself may live behind the Connector. AccountCreation_CreatedByUserId. + creation <- Future(AccountCreations.accountCreationProvider.vend.record(bankId.value, accountId.value, user.userId)) map { + x => unboxFullOrFail(x, Some(cc), s"$UnknownError The account ${accountId.value} was created, but who created it could not be recorded.", 500) + } + } yield JSONFactory700.createAccountJsonV700(ownerId, bankAccount, accountAttributes, creation) } val createAccountV700: Http4sRoute = Http4sRoute { @@ -5305,7 +5086,12 @@ object Http4s700 { AccountRoutingJsonV121("OBP", "8ca8a7e4-6d02-40e3-a129-0b2bf89de9f0"), AccountRoutingJsonV121("IBAN", "DE91100000000123456789") ), - account_attributes = Nil + account_attributes = Nil, + metadata = JSONFactory700.AccountMetadataJsonV700(JSONFactory700.AccountCreatedJsonV700( + at = "2026-09-30T10:12:00Z", + user_id = "a9f2c7e1-3d4b-4a5c-8e6f-7a8b9c0d1e2f", + on_behalf_of_user_id = "9ca9a7e4-6d02-40e3-a129-0b2bf89de9b1", + consent_reference_id = Some("4c7e8f2a-1b3d-4e5f-9a6b-7c8d9e0f1a2b"))) ) val createAccountDescriptionV700 = @@ -5321,6 +5107,8 @@ object Http4s700 { | |The balance amount MUST be zero. | + |The response `metadata.created` says who created the Account and for whom: `user_id` is the User who made the call and `on_behalf_of_user_id` the User it was made for. They differ when an application acting for a User, such as an AI agent, calls with a Consent; see [On Behalf Of](/glossary#On-Behalf-Of). `consent_reference_id` is then the Consent's reference id, the one Metrics record for the same call; it is omitted when the caller acted for themselves. OBP keeps this record itself, whichever Connector holds the Account. + | |Authentication is Required.""".stripMargin resourceDocs += ResourceDoc( @@ -5362,6 +5150,63 @@ object Http4s700 { http4sPartialFunction = Some(createAccountWithIdV700) ) + // Route: GET /obp/v7.0.0/banks/BANK_ID/account-directory + // The v6.0.0 Account Directory plus, for each account created through v7.0.0 Create Account, + // who created it and for whom (metadata.created, read from AccountCreations). + val getAccountDirectory: Http4sRoute = Http4sRoute { + case req @ GET -> `prefixPath` / "banks" / _ / "account-directory" => + EndpointHelpers.withUserAndBank(req) { (_, bank, cc) => + val allowedParams = List("limit", "offset", "sort_direction") + for { + httpParams <- NewStyle.function.extractHttpParamsFromUrl(req.uri.renderString) + (obpQueryParams, _) <- NewStyle.function.createObpParams(httpParams, allowedParams, Some(cc)) + (accounts, _) <- NewStyle.function.getAccountDirectory(bank.bankId, obpQueryParams, Some(cc)) + creations <- Future(AccountCreations.accountCreationProvider.vend.getForAccounts(bank.bankId.value, accounts.map(_.id))) + } yield { + val viewsPerAccount: Map[BankIdAccountId, List[String]] = accounts.map { account => + val bankIdAccountId = BankIdAccountId(BankId(account.bankId), AccountId(account.id)) + bankIdAccountId -> Views.views.vend.availableViewsForAccount(bankIdAccountId).map(_.viewId.value) + }.toMap + JSONFactory700.createAccountDirectoryJsonV700(accounts, viewsPerAccount, creations.map(c => c.accountId -> c).toMap) + } + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(getAccountDirectory), + "GET", + "/banks/BANK_ID/account-directory", + "Get Account Directory at Bank", + s"""Returns the accounts at the bank with their identifiers, for management UIs that need to list accounts without exposing sensitive data: balances and owners are not included. + | + |Each account has account_id, bank_id, label, account_number, account_type, branch_id, account_routings, account_attributes and view_ids, as in v6.0.0. + | + |An account created through v7.0.0 Create Account also has `metadata.created`, the same object Create Account returned: `at` is when it was created, `user_id` the User who made the call and `on_behalf_of_user_id` the User it was made for. They differ when an application acting for a User, such as an AI agent, called with a Consent; see [On Behalf Of](/glossary#On-Behalf-Of). `consent_reference_id` is then the Consent's reference id, and is omitted when the caller acted for themselves. `metadata` is omitted for an account OBP holds no creation record for, such as one created by an earlier API version, by the Connector, or the settlement accounts created with the bank. + | + |${urlParametersDocument(true, false)} + | + |Authentication is Required + |""".stripMargin, + EmptyBody, + JSONFactory700.AccountDirectoryJsonV700(List(JSONFactory700.AccountDirectoryItemJsonV700( + account_id = "8ca8a7e4-6d02-40e3-a129-0b2bf89de9f0", + bank_id = "gh.29.uk", + label = "My Account", + account_number = "123456789", + account_type = "CURRENT", + branch_id = "BRANCH_1", + account_routings = List(AccountRoutingJsonV121("OBP", "8ca8a7e4-6d02-40e3-a129-0b2bf89de9f0")), + account_attributes = List(com.openbankproject.commons.model.FastFirehoseAttributes(`type` = "STRING", code = "OVERDRAFT_LIMIT", value = "1000")), + view_ids = List("owner"), + metadata = Some(createAccountResponseExampleV700.metadata) + ))), + List($AuthenticatedUserIsRequired, $BankNotFound, UserHasMissingRoles, UnknownError), + apiTagAccount :: Nil, + Some(List(canGetAccountDirectoryAtOneBank)), + http4sPartialFunction = Some(getAccountDirectory) + ) + // ── OPEN_CORRIDOR per-bank broker registry (admin) ──────────────────────── // Operator endpoints for the per-bank RabbitMQ publish registry: each onboarded // bank's Bank Node consumes on its own vhost, so Interface C publishing needs @@ -5538,6 +5383,10 @@ object Http4s700 { | |Requires `open_corridor_enabled=true` on this instance and the `CanSettleOpenCorridor` role at BANK_ID. | + |**Who may call it.** A person with the Role CanSettleOpenCorridor at BANK_ID, or an application whose Consumer holds it + |as a Scope at BANK_ID and calls as itself, such as the bank's Bank Node with an OAuth2 client-credentials token. + |A person calling through a Consumer that holds the Scope still needs their own Entitlement (authMode UserXORApplication). + | |Authentication is Required.""".stripMargin, JSONFactory700.PostOpenCorridorSettlementJsonV700(other_bank_id = "ke.01.kcs", currency = "KES"), JSONFactory700.OpenCorridorSettleResultJsonV700( @@ -5557,6 +5406,7 @@ object Http4s700 { $BankNotFound, AmqpBankBrokerNotConfigured, OpenCorridorSettlementAddressMissing, UnknownError), apiTagTransactionRequest :: Nil, Some(List(canSettleOpenCorridor)), + authMode = UserXORApplication, http4sPartialFunction = Some(createOpenCorridorSettlement) ) @@ -5595,6 +5445,10 @@ object Http4s700 { | |Requires `open_corridor_enabled=true` and the `CanSettleOpenCorridor` role at BANK_ID (interim — a platform-operator role may replace this). | + |**Who may call it.** A person with the Role CanSettleOpenCorridor at BANK_ID, or an application whose Consumer holds it + |as a Scope at BANK_ID and calls as itself, such as the bank's Bank Node with an OAuth2 client-credentials token. + |A person calling through a Consumer that holds the Scope still needs their own Entitlement (authMode UserXORApplication). + | |Authentication is Required.""".stripMargin, JSONFactory700.PostOpenCorridorFeeSettlementJsonV700(currency = "KES"), code.opencorridorfees.OpenCorridorFeeSweepResultJsonV700( @@ -5610,6 +5464,7 @@ object Http4s700 { $BankNotFound, AmqpBankBrokerNotConfigured, OpenCorridorSettlementAddressMissing, UnknownError), apiTagTransactionRequest :: Nil, Some(List(canSettleOpenCorridor)), + authMode = UserXORApplication, http4sPartialFunction = Some(createOpenCorridorFeeSettlement) ) @@ -5646,6 +5501,10 @@ object Http4s700 { | |Requires `open_corridor_enabled=true` on this instance and the `CanSettleOpenCorridor` role at BANK_ID. | + |**Who may call it.** A person with the Role CanSettleOpenCorridor at BANK_ID, or an application whose Consumer holds it + |as a Scope at BANK_ID and calls as itself, such as the bank's Bank Node with an OAuth2 client-credentials token. + |A person calling through a Consumer that holds the Scope still needs their own Entitlement (authMode UserXORApplication). + | |Authentication is Required.""".stripMargin, EmptyBody, JSONFactory700.OpenCorridorSettlementStatusJsonV700( @@ -5665,12 +5524,16 @@ object Http4s700 { delivery_status = "PENDING", attempts = 3, last_error = "" - )) + )), + settlement_tx_id = Some("91e92fceea0d3b373989ae869fa2e12d77488ace8edcf70325a0ed4ecd500c40"), + settlement_asset = Some("ADA"), + settlement_asset_amount = Some("61943319") ), List($AuthenticatedUserIsRequired, UserHasMissingRoles, OpenCorridorDisabled, $BankNotFound, OpenCorridorSettlementNotFound, UnknownError), apiTagTransactionRequest :: Nil, Some(List(canSettleOpenCorridor)), + authMode = UserXORApplication, http4sPartialFunction = Some(getOpenCorridorSettlement) ) @@ -7810,9 +7673,21 @@ object Http4s700 { // The asset registry: the currencies, metals, accounting units, crypto assets and bank-issued assets amounts are held in. resourceDocs ++= Http4s700Assets.resourceDocs + // Files: documents and photographs stored in a space and attached to records, and the public ones served without login. + resourceDocs ++= Http4s700Files.resourceDocs + + // Send Email to Customer: a bank, or an application for it, tells one of its Customers about something. + resourceDocs ++= Http4s700CustomerEmails.resourceDocs + + // Tenants: the customers of the operator of this instance. + resourceDocs ++= Http4s700Tenants.resourceDocs + // What an OIDC provider reads from OBP-API to finish signing a User in for a Consumer. resourceDocs ++= Http4s700OpenIdConnect.resourceDocs + // Data retention: retention policies, runs and their approval, the disposal register, legal holds and the archive. + resourceDocs ++= Http4s700Retention.resourceDocs + // The order in which the routes are tried. ResourceDocMiddleware selects the doc of the first // route that serves a request, so it is given this same list: the doc that validates a request // is the doc of the route that runs it. diff --git a/obp-api/src/main/scala/code/api/v7_0_0/Http4s700Assets.scala b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700Assets.scala index bc9fa24b00..f23ad6b862 100644 --- a/obp-api/src/main/scala/code/api/v7_0_0/Http4s700Assets.scala +++ b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700Assets.scala @@ -27,17 +27,20 @@ TESOBE (http://www.tesobe.com/) package code.api.v7_0_0 import cats.effect.IO -import code.api.Constant.ApiPathZero +import code.api.Constant.{ApiPathZero, DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID} import code.api.util.APIUtil.{EmptyBody, ResourceDoc, unboxFullOrFail, Http4sRoute} +import code.api.util.ApiRole.canCreateAsset import code.api.util.ApiTag._ import code.api.util.ErrorMessages._ -import code.api.util.CustomJsonFormats +import code.api.util.{CallContext, CustomJsonFormats, NewStyle} import code.api.util.http4s.Http4sRequestAttributes.EndpointHelpers -import code.asset.{AssetStatuses, AssetTypes, Assets} +import code.asset.{AssetStatuses, AssetTrait, AssetTypes, Assets, ChainSchemes} import code.util.Helper import com.github.dwickern.macros.NameOf.nameOf import com.openbankproject.commons.ExecutionContext.Implicits.global +import com.openbankproject.commons.model.{BankId, ProductCode} import com.openbankproject.commons.util.ApiVersion +import net.liftweb.common.{Box, Empty, Failure, Full} import org.http4s._ import org.http4s.dsl.io._ import org.json4s.Formats @@ -51,8 +54,9 @@ import scala.util.Try * metals, accounting units, crypto assets and bank-issued assets that amounts can be held in (see * [[code.asset.Assets]] and ideas/ASSET_REGISTRY.md). * - * They need no authentication: the registry is reference data, and an app may need to check a code - * before anyone has logged in. Nothing here writes. + * The read endpoints need no authentication: the registry is reference data, and an app may need to + * check a code before anyone has logged in. Create Asset adds an asset to the registry; it needs the + * CanCreateAsset Role at the asset's administering bank, which is the bank in the path. * * It is declared in its own object to keep Http4s700's initialiser under the JVM's 64KB method limit. */ @@ -113,8 +117,7 @@ object Http4s700Assets { |accounting units and crypto assets, which are administered at the `SYS` bank), and its status: |`ACTIVE`, `SUSPENDED` or `RETIRED`. | - |The registry does not yet decide which currency codes OBP accepts or how many decimal places it - |gives them; it shows what it will decide once it does. + |The registry decides which currency codes OBP accepts and how many decimal places each one has. | |Query parameters, all optional; a filter matches exactly, apart from letter case for the first three: | @@ -200,4 +203,114 @@ object Http4s700Assets { None, http4sPartialFunction = Some(getAssetByChainIdentity) ) + + /** Treats a missing, empty or blank optional value as not set. */ + private def given(value: Option[String]): Option[String] = value.map(_.trim).filter(_.nonEmpty) + + /** Turns the registry's refusal into a 400 that says what was wrong with the asset. */ + private def createdOrFail(created: Box[AssetTrait], cc: CallContext): AssetTrait = created match { + case Full(asset) => asset + case Failure(message, _, _) => unboxFullOrFail(Empty, Some(cc), s"$InvalidAsset $message", 400) + case _ => unboxFullOrFail(Empty, Some(cc), s"$InvalidAsset", 400) + } + + // Route: POST /obp/v7.0.0/banks/BANK_ID/assets (201) + lazy val createAsset: Http4sRoute = Http4sRoute { + case req @ POST -> `prefixPath` / "banks" / bankIdInUrl / "assets" => + EndpointHelpers.withUserAndBodyCreated[PostAssetJsonV700, AssetJsonV700](req) { (user, body, cc) => + val atSystemBank = bankIdInUrl == DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID + val assetCode = Assets.normaliseCode(Option(body.asset_code).getOrElse("")) + val assetType = Option(body.asset_type).map(_.trim.toUpperCase).getOrElse("") + val issuerProductCode = given(body.issuer_product_code) + val chainScheme = given(body.chain_scheme).map(_.toUpperCase) + val chainAssetId = given(body.chain_asset_id) + val dti = given(body.dti).map(_.toUpperCase) + lazy val whyInvalid = Assets.whyInvalid(assetCode, assetType, Option(body.name).getOrElse(""), body.decimal_places, + if (atSystemBank) None else Some(bankIdInUrl), AssetStatuses.ACTIVE, issuerProductCode, chainScheme, chainAssetId, dti) + for { + _ <- Helper.booleanToFuture(s"$InvalidAsset At the SYS bank only an asset of type ${AssetTypes.CRYPTO} can be created. Current asset_type is $assetType.", 400, Some(cc)) { + !atSystemBank || assetType == AssetTypes.CRYPTO + } + _ <- Helper.booleanToFuture(s"$InvalidAsset At a bank, asset_type must be one of ${AssetTypes.issued.toList.sorted.mkString(", ")}. Current asset_type is $assetType.", 400, Some(cc)) { + atSystemBank || AssetTypes.issued.contains(assetType) + } + _ <- Helper.booleanToFuture(s"$InvalidAsset An asset at the SYS bank has no issuer, so it has no issuer_product_code.", 400, Some(cc)) { + !atSystemBank || issuerProductCode.isEmpty + } + // The registry's own rules before anything is looked up: an asset that could never be created + // is a 400 even if its code happens to be taken (a three-letter code is often an ISO currency) + _ <- Helper.booleanToFuture(s"$InvalidAsset ${whyInvalid.getOrElse("")}", 400, Some(cc)) { whyInvalid.isEmpty } + // The issuer's product describes the instrument, so it must exist at the issuing bank + _ <- issuerProductCode match { + case Some(productCode) => NewStyle.function.getProduct(BankId(bankIdInUrl), ProductCode(productCode), Some(cc)) + case None => Future.successful(()) + } + _ <- Helper.booleanToFuture(s"$AssetAlreadyExists Current asset_code is $assetCode.", 409, Some(cc)) { + Assets.getAsset(assetCode).isEmpty + } + _ <- Helper.booleanToFuture(s"$AssetChainIdentityAlreadyExists Current value is ${chainScheme.getOrElse("")}/${chainAssetId.getOrElse("")}.", 409, Some(cc)) { + (for { scheme <- chainScheme; id <- chainAssetId } yield Assets.getAssetByChainIdentity(scheme, id).isEmpty).getOrElse(true) + } + _ <- Helper.booleanToFuture(s"$AssetDtiAlreadyExists Current dti is ${dti.getOrElse("")}.", 409, Some(cc)) { + dti.forall(value => Assets.getAssetByDti(value).isEmpty) + } + created <- Future(Assets.createAsset( + assetCode = assetCode, + assetType = assetType, + name = Option(body.name).getOrElse(""), + decimalPlaces = body.decimal_places, + issuerBankId = if (atSystemBank) None else Some(bankIdInUrl), + status = AssetStatuses.ACTIVE, + createdByUserId = user.userId, + issuerProductCode = issuerProductCode, + chainScheme = chainScheme, + chainAssetId = chainAssetId, + dti = dti + )).map(createdOrFail(_, cc)) + } yield JSONFactory700Assets.createAssetJson(created) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(createAsset), + "POST", + "/banks/BANK_ID/assets", + "Create Asset", + s"""Adds an asset to the asset registry, so that accounts, transactions and product fees can hold + |amounts in it. Its code becomes a value OBP accepts in `currency` fields. + | + |BANK_ID is the bank that administers the asset, and the Role is checked there: + | + |- At a bank, the asset is one that bank issues, and the bank is recorded as its issuer. `asset_type` + | must be one of ${AssetTypes.issued.toList.sorted.mkString(", ")}. + |- At `SYS`, the asset has no issuer. Only a crypto asset (`asset_type` ${AssetTypes.CRYPTO}) can be + | created there. Currencies, precious metals and accounting units come from ISO 4217 and cannot be created. + | + |The fields: + | + |- `asset_code`: 3 to 10 letters or digits, stored upper case. A bank-issued asset's code is 4 to 10, so + | it can never be mistaken for an ISO 4217 currency code, which is always 3. It must not be taken. + |- `name`: the display name, at most ${Assets.MaxNameLength} characters. + |- `decimal_places`: how many digits an amount in this asset may have after the decimal point, from 0 to + | ${Assets.MaxDecimalPlaces}. It cannot be changed later. + |- `issuer_product_code`: optional, the product at BANK_ID that describes the instrument, such as a bond's + | coupon and maturity date. The product must exist. + |- `chain_scheme` and `chain_asset_id`: optional, both or neither. The chain and network the token is + | recorded on, one of ${ChainSchemes.all.toList.sorted.mkString(", ")}, and its identity there: on + | Cardano, the policy id and the hex asset name joined by a dot; on Ethereum, the token's contract + | address. No other asset may have the same chain identity. + |- `dti`: optional, the ISO 24165 Digital Token Identifier, 9 letters or digits. No other asset may have it. + | + |A new asset is `ACTIVE`. + | + |Authentication is Required.""".stripMargin, + JSONFactory700Assets.postAssetJsonExample, + JSONFactory700Assets.issuedAssetJsonExample, + List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidJsonFormat, InvalidAsset, + ProductNotFoundByProductCode, AssetAlreadyExists, AssetChainIdentityAlreadyExists, AssetDtiAlreadyExists, UnknownError), + List(apiTagAsset), + Some(List(canCreateAsset)), + http4sPartialFunction = Some(createAsset) + ).allowSystemSpace() } diff --git a/obp-api/src/main/scala/code/api/v7_0_0/Http4s700CustomerEmails.scala b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700CustomerEmails.scala new file mode 100644 index 0000000000..63697efcd2 --- /dev/null +++ b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700CustomerEmails.scala @@ -0,0 +1,188 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ +package code.api.v7_0_0 + +import cats.effect.IO +import code.api.Constant +import code.api.Constant.ApiPathZero +import code.api.util.APIUtil.{Http4sRoute, ResourceDoc, UserXORApplication, unboxFullOrFail} +import code.api.util.ApiRole.canSendEmailToCustomerAtOneBank +import code.api.util.ApiTag._ +import code.api.util.ErrorMessages._ +import code.api.util.http4s.Http4sRequestAttributes.EndpointHelpers +import code.api.util.{APIUtil, CommonsEmailWrapper, CustomJsonFormats} +import code.customer.CustomerX +import code.messageoutbox.MessageOutbox +import code.model.dataAccess.SelfServiceBank +import code.util.Helper +import com.github.dwickern.macros.NameOf.nameOf +import com.openbankproject.commons.ExecutionContext.Implicits.global +import com.openbankproject.commons.util.ApiVersion +import net.liftweb.mapper.{By, By_>} +import net.liftweb.util.Helpers.tryo +import org.http4s._ +import org.http4s.dsl.io._ +import org.json4s.Formats +import org.json4s.native.Serialization + +import scala.collection.mutable.ArrayBuffer +import scala.concurrent.Future + +/** + * This object holds Send Email to Customer: a bank, or an application acting for it such as a programme website, + * tells one of its Customers about something (their application was received, a decision was made). A Customer is + * the bank's legal person, so the email goes to someone the space has a relationship with. OBP-API reads the address + * from the Customer and queues the email in the message outbox, whose relay sends it; the caller never sees it. + * + * It is declared in its own object to keep Http4s700's initialiser under the JVM's 64KB method limit. + */ +object Http4s700CustomerEmails { + + implicit val formats: Formats = CustomJsonFormats.formats + + private val implementedInApiVersion = ApiVersion.v7_0_0 + private val prefixPath = Root / ApiPathZero.toString / implementedInApiVersion.toString + + val resourceDocs = ArrayBuffer[ResourceDoc]() + + val OperationCustomerEmail = "customer_email" + val SubjectTypeCustomerId = "customer_id" + val MaxSubjectLength = 200 + val MaxTextLength = 20000 + val MaxReferenceLength = 255 + + object CustomerEmails { + val DefaultMaxPerCustomerPerDay: Int = 20 + + def maxPerCustomerPerDay: Int = + APIUtil.getPropsAsIntValue("customer_emails.max_per_customer_per_day").openOr(DefaultMaxPerCustomerPerDay) + /** The From address: its own prop, else the one the "role granted" email uses. */ + def senderAddress: Option[String] = + APIUtil.getPropsValue("customer_emails.sender_address") + .or(APIUtil.getPropsValue("mail.api.consumer.registered.sender.address")) + .toOption.map(_.trim).filter(_.nonEmpty) + } + + /** An address the outbox can send to: one line, one @, no spaces. Not proof that it works. */ + private def plausibleAddress(email: String): Boolean = + email.length <= 254 && email.count(_ == '@') == 1 && !email.exists(c => c.isWhitespace || c == ',' || c == ';') && + email.indexOf('@') > 0 && email.indexOf('@') < email.length - 1 + + /** Emails queued for a Customer in the last 24 hours, by any application. */ + private def sentInLastDay(customerId: String): Long = + MessageOutbox.count( + By(MessageOutbox.OutboxType, MessageOutbox.TYPE_EMAIL), + By(MessageOutbox.OperationName, OperationCustomerEmail), + By(MessageOutbox.SubjectId, customerId), + By_>(MessageOutbox.CreatedAt, new java.util.Date(System.currentTimeMillis() - 24L * 60 * 60 * 1000)) + ) + + // Route: POST /obp/v7.0.0/banks/BANK_ID/customers/CUSTOMER_ID/emails + lazy val sendEmailToCustomer: Http4sRoute = Http4sRoute { + case req @ POST -> `prefixPath` / "banks" / bankIdInUrl / "customers" / customerId / "emails" if customerId.nonEmpty => + EndpointHelpers.withUserAndBodyCreated[PostCustomerEmailJsonV700, CustomerEmailJsonV700](req) { (caller, body, cc) => + val subject = Option(body.subject).map(_.trim).getOrElse("") + val text = Option(body.text).map(_.trim).getOrElse("") + val reference = body.reference.map(_.trim).filter(_.nonEmpty) + val sender = cc.consumer.map(_.name.get).toOption.filter(_.trim.nonEmpty).getOrElse("an application") + for { + // Anyone can make a self-service bank, hold the Role there and give its Customers any address. + _ <- Helper.booleanToFuture(s"$CustomerEmailsNotInSelfServiceBank Current bank_id is $bankIdInUrl.", 403, Some(cc)) { + !SelfServiceBank.isSelfService(bankIdInUrl) + } + from <- Future(CustomerEmails.senderAddress) + .map(unboxFullOrFail(_, Some(cc), CustomerEmailSenderNotConfigured, 503)) + _ <- Helper.booleanToFuture(s"$InvalidJsonFormat subject must be 1 to $MaxSubjectLength characters on one line.", 400, Some(cc)) { + subject.nonEmpty && subject.length <= MaxSubjectLength && !subject.exists(c => c == '\r' || c == '\n') + } + _ <- Helper.booleanToFuture(s"$InvalidJsonFormat text must be 1 to $MaxTextLength characters.", 400, Some(cc)) { + text.nonEmpty && text.length <= MaxTextLength + } + _ <- Helper.booleanToFuture(s"$InvalidJsonFormat reference must be at most $MaxReferenceLength characters.", 400, Some(cc)) { + reference.forall(_.length <= MaxReferenceLength) + } + // A Customer of another bank is not found here: the space only emails its own Customers. + customer <- Future(CustomerX.customerProvider.vend.getCustomerByCustomerId(customerId).filter(_.bankId == bankIdInUrl)) + .map(unboxFullOrFail(_, Some(cc), s"$CustomerNotFoundByCustomerId Current CUSTOMER_ID is $customerId", 404)) + address = Option(customer.email).map(_.trim).getOrElse("") + _ <- Helper.booleanToFuture(CustomerHasNoEmail, 400, Some(cc))(plausibleAddress(address)) + _ <- Helper.booleanToFuture(s"$CustomerEmailLimitReached At most ${CustomerEmails.maxPerCustomerPerDay} emails a day.", 429, Some(cc)) { + sentInLastDay(customer.customerId) < CustomerEmails.maxPerCustomerPerDay + } + footer = s"\n\n--\nSent by $sender through ${Constant.HostName}." + metadata = Serialization.write(Map( + "bank_id" -> Some(bankIdInUrl), + "consumer_id" -> cc.consumer.map(_.consumerId.get).toOption, + "sent_by_user_id" -> Some(caller.userId), + "reference" -> reference + ).collect { case (k, Some(v)) => k -> v }) + row <- Future(tryo(MessageOutbox.enqueueEmail( + subjectId = customer.customerId, + subjectIdType = SubjectTypeCustomerId, + operationName = OperationCustomerEmail, + CommonsEmailWrapper.EmailContent(from = from, to = List(address), subject = subject, + textContent = Some(text + footer)), + metadataJson = metadata + ))).map(unboxFullOrFail(_, Some(cc), s"$UnknownError Could not queue the email.", 400)) + } yield JSONFactory700CustomerEmails.createCustomerEmailJson(row, bankIdInUrl, customer.customerId, subject, reference) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(sendEmailToCustomer), + "POST", + "/banks/BANK_ID/customers/CUSTOMER_ID/emails", + "Send Email to Customer", + s"""Queues a plain-text email to the Customer CUSTOMER_ID of BANK_ID, at the email address on the Customer record, + |for example to confirm that their application was received or to tell them a decision was made. The address is + |never returned. The message outbox sends the email and retries it until it is delivered. + | + |The body has a `subject` (1 to $MaxSubjectLength characters, one line), the `text` (1 to $MaxTextLength characters) + |and an optional `reference` (at most $MaxReferenceLength characters), for example the record the email is about, + |kept with the queued message. A line saying which application sent the email is added to the text. + | + |**Limits.** Only a Customer of BANK_ID is emailed, and not in a self-service bank (one made with Create My Bank), + |where anyone could hold the Role and give Customers any address. At most `customer_emails.max_per_customer_per_day` (default ${CustomerEmails.DefaultMaxPerCustomerPerDay}) + |emails are queued for a Customer in 24 hours, by all applications together. The From address is + |`customer_emails.sender_address`, else `mail.api.consumer.registered.sender.address`. + | + |**Who may call it.** A User with the Role CanSendEmailToCustomerAtOneBank at BANK_ID, or an application acting as + |itself whose Consumer holds it as a Scope. + | + |Authentication is Required.""".stripMargin, + JSONFactory700CustomerEmails.postCustomerEmailJsonExample, + JSONFactory700CustomerEmails.customerEmailJsonExample, + List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidJsonFormat, CustomerEmailsNotInSelfServiceBank, + CustomerEmailSenderNotConfigured, CustomerNotFoundByCustomerId, CustomerHasNoEmail, CustomerEmailLimitReached, + UnknownError), + List(apiTagCustomer), + Some(List(canSendEmailToCustomerAtOneBank)), + authMode = UserXORApplication, + http4sPartialFunction = Some(sendEmailToCustomer) + ).allowSystemSpace() +} diff --git a/obp-api/src/main/scala/code/api/v7_0_0/Http4s700DynamicEntityDefinitions.scala b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700DynamicEntityDefinitions.scala index 6dfb6cdc40..2ea8da1125 100644 --- a/obp-api/src/main/scala/code/api/v7_0_0/Http4s700DynamicEntityDefinitions.scala +++ b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700DynamicEntityDefinitions.scala @@ -30,7 +30,7 @@ import cats.effect.IO import code.DynamicData.DynamicData import code.api.dynamic.entity.helper.DynamicEntitySpace import code.api.Constant.ApiPathZero -import code.api.util.APIUtil.{EmptyBody, ResourceDoc, UserOrApplication, Http4sRoute} +import code.api.util.APIUtil.{EmptyBody, ResourceDoc, UserXORApplication, Http4sRoute} import code.api.util.ApiRole._ import code.api.util.ApiTag._ import code.api.util.ErrorMessages._ @@ -161,7 +161,7 @@ object Http4s700DynamicEntityDefinitions { List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, UnknownError), apiTagManageDynamicEntity :: apiTagApi :: Nil, Some(canGetDynamicEntityDefinitions :: Nil), - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(getDynamicEntityDefinitions) ).allowSystemSpace() @@ -212,7 +212,7 @@ object Http4s700DynamicEntityDefinitions { List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidJsonFormat, UnknownError), apiTagManageDynamicEntity :: apiTagApi :: Nil, Some(canCreateDynamicEntityDefinition :: Nil), - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(createDynamicEntityDefinition) ).allowSystemSpace() @@ -255,7 +255,7 @@ object Http4s700DynamicEntityDefinitions { DynamicEntityNotFoundByDynamicEntityId, UnknownError), apiTagManageDynamicEntity :: apiTagApi :: Nil, Some(canUpdateDynamicEntityDefinition :: Nil), - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(updateDynamicEntityDefinition) ).allowSystemSpace() diff --git a/obp-api/src/main/scala/code/api/v7_0_0/Http4s700Files.scala b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700Files.scala new file mode 100644 index 0000000000..1d2c840d1e --- /dev/null +++ b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700Files.scala @@ -0,0 +1,1013 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ +package code.api.v7_0_0 + +import java.security.MessageDigest +import java.util.Base64 + +import cats.effect.IO +import code.api.Constant.{ApiPathZero, localIdentityProvider} +import code.api.dynamic.entity.helper.{DynamicEntityHelper, DynamicEntitySpace} +import code.api.util.APIUtil.{EmptyBody, Http4sRoute, ResourceDoc, UserXORApplication, unboxFullOrFail} +import code.api.util.ApiRole +import code.api.util.ApiRole.{canCreateFileAtOneBank, canCreatePublicFileAtOneBank, canGetFileAtOneBank, canUpdateFileScanStatusAtOneBank} +import code.api.util.ApiTag._ +import code.api.util.ErrorMessages._ +import code.api.util.http4s.Http4sRequestAttributes.EndpointHelpers +import code.api.util.http4s.Http4sRequestAttributes.EndpointHelpers.BytesResponse +import code.api.util.{APIUtil, CallContext, CustomJsonFormats, Glossary} +import code.DynamicData.DynamicDataProvider +import code.files.{FileT, Files, ScanStatus} +import code.model.dataAccess.{AuthUser, SelfServiceBank} +import code.users.Users +import code.util.Helper +import com.github.dwickern.macros.NameOf.nameOf +import com.openbankproject.commons.ExecutionContext.Implicits.global +import com.openbankproject.commons.model.User +import com.openbankproject.commons.util.ApiVersion +import net.liftweb.mapper.By +import net.liftweb.util.Helpers.tryo +import org.http4s._ +import org.http4s.dsl.io._ +import org.json4s.Formats + +import scala.collection.mutable.ArrayBuffer +import scala.concurrent.Future +import scala.util.Try + +/** + * This object holds the v7.0.0 file endpoints: documents such as PDFs and photographs that belong to a + * space (a bank, or SYS) and can be attached to records. See [[code.files.Files]] and the Glossary item + * "Files". + * + * Uploading needs CanCreateFileAtOneBank at the space, and making a file public also needs + * CanCreatePublicFileAtOneBank. There are no AtAnyBank Roles: a Role at SYS covers the system space. + * A logged-in caller may read a file they own, one they have been given access to, or any file of a + * space where they hold CanGetFileAtOneBank. Public files are served by separate endpoints, without + * logging in, which serve nothing else. + * + * It is declared in its own object to keep Http4s700's initialiser under the JVM's 64KB method limit. + */ +object Http4s700Files { + + implicit val formats: Formats = CustomJsonFormats.formats + + private val implementedInApiVersion = ApiVersion.v7_0_0 + private val prefixPath = Root / ApiPathZero.toString / implementedInApiVersion.toString + + val resourceDocs = ArrayBuffer[ResourceDoc]() + + /** The most this instance may ever allow. The props can narrow this list, never extend it. */ + val MaxAllowedMediaTypes: List[String] = List("image/jpeg", "image/png", "image/webp", "application/pdf") + + /** files.allowed_media_types, kept to the types in MaxAllowedMediaTypes; all of them when it is not set. */ + def allowedMediaTypes: List[String] = + APIUtil.getPropsValue("files.allowed_media_types").map(_.split(",").map(_.trim.toLowerCase).filter(_.nonEmpty).toList) + .map(configured => MaxAllowedMediaTypes.filter(configured.contains)) + .openOr(MaxAllowedMediaTypes) + + val DefaultMaxSizeInBytes: Long = 10L * 1024 * 1024 + + def maxSizeInBytes: Long = APIUtil.getPropsAsLongValue("files.max_size_in_bytes").openOr(DefaultMaxSizeInBytes) + + val MaxNameLength = 255 + + /** Whether the bytes start the way a file of this media type does, so a file cannot claim to be what it is not. */ + private def contentMatches(mediaType: String, bytes: Array[Byte]): Boolean = { + def startsWith(signature: Int*): Boolean = + bytes.length >= signature.length && signature.zipWithIndex.forall { case (b, i) => (bytes(i) & 0xff) == b } + mediaType match { + case "image/jpeg" => startsWith(0xff, 0xd8, 0xff) + case "image/png" => startsWith(0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a) + case "image/webp" => bytes.length >= 12 && startsWith(0x52, 0x49, 0x46, 0x46) && + new String(bytes.slice(8, 12), "US-ASCII") == "WEBP" + case "application/pdf" => startsWith(0x25, 0x50, 0x44, 0x46, 0x2d) // %PDF- + case _ => false + } + } + + private def sha256Hex(bytes: Array[Byte]): String = + MessageDigest.getInstance("SHA-256").digest(bytes).map("%02x".format(_)).mkString + + /** The ids the caller acts as: their own, and the User an agent acts for. */ + private def actingUserIds(user: User, cc: CallContext): Set[String] = + Set(user.userId, cc.onBehalfOfUserId).filter(_.nonEmpty) + + private def holdsRole(bankId: String, user: User, cc: CallContext, role: ApiRole): Boolean = + APIUtil.handleAccessControlWithAuthMode(bankId, user.userId, APIUtil.getConsumerPrimaryKey(Some(cc)), List(role), UserXORApplication) + + private def isOwner(file: FileT, user: User, cc: CallContext): Boolean = actingUserIds(user, cc).contains(file.ownerUserId) + + private def canRead(file: FileT, user: User, cc: CallContext): Boolean = + isOwner(file, user, cc) || + actingUserIds(user, cc).exists(id => Files.provider.vend.getAccess(file.fileId, id).isDefined) || + holdsRole(file.bankId, user, cc, canGetFileAtOneBank) || + (file.scanStatus != ScanStatus.Clean && holdsRole(file.bankId, user, cc, canUpdateFileScanStatusAtOneBank)) + + /** + * The file, if its content may be served to this caller: a clean file to anyone who may read it; a pending one + * only to its owner and to a holder of CanUpdateFileScanStatusAtOneBank, who scans it; a rejected one to nobody. + */ + private def servable(file: FileT, user: User, cc: CallContext): Future[FileT] = file.scanStatus match { + case ScanStatus.Clean => Future.successful(file) + case ScanStatus.Rejected => Helper.booleanToFuture(FileScanRejected, 403, Some(cc))(false).map(_ => file) + case _ => + Helper.booleanToFuture(FileNotYetScanned, 403, Some(cc)) { + isOwner(file, user, cc) || holdsRole(file.bankId, user, cc, canUpdateFileScanStatusAtOneBank) + }.map(_ => file) + } + + private def readDeniedMessage(bankId: String): String = + s"$UserHasMissingRoles $canGetFileAtOneBank at bank_id $bankId. Or be the file's owner, or be given access to it." + + /** The file, if the caller may read it: 404 when there is no such file, 403 when it is not theirs to read. */ + private def readableFile(bankId: String, fileId: String, user: User, cc: CallContext): Future[FileT] = + for { + file <- Future(Files.provider.vend.getFile(bankId, fileId)) + .map(unboxFullOrFail(_, Some(cc), s"$FileNotFound Current FILE_ID is $fileId", 404)) + _ <- Helper.booleanToFuture(readDeniedMessage(bankId), 403, Some(cc))(canRead(file, user, cc)) + } yield file + + private def publicFile(bankId: String, fileId: String, cc: CallContext): Future[FileT] = + Future(Files.provider.vend.getFile(bankId, fileId).filter(f => f.isPublic && f.scanStatus == ScanStatus.Clean)) + .map(unboxFullOrFail(_, Some(cc), s"$FileNotFound Current FILE_ID is $fileId", 404)) + + private def contentOf(file: FileT, inline: Boolean, cacheControl: String, cc: CallContext): Future[BytesResponse] = + Future(Files.provider.vend.getContent(file.fileId)) + .map(unboxFullOrFail(_, Some(cc), s"$FileNotFound The content of file ${file.fileId} is missing.", 404)) + .map(bytes => BytesResponse( + bytes = bytes, + mediaType = MediaType.parse(file.mediaType).getOrElse(MediaType.application.`octet-stream`), + fileName = file.fileName, + inline = inline, + etag = file.sha256, + cacheControl = cacheControl, + metricSummary = Map("file_id" -> file.fileId, "media_type" -> file.mediaType, "size_in_bytes" -> bytes.length))) + + private def inlineRequested(req: Request[IO]): Boolean = + req.uri.query.params.get("inline").exists(_.trim.equalsIgnoreCase("true")) + + /** record_type and record_id, both required. */ + private def recordOf(req: Request[IO], cc: CallContext): Future[(String, String)] = { + val parameters = req.uri.query.params + val recordType = parameters.get("record_type").map(_.trim).getOrElse("") + val recordId = parameters.get("record_id").map(_.trim).getOrElse("") + Helper.booleanToFuture(s"$InvalidFileQueryParameter record_type and record_id are both required.", 400, Some(cc)) { + recordType.nonEmpty && recordId.nonEmpty + }.map(_ => (recordType, recordId)) + } + + private val whoMayRead = + s"""**Who may call it.** The file's owner, a User the owner has given access to, or a User with the Role + |CanGetFileAtOneBank at BANK_ID (or an application acting as itself whose Consumer holds it as a Scope).""".stripMargin + + // Route: POST /obp/v7.0.0/banks/BANK_ID/files (201) + lazy val createFile: Http4sRoute = Http4sRoute { + case req @ POST -> `prefixPath` / "banks" / bankIdInUrl / "files" => + EndpointHelpers.withUserAndBodyCreated[PostFileJsonV700, FileJsonV700](req) { (user, body, cc) => + val fileName = Option(body.file_name).map(_.trim).getOrElse("") + val mediaType = Option(body.media_type).map(_.trim.toLowerCase).getOrElse("") + val isPublic = body.is_public.getOrElse(false) + val allowed = allowedMediaTypes + val maxSize = maxSizeInBytes + for { + _ <- Helper.booleanToFuture(s"$UserHasMissingRoles $canCreatePublicFileAtOneBank at bank_id $bankIdInUrl. It is needed to create a file with is_public true.", 403, Some(cc)) { + !isPublic || holdsRole(bankIdInUrl, user, cc, canCreatePublicFileAtOneBank) + } + _ <- Helper.booleanToFuture(s"$InvalidFile file_name must be 1 to $MaxNameLength characters.", 400, Some(cc)) { + fileName.nonEmpty && fileName.length <= MaxNameLength + } + _ <- Helper.booleanToFuture(s"$FileMediaTypeNotAllowed media_type must be one of ${allowed.mkString(", ")}. Current media_type is $mediaType.", 400, Some(cc)) { + allowed.contains(mediaType) + } + bytes <- Future(tryo(Base64.getMimeDecoder.decode(Option(body.content_base64).getOrElse("")))) + .map(unboxFullOrFail(_, Some(cc), s"$InvalidFile content_base64 is not valid base64.", 400)) + _ <- Helper.booleanToFuture(s"$InvalidFile The file is empty.", 400, Some(cc))(bytes.nonEmpty) + _ <- Helper.booleanToFuture(s"$FileTooLarge The most allowed is $maxSize bytes. The file has ${bytes.length}.", 400, Some(cc)) { + bytes.length <= maxSize + } + _ <- Helper.booleanToFuture(s"$InvalidFile The content is not a file of media_type $mediaType.", 400, Some(cc)) { + contentMatches(mediaType, bytes) + } + sha256 = sha256Hex(bytes) + _ <- Helper.booleanToFuture(s"$FileSha256Mismatch The content's sha256 is $sha256.", 400, Some(cc)) { + body.sha256.map(_.trim).filter(_.nonEmpty).forall(_.equalsIgnoreCase(sha256)) + } + // A Role in a self-service bank vouches for nothing, so the file waits for a scan there. + scanStatus = if (SelfServiceBank.isSelfService(bankIdInUrl)) ScanStatus.Pending else ScanStatus.Clean + file <- Future(Files.provider.vend.createFile(bankIdInUrl, sha256, mediaType, fileName, isPublic, + user.userId, bytes, scanStatus = scanStatus)) + .map(unboxFullOrFail(_, Some(cc), s"$UnknownError Could not store the file.", 400)) + } yield JSONFactory700Files.createFileJson(file) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(createFile), + "POST", + "/banks/BANK_ID/files", + "Create File", + s"""Stores a file, such as a PDF or a photograph, in a space: a bank, or `SYS` for the system space. + |The caller becomes its owner (the User an agent acts for, when one does). A file's content never + |changes once stored: a new version of a document is a new file. + | + |The fields: + | + |- `file_name`: 1 to $MaxNameLength characters. + |- `media_type`: one of the types this instance allows. At most these are ever allowed: + | ${MaxAllowedMediaTypes.mkString(", ")}; the prop `files.allowed_media_types` can narrow the list. The + | content must really be of that type. + |- `content_base64`: the file, base64 encoded. Its size is limited by the prop `files.max_size_in_bytes` + | (default $DefaultMaxSizeInBytes bytes). + |- `sha256`: optional, the hex SHA-256 of the content. When given, it must match. OBP computes and stores + | its own either way. + |- `is_public`: optional, default false. A public file can be read by anyone, without logging in, through + | Get Public File and Get Public File Content. Making a file public also needs the Role + | CanCreatePublicFileAtOneBank at BANK_ID. + | + |The file starts with `scan_status` clean, except in a bank made with Create My Bank (self-service), where anyone + |can grant themselves the Role: there it starts pending, like a personal file, until a scanner or a holder of + |CanUpdateFileScanStatusAtOneBank sets it. + | + |**Who may call it.** A User with the Role CanCreateFileAtOneBank at BANK_ID (`SYS` for the system space), + |or an application acting as itself whose Consumer holds it as a Scope. + | + |See ${Glossary.getGlossaryItemLink("Files")}. + | + |Authentication is Required.""".stripMargin, + JSONFactory700Files.postFileJsonExample, + JSONFactory700Files.fileJsonExample, + List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidJsonFormat, InvalidFile, + FileMediaTypeNotAllowed, FileTooLarge, FileSha256Mismatch, UnknownError), + List(apiTagFile), + Some(List(canCreateFileAtOneBank)), + authMode = UserXORApplication, + http4sPartialFunction = Some(createFile) + ).allowSystemSpace() + + /** + * Personal files: uploaded by a person, without a Role, within these limits (props, each with a default). They + * are never public, start pending until scanned, and are attached to one of the owner's own personal Dynamic + * Entity records. They are off unless the space is listed in files.personal.enabled_bank_ids. + */ + object PersonalFiles { + val DefaultMediaTypes: List[String] = List("application/pdf", "image/jpeg", "image/png") + val DefaultMaxSizeInBytes: Long = 5L * 1024 * 1024 + val DefaultMaxFilesPerUser: Int = 10 + val DefaultMaxTotalBytesPerUser: Long = 20L * 1024 * 1024 + + private def list(prop: String): Option[List[String]] = + APIUtil.getPropsValue(prop).map(_.split(",").map(_.trim).filter(_.nonEmpty).toList).toOption + + def enabledIn(bankId: String): Boolean = list("files.personal.enabled_bank_ids").exists(_.contains(bankId)) + /** Within allowedMediaTypes, so the personal list can only narrow what the space allows. */ + def mediaTypes: List[String] = { + val wanted = list("files.personal.allowed_media_types").map(_.map(_.toLowerCase)).getOrElse(DefaultMediaTypes) + allowedMediaTypes.filter(wanted.contains) + } + def maxSizeInBytes: Long = + math.min(Http4s700Files.maxSizeInBytes, APIUtil.getPropsAsLongValue("files.personal.max_size_in_bytes").openOr(DefaultMaxSizeInBytes)) + def maxFilesPerUser: Int = APIUtil.getPropsAsIntValue("files.personal.max_files_per_user").openOr(DefaultMaxFilesPerUser) + def maxTotalBytesPerUser: Long = + APIUtil.getPropsAsLongValue("files.personal.max_total_bytes_per_user").openOr(DefaultMaxTotalBytesPerUser) + } + + /** The person the caller acts for: the User an agent acts for, or the caller. Personal files are theirs. */ + private def personOf(user: User, cc: CallContext): String = + Option(cc.onBehalfOfUserId).filter(_.nonEmpty).getOrElse(user.userId) + + /** + * Whether the User's email address counts as validated: for a local User, OBP's own validation; for a User from + * another identity provider, that provider vouches for the address, so having one is enough. + */ + private def emailValidated(userId: String): Boolean = + Users.users.vend.getUserByUserId(userId).exists { u => + if (u.provider != localIdentityProvider) Option(u.emailAddress).exists(_.trim.nonEmpty) + else AuthUser.find(By(AuthUser.user, u.userPrimaryKey.value)).exists(_.validated.get) + } + + /** Whether record_type is a personal Dynamic Entity of the space and record_id one of ownerUserId's own records. */ + private def ownsPersonalRecord(bankId: String, recordType: String, recordId: String, ownerUserId: String): Boolean = { + val space = DynamicEntitySpace.bankIdOrNoneForSystem(bankId) + DynamicEntityHelper.definitionOf(space, recordType).exists(_.hasPersonalEntity) && + DynamicDataProvider.connectorMethodProvider.vend.get(space, recordType, recordId, Some(ownerUserId), isPersonalEntity = true).isDefined + } + + // Route: POST /obp/v7.0.0/banks/BANK_ID/my/files (201) + lazy val createMyFile: Http4sRoute = Http4sRoute { + case req @ POST -> `prefixPath` / "banks" / bankIdInUrl / "my" / "files" => + EndpointHelpers.withUserAndBodyCreated[PostMyFileJsonV700, FileJsonV700](req) { (user, body, cc) => + val fileName = Option(body.file_name).map(_.trim).getOrElse("") + val mediaType = Option(body.media_type).map(_.trim.toLowerCase).getOrElse("") + val recordType = Option(body.record_type).map(_.trim).getOrElse("") + val recordId = Option(body.record_id).map(_.trim).getOrElse("") + val person = personOf(user, cc) + val allowed = PersonalFiles.mediaTypes + val maxSize = PersonalFiles.maxSizeInBytes + for { + _ <- Helper.booleanToFuture(s"$PersonalFilesNotEnabled Current bank_id is $bankIdInUrl.", 403, Some(cc)) { + PersonalFiles.enabledIn(bankIdInUrl) + } + _ <- Helper.booleanToFuture(PersonalFileEmailNotValidated, 403, Some(cc)) { + emailValidated(person) + } + _ <- Helper.booleanToFuture(s"$InvalidFile file_name must be 1 to $MaxNameLength characters.", 400, Some(cc)) { + fileName.nonEmpty && fileName.length <= MaxNameLength + } + _ <- Helper.booleanToFuture(s"$FileMediaTypeNotAllowed media_type must be one of ${allowed.mkString(", ")}. Current media_type is $mediaType.", 400, Some(cc)) { + allowed.contains(mediaType) + } + _ <- Helper.booleanToFuture(s"$PersonalFileRecordNotOwned Current record is $recordType $recordId.", 403, Some(cc)) { + recordType.nonEmpty && recordId.nonEmpty && ownsPersonalRecord(bankIdInUrl, recordType, recordId, person) + } + bytes <- Future(tryo(Base64.getMimeDecoder.decode(Option(body.content_base64).getOrElse("")))) + .map(unboxFullOrFail(_, Some(cc), s"$InvalidFile content_base64 is not valid base64.", 400)) + _ <- Helper.booleanToFuture(s"$InvalidFile The file is empty.", 400, Some(cc))(bytes.nonEmpty) + _ <- Helper.booleanToFuture(s"$FileTooLarge The most allowed is $maxSize bytes. The file has ${bytes.length}.", 400, Some(cc)) { + bytes.length <= maxSize + } + _ <- Helper.booleanToFuture(s"$InvalidFile The content is not a file of media_type $mediaType.", 400, Some(cc)) { + contentMatches(mediaType, bytes) + } + (count, total) = Files.provider.vend.personalUsage(bankIdInUrl, person) + _ <- Helper.booleanToFuture(s"$PersonalFileQuotaExceeded At most ${PersonalFiles.maxFilesPerUser} files and ${PersonalFiles.maxTotalBytesPerUser} bytes; this User has $count files and $total bytes.", 400, Some(cc)) { + count < PersonalFiles.maxFilesPerUser && total + bytes.length <= PersonalFiles.maxTotalBytesPerUser + } + sha256 = sha256Hex(bytes) + _ <- Helper.booleanToFuture(s"$FileSha256Mismatch The content's sha256 is $sha256.", 400, Some(cc)) { + body.sha256.map(_.trim).filter(_.nonEmpty).forall(_.equalsIgnoreCase(sha256)) + } + file <- Future(Files.provider.vend.createFile(bankIdInUrl, sha256, mediaType, fileName, isPublic = false, + user.userId, bytes, isPersonal = true, scanStatus = ScanStatus.Pending)) + .map(unboxFullOrFail(_, Some(cc), s"$UnknownError Could not store the file.", 400)) + _ <- Future(Files.provider.vend.createAttachment(file, recordType, recordId, user.userId)) + .map(unboxFullOrFail(_, Some(cc), s"$UnknownError Could not attach the file.", 400)) + } yield JSONFactory700Files.createFileJson(file) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(createMyFile), + "POST", + "/banks/BANK_ID/my/files", + "Create My File", + s"""Stores a personal file: one a person uploads in their own name, without a Role, for example a pitch deck for + |an application or a document for a KYC check. The caller becomes its owner (the User an agent acts for, when + |one does), and the file is attached at once to one of the owner's own personal Dynamic Entity records: + |`record_type` is the entity's name and `record_id` the record's id. + | + |A personal file: + | + |- is allowed only in the spaces listed in the prop `files.personal.enabled_bank_ids` (none by default); + |- needs the owner's email address to be validated (for a User of another identity provider, having one is enough); + |- is one of the media types in `files.personal.allowed_media_types` (default ${PersonalFiles.DefaultMediaTypes.mkString(", ")}), + | within those the space allows; + |- is at most `files.personal.max_size_in_bytes` (default ${PersonalFiles.DefaultMaxSizeInBytes} bytes), and each User + | holds at most `files.personal.max_files_per_user` (default ${PersonalFiles.DefaultMaxFilesPerUser}) files and + | `files.personal.max_total_bytes_per_user` (default ${PersonalFiles.DefaultMaxTotalBytesPerUser} bytes) per space; + |- is never public; + |- starts with `scan_status` pending. Until it is clean, its content is served only to its owner and to holders of + | CanUpdateFileScanStatusAtOneBank; see Update File Scan Status. + | + |See ${Glossary.getGlossaryItemLink("Files")}. + | + |Authentication is Required.""".stripMargin, + JSONFactory700Files.postMyFileJsonExample, + JSONFactory700Files.myFileJsonExample, + List($BankNotFound, $AuthenticatedUserIsRequired, InvalidJsonFormat, PersonalFilesNotEnabled, + PersonalFileEmailNotValidated, InvalidFile, FileMediaTypeNotAllowed, PersonalFileRecordNotOwned, FileTooLarge, + PersonalFileQuotaExceeded, FileSha256Mismatch, UnknownError), + List(apiTagFile), + None, + http4sPartialFunction = Some(createMyFile) + ).allowSystemSpace() + + // Route: PUT /obp/v7.0.0/banks/BANK_ID/files/FILE_ID/scan-status + lazy val updateFileScanStatus: Http4sRoute = Http4sRoute { + case req @ PUT -> `prefixPath` / "banks" / bankIdInUrl / "files" / fileId / "scan-status" if fileId.nonEmpty => + EndpointHelpers.withUserAndBody[PutFileScanStatusJsonV700, FileJsonV700](req) { (user, body, cc) => + val status = Option(body.scan_status).map(_.trim.toLowerCase).getOrElse("") + val reason = body.reason.map(_.trim).getOrElse("") + for { + _ <- Helper.booleanToFuture(InvalidFileScanStatus, 400, Some(cc))(ScanStatus.all.contains(status)) + _ <- Helper.booleanToFuture(s"$InvalidFile reason must be at most 1000 characters.", 400, Some(cc))(reason.length <= 1000) + _ <- Future(Files.provider.vend.getFile(bankIdInUrl, fileId)) + .map(unboxFullOrFail(_, Some(cc), s"$FileNotFound Current FILE_ID is $fileId", 404)) + file <- Future(Files.provider.vend.setScanStatus(bankIdInUrl, fileId, status, reason, user.userId)) + .map(unboxFullOrFail(_, Some(cc), s"$UnknownError Could not update the file.", 400)) + } yield JSONFactory700Files.createFileJson(file) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(updateFileScanStatus), + "PUT", + "/banks/BANK_ID/files/FILE_ID/scan-status", + "Update File Scan Status", + s"""Sets a file's `scan_status`: pending, clean or rejected, with an optional `reason` (at most 1000 characters), + |for example the signature a scanner matched. A scanner sets it once it has scanned a file; a member of staff sets + |it to approve a file when no scanner is configured, to reject a reported file, or to clear a false positive. + | + |Only a clean file's content is served to everyone who may read the file. A pending file's content is served to + |its owner and to holders of CanUpdateFileScanStatusAtOneBank; a rejected file's to nobody. Get Files of a Space + |with `scan_status=pending` lists the files waiting. + | + |**Who may call it.** A User with the Role CanUpdateFileScanStatusAtOneBank at BANK_ID, or an application acting + |as itself whose Consumer holds it as a Scope. + | + |Authentication is Required.""".stripMargin, + JSONFactory700Files.putFileScanStatusJsonExample, + JSONFactory700Files.fileJsonExample, + List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidJsonFormat, InvalidFileScanStatus, + InvalidFile, FileNotFound, UnknownError), + List(apiTagFile), + Some(List(canUpdateFileScanStatusAtOneBank)), + authMode = UserXORApplication, + http4sPartialFunction = Some(updateFileScanStatus) + ).allowSystemSpace() + + // Route: GET /obp/v7.0.0/banks/BANK_ID/files?record_type=&record_id= + lazy val getFilesAttachedToRecord: Http4sRoute = Http4sRoute { + case req @ GET -> `prefixPath` / "banks" / bankIdInUrl / "files" => + EndpointHelpers.withUser(req) { (user, cc) => + for { + (recordType, recordId) <- recordOf(req, cc) + files <- Future(Files.provider.vend.getFilesAttachedTo(bankIdInUrl, recordType, recordId)) + } yield JSONFactory700Files.createFilesJson(files.filter(canRead(_, user, cc))) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(getFilesAttachedToRecord), + "GET", + "/banks/BANK_ID/files", + "Get Files Attached to a Record", + s"""Lists the files attached to one record, oldest attachment first. Both query parameters are required: + | + |- `record_type`: the kind of record, as given when the file was attached, e.g. `customer` + |- `record_id`: the record's id + | + |Only the files the caller may read are listed: those they own, those they have been given access to, or + |all of them with the Role CanGetFileAtOneBank at BANK_ID. + | + |Authentication is Required.""".stripMargin, + EmptyBody, + JSONFactory700Files.filesJsonExample, + List($BankNotFound, $AuthenticatedUserIsRequired, InvalidFileQueryParameter, UnknownError), + List(apiTagFile), + None, + authMode = UserXORApplication, + http4sPartialFunction = Some(getFilesAttachedToRecord) + ).allowSystemSpace() + + /** The most files one page of Get Files of a Space returns, and the default page size. */ + val MaxPageSize = 500 + + // Route: GET /obp/v7.0.0/management/banks/BANK_ID/files + lazy val getFilesOfSpace: Http4sRoute = Http4sRoute { + case req @ GET -> `prefixPath` / "management" / "banks" / bankIdInUrl / "files" => + EndpointHelpers.withUser(req) { (user, cc) => + val parameters = req.uri.query.params + def number(name: String, default: Int): Int = + parameters.get(name).map(_.trim).filter(_.nonEmpty).map(value => Try(value.toInt).getOrElse(-1)).getOrElse(default) + val limit = number("limit", MaxPageSize) + val offset = number("offset", 0) + val scanStatus = parameters.get("scan_status").map(_.trim.toLowerCase).filter(_.nonEmpty) + for { + _ <- Helper.booleanToFuture(s"$InvalidFileQueryParameter scan_status must be pending or rejected.", 400, Some(cc)) { + scanStatus.forall(s => s == ScanStatus.Pending || s == ScanStatus.Rejected) + } + // A scanner holding only CanUpdateFileScanStatusAtOneBank lists the files it has to deal with, not every file. + _ <- Helper.booleanToFuture(s"$UserHasMissingRoles $canGetFileAtOneBank at bank_id $bankIdInUrl, or give scan_status.", 403, Some(cc)) { + scanStatus.isDefined || holdsRole(bankIdInUrl, user, cc, canGetFileAtOneBank) + } + _ <- Helper.booleanToFuture(s"$InvalidFileQueryParameter limit must be a whole number from 1 to $MaxPageSize.", 400, Some(cc)) { + limit >= 1 && limit <= MaxPageSize + } + _ <- Helper.booleanToFuture(s"$InvalidFileQueryParameter offset must be a whole number, 0 or more.", 400, Some(cc)) { + offset >= 0 + } + (files, total) <- Future(Files.provider.vend.getFiles(bankIdInUrl, limit, offset, scanStatus)) + } yield JSONFactory700Files.createFilesPageJson(files, total, limit, offset) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(getFilesOfSpace), + "GET", + "/management/banks/BANK_ID/files", + "Get Files of a Space", + s"""Lists all the files of a space (a bank, or `SYS` for the system space), newest first, whoever owns them. + | + |Query parameters, both optional: + | + |- `limit`: the most files to return, from 1 to $MaxPageSize (default $MaxPageSize) + |- `offset`: how many files to skip (default 0) + |- `scan_status`: `pending` or `rejected`, to list only those files, e.g. for a scanner + | + |`pagination.total` is the number of files listed in all pages. + | + |**Who may call it.** A User with the Role CanGetFileAtOneBank at BANK_ID, or an application acting as itself + |whose Consumer holds it as a Scope. A holder of CanUpdateFileScanStatusAtOneBank may call it with `scan_status`. + | + |Authentication is Required.""".stripMargin, + EmptyBody, + JSONFactory700Files.filesPageJsonExample, + List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidFileQueryParameter, UnknownError), + List(apiTagFile), + Some(List(canGetFileAtOneBank, canUpdateFileScanStatusAtOneBank)), + authMode = UserXORApplication, + http4sPartialFunction = Some(getFilesOfSpace) + ).allowSystemSpace() + + // Route: GET /obp/v7.0.0/banks/BANK_ID/my/files + lazy val getMyFiles: Http4sRoute = Http4sRoute { + case req @ GET -> `prefixPath` / "banks" / bankIdInUrl / "my" / "files" => + EndpointHelpers.withUser(req) { (user, cc) => + Future(actingUserIds(user, cc).toList.flatMap(Files.provider.vend.getFilesOwnedOrSharedWith(bankIdInUrl, _))) + .map(files => JSONFactory700Files.createFilesJson( + files.groupBy(_.fileId).values.map(_.head).toList.sortBy(_.createdAt.getTime))) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(getMyFiles), + "GET", + "/banks/BANK_ID/my/files", + "Get My Files", + s"""Lists the files of the space that the caller owns or has been given access to, oldest first. + | + |Authentication is Required.""".stripMargin, + EmptyBody, + JSONFactory700Files.filesJsonExample, + List($BankNotFound, $AuthenticatedUserIsRequired, UnknownError), + List(apiTagFile), + None, + authMode = UserXORApplication, + http4sPartialFunction = Some(getMyFiles) + ).allowSystemSpace() + + // Route: GET /obp/v7.0.0/banks/BANK_ID/files/FILE_ID + lazy val getFile: Http4sRoute = Http4sRoute { + case req @ GET -> `prefixPath` / "banks" / bankIdInUrl / "files" / fileId if fileId.nonEmpty => + EndpointHelpers.withUser(req) { (user, cc) => + readableFile(bankIdInUrl, fileId, user, cc).map(JSONFactory700Files.createFileJson) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(getFile), + "GET", + "/banks/BANK_ID/files/FILE_ID", + "Get File", + s"""Returns what is known about a file: its name, media type, size, SHA-256, whether it is public, its owner + |and when it was stored. Get File Content returns the file itself. + | + |$whoMayRead + | + |Authentication is Required.""".stripMargin, + EmptyBody, + JSONFactory700Files.fileJsonExample, + List($BankNotFound, $AuthenticatedUserIsRequired, FileNotFound, UserHasMissingRoles, UnknownError), + List(apiTagFile), + None, + authMode = UserXORApplication, + http4sPartialFunction = Some(getFile) + ).allowSystemSpace() + + // Route: GET /obp/v7.0.0/banks/BANK_ID/files/FILE_ID/content + lazy val getFileContent: Http4sRoute = Http4sRoute { + case req @ GET -> `prefixPath` / "banks" / bankIdInUrl / "files" / fileId / "content" if fileId.nonEmpty => + EndpointHelpers.withUserBytes(req) { (user, cc) => + readableFile(bankIdInUrl, fileId, user, cc).flatMap(servable(_, user, cc)) + .flatMap(contentOf(_, inlineRequested(req), "private, no-store", cc)) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(getFileContent), + "GET", + "/banks/BANK_ID/files/FILE_ID/content", + "Get File Content", + s"""Returns the file itself, not JSON: the body is the file, with the file's own `Content-Type`, its SHA-256 as + |the `ETag`, and `Content-Disposition` naming the file. Add `?inline=true` to have a browser show it rather + |than save it. Errors are JSON, as everywhere else. + | + |$whoMayRead + | + |Authentication is Required.""".stripMargin, + EmptyBody, + EmptyBody, + List($BankNotFound, $AuthenticatedUserIsRequired, FileNotFound, UserHasMissingRoles, UnknownError), + List(apiTagFile), + None, + authMode = UserXORApplication, + http4sPartialFunction = Some(getFileContent) + ).allowSystemSpace() + + // Route: PUT /obp/v7.0.0/banks/BANK_ID/files/FILE_ID/is-public + lazy val updateFileIsPublic: Http4sRoute = Http4sRoute { + case req @ PUT -> `prefixPath` / "banks" / bankIdInUrl / "files" / fileId / "is-public" if fileId.nonEmpty => + EndpointHelpers.withUserAndBody[PutFileIsPublicJsonV700, FileJsonV700](req) { (_, body, cc) => + Future(Files.provider.vend.getFile(bankIdInUrl, fileId)) + .map(unboxFullOrFail(_, Some(cc), s"$FileNotFound Current FILE_ID is $fileId", 404)) + .map(_ => Files.provider.vend.setPublic(bankIdInUrl, fileId, body.is_public)) + .map(unboxFullOrFail(_, Some(cc), s"$UnknownError Could not update the file.", 400)) + .map(JSONFactory700Files.createFileJson) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(updateFileIsPublic), + "PUT", + "/banks/BANK_ID/files/FILE_ID/is-public", + "Update File Is Public", + s"""Makes a file public, or withdraws it, for example one published by mistake. Its content does not change. + | + |Get Public File Content lets clients cache a public file for up to an hour, so a copy may still be served + |from a cache for that long after the file is withdrawn. + | + |**Who may call it.** A User with the Role CanCreatePublicFileAtOneBank at BANK_ID, or an application acting + |as itself whose Consumer holds it as a Scope. + | + |Authentication is Required.""".stripMargin, + JSONFactory700Files.putFileIsPublicJsonExample, + JSONFactory700Files.fileJsonExample, + List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidJsonFormat, FileNotFound, UnknownError), + List(apiTagFile), + Some(List(canCreatePublicFileAtOneBank)), + authMode = UserXORApplication, + http4sPartialFunction = Some(updateFileIsPublic) + ).allowSystemSpace() + + private def manageDeniedMessage(bankId: String): String = + s"$UserHasMissingRoles $canCreateFileAtOneBank at bank_id $bankId. Or be the file's owner." + + // Route: POST /obp/v7.0.0/banks/BANK_ID/files/FILE_ID/attachments (201) + lazy val createFileAttachment: Http4sRoute = Http4sRoute { + case req @ POST -> `prefixPath` / "banks" / bankIdInUrl / "files" / fileId / "attachments" if fileId.nonEmpty => + EndpointHelpers.withUserAndBodyCreated[PostFileAttachmentJsonV700, FileAttachmentJsonV700](req) { (user, body, cc) => + val recordType = Option(body.record_type).map(_.trim).getOrElse("") + val recordId = Option(body.record_id).map(_.trim).getOrElse("") + for { + file <- Future(Files.provider.vend.getFile(bankIdInUrl, fileId)) + .map(unboxFullOrFail(_, Some(cc), s"$FileNotFound Current FILE_ID is $fileId", 404)) + _ <- Helper.booleanToFuture(manageDeniedMessage(bankIdInUrl), 403, Some(cc)) { + isOwner(file, user, cc) || holdsRole(bankIdInUrl, user, cc, canCreateFileAtOneBank) + } + _ <- Helper.booleanToFuture(s"$InvalidFile record_type and record_id must each be 1 to 255 characters.", 400, Some(cc)) { + recordType.nonEmpty && recordType.length <= 255 && recordId.nonEmpty && recordId.length <= 255 + } + // A personal file goes only on its owner's own personal records, unless the Role attaches it. + _ <- Helper.booleanToFuture(s"$PersonalFileRecordNotOwned Current record is $recordType $recordId.", 403, Some(cc)) { + !file.isPersonal || holdsRole(bankIdInUrl, user, cc, canCreateFileAtOneBank) || + ownsPersonalRecord(bankIdInUrl, recordType, recordId, file.ownerUserId) + } + _ <- Helper.booleanToFuture(s"$FileAttachmentAlreadyExists Current record is $recordType $recordId.", 409, Some(cc)) { + !Files.provider.vend.attachmentExists(fileId, recordType, recordId) + } + attachment <- Future(Files.provider.vend.createAttachment(file, recordType, recordId, user.userId)) + .map(unboxFullOrFail(_, Some(cc), s"$UnknownError Could not attach the file.", 400)) + } yield JSONFactory700Files.createFileAttachmentJson(attachment) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(createFileAttachment), + "POST", + "/banks/BANK_ID/files/FILE_ID/attachments", + "Create File Attachment", + s"""Attaches a file to a record, such as a Customer or a Dynamic Entity record. `record_type` names the kind + |of record (e.g. `customer`, or a Dynamic Entity's name) and `record_id` its id; OBP does not look the + |record up. A file is attached to a record at most once. + | + |**Who may call it.** The file's owner, or a User with the Role CanCreateFileAtOneBank at BANK_ID (or an + |application acting as itself whose Consumer holds it as a Scope). + | + |Authentication is Required.""".stripMargin, + JSONFactory700Files.postFileAttachmentJsonExample, + JSONFactory700Files.fileAttachmentJsonExample, + List($BankNotFound, $AuthenticatedUserIsRequired, InvalidJsonFormat, FileNotFound, UserHasMissingRoles, + InvalidFile, FileAttachmentAlreadyExists, UnknownError), + List(apiTagFile), + None, + authMode = UserXORApplication, + http4sPartialFunction = Some(createFileAttachment) + ).allowSystemSpace() + + // Route: GET /obp/v7.0.0/banks/BANK_ID/files/FILE_ID/attachments + lazy val getFileAttachments: Http4sRoute = Http4sRoute { + case req @ GET -> `prefixPath` / "banks" / bankIdInUrl / "files" / fileId / "attachments" if fileId.nonEmpty => + EndpointHelpers.withUser(req) { (user, cc) => + readableFile(bankIdInUrl, fileId, user, cc) + .map(_ => JSONFactory700Files.createFileAttachmentsJson(Files.provider.vend.getAttachments(bankIdInUrl, fileId))) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(getFileAttachments), + "GET", + "/banks/BANK_ID/files/FILE_ID/attachments", + "Get File Attachments", + s"""Lists the records a file is attached to, oldest first. + | + |$whoMayRead + | + |Authentication is Required.""".stripMargin, + EmptyBody, + JSONFactory700Files.fileAttachmentsJsonExample, + List($BankNotFound, $AuthenticatedUserIsRequired, FileNotFound, UserHasMissingRoles, UnknownError), + List(apiTagFile), + None, + authMode = UserXORApplication, + http4sPartialFunction = Some(getFileAttachments) + ).allowSystemSpace() + + // Route: DELETE /obp/v7.0.0/banks/BANK_ID/files/FILE_ID/attachments/FILE_ATTACHMENT_ID (204) + lazy val deleteFileAttachment: Http4sRoute = Http4sRoute { + case req @ DELETE -> `prefixPath` / "banks" / bankIdInUrl / "files" / fileId / "attachments" / fileAttachmentId + if fileId.nonEmpty && fileAttachmentId.nonEmpty => + EndpointHelpers.withUserDelete(req) { (user, cc) => + for { + file <- Future(Files.provider.vend.getFile(bankIdInUrl, fileId)) + .map(unboxFullOrFail(_, Some(cc), s"$FileNotFound Current FILE_ID is $fileId", 404)) + _ <- Helper.booleanToFuture(manageDeniedMessage(bankIdInUrl), 403, Some(cc)) { + isOwner(file, user, cc) || holdsRole(bankIdInUrl, user, cc, canCreateFileAtOneBank) + } + attachment <- Future(Files.provider.vend.getAttachment(bankIdInUrl, fileId, fileAttachmentId)) + .map(unboxFullOrFail(_, Some(cc), s"$FileAttachmentNotFound Current FILE_ATTACHMENT_ID is $fileAttachmentId", 404)) + deleted <- Future(Files.provider.vend.deleteAttachment(attachment.fileAttachmentId)) + .map(unboxFullOrFail(_, Some(cc), s"$UnknownError Could not detach the file.", 400)) + } yield deleted + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(deleteFileAttachment), + "DELETE", + "/banks/BANK_ID/files/FILE_ID/attachments/FILE_ATTACHMENT_ID", + "Delete File Attachment", + s"""Detaches a file from a record. The file itself is kept. + | + |**Who may call it.** The file's owner, or a User with the Role CanCreateFileAtOneBank at BANK_ID (or an + |application acting as itself whose Consumer holds it as a Scope). + | + |Authentication is Required.""".stripMargin, + EmptyBody, + EmptyBody, + List($BankNotFound, $AuthenticatedUserIsRequired, FileNotFound, UserHasMissingRoles, FileAttachmentNotFound, UnknownError), + List(apiTagFile), + None, + authMode = UserXORApplication, + http4sPartialFunction = Some(deleteFileAttachment) + ).allowSystemSpace() + + private def ownedFile(bankId: String, fileId: String, user: User, cc: CallContext): Future[FileT] = + for { + file <- Future(Files.provider.vend.getFile(bankId, fileId)) + .map(unboxFullOrFail(_, Some(cc), s"$FileNotFound Current FILE_ID is $fileId", 404)) + _ <- Helper.booleanToFuture(s"$FileOwnerRequired Only the owner may change who has access to a file.", 403, Some(cc)) { + isOwner(file, user, cc) + } + } yield file + + // Route: POST /obp/v7.0.0/banks/BANK_ID/files/FILE_ID/access (201) + lazy val createFileAccess: Http4sRoute = Http4sRoute { + case req @ POST -> `prefixPath` / "banks" / bankIdInUrl / "files" / fileId / "access" if fileId.nonEmpty => + EndpointHelpers.withUserAndBodyCreated[PostFileAccessJsonV700, FileAccessJsonV700](req) { (user, body, cc) => + val granteeUserId = Option(body.user_id).map(_.trim).getOrElse("") + for { + file <- ownedFile(bankIdInUrl, fileId, user, cc) + grantee <- Future(code.users.Users.users.vend.getUserByUserId(granteeUserId)) + .map(unboxFullOrFail(_, Some(cc), s"$UserNotFoundByUserId Current user_id is $granteeUserId", 404)) + // Explicit target: fail loud rather than redirect (see the entitlement endpoints). + _ <- Helper.booleanToFuture( + s"$InvalidUserId user_id names a consent user (an agent identity minted by a Consent). Access to a file is given to humans - use the granting user's USER_ID.", + 400, Some(cc))(!grantee.isConsentUser) + _ <- Helper.booleanToFuture(s"$FileAccessAlreadyExists Current user_id is $granteeUserId.", 409, Some(cc)) { + Files.provider.vend.getAccess(file.fileId, granteeUserId).isEmpty + } + access <- Future(Files.provider.vend.grantAccess(file.fileId, granteeUserId, user.userId)) + .map(unboxFullOrFail(_, Some(cc), s"$UnknownError Could not give access to the file.", 400)) + } yield JSONFactory700Files.createFileAccessJson(access) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(createFileAccess), + "POST", + "/banks/BANK_ID/files/FILE_ID/access", + "Create File Access", + s"""Gives a User read access to a file: they can then see it with Get File, Get File Content, Get File + |Attachments and Get My Files. The User must be a person: a consent user is refused, since access given to + |one would end with its Consent. + | + |**Who may call it.** The file's owner only. + | + |Authentication is Required.""".stripMargin, + JSONFactory700Files.postFileAccessJsonExample, + JSONFactory700Files.fileAccessJsonExample, + List($BankNotFound, $AuthenticatedUserIsRequired, InvalidJsonFormat, FileNotFound, FileOwnerRequired, + UserNotFoundByUserId, InvalidUserId, FileAccessAlreadyExists, UnknownError), + List(apiTagFile), + None, + authMode = UserXORApplication, + http4sPartialFunction = Some(createFileAccess) + ).allowSystemSpace() + + // Route: GET /obp/v7.0.0/banks/BANK_ID/files/FILE_ID/access + lazy val getFileAccess: Http4sRoute = Http4sRoute { + case req @ GET -> `prefixPath` / "banks" / bankIdInUrl / "files" / fileId / "access" if fileId.nonEmpty => + EndpointHelpers.withUser(req) { (user, cc) => + for { + file <- Future(Files.provider.vend.getFile(bankIdInUrl, fileId)) + .map(unboxFullOrFail(_, Some(cc), s"$FileNotFound Current FILE_ID is $fileId", 404)) + _ <- Helper.booleanToFuture(s"$UserHasMissingRoles $canGetFileAtOneBank at bank_id $bankIdInUrl. Or be the file's owner.", 403, Some(cc)) { + isOwner(file, user, cc) || holdsRole(bankIdInUrl, user, cc, canGetFileAtOneBank) + } + } yield JSONFactory700Files.createFileAccessListJson(Files.provider.vend.getAccessList(file.fileId)) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(getFileAccess), + "GET", + "/banks/BANK_ID/files/FILE_ID/access", + "Get File Access", + s"""Lists the Users who have been given access to a file, and who gave it. + | + |**Who may call it.** The file's owner, or a User with the Role CanGetFileAtOneBank at BANK_ID (or an + |application acting as itself whose Consumer holds it as a Scope). + | + |Authentication is Required.""".stripMargin, + EmptyBody, + JSONFactory700Files.fileAccessListJsonExample, + List($BankNotFound, $AuthenticatedUserIsRequired, FileNotFound, UserHasMissingRoles, UnknownError), + List(apiTagFile), + None, + authMode = UserXORApplication, + http4sPartialFunction = Some(getFileAccess) + ).allowSystemSpace() + + // Route: DELETE /obp/v7.0.0/banks/BANK_ID/files/FILE_ID/access/USER_ID (204) + lazy val deleteFileAccess: Http4sRoute = Http4sRoute { + case req @ DELETE -> `prefixPath` / "banks" / bankIdInUrl / "files" / fileId / "access" / granteeUserId + if fileId.nonEmpty && granteeUserId.nonEmpty => + EndpointHelpers.withUserDelete(req) { (user, cc) => + for { + file <- ownedFile(bankIdInUrl, fileId, user, cc) + _ <- Future(Files.provider.vend.getAccess(file.fileId, granteeUserId)) + .map(unboxFullOrFail(_, Some(cc), s"$FileAccessNotFound Current USER_ID is $granteeUserId", 404)) + revoked <- Future(Files.provider.vend.revokeAccess(file.fileId, granteeUserId)) + .map(unboxFullOrFail(_, Some(cc), s"$UnknownError Could not remove access to the file.", 400)) + } yield revoked + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(deleteFileAccess), + "DELETE", + "/banks/BANK_ID/files/FILE_ID/access/USER_ID", + "Delete File Access", + s"""Takes away a User's access to a file. + | + |**Who may call it.** The file's owner only. + | + |Authentication is Required.""".stripMargin, + EmptyBody, + EmptyBody, + List($BankNotFound, $AuthenticatedUserIsRequired, FileNotFound, FileOwnerRequired, FileAccessNotFound, UnknownError), + List(apiTagFile), + None, + authMode = UserXORApplication, + http4sPartialFunction = Some(deleteFileAccess) + ).allowSystemSpace() + + // ─── Public files: no authentication, and only files that are public ─── + + /** A public file's content may be cached, but only for an hour, since a file can be withdrawn. */ + private val PublicCacheControl = "public, max-age=3600" + + // Route: GET /obp/v7.0.0/banks/BANK_ID/public-files?record_type=&record_id= + lazy val getPublicFilesAttachedToRecord: Http4sRoute = Http4sRoute { + case req @ GET -> `prefixPath` / "banks" / bankIdInUrl / "public-files" => + EndpointHelpers.executeAndRespond(req) { cc => + for { + (recordType, recordId) <- recordOf(req, cc) + files <- Future(Files.provider.vend.getFilesAttachedTo(bankIdInUrl, recordType, recordId)) + } yield JSONFactory700Files.createPublicFilesJson(files.filter(f => f.isPublic && f.scanStatus == ScanStatus.Clean)) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(getPublicFilesAttachedToRecord), + "GET", + "/banks/BANK_ID/public-files", + "Get Public Files Attached to a Record", + s"""Lists the public files attached to one record, oldest attachment first. Files that are not public are + |left out. Both query parameters are required: `record_type` and `record_id`. + | + |No Authentication is Required.""".stripMargin, + EmptyBody, + JSONFactory700Files.publicFilesJsonExample, + List($BankNotFound, InvalidFileQueryParameter, UnknownError), + List(apiTagFile), + None, + http4sPartialFunction = Some(getPublicFilesAttachedToRecord) + ).allowSystemSpace() + + // Route: GET /obp/v7.0.0/banks/BANK_ID/public-files/FILE_ID + lazy val getPublicFile: Http4sRoute = Http4sRoute { + case req @ GET -> `prefixPath` / "banks" / bankIdInUrl / "public-files" / fileId if fileId.nonEmpty => + EndpointHelpers.executeAndRespond(req) { cc => + publicFile(bankIdInUrl, fileId, cc).map(JSONFactory700Files.createPublicFileJson) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(getPublicFile), + "GET", + "/banks/BANK_ID/public-files/FILE_ID", + "Get Public File", + s"""Returns what is known about a public file: its name, media type, size, SHA-256 and when it was stored, + |but not who stored it. A file that is not public is not found here. + | + |No Authentication is Required.""".stripMargin, + EmptyBody, + JSONFactory700Files.publicFileJsonExample, + List($BankNotFound, FileNotFound, UnknownError), + List(apiTagFile), + None, + http4sPartialFunction = Some(getPublicFile) + ).allowSystemSpace() + + // Route: GET /obp/v7.0.0/banks/BANK_ID/public-files/FILE_ID/content + lazy val getPublicFileContent: Http4sRoute = Http4sRoute { + case req @ GET -> `prefixPath` / "banks" / bankIdInUrl / "public-files" / fileId / "content" if fileId.nonEmpty => + EndpointHelpers.executeAndRespondBytes(req) { cc => + publicFile(bankIdInUrl, fileId, cc).flatMap(contentOf(_, inlineRequested(req), PublicCacheControl, cc)) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(getPublicFileContent), + "GET", + "/banks/BANK_ID/public-files/FILE_ID/content", + "Get Public File Content", + s"""Returns a public file itself, not JSON: the body is the file, with the file's own `Content-Type`, its + |SHA-256 as the `ETag`, and `Content-Disposition` naming the file. Add `?inline=true` to have a browser show + |it rather than save it. A file that is not public is not found here. Errors are JSON. + | + |The response may be cached for up to an hour (`Cache-Control: $PublicCacheControl`). + | + |No Authentication is Required.""".stripMargin, + EmptyBody, + EmptyBody, + List($BankNotFound, FileNotFound, UnknownError), + List(apiTagFile), + None, + http4sPartialFunction = Some(getPublicFileContent) + ).allowSystemSpace() +} diff --git a/obp-api/src/main/scala/code/api/v7_0_0/Http4s700Grants.scala b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700Grants.scala index 9fe07566a7..1813a01301 100644 --- a/obp-api/src/main/scala/code/api/v7_0_0/Http4s700Grants.scala +++ b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700Grants.scala @@ -28,7 +28,7 @@ package code.api.v7_0_0 import cats.effect.IO import code.api.Constant.ApiPathZero -import code.api.util.APIUtil.{Http4sRoute, EmptyBody, ResourceDoc, UserOrApplication} +import code.api.util.APIUtil.{Http4sRoute, EmptyBody, ResourceDoc, UserXORApplication} import code.api.util.ApiRole._ import code.api.util.ApiTag._ import code.api.util.ErrorMessages._ @@ -87,7 +87,7 @@ object Http4s700Grants { List($AuthenticatedUserIsRequired, UserHasMissingRoles, UnknownError), List(apiTagScope, apiTagRole), Some(List(canGetAllScopes)), - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(getAllScopes) ) @@ -125,7 +125,7 @@ object Http4s700Grants { List($AuthenticatedUserIsRequired, UserHasMissingRoles, UnknownError), List(apiTagRole, apiTagEntitlement), Some(List(canGetReachableRoles)), - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(getReachableRoles) ) } diff --git a/obp-api/src/main/scala/code/api/v7_0_0/Http4s700MyRoles.scala b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700MyRoles.scala index c65d1efc18..ce382929c3 100644 --- a/obp-api/src/main/scala/code/api/v7_0_0/Http4s700MyRoles.scala +++ b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700MyRoles.scala @@ -74,7 +74,7 @@ object Http4s700MyRoles { val resourceDocs = ArrayBuffer[ResourceDoc]() - private val authModes: List[EndpointAuthMode] = List(UserOnly, ApplicationOnly, UserOrApplication, UserAndApplication) + private val authModes: List[EndpointAuthMode] = List(UserOnly, ApplicationOnly, UserXORApplication, UserAndApplication) /** The same decision as handleAccessControlWithAuthMode, for each auth mode, without granting * just-in-time Entitlements: a GET must not create them. */ @@ -93,7 +93,7 @@ object Http4s700MyRoles { val user = cc.user.toOption val userId = user.map(_.userId).getOrElse("") // A client-credentials token still brings a pseudo-user, named after the Consumer's client id. - val applicationOnly = user.forall(u => Set(Option(consumer.key.get), Option(consumer.azp.get)).flatten.contains(u.idGivenByProvider)) + val applicationOnly = user.forall(u => isApplicationsOwnUser(u, consumer)) val virtualRoles = if (userId.isEmpty) Nil else if (isSuperAdmin(userId)) superAdminVirtualRoles else if (isOidcOperator(userId)) oidcOperatorVirtualRoles @@ -104,10 +104,13 @@ object Http4s700MyRoles { val listed = getPropsValue("require_scopes_for_listed_roles", "").split(",").contains(role.toString) val scopesRequired = ApiPropsWithAlias.requireScopesForAllRoles || listed val allowed = authModes.filter { mode => - virtual || (if (scopesRequired) heldByUser && heldByConsumer else mode match { + virtual || (if (scopesRequired) mode match { + case ApplicationOnly | UserXORApplication if applicationOnly => heldByConsumer + case _ => heldByUser && heldByConsumer + } else mode match { case UserOnly => heldByUser case ApplicationOnly => heldByConsumer - case UserOrApplication => heldByUser || heldByConsumer + case UserXORApplication => if (applicationOnly) heldByConsumer else heldByUser case UserAndApplication => heldByUser && heldByConsumer }) } @@ -127,7 +130,7 @@ object Http4s700MyRoles { caller = MyRoleCallerJsonV700(user_id = "9ca9a7e4-6d02-40e3-a129-0b2bf89de9b1", consumer_id = "7uy8a7e4-6d02-40e3-a129-0b2bf89de8uh", application_only = true), held_by = MyRoleHeldByJsonV700(user = false, consumer = true), scopes_required = false, - allowed_for_auth_modes = List("ApplicationOnly", "UserOrApplication") + allowed_for_auth_modes = List("ApplicationOnly", "UserXORApplication") ) resourceDocs += ResourceDoc( @@ -151,7 +154,7 @@ object Http4s700MyRoles { |* `held_by.consumer`: the Consumer holds the Role as a Scope. |* `scopes_required`: the props `require_scopes_for_all_roles` or `require_scopes_for_listed_roles` make | every endpoint need both. - |* `allowed_for_auth_modes`: the endpoint auth modes (`UserOnly`, `ApplicationOnly`, `UserOrApplication`, + |* `allowed_for_auth_modes`: the endpoint auth modes (`UserOnly`, `ApplicationOnly`, `UserXORApplication`, | `UserAndApplication`) under which an endpoint that needs this Role would let the caller in. | An endpoint that accepts several Roles lets the caller in if any one of them is allowed. | @@ -166,7 +169,7 @@ object Http4s700MyRoles { List(ApplicationNotIdentified, BankNotFound, IncorrectRoleName, EntitlementIsSystemRole, UnknownError), List(apiTagRole, apiTagEntitlement, apiTagScope), None, - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(getMyRole) ).allowSystemSpace() } diff --git a/obp-api/src/main/scala/code/api/v7_0_0/Http4s700OpenIdConnect.scala b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700OpenIdConnect.scala index 5ef9e703dc..32018de7f8 100644 --- a/obp-api/src/main/scala/code/api/v7_0_0/Http4s700OpenIdConnect.scala +++ b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700OpenIdConnect.scala @@ -28,7 +28,7 @@ package code.api.v7_0_0 import cats.effect.IO import code.api.Constant.ApiPathZero -import code.api.util.APIUtil.{Http4sRoute, EmptyBody, ResourceDoc, UserOrApplication, unboxFullOrFail, userAuthenticationMessage} +import code.api.util.APIUtil.{Http4sRoute, EmptyBody, ResourceDoc, UserXORApplication, unboxFullOrFail, userAuthenticationMessage} import code.api.util.ApiRole._ import code.api.util.ApiTag._ import code.api.util.ErrorMessages._ @@ -118,7 +118,7 @@ object Http4s700OpenIdConnect { List($AuthenticatedUserIsRequired, UserHasMissingRoles, ConsentNotFound, UserNotFoundByUserId, UnknownError), apiTagOIDC :: apiTagConsent :: apiTagOAuth :: Nil, Some(canGetOidcConsent :: Nil), - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(getOidcConsent) ) } diff --git a/obp-api/src/main/scala/code/api/v7_0_0/Http4s700PlatformApps.scala b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700PlatformApps.scala index d325914cca..0e080f5710 100644 --- a/obp-api/src/main/scala/code/api/v7_0_0/Http4s700PlatformApps.scala +++ b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700PlatformApps.scala @@ -28,7 +28,7 @@ package code.api.v7_0_0 import cats.effect.IO import code.api.Constant.ApiPathZero -import code.api.util.APIUtil.{EmptyBody, ResourceDoc, UserOrApplication, Http4sRoute} +import code.api.util.APIUtil.{EmptyBody, ResourceDoc, UserXORApplication, Http4sRoute} import code.api.util.ApiRole._ import code.api.util.ApiTag._ import code.api.util.ErrorMessages._ @@ -265,7 +265,7 @@ object Http4s700PlatformApps { List(ApplicationNotIdentified, InvalidJsonFormat, PlatformAppNotFound, InvalidPlatformAppDeclaration, UnknownError), List(apiTagConsumer, apiTagScope), None, - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(updateCurrentConsumerPlatformApp) ) diff --git a/obp-api/src/main/scala/code/api/v7_0_0/Http4s700Retention.scala b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700Retention.scala new file mode 100644 index 0000000000..245a7b9ad5 --- /dev/null +++ b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700Retention.scala @@ -0,0 +1,854 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ +package code.api.v7_0_0 + +import java.util.Date + +import cats.effect.IO +import code.api.Constant.ApiPathZero +import code.api.dynamic.entity.helper.DynamicEntitySpace +import code.api.util.APIUtil.{EmptyBody, Http4sRoute, ResourceDoc, generateUUID} +import code.api.util.ApiRole._ +import code.api.util.ApiTag._ +import code.api.util.ErrorMessages._ +import code.api.util.http4s.Http4sRequestAttributes.EndpointHelpers +import code.api.util.{CallContext, CustomJsonFormats} +import code.dynamicEntity.DynamicEntityProvider +import code.retention.Retention._ +import code.retention._ +import code.util.Helper +import com.github.dwickern.macros.NameOf.nameOf +import com.openbankproject.commons.ExecutionContext.Implicits.global +import com.openbankproject.commons.util.ApiVersion +import net.liftweb.common.Box +import net.liftweb.mapper._ +import net.liftweb.mapper.QueryParam +import org.http4s._ +import org.http4s.dsl.io._ +import org.json4s.Formats + +import scala.collection.mutable.ArrayBuffer +import scala.concurrent.Future +import scala.util.Try + +/** + * This object holds the v7.0.0 data retention endpoints: retention policies, their previews and runs, + * the approval of runs by a second user, the disposal register, legal holds and the retention + * archive. See RETENTION_PLAN.md for the model and code.retention for the work itself. + * + * BANK_ID is the space: a bank's id, or SYS for the system space. The ResourceDocs declare + * allowSystemSpace(), so the middleware lets SYS through and checks each Role at the BANK_ID in the + * URL. Everything is looked up within that space: an id of another space is not found. + * + * Declared in its own object to keep Http4s700's initialiser under the JVM's 64KB method limit. + */ +object Http4s700Retention { + + implicit val formats: Formats = CustomJsonFormats.formats + + private val implementedInApiVersion = ApiVersion.v7_0_0 + private val prefixPath = Root / ApiPathZero.toString / implementedInApiVersion.toString + + val resourceDocs = ArrayBuffer[ResourceDoc]() + + private val MaxPageSize = 500 + private val DefaultPageSize = 50 + private val MaxNameLength = 255 + private val MaxReasonLength = 2000 + + private val tags = apiTagDataRetention :: apiTagApi :: Nil + + // ---- lookups, always within the space of the URL + + private def policyFound(bankId: String, policyId: String, cc: CallContext): Future[RetentionPolicy] = + Future(RetentionPolicy.find(By(RetentionPolicy.BankId, bankId), By(RetentionPolicy.RetentionPolicyId, policyId))) + .map(box => code.api.util.APIUtil.unboxFullOrFail(box, Some(cc), s"$RetentionPolicyNotFound Current RETENTION_POLICY_ID is $policyId.", 404)) + + private def runFound(bankId: String, runId: String, cc: CallContext): Future[RetentionRun] = + Future(RetentionRun.find(By(RetentionRun.BankId, bankId), By(RetentionRun.RetentionRunId, runId))) + .map(box => code.api.util.APIUtil.unboxFullOrFail(box, Some(cc), s"$RetentionRunNotFound Current RETENTION_RUN_ID is $runId.", 404)) + + private def holdFound(bankId: String, holdId: String, cc: CallContext): Future[RetentionHold] = + Future(RetentionHold.find(By(RetentionHold.BankId, bankId), By(RetentionHold.RetentionHoldId, holdId))) + .map(box => code.api.util.APIUtil.unboxFullOrFail(box, Some(cc), s"$RetentionHoldNotFound Current RETENTION_HOLD_ID is $holdId.", 404)) + + private def archiveEntryFound(bankId: String, archiveId: String, cc: CallContext): Future[RetentionArchive] = + Future(RetentionArchive.find(By(RetentionArchive.BankId, bankId), By(RetentionArchive.RetentionArchiveId, archiveId))) + .map(box => code.api.util.APIUtil.unboxFullOrFail(box, Some(cc), s"$RetentionArchiveEntryNotFound Current RETENTION_ARCHIVE_ID is $archiveId.", 404)) + + private def check(problem: Option[String], prefix: String, cc: CallContext): Future[Box[Unit]] = + Helper.booleanToFuture(s"$prefix${problem.getOrElse("")}", 400, Some(cc))(problem.isEmpty) + + // ---- query parameters + + private case class Page(limit: Int, offset: Int) + + private def params(req: Request[IO]): Map[String, String] = + req.uri.query.params.map { case (k, v) => k -> v.trim }.filter(_._2.nonEmpty) + + private def page(req: Request[IO], cc: CallContext): Future[Page] = { + val p = params(req) + val limit = p.get("limit").map(v => Try(v.toInt).getOrElse(-1)).getOrElse(DefaultPageSize) + val offset = p.get("offset").map(v => Try(v.toInt).getOrElse(-1)).getOrElse(0) + for { + _ <- Helper.booleanToFuture(s"${InvalidRetentionQueryParameter}limit must be a whole number from 1 to $MaxPageSize.", 400, Some(cc)) { + limit >= 1 && limit <= MaxPageSize + } + _ <- Helper.booleanToFuture(s"${InvalidRetentionQueryParameter}offset must be a whole number, 0 or more.", 400, Some(cc)) { + offset >= 0 + } + } yield Page(limit, offset) + } + + // ---- policy validation + + private case class PolicySettings(resourceType: String, entityName: Option[String], dateBasis: String, + condition: Option[RetentionCondition], retainDays: Int, archiveDays: Option[Int], + action: String, name: String, description: String) + + /** What is wrong with a policy's settings, if anything. */ + private def policyProblem(bankId: String, s: PolicySettings): Option[String] = { + def entityExists(name: String): Boolean = + DynamicEntityProvider.connectorMethodProvider.vend + .getByEntityName(DynamicEntitySpace.bankIdOrNoneForSystem(bankId), name).isDefined + if (!resourceTypes.contains(s.resourceType)) Some(s"resource_type must be one of ${resourceTypes.mkString(", ")}.") + else if (s.resourceType == ResourceDynamicEntityRecord && s.entityName.isEmpty) + Some("entity_name is required for dynamic_entity_record.") + else if (s.resourceType != ResourceDynamicEntityRecord && s.entityName.isDefined) + Some("entity_name is only allowed for dynamic_entity_record.") + else if (s.resourceType == ResourceDynamicEntityRecord && !entityExists(s.entityName.getOrElse(""))) + Some(s"There is no dynamic entity named ${s.entityName.getOrElse("")} in the space $bankId.") + else if (s.resourceType == ResourceCustomer && bankId == SystemSpace) + Some("A Customer belongs to a bank, so a customer policy names a bank, not SYS.") + else if (!isValidDateBasis(s.resourceType, s.dateBasis)) + Some(s"date_basis must be one of ${dateBases(s.resourceType).mkString(", ")}" + + (if (s.resourceType == ResourceDynamicEntityRecord) s", or $BasisFieldPrefix." else ".")) + else if (!actionsFor(s.resourceType).contains(s.action)) + Some(s"action must be one of ${actionsFor(s.resourceType).mkString(", ")} for ${s.resourceType}.") + else if (s.retainDays < RetentionProps.minimumRetainDays || s.retainDays > MaxDays) + Some(s"retain_days must be a whole number from ${RetentionProps.minimumRetainDays} (retention.minimum_retain_days) to $MaxDays.") + else if (s.action == ActionArchive && s.archiveDays.isEmpty) + Some("archive_days is required for the archive action: how long an archived resource is kept before it is deleted.") + else if (s.action != ActionArchive && s.archiveDays.isDefined) + Some("archive_days is only allowed for the archive action.") + else if (s.archiveDays.exists(d => d < 0 || d > MaxDays)) Some(s"archive_days must be a whole number from 0 to $MaxDays.") + else if (s.condition.exists(_.value.length > 2000)) Some("condition.value must be at most 2000 characters.") + else if (s.name.length > MaxNameLength) Some(s"name must be at most $MaxNameLength characters.") + else None + } + + private def conditionOf(json: Option[RetentionConditionJsonV700]): Option[RetentionCondition] = + json.map(c => RetentionCondition(Option(c.field).map(_.trim).getOrElse(""), Option(c.value).getOrElse(""))) + .filter(_.field.nonEmpty) + + private def save(policy: RetentionPolicy, s: PolicySettings, cascade: Boolean, requiresApproval: Boolean, enabled: Boolean): RetentionPolicy = + policy + .Name(s.name) + .Description(s.description) + .ResourceType(s.resourceType) + .EntityName(s.entityName.getOrElse("")) + .DateBasis(s.dateBasis) + .ConditionField(s.condition.map(_.field).getOrElse("")) + .ConditionValue(s.condition.map(_.value).getOrElse("")) + .RetainDays(s.retainDays) + .ArchiveDays(s.archiveDays.getOrElse(-1)) + .Action(s.action) + .Cascade(cascade) + .RequiresApproval(requiresApproval) + .Enabled(enabled) + .saveMe() + + private val policyDescription = + s"""A retention policy says how long one kind of resource of a space stays live, and what happens to it then. + |BANK_ID is the space: a bank's id, or `SYS` for the system space. The Role is checked at that BANK_ID. + |Periods are configuration, set here per space, never in code. + | + |* `resource_type`: one of ${resourceTypes.mkString(", ")}. + |* `entity_name`: the dynamic entity, for `dynamic_entity_record` only. It must exist in the space. + |* `date_basis`: what the period is counted from. `created` or `updated` for dynamic entity records, + | chat messages and customers; or, for dynamic entity records, `field:`, the ISO 8601 date in that + | field of the record. `created` for files. `created`, `updated` or `last_activity` (the last message, + | or creation when there was none) for chat rooms. `created`, `updated` or `expires` (the expiry of the + | Consent's token; the default) for consents. Defaults to `created` otherwise. + |* `condition`: optional, `{"field": "status", "value": "CLOSED"}`. Only resources whose field has this + | value (compared as text) are governed. For a dynamic entity record the field is one of the record's own; + | for the other types it is a field of the resource's JSON copy, such as `media_type`, `message_type`, + | `customer_type` or `status`. + |* `retain_days`: how long a resource stays live after its trigger date. At least `retention.minimum_retain_days` + | (1825 days, five years, by default). + |* `action`: `archive` (move it out of the live tables into the retention archive, readable only with + | CanGetRetentionArchive, then delete it after `archive_days`), `delete` (final), or, for customers + | only, `anonymise` (clear the personal fields and delete the attributes, addresses, dependants, tax + | residences and links to Users, keeping the customer id so that references stay valid). + | Archiving is not an undo: nothing is ever restored from the archive. + |* `archive_days`: required for `archive`, not allowed otherwise. + |* `cascade`: files attached to a disposed resource go with it (a dynamic entity record's files are attached + | with the entity name as record type; a customer's with `customer`). A file still attached to another + | record stays. Default false: the attachments go, the files stay. + |* `requires_approval`: an execution waits as pending until a second, different user approves it. + |* `enabled`: whether the retention scheduler runs the policy. Default true. A run can always be + | started by hand. + | + |Every disposal is written to the disposal register. Resources on legal hold are skipped by every run. + |A run that would archive is refused unless `retention.archive.enabled` is true, and one that would delete + |or anonymise unless `retention.delete.enabled` is true. See RETENTION_PLAN.md.""".stripMargin + + private val errorsOnPolicyWrite = List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidJsonFormat, + InvalidRetentionPolicy, UnknownError) + + // Route: POST /obp/v7.0.0/management/banks/BANK_ID/retention-policies (201) + lazy val createRetentionPolicy: Http4sRoute = Http4sRoute { + case req @ POST -> `prefixPath` / "management" / "banks" / bankId / "retention-policies" => + EndpointHelpers.withUserAndBodyCreated[PostRetentionPolicyJsonV700, RetentionPolicyJsonV700](req) { (user, body, cc) => + val resourceType = Option(body.resource_type).map(_.trim).getOrElse("") + val settings = PolicySettings( + resourceType = resourceType, + entityName = body.entity_name.map(_.trim).filter(_.nonEmpty), + dateBasis = body.date_basis.map(_.trim).filter(_.nonEmpty).getOrElse(defaultDateBasis(resourceType)), + condition = conditionOf(body.condition), + retainDays = body.retain_days, + archiveDays = body.archive_days, + action = Option(body.action).map(_.trim).getOrElse(""), + name = body.name.map(_.trim).getOrElse(""), + description = body.description.getOrElse("")) + for { + problem <- Future(policyProblem(bankId, settings)) + _ <- check(problem, InvalidRetentionPolicy, cc) + created <- Future(save(RetentionPolicy.create.RetentionPolicyId(generateUUID()).BankId(bankId).CreatedByUserId(user.userId), + settings, body.cascade.getOrElse(false), body.requires_approval.getOrElse(false), body.enabled.getOrElse(true))) + } yield JSONFactory700Retention.createPolicyJson(created) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(createRetentionPolicy), + "POST", + "/management/banks/BANK_ID/retention-policies", + "Create Retention Policy", + s"""Create a retention policy for a space. + | + |Only `resource_type`, `retain_days` and `action` are required (and `entity_name` and `archive_days` where + |the type and action call for them). + | + |$policyDescription""".stripMargin, + JSONFactory700Retention.postRetentionPolicyJsonV700Example, + JSONFactory700Retention.retentionPolicyJsonV700Example, + errorsOnPolicyWrite, + tags, + Some(canManageRetentionPolicy :: Nil), + http4sPartialFunction = Some(createRetentionPolicy) + ).allowSystemSpace() + + // Route: GET /obp/v7.0.0/management/banks/BANK_ID/retention-policies + lazy val getRetentionPolicies: Http4sRoute = Http4sRoute { + case req @ GET -> `prefixPath` / "management" / "banks" / bankId / "retention-policies" => + EndpointHelpers.withUser(req) { (_, _) => + Future(RetentionPolicy.findAll(By(RetentionPolicy.BankId, bankId), OrderBy(RetentionPolicy.id, Ascending))) + .map(policies => RetentionPoliciesJsonV700(policies.map(JSONFactory700Retention.createPolicyJson))) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(getRetentionPolicies), + "GET", + "/management/banks/BANK_ID/retention-policies", + "Get Retention Policies", + s"""The retention policies of a space, oldest first. + | + |$policyDescription""".stripMargin, + EmptyBody, + JSONFactory700Retention.retentionPoliciesJsonV700Example, + List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, UnknownError), + tags, + Some(canGetRetentionPolicies :: canManageRetentionPolicy :: Nil), + http4sPartialFunction = Some(getRetentionPolicies) + ).allowSystemSpace() + + // Route: GET /obp/v7.0.0/management/banks/BANK_ID/retention-policies/RETENTION_POLICY_ID + lazy val getRetentionPolicy: Http4sRoute = Http4sRoute { + case req @ GET -> `prefixPath` / "management" / "banks" / bankId / "retention-policies" / policyId if policyId.nonEmpty => + EndpointHelpers.withUser(req) { (_, cc) => + policyFound(bankId, policyId, cc).map(JSONFactory700Retention.createPolicyJson) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(getRetentionPolicy), + "GET", + "/management/banks/BANK_ID/retention-policies/RETENTION_POLICY_ID", + "Get Retention Policy", + s"""One retention policy of a space. + | + |$policyDescription""".stripMargin, + EmptyBody, + JSONFactory700Retention.retentionPolicyJsonV700Example, + List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, RetentionPolicyNotFound, UnknownError), + tags, + Some(canGetRetentionPolicies :: canManageRetentionPolicy :: Nil), + http4sPartialFunction = Some(getRetentionPolicy) + ).allowSystemSpace() + + // Route: PUT /obp/v7.0.0/management/banks/BANK_ID/retention-policies/RETENTION_POLICY_ID + lazy val updateRetentionPolicy: Http4sRoute = Http4sRoute { + case req @ PUT -> `prefixPath` / "management" / "banks" / bankId / "retention-policies" / policyId if policyId.nonEmpty => + EndpointHelpers.withUserAndBody[PutRetentionPolicyJsonV700, RetentionPolicyJsonV700](req) { (_, body, cc) => + for { + policy <- policyFound(bankId, policyId, cc) + action = body.action.map(_.trim).getOrElse(policy.action) + settings = PolicySettings( + resourceType = policy.resourceType, + entityName = policy.entityName, + dateBasis = body.date_basis.map(_.trim).filter(_.nonEmpty).getOrElse(policy.dateBasis), + condition = if (body.condition.isDefined) conditionOf(body.condition) else policy.condition, + retainDays = body.retain_days.getOrElse(policy.retainDays), + archiveDays = if (action != ActionArchive) None else body.archive_days.orElse(policy.archiveDays), + action = action, + name = body.name.map(_.trim).getOrElse(policy.name), + description = body.description.getOrElse(policy.description)) + problem <- Future(policyProblem(bankId, settings)) + _ <- check(problem, InvalidRetentionPolicy, cc) + updated <- Future(save(policy, settings, body.cascade.getOrElse(policy.cascade), + body.requires_approval.getOrElse(policy.requiresApproval), body.enabled.getOrElse(policy.enabled))) + } yield JSONFactory700Retention.createPolicyJson(updated) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(updateRetentionPolicy), + "PUT", + "/management/banks/BANK_ID/retention-policies/RETENTION_POLICY_ID", + "Update Retention Policy", + s"""Change a retention policy. Every field is optional, and a field left out keeps its value. The resource type + |and the entity name cannot change: create another policy instead. A `condition` whose `field` is empty + |removes the condition. `archive_days` is cleared when the action becomes anything but `archive`. + | + |A change applies to runs started after it; a pending run executes with the policy as it is when approved. + | + |$policyDescription""".stripMargin, + JSONFactory700Retention.putRetentionPolicyJsonV700Example, + JSONFactory700Retention.retentionPolicyJsonV700Example, + RetentionPolicyNotFound :: errorsOnPolicyWrite, + tags, + Some(canManageRetentionPolicy :: Nil), + http4sPartialFunction = Some(updateRetentionPolicy) + ).allowSystemSpace() + + // Route: DELETE /obp/v7.0.0/management/banks/BANK_ID/retention-policies/RETENTION_POLICY_ID (204) + lazy val deleteRetentionPolicy: Http4sRoute = Http4sRoute { + case req @ DELETE -> `prefixPath` / "management" / "banks" / bankId / "retention-policies" / policyId if policyId.nonEmpty => + EndpointHelpers.withUserDelete(req) { (_, cc) => + for { + policy <- policyFound(bankId, policyId, cc) + _ <- Future { + RetentionRun.findAll(By(RetentionRun.RetentionPolicyId, policyId), By(RetentionRun.Status, StatusPending)).foreach( + _.Status(StatusRefused).EndedAt(new Date()).Message("The policy of this run has been deleted.").saveMe()) + policy.delete_! + } + } yield () + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(deleteRetentionPolicy), + "DELETE", + "/management/banks/BANK_ID/retention-policies/RETENTION_POLICY_ID", + "Delete Retention Policy", + s"""Delete a retention policy. Its pending runs are refused. Its runs, the disposal register and what it + |archived stay; archived resources are still deleted from the archive when their archive period ends (by the + |retention scheduler, with `retention.delete.enabled`). + | + |BANK_ID is the space: a bank's id, or `SYS` for the system space. The Role is checked at that BANK_ID.""".stripMargin, + EmptyBody, + EmptyBody, + List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, RetentionPolicyNotFound, UnknownError), + tags, + Some(canManageRetentionPolicy :: Nil), + http4sPartialFunction = Some(deleteRetentionPolicy) + ).allowSystemSpace() + + // Route: GET /obp/v7.0.0/management/banks/BANK_ID/retention-policies/RETENTION_POLICY_ID/preview + lazy val getRetentionPolicyPreview: Http4sRoute = Http4sRoute { + case req @ GET -> `prefixPath` / "management" / "banks" / bankId / "retention-policies" / policyId / "preview" => + EndpointHelpers.withUser(req) { (_, cc) => + for { + policy <- policyFound(bankId, policyId, cc) + result <- Future(RetentionEngine.preview(policy)) + } yield JSONFactory700Retention.createPreviewJson(policy, result) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(getRetentionPolicyPreview), + "GET", + "/management/banks/BANK_ID/retention-policies/RETENTION_POLICY_ID/preview", + "Get Retention Policy Preview", + s"""A dry run: what a run of the policy would dispose of now. Nothing is changed or recorded, and a preview works + |whatever the props say. + | + |* `cutoff_date`: a resource whose trigger date is on or before it is due. It counts `retain_days`, or + | `minimum_retain_days` (`retention.minimum_retain_days`) if that is longer. + |* `matched_count`: the resources due; `held_count` of them are on legal hold and would be skipped; + | `due_count` is the rest. One run disposes of at most `max_items_per_run`. + |* `oldest_trigger_date` and up to ${RetentionEngine.SampleSize} `sample_resource_ids` not on hold. + |* `archive_entries_due_count`: resources this policy archived whose archive period has ended. + |* `count_is_capped`: counting stopped at ${RetentionEngine.PreviewScanLimit}. + |* `can_execute`: whether `retention.archive.enabled` / `retention.delete.enabled` allow an execution now. + | + |BANK_ID is the space: a bank's id, or `SYS` for the system space. The Role is checked at that BANK_ID.""".stripMargin, + EmptyBody, + JSONFactory700Retention.retentionPolicyPreviewJsonV700Example, + List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, RetentionPolicyNotFound, UnknownError), + tags, + Some(canGetRetentionPolicies :: canRunRetentionPolicy :: canManageRetentionPolicy :: Nil), + http4sPartialFunction = Some(getRetentionPolicyPreview) + ).allowSystemSpace() + + /** A refused run is recorded, then answered with the reason and the run's id. */ + private def answerRun(run: RetentionRun, cc: CallContext): Future[RetentionRunJsonV700] = + Helper.booleanToFuture(s"${run.message} The refused run is recorded as RETENTION_RUN_ID ${run.retentionRunId}.", 400, Some(cc)) { + run.status != StatusRefused + }.map(_ => JSONFactory700Retention.createRunJson(run)) + + // Route: POST /obp/v7.0.0/management/banks/BANK_ID/retention-policies/RETENTION_POLICY_ID/runs (201) + lazy val createRetentionRun: Http4sRoute = Http4sRoute { + case req @ POST -> `prefixPath` / "management" / "banks" / bankId / "retention-policies" / policyId / "runs" => + EndpointHelpers.withUserAndStatus(req) { (user, cc) => + val rawBody = cc.httpBody.getOrElse("").trim + val body: Option[PostRetentionRunJsonV700] = + if (rawBody.isEmpty) Some(PostRetentionRunJsonV700()) + else Try(com.openbankproject.commons.util.JsonAliases.parse(rawBody).extract[PostRetentionRunJsonV700]).toOption + val mode = body.flatMap(_.mode).map(_.trim).filter(_.nonEmpty).getOrElse(ModeExecute) + for { + _ <- Helper.booleanToFuture(s"$InvalidJsonFormat The Json body should be the PostRetentionRunJsonV700 ", 400, Some(cc))(body.isDefined) + _ <- Helper.booleanToFuture(InvalidRetentionRun, 400, Some(cc))(modes.contains(mode)) + policy <- policyFound(bankId, policyId, cc) + _ <- Helper.booleanToFuture(RetentionRunAlreadyInProgress, 409, Some(cc)) { + mode == ModePreview || !RetentionEngine.isRunning(policy.retentionPolicyId) + } + run <- Future(RetentionEngine.requestRun(policy, mode, user.userId)) + json <- answerRun(run, cc) + } yield (json, 201) + } + } + + private val runDescription = + s"""A run has a `mode`, `preview` (counts only, nothing changes) or `execute`, and a `status`: `pending` (waiting for + |approval by a second user), `running`, `done`, `refused` (the props switch its action off, the approver rejected + |it, or its policy was deleted) or `failed` (some resources could not be disposed of and were left in place; see + |`message`). The counts include what went with a resource: cascaded files, and a chat room's messages. + |`requested_by_user_id` is left out when the retention scheduler requested the run. + | + |BANK_ID is the space: a bank's id, or `SYS` for the system space. The Role is checked at that BANK_ID.""".stripMargin + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(createRetentionRun), + "POST", + "/management/banks/BANK_ID/retention-policies/RETENTION_POLICY_ID/runs", + "Create Retention Run", + s"""Run a retention policy now. The body is optional: `{"mode": "execute"}` (the default) or `{"mode": "preview"}`. + | + |An execution of a policy that requires approval is created as `pending` and runs when a different user approves + |it with Update Retention Run Approval. Otherwise it runs now and the response shows its outcome. One run disposes + |of at most `retention.run.max_items` resources; the next run continues. + | + |An execution is refused, with 400 and a recorded `refused` run, when `retention.archive.enabled` (for `archive`) + |or `retention.delete.enabled` (for `delete` and `anonymise`) is not true. A preview is never refused. + | + |$runDescription""".stripMargin, + JSONFactory700Retention.postRetentionRunJsonV700Example, + JSONFactory700Retention.retentionRunJsonV700Example, + List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidJsonFormat, InvalidRetentionRun, + RetentionPolicyNotFound, RetentionRunAlreadyInProgress, RetentionArchiveIsDisabled, RetentionDeleteIsDisabled, UnknownError), + tags, + Some(canRunRetentionPolicy :: Nil), + http4sPartialFunction = Some(createRetentionRun) + ).allowSystemSpace() + + // Route: GET /obp/v7.0.0/management/banks/BANK_ID/retention-runs + lazy val getRetentionRuns: Http4sRoute = Http4sRoute { + case req @ GET -> `prefixPath` / "management" / "banks" / bankId / "retention-runs" => + EndpointHelpers.withUser(req) { (_, cc) => + val p = params(req) + for { + pg <- page(req, cc) + _ <- Helper.booleanToFuture(s"${InvalidRetentionQueryParameter}status must be one of ${statuses.mkString(", ")}.", 400, Some(cc)) { + p.get("status").forall(statuses.contains) + } + runs <- Future(RetentionRun.findAll( + (List[QueryParam[RetentionRun]](By(RetentionRun.BankId, bankId)) ::: + p.get("retention_policy_id").map(id => By(RetentionRun.RetentionPolicyId, id)).toList ::: + p.get("status").map(s => By(RetentionRun.Status, s)).toList ::: + List(OrderBy(RetentionRun.id, Descending), StartAt[RetentionRun](pg.offset), MaxRows[RetentionRun](pg.limit))): _*)) + } yield RetentionRunsJsonV700(runs.map(JSONFactory700Retention.createRunJson)) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(getRetentionRuns), + "GET", + "/management/banks/BANK_ID/retention-runs", + "Get Retention Runs", + s"""The retention runs of a space, newest first. Optional query parameters: `retention_policy_id`, `status`, + |`limit` (1 to $MaxPageSize, default $DefaultPageSize) and `offset` (default 0). + | + |$runDescription""".stripMargin, + EmptyBody, + JSONFactory700Retention.retentionRunsJsonV700Example, + List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidRetentionQueryParameter, UnknownError), + tags, + Some(canGetRetentionRuns :: Nil), + http4sPartialFunction = Some(getRetentionRuns) + ).allowSystemSpace() + + // Route: GET /obp/v7.0.0/management/banks/BANK_ID/retention-runs/RETENTION_RUN_ID + lazy val getRetentionRun: Http4sRoute = Http4sRoute { + case req @ GET -> `prefixPath` / "management" / "banks" / bankId / "retention-runs" / runId if runId.nonEmpty => + EndpointHelpers.withUser(req) { (_, cc) => + runFound(bankId, runId, cc).map(JSONFactory700Retention.createRunJson) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(getRetentionRun), + "GET", + "/management/banks/BANK_ID/retention-runs/RETENTION_RUN_ID", + "Get Retention Run", + s"""One retention run of a space. + | + |$runDescription""".stripMargin, + EmptyBody, + JSONFactory700Retention.retentionRunJsonV700Example, + List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, RetentionRunNotFound, UnknownError), + tags, + Some(canGetRetentionRuns :: Nil), + http4sPartialFunction = Some(getRetentionRun) + ).allowSystemSpace() + + // Route: PUT /obp/v7.0.0/management/banks/BANK_ID/retention-runs/RETENTION_RUN_ID/approval + lazy val updateRetentionRunApproval: Http4sRoute = Http4sRoute { + case req @ PUT -> `prefixPath` / "management" / "banks" / bankId / "retention-runs" / runId / "approval" => + EndpointHelpers.withUserAndBody[PutRetentionRunApprovalJsonV700, RetentionRunJsonV700](req) { (user, body, cc) => + val decision = Option(body.decision).map(_.trim.toLowerCase).getOrElse("") + for { + _ <- Helper.booleanToFuture(InvalidRetentionRunApproval, 400, Some(cc)) { + decision == DecisionApprove || decision == DecisionReject + } + run <- runFound(bankId, runId, cc) + _ <- Helper.booleanToFuture(s"$RetentionRunNotPending Current status is ${run.status}.", 400, Some(cc))(run.status == StatusPending) + _ <- Helper.booleanToFuture(RetentionRunApproverIsRequester, 403, Some(cc))(run.requestedByUserId != user.userId) + policy = run.retentionPolicyId.flatMap(id => + RetentionPolicy.find(By(RetentionPolicy.BankId, bankId), By(RetentionPolicy.RetentionPolicyId, id)).toOption) + _ <- Helper.booleanToFuture(RetentionRunAlreadyInProgress, 409, Some(cc)) { + decision == DecisionReject || policy.forall(p => !RetentionEngine.isRunning(p.retentionPolicyId)) + } + decided <- Future(RetentionEngine.decide(run, policy, user.userId, decision == DecisionApprove, + body.comment.map(_.trim).filter(_.nonEmpty))) + json <- if (decision == DecisionReject) Future.successful(JSONFactory700Retention.createRunJson(decided)) + else answerRun(decided, cc) + } yield json + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(updateRetentionRunApproval), + "PUT", + "/management/banks/BANK_ID/retention-runs/RETENTION_RUN_ID/approval", + "Update Retention Run Approval", + s"""Approve or reject a pending run: `{"decision": "approve"}` or `{"decision": "reject"}`, with an optional + |`comment`. The approver must be a different user from the one who requested the run (403 otherwise). A run the + |retention scheduler requested may be approved by any user holding the Role. + | + |An approved run executes now and the response shows its outcome. It is refused, with 400, if the props have + |switched its action off since it was requested, or if its policy has been deleted. A rejected run ends as + |`refused` with `approval_decision` `rejected`. + | + |$runDescription""".stripMargin, + JSONFactory700Retention.putRetentionRunApprovalJsonV700Example, + JSONFactory700Retention.retentionRunJsonV700Example, + List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidJsonFormat, InvalidRetentionRunApproval, + RetentionRunNotFound, RetentionRunNotPending, RetentionRunApproverIsRequester, RetentionRunAlreadyInProgress, + RetentionArchiveIsDisabled, RetentionDeleteIsDisabled, UnknownError), + tags, + Some(canApproveRetentionRun :: Nil), + http4sPartialFunction = Some(updateRetentionRunApproval) + ).allowSystemSpace() + + // Route: GET /obp/v7.0.0/management/banks/BANK_ID/retention-runs/RETENTION_RUN_ID/items + lazy val getRetentionRunItems: Http4sRoute = Http4sRoute { + case req @ GET -> `prefixPath` / "management" / "banks" / bankId / "retention-runs" / runId / "items" => + EndpointHelpers.withUser(req) { (_, cc) => + for { + pg <- page(req, cc) + run <- runFound(bankId, runId, cc) + items <- Future(RetentionRunItem.findAll(By(RetentionRunItem.RetentionRunId, run.retentionRunId), + OrderBy(RetentionRunItem.id, Ascending), StartAt[RetentionRunItem](pg.offset), MaxRows[RetentionRunItem](pg.limit))) + } yield RetentionRunItemsJsonV700(items.map(JSONFactory700Retention.createRunItemJson)) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(getRetentionRunItems), + "GET", + "/management/banks/BANK_ID/retention-runs/RETENTION_RUN_ID/items", + "Get Retention Run Items", + s"""The disposal register of one run: one item per resource it archived, anonymised or deleted, and per archived + |resource it deleted from the archive (`delete_from_archive`), in the order they were disposed of. Identifiers + |only, never content, so the register outlives the resources. A cascaded file, or a message of a disposed chat + |room, names its parent in `parent_resource_type` and `parent_resource_id`. Optional query parameters: `limit` + |(1 to $MaxPageSize, default $DefaultPageSize) and `offset` (default 0). + | + |BANK_ID is the space: a bank's id, or `SYS` for the system space. The Role is checked at that BANK_ID.""".stripMargin, + EmptyBody, + JSONFactory700Retention.retentionRunItemsJsonV700Example, + List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidRetentionQueryParameter, RetentionRunNotFound, UnknownError), + tags, + Some(canGetRetentionRuns :: Nil), + http4sPartialFunction = Some(getRetentionRunItems) + ).allowSystemSpace() + + private val holdDescription = + s"""A legal hold keeps resources out of every retention run until it is released: they are not archived, + |anonymised or deleted, and an archived resource on hold is not deleted from the archive. A hold names one + |resource by `resource_type` and `resource_id` (for a dynamic entity record, `entity_name` may be given too), or, + |for `dynamic_entity_record`, every record of one dynamic entity by `entity_name` alone. A consent is named by its + |`consent_reference_id`. A chat room with a held message is kept whole. + | + |BANK_ID is the space: a bank's id, or `SYS` for the system space. The Role is checked at that BANK_ID.""".stripMargin + + // Route: POST /obp/v7.0.0/management/banks/BANK_ID/retention-holds (201) + lazy val createRetentionHold: Http4sRoute = Http4sRoute { + case req @ POST -> `prefixPath` / "management" / "banks" / bankId / "retention-holds" => + EndpointHelpers.withUserAndBodyCreated[PostRetentionHoldJsonV700, RetentionHoldJsonV700](req) { (user, body, cc) => + val resourceType = Option(body.resource_type).map(_.trim).getOrElse("") + val resourceId = body.resource_id.map(_.trim).filter(_.nonEmpty) + val entityName = body.entity_name.map(_.trim).filter(_.nonEmpty) + val reason = Option(body.reason).map(_.trim).getOrElse("") + val problem = + if (!resourceTypes.contains(resourceType)) Some(s"resource_type must be one of ${resourceTypes.mkString(", ")}.") + else if (entityName.isDefined && resourceType != ResourceDynamicEntityRecord) Some("entity_name is only allowed for dynamic_entity_record.") + else if (resourceId.isEmpty && entityName.isEmpty) Some("resource_id is required, or, for dynamic_entity_record, entity_name.") + else if (reason.isEmpty) Some("reason is required.") + else if (reason.length > MaxReasonLength) Some(s"reason must be at most $MaxReasonLength characters.") + else None + for { + _ <- check(problem, InvalidRetentionHold, cc) + hold <- Future(RetentionHold.create + .RetentionHoldId(generateUUID()) + .BankId(bankId) + .ResourceType(resourceType) + .ResourceId(resourceId.getOrElse("")) + .EntityName(entityName.getOrElse("")) + .Reason(reason) + .PlacedByUserId(user.userId) + .PlacedAt(new Date()) + .saveMe()) + } yield JSONFactory700Retention.createHoldJson(hold) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(createRetentionHold), + "POST", + "/management/banks/BANK_ID/retention-holds", + "Create Retention Hold", + s"""Place a legal hold. `reason` is required. + | + |$holdDescription""".stripMargin, + JSONFactory700Retention.postRetentionHoldJsonV700Example, + JSONFactory700Retention.retentionHoldJsonV700Example, + List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidJsonFormat, InvalidRetentionHold, UnknownError), + tags, + Some(canManageRetentionHold :: Nil), + http4sPartialFunction = Some(createRetentionHold) + ).allowSystemSpace() + + // Route: GET /obp/v7.0.0/management/banks/BANK_ID/retention-holds + lazy val getRetentionHolds: Http4sRoute = Http4sRoute { + case req @ GET -> `prefixPath` / "management" / "banks" / bankId / "retention-holds" => + EndpointHelpers.withUser(req) { (_, cc) => + val p = params(req) + for { + pg <- page(req, cc) + _ <- Helper.booleanToFuture(s"${InvalidRetentionQueryParameter}is_active must be true or false.", 400, Some(cc)) { + p.get("is_active").forall(v => v == "true" || v == "false") + } + holds <- Future(RetentionHold.findAll( + (List[QueryParam[RetentionHold]](By(RetentionHold.BankId, bankId)) ::: + p.get("resource_type").map(t => By(RetentionHold.ResourceType, t)).toList ::: + p.get("resource_id").map(id => By(RetentionHold.ResourceId, id)).toList ::: + p.get("is_active").map(v => if (v == "true") NullRef(RetentionHold.ReleasedAt) else NotNullRef(RetentionHold.ReleasedAt)).toList ::: + List(OrderBy(RetentionHold.id, Descending), StartAt[RetentionHold](pg.offset), MaxRows[RetentionHold](pg.limit))): _*)) + } yield RetentionHoldsJsonV700(holds.map(JSONFactory700Retention.createHoldJson)) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(getRetentionHolds), + "GET", + "/management/banks/BANK_ID/retention-holds", + "Get Retention Holds", + s"""The legal holds of a space, newest first, released ones included. Optional query parameters: `is_active` + |(true or false), `resource_type`, `resource_id`, `limit` (1 to $MaxPageSize, default $DefaultPageSize) and + |`offset` (default 0). + | + |$holdDescription""".stripMargin, + EmptyBody, + JSONFactory700Retention.retentionHoldsJsonV700Example, + List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidRetentionQueryParameter, UnknownError), + tags, + Some(canManageRetentionHold :: canGetRetentionPolicies :: Nil), + http4sPartialFunction = Some(getRetentionHolds) + ).allowSystemSpace() + + // Route: DELETE /obp/v7.0.0/management/banks/BANK_ID/retention-holds/RETENTION_HOLD_ID (204) + lazy val deleteRetentionHold: Http4sRoute = Http4sRoute { + case req @ DELETE -> `prefixPath` / "management" / "banks" / bankId / "retention-holds" / holdId if holdId.nonEmpty => + EndpointHelpers.withUserDelete(req) { (user, cc) => + for { + hold <- holdFound(bankId, holdId, cc) + _ <- Helper.booleanToFuture(RetentionHoldAlreadyReleased, 400, Some(cc))(hold.isActive) + _ <- Future(hold.ReleasedByUserId(user.userId).ReleasedAt(new Date()).saveMe()) + } yield () + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(deleteRetentionHold), + "DELETE", + "/management/banks/BANK_ID/retention-holds/RETENTION_HOLD_ID", + "Delete Retention Hold", + s"""Release a legal hold. The hold is kept, with who released it and when, and is listed by Get Retention Holds + |with `is_active` false; the next run no longer skips what it named. + | + |$holdDescription""".stripMargin, + EmptyBody, + EmptyBody, + List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, RetentionHoldNotFound, RetentionHoldAlreadyReleased, UnknownError), + tags, + Some(canManageRetentionHold :: Nil), + http4sPartialFunction = Some(deleteRetentionHold) + ).allowSystemSpace() + + private val archiveDescription = + s"""The retention archive holds what `archive` policies moved out of the live tables: a read-only JSON copy of each + |resource (and a file's bytes), until `delete_after`, when it is deleted. Nothing in it is reachable through any + |other endpoint, and nothing is ever restored from it. A cascaded file names the record it went with in + |`parent_resource_type` and `parent_resource_id`; a chat room's messages are inside the room's copy. + | + |BANK_ID is the space: a bank's id, or `SYS` for the system space. The Role is checked at that BANK_ID.""".stripMargin + + // Route: GET /obp/v7.0.0/management/banks/BANK_ID/retention-archive + lazy val getRetentionArchiveEntries: Http4sRoute = Http4sRoute { + case req @ GET -> `prefixPath` / "management" / "banks" / bankId / "retention-archive" => + EndpointHelpers.withUser(req) { (_, cc) => + val p = params(req) + for { + pg <- page(req, cc) + entries <- Future(RetentionArchive.findAll( + (List[QueryParam[RetentionArchive]](By(RetentionArchive.BankId, bankId)) ::: + p.get("resource_type").map(v => By(RetentionArchive.ResourceType, v)).toList ::: + p.get("resource_id").map(v => By(RetentionArchive.ResourceId, v)).toList ::: + p.get("entity_name").map(v => By(RetentionArchive.EntityName, v)).toList ::: + p.get("owner_user_id").map(v => By(RetentionArchive.OwnerUserId, v)).toList ::: + p.get("retention_policy_id").map(v => By(RetentionArchive.RetentionPolicyId, v)).toList ::: + p.get("retention_run_id").map(v => By(RetentionArchive.RetentionRunId, v)).toList ::: + List(OrderBy(RetentionArchive.id, Descending), StartAt[RetentionArchive](pg.offset), MaxRows[RetentionArchive](pg.limit))): _*)) + } yield RetentionArchiveEntriesJsonV700(entries.map(JSONFactory700Retention.createArchiveEntryJson(_, withData = false, None))) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(getRetentionArchiveEntries), + "GET", + "/management/banks/BANK_ID/retention-archive", + "Get Retention Archive Entries", + s"""The entries of a space's retention archive, newest first, without their content. Optional query parameters, + |each matching exactly: `resource_type`, `resource_id`, `entity_name`, `owner_user_id`, `retention_policy_id`, + |`retention_run_id`; and `limit` (1 to $MaxPageSize, default $DefaultPageSize) and `offset` (default 0). + | + |$archiveDescription""".stripMargin, + EmptyBody, + JSONFactory700Retention.retentionArchiveEntriesJsonV700Example, + List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidRetentionQueryParameter, UnknownError), + tags, + Some(canGetRetentionArchive :: Nil), + http4sPartialFunction = Some(getRetentionArchiveEntries) + ).allowSystemSpace() + + // Route: GET /obp/v7.0.0/management/banks/BANK_ID/retention-archive/RETENTION_ARCHIVE_ID + lazy val getRetentionArchiveEntry: Http4sRoute = Http4sRoute { + case req @ GET -> `prefixPath` / "management" / "banks" / bankId / "retention-archive" / archiveId if archiveId.nonEmpty => + EndpointHelpers.withUser(req) { (_, cc) => + val includeContent = params(req).get("include_content") + for { + _ <- Helper.booleanToFuture(s"${InvalidRetentionQueryParameter}include_content must be true or false.", 400, Some(cc)) { + includeContent.forall(v => v == "true" || v == "false") + } + entry <- archiveEntryFound(bankId, archiveId, cc) + content <- Future( + if (includeContent.contains("true") && entry.hasContent) + RetentionArchiveContent.find(By(RetentionArchiveContent.RetentionArchiveId, entry.retentionArchiveId)) + .toOption.flatMap(_.content) + else None) + } yield JSONFactory700Retention.createArchiveEntryJson(entry, withData = true, content) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(getRetentionArchiveEntry), + "GET", + "/management/banks/BANK_ID/retention-archive/RETENTION_ARCHIVE_ID", + "Get Retention Archive Entry", + s"""One entry of a space's retention archive, with its JSON copy in `data`. With `include_content=true`, a file's + |bytes are given too, Base64 encoded, in `content_base64`. + | + |$archiveDescription""".stripMargin, + EmptyBody, + JSONFactory700Retention.retentionArchiveEntryJsonV700Example, + List($BankNotFound, $AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidRetentionQueryParameter, + RetentionArchiveEntryNotFound, UnknownError), + tags, + Some(canGetRetentionArchive :: Nil), + http4sPartialFunction = Some(getRetentionArchiveEntry) + ).allowSystemSpace() +} diff --git a/obp-api/src/main/scala/code/api/v7_0_0/Http4s700Telemetry.scala b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700Telemetry.scala index 72abd401e9..dc9f42e52b 100644 --- a/obp-api/src/main/scala/code/api/v7_0_0/Http4s700Telemetry.scala +++ b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700Telemetry.scala @@ -28,7 +28,7 @@ package code.api.v7_0_0 import cats.effect.IO import code.api.Constant.ApiPathZero -import code.api.util.APIUtil.{EmptyBody, ResourceDoc, UserOrApplication, Http4sRoute} +import code.api.util.APIUtil.{EmptyBody, ResourceDoc, UserXORApplication, Http4sRoute} import code.api.util.ApiRole._ import code.api.util.ApiTag._ import code.api.util.ErrorMessages._ @@ -119,7 +119,7 @@ object Http4s700Telemetry { List($AuthenticatedUserIsRequired, UserHasMissingRoles, UnknownError), List(apiTagApi, apiTagSystem), Some(List(canGetTelemetry)), - authMode = UserOrApplication, + authMode = UserXORApplication, http4sPartialFunction = Some(getTelemetry) ) } diff --git a/obp-api/src/main/scala/code/api/v7_0_0/Http4s700Tenants.scala b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700Tenants.scala new file mode 100644 index 0000000000..b2c5b2e644 --- /dev/null +++ b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700Tenants.scala @@ -0,0 +1,253 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ +package code.api.v7_0_0 + +import cats.effect.IO +import code.api.Constant.ApiPathZero +import code.api.util.APIUtil.{EmptyBody, Http4sRoute, ResourceDoc, unboxFullOrFail} +import code.api.util.ApiRole.{canCreateTenant, canDeleteTenant, canGetTenant, canUpdateTenant} +import code.api.util.ApiTag._ +import code.api.util.ErrorMessages._ +import code.api.util.http4s.Http4sRequestAttributes.EndpointHelpers +import code.api.util.{APIUtil, CallContext, CustomJsonFormats, NewStyle} +import code.tenant.{TenantTrait, Tenants} +import code.util.Helper +import com.github.dwickern.macros.NameOf.nameOf +import com.openbankproject.commons.ExecutionContext.Implicits.global +import com.openbankproject.commons.util.ApiVersion +import org.http4s._ +import org.http4s.dsl.io._ +import org.json4s.Formats + +import scala.collection.mutable.ArrayBuffer +import scala.concurrent.Future + +/** + * This object holds the v7.0.0 Tenant endpoints. A Tenant is a customer of the operator of this OBP instance: an + * organisation that uses it. Tenants belong to no bank, so their Roles are system Roles; later, banks will name the + * Tenant they belong to. + * + * It is declared in its own object to keep Http4s700's initialiser under the JVM's 64KB method limit. + */ +object Http4s700Tenants { + + implicit val formats: Formats = CustomJsonFormats.formats + + private val implementedInApiVersion = ApiVersion.v7_0_0 + private val prefixPath = Root / ApiPathZero.toString / implementedInApiVersion.toString + + val resourceDocs = ArrayBuffer[ResourceDoc]() + + private val ValidTenantStatuses = Set("active", "suspended", "archived") + private val ValidTenantVisibilities = Set("public", "unlisted", "private") + private val TenantIdRegex = "^[a-zA-Z0-9._-]{2,64}$".r + + private val tenantDescription = + """A Tenant is a customer of the operator of this OBP instance: an organisation that uses it. Tenants belong to no + |bank, and their Roles are system Roles. The tenant_id is unique and immutable.""".stripMargin + + private def tenantOf(tenantId: String, cc: CallContext): Future[TenantTrait] = + Future(Tenants.tenant.vend.getTenant(tenantId)).map(unboxFullOrFail(_, Some(cc), TenantNotFound, 404)) + + // Route: POST /obp/v7.0.0/tenants (201) + lazy val createTenant: Http4sRoute = Http4sRoute { + case req @ POST -> `prefixPath` / "tenants" => + EndpointHelpers.withUserAndBodyCreated[PostTenantJsonV700, TenantJsonV700](req) { (user, body, cc) => + val status = body.status.getOrElse("active") + val visibility = body.visibility.getOrElse("public") + for { + _ <- Helper.booleanToFuture(InvalidTenantIdFormat, 400, Some(cc))(TenantIdRegex.findFirstIn(body.tenant_id).isDefined) + _ <- Helper.booleanToFuture(InvalidTenantStatus, 400, Some(cc))(ValidTenantStatuses.contains(status)) + _ <- Helper.booleanToFuture(InvalidTenantVisibility, 400, Some(cc))(ValidTenantVisibilities.contains(visibility)) + existing <- Future(Tenants.tenant.vend.getTenant(body.tenant_id)) + _ <- Helper.booleanToFuture(TenantAlreadyExists, 409, Some(cc))(existing.isEmpty) + created <- Future { + Tenants.tenant.vend.createTenant(body.tenant_id, body.name, body.website, body.logo_url, status, visibility, user.userId) + }.map(unboxFullOrFail(_, Some(cc), CreateTenantError, 400)) + } yield JSONFactory700Tenants.createTenantJsonV700(created) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(createTenant), + "POST", + "/tenants", + "Create Tenant", + s"""Create a Tenant. + | + |$tenantDescription + | + |The tenant_id must be a URL-safe string (a-z, A-Z, 0-9, '-', '.', '_'), between 2 and 64 characters in length. + | + |Optional fields: + |- status: one of active, suspended, archived (defaults to active) + |- visibility: one of public, unlisted, private (defaults to public) + |- website, logo_url + | + |Authentication is Required.""".stripMargin, + PostTenantJsonV700("tesobe", "TESOBE GmbH", Some("https://www.tesobe.com"), None, Some("active"), Some("public")), + JSONFactory700Tenants.tenantJsonV700Example, + List($AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidJsonFormat, InvalidTenantIdFormat, InvalidTenantStatus, + InvalidTenantVisibility, TenantAlreadyExists, CreateTenantError, UnknownError), + apiTagTenant :: Nil, + Some(List(canCreateTenant)), + http4sPartialFunction = Some(createTenant) + ) + + // Route: GET /obp/v7.0.0/tenants + lazy val getTenants: Http4sRoute = Http4sRoute { + case req @ GET -> `prefixPath` / "tenants" => + EndpointHelpers.withUser(req) { (user, cc) => + for { + tenants <- Tenants.tenant.vend.getTenants().map(unboxFullOrFail(_, Some(cc), UnknownError, 500)) + hasGetRole = APIUtil.hasAtLeastOneEntitlement("", user.userId, List(canGetTenant)) + } yield JSONFactory700Tenants.createTenantsJsonV700(if (hasGetRole) tenants else tenants.filter(_.visibility == "public")) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(getTenants), + "GET", + "/tenants", + "Get Tenants", + s"""Returns the Tenants. + | + |$tenantDescription + | + |By default returns only Tenants whose visibility is `public`. Users with the Role CanGetTenant see all of them, + |including unlisted and private ones. + | + |Authentication is Required.""".stripMargin, + EmptyBody, + TenantsJsonV700(List(JSONFactory700Tenants.tenantJsonV700Example)), + List($AuthenticatedUserIsRequired, UnknownError), + apiTagTenant :: Nil, + None, + http4sPartialFunction = Some(getTenants) + ) + + // Route: GET /obp/v7.0.0/tenants/TENANT_ID + lazy val getTenant: Http4sRoute = Http4sRoute { + case req @ GET -> `prefixPath` / "tenants" / tenantId => + EndpointHelpers.withUser(req) { (user, cc) => + for { + tenant <- tenantOf(tenantId, cc) + _ <- if (tenant.visibility == "private") NewStyle.function.hasEntitlement("", user.userId, canGetTenant, Some(cc)) + else Future.successful(()) + } yield JSONFactory700Tenants.createTenantJsonV700(tenant) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(getTenant), + "GET", + "/tenants/TENANT_ID", + "Get Tenant", + s"""Returns the Tenant TENANT_ID. + | + |$tenantDescription + | + |Tenants with visibility `public` or `unlisted` are visible to any authenticated user. Tenants with visibility + |`private` need the Role CanGetTenant. + | + |Authentication is Required.""".stripMargin, + EmptyBody, + JSONFactory700Tenants.tenantJsonV700Example, + List($AuthenticatedUserIsRequired, UserHasMissingRoles, TenantNotFound, UnknownError), + apiTagTenant :: Nil, + None, + http4sPartialFunction = Some(getTenant) + ) + + // Route: PUT /obp/v7.0.0/tenants/TENANT_ID + lazy val updateTenant: Http4sRoute = Http4sRoute { + case req @ PUT -> `prefixPath` / "tenants" / tenantId => + EndpointHelpers.withUserAndBody[PutTenantJsonV700, TenantJsonV700](req) { (_, body, cc) => + for { + _ <- tenantOf(tenantId, cc) + _ <- Helper.booleanToFuture(InvalidTenantStatus, 400, Some(cc))(body.status.forall(ValidTenantStatuses.contains)) + _ <- Helper.booleanToFuture(InvalidTenantVisibility, 400, Some(cc))(body.visibility.forall(ValidTenantVisibilities.contains)) + updated <- Future { + Tenants.tenant.vend.updateTenant(tenantId, body.name, body.website, body.logo_url, body.status, body.visibility) + }.map(unboxFullOrFail(_, Some(cc), UpdateTenantError, 400)) + } yield JSONFactory700Tenants.createTenantJsonV700(updated) + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(updateTenant), + "PUT", + "/tenants/TENANT_ID", + "Update Tenant", + s"""Update the Tenant TENANT_ID. All body fields are optional; the tenant_id is immutable. + | + |$tenantDescription + | + |Authentication is Required.""".stripMargin, + PutTenantJsonV700(Some("TESOBE GmbH"), Some("https://www.tesobe.com"), None, Some("active"), Some("public")), + JSONFactory700Tenants.tenantJsonV700Example, + List($AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidJsonFormat, TenantNotFound, InvalidTenantStatus, + InvalidTenantVisibility, UpdateTenantError, UnknownError), + apiTagTenant :: Nil, + Some(List(canUpdateTenant)), + http4sPartialFunction = Some(updateTenant) + ) + + // Route: DELETE /obp/v7.0.0/tenants/TENANT_ID (204) + lazy val deleteTenant: Http4sRoute = Http4sRoute { + case req @ DELETE -> `prefixPath` / "tenants" / tenantId => + EndpointHelpers.withUserDelete(req) { (_, cc) => + for { + _ <- tenantOf(tenantId, cc) + _ <- Future(Tenants.tenant.vend.deleteTenant(tenantId)).map(unboxFullOrFail(_, Some(cc), DeleteTenantError, 400)) + } yield () + } + } + + resourceDocs += ResourceDoc( + implementedInApiVersion, + nameOf(deleteTenant), + "DELETE", + "/tenants/TENANT_ID", + "Delete Tenant", + s"""Delete the Tenant TENANT_ID. + | + |$tenantDescription + | + |Authentication is Required.""".stripMargin, + EmptyBody, + EmptyBody, + List($AuthenticatedUserIsRequired, UserHasMissingRoles, TenantNotFound, DeleteTenantError, UnknownError), + apiTagTenant :: Nil, + Some(List(canDeleteTenant)), + http4sPartialFunction = Some(deleteTenant) + ) +} diff --git a/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory7.0.0.scala b/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory7.0.0.scala index accf19115f..3ef3c925ce 100644 --- a/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory7.0.0.scala +++ b/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory7.0.0.scala @@ -790,55 +790,6 @@ object JSONFactory700 extends MdcLoggable with code.api.util.CustomJsonFormats { abac_trace: AbacEvaluationTraceJsonV700 ) - // Organisation JSON case classes - case class PostOrganisationJsonV700( - organisation_id: String, - name: String, - website: Option[String], - logo_url: Option[String], - status: Option[String], - visibility: Option[String] - ) - - case class PutOrganisationJsonV700( - name: Option[String], - website: Option[String], - logo_url: Option[String], - status: Option[String], - visibility: Option[String] - ) - - case class OrganisationJsonV700( - organisation_id: String, - name: String, - website: Option[String], - logo_url: Option[String], - status: String, - visibility: String, - created_by_user_id: String, - created_at: java.util.Date, - updated_at: java.util.Date - ) - - case class OrganisationsJsonV700(organisations: List[OrganisationJsonV700]) - - def createOrganisationJsonV700(o: code.organisation.OrganisationTrait): OrganisationJsonV700 = { - OrganisationJsonV700( - organisation_id = o.organisationId, - name = o.name, - website = o.website, - logo_url = o.logoUrl, - status = o.status, - visibility = o.visibility, - created_by_user_id = o.createdByUserId, - created_at = o.createdAt, - updated_at = o.updatedAt - ) - } - - def createOrganisationsJsonV700(orgs: List[code.organisation.OrganisationTrait]): OrganisationsJsonV700 = { - OrganisationsJsonV700(orgs.map(createOrganisationJsonV700)) - } // ── Routing Scheme JSON case classes ───────────────────────────────────────── @@ -1545,13 +1496,27 @@ object JSONFactory700 extends MdcLoggable with code.api.util.CustomJsonFormats { balance: AmountOfMoneyJsonV121, branch_id: String, account_routings: List[AccountRoutingJsonV121], - account_attributes: List[AccountAttributeResponseJson] + account_attributes: List[AccountAttributeResponseJson], + metadata: AccountMetadataJsonV700 ) + /** Who created the account and for whom, as in a Dynamic Entity record's metadata: `user_id` made + * the call (an agent's own id under a Consent), `on_behalf_of_user_id` is the person it was for. */ + case class AccountCreatedJsonV700(at: String, user_id: String, on_behalf_of_user_id: String, consent_reference_id: Option[String]) + case class AccountMetadataJsonV700(created: AccountCreatedJsonV700) + + def accountMetadataJsonV700(creation: code.accountcreation.AccountCreationTrait): AccountMetadataJsonV700 = + AccountMetadataJsonV700(AccountCreatedJsonV700( + at = java.time.format.DateTimeFormatter.ISO_INSTANT.format(creation.createdAt.toInstant.truncatedTo(java.time.temporal.ChronoUnit.SECONDS)), + user_id = creation.createdByUserId, + on_behalf_of_user_id = creation.createdByOnBehalfOfUserId, + consent_reference_id = creation.consentReferenceId)) + def createAccountJsonV700( userId: String, account: BankAccount, - accountAttributes: List[AccountAttribute] + accountAttributes: List[AccountAttribute], + creation: code.accountcreation.AccountCreationTrait ): CreateAccountResponseJsonV700 = CreateAccountResponseJsonV700( account_id = account.accountId.value, @@ -1565,9 +1530,50 @@ object JSONFactory700 extends MdcLoggable with code.api.util.CustomJsonFormats { account.accountId.value, account.accountRoutings.map(r => AccountRoutingJsonV121(r.scheme, r.address)) ), - account_attributes = accountAttributes.map(JSONFactory310.createAccountAttributeJson) + account_attributes = accountAttributes.map(JSONFactory310.createAccountAttributeJson), + metadata = accountMetadataJsonV700(creation) ) + /** One account in the v7.0.0 Account Directory: the v6.0.0 fields plus `metadata.created`, which + * is present only for an account created through v7.0.0 Create Account (see AccountCreations). */ + case class AccountDirectoryItemJsonV700( + account_id: String, + bank_id: String, + label: String, + account_number: String, + account_type: String, + branch_id: String, + account_routings: List[AccountRoutingJsonV121], + account_attributes: List[com.openbankproject.commons.model.FastFirehoseAttributes], + view_ids: List[String], + metadata: Option[AccountMetadataJsonV700] + ) + + case class AccountDirectoryJsonV700(accounts: List[AccountDirectoryItemJsonV700]) + + def createAccountDirectoryJsonV700( + accounts: List[com.openbankproject.commons.model.AccountDirectoryItem], + viewsPerAccount: Map[BankIdAccountId, List[String]], + creationPerAccountId: Map[String, code.accountcreation.AccountCreationTrait] + ): AccountDirectoryJsonV700 = + AccountDirectoryJsonV700(accounts.map { account => + AccountDirectoryItemJsonV700( + account_id = account.id, + bank_id = account.bankId, + label = account.label, + account_number = account.number, + account_type = account.productCode, + branch_id = account.branchId, + account_routings = Constant.accountRoutingsWithImplicitOBP( + account.id, + account.accountRoutings.map(r => AccountRoutingJsonV121(r.scheme, r.address)) + ), + account_attributes = account.accountAttributes, + view_ids = viewsPerAccount.getOrElse(BankIdAccountId(BankId(account.bankId), AccountId(account.id)), Nil), + metadata = creationPerAccountId.get(account.id).map(accountMetadataJsonV700) + ) + }) + // ─── OPEN_CORRIDOR per-bank broker registry (admin) ──────────────────────── // Transport coordinates only. The settlement address is NOT part of the broker @@ -1703,7 +1709,13 @@ object JSONFactory700 extends MdcLoggable with code.api.util.CustomJsonFormats { settlement_status: String, settlement_depth: Option[Int], covered_transaction_request_ids: List[String], - messages: List[OpenCorridorSettlementMessageJsonV700] + messages: List[OpenCorridorSettlementMessageJsonV700], + // What the debtor bank's node reported paying on the settlement rail, from its last reply to the + // settlement instruction: the rail transaction id, the asset and the amount in the asset's smallest + // unit (lovelace for ADA). The creditor bank checks its incoming payment against these. None until reported. + settlement_tx_id: Option[String] = None, + settlement_asset: Option[String] = None, + settlement_asset_amount: Option[String] = None ) // Build the originator block for a TR response. Returns None when there's no diff --git a/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory700Assets.scala b/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory700Assets.scala index c4bed7d434..ec46467eb6 100644 --- a/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory700Assets.scala +++ b/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory700Assets.scala @@ -55,6 +55,21 @@ case class AssetJsonV700( case class AssetsPaginationJsonV700(total: Int, limit: Int, offset: Int) +/** + * The body of Create Asset. The issuer is the bank in the path, so it is not in the body; the status of + * a new asset is always ACTIVE. + */ +case class PostAssetJsonV700( + asset_code: String, + asset_type: String, + name: String, + decimal_places: Int, + issuer_product_code: Option[String], + chain_scheme: Option[String], + chain_asset_id: Option[String], + dti: Option[String] +) + case class AssetsJsonV700(assets: List[AssetJsonV700], pagination: AssetsPaginationJsonV700) object JSONFactory700Assets { @@ -116,6 +131,18 @@ object JSONFactory700Assets { updated_at = new Date() ) + /** A bank's bond token, the body Create Asset takes at that bank. */ + val postAssetJsonExample: PostAssetJsonV700 = PostAssetJsonV700( + asset_code = "TZBOND29", + asset_type = "DEBT_SECURITY", + name = "Example Bank 2029 Fixed Rate Note", + decimal_places = 0, + issuer_product_code = Some("TZBOND29"), + chain_scheme = Some("CARDANO_MAINNET"), + chain_asset_id = Some("f0ff48bbb7bbe9d59a40f1ce90e9e9d0ff5002ec48f232b49ca0fb9a.545a424f4e443239"), + dti = None + ) + val assetsJsonExample: AssetsJsonV700 = AssetsJsonV700(List(assetJsonExample), AssetsPaginationJsonV700(total = 1, limit = 500, offset = 0)) } diff --git a/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory700CustomerEmails.scala b/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory700CustomerEmails.scala new file mode 100644 index 0000000000..46400f6690 --- /dev/null +++ b/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory700CustomerEmails.scala @@ -0,0 +1,70 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ +package code.api.v7_0_0 + +import java.util.Date + +import code.messageoutbox.MessageOutbox + +/** The body of Send Email to Customer: plain text only. */ +case class PostCustomerEmailJsonV700( + subject: String, + text: String, + reference: Option[String] +) + +/** A queued email, as Send Email to Customer returns it. The address is never returned. */ +case class CustomerEmailJsonV700( + message_id: String, + bank_id: String, + customer_id: String, + subject: String, + reference: Option[String], + status: String, + created_at: Date +) + +object JSONFactory700CustomerEmails { + val postCustomerEmailJsonExample: PostCustomerEmailJsonV700 = PostCustomerEmailJsonV700( + subject = "Your application was received", + text = "Dear applicant,\n\nThank you: your application to Cohort 1 was received on 20 November.", + reference = Some("cohort_application:6fb17583-1e49-4435-bb74-a14fe0996723") + ) + + val customerEmailJsonExample: CustomerEmailJsonV700 = CustomerEmailJsonV700( + message_id = "1042", + bank_id = "cohort", + customer_id = "7uy8a7e4-6d02-40e3-a129-0b2bf89de9f0", + subject = "Your application was received", + reference = Some("cohort_application:6fb17583-1e49-4435-bb74-a14fe0996723"), + status = MessageOutbox.STATUS_PENDING, + created_at = new Date(1795000000000L) + ) + + def createCustomerEmailJson(row: MessageOutbox, bankId: String, customerId: String, subject: String, reference: Option[String]): CustomerEmailJsonV700 = + CustomerEmailJsonV700(row.id.get.toString, bankId, customerId, subject, reference, row.status, row.CreatedAt.get) +} diff --git a/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory700Files.scala b/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory700Files.scala new file mode 100644 index 0000000000..74aed51754 --- /dev/null +++ b/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory700Files.scala @@ -0,0 +1,290 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ +package code.api.v7_0_0 + +import java.util.Date + +import code.files.{FileAccessT, FileAttachmentT, FileT} + +/* + * The JSON of the v7.0.0 file endpoints. Package-level case classes, for the reason given in + * JSONFactory700Operations. + */ + +/** One stored file: what it is and who owns it. Never the bytes, which Get File Content returns. */ +case class FileJsonV700( + file_id: String, + bank_id: String, + file_name: String, + media_type: String, + size_in_bytes: Long, + sha256: String, + is_public: Boolean, + owner_user_id: String, + user_id: String, + on_behalf_of_user_id: String, + created_at: Date, + is_personal: Boolean, + scan_status: String, + scan_status_reason: String, + scan_status_updated_at: Option[Date] +) + +case class FilesJsonV700(files: List[FileJsonV700]) + +case class FilesPaginationJsonV700(total: Long, limit: Int, offset: Int) + +/** One page of all the files of a space, for administrators. */ +case class FilesPageJsonV700(files: List[FileJsonV700], pagination: FilesPaginationJsonV700) + +/** + * A public file as anyone may see it, without logging in: what it is, but not who uploaded it. + */ +case class PublicFileJsonV700( + file_id: String, + bank_id: String, + file_name: String, + media_type: String, + size_in_bytes: Long, + sha256: String, + created_at: Date +) + +case class PublicFilesJsonV700(files: List[PublicFileJsonV700]) + +/** + * The body of Create File. The bytes travel base64 encoded. sha256, when given, is checked against the + * decoded bytes; OBP computes and stores its own either way. + */ +case class PostFileJsonV700( + file_name: String, + media_type: String, + content_base64: String, + is_public: Option[Boolean], + sha256: Option[String] +) + +case class PutFileIsPublicJsonV700(is_public: Boolean) + +/** + * The body of Create My File: a personal file, uploaded without a Role, attached at once to one of the + * owner's own personal Dynamic Entity records (record_type is the entity's name, record_id the record's id). + */ +case class PostMyFileJsonV700( + file_name: String, + media_type: String, + content_base64: String, + sha256: Option[String], + record_type: String, + record_id: String +) + +case class PutFileScanStatusJsonV700(scan_status: String, reason: Option[String]) + +case class PostFileAttachmentJsonV700(record_type: String, record_id: String) + +case class FileAttachmentJsonV700( + file_attachment_id: String, + file_id: String, + bank_id: String, + record_type: String, + record_id: String, + user_id: String, + on_behalf_of_user_id: String, + created_at: Date +) + +case class FileAttachmentsJsonV700(attachments: List[FileAttachmentJsonV700]) + +case class PostFileAccessJsonV700(user_id: String) + +case class FileAccessJsonV700( + file_id: String, + user_id: String, + granted_by_user_id: String, + on_behalf_of_user_id: String, + created_at: Date +) + +case class FileAccessListJsonV700(access: List[FileAccessJsonV700]) + +object JSONFactory700Files { + + def createFileJson(file: FileT): FileJsonV700 = + FileJsonV700( + file_id = file.fileId, + bank_id = file.bankId, + file_name = file.fileName, + media_type = file.mediaType, + size_in_bytes = file.sizeInBytes, + sha256 = file.sha256, + is_public = file.isPublic, + owner_user_id = file.ownerUserId, + user_id = file.userId, + on_behalf_of_user_id = file.onBehalfOfUserId, + created_at = file.createdAt, + is_personal = file.isPersonal, + scan_status = file.scanStatus, + scan_status_reason = file.scanStatusReason, + scan_status_updated_at = file.scanStatusUpdatedAt + ) + + def createFilesJson(files: List[FileT]): FilesJsonV700 = FilesJsonV700(files.map(createFileJson)) + + def createFilesPageJson(files: List[FileT], total: Long, limit: Int, offset: Int): FilesPageJsonV700 = + FilesPageJsonV700(files.map(createFileJson), FilesPaginationJsonV700(total, limit, offset)) + + def createPublicFileJson(file: FileT): PublicFileJsonV700 = + PublicFileJsonV700( + file_id = file.fileId, + bank_id = file.bankId, + file_name = file.fileName, + media_type = file.mediaType, + size_in_bytes = file.sizeInBytes, + sha256 = file.sha256, + created_at = file.createdAt + ) + + def createPublicFilesJson(files: List[FileT]): PublicFilesJsonV700 = PublicFilesJsonV700(files.map(createPublicFileJson)) + + def createFileAttachmentJson(attachment: FileAttachmentT): FileAttachmentJsonV700 = + FileAttachmentJsonV700( + file_attachment_id = attachment.fileAttachmentId, + file_id = attachment.fileId, + bank_id = attachment.bankId, + record_type = attachment.recordType, + record_id = attachment.recordId, + user_id = attachment.userId, + on_behalf_of_user_id = attachment.onBehalfOfUserId, + created_at = attachment.createdAt + ) + + def createFileAttachmentsJson(attachments: List[FileAttachmentT]): FileAttachmentsJsonV700 = + FileAttachmentsJsonV700(attachments.map(createFileAttachmentJson)) + + def createFileAccessJson(access: FileAccessT): FileAccessJsonV700 = + FileAccessJsonV700( + file_id = access.fileId, + user_id = access.granteeUserId, + granted_by_user_id = access.grantedByUserId, + on_behalf_of_user_id = access.onBehalfOfUserId, + created_at = access.createdAt + ) + + def createFileAccessListJson(access: List[FileAccessT]): FileAccessListJsonV700 = + FileAccessListJsonV700(access.map(createFileAccessJson)) + + private val exampleDate = new Date(1792195200000L) // 2026-10-17T00:00:00Z + + val fileJsonExample: FileJsonV700 = FileJsonV700( + file_id = "b3a2a1e4-5c6d-4e7f-8a9b-0c1d2e3f4a5b", + bank_id = "gh.29.uk", + file_name = "site_survey.pdf", + media_type = "application/pdf", + size_in_bytes = 482113L, + sha256 = "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", + is_public = false, + owner_user_id = "9ca9a7e4-6d02-40e3-a129-0b2bf89de9b1", + user_id = "9ca9a7e4-6d02-40e3-a129-0b2bf89de9b1", + on_behalf_of_user_id = "", + created_at = exampleDate, + is_personal = false, + scan_status = "clean", + scan_status_reason = "", + scan_status_updated_at = None + ) + + val filesJsonExample: FilesJsonV700 = FilesJsonV700(List(fileJsonExample)) + + val filesPageJsonExample: FilesPageJsonV700 = + FilesPageJsonV700(List(fileJsonExample), FilesPaginationJsonV700(total = 1, limit = 500, offset = 0)) + + val publicFileJsonExample: PublicFileJsonV700 = PublicFileJsonV700( + file_id = "c4b3b2f5-6d7e-4f80-9bac-1d2e3f4a5b6c", + bank_id = "gh.29.uk", + file_name = "site_photo.jpg", + media_type = "image/jpeg", + size_in_bytes = 1048576L, + sha256 = "2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae", + created_at = exampleDate + ) + + val publicFilesJsonExample: PublicFilesJsonV700 = PublicFilesJsonV700(List(publicFileJsonExample)) + + val postFileJsonExample: PostFileJsonV700 = PostFileJsonV700( + file_name = "site_survey.pdf", + media_type = "application/pdf", + content_base64 = "JVBERi0xLjcKJcfsj6IK", + is_public = Some(false), + sha256 = Some("9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08") + ) + + val putFileIsPublicJsonExample: PutFileIsPublicJsonV700 = PutFileIsPublicJsonV700(is_public = true) + + val postMyFileJsonExample: PostMyFileJsonV700 = PostMyFileJsonV700( + file_name = "pitch_deck.pdf", + media_type = "application/pdf", + content_base64 = "JVBERi0xLjcKJcfsj6IK", + sha256 = None, + record_type = "cohort_application", + record_id = "6fb17583-1e49-4435-bb74-a14fe0996723" + ) + + lazy val myFileJsonExample: FileJsonV700 = fileJsonExample.copy( + file_name = "pitch_deck.pdf", is_personal = true, scan_status = "pending") + + val putFileScanStatusJsonExample: PutFileScanStatusJsonV700 = + PutFileScanStatusJsonV700(scan_status = "clean", reason = Some("ClamAV 1.4: no threats found")) + + val postFileAttachmentJsonExample: PostFileAttachmentJsonV700 = + PostFileAttachmentJsonV700(record_type = "customer", record_id = "7uy8a7e4-6d02-40e3-a129-0b2bf89de8uh") + + val fileAttachmentJsonExample: FileAttachmentJsonV700 = FileAttachmentJsonV700( + file_attachment_id = "d5c4c3a6-7e8f-4091-acbd-2e3f4a5b6c7d", + file_id = fileJsonExample.file_id, + bank_id = "gh.29.uk", + record_type = "customer", + record_id = "7uy8a7e4-6d02-40e3-a129-0b2bf89de8uh", + user_id = "9ca9a7e4-6d02-40e3-a129-0b2bf89de9b1", + on_behalf_of_user_id = "", + created_at = exampleDate + ) + + val fileAttachmentsJsonExample: FileAttachmentsJsonV700 = FileAttachmentsJsonV700(List(fileAttachmentJsonExample)) + + val postFileAccessJsonExample: PostFileAccessJsonV700 = PostFileAccessJsonV700(user_id = "6f5e4d3c-2b1a-4098-8765-43210fedcba9") + + val fileAccessJsonExample: FileAccessJsonV700 = FileAccessJsonV700( + file_id = fileJsonExample.file_id, + user_id = "6f5e4d3c-2b1a-4098-8765-43210fedcba9", + granted_by_user_id = "9ca9a7e4-6d02-40e3-a129-0b2bf89de9b1", + on_behalf_of_user_id = "", + created_at = exampleDate + ) + + val fileAccessListJsonExample: FileAccessListJsonV700 = FileAccessListJsonV700(List(fileAccessJsonExample)) +} diff --git a/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory700Retention.scala b/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory700Retention.scala new file mode 100644 index 0000000000..65cda11eb4 --- /dev/null +++ b/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory700Retention.scala @@ -0,0 +1,517 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ +package code.api.v7_0_0 + +import java.util.{Base64, Date} + +import code.retention._ +import com.openbankproject.commons.util.JsonAliases +import org.json4s.JsonAST.JObject +import org.json4s.JsonDSL._ +import org.json4s.{JNothing, JValue} + +import scala.util.Try + +/* + * The JSON of the v7.0.0 data retention endpoints. Package-level case classes, for the reason given in + * JSONFactory700Operations. + */ + +/** A condition on a resource: its field `field` has the value `value`, compared as text. */ +case class RetentionConditionJsonV700(field: String, value: String) + +/** + * Create a retention policy. Only `resource_type`, `retain_days` and `action` are required. + * `entity_name` is required for, and only allowed with, `dynamic_entity_record`. `archive_days` is + * required for, and only allowed with, the `archive` action. + */ +case class PostRetentionPolicyJsonV700( + name: Option[String] = None, + description: Option[String] = None, + resource_type: String, + entity_name: Option[String] = None, + date_basis: Option[String] = None, + condition: Option[RetentionConditionJsonV700] = None, + retain_days: Int, + archive_days: Option[Int] = None, + action: String, + cascade: Option[Boolean] = None, + requires_approval: Option[Boolean] = None, + enabled: Option[Boolean] = None +) + +/** + * Change a retention policy. Every field is optional and a field left out keeps its value. The + * resource type and entity name cannot change. A condition whose field is empty removes the + * condition. `archive_days` is cleared when the action becomes anything but `archive`. + */ +case class PutRetentionPolicyJsonV700( + name: Option[String] = None, + description: Option[String] = None, + date_basis: Option[String] = None, + condition: Option[RetentionConditionJsonV700] = None, + retain_days: Option[Int] = None, + archive_days: Option[Int] = None, + action: Option[String] = None, + cascade: Option[Boolean] = None, + requires_approval: Option[Boolean] = None, + enabled: Option[Boolean] = None +) + +case class RetentionPolicyJsonV700( + retention_policy_id: String, + bank_id: String, + name: String, + description: String, + resource_type: String, + entity_name: Option[String], + date_basis: String, + condition: Option[RetentionConditionJsonV700], + retain_days: Int, + archive_days: Option[Int], + action: String, + cascade: Boolean, + requires_approval: Boolean, + enabled: Boolean, + created_by_user_id: String, + created_at: Date, + updated_at: Date +) + +case class RetentionPoliciesJsonV700(retention_policies: List[RetentionPolicyJsonV700]) + +/** What a run of the policy would dispose of now, and whether this instance would let it. */ +case class RetentionPolicyPreviewJsonV700( + retention_policy_id: String, + bank_id: String, + resource_type: String, + entity_name: Option[String], + action: String, + /** A resource whose trigger date is on or before this date is due. */ + cutoff_date: Date, + matched_count: Int, + held_count: Int, + /** matched_count less held_count, though one run disposes of at most max_items_per_run of them. */ + due_count: Int, + oldest_trigger_date: Option[Date], + sample_resource_ids: List[String], + /** Resources this policy archived whose archive period has ended, so the next run deletes them. */ + archive_entries_due_count: Int, + count_is_capped: Boolean, + max_items_per_run: Int, + /** retention.minimum_retain_days: a run never disposes of a resource sooner than this after its trigger date. */ + minimum_retain_days: Int, + archive_enabled: Boolean, + delete_enabled: Boolean, + /** Whether an execution of this policy would be allowed by the props now. */ + can_execute: Boolean, + requires_approval: Boolean +) + +/** Run a policy now. `mode` is execute (the default) or preview. */ +case class PostRetentionRunJsonV700(mode: Option[String] = None) + +case class RetentionRunJsonV700( + retention_run_id: String, + /** Left out for a run of the scheduler that deleted archived resources across policies. */ + retention_policy_id: Option[String], + bank_id: String, + mode: String, + status: String, + /** Left out when the scheduler requested the run. */ + requested_by_user_id: Option[String], + approved_by_user_id: Option[String], + approval_decision: Option[String], + approval_comment: Option[String], + matched_count: Int, + archived_count: Int, + anonymised_count: Int, + deleted_count: Int, + held_count: Int, + failed_count: Int, + requested_at: Date, + started_at: Option[Date], + ended_at: Option[Date], + message: String +) + +case class RetentionRunsJsonV700(retention_runs: List[RetentionRunJsonV700]) + +/** Approve or reject a pending run: `decision` is approve or reject. */ +case class PutRetentionRunApprovalJsonV700(decision: String, comment: Option[String] = None) + +/** One row of the disposal register. Identifiers only, never content. */ +case class RetentionRunItemJsonV700( + retention_run_item_id: String, + retention_run_id: String, + retention_policy_id: Option[String], + bank_id: String, + resource_type: String, + resource_id: String, + entity_name: Option[String], + owner_user_id: Option[String], + action: String, + parent_resource_type: Option[String], + parent_resource_id: Option[String], + retention_archive_id: Option[String], + disposed_at: Date +) + +case class RetentionRunItemsJsonV700(retention_run_items: List[RetentionRunItemJsonV700]) + +/** + * Place a legal hold. Give `resource_id` to hold one resource (with `entity_name` too for a dynamic + * entity record, optionally), or, for `dynamic_entity_record`, `entity_name` alone to hold every + * record of that dynamic entity. + */ +case class PostRetentionHoldJsonV700( + resource_type: String, + resource_id: Option[String] = None, + entity_name: Option[String] = None, + reason: String +) + +case class RetentionHoldJsonV700( + retention_hold_id: String, + bank_id: String, + resource_type: String, + resource_id: Option[String], + entity_name: Option[String], + reason: String, + placed_by_user_id: String, + placed_at: Date, + released_by_user_id: Option[String], + released_at: Option[Date], + is_active: Boolean +) + +case class RetentionHoldsJsonV700(retention_holds: List[RetentionHoldJsonV700]) + +/** + * One archived resource. `data` (the JSON copy) and `content_base64` (a file's bytes) are given only + * by Get Retention Archive Entry, the latter only with `include_content=true`. + */ +case class RetentionArchiveEntryJsonV700( + retention_archive_id: String, + retention_policy_id: Option[String], + retention_run_id: String, + bank_id: String, + resource_type: String, + resource_id: String, + entity_name: Option[String], + owner_user_id: Option[String], + parent_resource_type: Option[String], + parent_resource_id: Option[String], + archived_at: Date, + delete_after: Option[Date], + has_content: Boolean, + content_size_in_bytes: Option[Long], + data: Option[JValue] = None, + content_base64: Option[String] = None +) + +case class RetentionArchiveEntriesJsonV700(retention_archive_entries: List[RetentionArchiveEntryJsonV700]) + +object JSONFactory700Retention { + + def createPolicyJson(p: RetentionPolicy): RetentionPolicyJsonV700 = RetentionPolicyJsonV700( + retention_policy_id = p.retentionPolicyId, + bank_id = p.bankId, + name = p.name, + description = p.description, + resource_type = p.resourceType, + entity_name = p.entityName, + date_basis = p.dateBasis, + condition = p.condition.map(c => RetentionConditionJsonV700(c.field, c.value)), + retain_days = p.retainDays, + archive_days = p.archiveDays, + action = p.action, + cascade = p.cascade, + requires_approval = p.requiresApproval, + enabled = p.enabled, + created_by_user_id = p.createdByUserId, + created_at = p.createdAt.get, + updated_at = p.updatedAt.get + ) + + def createPreviewJson(p: RetentionPolicy, r: RetentionEngine.PreviewResult): RetentionPolicyPreviewJsonV700 = + RetentionPolicyPreviewJsonV700( + retention_policy_id = p.retentionPolicyId, + bank_id = p.bankId, + resource_type = p.resourceType, + entity_name = p.entityName, + action = p.action, + cutoff_date = r.cutoff, + matched_count = r.matched, + held_count = r.held, + due_count = r.matched - r.held, + oldest_trigger_date = r.oldestTriggerDate, + sample_resource_ids = r.sampleResourceIds, + archive_entries_due_count = r.archiveEntriesDue, + count_is_capped = r.scanLimitReached, + max_items_per_run = RetentionProps.maxItemsPerRun, + minimum_retain_days = RetentionProps.minimumRetainDays, + archive_enabled = RetentionProps.archiveEnabled, + delete_enabled = RetentionProps.deleteEnabled, + can_execute = RetentionEngine.refusal(p).isEmpty, + requires_approval = p.requiresApproval + ) + + def createRunJson(r: RetentionRun): RetentionRunJsonV700 = RetentionRunJsonV700( + retention_run_id = r.retentionRunId, + retention_policy_id = r.retentionPolicyId, + bank_id = r.bankId, + mode = r.mode, + status = r.status, + requested_by_user_id = Option(r.requestedByUserId).filter(_.nonEmpty), + approved_by_user_id = r.approvedByUserId, + approval_decision = r.approvalDecision, + approval_comment = r.approvalComment, + matched_count = r.MatchedCount.get, + archived_count = r.ArchivedCount.get, + anonymised_count = r.AnonymisedCount.get, + deleted_count = r.DeletedCount.get, + held_count = r.HeldCount.get, + failed_count = r.FailedCount.get, + requested_at = r.requestedAt, + started_at = r.startedAt, + ended_at = r.endedAt, + message = r.message + ) + + def createRunItemJson(i: RetentionRunItem): RetentionRunItemJsonV700 = RetentionRunItemJsonV700( + retention_run_item_id = i.retentionRunItemId, + retention_run_id = i.retentionRunId, + retention_policy_id = i.retentionPolicyId, + bank_id = i.bankId, + resource_type = i.resourceType, + resource_id = i.resourceId, + entity_name = i.entityName, + owner_user_id = i.ownerUserId, + action = i.action, + parent_resource_type = i.parentResourceType, + parent_resource_id = i.parentResourceId, + retention_archive_id = i.retentionArchiveId, + disposed_at = i.disposedAt + ) + + def createHoldJson(h: RetentionHold): RetentionHoldJsonV700 = RetentionHoldJsonV700( + retention_hold_id = h.retentionHoldId, + bank_id = h.bankId, + resource_type = h.resourceType, + resource_id = h.resourceId, + entity_name = h.entityName, + reason = h.reason, + placed_by_user_id = h.placedByUserId, + placed_at = h.placedAt, + released_by_user_id = h.releasedByUserId, + released_at = h.releasedAt, + is_active = h.isActive + ) + + def createArchiveEntryJson(a: RetentionArchive, withData: Boolean, content: Option[Array[Byte]]): RetentionArchiveEntryJsonV700 = + RetentionArchiveEntryJsonV700( + retention_archive_id = a.retentionArchiveId, + retention_policy_id = a.retentionPolicyId, + retention_run_id = a.retentionRunId, + bank_id = a.bankId, + resource_type = a.resourceType, + resource_id = a.resourceId, + entity_name = a.entityName, + owner_user_id = a.ownerUserId, + parent_resource_type = a.parentResourceType, + parent_resource_id = a.parentResourceId, + archived_at = a.archivedAt, + delete_after = a.deleteAfter, + has_content = a.hasContent, + content_size_in_bytes = if (a.hasContent) Some(a.contentSizeInBytes) else None, + data = if (withData) Try(JsonAliases.parse(a.jsonCopy)).toOption.filterNot(_ == JNothing) else None, + content_base64 = content.map(bytes => Base64.getEncoder.encodeToString(bytes)) + ) + + // ---- examples for the ResourceDocs. Neutral values only. + + private val exampleDate = new Date(1767225600000L) // 2026-01-01T00:00:00Z + private val examplePolicyId = "6a1f0e2c-3b8d-4c5e-9f7a-1d2e3f4a5b6c" + private val exampleRunId = "0b9c8d7e-6f5a-4b3c-2d1e-0f9e8d7c6b5a" + private val exampleUserId = "9ca9a7e4-6d02-40e3-a129-0b2bf89de9b1" + private val exampleOtherUserId = "4f3e2d1c-0b9a-4876-9543-210fedcba987" + + val postRetentionPolicyJsonV700Example = PostRetentionPolicyJsonV700( + name = Some("Close applications after two years"), + description = Some("Applications are kept for two years after they were last changed, then archived for five more."), + resource_type = Retention.ResourceDynamicEntityRecord, + entity_name = Some("application"), + date_basis = Some(Retention.BasisUpdated), + condition = Some(RetentionConditionJsonV700("status", "CLOSED")), + retain_days = 2555, + archive_days = Some(1825), + action = Retention.ActionArchive, + cascade = Some(true), + requires_approval = Some(true), + enabled = Some(true) + ) + + val putRetentionPolicyJsonV700Example = PutRetentionPolicyJsonV700( + retain_days = Some(3650), + requires_approval = Some(true) + ) + + val retentionPolicyJsonV700Example = RetentionPolicyJsonV700( + retention_policy_id = examplePolicyId, + bank_id = "SYS", + name = "Close applications after two years", + description = "Applications are kept for two years after they were last changed, then archived for five more.", + resource_type = Retention.ResourceDynamicEntityRecord, + entity_name = Some("application"), + date_basis = Retention.BasisUpdated, + condition = Some(RetentionConditionJsonV700("status", "CLOSED")), + retain_days = 2555, + archive_days = Some(1825), + action = Retention.ActionArchive, + cascade = true, + requires_approval = true, + enabled = true, + created_by_user_id = exampleUserId, + created_at = exampleDate, + updated_at = exampleDate + ) + + val retentionPoliciesJsonV700Example = RetentionPoliciesJsonV700(List(retentionPolicyJsonV700Example)) + + val retentionPolicyPreviewJsonV700Example = RetentionPolicyPreviewJsonV700( + retention_policy_id = examplePolicyId, + bank_id = "SYS", + resource_type = Retention.ResourceDynamicEntityRecord, + entity_name = Some("application"), + action = Retention.ActionArchive, + cutoff_date = exampleDate, + matched_count = 42, + held_count = 2, + due_count = 40, + oldest_trigger_date = Some(new Date(1704067200000L)), + sample_resource_ids = List("3d2c1b0a-9f8e-4d7c-6b5a-4f3e2d1c0b9a"), + archive_entries_due_count = 0, + count_is_capped = false, + max_items_per_run = RetentionProps.MaxItemsPerRunDefault, + minimum_retain_days = RetentionProps.MinimumRetainDaysDefault, + archive_enabled = true, + delete_enabled = false, + can_execute = true, + requires_approval = true + ) + + val postRetentionRunJsonV700Example = PostRetentionRunJsonV700(mode = Some(Retention.ModeExecute)) + + val retentionRunJsonV700Example = RetentionRunJsonV700( + retention_run_id = exampleRunId, + retention_policy_id = Some(examplePolicyId), + bank_id = "SYS", + mode = Retention.ModeExecute, + status = Retention.StatusDone, + requested_by_user_id = Some(exampleUserId), + approved_by_user_id = Some(exampleOtherUserId), + approval_decision = Some(Retention.DecisionApproved), + approval_comment = Some("Checked against the preview."), + matched_count = 42, + archived_count = 43, + anonymised_count = 0, + deleted_count = 0, + held_count = 2, + failed_count = 0, + requested_at = exampleDate, + started_at = Some(exampleDate), + ended_at = Some(exampleDate), + message = "Done." + ) + + val retentionRunsJsonV700Example = RetentionRunsJsonV700(List(retentionRunJsonV700Example)) + + val putRetentionRunApprovalJsonV700Example = PutRetentionRunApprovalJsonV700(Retention.DecisionApprove, Some("Checked against the preview.")) + + val retentionRunItemsJsonV700Example = RetentionRunItemsJsonV700(List(RetentionRunItemJsonV700( + retention_run_item_id = "7e6d5c4b-3a29-4180-9f8e-7d6c5b4a3928", + retention_run_id = exampleRunId, + retention_policy_id = Some(examplePolicyId), + bank_id = "SYS", + resource_type = Retention.ResourceFile, + resource_id = "1a2b3c4d-5e6f-4a8b-9c0d-1e2f3a4b5c6d", + entity_name = None, + owner_user_id = Some(exampleUserId), + action = Retention.ActionArchive, + parent_resource_type = Some(Retention.ResourceDynamicEntityRecord), + parent_resource_id = Some("3d2c1b0a-9f8e-4d7c-6b5a-4f3e2d1c0b9a"), + retention_archive_id = Some("5c4b3a29-1807-4f6e-8d5c-4b3a29180706"), + disposed_at = exampleDate + ))) + + val postRetentionHoldJsonV700Example = PostRetentionHoldJsonV700( + resource_type = Retention.ResourceDynamicEntityRecord, + resource_id = Some("3d2c1b0a-9f8e-4d7c-6b5a-4f3e2d1c0b9a"), + entity_name = Some("application"), + reason = "Requested by the records officer while a complaint is open." + ) + + val retentionHoldJsonV700Example = RetentionHoldJsonV700( + retention_hold_id = "2b1a0f9e-8d7c-4b6a-a594-837261504f3e", + bank_id = "SYS", + resource_type = Retention.ResourceDynamicEntityRecord, + resource_id = Some("3d2c1b0a-9f8e-4d7c-6b5a-4f3e2d1c0b9a"), + entity_name = Some("application"), + reason = "Requested by the records officer while a complaint is open.", + placed_by_user_id = exampleUserId, + placed_at = exampleDate, + released_by_user_id = None, + released_at = None, + is_active = true + ) + + val retentionHoldsJsonV700Example = RetentionHoldsJsonV700(List(retentionHoldJsonV700Example)) + + val retentionArchiveEntryJsonV700Example = RetentionArchiveEntryJsonV700( + retention_archive_id = "5c4b3a29-1807-4f6e-8d5c-4b3a29180706", + retention_policy_id = Some(examplePolicyId), + retention_run_id = exampleRunId, + bank_id = "SYS", + resource_type = Retention.ResourceDynamicEntityRecord, + resource_id = "3d2c1b0a-9f8e-4d7c-6b5a-4f3e2d1c0b9a", + entity_name = Some("application"), + owner_user_id = Some(exampleUserId), + parent_resource_type = None, + parent_resource_id = None, + archived_at = exampleDate, + delete_after = Some(new Date(1924992000000L)), + has_content = false, + content_size_in_bytes = None, + data = Some(("dynamic_data_id" -> "3d2c1b0a-9f8e-4d7c-6b5a-4f3e2d1c0b9a") ~ ("entity_name" -> "application") ~ + ("data" -> (("status" -> "CLOSED") ~ ("title" -> "Example application")))), + content_base64 = None + ) + + val retentionArchiveEntriesJsonV700Example = RetentionArchiveEntriesJsonV700( + List(retentionArchiveEntryJsonV700Example.copy(data = None))) +} diff --git a/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory700Tenants.scala b/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory700Tenants.scala new file mode 100644 index 0000000000..e1fe1201f3 --- /dev/null +++ b/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory700Tenants.scala @@ -0,0 +1,90 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ +package code.api.v7_0_0 + +import code.tenant.TenantTrait + +case class PostTenantJsonV700( + tenant_id: String, + name: String, + website: Option[String], + logo_url: Option[String], + status: Option[String], + visibility: Option[String] +) + +case class PutTenantJsonV700( + name: Option[String], + website: Option[String], + logo_url: Option[String], + status: Option[String], + visibility: Option[String] +) + +case class TenantJsonV700( + tenant_id: String, + name: String, + website: Option[String], + logo_url: Option[String], + status: String, + visibility: String, + created_by_user_id: String, + created_at: java.util.Date, + updated_at: java.util.Date +) + +case class TenantsJsonV700(tenants: List[TenantJsonV700]) + +object JSONFactory700Tenants { + lazy val tenantJsonV700Example: TenantJsonV700 = TenantJsonV700( + tenant_id = "tesobe", + name = "TESOBE GmbH", + website = Some("https://www.tesobe.com"), + logo_url = None, + status = "active", + visibility = "public", + created_by_user_id = "9ca9a7e4-6d02-40e3-a129-0b2bf89de9b1", + created_at = new java.util.Date(1795000000000L), + updated_at = new java.util.Date(1795000000000L) + ) + + def createTenantJsonV700(t: TenantTrait): TenantJsonV700 = + TenantJsonV700( + tenant_id = t.tenantId, + name = t.name, + website = t.website, + logo_url = t.logoUrl, + status = t.status, + visibility = t.visibility, + created_by_user_id = t.createdByUserId, + created_at = t.createdAt, + updated_at = t.updatedAt + ) + + def createTenantsJsonV700(tenants: List[TenantTrait]): TenantsJsonV700 = + TenantsJsonV700(tenants.map(createTenantJsonV700)) +} diff --git a/obp-api/src/main/scala/code/asset/Asset.scala b/obp-api/src/main/scala/code/asset/Asset.scala index d10f66fbf9..ba8c1e33ac 100644 --- a/obp-api/src/main/scala/code/asset/Asset.scala +++ b/obp-api/src/main/scala/code/asset/Asset.scala @@ -85,7 +85,7 @@ class Asset extends AssetTrait with LongKeyedMapper[Asset] with IdPK { /** * The unique indexes on (chain_scheme, chain_asset_id) and on dti that the design describes are not * declared here: they must ignore rows where the value is not set, and Mapper cannot declare such a - * partial index. Nothing writes those columns yet; the write endpoints will enforce the uniqueness. + * partial index. The Create Asset endpoint checks both before it writes, and answers 409 when one is taken. */ object Asset extends Asset with LongKeyedMetaMapper[Asset] { override def dbTableName = "Asset" diff --git a/obp-api/src/main/scala/code/asset/Assets.scala b/obp-api/src/main/scala/code/asset/Assets.scala index 76cf399e64..2ff9f14bb3 100644 --- a/obp-api/src/main/scala/code/asset/Assets.scala +++ b/obp-api/src/main/scala/code/asset/Assets.scala @@ -80,6 +80,22 @@ object AssetStatuses { val all: Set[String] = Set(ACTIVE, SUSPENDED, RETIRED) } +/** + * This object lists the chains and networks an asset can be recorded on. Each value names the network + * as well as the chain, because the same token identity can be a different token on another network, + * and OBP cannot tell which network a node it talks to is on. The list grows when OBP supports + * another network. + */ +object ChainSchemes { + val CARDANO_MAINNET = "CARDANO_MAINNET" + val CARDANO_PREPROD = "CARDANO_PREPROD" + val CARDANO_PREVIEW = "CARDANO_PREVIEW" + val ETHEREUM_MAINNET = "ETHEREUM_MAINNET" + val ETHEREUM_SEPOLIA = "ETHEREUM_SEPOLIA" + + val all: Set[String] = Set(CARDANO_MAINNET, CARDANO_PREPROD, CARDANO_PREVIEW, ETHEREUM_MAINNET, ETHEREUM_SEPOLIA) +} + /** This trait is the set of operations on the asset registry. */ trait AssetProvider { /** Finds an asset by its code, ignoring case: `eur` finds `EUR`. */ @@ -99,6 +115,8 @@ trait AssetProvider { ): (List[AssetTrait], Int) /** Finds the asset with this on-chain identity, for example a Cardano token's policy id and asset name. */ def getAssetByChainIdentity(chainScheme: String, chainAssetId: String): Box[AssetTrait] + /** Finds the asset with this ISO 24165 Digital Token Identifier. */ + def getAssetByDti(dti: String): Box[AssetTrait] def createAsset( assetCode: String, assetType: String, @@ -106,7 +124,11 @@ trait AssetProvider { decimalPlaces: Int, issuerBankId: Option[String], status: String, - createdByUserId: String + createdByUserId: String, + issuerProductCode: Option[String] = None, + chainScheme: Option[String] = None, + chainAssetId: Option[String] = None, + dti: Option[String] = None ): Box[AssetTrait] } @@ -120,6 +142,18 @@ object Assets extends AssetProvider { /** Codes are 3 to 10 upper case letters or digits once upper-cased. */ private val AssetCodePattern = "^[A-Z0-9]{3,10}$".r + /** + * A bank-issued asset's code is 4 to 10 characters, so it can never collide with a current or + * future ISO 4217 code, which is always 3. + */ + private val IssuedAssetCodePattern = "^[A-Z0-9]{4,10}$".r + + /** A Digital Token Identifier is 9 upper case letters or digits. */ + private val DtiPattern = "^[A-Z0-9]{9}$".r + + /** The longest name the Name column holds. */ + val MaxNameLength = 125 + /** * Amounts will be stored as DECIMAL(38, 18) (ideas/ASSET_REGISTRY.md, section 5), so no asset can * use more than 18 decimal places. @@ -156,45 +190,116 @@ object Assets extends AssetProvider { if (chainScheme.isEmpty || chainAssetId.isEmpty) Failure("A chain identity needs both a chain scheme and a chain asset id.") else Asset.find(By(Asset.ChainScheme, chainScheme), By(Asset.ChainAssetId, chainAssetId)) - override def createAsset( + override def getAssetByDti(dti: String): Box[AssetTrait] = + if (dti.isEmpty) Failure("A DTI must not be empty.") + else Asset.find(By(Asset.Dti, dti)) + + /** Treats a missing, empty or blank optional value as not set. */ + private def given(value: Option[String]): Option[String] = value.map(_.trim).filter(_.nonEmpty) + + /** + * The first rule of the registry that this asset breaks, or None when it breaks none. createAsset + * checks it before writing, and the Create Asset endpoint checks it before looking for a taken code, + * chain identity or DTI, so an asset that could never be created is a 400, never a 409. + */ + def whyInvalid( assetCode: String, assetType: String, name: String, decimalPlaces: Int, issuerBankId: Option[String], status: String, - createdByUserId: String - ): Box[AssetTrait] = { + issuerProductCode: Option[String], + chainScheme: Option[String], + chainAssetId: Option[String], + dti: Option[String] + ): Option[String] = { val code = normaliseCode(assetCode) - val issuer = issuerBankId.map(_.trim).filter(_.nonEmpty) + val issuer = given(issuerBankId) + val productCode = given(issuerProductCode) + val scheme = given(chainScheme).map(_.toUpperCase) + val chainId = given(chainAssetId) + val tokenIdentifier = given(dti).map(_.toUpperCase) if (AssetCodePattern.findFirstIn(code).isEmpty) - Failure(s"Asset code '$assetCode' must be 3 to 10 letters or digits.") + Some(s"Asset code '$assetCode' must be 3 to 10 letters or digits.") + else if (AssetTypes.issued.contains(assetType) && IssuedAssetCodePattern.findFirstIn(code).isEmpty) + Some(s"Asset code '$assetCode' of a bank-issued asset must be 4 to 10 letters or digits, so it cannot be mistaken for an ISO 4217 currency code.") + else if (Option(name).forall(_.trim.isEmpty) || name.trim.length > MaxNameLength) + Some(s"Name must not be empty and must be at most $MaxNameLength characters.") else if (!AssetTypes.all.contains(assetType)) - Failure(s"Asset type '$assetType' must be one of ${AssetTypes.all.toList.sorted.mkString(", ")}.") + Some(s"Asset type '$assetType' must be one of ${AssetTypes.all.toList.sorted.mkString(", ")}.") else if (decimalPlaces < 0 || decimalPlaces > MaxDecimalPlaces) - Failure(s"Decimal places $decimalPlaces must be between 0 and $MaxDecimalPlaces.") + Some(s"Decimal places $decimalPlaces must be between 0 and $MaxDecimalPlaces.") else if (AssetTypes.issued.contains(assetType) && issuer.isEmpty) - Failure(s"An asset of type $assetType must have an issuer bank.") + Some(s"An asset of type $assetType must have an issuer bank.") else if (AssetTypes.withoutIssuer.contains(assetType) && issuer.nonEmpty) - Failure(s"An asset of type $assetType has no issuer bank; it is administered at the SYS bank.") + Some(s"An asset of type $assetType has no issuer bank; it is administered at the SYS bank.") else if (!AssetStatuses.all.contains(status)) - Failure(s"Status '$status' must be one of ${AssetStatuses.all.toList.sorted.mkString(", ")}.") + Some(s"Status '$status' must be one of ${AssetStatuses.all.toList.sorted.mkString(", ")}.") + else if (productCode.nonEmpty && issuer.isEmpty) + Some("An issuer product code needs an issuer bank, where the product is.") + else if (scheme.isDefined != chainId.isDefined) + Some("A chain identity needs both a chain scheme and a chain asset id, or neither.") + else if (scheme.exists(value => !ChainSchemes.all.contains(value))) + Some(s"Chain scheme '${scheme.getOrElse("")}' must be one of ${ChainSchemes.all.toList.sorted.mkString(", ")}.") + else if (tokenIdentifier.exists(value => DtiPattern.findFirstIn(value).isEmpty)) + Some(s"DTI '${tokenIdentifier.getOrElse("")}' must be 9 letters or digits.") else - tryo { - Asset.create - .AssetCode(code) - .AssetType(assetType) - .Name(name) - .DecimalPlaces(decimalPlaces) - .IssuerBankId(issuer.getOrElse("")) - .Status(status) - .CreatedByUserId(createdByUserId) - .saveMe() - } match { - case created @ Full(_) => - AssetLookup.invalidate() - created - case other => other - } + None + } + + override def createAsset( + assetCode: String, + assetType: String, + name: String, + decimalPlaces: Int, + issuerBankId: Option[String], + status: String, + createdByUserId: String, + issuerProductCode: Option[String], + chainScheme: Option[String], + chainAssetId: Option[String], + dti: Option[String] + ): Box[AssetTrait] = { + val code = normaliseCode(assetCode) + val issuer = given(issuerBankId) + val productCode = given(issuerProductCode) + val scheme = given(chainScheme).map(_.toUpperCase) + val chainId = given(chainAssetId) + val tokenIdentifier = given(dti).map(_.toUpperCase) + whyInvalid(assetCode, assetType, name, decimalPlaces, issuerBankId, status, issuerProductCode, chainScheme, chainAssetId, dti) match { + case Some(rule) => Failure(rule) + case None => + tryo { + Asset.create + .AssetCode(code) + .AssetType(assetType) + .Name(name.trim) + .DecimalPlaces(decimalPlaces) + .IssuerBankId(issuer.getOrElse("")) + .IssuerProductCode(productCode.getOrElse("")) + .ChainScheme(scheme.getOrElse("")) + .ChainAssetId(chainId.getOrElse("")) + .Dti(tokenIdentifier.getOrElse("")) + .Status(status) + .CreatedByUserId(creatorToStore(createdByUserId)) + .saveMe() + } match { + case created @ Full(_) => + AssetLookup.invalidate() + created + case other => other + } + } } + + /** + * Returns the user id to store as the asset's creator. An asset outlives the Consent of a consent + * user that created it, so the creator is the person that user acts for. The boot seed is not a + * user and is stored as it is. + */ + private def creatorToStore(createdByUserId: String): String = + if (createdByUserId == AssetSeed.SeedActor) createdByUserId + else code.users.Users.users.vend.attributedUserId(createdByUserId, code.users.UserReference.Asset_CreatedByUserId) + .openOr(createdByUserId) } diff --git a/obp-api/src/main/scala/code/bankconnectors/LocalMappedConnector.scala b/obp-api/src/main/scala/code/bankconnectors/LocalMappedConnector.scala index 29e1ad1436..d8906171b8 100644 --- a/obp-api/src/main/scala/code/bankconnectors/LocalMappedConnector.scala +++ b/obp-api/src/main/scala/code/bankconnectors/LocalMappedConnector.scala @@ -5337,7 +5337,7 @@ object LocalMappedConnector extends Connector with MdcLoggable { TransactionRequests.transactionRequestProvider.vend.getTransactionRequest(transactionRequestId).map(transactionRequest => (transactionRequest, callContext)) override def getTransactionRequestTypes(initiator: User, fromAccount: BankAccount, callContext: Option[CallContext]):Box[(List[TransactionRequestType], Option[CallContext])] = { - Full((APIUtil.getPropsValue("transactionRequests_supported_types", "").split(",").map(x => TransactionRequestType(x)).toList, callContext)) + Full((APIUtil.advertisedTransactionRequestTypes.map(x => TransactionRequestType(x)), callContext)) } /** diff --git a/obp-api/src/main/scala/code/bankconnectors/opencorridor/OpenCorridorSettlement.scala b/obp-api/src/main/scala/code/bankconnectors/opencorridor/OpenCorridorSettlement.scala index de7c887952..bb990a3075 100644 --- a/obp-api/src/main/scala/code/bankconnectors/opencorridor/OpenCorridorSettlement.scala +++ b/obp-api/src/main/scala/code/bankconnectors/opencorridor/OpenCorridorSettlement.scala @@ -414,7 +414,10 @@ object OpenCorridorSettlement extends MdcLoggable { delivery_status = row.status, attempts = row.attempts, last_error = row.LastError.get - )) + )), + settlement_tx_id = instructionRow.flatMap(nodeReportedField(_, "tx_id")).filter(_.nonEmpty), + settlement_asset = instructionRow.flatMap(nodeReportedField(_, "asset")).filter(_.nonEmpty), + settlement_asset_amount = instructionRow.flatMap(nodeReportedField(_, "asset_amount")).filter(_.nonEmpty) ), callContext) } diff --git a/obp-api/src/main/scala/code/dynamicEntity/DynamicEntityProvider.scala b/obp-api/src/main/scala/code/dynamicEntity/DynamicEntityProvider.scala index 97323493c3..8c64217177 100644 --- a/obp-api/src/main/scala/code/dynamicEntity/DynamicEntityProvider.scala +++ b/obp-api/src/main/scala/code/dynamicEntity/DynamicEntityProvider.scala @@ -487,13 +487,19 @@ object ReferenceType extends MdcLoggable { object DynamicEntityAuthMode { val UserOnly = "UserOnly" val ApplicationOnly = "ApplicationOnly" - val UserOrApplication = "UserOrApplication" + val UserXORApplication = "UserXORApplication" val UserAndApplication = "UserAndApplication" - val all: List[String] = List(UserOnly, ApplicationOnly, UserOrApplication, UserAndApplication) + /** The former name of UserXORApplication, still accepted on input and in stored rows. */ + val LegacyUserOrApplication = "UserOrApplication" + val all: List[String] = List(UserOnly, ApplicationOnly, UserXORApplication, UserAndApplication) val default: String = UserOnly def isValid(value: String): Boolean = all.contains(value) - /** null / empty (rows created before the column existed) read as the default. */ - def normalise(value: String): String = Option(value).map(_.trim).filter(_.nonEmpty).getOrElse(default) + /** null / empty (rows created before the column existed) read as the default; the legacy name reads as UserXORApplication. */ + def normalise(value: String): String = Option(value).map(_.trim).filter(_.nonEmpty) match { + case Some(LegacyUserOrApplication) => UserXORApplication + case Some(v) => v + case None => default + } } case class DynamicEntityCommons(entityName: String, diff --git a/obp-api/src/main/scala/code/files/Files.scala b/obp-api/src/main/scala/code/files/Files.scala index c5d519e12e..01f671ac5f 100644 --- a/obp-api/src/main/scala/code/files/Files.scala +++ b/obp-api/src/main/scala/code/files/Files.scala @@ -34,7 +34,7 @@ import net.liftweb.util.SimpleInjector /** * This file holds the model for Files: documents such as PDFs and photographs that belong to a space * (a bank, or SYS for the system space) and can be attached to OBP records. See the Glossary item - * "Files" and ideas/ogcr_file_storage.md. + * "Files". * * A file's bytes never change once stored. A new version of a document is a new file. What can * change is which records the file is attached to and which Users have been given access to it. @@ -61,10 +61,33 @@ trait FileT { def userId: String /** The User an agent uploaded the file for. Empty when nobody was delegating. */ def onBehalfOfUserId: String + /** Anyone may read a public file, without logging in, through the public file endpoints. */ + def isPublic: Boolean def createdAt: Date /** The file's owner: the User an agent acted for, or else the User who uploaded it. */ def ownerUserId: String = if (onBehalfOfUserId.nonEmpty) onBehalfOfUserId else userId + /** Uploaded by a person through Create My File, within the personal files limits, rather than with a Role. */ + def isPersonal: Boolean + /** pending, clean or rejected (see [[ScanStatus]]). The content is served to everyone allowed only when clean. */ + def scanStatus: String + /** Why the status was set, e.g. the signature a scanner matched. Empty when no reason was given. */ + def scanStatusReason: String + def scanStatusUpdatedAt: Option[Date] + /** Who last set the status: the scanner's or the staff member's own id. Empty while nobody has. */ + def scanStatusUpdatedByUserId: String +} + +/** + * Whether a file's content may be served. A personal file starts pending and waits for a scanner or a member + * of staff; a file uploaded with a Role starts clean. Until a file is clean its content is served only to its + * owner and to holders of CanUpdateFileScanStatusAtOneBank; a rejected file is not served at all. + */ +object ScanStatus { + val Pending = "pending" + val Clean = "clean" + val Rejected = "rejected" + val all: List[String] = List(Pending, Clean, Rejected) } /** A link between a file and one record, such as a Customer or a Dynamic Entity record. */ @@ -102,21 +125,34 @@ trait FileStore { } trait FilesProvider { - /** Store the bytes and the metadata of a new file, together. */ - def createFile(bankId: String, sha256: String, mediaType: String, fileName: String, - userId: String, onBehalfOfUserId: String, bytes: Array[Byte]): Box[FileT] + /** Store the bytes and the metadata of a new file, together. userId is the caller; the provider records + * the User an agent acted for, by the attribution policy of File_UserId. */ + def createFile(bankId: String, sha256: String, mediaType: String, fileName: String, isPublic: Boolean, + userId: String, bytes: Array[Byte], isPersonal: Boolean = false, + scanStatus: String = ScanStatus.Clean): Box[FileT] def getFile(bankId: String, fileId: String): Box[FileT] def getContent(fileId: String): Box[Array[Byte]] - - def createAttachment(file: FileT, recordType: String, recordId: String, - userId: String, onBehalfOfUserId: String): Box[FileAttachmentT] + /** Whether the file is public is the one thing about it that can change; its bytes never do. */ + def setPublic(bankId: String, fileId: String, isPublic: Boolean): Box[FileT] + /** One page of the files of a space, newest first, and how many there are in all; only those with + * scanStatus when it is given (pending or rejected, which are always stored explicitly). */ + def getFiles(bankId: String, limit: Int, offset: Int, scanStatus: Option[String] = None): (List[FileT], Long) + /** Set the scan status. userId is the caller, recorded as it is (an audit of who decided). */ + def setScanStatus(bankId: String, fileId: String, scanStatus: String, reason: String, userId: String): Box[FileT] + /** How many personal files a User owns in a space, and their total size in bytes. */ + def personalUsage(bankId: String, ownerUserId: String): (Int, Long) + /** The files of a space that the User owns or has been given access to, oldest first. */ + def getFilesOwnedOrSharedWith(bankId: String, userId: String): List[FileT] + + def createAttachment(file: FileT, recordType: String, recordId: String, userId: String): Box[FileAttachmentT] def getAttachment(bankId: String, fileId: String, fileAttachmentId: String): Box[FileAttachmentT] + def getAttachments(bankId: String, fileId: String): List[FileAttachmentT] def attachmentExists(fileId: String, recordType: String, recordId: String): Boolean def deleteAttachment(fileAttachmentId: String): Box[Boolean] /** The files attached to one record of a space. */ def getFilesAttachedTo(bankId: String, recordType: String, recordId: String): List[FileT] - def grantAccess(fileId: String, granteeUserId: String, grantedByUserId: String, onBehalfOfUserId: String): Box[FileAccessT] + def grantAccess(fileId: String, granteeUserId: String, grantedByUserId: String): Box[FileAccessT] def getAccess(fileId: String, granteeUserId: String): Box[FileAccessT] def getAccessList(fileId: String): List[FileAccessT] def revokeAccess(fileId: String, granteeUserId: String): Box[Boolean] diff --git a/obp-api/src/main/scala/code/files/MappedFilesProvider.scala b/obp-api/src/main/scala/code/files/MappedFilesProvider.scala index 6227e88126..bdb428c8be 100644 --- a/obp-api/src/main/scala/code/files/MappedFilesProvider.scala +++ b/obp-api/src/main/scala/code/files/MappedFilesProvider.scala @@ -28,6 +28,7 @@ package code.files import java.util.{Date, UUID} +import code.users.{UserReference, Users} import net.liftweb.common.{Box, Empty, Full} import net.liftweb.mapper._ import net.liftweb.util.Helpers.tryo @@ -41,10 +42,16 @@ object MappedFilesProvider extends FilesProvider { /** Where the bytes go. Replaced by an object storage implementation when the volume calls for it. */ val store: FileStore = PostgresFileStore - override def createFile(bankId: String, sha256: String, mediaType: String, fileName: String, - userId: String, onBehalfOfUserId: String, bytes: Array[Byte]): Box[FileT] = { + /** The caller and, when an agent called for somebody, that somebody (empty otherwise), by the + * reference's attribution policy: these tables record both. */ + private def attributed(userId: String, ref: UserReference): Box[(String, String)] = + Users.users.vend.attributionOf(userId, ref).map(a => (a.userId, if (a.isDelegated) a.onBehalfOfUserId else "")) + + override def createFile(bankId: String, sha256: String, mediaType: String, fileName: String, isPublic: Boolean, + userId: String, bytes: Array[Byte], isPersonal: Boolean, scanStatus: String): Box[FileT] = { val fileId = UUID.randomUUID().toString for { + (callerId, onBehalfOfUserId) <- attributed(userId, UserReference.File_UserId) _ <- store.write(fileId, bytes) file <- tryo { File.create @@ -54,7 +61,10 @@ object MappedFilesProvider extends FilesProvider { .SizeInBytes(bytes.length.toLong) .MediaType(mediaType) .FileName(fileName) - .UserId(userId) + .IsPublic(isPublic) + .IsPersonal(isPersonal) + .ScanStatus(scanStatus) + .UserId(callerId) .OnBehalfOfUserId(onBehalfOfUserId) .CreatedAt(new Date()) .saveMe() @@ -67,23 +77,57 @@ object MappedFilesProvider extends FilesProvider { override def getContent(fileId: String): Box[Array[Byte]] = store.read(fileId) - override def createAttachment(file: FileT, recordType: String, recordId: String, - userId: String, onBehalfOfUserId: String): Box[FileAttachmentT] = tryo { - FileAttachment.create - .FileId(file.fileId) - .BankId(file.bankId) - .RecordType(recordType) - .RecordId(recordId) - .UserId(userId) - .OnBehalfOfUserId(onBehalfOfUserId) - .CreatedAt(new Date()) - .saveMe() + override def setPublic(bankId: String, fileId: String, isPublic: Boolean): Box[FileT] = + File.find(By(File.BankId, bankId), By(File.FileId, fileId)).flatMap(file => tryo(file.IsPublic(isPublic).saveMe())) + + override def getFiles(bankId: String, limit: Int, offset: Int, scanStatus: Option[String]): (List[FileT], Long) = { + val where: List[QueryParam[File]] = By(File.BankId, bankId) :: scanStatus.map(By(File.ScanStatus, _)).toList + (File.findAll(where ++ List(OrderBy(File.CreatedAt, Descending), MaxRows[File](limit), StartAt[File](offset)): _*), + File.count(where: _*)) } + override def setScanStatus(bankId: String, fileId: String, scanStatus: String, reason: String, userId: String): Box[FileT] = + File.find(By(File.BankId, bankId), By(File.FileId, fileId)).flatMap { file => + tryo(file.ScanStatus(scanStatus).ScanStatusReason(reason).ScanStatusUpdatedAt(new Date()) + .ScanStatusUpdatedByUserId(userId).saveMe()) + } + + override def personalUsage(bankId: String, ownerUserId: String): (Int, Long) = { + val owned = File.findAll(By(File.BankId, bankId), By(File.IsPersonal, true)).filter(_.ownerUserId == ownerUserId) + (owned.size, owned.map(_.sizeInBytes).sum) + } + + override def getFilesOwnedOrSharedWith(bankId: String, userId: String): List[FileT] = { + val uploaded = File.findAll(By(File.BankId, bankId), By(File.UserId, userId), By(File.OnBehalfOfUserId, "")) + val uploadedFor = File.findAll(By(File.BankId, bankId), By(File.OnBehalfOfUserId, userId)) + val sharedFileIds = FileAccess.findAll(By(FileAccess.GranteeUserId, userId)).map(_.FileId.get).distinct + val shared = if (sharedFileIds.isEmpty) Nil else File.findAll(By(File.BankId, bankId), ByList(File.FileId, sharedFileIds)) + (uploaded ++ uploadedFor ++ shared).groupBy(_.fileId).values.map(_.head).toList.sortBy(_.createdAt.getTime) + } + + override def createAttachment(file: FileT, recordType: String, recordId: String, userId: String): Box[FileAttachmentT] = + attributed(userId, UserReference.FileAttachment_UserId).flatMap { case (callerId, onBehalfOfUserId) => + tryo { + FileAttachment.create + .FileId(file.fileId) + .BankId(file.bankId) + .RecordType(recordType) + .RecordId(recordId) + .UserId(callerId) + .OnBehalfOfUserId(onBehalfOfUserId) + .CreatedAt(new Date()) + .saveMe() + } + } + override def getAttachment(bankId: String, fileId: String, fileAttachmentId: String): Box[FileAttachmentT] = FileAttachment.find(By(FileAttachment.BankId, bankId), By(FileAttachment.FileId, fileId), By(FileAttachment.FileAttachmentId, fileAttachmentId)) + override def getAttachments(bankId: String, fileId: String): List[FileAttachmentT] = + FileAttachment.findAll(By(FileAttachment.BankId, bankId), By(FileAttachment.FileId, fileId), + OrderBy(FileAttachment.CreatedAt, Ascending)) + override def attachmentExists(fileId: String, recordType: String, recordId: String): Boolean = FileAttachment.find(By(FileAttachment.FileId, fileId), By(FileAttachment.RecordType, recordType), By(FileAttachment.RecordId, recordId)).isDefined @@ -102,15 +146,20 @@ object MappedFilesProvider extends FilesProvider { } } - override def grantAccess(fileId: String, granteeUserId: String, grantedByUserId: String, onBehalfOfUserId: String): Box[FileAccessT] = tryo { - FileAccess.create - .FileId(fileId) - .GranteeUserId(granteeUserId) - .GrantedByUserId(grantedByUserId) - .OnBehalfOfUserId(onBehalfOfUserId) - .CreatedAt(new Date()) - .saveMe() - } + // The grantee is an explicit target named in the request: the endpoint refuses a consent user rather + // than redirecting, since access given to one would die with its Consent. + override def grantAccess(fileId: String, granteeUserId: String, grantedByUserId: String): Box[FileAccessT] = + attributed(grantedByUserId, UserReference.FileAccess_GrantedByUserId).flatMap { case (callerId, onBehalfOfUserId) => + tryo { + FileAccess.create + .FileId(fileId) + .GranteeUserId(granteeUserId) + .GrantedByUserId(callerId) + .OnBehalfOfUserId(onBehalfOfUserId) + .CreatedAt(new Date()) + .saveMe() + } + } override def getAccess(fileId: String, granteeUserId: String): Box[FileAccessT] = FileAccess.find(By(FileAccess.FileId, fileId), By(FileAccess.GranteeUserId, granteeUserId)) @@ -150,7 +199,18 @@ class File extends FileT with LongKeyedMapper[File] with IdPK { object FileName extends MappedString(this, 255) object UserId extends MappedString(this, 255) object OnBehalfOfUserId extends MappedString(this, 255) + object IsPublic extends MappedBoolean(this) { + override def defaultValue = false + } object CreatedAt extends MappedDateTime(this) + object IsPersonal extends MappedBoolean(this) { + override def defaultValue = false + } + // NULL on rows written before scanning existed, which count as clean. + object ScanStatus extends MappedString(this, 20) + object ScanStatusReason extends MappedString(this, 1000) + object ScanStatusUpdatedAt extends MappedDateTime(this) + object ScanStatusUpdatedByUserId extends MappedString(this, 255) override def fileId: String = FileId.get override def bankId: String = BankId.get @@ -160,7 +220,13 @@ class File extends FileT with LongKeyedMapper[File] with IdPK { override def fileName: String = FileName.get override def userId: String = UserId.get override def onBehalfOfUserId: String = Option(OnBehalfOfUserId.get).getOrElse("") + override def isPublic: Boolean = IsPublic.get override def createdAt: Date = CreatedAt.get + override def isPersonal: Boolean = Option(IsPersonal.get).exists(b => b) + override def scanStatus: String = Option(ScanStatus.get).filter(_.nonEmpty).getOrElse(code.files.ScanStatus.Clean) + override def scanStatusReason: String = Option(ScanStatusReason.get).getOrElse("") + override def scanStatusUpdatedAt: Option[Date] = Option(ScanStatusUpdatedAt.get) + override def scanStatusUpdatedByUserId: String = Option(ScanStatusUpdatedByUserId.get).getOrElse("") } object File extends File with LongKeyedMetaMapper[File] { @@ -171,7 +237,13 @@ class FileContent extends LongKeyedMapper[FileContent] with IdPK { override def getSingleton = FileContent object FileId extends MappedString(this, 36) - object Content extends MappedBinary(this) + object Content extends MappedBinary(this) { + // Lift creates a binary column as BINARY on H2, which H2 2.x reads as BINARY(1): one byte. Use VARBINARY + // there (not BLOB, which Mapper reads back as a Blob, not bytes). Other databases keep Lift's type + // (BYTEA on PostgreSQL). + override def fieldCreatorString(dbType: net.liftweb.db.DriverType, colName: String): String = + if (dbType == net.liftweb.db.H2Driver) colName + " VARBINARY" else super.fieldCreatorString(dbType, colName) + } } object FileContent extends FileContent with LongKeyedMetaMapper[FileContent] { diff --git a/obp-api/src/main/scala/code/messageoutbox/MessageOutbox.scala b/obp-api/src/main/scala/code/messageoutbox/MessageOutbox.scala index eafaa70e1b..2d61217b20 100644 --- a/obp-api/src/main/scala/code/messageoutbox/MessageOutbox.scala +++ b/obp-api/src/main/scala/code/messageoutbox/MessageOutbox.scala @@ -203,11 +203,12 @@ object MessageOutbox extends MessageOutbox with LongKeyedMetaMapper[MessageOutbo .saveMe() /** Queue an email; the relay sends it (attachments are not supported). */ - def enqueueEmail(subjectId: String, subjectIdType: String, operationName: String, email: EmailContent): MessageOutbox = { + def enqueueEmail(subjectId: String, subjectIdType: String, operationName: String, email: EmailContent, + metadataJson: String = ""): MessageOutbox = { val payload = OutboxEmailPayload(email.from, email.to, email.cc, email.bcc, email.subject, email.textContent, email.htmlContent) enqueue(TYPE_EMAIL, subjectId, subjectIdType, operationName, - email.to.mkString(",").take(255), Serialization.write(payload)(emailFormats)) + email.to.mkString(",").take(255), Serialization.write(payload)(emailFormats), metadataJson) } def emailPayload(row: MessageOutbox): OutboxEmailPayload = diff --git a/obp-api/src/main/scala/code/model/dataAccess/SelfServiceBank.scala b/obp-api/src/main/scala/code/model/dataAccess/SelfServiceBank.scala new file mode 100644 index 0000000000..1d9b516d74 --- /dev/null +++ b/obp-api/src/main/scala/code/model/dataAccess/SelfServiceBank.scala @@ -0,0 +1,59 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ +package code.model.dataAccess + +import net.liftweb.mapper._ +import net.liftweb.util.Helpers.tryo + +/** + * The banks made with Create My Bank (POST /my/banks): a space of their own for developers, without CanCreateBank. + * Anyone may make one and grant themselves its Roles, so those Roles vouch for nothing: files stored there start + * pending, for example. The Create My Bank quota counts these rows. + * + * Create My Bank refuses unless OBP stores banks itself (the mapped connector), so a self-service bank is never on a + * backend system. A bank from a backend is never in this record, and counts as a real bank. + */ +class SelfServiceBank extends LongKeyedMapper[SelfServiceBank] with IdPK { + def getSingleton = SelfServiceBank + + object BankId extends MappedString(this, 255) + object CreatedByUserId extends MappedString(this, 255) + object CreatedAt extends MappedDateTime(this) { + override def defaultValue = new java.util.Date() + } +} + +object SelfServiceBank extends SelfServiceBank with LongKeyedMetaMapper[SelfServiceBank] { + override def dbTableName = "SelfServiceBank" + override def dbIndexes = UniqueIndex(BankId) :: super.dbIndexes + + /** Whether the bank was made with Create My Bank. SYS and banks from a backend are not. */ + def isSelfService(bankId: String): Boolean = find(By(BankId, bankId)).isDefined + + def record(bankId: String, createdByUserId: String): Unit = + if (!isSelfService(bankId)) tryo(create.BankId(bankId).CreatedByUserId(createdByUserId).saveMe()) +} diff --git a/obp-api/src/main/scala/code/retention/Retention.scala b/obp-api/src/main/scala/code/retention/Retention.scala new file mode 100644 index 0000000000..e24cc3c1d7 --- /dev/null +++ b/obp-api/src/main/scala/code/retention/Retention.scala @@ -0,0 +1,190 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ +package code.retention + +import java.time.{Instant, LocalDate, LocalDateTime, OffsetDateTime, ZoneOffset} +import java.util.Date + +import code.api.Constant.DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID +import code.api.util.APIUtil +import org.json4s._ +import org.json4s.native.JsonMethods.{compact, render} + +import scala.util.Try + +/** + * The vocabulary of data retention: the resource types a policy can govern, the actions, the date a + * resource's retention period is counted from, and the run modes and statuses. See RETENTION_PLAN.md. + */ +object Retention { + + val ResourceDynamicEntityRecord = "dynamic_entity_record" + val ResourceFile = "file" + val ResourceChatMessage = "chat_message" + val ResourceChatRoom = "chat_room" + val ResourceCustomer = "customer" + val ResourceConsent = "consent" + val resourceTypes: List[String] = + List(ResourceDynamicEntityRecord, ResourceFile, ResourceChatMessage, ResourceChatRoom, ResourceCustomer, ResourceConsent) + + val ActionArchive = "archive" + val ActionDelete = "delete" + val ActionAnonymise = "anonymise" + val actions: List[String] = List(ActionArchive, ActionDelete, ActionAnonymise) + /** The register's action for an archived resource deleted from the archive when its archive period ended. */ + val ActionDeleteFromArchive = "delete_from_archive" + + val BasisCreated = "created" + val BasisUpdated = "updated" + val BasisLastActivity = "last_activity" + val BasisExpires = "expires" + val BasisFieldPrefix = "field:" + + val ModePreview = "preview" + val ModeExecute = "execute" + val modes: List[String] = List(ModePreview, ModeExecute) + + val StatusPending = "pending" + val StatusRunning = "running" + val StatusDone = "done" + val StatusRefused = "refused" + val StatusFailed = "failed" + val statuses: List[String] = List(StatusPending, StatusRunning, StatusDone, StatusRefused, StatusFailed) + + val DecisionApprove = "approve" + val DecisionReject = "reject" + val DecisionApproved = "approved" + val DecisionRejected = "rejected" + + /** What an anonymised Customer's legal name becomes, and how a run recognises one it already anonymised. */ + val AnonymisedLegalName = "ANONYMISED" + + /** The longest period a policy may name, in days: a hundred years. */ + val MaxDays = 36500 + + val SystemSpace: String = DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID + + /** + * The bank ids a space is stored under, for the tables that do not all use the SYS sentinel. A + * system chat room is stored with an empty bank id, and a file of the system space may be. + */ + def storedBankIds(bankId: String): List[String] = + if (bankId == SystemSpace) List(SystemSpace, "") else List(bankId) + + /** The date bases a policy for this resource type may count from. `field:` is accepted on top for dynamic entity records. */ + def dateBases(resourceType: String): List[String] = resourceType match { + case ResourceDynamicEntityRecord => List(BasisCreated, BasisUpdated) + case ResourceFile => List(BasisCreated) + case ResourceChatMessage => List(BasisCreated, BasisUpdated) + case ResourceChatRoom => List(BasisCreated, BasisUpdated, BasisLastActivity) + case ResourceCustomer => List(BasisCreated, BasisUpdated) + case ResourceConsent => List(BasisCreated, BasisUpdated, BasisExpires) + case _ => Nil + } + + def defaultDateBasis(resourceType: String): String = + if (resourceType == ResourceConsent) BasisExpires else BasisCreated + + def isValidDateBasis(resourceType: String, basis: String): Boolean = + dateBases(resourceType).contains(basis) || + (resourceType == ResourceDynamicEntityRecord && basis.startsWith(BasisFieldPrefix) && basis.stripPrefix(BasisFieldPrefix).trim.nonEmpty) + + def actionsFor(resourceType: String): List[String] = + if (resourceType == ResourceCustomer) actions else List(ActionArchive, ActionDelete) + + /** The end of the retention period: a resource whose trigger date is on or before this date is due. */ + def cutoff(retainDays: Int, now: Date = new Date()): Date = + new Date(now.getTime - retainDays.toLong * 86400000L) + + def plusDays(date: Date, days: Int): Date = new Date(date.getTime + days.toLong * 86400000L) + + /** + * Does the resource match the policy's condition? The field is read from the top level of the + * resource's JSON (a dynamic entity record's own fields; for the other types, the fields of their + * JSON copy), and compared with the value as text: `true`, `42` and `CLOSED` all match. + */ + def conditionMatches(condition: Option[RetentionCondition], view: JValue): Boolean = condition match { + case None => true + case Some(RetentionCondition(field, value)) => (view \ field) match { + case JNothing => false + case JNull => value == "null" + case JString(s) => s == value + case JBool(b) => b.toString == value + case JInt(i) => i.toString == value + case JDouble(d) => Try(BigDecimal(value) == BigDecimal(d)).getOrElse(false) + case JDecimal(d) => Try(BigDecimal(value) == d).getOrElse(false) + case other => compact(render(other)) == value + } + } + + /** + * The date in a dynamic entity record's field, for a `field:` date basis: an ISO 8601 date + * (`2026-01-31`, read as midnight UTC) or date and time (`2026-01-31T10:00:00Z`). A record whose + * field is missing or is not such a date has no trigger date, and no policy counting from that field + * matches it. + */ + def dateIn(view: JValue, field: String): Option[Date] = (view \ field) match { + case JString(s) => + val text = s.trim + Try(Date.from(OffsetDateTime.parse(text).toInstant)) + .orElse(Try(Date.from(Instant.parse(text)))) + .orElse(Try(Date.from(LocalDateTime.parse(text).toInstant(ZoneOffset.UTC)))) + .orElse(Try(Date.from(LocalDate.parse(text).atStartOfDay(ZoneOffset.UTC).toInstant))) + .toOption + case _ => None + } + + def iso(date: Date): JValue = Option(date).map(d => JString(Instant.ofEpochMilli(d.getTime).toString)).getOrElse(JNull) + def text(value: String): JValue = Option(value).map(JString(_)).getOrElse(JNull) +} + +/** + * The props of data retention. Every value is read when it is used, so a change to the props takes + * effect without a restart where the props source allows it. All of them are documented in + * sample.props.template. + */ +object RetentionProps { + /** Runs that would move resources into the retention archive are refused unless this is true. */ + def archiveEnabled: Boolean = APIUtil.getPropsAsBoolValue("retention.archive.enabled", false) + /** Runs that would delete or anonymise resources, or delete them from the archive, are refused unless this is true. */ + def deleteEnabled: Boolean = APIUtil.getPropsAsBoolValue("retention.delete.enabled", false) + /** + * No resource leaves the live tables, by any action, sooner than this many days after its trigger + * date: a policy with a shorter retain_days is refused, and an existing one (from before the prop was + * raised) runs as though its retain_days were this. 1825 days, five years, by default. + */ + val MinimumRetainDaysDefault = 1825 + def minimumRetainDays: Int = math.max(0, APIUtil.getPropsAsIntValue("retention.minimum_retain_days", MinimumRetainDaysDefault)) + def schedulerEnabled: Boolean = APIUtil.getPropsAsBoolValue("retention.scheduler.enabled", false) + /** 3593 seconds, a prime just under an hour, so the runs drift across the clock rather than lining up with other schedulers. */ + val SchedulerIntervalInSecondsDefault = 3593L + def schedulerIntervalInSeconds: Long = + APIUtil.getPropsAsLongValue("retention.scheduler.interval_in_seconds", SchedulerIntervalInSecondsDefault) + /** The most resources one run disposes of. What is left is disposed of by the next run. */ + val MaxItemsPerRunDefault = 1000 + def maxItemsPerRun: Int = math.max(1, APIUtil.getPropsAsIntValue("retention.run.max_items", MaxItemsPerRunDefault)) +} diff --git a/obp-api/src/main/scala/code/retention/RetentionEngine.scala b/obp-api/src/main/scala/code/retention/RetentionEngine.scala new file mode 100644 index 0000000000..8985dc77ef --- /dev/null +++ b/obp-api/src/main/scala/code/retention/RetentionEngine.scala @@ -0,0 +1,366 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ +package code.retention + +import java.util.Date + +import code.api.util.APIUtil.generateUUID +import code.api.util.ErrorMessages.{RetentionArchiveIsDisabled, RetentionDeleteIsDisabled} +import code.retention.Retention._ +import code.util.Helper.MdcLoggable +import net.liftweb.mapper._ +import org.json4s.native.JsonMethods.{compact, render} + +import scala.util.control.NonFatal + +/** The active legal holds of one space, and the question every run asks of them. */ +class RetentionHolds(holds: List[RetentionHold]) { + /** + * Is this resource on hold? A hold names one resource by id (and, for a dynamic entity record, + * optionally the entity), or every record of one dynamic entity by its name alone. + */ + def isHeld(resourceType: String, resourceId: String, entityName: Option[String]): Boolean = + holds.exists(hold => hold.resourceType == resourceType && (hold.resourceId match { + case Some(id) => id == resourceId && (hold.entityName.isEmpty || hold.entityName == entityName) + case None => hold.entityName.isDefined && hold.entityName == entityName + })) +} + +object RetentionHolds { + def active(bankId: String): RetentionHolds = + new RetentionHolds(RetentionHold.findAll(By(RetentionHold.BankId, bankId), NullRef(RetentionHold.ReleasedAt))) +} + +/** + * What one execution of a run does to the archive and the disposal register, and what it counts. + * The resource handlers call it for every resource they dispose of, the cascaded ones included. + */ +class Disposal(val run: RetentionRun, val policy: Option[RetentionPolicy], holds: RetentionHolds) { + var archived = 0 + var anonymised = 0 + var deleted = 0 + var held = 0 + + def countHeld(): Unit = held += 1 + + def isHeld(resourceType: String, resourceId: String, entityName: Option[String]): Boolean = + holds.isHeld(resourceType, resourceId, entityName) + + def isHeld(candidate: RetentionCandidate): Boolean = + isHeld(candidate.resourceType, candidate.resourceId, candidate.entityName) + + /** Write the candidate's JSON copy (and a file's bytes) to the retention archive; its archive id. */ + def archive(candidate: RetentionCandidate, content: Option[Array[Byte]], parent: Option[(String, String)]): String = { + val archiveId = generateUUID() + val now = new Date() + RetentionArchive.create + .RetentionArchiveId(archiveId) + .RetentionPolicyId(policy.map(_.retentionPolicyId).getOrElse("")) + .RetentionRunId(run.retentionRunId) + .BankId(candidate.bankId) + .ResourceType(candidate.resourceType) + .ResourceId(candidate.resourceId) + .EntityName(candidate.entityName.getOrElse("")) + .OwnerUserId(candidate.ownerUserId.getOrElse("")) + .ParentResourceType(parent.map(_._1).getOrElse("")) + .ParentResourceId(parent.map(_._2).getOrElse("")) + .JsonCopy(compact(render(candidate.jsonCopy()))) + .HasContent(content.isDefined) + .ContentSizeInBytes(content.map(_.length.toLong).getOrElse(0L)) + .ArchivedAt(now) + .DeleteAfter(policy.flatMap(_.archiveDays).map(days => plusDays(now, days)).orNull) + .saveMe() + content.foreach(bytes => RetentionArchiveContent.create.RetentionArchiveId(archiveId) + .ContentBase64(java.util.Base64.getEncoder.encodeToString(bytes)).saveMe()) + archiveId + } + + /** Write one row of the disposal register, and count it. */ + def register(candidate: RetentionCandidate, action: String, archiveId: Option[String], parent: Option[(String, String)]): Unit = { + writeItem(candidate.bankId, candidate.resourceType, candidate.resourceId, candidate.entityName, candidate.ownerUserId, + action, archiveId, parent) + action match { + case ActionArchive => archived += 1 + case ActionAnonymise => anonymised += 1 + case _ => deleted += 1 + } + } + + /** Register an archived resource deleted from the archive at the end of its archive period. */ + def registerDeletedFromArchive(entry: RetentionArchive): Unit = { + writeItem(entry.bankId, entry.resourceType, entry.resourceId, entry.entityName, entry.ownerUserId, + ActionDeleteFromArchive, Some(entry.retentionArchiveId), + entry.parentResourceType.flatMap(t => entry.parentResourceId.map(t -> _))) + deleted += 1 + } + + private def writeItem(bankId: String, resourceType: String, resourceId: String, entityName: Option[String], + ownerUserId: Option[String], action: String, archiveId: Option[String], + parent: Option[(String, String)]): Unit = + RetentionRunItem.create + .RetentionRunItemId(generateUUID()) + .RetentionRunId(run.retentionRunId) + .RetentionPolicyId(policy.map(_.retentionPolicyId).getOrElse("")) + .BankId(bankId) + .ResourceType(resourceType) + .ResourceId(resourceId) + .EntityName(entityName.getOrElse("")) + .OwnerUserId(ownerUserId.getOrElse("")) + .Action(action) + .ParentResourceType(parent.map(_._1).getOrElse("")) + .ParentResourceId(parent.map(_._2).getOrElse("")) + .RetentionArchiveId(archiveId.getOrElse("")) + .DisposedAt(new Date()) + .saveMe() +} + +/** + * This object carries out retention: previews, runs, approvals and the deletion of archived + * resources whose archive period has ended. The endpoints and RetentionScheduler both call it, so a + * run started by hand and one started by the scheduler obey the same rules. + */ +object RetentionEngine extends MdcLoggable { + + /** The most resources a preview reads before it stops counting. */ + val PreviewScanLimit = 100000 + val SampleSize = 10 + /** A run still marked running after this long was cut short (the JVM died) and no longer blocks the next one. */ + val StaleRunMillis: Long = 6L * 3600L * 1000L + + case class PreviewResult(cutoff: Date, matched: Int, held: Int, oldestTriggerDate: Option[Date], + sampleResourceIds: List[String], archiveEntriesDue: Int, scanLimitReached: Boolean) + + def handlerOf(policy: RetentionPolicy): RetentionResourceHandler = RetentionResources.handlers(policy.resourceType) + + /** The retain days a run uses: the policy's, but never fewer than retention.minimum_retain_days. */ + def retainDaysOf(policy: RetentionPolicy): Int = math.max(policy.retainDays, RetentionProps.minimumRetainDays) + + /** What a run of the policy would dispose of now. Changes nothing. */ + def preview(policy: RetentionPolicy): PreviewResult = { + val cutoff = Retention.cutoff(retainDaysOf(policy)) + val holds = RetentionHolds.active(policy.bankId) + var matched = 0 + var held = 0 + var oldest: Option[Date] = None + var samples = List.empty[String] + handlerOf(policy).scan(policy, cutoff) { candidate => + matched += 1 + if (holds.isHeld(candidate.resourceType, candidate.resourceId, candidate.entityName)) held += 1 + else if (samples.size < SampleSize) samples = samples :+ candidate.resourceId + val trigger = Option(candidate.triggerDate) + if (trigger.exists(t => oldest.forall(t.before))) oldest = trigger + matched < PreviewScanLimit + } + PreviewResult(cutoff, matched, held, oldest, samples, archiveEntriesDue(policy), matched >= PreviewScanLimit) + } + + def archiveEntriesDue(policy: RetentionPolicy): Int = + RetentionArchive.count(By(RetentionArchive.RetentionPolicyId, policy.retentionPolicyId), + By_<=(RetentionArchive.DeleteAfter, new Date())).toInt + + /** + * Why an execution of this policy must be refused on this instance: archiving or deleting is + * switched off in the props. Anonymising is irreversible, so it counts as deleting. A preview is + * never refused. + */ + def refusal(policy: RetentionPolicy): Option[String] = policy.action match { + case ActionArchive if !RetentionProps.archiveEnabled => Some(RetentionArchiveIsDisabled) + case ActionDelete | ActionAnonymise if !RetentionProps.deleteEnabled => Some(RetentionDeleteIsDisabled) + case _ => None + } + + /** Is an execution of this policy under way? */ + def isRunning(policyId: String): Boolean = + RetentionRun.find(By(RetentionRun.RetentionPolicyId, policyId), By(RetentionRun.Status, StatusRunning), + By_>(RetentionRun.StartedAt, new Date(System.currentTimeMillis() - StaleRunMillis))).isDefined + + private def newRun(bankId: String, policyId: Option[String], mode: String, status: String, + requestedByUserId: String, message: String): RetentionRun = { + val now = new Date() + RetentionRun.create + .RetentionRunId(generateUUID()) + .RetentionPolicyId(policyId.getOrElse("")) + .BankId(bankId) + .Mode(mode) + .Status(status) + .RequestedByUserId(requestedByUserId) + .RequestedAt(now) + .StartedAt(if (status == StatusRunning) now else null) + .EndedAt(if (status == StatusRefused) now else null) + .Message(message) + .saveMe() + } + + /** + * Run the policy now, for `requestedByUserId` (empty for the scheduler). A preview records its + * counts and changes nothing. An execution is refused when the props switch its action off, waits + * as pending when the policy requires approval, and otherwise runs straight away. + */ + def requestRun(policy: RetentionPolicy, mode: String, requestedByUserId: String): RetentionRun = + if (mode == ModePreview) { + val run = newRun(policy.bankId, Some(policy.retentionPolicyId), ModePreview, StatusRunning, requestedByUserId, "") + val result = preview(policy) + run.MatchedCount(result.matched).HeldCount(result.held).Status(StatusDone).EndedAt(new Date()) + .Message(s"Preview only: ${result.matched - result.held} resource(s) would be disposed of by '${policy.action}', ${result.held} on hold." + + (if (result.scanLimitReached) s" Counting stopped at $PreviewScanLimit." else "")) + .saveMe() + } else refusal(policy) match { + case Some(reason) => + newRun(policy.bankId, Some(policy.retentionPolicyId), ModeExecute, StatusRefused, requestedByUserId, reason) + case None if policy.requiresApproval => + newRun(policy.bankId, Some(policy.retentionPolicyId), ModeExecute, StatusPending, requestedByUserId, + "This policy requires approval: the run starts when a different user approves it.") + case None => + execute(newRun(policy.bankId, Some(policy.retentionPolicyId), ModeExecute, StatusRunning, requestedByUserId, ""), policy) + } + + /** + * Approve or reject a pending run. The caller has already checked that the approver is not the + * user who requested it. An approved run executes now, unless the props have switched its action + * off since it was requested, in which case it is refused. + */ + def decide(run: RetentionRun, policy: Option[RetentionPolicy], approverUserId: String, approve: Boolean, + comment: Option[String]): RetentionRun = { + val decided = run.ApprovedByUserId(approverUserId) + .ApprovalDecision(if (approve) DecisionApproved else DecisionRejected) + .ApprovalComment(comment.getOrElse("")) + if (!approve) decided.Status(StatusRefused).EndedAt(new Date()).Message("Rejected by the approver.").saveMe() + else policy match { + case None => + decided.Status(StatusRefused).EndedAt(new Date()).Message("The policy of this run has been deleted.").saveMe() + case Some(p) => refusal(p) match { + case Some(reason) => decided.Status(StatusRefused).EndedAt(new Date()).Message(reason).saveMe() + case None => execute(decided.Status(StatusRunning).StartedAt(new Date()).saveMe(), p) + } + } + } + + /** + * Execute a run: dispose of what is due, up to retention.run.max_items resources, skipping what is + * on hold; then delete from the archive what this policy archived and whose archive period has + * ended. A resource that fails is left in place and counted; the run then ends as failed. + */ + def execute(run: RetentionRun, policy: RetentionPolicy): RetentionRun = { + val disposal = new Disposal(run, Some(policy), RetentionHolds.active(policy.bankId)) + var matched = 0 + var failed = 0 + var limitReached = false + val maxItems = RetentionProps.maxItemsPerRun + try { + handlerOf(policy).scan(policy, Retention.cutoff(retainDaysOf(policy))) { candidate => + matched += 1 + if (disposal.isHeld(candidate)) disposal.countHeld() + else try { + if (!candidate.dispose(policy.action, disposal)) disposal.countHeld() + } catch { + case NonFatal(e) => + failed += 1 + logger.error(s"RetentionEngine: could not ${policy.action} ${candidate.resourceType} ${candidate.resourceId} " + + s"(policy ${policy.retentionPolicyId}, run ${run.retentionRunId})", e) + } + limitReached = matched >= maxItems + !limitReached + } + val archiveNote = + if (RetentionProps.deleteEnabled) { + deleteDueFromArchive(disposal, List(By(RetentionArchive.RetentionPolicyId, policy.retentionPolicyId))) + "" + } else if (archiveEntriesDue(policy) > 0) + " Archived resources whose archive period has ended were kept, because retention.delete.enabled is false." + else "" + val message = + (if (failed > 0) s"$failed resource(s) could not be disposed of and were left in place." else "Done.") + + (if (limitReached) s" The run stopped at retention.run.max_items ($maxItems); the next run continues." else "") + + archiveNote + finish(run, disposal, matched, failed, if (failed > 0) StatusFailed else StatusDone, message) + } catch { + case NonFatal(e) => + logger.error(s"RetentionEngine: run ${run.retentionRunId} of policy ${policy.retentionPolicyId} failed", e) + finish(run, disposal, matched, failed, StatusFailed, Option(e.getMessage).getOrElse(e.toString)) + } + } + + private def finish(run: RetentionRun, disposal: Disposal, matched: Int, failed: Int, status: String, message: String): RetentionRun = + run.MatchedCount(matched) + .ArchivedCount(disposal.archived) + .AnonymisedCount(disposal.anonymised) + .DeletedCount(disposal.deleted) + .HeldCount(disposal.held) + .FailedCount(failed) + .Status(status) + .EndedAt(new Date()) + .Message(message) + .saveMe() + + /** Delete the archive entries matching `filter` whose archive period has ended, unless on hold. */ + private def deleteDueFromArchive(disposal: Disposal, filter: List[QueryParam[RetentionArchive]]): Unit = { + val due = RetentionArchive.findAll((filter ::: List(By_<=(RetentionArchive.DeleteAfter, new Date()), + OrderBy(RetentionArchive.DeleteAfter, Ascending), MaxRows[RetentionArchive](RetentionProps.maxItemsPerRun))): _*) + due.foreach { entry => + val parentHeld = entry.parentResourceType.exists(t => entry.parentResourceId.exists(id => disposal.isHeld(t, id, entry.entityName))) + if (disposal.isHeld(entry.resourceType, entry.resourceId, entry.entityName) || parentHeld) disposal.countHeld() + else { + RetentionArchiveContent.bulkDelete_!!(By(RetentionArchiveContent.RetentionArchiveId, entry.retentionArchiveId)) + entry.delete_! + disposal.registerDeletedFromArchive(entry) + } + } + } + + /** + * Delete from the archive every entry whose archive period has ended, whichever policy archived it, + * so that deleting a policy never leaves its archive behind. One run per space is recorded, with no + * policy. Only called with retention.delete.enabled true. + */ + def deleteAllDueFromArchive(): List[RetentionRun] = { + val banks = RetentionArchive.findAll(By_<=(RetentionArchive.DeleteAfter, new Date()), + MaxRows[RetentionArchive](RetentionProps.maxItemsPerRun)).map(_.bankId).distinct + banks.map { bankId => + val run = newRun(bankId, None, ModeExecute, StatusRunning, "", "") + val disposal = new Disposal(run, None, RetentionHolds.active(bankId)) + try { + deleteDueFromArchive(disposal, List(By(RetentionArchive.BankId, bankId))) + finish(run, disposal, 0, 0, StatusDone, "Archived resources whose archive period has ended were deleted from the archive.") + } catch { + case NonFatal(e) => + logger.error(s"RetentionEngine: deleting from the archive of $bankId failed", e) + finish(run, disposal, 0, 1, StatusFailed, Option(e.getMessage).getOrElse(e.toString)) + } + } + } + + /** Has the policy anything to do now? Reads no further than the first resource not on hold. */ + def hasWork(policy: RetentionPolicy): Boolean = { + val holds = RetentionHolds.active(policy.bankId) + var found = false + handlerOf(policy).scan(policy, Retention.cutoff(retainDaysOf(policy))) { candidate => + found = !holds.isHeld(candidate.resourceType, candidate.resourceId, candidate.entityName) + !found + } + found || (RetentionProps.deleteEnabled && archiveEntriesDue(policy) > 0) + } +} diff --git a/obp-api/src/main/scala/code/retention/RetentionModels.scala b/obp-api/src/main/scala/code/retention/RetentionModels.scala new file mode 100644 index 0000000000..ac55dfcbc2 --- /dev/null +++ b/obp-api/src/main/scala/code/retention/RetentionModels.scala @@ -0,0 +1,293 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ +package code.retention + +import java.util.Date + +import code.util.MappedUUID +import net.liftweb.mapper._ + +/* + * The tables of data retention. See RETENTION_PLAN.md at the repository root for the model. + * + * Naming: these are DB entities, so no class starts with `Mapped` and no column object starts with + * `m` + uppercase (see MappedClassNameTest). None of them adds a column to an existing table. + */ + +/** + * One retention policy: how long one kind of resource of one space (a bank, or SYS) stays live, and + * what happens to it then. + * + * ArchiveDays holds -1 when the policy has no archive stage, because Mapper has no nullable int; the + * readers below turn that back into None. + */ +class RetentionPolicy extends LongKeyedMapper[RetentionPolicy] with IdPK with CreatedUpdated { + def getSingleton = RetentionPolicy + + object RetentionPolicyId extends MappedUUID(this) + object BankId extends MappedString(this, 255) + object Name extends MappedString(this, 255) + object Description extends MappedText(this) + object ResourceType extends MappedString(this, 64) + object EntityName extends MappedString(this, 255) + object DateBasis extends MappedString(this, 255) + object ConditionField extends MappedString(this, 255) + object ConditionValue extends MappedString(this, 2000) + object RetainDays extends MappedInt(this) + object ArchiveDays extends MappedInt(this) { + override def defaultValue = -1 + } + object Action extends MappedString(this, 32) + object Cascade extends MappedBoolean(this) + object RequiresApproval extends MappedBoolean(this) + object Enabled extends MappedBoolean(this) + object CreatedByUserId extends MappedString(this, 255) + + def retentionPolicyId: String = RetentionPolicyId.get + def bankId: String = BankId.get + def name: String = Option(Name.get).getOrElse("") + def description: String = Option(Description.get).getOrElse("") + def resourceType: String = ResourceType.get + def entityName: Option[String] = Option(EntityName.get).filter(_.nonEmpty) + def dateBasis: String = DateBasis.get + def condition: Option[RetentionCondition] = + Option(ConditionField.get).filter(_.nonEmpty).map(field => RetentionCondition(field, Option(ConditionValue.get).getOrElse(""))) + def retainDays: Int = RetainDays.get + def archiveDays: Option[Int] = Option(ArchiveDays.get).filter(_ >= 0) + def action: String = Action.get + def cascade: Boolean = Cascade.get + def requiresApproval: Boolean = RequiresApproval.get + def enabled: Boolean = Enabled.get + def createdByUserId: String = Option(CreatedByUserId.get).getOrElse("") +} + +object RetentionPolicy extends RetentionPolicy with LongKeyedMetaMapper[RetentionPolicy] { + override def dbIndexes: List[BaseIndex[RetentionPolicy]] = + UniqueIndex(RetentionPolicyId) :: Index(BankId) :: super.dbIndexes +} + +/** A simple condition on a resource: the field `field` must have the value `value`. */ +case class RetentionCondition(field: String, value: String) + +/** + * One archived resource: a JSON copy of a record that has left the live tables. It is read-only and + * reachable only through the retention archive endpoints. It stays here until DeleteAfter, and is + * then deleted. A file's bytes are kept apart, in RetentionArchiveContent, so that listing the + * archive never loads them. + */ +class RetentionArchive extends LongKeyedMapper[RetentionArchive] with IdPK { + def getSingleton = RetentionArchive + + object RetentionArchiveId extends MappedUUID(this) + object RetentionPolicyId extends MappedString(this, 36) + object RetentionRunId extends MappedString(this, 36) + object BankId extends MappedString(this, 255) + object ResourceType extends MappedString(this, 64) + object ResourceId extends MappedString(this, 255) + object EntityName extends MappedString(this, 255) + /** The owner of the archived resource, as it was stored on the resource. Copied, never the caller. */ + object OwnerUserId extends MappedString(this, 255) + object ParentResourceType extends MappedString(this, 64) + object ParentResourceId extends MappedString(this, 255) + object JsonCopy extends MappedText(this) + object HasContent extends MappedBoolean(this) + object ContentSizeInBytes extends MappedLong(this) + object ArchivedAt extends MappedDateTime(this) + object DeleteAfter extends MappedDateTime(this) + + def retentionArchiveId: String = RetentionArchiveId.get + def retentionPolicyId: Option[String] = Option(RetentionPolicyId.get).filter(_.nonEmpty) + def retentionRunId: String = Option(RetentionRunId.get).getOrElse("") + def bankId: String = BankId.get + def resourceType: String = ResourceType.get + def resourceId: String = ResourceId.get + def entityName: Option[String] = Option(EntityName.get).filter(_.nonEmpty) + def ownerUserId: Option[String] = Option(OwnerUserId.get).filter(_.nonEmpty) + def parentResourceType: Option[String] = Option(ParentResourceType.get).filter(_.nonEmpty) + def parentResourceId: Option[String] = Option(ParentResourceId.get).filter(_.nonEmpty) + def jsonCopy: String = Option(JsonCopy.get).getOrElse("{}") + def hasContent: Boolean = HasContent.get + def contentSizeInBytes: Long = ContentSizeInBytes.get + def archivedAt: Date = ArchivedAt.get + def deleteAfter: Option[Date] = Option(DeleteAfter.get) +} + +object RetentionArchive extends RetentionArchive with LongKeyedMetaMapper[RetentionArchive] { + override def dbIndexes: List[BaseIndex[RetentionArchive]] = + UniqueIndex(RetentionArchiveId) :: + Index(BankId, ResourceType, ResourceId) :: + Index(RetentionPolicyId) :: + Index(DeleteAfter) :: + super.dbIndexes +} + +/** + * The bytes of an archived file, one row per RetentionArchive row that has content. They are kept + * Base64 encoded in a text column rather than in a MappedBinary: Lift maps a binary column to a + * one-byte BINARY on H2, which the test databases use, while a text column holds any size on every + * database OBP supports. + */ +class RetentionArchiveContent extends LongKeyedMapper[RetentionArchiveContent] with IdPK { + def getSingleton = RetentionArchiveContent + + object RetentionArchiveId extends MappedString(this, 36) + object ContentBase64 extends MappedText(this) + + def content: Option[Array[Byte]] = + Option(ContentBase64.get).flatMap(text => scala.util.Try(java.util.Base64.getDecoder.decode(text)).toOption) +} + +object RetentionArchiveContent extends RetentionArchiveContent with LongKeyedMetaMapper[RetentionArchiveContent] { + override def dbIndexes: List[BaseIndex[RetentionArchiveContent]] = UniqueIndex(RetentionArchiveId) :: super.dbIndexes +} + +/** + * A legal hold: while it is active (ReleasedAt is empty) no retention run archives, anonymises or + * deletes what it names. It names either one resource (ResourceId), or, for dynamic entity records, + * every record of one dynamic entity of the space (EntityName with no ResourceId). + */ +class RetentionHold extends LongKeyedMapper[RetentionHold] with IdPK { + def getSingleton = RetentionHold + + object RetentionHoldId extends MappedUUID(this) + object BankId extends MappedString(this, 255) + object ResourceType extends MappedString(this, 64) + object ResourceId extends MappedString(this, 255) + object EntityName extends MappedString(this, 255) + object Reason extends MappedText(this) + object PlacedByUserId extends MappedString(this, 255) + object PlacedAt extends MappedDateTime(this) + object ReleasedByUserId extends MappedString(this, 255) + object ReleasedAt extends MappedDateTime(this) + + def retentionHoldId: String = RetentionHoldId.get + def bankId: String = BankId.get + def resourceType: String = ResourceType.get + def resourceId: Option[String] = Option(ResourceId.get).filter(_.nonEmpty) + def entityName: Option[String] = Option(EntityName.get).filter(_.nonEmpty) + def reason: String = Option(Reason.get).getOrElse("") + def placedByUserId: String = Option(PlacedByUserId.get).getOrElse("") + def placedAt: Date = PlacedAt.get + def releasedByUserId: Option[String] = Option(ReleasedByUserId.get).filter(_.nonEmpty) + def releasedAt: Option[Date] = Option(ReleasedAt.get) + def isActive: Boolean = releasedAt.isEmpty +} + +object RetentionHold extends RetentionHold with LongKeyedMetaMapper[RetentionHold] { + override def dbIndexes: List[BaseIndex[RetentionHold]] = + UniqueIndex(RetentionHoldId) :: Index(BankId, ResourceType) :: super.dbIndexes +} + +/** + * One run of a policy: a preview (counts only) or an execution. An execution of a policy that + * requires approval starts as pending and runs when a second, different user approves it. + */ +class RetentionRun extends LongKeyedMapper[RetentionRun] with IdPK { + def getSingleton = RetentionRun + + object RetentionRunId extends MappedUUID(this) + object RetentionPolicyId extends MappedString(this, 36) + object BankId extends MappedString(this, 255) + object Mode extends MappedString(this, 16) + object Status extends MappedString(this, 16) + object RequestedByUserId extends MappedString(this, 255) + object ApprovedByUserId extends MappedString(this, 255) + object ApprovalDecision extends MappedString(this, 16) + object ApprovalComment extends MappedText(this) + object MatchedCount extends MappedInt(this) + object ArchivedCount extends MappedInt(this) + object AnonymisedCount extends MappedInt(this) + object DeletedCount extends MappedInt(this) + object HeldCount extends MappedInt(this) + object FailedCount extends MappedInt(this) + object RequestedAt extends MappedDateTime(this) + object StartedAt extends MappedDateTime(this) + object EndedAt extends MappedDateTime(this) + object Message extends MappedText(this) + + def retentionRunId: String = RetentionRunId.get + def retentionPolicyId: Option[String] = Option(RetentionPolicyId.get).filter(_.nonEmpty) + def bankId: String = BankId.get + def mode: String = Mode.get + def status: String = Status.get + def requestedByUserId: String = Option(RequestedByUserId.get).getOrElse("") + def approvedByUserId: Option[String] = Option(ApprovedByUserId.get).filter(_.nonEmpty) + def approvalDecision: Option[String] = Option(ApprovalDecision.get).filter(_.nonEmpty) + def approvalComment: Option[String] = Option(ApprovalComment.get).filter(_.nonEmpty) + def requestedAt: Date = RequestedAt.get + def startedAt: Option[Date] = Option(StartedAt.get) + def endedAt: Option[Date] = Option(EndedAt.get) + def message: String = Option(Message.get).getOrElse("") +} + +object RetentionRun extends RetentionRun with LongKeyedMetaMapper[RetentionRun] { + override def dbIndexes: List[BaseIndex[RetentionRun]] = + UniqueIndex(RetentionRunId) :: Index(BankId, RequestedAt) :: Index(RetentionPolicyId, Status) :: super.dbIndexes +} + +/** + * The disposal register: one row per resource a run archived, anonymised or deleted, and per + * archived resource later deleted from the archive. It holds identifiers only, never content, so it + * can be kept after the resource itself is gone. + */ +class RetentionRunItem extends LongKeyedMapper[RetentionRunItem] with IdPK { + def getSingleton = RetentionRunItem + + object RetentionRunItemId extends MappedUUID(this) + object RetentionRunId extends MappedString(this, 36) + object RetentionPolicyId extends MappedString(this, 36) + object BankId extends MappedString(this, 255) + object ResourceType extends MappedString(this, 64) + object ResourceId extends MappedString(this, 255) + object EntityName extends MappedString(this, 255) + /** The owner of the disposed resource, as it was stored on the resource. Copied, never the caller. */ + object OwnerUserId extends MappedString(this, 255) + object Action extends MappedString(this, 32) + object ParentResourceType extends MappedString(this, 64) + object ParentResourceId extends MappedString(this, 255) + object RetentionArchiveId extends MappedString(this, 36) + object DisposedAt extends MappedDateTime(this) + + def retentionRunItemId: String = RetentionRunItemId.get + def retentionRunId: String = RetentionRunId.get + def retentionPolicyId: Option[String] = Option(RetentionPolicyId.get).filter(_.nonEmpty) + def bankId: String = BankId.get + def resourceType: String = ResourceType.get + def resourceId: String = ResourceId.get + def entityName: Option[String] = Option(EntityName.get).filter(_.nonEmpty) + def ownerUserId: Option[String] = Option(OwnerUserId.get).filter(_.nonEmpty) + def action: String = Action.get + def parentResourceType: Option[String] = Option(ParentResourceType.get).filter(_.nonEmpty) + def parentResourceId: Option[String] = Option(ParentResourceId.get).filter(_.nonEmpty) + def retentionArchiveId: Option[String] = Option(RetentionArchiveId.get).filter(_.nonEmpty) + def disposedAt: Date = DisposedAt.get +} + +object RetentionRunItem extends RetentionRunItem with LongKeyedMetaMapper[RetentionRunItem] { + override def dbIndexes: List[BaseIndex[RetentionRunItem]] = + UniqueIndex(RetentionRunItemId) :: Index(RetentionRunId) :: Index(BankId, ResourceType, ResourceId) :: super.dbIndexes +} diff --git a/obp-api/src/main/scala/code/retention/RetentionResources.scala b/obp-api/src/main/scala/code/retention/RetentionResources.scala new file mode 100644 index 0000000000..a433e4cc7a --- /dev/null +++ b/obp-api/src/main/scala/code/retention/RetentionResources.scala @@ -0,0 +1,576 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ +package code.retention + +import java.util.Date + +import code.CustomerDependants.MappedCustomerDependant +import code.DynamicData.{DynamicData, DynamicDataAccess} +import code.chat.{ChatMessage, ChatRoom, Participant, Reaction} +import code.consent.{ConsentItem, MappedConsent} +import code.context.MappedConsentAuthContext +import code.customer.MappedCustomer +import code.customeraccountlinks.CustomerAccountLink +import code.customeraddress.MappedCustomerAddress +import code.customerattribute.MappedCustomerAttribute +import code.files.{File, FileAccess, FileAttachment, FileContent} +import code.retention.Retention._ +import code.taxresidence.MappedTaxResidence +import code.usercustomerlinks.MappedUserCustomerLink +import com.openbankproject.commons.util.JsonAliases +import net.liftweb.mapper._ +import org.json4s._ +import org.json4s.JsonDSL._ + +import scala.util.Try + +/** + * One resource a policy has found due: its identity, the date its retention period is counted from, + * and how to dispose of it. + * + * `conditionView` is what the policy's condition is checked against, and is cheap to build. + * `jsonCopy` is the full copy written to the retention archive, built only when the resource is + * archived. `dispose` carries out an action on the live tables and returns false when the resource + * was left in place because something it would take with it is on hold. + */ +case class RetentionCandidate(resourceType: String, + resourceId: String, + bankId: String, + entityName: Option[String], + ownerUserId: Option[String], + triggerDate: Date, + conditionView: JValue)(val jsonCopy: () => JObject, + val dispose: (String, Disposal) => Boolean) + +/** How one resource type is found and disposed of. */ +trait RetentionResourceHandler { + def resourceType: String + + /** + * Hand each resource of the policy's space whose trigger date is on or before `cutoff`, and which + * matches the policy's condition, to `visit`, oldest row first, until `visit` returns false. + * Resources may be disposed of from inside `visit`. + */ + def scan(policy: RetentionPolicy, cutoff: Date)(visit: RetentionCandidate => Boolean): Unit +} + +object RetentionResources { + + val handlers: Map[String, RetentionResourceHandler] = List( + DynamicEntityRecordRetention, FileRetention, ChatMessageRetention, ChatRoomRetention, CustomerRetention, ConsentRetention + ).map(h => h.resourceType -> h).toMap + + private val PageSize = 500 + + /** + * Read a table in pages ordered by primary key, so that rows deleted while the scan runs never make + * it skip or repeat one. `visit` returns false to stop. + */ + private[retention] def scanById[T <: LongKeyedMapper[T]](meta: LongKeyedMetaMapper[T], idField: MappedField[Long, T], + params: List[QueryParam[T]])(visit: T => Boolean): Unit = { + var lastId = -1L + var go = true + while (go) { + val page = meta.findAll((params ::: List(By_>(idField, lastId), OrderBy(idField, Ascending), MaxRows[T](PageSize))): _*) + val rows = page.iterator + while (go && rows.hasNext) { + val row = rows.next() + lastId = row.primaryKeyField.get + go = visit(row) + } + if (page.size < PageSize) go = false + } + } + + private[retention] def parse(json: String): JValue = + Try(JsonAliases.parse(Option(json).getOrElse(""))).getOrElse(JNothing) + + private[retention] def nonEmpty(values: String*): Option[String] = values.find(v => v != null && v.nonEmpty) +} + +import code.retention.RetentionResources._ + +object DynamicEntityRecordRetention extends RetentionResourceHandler { + val resourceType: String = ResourceDynamicEntityRecord + + def scan(policy: RetentionPolicy, cutoff: Date)(visit: RetentionCandidate => Boolean): Unit = { + val space = policy.bankId + val entityName = policy.entityName.getOrElse("") + val dateFilter: List[QueryParam[DynamicData]] = policy.dateBasis match { + case BasisCreated => List(By_<=(DynamicData.createdAt, cutoff)) + case BasisUpdated => List(By_<=(DynamicData.updatedAt, cutoff)) + case _ => Nil + } + scanById(DynamicData, DynamicData.id, + By(DynamicData.BankId, space) :: By(DynamicData.DynamicEntityName, entityName) :: dateFilter) { row => + val data = parse(row.DataJson.get) + val trigger: Option[Date] = policy.dateBasis match { + case BasisCreated => Option(row.createdAt.get) + case BasisUpdated => Option(row.updatedAt.get) + case basis => dateIn(data, basis.stripPrefix(BasisFieldPrefix).trim) + } + trigger.filter(t => !t.after(cutoff) && conditionMatches(policy.condition, data)) match { + case Some(t) => visit(candidate(policy, row, data, t)) + case None => true + } + } + } + + private def candidate(policy: RetentionPolicy, row: DynamicData, data: JValue, trigger: Date): RetentionCandidate = { + val space = policy.bankId + val entityName = row.DynamicEntityName.get + val recordId = row.DynamicDataId.get + val owner = + if (row.IsPersonalEntity.get) nonEmpty(row.UserId.get) + else nonEmpty(row.CreatedByOnBehalfOfUserId.get, row.CreatedByUserId.get, row.UserId.get) + lazy val self: RetentionCandidate = RetentionCandidate(resourceType, recordId, space, Some(entityName), owner, trigger, data)( + jsonCopy = () => + ("dynamic_data_id" -> recordId) ~ + ("entity_name" -> entityName) ~ + ("bank_id" -> space) ~ + ("user_id" -> text(row.UserId.get)) ~ + ("is_personal_entity" -> row.IsPersonalEntity.get) ~ + ("created_at" -> iso(row.createdAt.get)) ~ + ("updated_at" -> iso(row.updatedAt.get)) ~ + ("created_by_user_id" -> text(row.CreatedByUserId.get)) ~ + ("created_by_on_behalf_of_user_id" -> text(row.CreatedByOnBehalfOfUserId.get)) ~ + ("updated_by_user_id" -> text(row.UpdatedByUserId.get)) ~ + ("updated_by_on_behalf_of_user_id" -> text(row.UpdatedByOnBehalfOfUserId.get)) ~ + ("file_attachments" -> FileRetention.attachmentsJson(space, entityName, recordId)) ~ + ("data" -> data), + dispose = (action, disposal) => { + val archiveId = if (action == ActionArchive) Some(disposal.archive(self, None, None)) else None + FileRetention.disposeAttached(space, entityName, recordId, policy.cascade, action, disposal, resourceType) + DynamicDataAccess.bulkDelete_!!(By(DynamicDataAccess.BankId, space), By(DynamicDataAccess.EntityName, entityName), + By(DynamicDataAccess.DynamicDataId, recordId)) + row.delete_! + // Keep the SQL projection of the entity in step, as the record endpoints do. No webhook is sent: + // a retention run is not a write through the endpoints. + code.api.dynamic.entity.projection.ProjectionDualWrite.onDelete( + Option(space).filterNot(_ == SystemSpace), entityName, recordId) + disposal.register(self, action, archiveId, None) + true + } + ) + self + } +} + +object FileRetention extends RetentionResourceHandler { + val resourceType: String = ResourceFile + + def scan(policy: RetentionPolicy, cutoff: Date)(visit: RetentionCandidate => Boolean): Unit = + scanById(File, File.id, List(ByList(File.BankId, storedBankIds(policy.bankId)), By_<=(File.CreatedAt, cutoff))) { file => + val c = candidate(file, None) + if (conditionMatches(policy.condition, c.conditionView)) visit(c) else true + } + + def attachmentsJson(bankId: String, recordType: String, recordId: String): JValue = + JArray(FileAttachment.findAll(ByList(FileAttachment.BankId, storedBankIds(bankId)), By(FileAttachment.RecordType, recordType), + By(FileAttachment.RecordId, recordId)).map(a => + ("file_attachment_id" -> a.fileAttachmentId) ~ ("file_id" -> a.fileId): JValue)) + + private def summary(file: File): JObject = + ("file_id" -> file.fileId) ~ + ("bank_id" -> file.bankId) ~ + ("file_name" -> text(file.FileName.get)) ~ + ("media_type" -> text(file.MediaType.get)) ~ + ("size_in_bytes" -> file.SizeInBytes.get) ~ + ("sha256" -> text(file.Sha256.get)) ~ + ("user_id" -> text(file.UserId.get)) ~ + ("on_behalf_of_user_id" -> text(file.OnBehalfOfUserId.get)) ~ + ("created_at" -> iso(file.CreatedAt.get)) + + private def candidate(file: File, parent: Option[(String, String)]): RetentionCandidate = + RetentionCandidate(resourceType, file.fileId, file.bankId, None, nonEmpty(file.ownerUserId), file.CreatedAt.get, summary(file))( + jsonCopy = () => summary(file) ~ + ("attachments" -> JArray(FileAttachment.findAll(By(FileAttachment.FileId, file.fileId)).map(a => + ("file_attachment_id" -> a.fileAttachmentId) ~ ("record_type" -> a.recordType) ~ ("record_id" -> a.recordId): JValue))) ~ + ("access" -> JArray(FileAccess.findAll(By(FileAccess.FileId, file.fileId)).map(a => + ("grantee_user_id" -> a.granteeUserId) ~ ("granted_by_user_id" -> a.grantedByUserId): JValue))), + dispose = (action, disposal) => { + disposeFile(file, action, disposal, parent) + true + } + ) + + /** Archive or delete one file with its bytes, its attachments and its access list. */ + private def disposeFile(file: File, action: String, disposal: Disposal, parent: Option[(String, String)]): Unit = { + val self = candidate(file, parent) + val fileAction = if (action == ActionArchive) ActionArchive else ActionDelete + val archiveId = + if (fileAction == ActionArchive) { + val bytes = FileContent.find(By(FileContent.FileId, file.fileId)).toOption.flatMap(row => Option(row.Content.get)) + Some(disposal.archive(self, bytes, parent)) + } else None + FileAccess.bulkDelete_!!(By(FileAccess.FileId, file.fileId)) + FileAttachment.bulkDelete_!!(By(FileAttachment.FileId, file.fileId)) + FileContent.bulkDelete_!!(By(FileContent.FileId, file.fileId)) + file.delete_! + disposal.register(self, fileAction, archiveId, parent) + } + + /** + * The files attached to a record that is being disposed of. The record's attachments always go. + * With `cascade`, each file goes too, with the record's action (archive stays archive; delete and + * anonymise delete the file) — unless it is still attached to another record, which keeps it, or + * it is on hold, which keeps it and is counted as held. + */ + def disposeAttached(bankId: String, recordType: String, recordId: String, cascade: Boolean, action: String, + disposal: Disposal, parentType: String): Unit = { + val attachments = FileAttachment.findAll(ByList(FileAttachment.BankId, storedBankIds(bankId)), + By(FileAttachment.RecordType, recordType), By(FileAttachment.RecordId, recordId)) + attachments.foreach { attachment => + attachment.delete_! + if (cascade && FileAttachment.count(By(FileAttachment.FileId, attachment.fileId)) == 0) { + File.find(By(File.FileId, attachment.fileId)).foreach { file => + if (disposal.isHeld(resourceType, file.fileId, None)) disposal.countHeld() + else disposeFile(file, action, disposal, Some(parentType -> recordId)) + } + } + } + } +} + +object ChatMessageRetention extends RetentionResourceHandler { + val resourceType: String = ResourceChatMessage + + def scan(policy: RetentionPolicy, cutoff: Date)(visit: RetentionCandidate => Boolean): Unit = { + val rooms = ChatRoom.findAll(ByList(ChatRoom.BankId, storedBankIds(policy.bankId))) + val dateFilter: QueryParam[ChatMessage] = + if (policy.dateBasis == BasisUpdated) By_<=(ChatMessage.updatedAt, cutoff) else By_<=(ChatMessage.createdAt, cutoff) + var go = true + rooms.map(_.chatRoomId).grouped(500).foreach { roomIds => + if (go) scanById(ChatMessage, ChatMessage.id, List(ByList(ChatMessage.ChatRoomId, roomIds), dateFilter)) { message => + val trigger = if (policy.dateBasis == BasisUpdated) message.updatedAt.get else message.createdAt.get + val c = candidate(policy.bankId, message, trigger) + go = if (conditionMatches(policy.condition, c.conditionView)) visit(c) else true + go + } + } + } + + private[retention] def summary(message: ChatMessage): JObject = + ("chat_message_id" -> message.chatMessageId) ~ + ("chat_room_id" -> message.chatRoomId) ~ + ("sender_user_id" -> text(message.SenderUserId.get)) ~ + ("sender_consumer_id" -> text(message.SenderConsumerId.get)) ~ + ("message_type" -> text(message.MessageType.get)) ~ + ("reply_to_message_id" -> text(message.ReplyToMessageId.get)) ~ + ("thread_id" -> text(message.ThreadId.get)) ~ + ("is_deleted" -> message.IsDeleted.get) ~ + ("created_at" -> iso(message.createdAt.get)) ~ + ("updated_at" -> iso(message.updatedAt.get)) + + private[retention] def fullJson(message: ChatMessage): JObject = + summary(message) ~ + ("content" -> text(message.Content.get)) ~ + ("mentioned_user_ids" -> text(message.MentionedUserIds.get)) ~ + ("reactions" -> JArray(Reaction.findAll(By(Reaction.ChatMessageId, message.chatMessageId)).map(r => + ("reaction_id" -> r.reactionId) ~ ("user_id" -> r.userId) ~ ("emoji" -> r.emoji) ~ ("created_at" -> iso(r.createdAt.get)): JValue))) + + private[retention] def candidate(bankId: String, message: ChatMessage, trigger: Date): RetentionCandidate = { + lazy val self: RetentionCandidate = RetentionCandidate(resourceType, message.chatMessageId, bankId, None, + nonEmpty(message.SenderUserId.get), trigger, summary(message))( + jsonCopy = () => fullJson(message), + dispose = (action, disposal) => { + val archiveId = if (action == ActionArchive) Some(disposal.archive(self, None, None)) else None + Reaction.bulkDelete_!!(By(Reaction.ChatMessageId, message.chatMessageId)) + message.delete_! + // The room shows a preview of its newest message; it must not keep the text of one disposed of. + ChatRoom.find(By(ChatRoom.ChatRoomId, message.chatRoomId)).foreach { room => + val lastAt = Option(room.LastMessageAt.get) + if (lastAt.forall(at => Option(message.createdAt.get).forall(created => !at.after(created)))) + room.LastMessagePreview("").LastMessageSenderUsername("").saveMe() + } + disposal.register(self, action, archiveId, None) + true + } + ) + self + } +} + +object ChatRoomRetention extends RetentionResourceHandler { + val resourceType: String = ResourceChatRoom + + def scan(policy: RetentionPolicy, cutoff: Date)(visit: RetentionCandidate => Boolean): Unit = { + val dateFilter: List[QueryParam[ChatRoom]] = policy.dateBasis match { + case BasisCreated => List(By_<=(ChatRoom.createdAt, cutoff)) + case BasisUpdated => List(By_<=(ChatRoom.updatedAt, cutoff)) + case _ => Nil + } + scanById(ChatRoom, ChatRoom.id, ByList(ChatRoom.BankId, storedBankIds(policy.bankId)) :: dateFilter) { room => + val trigger: Option[Date] = policy.dateBasis match { + case BasisCreated => Option(room.createdAt.get) + case BasisUpdated => Option(room.updatedAt.get) + case _ => Option(room.LastMessageAt.get).orElse(Option(room.createdAt.get)) + } + val c = trigger.filterNot(_.after(cutoff)).map(t => candidate(policy.bankId, room, t)) + c.filter(c => conditionMatches(policy.condition, c.conditionView)).map(visit).getOrElse(true) + } + } + + private def summary(room: ChatRoom): JObject = + ("chat_room_id" -> room.chatRoomId) ~ + ("bank_id" -> text(room.BankId.get)) ~ + ("name" -> text(room.Name.get)) ~ + ("created_by_user_id" -> text(room.CreatedByUserId.get)) ~ + ("is_open_room" -> room.IsOpenRoom.get) ~ + ("is_archived" -> room.IsArchived.get) ~ + ("last_message_at" -> iso(room.LastMessageAt.get)) ~ + ("created_at" -> iso(room.createdAt.get)) ~ + ("updated_at" -> iso(room.updatedAt.get)) + + private def candidate(bankId: String, room: ChatRoom, trigger: Date): RetentionCandidate = { + lazy val self: RetentionCandidate = RetentionCandidate(resourceType, room.chatRoomId, bankId, None, nonEmpty(room.CreatedByUserId.get), trigger, summary(room))( + // The joining key is left out: it is how people get into the room, not a record of it. + jsonCopy = () => summary(room) ~ + ("description" -> text(room.Description.get)) ~ + ("participants" -> JArray(Participant.findAll(By(Participant.ChatRoomId, room.chatRoomId)).map(p => + ("participant_id" -> p.participantId) ~ ("user_id" -> text(p.UserId.get)) ~ ("consumer_id" -> text(p.ConsumerId.get)) ~ + ("permissions" -> text(p.Permissions.get)) ~ ("joined_at" -> iso(p.JoinedAt.get)): JValue))) ~ + ("messages" -> JArray(ChatMessage.findAll(By(ChatMessage.ChatRoomId, room.chatRoomId), OrderBy(ChatMessage.id, Ascending)) + .map(m => ChatMessageRetention.fullJson(m): JValue))), + dispose = (action, disposal) => { + val messages = ChatMessage.findAll(By(ChatMessage.ChatRoomId, room.chatRoomId)) + // A room goes with all its messages, so a room holding a message on hold stays whole. + if (messages.exists(m => disposal.isHeld(ResourceChatMessage, m.chatMessageId, None))) false + else { + val archiveId = if (action == ActionArchive) Some(disposal.archive(self, None, None)) else None + messages.foreach { m => + Reaction.bulkDelete_!!(By(Reaction.ChatMessageId, m.chatMessageId)) + m.delete_! + disposal.register(ChatMessageRetention.candidate(bankId, m, m.createdAt.get), action, archiveId, Some(resourceType -> room.chatRoomId)) + } + Participant.bulkDelete_!!(By(Participant.ChatRoomId, room.chatRoomId)) + room.delete_! + disposal.register(self, action, archiveId, None) + true + } + } + ) + self + } +} + +object CustomerRetention extends RetentionResourceHandler { + val resourceType: String = ResourceCustomer + /** Files about a Customer are attached with this record type. */ + val FileRecordType = "customer" + + def scan(policy: RetentionPolicy, cutoff: Date)(visit: RetentionCandidate => Boolean): Unit = { + val dateFilter: QueryParam[MappedCustomer] = + if (policy.dateBasis == BasisUpdated) By_<=(MappedCustomer.updatedAt, cutoff) else By_<=(MappedCustomer.createdAt, cutoff) + scanById(MappedCustomer, MappedCustomer.id, List(By(MappedCustomer.mBank, policy.bankId), dateFilter)) { customer => + // A Customer already anonymised is not anonymised again on every run. + val alreadyDone = policy.action == ActionAnonymise && customer.mLegalName.get == AnonymisedLegalName + val trigger = Option(if (policy.dateBasis == BasisUpdated) customer.updatedAt.get else customer.createdAt.get) + val c = trigger.filterNot(_ => alreadyDone).map(t => candidate(policy, customer, t)) + c.filter(c => conditionMatches(policy.condition, c.conditionView)).map(visit).getOrElse(true) + } + } + + private def summary(customer: MappedCustomer): JObject = + ("customer_id" -> customer.customerId) ~ + ("bank_id" -> customer.bankId) ~ + ("customer_number" -> text(customer.mNumber.get)) ~ + ("customer_type" -> text(customer.mCustomerType.get)) ~ + ("branch_id" -> text(customer.mBranchId.get)) ~ + ("kyc_status" -> customer.mKycStatus.get) ~ + ("created_at" -> iso(customer.createdAt.get)) ~ + ("updated_at" -> iso(customer.updatedAt.get)) + + private def fullJson(customer: MappedCustomer): JObject = { + val pk = customer.id.get + summary(customer) ~ + ("legal_name" -> text(customer.mLegalName.get)) ~ + ("mobile_phone_number" -> text(customer.mMobileNumber.get)) ~ + ("email" -> text(customer.mEmail.get)) ~ + ("face_image_url" -> text(customer.mFaceImageUrl.get)) ~ + ("date_of_birth" -> iso(customer.mDateOfBirth.get)) ~ + ("relationship_status" -> text(customer.mRelationshipStatus.get)) ~ + ("dependants" -> customer.mDependents.get) ~ + ("highest_education_attained" -> text(customer.mHighestEducationAttained.get)) ~ + ("employment_status" -> text(customer.mEmploymentStatus.get)) ~ + ("credit_rating" -> (("rating" -> text(customer.mCreditRating.get)) ~ ("source" -> text(customer.mCreditSource.get)))) ~ + ("credit_limit" -> (("currency" -> text(customer.mCreditLimitCurrency.get)) ~ ("amount" -> text(customer.mCreditLimitAmount.get)))) ~ + ("last_ok_date" -> iso(customer.mLastOkDate.get)) ~ + ("title" -> text(customer.mTitle.get)) ~ + ("name_suffix" -> text(customer.mNameSuffix.get)) ~ + ("parent_customer_id" -> text(customer.mParentCustomerId.get)) ~ + ("customer_attributes" -> JArray(MappedCustomerAttribute.findAll(By(MappedCustomerAttribute.mCustomerId, customer.customerId)).map(a => + ("customer_attribute_id" -> a.customerAttributeId) ~ ("name" -> text(a.mName.get)) ~ ("type" -> text(a.mType.get)) ~ + ("value" -> text(a.mValue.get)): JValue))) ~ + ("addresses" -> JArray(MappedCustomerAddress.findAll(By(MappedCustomerAddress.mCustomerId, pk)).map(a => + ("customer_address_id" -> a.customerAddressId) ~ ("line_1" -> text(a.line1)) ~ ("line_2" -> text(a.line2)) ~ + ("line_3" -> text(a.line3)) ~ ("city" -> text(a.city)) ~ ("county" -> text(a.county)) ~ ("state" -> text(a.state)) ~ + ("postcode" -> text(a.postcode)) ~ ("country_code" -> text(a.countryCode)) ~ ("tags" -> text(a.tags)) ~ + ("status" -> text(a.status)): JValue))) ~ + ("dependant_dates_of_birth" -> JArray(MappedCustomerDependant.findAll(By(MappedCustomerDependant.mCustomer, pk)) + .map(d => iso(d.mDateOfBirth.get)))) ~ + ("tax_residences" -> JArray(MappedTaxResidence.findAll(By(MappedTaxResidence.mCustomerId, pk)).map(t => + ("tax_residence_id" -> t.taxResidenceId) ~ ("domain" -> text(t.domain)) ~ ("tax_number" -> text(t.taxNumber)): JValue))) ~ + ("user_customer_links" -> JArray(userLinks(customer).map(l => + ("user_customer_link_id" -> l.userCustomerLinkId) ~ ("user_id" -> l.userId) ~ ("is_active" -> l.isActive) ~ + ("date_inserted" -> iso(l.dateInserted)): JValue))) ~ + ("customer_account_links" -> JArray(CustomerAccountLink.findAll(By(CustomerAccountLink.CustomerId, customer.customerId)).map(l => + ("customer_account_link_id" -> l.customerAccountLinkId) ~ ("bank_id" -> text(l.BankId.get)) ~ + ("account_id" -> text(l.AccountId.get)) ~ ("relationship_type" -> text(l.RelationshipType.get)): JValue))) ~ + ("file_attachments" -> FileRetention.attachmentsJson(customer.bankId, FileRecordType, customer.customerId)) + } + + private def userLinks(customer: MappedCustomer): List[MappedUserCustomerLink] = + MappedUserCustomerLink.findAll(By(MappedUserCustomerLink.mCustomerId, customer.customerId)) + + private def candidate(policy: RetentionPolicy, customer: MappedCustomer, trigger: Date): RetentionCandidate = { + val owner = userLinks(customer).sortBy(l => !l.isActive).headOption.map(_.userId).filter(_.nonEmpty) + lazy val self: RetentionCandidate = RetentionCandidate(resourceType, customer.customerId, customer.bankId, None, owner, trigger, summary(customer))( + jsonCopy = () => fullJson(customer), + dispose = (action, disposal) => { + val archiveId = if (action == ActionArchive) Some(disposal.archive(self, None, None)) else None + if (policy.cascade || action != ActionAnonymise) + FileRetention.disposeAttached(customer.bankId, FileRecordType, customer.customerId, policy.cascade, action, disposal, resourceType) + removePersonalRecords(customer) + if (action == ActionAnonymise) anonymise(customer) + else { + CustomerAccountLink.bulkDelete_!!(By(CustomerAccountLink.CustomerId, customer.customerId)) + customer.delete_! + } + disposal.register(self, action, archiveId, None) + true + } + ) + self + } + + /** The records that describe the person: attributes, addresses, dependants, tax residences and the links to Users. */ + private def removePersonalRecords(customer: MappedCustomer): Unit = { + val pk = customer.id.get + MappedCustomerAttribute.bulkDelete_!!(By(MappedCustomerAttribute.mCustomerId, customer.customerId)) + MappedCustomerAddress.bulkDelete_!!(By(MappedCustomerAddress.mCustomerId, pk)) + MappedCustomerDependant.bulkDelete_!!(By(MappedCustomerDependant.mCustomer, pk)) + MappedTaxResidence.bulkDelete_!!(By(MappedTaxResidence.mCustomerId, pk)) + MappedUserCustomerLink.bulkDelete_!!(By(MappedUserCustomerLink.mCustomerId, customer.customerId)) + } + + /** + * Clear the personal fields and keep the row, so that the customer id, the bank, the customer + * number and the links to accounts stay valid for whatever refers to them. + */ + private def anonymise(customer: MappedCustomer): Unit = + customer + .mLegalName(AnonymisedLegalName) + .mMobileNumber("") + .mEmail("") + .mFaceImageUrl("") + .mFaceImageTime(null) + .mDateOfBirth(null) + .mRelationshipStatus("") + .mDependents(0) + .mHighestEducationAttained("") + .mEmploymentStatus("") + .mCreditRating("") + .mCreditSource("") + .mCreditLimitCurrency("") + .mCreditLimitAmount("") + .mTitle("") + .mNameSuffix("") + .saveMe() +} + +object ConsentRetention extends RetentionResourceHandler { + val resourceType: String = ResourceConsent + + /** + * A Consent belongs to the banks its items name. A policy of a bank governs the Consents that name + * that bank and no other; a policy of SYS governs the Consents that name no bank at all. + */ + private def belongsTo(bankId: String, items: List[ConsentItem]): Boolean = { + val banks = items.map(i => Option(i.bankId.get).getOrElse("")).filter(b => b.nonEmpty && b != SystemSpace).distinct + if (bankId == SystemSpace) banks.isEmpty else banks == List(bankId) + } + + def scan(policy: RetentionPolicy, cutoff: Date)(visit: RetentionCandidate => Boolean): Unit = { + val dateFilter: QueryParam[MappedConsent] = policy.dateBasis match { + case BasisCreated => By_<=(MappedConsent.createdAt, cutoff) + case BasisUpdated => By_<=(MappedConsent.updatedAt, cutoff) + case _ => By_<=(MappedConsent.mJwtExpiresAt, cutoff) + } + scanById(MappedConsent, MappedConsent.id, List(dateFilter)) { consent => + val referenceId = Option(consent.mConsentReferenceId.get).getOrElse("") + lazy val items = ConsentItem.findAll(By(ConsentItem.consentReferenceId, referenceId)) + val trigger: Option[Date] = policy.dateBasis match { + case BasisCreated => Option(consent.createdAt.get) + case BasisUpdated => Option(consent.updatedAt.get) + case _ => Option(consent.mJwtExpiresAt.get) + } + if (referenceId.isEmpty || trigger.isEmpty || !belongsTo(policy.bankId, items)) true + else { + val c = candidate(policy.bankId, consent, referenceId, items, trigger.get) + if (conditionMatches(policy.condition, c.conditionView)) visit(c) else true + } + } + } + + // The consent id, the secret, the challenge and the JWT are credentials and are never copied: the + // Consent is identified by its consent_reference_id everywhere in retention. + private def summary(consent: MappedConsent, referenceId: String): JObject = + ("consent_reference_id" -> referenceId) ~ + ("user_id" -> text(consent.mUserId.get)) ~ + ("consumer_id" -> text(consent.mConsumerId.get)) ~ + ("status" -> text(consent.mStatus.get)) ~ + ("api_standard" -> text(consent.mApiStandard.get)) ~ + ("api_version" -> text(consent.mApiVersion.get)) ~ + ("created_at" -> iso(consent.createdAt.get)) ~ + ("updated_at" -> iso(consent.updatedAt.get)) ~ + ("jwt_expires_at" -> iso(consent.mJwtExpiresAt.get)) + + private def candidate(bankId: String, consent: MappedConsent, referenceId: String, items: List[ConsentItem], trigger: Date): RetentionCandidate = { + lazy val self: RetentionCandidate = RetentionCandidate(resourceType, referenceId, bankId, None, nonEmpty(consent.mUserId.get), trigger, summary(consent, referenceId))( + jsonCopy = () => summary(consent, referenceId) ~ + ("status_updated_at" -> iso(consent.mStatusUpdateDateTime.get)) ~ + ("valid_until" -> iso(consent.mValidUntil.get)) ~ + ("expiration_date_time" -> iso(consent.mExpirationDateTime.get)) ~ + ("recurring_indicator" -> consent.mRecurringIndicator.get) ~ + ("frequency_per_day" -> consent.mFrequencyPerDay.get) ~ + ("note" -> text(consent.mNote.get)) ~ + ("consent_items" -> JArray(items.map(i => + ("item_type" -> text(i.itemType.get)) ~ ("bank_id" -> text(i.bankId.get)) ~ ("account_id" -> text(i.accountId.get)) ~ + ("view_id" -> text(i.viewId.get)) ~ ("role_name" -> text(i.roleName.get)): JValue))), + dispose = (action, disposal) => { + val archiveId = if (action == ActionArchive) Some(disposal.archive(self, None, None)) else None + ConsentItem.bulkDelete_!!(By(ConsentItem.consentReferenceId, referenceId)) + MappedConsentAuthContext.bulkDelete_!!(By(MappedConsentAuthContext.ConsentId, consent.mConsentId.get)) + consent.delete_! + disposal.register(self, action, archiveId, None) + true + } + ) + self + } +} diff --git a/obp-api/src/main/scala/code/scheduler/ConsentScheduler.scala b/obp-api/src/main/scala/code/scheduler/ConsentScheduler.scala index 82e893139b..db7f5f490e 100644 --- a/obp-api/src/main/scala/code/scheduler/ConsentScheduler.scala +++ b/obp-api/src/main/scala/code/scheduler/ConsentScheduler.scala @@ -78,6 +78,16 @@ object ConsentScheduler extends MdcLoggable { logger.warn("|---> Skipping expiredObpConsents task: obp_expired_consents_interval_in_seconds set to 0") } + // OBP consents still INITIATED: their challenge was never answered (or could not be), so they are rejected + // after a while instead of waiting forever. An app that reuses a pending consent is then not blocked by one. + val obpUnfinishedInterval = APIUtil.getPropsAsIntValue("obp_unfinished_consents_interval_in_seconds", 593) + if (obpUnfinishedInterval > 0) { + val time = APIUtil.getPropsAsIntValue("obp_unfinished_consents_time_in_seconds", 21600) + SchedulerUtil.startTask(interval = obpUnfinishedInterval, () => unfinishedObpConsents(time), initialDelay) + initialDelay = initialDelay + 10 + } else { + logger.warn("|---> Skipping unfinishedObpConsents task: obp_unfinished_consents_interval_in_seconds set to 0") + } // UK Open Banking. checkUKConsent (ConsentUtil.scala) also reactively rejects an expired // AUTHORISED consent on every request regardless of this task's timing -- this proactive // sweep just keeps the stored status accurate for GET/dashboard purposes. @@ -212,6 +222,42 @@ object ConsentScheduler extends MdcLoggable { case Success(_) => logger.debug("|---> Task executed successfully") } } + /** + * Rejects OBP consents that are still INITIATED and have not changed for `seconds`. An INITIATED consent waits + * only for the answer to its challenge; it never granted any access, so there is nothing to give back. + */ + private[scheduler] def unfinishedObpConsents(seconds: Int): Unit = { + Try { + logger.debug("|---> Checking for unfinished OBP consents...") + val unfinishedConsents = MappedConsent.findAll( + By(MappedConsent.mStatus, ConsentStatus.INITIATED.toString), + By(MappedConsent.mApiStandard, ApiStandards.obp.toString), + By_<(MappedConsent.updatedAt, SchedulerUtil.someSecondsAgo(seconds)) + ) + logger.debug(s"|---> Found ${unfinishedConsents.size} unfinished OBP consents") + unfinishedConsents.foreach { consent => + Try { + val message = s"|---> Changed status from ${consent.status} to ${ConsentStatus.REJECTED} for consent ID: ${consent.id}: not accepted within $seconds seconds" + val newNote = s"$currentDate\n$message\n" + Option(consent.note).getOrElse("") + val rows = code.bankconnectors.DoobieConsentSchedulerQueries.conditionallyUpdateStatus( + consentPrimaryKey = consent.id.get, + guardStatus = ConsentStatus.INITIATED.toString, + newStatus = ConsentStatus.REJECTED.toString, + newNote = newNote + ) + if (rows > 0) logger.warn(message) + else logger.debug(s"|---> Skipped stale update for OBP consent ${consent.id}: status already changed") + } match { + case Failure(ex) => logger.error(s"Failed to update consent ID: ${consent.id}", ex) + case Success(_) => // Already logged + } + } + } match { + case Failure(ex) => logger.error("Error in unfinishedObpConsents!", ex) + case Success(_) => logger.debug("|---> Task executed successfully") + } + } + private def expiredUKConsents(): Unit = { Try { logger.debug("|---> Checking for expired UK Open Banking consents...") diff --git a/obp-api/src/main/scala/code/scheduler/RetentionScheduler.scala b/obp-api/src/main/scala/code/scheduler/RetentionScheduler.scala new file mode 100644 index 0000000000..dcbb4d763b --- /dev/null +++ b/obp-api/src/main/scala/code/scheduler/RetentionScheduler.scala @@ -0,0 +1,127 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ + +package code.scheduler + +import java.util.concurrent.TimeUnit + +import code.actorsystem.ObpActorSystem +import code.api.Constant +import code.api.util.APIUtil.generateUUID +import code.retention.Retention.{ModeExecute, StatusPending} +import code.retention.{RetentionEngine, RetentionPolicy, RetentionProps, RetentionRun} +import code.util.Helper.MdcLoggable +import net.liftweb.common.Full +import net.liftweb.mapper.By + +import scala.concurrent.duration._ +import scala.util.control.NonFatal + +/** + * Runs the enabled retention policies on a timer, modelled on [[MetricsArchiveScheduler]]. Started + * from Boot only when retention.scheduler.enabled is true, every + * retention.scheduler.interval_in_seconds. + * + * Each tick, for every enabled policy with something due: a policy that does not require approval + * is executed, unless the props switch its action off (then it is skipped and logged, not recorded + * as a refused run every tick); a policy that requires approval gets one pending run, which waits + * for a user to approve it. Then, with retention.delete.enabled, archived resources whose archive + * period has ended are deleted from the archive. + * + * A JobScheduler lock row keeps two nodes, or two ticks, from running at once. + */ +object RetentionScheduler extends MdcLoggable { + + private lazy val actorSystem = ObpActorSystem.localActorSystem + implicit lazy val executor = actorSystem.dispatcher + private lazy val scheduler = actorSystem.scheduler + private val jobName = "RetentionScheduler" + private val apiInstanceId = Constant.ApiInstanceId + + /** Start the scheduler when retention.scheduler.enabled is true (it is false by default). Returns whether it started. */ + def startIfEnabled(): Boolean = + if (RetentionProps.schedulerEnabled) { + start(intervalInSeconds = RetentionProps.schedulerIntervalInSeconds) + true + } else { + logger.info("RetentionScheduler not started: retention.scheduler.enabled is false") + false + } + + def start(intervalInSeconds: Long): Unit = { + logger.info(s"RetentionScheduler.start: every $intervalInSeconds seconds") + // This instance cannot have a run in progress at boot, so a lock row of its own is a leftover. + JobScheduler.findAll(By(JobScheduler.ApiInstanceId, apiInstanceId), By(JobScheduler.Name, jobName)).foreach(_.delete_!) + scheduler.schedule( + initialDelay = Duration(intervalInSeconds, TimeUnit.SECONDS), + interval = Duration(intervalInSeconds, TimeUnit.SECONDS), + runnable = new Runnable { + def run(): Unit = { runOnce(); () } + } + ) + } + + /** One tick. Returns false when it was skipped because another tick holds the lock. */ + def runOnce(): Boolean = + JobScheduler.find(By(JobScheduler.Name, jobName)) match { + case Full(job) => + logger.info(s"RetentionScheduler.runOnce skipped: job ${job.JobId.get} started at ${job.createdAt.get} on ${job.ApiInstanceId.get} is in progress") + false + case _ => + val job = JobScheduler.create.JobId(generateUUID()).Name(jobName).ApiInstanceId(apiInstanceId).saveMe() + try { + RetentionPolicy.findAll(By(RetentionPolicy.Enabled, true)).foreach(runPolicy) + if (RetentionProps.deleteEnabled) RetentionEngine.deleteAllDueFromArchive() + } catch { + case NonFatal(e) => logger.error("RetentionScheduler.runOnce failed", e) + } finally { + JobScheduler.delete_!(job) + } + true + } + + private def runPolicy(policy: RetentionPolicy): Unit = + try { + val id = policy.retentionPolicyId + if (RetentionEngine.isRunning(id)) logger.info(s"RetentionScheduler: policy $id skipped, a run is in progress") + else if (RetentionEngine.refusal(policy).isDefined) + logger.info(s"RetentionScheduler: policy $id skipped: ${RetentionEngine.refusal(policy).getOrElse("")}") + else if (!RetentionEngine.hasWork(policy)) () + else if (policy.requiresApproval) { + val alreadyPending = RetentionRun.find(By(RetentionRun.RetentionPolicyId, id), By(RetentionRun.Status, StatusPending)).isDefined + if (!alreadyPending) { + val run = RetentionEngine.requestRun(policy, ModeExecute, "") + logger.info(s"RetentionScheduler: policy $id has resources due; run ${run.retentionRunId} is waiting for approval") + } + } else { + val run = RetentionEngine.requestRun(policy, ModeExecute, "") + logger.info(s"RetentionScheduler: policy $id run ${run.retentionRunId} ended ${run.status}: ${run.message}") + } + } catch { + case NonFatal(e) => logger.error(s"RetentionScheduler: policy ${policy.retentionPolicyId} failed", e) + } +} diff --git a/obp-api/src/main/scala/code/organisation/Organisation.scala b/obp-api/src/main/scala/code/tenant/Tenant.scala similarity index 62% rename from obp-api/src/main/scala/code/organisation/Organisation.scala rename to obp-api/src/main/scala/code/tenant/Tenant.scala index 3e3121f079..1bdb754f42 100644 --- a/obp-api/src/main/scala/code/organisation/Organisation.scala +++ b/obp-api/src/main/scala/code/tenant/Tenant.scala @@ -25,7 +25,7 @@ TESOBE (http://www.tesobe.com/) */ -package code.organisation +package code.tenant import net.liftweb.common.Box import net.liftweb.mapper._ @@ -34,20 +34,20 @@ import com.openbankproject.commons.ExecutionContext.Implicits.global import scala.concurrent.Future -object MappedOrganisationProvider extends OrganisationProvider { +object MappedTenantProvider extends TenantProvider { - override def createOrganisation( - organisationId: String, + override def createTenant( + tenantId: String, name: String, website: Option[String], logoUrl: Option[String], status: String, visibility: String, createdByUserId: String - ): Box[OrganisationTrait] = { + ): Box[TenantTrait] = { tryo { - Organisation.create - .OrganisationId(organisationId) + Tenant.create + .TenantId(tenantId) .Name(name) .Website(website.getOrElse("")) .LogoUrl(logoUrl.getOrElse("")) @@ -58,49 +58,51 @@ object MappedOrganisationProvider extends OrganisationProvider { } } - override def getOrganisation(organisationId: String): Box[OrganisationTrait] = { - Organisation.find(By(Organisation.OrganisationId, organisationId)) + override def getTenant(tenantId: String): Box[TenantTrait] = { + Tenant.find(By(Tenant.TenantId, tenantId)) } - override def getAllOrganisations(): Future[Box[List[OrganisationTrait]]] = { + override def getTenants(): Future[Box[List[TenantTrait]]] = { Future { - tryo { Organisation.findAll() } + tryo { Tenant.findAll() } } } - override def updateOrganisation( - organisationId: String, + override def updateTenant( + tenantId: String, name: Option[String], website: Option[String], logoUrl: Option[String], status: Option[String], visibility: Option[String] - ): Box[OrganisationTrait] = { - Organisation.find(By(Organisation.OrganisationId, organisationId)).flatMap { org => + ): Box[TenantTrait] = { + Tenant.find(By(Tenant.TenantId, tenantId)).flatMap { tenant => tryo { - name.foreach(v => org.Name(v)) - website.foreach(v => org.Website(v)) - logoUrl.foreach(v => org.LogoUrl(v)) - status.foreach(v => org.Status(v)) - visibility.foreach(v => org.Visibility(v)) - org.LastUpdate(new java.util.Date()) - org.saveMe() + name.foreach(v => tenant.Name(v)) + website.foreach(v => tenant.Website(v)) + logoUrl.foreach(v => tenant.LogoUrl(v)) + status.foreach(v => tenant.Status(v)) + visibility.foreach(v => tenant.Visibility(v)) + tenant.LastUpdate(new java.util.Date()) + tenant.saveMe() } } } - override def deleteOrganisation(organisationId: String): Box[Boolean] = { - Organisation.find(By(Organisation.OrganisationId, organisationId)).flatMap { org => - tryo { org.delete_! } + override def deleteTenant(tenantId: String): Box[Boolean] = { + Tenant.find(By(Tenant.TenantId, tenantId)).flatMap { tenant => + tryo { + tenant.delete_! + } } } } -class Organisation extends OrganisationTrait with LongKeyedMapper[Organisation] with IdPK { +class Tenant extends TenantTrait with LongKeyedMapper[Tenant] with IdPK { - def getSingleton = Organisation + def getSingleton = Tenant - object OrganisationId extends MappedString(this, 64) + object TenantId extends MappedString(this, 64) object Name extends MappedString(this, 255) object Website extends MappedString(this, 1024) object LogoUrl extends MappedString(this, 1024) @@ -114,7 +116,7 @@ class Organisation extends OrganisationTrait with LongKeyedMapper[Organisation] override def defaultValue = new java.util.Date() } - override def organisationId: String = OrganisationId.get + override def tenantId: String = TenantId.get override def name: String = Name.get override def website: Option[String] = { val v = Website.get @@ -131,7 +133,7 @@ class Organisation extends OrganisationTrait with LongKeyedMapper[Organisation] override def updatedAt: java.util.Date = LastUpdate.get } -object Organisation extends Organisation with LongKeyedMetaMapper[Organisation] { - override def dbTableName = "Organisation" - override def dbIndexes = UniqueIndex(OrganisationId) :: super.dbIndexes +object Tenant extends Tenant with LongKeyedMetaMapper[Tenant] { + override def dbTableName = "Tenant" + override def dbIndexes = UniqueIndex(TenantId) :: super.dbIndexes } diff --git a/obp-api/src/main/scala/code/organisation/OrganisationTrait.scala b/obp-api/src/main/scala/code/tenant/TenantTrait.scala similarity index 67% rename from obp-api/src/main/scala/code/organisation/OrganisationTrait.scala rename to obp-api/src/main/scala/code/tenant/TenantTrait.scala index 88c9e7a5e1..612bf592ec 100644 --- a/obp-api/src/main/scala/code/organisation/OrganisationTrait.scala +++ b/obp-api/src/main/scala/code/tenant/TenantTrait.scala @@ -25,48 +25,53 @@ TESOBE (http://www.tesobe.com/) */ -package code.organisation +package code.tenant import net.liftweb.common.Box import net.liftweb.util.SimpleInjector import scala.concurrent.Future -object Organisations extends SimpleInjector { - val organisation = new Inject(() => buildOne) {} +object Tenants extends SimpleInjector { + val tenant = new Inject(() => buildOne) {} - def buildOne: OrganisationProvider = MappedOrganisationProvider + def buildOne: TenantProvider = MappedTenantProvider } -trait OrganisationProvider { - def createOrganisation( - organisationId: String, +/** + * A Tenant is a customer of the operator of this OBP instance: an organisation that uses it. Tenants belong to + * no bank; later, banks will name the Tenant they belong to. The tenant_id is + * unique and immutable. + */ +trait TenantProvider { + def createTenant( + tenantId: String, name: String, website: Option[String], logoUrl: Option[String], status: String, visibility: String, createdByUserId: String - ): Box[OrganisationTrait] + ): Box[TenantTrait] - def getOrganisation(organisationId: String): Box[OrganisationTrait] + def getTenant(tenantId: String): Box[TenantTrait] - def getAllOrganisations(): Future[Box[List[OrganisationTrait]]] + def getTenants(): Future[Box[List[TenantTrait]]] - def updateOrganisation( - organisationId: String, + def updateTenant( + tenantId: String, name: Option[String], website: Option[String], logoUrl: Option[String], status: Option[String], visibility: Option[String] - ): Box[OrganisationTrait] + ): Box[TenantTrait] - def deleteOrganisation(organisationId: String): Box[Boolean] + def deleteTenant(tenantId: String): Box[Boolean] } -trait OrganisationTrait { - def organisationId: String +trait TenantTrait { + def tenantId: String def name: String def website: Option[String] def logoUrl: Option[String] diff --git a/obp-api/src/main/scala/code/users/UserReference.scala b/obp-api/src/main/scala/code/users/UserReference.scala index 1fe9c0db9d..760c2e9536 100644 --- a/obp-api/src/main/scala/code/users/UserReference.scala +++ b/obp-api/src/main/scala/code/users/UserReference.scala @@ -162,6 +162,12 @@ object UserReference { case object PlatformApp_MarkedByUserId extends UserReference(UseAuthenticatedUserId, "code.platformapp.PlatformApp", List("MarkedByUserId"), "audit: who marked the Consumer as a platform app") case object GroupMembership_CreatedByUserId extends UserReference(UseAuthenticatedUserId, "code.group.GroupMembership", List("CreatedByUserId"), "audit: who added the member, as Entitlement_GrantedByUserId") case object AssetStatusHistory_ChangedByUserId extends UserReference(UseAuthenticatedUserId, "code.asset.AssetStatusHistory", List("ChangedByUserId"), "audit: who changed the asset's status, the question a regulator asks") + case object File_ScanStatusUpdatedByUserId extends UserReference(UseAuthenticatedUserId, "code.files.File", List("ScanStatusUpdatedByUserId"), "audit: who set the scan status, the scanner or the staff member who decided") + case object RetentionPolicy_CreatedByUserId extends UserReference(UseAuthenticatedUserId, "code.retention.RetentionPolicy", List("CreatedByUserId"), "audit: who created the retention policy") + case object RetentionHold_PlacedByUserId extends UserReference(UseAuthenticatedUserId, "code.retention.RetentionHold", List("PlacedByUserId", "ReleasedByUserId"), "audit: who placed, and who released, a legal hold") + case object RetentionRun_RequestedByUserId extends UserReference(UseAuthenticatedUserId, "code.retention.RetentionRun", List("RequestedByUserId", "ApprovedByUserId"), "audit: who requested a run and who approved it; the two must differ, which is only meaningful for the users who actually acted") + case object RetentionArchive_OwnerUserId extends UserReference(UseAuthenticatedUserId, "code.retention.RetentionArchive", List("OwnerUserId"), "copied from the archived resource as it was stored there; never written from the caller") + case object RetentionRunItem_OwnerUserId extends UserReference(UseAuthenticatedUserId, "code.retention.RetentionRunItem", List("OwnerUserId"), "disposal register: copied from the disposed resource as it was stored there; never written from the caller") // ---- UseOnBehalfOfUserId: the row belongs to the person, so it must outlive the Consent that // ---- created it. A handful of these tables keep both ids, and those name two fields. @@ -201,8 +207,10 @@ object UserReference { case object DynamicGlossaryItem_CreatedByUserId extends UserReference(UseOnBehalfOfUserId , "code.glossaryitem.DynamicGlossaryItem", List("CreatedByUserId"), "outlives the Consent that created it") case object DomainApi_CreatedByUserId extends UserReference(UseOnBehalfOfUserId , "code.domainapi.DomainApi", List("CreatedByUserId"), "the Domain API's creator; a published API outlives the Consent that created it") case object DynamicEntityWebhook_CreatedByUserId extends UserReference(UseOnBehalfOfUserId , "code.dynamicentitywebhook.DynamicEntityWebhook", List("CreatedByUserId", "CreatedByOnBehalfOfUserId"), "record both, like Counterparty: a webhook sends data out of OBP and outlives the Consent that made it, so the agent stays on CreatedByUserId (published as created_by_user_id) and the person it acted for goes to CreatedByOnBehalfOfUserId") + case object AccountCreation_CreatedByUserId extends UserReference(UseOnBehalfOfUserId , "code.accountcreation.AccountCreation", List("CreatedByUserId", "CreatedByOnBehalfOfUserId"), "record both, like Counterparty: who made the call that created the account, agent included, and the person it was made for; published in the v7.0.0 Create Account metadata. The holder is AccountHolders_User") case object Bank_CreatedByUserId extends UserReference(UseOnBehalfOfUserId , "code.model.dataAccess.MappedBank", List("CreatedByUserId"), "creator grant already resolved at the endpoint") - case object Organisation_CreatedByUserId extends UserReference(UseOnBehalfOfUserId , "code.organisation.Organisation", List("CreatedByUserId"), "outlives the Consent that created it") + case object SelfServiceBank_CreatedByUserId extends UserReference(UseOnBehalfOfUserId , "code.model.dataAccess.SelfServiceBank", List("CreatedByUserId"), "as Bank_CreatedByUserId; the Create My Bank quota counts these rows per creator") + case object Tenant_CreatedByUserId extends UserReference(UseOnBehalfOfUserId , "code.tenant.Tenant", List("CreatedByUserId"), "outlives the Consent that created it") case object Asset_CreatedByUserId extends UserReference(UseOnBehalfOfUserId , "code.asset.Asset", List("CreatedByUserId"), "outlives the Consent that created it, as RoutingScheme_CreatedByUserId; seeded rows hold system:asset-seed") case object File_UserId extends UserReference(UseOnBehalfOfUserId , "code.files.File", List("UserId", "OnBehalfOfUserId"), "record both: UserId = who uploaded (the agent), OnBehalfOfUserId = the human, who owns the file") case object FileAttachment_UserId extends UserReference(UseOnBehalfOfUserId , "code.files.FileAttachment", List("UserId", "OnBehalfOfUserId"), "record both: UserId = who attached (the agent), OnBehalfOfUserId = the human") @@ -263,6 +271,11 @@ object UserReference { PlatformApp_MarkedByUserId, GroupMembership_CreatedByUserId, AssetStatusHistory_ChangedByUserId, + RetentionPolicy_CreatedByUserId, + RetentionHold_PlacedByUserId, + RetentionRun_RequestedByUserId, + RetentionArchive_OwnerUserId, + RetentionRunItem_OwnerUserId, TransactionRequest_UserId, Entitlement_UserId, GroupMembership_UserId, @@ -298,10 +311,13 @@ object UserReference { DynamicGlossaryItem_CreatedByUserId, DomainApi_CreatedByUserId, DynamicEntityWebhook_CreatedByUserId, + AccountCreation_CreatedByUserId, Bank_CreatedByUserId, - Organisation_CreatedByUserId, + SelfServiceBank_CreatedByUserId, + Tenant_CreatedByUserId, Asset_CreatedByUserId, File_UserId, + File_ScanStatusUpdatedByUserId, FileAttachment_UserId, FileAccess_GrantedByUserId, FileAccess_GranteeUserId, @@ -359,9 +375,6 @@ object UserReference { "deferred 2026-09-16, not mechanical: agreed as record-both (like Counterparty), but the webhook " + "code has two dead paths already and is not to be touched in a hurry. todo/webhook_attribution.md.") - private val filesNotYetServed = NotYetWired.NotReachedByConsentUser( - "the Files tables (code.files) exist but no endpoint writes them yet. The endpoints must store " + - "cc.onBehalfOfUserId as the owner (ideas/ogcr_file_storage.md) and reject a consent user as grantee.") private val rejectNotEnforced = NotYetWired.StoresCaller( "Reject, not yet enforced: attributionOf already returns Failure for it (AgentDelegationTest pins " + @@ -401,13 +414,8 @@ object UserReference { CounterpartyWhereTag_User -> mechanicalBatch, ApiProductSubscription_CreatedByUserId -> mechanicalBatch, DynamicGlossaryItem_CreatedByUserId -> mechanicalBatch, - Organisation_CreatedByUserId -> mechanicalBatch, - Asset_CreatedByUserId -> NotYetWired.NotReachedByConsentUser( - "no endpoint writes the asset registry yet; only the boot seed does. Wire it with the asset write endpoints."), - File_UserId -> filesNotYetServed, - FileAttachment_UserId -> filesNotYetServed, - FileAccess_GrantedByUserId -> filesNotYetServed, - FileAccess_GranteeUserId -> filesNotYetServed, + Tenant_CreatedByUserId -> mechanicalBatch, + SelfServiceBank_CreatedByUserId -> mechanicalBatch, PayeeLookup_CreatedByUserId -> mechanicalBatch, RoutingScheme_CreatedByUserId -> mechanicalBatch, UtilityPaymentCallback_CreatedByUserId -> mechanicalBatch, @@ -438,7 +446,10 @@ object UserReference { AccountAccessRequest_TargetUserId -> NotYetWired.NotReachedByConsentUser( "explicit target, so the endpoint refuses a consent user rather than redirecting -- covered " + "by ExplicitTargetConsentUserSweepTest. The provider redirect is unreachable from the API."), - GroupMembership_UserId -> NotYetWired.NotReachedByConsentUser( + FileAccess_GranteeUserId -> NotYetWired.NotReachedByConsentUser( + "explicit target: Create File Access refuses a consent user as grantee rather than redirecting -- " + + "covered by ExplicitTargetConsentUserSweepTest. The provider redirect is unreachable from the API."), + GroupMembership_UserId -> NotYetWired.NotReachedByConsentUser( "the member is never the caller: the Group member sync endpoint reads each member's user id " + "from the Group's existing Entitlements and membership rows, so there is nothing to redirect."), Consent_UserId -> NotYetWired.StoresCaller( diff --git a/obp-api/src/main/scala/code/util/BankNameGenerator.scala b/obp-api/src/main/scala/code/util/BankNameGenerator.scala index e0bdea2562..37242e3001 100644 --- a/obp-api/src/main/scala/code/util/BankNameGenerator.scala +++ b/obp-api/src/main/scala/code/util/BankNameGenerator.scala @@ -108,6 +108,17 @@ object BankNameGenerator { private def pickRandomWord(words: List[String]): String = words(Random.nextInt(words.size)) + /** + * Whether a bank id has the shape generate() gives it: an adjective, a celestial and an animal from these lists, + * then the hex suffix. Used to recognise banks made with Create My Bank before banks recorded it. + */ + def isGenerated(bankId: String): Boolean = bankId.split("-").toList match { + case List(a, c, n, suffix) => + adjectives.contains(a) && celestials.contains(c) && animals.contains(n) && + suffix.length == suffixLength && suffix.forall(suffixAlphabet.contains(_)) + case _ => false + } + def generate(): GeneratedBankName = { val words = List(pickRandomWord(adjectives), pickRandomWord(celestials), pickRandomWord(animals)) val suffix = List.fill(suffixLength)(suffixAlphabet(Random.nextInt(suffixAlphabet.length))).mkString diff --git a/obp-api/src/test/scala/code/api/UKOpenBanking/v3_1_0/UKOpenBankingV310AisTests.scala b/obp-api/src/test/scala/code/api/UKOpenBanking/v3_1_0/UKOpenBankingV310AisTests.scala index a1e1a8b695..b75a92361b 100644 --- a/obp-api/src/test/scala/code/api/UKOpenBanking/v3_1_0/UKOpenBankingV310AisTests.scala +++ b/obp-api/src/test/scala/code/api/UKOpenBanking/v3_1_0/UKOpenBankingV310AisTests.scala @@ -27,7 +27,7 @@ TESOBE (http://www.tesobe.com/) package code.api.UKOpenBanking.v3_1_0 -import code.api.util.APIUtil.{DateWithDayFormat, ResourceDoc, UserOrApplication, buildOperationId} +import code.api.util.APIUtil.{DateWithDayFormat, ResourceDoc, UserXORApplication, buildOperationId} import code.api.util.ErrorMessages.ConsentNotFound import code.consent.Consents import com.openbankproject.commons.model.ErrorMessage @@ -82,11 +82,11 @@ class UKOpenBankingV310AisTests extends UKOpenBankingV310ServerSetup { // See the twin scenario in UKOpenBankingV401AccountInfoTests for why this is pinned: lodging is // a client-credentials call with no PSU, and the ResourceDoc default (UserOnly) would 401 it as // soon as a client-credentials token stops auto-vivifying a user. - scenario("ResourceDoc declares UserOrApplication so a PSU-less TPP call is not rejected", UKOpenBankingV310) { + scenario("ResourceDoc declares UserXORApplication so a PSU-less TPP call is not rejected", UKOpenBankingV310) { val docs = ResourceDoc.getResourceDocs( List(buildOperationId(ApiVersion.ukOpenBankingV31, "createAccountAccessConsents"))) docs should not be empty - docs.foreach(_.authMode should equal(UserOrApplication)) + docs.foreach(_.authMode should equal(UserXORApplication)) } } feature("UKOB v3.1 DELETE /account-access-consents/CONSENT_ID") { diff --git a/obp-api/src/test/scala/code/api/UKOpenBanking/v4_0_1/UKOpenBankingV401AccountInfoTests.scala b/obp-api/src/test/scala/code/api/UKOpenBanking/v4_0_1/UKOpenBankingV401AccountInfoTests.scala index 96409f8b2e..bc059dbaf9 100644 --- a/obp-api/src/test/scala/code/api/UKOpenBanking/v4_0_1/UKOpenBankingV401AccountInfoTests.scala +++ b/obp-api/src/test/scala/code/api/UKOpenBanking/v4_0_1/UKOpenBankingV401AccountInfoTests.scala @@ -28,7 +28,7 @@ TESOBE (http://www.tesobe.com/) package code.api.UKOpenBanking.v4_0_1 import code.api.Constant -import code.api.util.APIUtil.{DateWithDayFormat, ResourceDoc, UserOrApplication, buildOperationId} +import code.api.util.APIUtil.{DateWithDayFormat, ResourceDoc, UserXORApplication, buildOperationId} import code.api.util.ErrorMessages.{ConsentDoesNotMatchConsumer, ConsentDoesNotMatchUser, ConsentExpiredIssue, ConsentIdClaimMissing, ConsentNotFound} import code.api.util.{CallContext, CertificateUtil, Consent} import code.consent.{ConsentStatus, Consents} @@ -190,11 +190,11 @@ class UKOpenBankingV401AccountInfoTests extends UKOpenBankingV401ServerSetup { // silently reverting to the default -- the endpoint would keep working for as long as OAuth2 // token parsing auto-vivifies a user for a client-credentials token, and start 401ing the day // that stops. - scenario("ResourceDoc declares UserOrApplication so a PSU-less TPP call is not rejected", UKOpenBankingV401AccountInfo) { + scenario("ResourceDoc declares UserXORApplication so a PSU-less TPP call is not rejected", UKOpenBankingV401AccountInfo) { val docs = ResourceDoc.getResourceDocs( List(buildOperationId(ApiVersion.ukOpenBankingV401, "createAccountAccessConsents"))) docs should not be empty - docs.foreach(_.authMode should equal(UserOrApplication)) + docs.foreach(_.authMode should equal(UserXORApplication)) } scenario("all three datetime fields omitted -> 201, open-ended (no expiry/date restriction)", UKOpenBankingV401AccountInfo) { val bodyWithoutDates = diff --git a/obp-api/src/test/scala/code/api/UKOpenBanking/v4_0_1/UKOpenBankingV401ConsentAccessTests.scala b/obp-api/src/test/scala/code/api/UKOpenBanking/v4_0_1/UKOpenBankingV401ConsentAccessTests.scala index 5037a921d8..1dc732d13a 100644 --- a/obp-api/src/test/scala/code/api/UKOpenBanking/v4_0_1/UKOpenBankingV401ConsentAccessTests.scala +++ b/obp-api/src/test/scala/code/api/UKOpenBanking/v4_0_1/UKOpenBankingV401ConsentAccessTests.scala @@ -27,7 +27,7 @@ TESOBE (http://www.tesobe.com/) package code.api.UKOpenBanking.v4_0_1 -import code.api.util.APIUtil.{ResourceDoc, UserOrApplication, buildOperationId} +import code.api.util.APIUtil.{ResourceDoc, UserXORApplication, buildOperationId} import code.api.util.{CallContext, Consent} import code.api.util.ErrorMessages.{ConsentDoesNotMatchConsumer, ConsentDoesNotMatchUser} import code.model.UserX @@ -294,15 +294,15 @@ class UKOpenBankingV401ConsentAccessTests extends UKOpenBankingV401ServerSetup { // reached. Pinned because nothing else would notice a revert: these endpoints keep working for // as long as OAuth2 token parsing auto-vivifies a user for a client-credentials token. for (name <- List("getAccountAccessConsentsConsentId", "deleteAccountAccessConsentsConsentId")) { - scenario(s"v4.0.1 $name declares UserOrApplication", UKOpenBankingV401ConsentAccess) { + scenario(s"v4.0.1 $name declares UserXORApplication", UKOpenBankingV401ConsentAccess) { val docs = ResourceDoc.getResourceDocs(List(buildOperationId(ApiVersion.ukOpenBankingV401, name))) docs should not be empty - docs.foreach(_.authMode should equal(UserOrApplication)) + docs.foreach(_.authMode should equal(UserXORApplication)) } - scenario(s"v3.1 $name declares UserOrApplication", UKOpenBankingV401ConsentAccess) { + scenario(s"v3.1 $name declares UserXORApplication", UKOpenBankingV401ConsentAccess) { val docs = ResourceDoc.getResourceDocs(List(buildOperationId(ApiVersion.ukOpenBankingV31, name))) docs should not be empty - docs.foreach(_.authMode should equal(UserOrApplication)) + docs.foreach(_.authMode should equal(UserXORApplication)) } } } diff --git a/obp-api/src/test/scala/code/api/berlin/group/v1_3/AccountInformationServiceAISApiTest.scala b/obp-api/src/test/scala/code/api/berlin/group/v1_3/AccountInformationServiceAISApiTest.scala index fd24972860..80155de1c1 100644 --- a/obp-api/src/test/scala/code/api/berlin/group/v1_3/AccountInformationServiceAISApiTest.scala +++ b/obp-api/src/test/scala/code/api/berlin/group/v1_3/AccountInformationServiceAISApiTest.scala @@ -980,15 +980,15 @@ class AccountInformationServiceAISApiTest extends BerlinGroupConsentFixtures { // // Pinned because nothing else would notice a revert: these keep working for as long as OAuth2 // token parsing auto-vivifies a user for a client-credentials token, and start 401ing the day - // that stops. The consent endpoints in the same file have been UserOrApplication all along, so + // that stops. The consent endpoints in the same file have been UserXORApplication all along, so // this is also a consistency guard within the family. feature("BG v1.3 - consent authorisation sub-resources accept a client-credentials caller") { for (name <- List(nameOf(Http4sBGv13AIS.getConsentAuthorisation), nameOf(Http4sBGv13AIS.getConsentScaStatus))) { - scenario(s"$name declares UserOrApplication", BerlinGroupV1_3) { + scenario(s"$name declares UserXORApplication", BerlinGroupV1_3) { val docs = APIUtil.ResourceDoc.getResourceDocs( List(APIUtil.buildOperationId(ConstantsBG.berlinGroupVersion1, name))) docs should not be empty - docs.foreach(_.authMode should equal(APIUtil.UserOrApplication)) + docs.foreach(_.authMode should equal(APIUtil.UserXORApplication)) } } } diff --git a/obp-api/src/test/scala/code/api/berlin/group/v1_3/BerlinGroupV13ConsentAccessTests.scala b/obp-api/src/test/scala/code/api/berlin/group/v1_3/BerlinGroupV13ConsentAccessTests.scala index f27dd88d59..aba8763e66 100644 --- a/obp-api/src/test/scala/code/api/berlin/group/v1_3/BerlinGroupV13ConsentAccessTests.scala +++ b/obp-api/src/test/scala/code/api/berlin/group/v1_3/BerlinGroupV13ConsentAccessTests.scala @@ -727,7 +727,7 @@ class BerlinGroupV13ConsentAccessTests extends BerlinGroupConsentFixtures { } } - // The GET siblings were brought to UserOrApplication on their own; these two were held back + // The GET siblings were brought to UserXORApplication on their own; these two were held back // because a doc's auth mode says nothing useful until the handler has an answer to which PSU an // authorisation is for. They now do, and the answer does not come from the session, so the docs // can state what these calls have always been: the TPP acting as itself. @@ -746,11 +746,11 @@ class BerlinGroupV13ConsentAccessTests extends BerlinGroupConsentFixtures { "updateConsentsPsuDataUpdateAuthorisationConfirmation" ) for (name <- authorisationDocs) { - scenario(s"$name declares UserOrApplication", BerlinGroupV13ConsentAccess) { + scenario(s"$name declares UserXORApplication", BerlinGroupV13ConsentAccess) { val docs = APIUtil.ResourceDoc.getResourceDocs( List(APIUtil.buildOperationId(ConstantsBG.berlinGroupVersion1, name))) docs should not be empty - docs.foreach(_.authMode should equal(APIUtil.UserOrApplication)) + docs.foreach(_.authMode should equal(APIUtil.UserXORApplication)) } } } diff --git a/obp-api/src/test/scala/code/api/sweep/ExplicitTargetConsentUserSweepTest.scala b/obp-api/src/test/scala/code/api/sweep/ExplicitTargetConsentUserSweepTest.scala index f4e9ead86c..a4628ab53a 100644 --- a/obp-api/src/test/scala/code/api/sweep/ExplicitTargetConsentUserSweepTest.scala +++ b/obp-api/src/test/scala/code/api/sweep/ExplicitTargetConsentUserSweepTest.scala @@ -67,9 +67,9 @@ import scala.io.Source * would be wrong: the caller asked for a specific target, and if that target is an agent identity * the request is a mistake the caller needs told about. So the endpoint answers 400 OBP-30107. * - * Why a sweep and not a scenario per version file: the guard is twenty near-identical three-line + * Why a sweep and not a scenario per version file: the guard is twenty-one near-identical three-line * blocks spread over ten Http4s objects, and the failure mode that matters is not one of them - * being wrong -- it is a twenty-first being added, or an existing one being dropped, with nobody + * being wrong -- it is a twenty-second being added, or an existing one being dropped, with nobody * noticing. So this suite pairs each probe with a source scan (the last scenario) that counts the * guards in main and fails if the count and this table disagree. * @@ -227,6 +227,19 @@ class ExplicitTargetConsentUserSweepTest extends ServerSetupWithTestData with De )) } + scenario("file access", ExplicitTargetGuard) { + val agentId = newConsentUser().userId + // Only a file's owner gives access to it, so the file is stored for the caller (resourceUser1). + val fileId = code.files.Files.provider.vend.createFile( + bankId, "0" * 64, "image/png", "explicit-target-sweep.png", isPublic = false, + resourceUser1.userId, Array[Byte](1, 2, 3) + ).openOrThrowException("expected the file to be stored").fileId + sweep(List( + Probe("OBPv7.0.0-createFileAccess", "POST", s"/obp/v7.0.0/banks/$bankId/files/$fileId/access", + code.api.v7_0_0.PostFileAccessJsonV700(user_id = agentId)) + )) + } + scenario("account creation", ExplicitTargetGuard) { val agentId = newConsentUser().userId def v310Body = CreateAccountRequestJsonV310( @@ -344,11 +357,11 @@ class ExplicitTargetConsentUserSweepTest extends ServerSetupWithTestData with De finally source.close() } - // Twenty-one probes cover twenty guards: v7.0.0 registers createAccountV700 (POST) and + // Twenty-two probes cover twenty-one guards: v7.0.0 registers createAccountV700 (POST) and // createAccountWithIdV700 (PUT) over one shared createAccountCommon, so the two probes // reach the same guard by two different routes -- the PUT arm validates the account id // first, which is the part worth probing separately. - val guardsInMain = 20 + val guardsInMain = 21 withClue(s"found ${guardLines.size} explicit-target guards in main, but this suite is " + s"written against $guardsInMain. A guard added without a probe is untested; a " + diff --git a/obp-api/src/test/scala/code/api/sweep/OnBehalfOfOwnershipSweepTest.scala b/obp-api/src/test/scala/code/api/sweep/OnBehalfOfOwnershipSweepTest.scala index 37c091fc7b..1181ff3acf 100644 --- a/obp-api/src/test/scala/code/api/sweep/OnBehalfOfOwnershipSweepTest.scala +++ b/obp-api/src/test/scala/code/api/sweep/OnBehalfOfOwnershipSweepTest.scala @@ -218,11 +218,17 @@ class OnBehalfOfOwnershipSweepTest extends ServerSetupWithTestData with DefaultU * (SignatoryPanel.UserIds). DBUtil.runQuery stringifies every column, so one comparison covers * all three: equal to the primary key, or containing the user_id. */ + // A record-both reference (Counterparty, DynamicData, AccountCreation, ...) keeps the agent in its + // first column on purpose -- that is the audit half -- and the person in its OnBehalfOf column. Only + // the OnBehalfOf column is about ownership, so only it may not hold the consent user. + private def ownershipFields(ref: UserReference): List[String] = + if (ref.fields.exists(_.contains("OnBehalfOf"))) ref.fields.filter(_.contains("OnBehalfOf")) else ref.fields + private def referencesOwnedBy(userId: String, primaryKey: String): List[String] = for { ref <- UserReference.byPolicy(AttributionPolicy.UseOnBehalfOfUserId) meta <- metaByClassName.get(ref.mapperClass).toList - field <- ref.fields.flatMap(name => meta.mappedFields.find(_.name == name)) + field <- ownershipFields(ref).flatMap(name => meta.mappedFields.find(_.name == name)) where = sharedColumnDiscriminator.get(ref.name).flatMap { case (discriminator, otherValue) => meta.mappedFields.find(_.name == discriminator).map(d => s" WHERE ${d.dbColumnName} IS NULL OR ${d.dbColumnName} <> '$otherValue'") @@ -372,6 +378,20 @@ class OnBehalfOfOwnershipSweepTest extends ServerSetupWithTestData with DefaultU holders should not contain agentId } + // AccountCreation_CreatedByUserId records both: the agent made the call, for resourceUser1. + withClue(s"metadata.created of the account the agent created: ${compact(render(body \ "metadata"))} ") { + (body \ "metadata" \ "created" \ "user_id").extract[String] should equal(agentId) + (body \ "metadata" \ "created" \ "on_behalf_of_user_id").extract[String] should equal(resourceUser1.userId) + } + + // ... and names the Consent it acted under, by the reference id the metrics record for the call. + val agentConsentReferenceId = code.model.dataAccess.ResourceUser + .find(net.liftweb.mapper.By(code.model.dataAccess.ResourceUser.userId_, agentId)) + .flatMap(ru => code.consent.Consents.consentProvider.vend.getConsentByConsentId(ru.CreatedByConsentId.get)) + .map(_.consentReferenceId) + agentConsentReferenceId.isDefined should be(true) + (body \ "metadata" \ "created" \ "consent_reference_id").extractOpt[String] should equal(agentConsentReferenceId.toOption) + val leaked = referencesOwnedBy(agentId, agentKey).filterNot(notYetWired.contains) withClue( s"""|${leaked.size} wired reference(s) still hold rows owned by the consent user diff --git a/obp-api/src/test/scala/code/api/v3_0_0/EntitlementRequestsTest.scala b/obp-api/src/test/scala/code/api/v3_0_0/EntitlementRequestsTest.scala index 3e90ef9f3a..38940291cb 100644 --- a/obp-api/src/test/scala/code/api/v3_0_0/EntitlementRequestsTest.scala +++ b/obp-api/src/test/scala/code/api/v3_0_0/EntitlementRequestsTest.scala @@ -268,6 +268,29 @@ class EntitlementRequestsTest extends V300ServerSetup with DefaultUsers { resultGetMyUserALl.entitlement_requests.length should be (0) } + scenario("a super admin sees and deletes entitlement requests without stored Entitlements", VersionOfApi, + ApiEndpoint1, ApiEndpoint2, ApiEndpoint3) { + Given("user1 is listed in super_admin_user_ids and holds no Entitlements") + setPropsValues("super_admin_user_ids" -> resourceUser1.userId) + Entitlement.entitlement.vend.getEntitlementsByUserId(resourceUser1.userId).map(_.size).openOr(0) should equal(0) + + val postJson = s"""{"bank_id":"${testBankId1.value}", "role_name":"CanCreateBankLevelEndpointTag"}""" + val response300 = makePostRequest((v3_0Request / "entitlement-requests").POST <@(user1), postJson) + response300.code should equal(201) + val entitlementRequestId = response300.body.extract[EntitlementRequestJSON].entitlement_request_id + + When("they list every entitlement request") + val responseGetAll = makeGetRequest((v3_0Request / "entitlement-requests").GET <@(user1)) + Then("the virtual CanGetEntitlementRequestsAtAnyBank lets them") + responseGetAll.code should equal(200) + responseGetAll.body.extract[EntitlementRequestsJSON].entitlement_requests.map(_.entitlement_request_id) should contain (entitlementRequestId) + + When("they delete the request, as accepting one does") + val deleteResponse = makeDeleteRequest((v3_0Request / "entitlement-requests" / entitlementRequestId).DELETE <@(user1)) + Then("the virtual CanDeleteEntitlementRequestsAtAnyBank lets them") + deleteResponse.code should equal(200) + } + } feature(s"Pagination and sorting for entitlement request endpoints") { diff --git a/obp-api/src/test/scala/code/api/v4_0_0/UserCustomerLinkTest.scala b/obp-api/src/test/scala/code/api/v4_0_0/UserCustomerLinkTest.scala index a3ddd7b922..8e273ed0f3 100644 --- a/obp-api/src/test/scala/code/api/v4_0_0/UserCustomerLinkTest.scala +++ b/obp-api/src/test/scala/code/api/v4_0_0/UserCustomerLinkTest.scala @@ -31,7 +31,7 @@ import org.json4s._ import code.api.ResourceDocs1_4_0.SwaggerDefinitionsJSON import code.api.util.APIUtil.OAuth._ import code.api.util.ApiRole.{CanCreateUserCustomerLink, CanDeleteUserCustomerLink, CanGetUserCustomerLink} -import code.api.util.ErrorMessages.{UserHasMissingRoles, AuthenticatedUserIsRequired} +import code.api.util.ErrorMessages.{ApplicationNotIdentified, UserHasMissingRoles, AuthenticatedUserIsRequired} import code.api.v2_0_0.UserCustomerLinksJson import code.api.v4_0_0.Http4s400.Implementations4_0_0 import code.entitlement.Entitlement @@ -143,7 +143,8 @@ class UserCustomerLinkTest extends V400ServerSetup { val createResponse = makePostRequest(createRequest, write(postJson)) Then("We should get a 401") createResponse.code should equal(401) - createResponse.body.extract[ErrorMessage].message should equal(AuthenticatedUserIsRequired) + // The endpoint is UserXORApplication: with neither a User nor an application, the application is unknown. + createResponse.body.extract[ErrorMessage].message should equal(ApplicationNotIdentified) } } feature(s"test $ApiEndpoint4 version $VersionOfApi - Authorized access") { diff --git a/obp-api/src/test/scala/code/api/v5_1_0/ConsentsDisabledTest.scala b/obp-api/src/test/scala/code/api/v5_1_0/ConsentsDisabledTest.scala new file mode 100644 index 0000000000..28a094b3f6 --- /dev/null +++ b/obp-api/src/test/scala/code/api/v5_1_0/ConsentsDisabledTest.scala @@ -0,0 +1,74 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2019, TESOBE GmbH + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + */ +package code.api.v5_1_0 + +import code.api.RequestHeader +import code.api.ResourceDocs1_4_0.SwaggerDefinitionsJSON +import code.api.util.APIUtil.OAuth._ +import code.api.util.ErrorMessages.ConsentDisabled +import code.api.v5_1_0.Http4s510.Implementations5_1_0 +import code.consent.MappedConsent +import com.github.dwickern.macros.NameOf.nameOf +import com.openbankproject.commons.model.ErrorMessage +import com.openbankproject.commons.util.ApiVersion +import org.json4s.native.Serialization.write +import org.scalatest.Tag + +/** + * With consents.allowed off a consent can never be used as a login, so none is created either: the create endpoints + * answer OBP-35007 before sending a challenge or storing anything. + */ +class ConsentsDisabledTest extends V510ServerSetup { + + object VersionOfApi extends Tag(ApiVersion.v5_1_0.toString) + object CreateConsent extends Tag(nameOf(Implementations5_1_0.createConsent)) + + private val consumerKeyHeader = List((RequestHeader.`Consumer-Key`, user1.map(_._1.key).getOrElse("SHOULD_NOT_HAPPEN"))) + private lazy val body = write(SwaggerDefinitionsJSON.postConsentImplicitJsonV310.copy(entitlements = Nil, views = Nil, + consumer_id = Some(testConsumer.consumerId.get))) + + feature(s"Creating a consent while consents are not allowed - $VersionOfApi") { + + scenario("v5.1.0 and v6.0.0 Create Consent (IMPLICIT) are refused with OBP-35007 and store nothing", CreateConsent, VersionOfApi) { + setPropsValues("consents.allowed" -> "false") + val before = MappedConsent.count + for (version <- List("v5.1.0", "v6.0.0")) { + val request = (baseRequest / "obp" / version / "my" / "consents" / "IMPLICIT").POST <@ (user1) + val response = makePostRequest(request, body, consumerKeyHeader) + response.code should equal(401) + response.body.extract[ErrorMessage].message should include(ConsentDisabled.trim) + } + MappedConsent.count should equal(before) + } + + scenario("With consents allowed the same request gets past the switch", CreateConsent, VersionOfApi) { + setPropsValues("consents.allowed" -> "true") + val request = (v5_1_0_Request / "my" / "consents" / "IMPLICIT").POST <@ (user1) + val response = makePostRequest(request, body, consumerKeyHeader) + (response.body \ "message").extractOpt[String].getOrElse("") should not include "OBP-35007" + } + } +} diff --git a/obp-api/src/test/scala/code/api/v5_1_0/ConsumerTest.scala b/obp-api/src/test/scala/code/api/v5_1_0/ConsumerTest.scala index 826d16a68f..0a9de956fb 100644 --- a/obp-api/src/test/scala/code/api/v5_1_0/ConsumerTest.scala +++ b/obp-api/src/test/scala/code/api/v5_1_0/ConsumerTest.scala @@ -88,7 +88,7 @@ class ConsumerTest extends V510ServerSetup { responseApiEndpoint3.code should equal(401) responseApiEndpoint4.code should equal(401) responseApiEndpoint6.code should equal(401) - // createConsumer and getConsumers use authMode=UserOrApplication, so unauthenticated requests get ApplicationNotIdentified + // createConsumer and getConsumers use authMode=UserXORApplication, so unauthenticated requests get ApplicationNotIdentified responseApiEndpoint1.body.toString contains(s"$ApplicationNotIdentified") should be (true) responseApiEndpoint2.body.toString contains(s"$ApplicationNotIdentified") should be (true) responseApiEndpoint3.body.toString contains(s"$AuthenticatedUserIsRequired") should be (true) diff --git a/obp-api/src/test/scala/code/api/v5_1_0/LogCacheEndpointTest.scala b/obp-api/src/test/scala/code/api/v5_1_0/LogCacheEndpointTest.scala index 41b04c177c..914b86ac6d 100644 --- a/obp-api/src/test/scala/code/api/v5_1_0/LogCacheEndpointTest.scala +++ b/obp-api/src/test/scala/code/api/v5_1_0/LogCacheEndpointTest.scala @@ -274,21 +274,24 @@ class LogCacheEndpointTest extends V510ServerSetup { } feature(s"test $ApiEndpoint1 version $VersionOfApi - Application access with a Scope") { - scenario("A Consumer holding the Role as a Scope may read the log cache without an Entitlement", ApiEndpoint1, VersionOfApi) { - val request = (v5_1_0_Request / "system" / "log-cache" / "error").GET <@(user2) - Given("user2 holds no log cache Entitlement and testConsumer2 no Scope") + scenario("A Consumer holding the Role as a Scope may read the log cache as itself, without an Entitlement", ApiEndpoint1, VersionOfApi) { + val request = (v5_1_0_Request / "system" / "log-cache" / "error").GET <@(application2) + Given("testConsumer2 holds no Scope") makeGetRequest(request).code should equal(403) - When("testConsumer2, which user2 signs with, is granted CanGetSystemLogCacheError as a Scope") + When("testConsumer2 is granted CanGetSystemLogCacheError as a Scope") val granted = Scope.scope.vend.addScope("", testConsumer2.id.get.toString, CanGetSystemLogCacheError.toString) try { val response = makeGetRequest(request) - Then("the error level log cache is returned") + Then("the error level log cache is returned to the Consumer acting as itself") response.code should equal(200) (response.body \ "entries") shouldBe a[JArray] And("the Scope covers only its own level") - makeGetRequest((v5_1_0_Request / "system" / "log-cache" / "info").GET <@(user2)).code should equal(403) + makeGetRequest((v5_1_0_Request / "system" / "log-cache" / "info").GET <@(application2)).code should equal(403) + + And("a person calling through testConsumer2 does not get its Scope") + makeGetRequest((v5_1_0_Request / "system" / "log-cache" / "error").GET <@(user2)).code should equal(403) } finally Scope.scope.vend.deleteScope(granted) } } diff --git a/obp-api/src/test/scala/code/api/v6_0_0/AggregateMetricsTest.scala b/obp-api/src/test/scala/code/api/v6_0_0/AggregateMetricsTest.scala index 187c0a0c9e..1796c96b42 100644 --- a/obp-api/src/test/scala/code/api/v6_0_0/AggregateMetricsTest.scala +++ b/obp-api/src/test/scala/code/api/v6_0_0/AggregateMetricsTest.scala @@ -128,16 +128,18 @@ class AggregateMetricsTest extends V600ServerSetup { } feature(s"test $ApiEndpoint1 version $VersionOfApi - Application access with a Scope") { - scenario("A Consumer holding the Role as a Scope may read aggregate metrics without an Entitlement", ApiEndpoint1, VersionOfApi) { - val request = (v6_0_0_Request / "management" / "aggregate-metrics").GET <@ (user2) - Given("user2 holds no CanReadAggregateMetrics Entitlement and testConsumer2 no Scope") - makeGetRequest(request).code should equal(403) + scenario("A Consumer holding the Role as a Scope reads aggregate metrics as itself, but a person calling through it does not", ApiEndpoint1, VersionOfApi) { + val aggregateMetrics = v6_0_0_Request / "management" / "aggregate-metrics" + Given("testConsumer2 holds no CanReadAggregateMetrics Scope") + makeGetRequest(aggregateMetrics.GET <@ (application2)).code should equal(403) - When("testConsumer2, which user2 signs with, is granted CanReadAggregateMetrics as a Scope") + When("testConsumer2 is granted CanReadAggregateMetrics as a Scope") val granted = Scope.scope.vend.addScope("", testConsumer2.id.get.toString, CanReadAggregateMetrics.toString) try { - Then("aggregate metrics are returned") - makeGetRequest(request).code should equal(200) + Then("the application acting as itself gets aggregate metrics") + makeGetRequest(aggregateMetrics.GET <@ (application2)).code should equal(200) + And("user2, a person signing with testConsumer2, still needs an Entitlement of their own (UserXORApplication)") + makeGetRequest(aggregateMetrics.GET <@ (user2)).code should equal(403) } finally Scope.scope.vend.deleteScope(granted) } } diff --git a/obp-api/src/test/scala/code/api/v6_0_0/AppDirectoryTest.scala b/obp-api/src/test/scala/code/api/v6_0_0/AppDirectoryTest.scala index 0698b96517..549a1ed30a 100644 --- a/obp-api/src/test/scala/code/api/v6_0_0/AppDirectoryTest.scala +++ b/obp-api/src/test/scala/code/api/v6_0_0/AppDirectoryTest.scala @@ -198,6 +198,7 @@ class AppDirectoryTest extends V600ServerSetup { APIUtil.publicAppUrlPropNames should contain("public_obp_portal_url") APIUtil.publicAppUrlPropNames should contain("public_obp_api_explorer_url") APIUtil.publicAppUrlPropNames should contain("public_obp_api_manager_url") + APIUtil.publicAppUrlPropNames should contain("public_obp_cohort_url") APIUtil.publicAppUrlPropNames should contain("public_obp_sandbox_populator_url") APIUtil.publicAppUrlPropNames should contain("public_obp_oidc_url") APIUtil.publicAppUrlPropNames should contain("public_keycloak_url") diff --git a/obp-api/src/test/scala/code/api/v6_0_0/ChatThreadAndReactionsTest.scala b/obp-api/src/test/scala/code/api/v6_0_0/ChatThreadAndReactionsTest.scala new file mode 100644 index 0000000000..c95b730e0a --- /dev/null +++ b/obp-api/src/test/scala/code/api/v6_0_0/ChatThreadAndReactionsTest.scala @@ -0,0 +1,108 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ + +package code.api.v6_0_0 + +import code.api.util.APIUtil.OAuth._ +import code.api.util.ErrorMessages.ChatMessageNotFound +import code.chat.{ChatMessageTrait, ChatRoomTrait, ParticipantTrait} +import com.openbankproject.commons.model.ErrorMessage +import com.openbankproject.commons.util.ApiVersion +import org.scalatest.Tag + +/** + * Thread replies and reactions of a message can only be read through the room the message belongs to. + */ +class ChatThreadAndReactionsTest extends V600ServerSetup { + + object VersionOfApi extends Tag(ApiVersion.v6_0_0.toString) + object ApiEndpointBankThread extends Tag("getBankThreadReplies") + object ApiEndpointSystemThread extends Tag("getSystemThreadReplies") + object ApiEndpointBankReactions extends Tag("getBankReactions") + object ApiEndpointSystemReactions extends Tag("getSystemReactions") + + // a room at bankId in which user1 is a participant + private def createRoomWithUser1(bankId: String): String = { + val room = ChatRoomTrait.chatRoomProvider.vend.createChatRoom( + bankId, s"room-${java.util.UUID.randomUUID()}", "test room", resourceUser2.userId).openOrThrowException("room") + ParticipantTrait.participantProvider.vend.addParticipant( + room.chatRoomId, resourceUser1.userId, "", Nil, "").openOrThrowException("participant") + room.chatRoomId + } + + // a message by user2 in a room user1 is not in + private def createMessageInOtherRoom(bankId: String): String = { + val otherRoom = ChatRoomTrait.chatRoomProvider.vend.createChatRoom( + bankId, s"other-${java.util.UUID.randomUUID()}", "other room", resourceUser2.userId).openOrThrowException("room") + ChatMessageTrait.chatMessageProvider.vend.createMessage( + otherRoom.chatRoomId, resourceUser2.userId, "", "secret", "text", Nil, "", "").openOrThrowException("message").chatMessageId + } + + feature(s"Get Bank Thread Replies - GET /obp/v6.0.0/banks/BANK_ID/chat-rooms/CHAT_ROOM_ID/messages/CHAT_MESSAGE_ID/thread - $VersionOfApi") { + scenario("A message of another room is refused with 404", ApiEndpointBankThread, VersionOfApi) { + val roomId = createRoomWithUser1(testBankId1.value) + val messageId = createMessageInOtherRoom(testBankId1.value) + val request = (v6_0_0_Request / "banks" / testBankId1.value / "chat-rooms" / roomId / "messages" / messageId / "thread").GET <@ (user1) + val response = makeGetRequest(request) + response.code should equal(404) + response.body.extract[ErrorMessage].message should equal(ChatMessageNotFound) + } + } + + feature(s"Get System Thread Replies - GET /obp/v6.0.0/chat-rooms/CHAT_ROOM_ID/messages/CHAT_MESSAGE_ID/thread - $VersionOfApi") { + scenario("A message of another room is refused with 404", ApiEndpointSystemThread, VersionOfApi) { + val roomId = createRoomWithUser1("") + val messageId = createMessageInOtherRoom("") + val request = (v6_0_0_Request / "chat-rooms" / roomId / "messages" / messageId / "thread").GET <@ (user1) + val response = makeGetRequest(request) + response.code should equal(404) + response.body.extract[ErrorMessage].message should equal(ChatMessageNotFound) + } + } + + feature(s"Get Bank Reactions - GET /obp/v6.0.0/banks/BANK_ID/chat-rooms/CHAT_ROOM_ID/messages/CHAT_MESSAGE_ID/reactions - $VersionOfApi") { + scenario("A message of another room is refused with 404", ApiEndpointBankReactions, VersionOfApi) { + val roomId = createRoomWithUser1(testBankId1.value) + val messageId = createMessageInOtherRoom(testBankId1.value) + val request = (v6_0_0_Request / "banks" / testBankId1.value / "chat-rooms" / roomId / "messages" / messageId / "reactions").GET <@ (user1) + val response = makeGetRequest(request) + response.code should equal(404) + response.body.extract[ErrorMessage].message should equal(ChatMessageNotFound) + } + } + + feature(s"Get System Reactions - GET /obp/v6.0.0/chat-rooms/CHAT_ROOM_ID/messages/CHAT_MESSAGE_ID/reactions - $VersionOfApi") { + scenario("A message of another room is refused with 404", ApiEndpointSystemReactions, VersionOfApi) { + val roomId = createRoomWithUser1("") + val messageId = createMessageInOtherRoom("") + val request = (v6_0_0_Request / "chat-rooms" / roomId / "messages" / messageId / "reactions").GET <@ (user1) + val response = makeGetRequest(request) + response.code should equal(404) + response.body.extract[ErrorMessage].message should equal(ChatMessageNotFound) + } + } +} diff --git a/obp-api/src/test/scala/code/api/v6_0_0/DynamicEntityAuthModeTest.scala b/obp-api/src/test/scala/code/api/v6_0_0/DynamicEntityAuthModeTest.scala index adc0fcee1b..5f95b1f6ae 100644 --- a/obp-api/src/test/scala/code/api/v6_0_0/DynamicEntityAuthModeTest.scala +++ b/obp-api/src/test/scala/code/api/v6_0_0/DynamicEntityAuthModeTest.scala @@ -44,7 +44,8 @@ import code.api.Constant.DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID /** * auth_mode on a Dynamic Entity decides who may hold the roles guarding its data endpoints: * the User's Entitlements (UserOnly, the default), the Consumer's Scopes (ApplicationOnly), - * either (UserOrApplication) or both (UserAndApplication). + * the Consumer's Scopes when it acts as itself and the User's Entitlements when a person acts (UserXORApplication), + * or both (UserAndApplication). */ class DynamicEntityAuthModeTest extends V600ServerSetup { // user1 defines the entities (and may hold broad roles in the shared test DB); @@ -92,10 +93,17 @@ class DynamicEntityAuthModeTest extends V600ServerSetup { deleteSystemEntity((body \ "dynamic_entity_id").extract[String]) } - scenario("accepts UserOrApplication and returns it", VersionOfApi) { - val (code, body) = createSystemEntity(entityJson("am_either", Some("UserOrApplication"))) + scenario("accepts UserXORApplication and returns it", VersionOfApi) { + val (code, body) = createSystemEntity(entityJson("am_either", Some("UserXORApplication"))) code should equal(201) - (body \ "auth_mode").extract[String] should equal("UserOrApplication") + (body \ "auth_mode").extract[String] should equal("UserXORApplication") + deleteSystemEntity((body \ "dynamic_entity_id").extract[String]) + } + + scenario("accepts the former name UserOrApplication and stores it as UserXORApplication", VersionOfApi) { + val (code, body) = createSystemEntity(entityJson("am_legacy", Some("UserOrApplication"))) + code should equal(201) + (body \ "auth_mode").extract[String] should equal("UserXORApplication") deleteSystemEntity((body \ "dynamic_entity_id").extract[String]) } @@ -130,14 +138,14 @@ class DynamicEntityAuthModeTest extends V600ServerSetup { } } - scenario("UserOrApplication: a Consumer Scope alone is enough", VersionOfApi) { + scenario("UserXORApplication: a Consumer Scope alone is enough for the Consumer acting as itself", VersionOfApi) { val entityName = "am_either_data" - val (code, body) = createSystemEntity(entityJson(entityName, Some("UserOrApplication"))) + val (code, body) = createSystemEntity(entityJson(entityName, Some("UserXORApplication"))) code should equal(201) val entityId = (body \ "dynamic_entity_id").extract[String] val scope = Scope.scope.vend.addScope(DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID, testConsumer2.id.get.toString, getRoleName(entityName)) try { - val response = makeGetRequest((dynamicEntity_Request / entityName).GET <@(user2)) + val response = makeGetRequest((dynamicEntity_Request / entityName).GET <@(application2)) response.code should equal(200) } finally { Scope.scope.vend.deleteScope(scope) @@ -145,9 +153,25 @@ class DynamicEntityAuthModeTest extends V600ServerSetup { } } - scenario("UserOrApplication: a User Entitlement alone is still enough", VersionOfApi) { + scenario("UserXORApplication: a person does not get the Scope of the Consumer they call through", VersionOfApi) { + val entityName = "am_either_person" + val (code, body) = createSystemEntity(entityJson(entityName, Some("UserXORApplication"))) + code should equal(201) + val entityId = (body \ "dynamic_entity_id").extract[String] + val scope = Scope.scope.vend.addScope(DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID, testConsumer2.id.get.toString, getRoleName(entityName)) + try { + val response = makeGetRequest((dynamicEntity_Request / entityName).GET <@(user2)) + response.code should equal(403) + response.body.extract[ErrorMessage].message should include(UserHasMissingRoles) + } finally { + Scope.scope.vend.deleteScope(scope) + deleteSystemEntity(entityId) + } + } + + scenario("UserXORApplication: a User Entitlement alone is still enough", VersionOfApi) { val entityName = "am_either_user" - val (code, body) = createSystemEntity(entityJson(entityName, Some("UserOrApplication"))) + val (code, body) = createSystemEntity(entityJson(entityName, Some("UserXORApplication"))) code should equal(201) val entityId = (body \ "dynamic_entity_id").extract[String] Entitlement.entitlement.vend.addEntitlement(DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID, resourceUser2.userId, getRoleName(entityName)) @@ -159,14 +183,14 @@ class DynamicEntityAuthModeTest extends V600ServerSetup { } } - scenario("UserOrApplication: the Scope only covers the role it names (Get, not Create)", VersionOfApi) { + scenario("UserXORApplication: the Scope only covers the role it names (Get, not Create)", VersionOfApi) { val entityName = "am_either_scoped" - val (code, body) = createSystemEntity(entityJson(entityName, Some("UserOrApplication"))) + val (code, body) = createSystemEntity(entityJson(entityName, Some("UserXORApplication"))) code should equal(201) val entityId = (body \ "dynamic_entity_id").extract[String] val scope = Scope.scope.vend.addScope(DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID, testConsumer2.id.get.toString, getRoleName(entityName)) try { - val response = makePostRequest((dynamicEntity_Request / entityName).POST <@(user2), write(("name" -> "x"): JObject)) + val response = makePostRequest((dynamicEntity_Request / entityName).POST <@(application2), write(("name" -> "x"): JObject)) response.code should equal(403) response.body.extract[ErrorMessage].message should include(createRoleName(entityName)) } finally { diff --git a/obp-api/src/test/scala/code/api/v6_0_0/DynamicEntityTest.scala b/obp-api/src/test/scala/code/api/v6_0_0/DynamicEntityTest.scala index fcfe150015..91eefb4cdc 100644 --- a/obp-api/src/test/scala/code/api/v6_0_0/DynamicEntityTest.scala +++ b/obp-api/src/test/scala/code/api/v6_0_0/DynamicEntityTest.scala @@ -194,11 +194,11 @@ class DynamicEntityTest extends V600ServerSetup { } scenario("Create System Dynamic Entity with consumer scope (no user entitlement)", ApiEndpoint1, VersionOfApi) { - // Add scope to consumer instead of entitlement to user — UserOrApplication should accept this + // Add scope to consumer instead of entitlement to user — UserXORApplication accepts it when the Consumer acts as itself val addedScope = Scope.scope.vend.addScope(DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID, testConsumer.id.get.toString, ApiRole.CanCreateDynamicEntityDefinition.toString) When("We create a dynamic entity using consumer with scope") - val request = (v6_0_0_Request / "management" / "system-dynamic-entities").POST <@(user1) + val request = (v6_0_0_Request / "management" / "system-dynamic-entities").POST <@(application1) val response = try { makePostRequest(request, write(rightEntityV600)) } finally { diff --git a/obp-api/src/test/scala/code/api/v6_0_0/EndpointAuthModeTest.scala b/obp-api/src/test/scala/code/api/v6_0_0/EndpointAuthModeTest.scala index c4d8c05171..e86bb20a6b 100644 --- a/obp-api/src/test/scala/code/api/v6_0_0/EndpointAuthModeTest.scala +++ b/obp-api/src/test/scala/code/api/v6_0_0/EndpointAuthModeTest.scala @@ -45,7 +45,7 @@ class EndpointAuthModeTest extends V600ServerSetup { scenario("All four auth modes should be defined", VersionOfApi) { val userOnly: EndpointAuthMode = UserOnly val appOnly: EndpointAuthMode = ApplicationOnly - val userOrApp: EndpointAuthMode = UserOrApplication + val userOrApp: EndpointAuthMode = UserXORApplication val userAndApp: EndpointAuthMode = UserAndApplication userOnly shouldBe a[EndpointAuthMode] @@ -54,13 +54,13 @@ class EndpointAuthModeTest extends V600ServerSetup { userAndApp shouldBe a[EndpointAuthMode] } - scenario("verifyUserCredentials ResourceDoc should have UserOrApplication authMode", VersionOfApi) { + scenario("verifyUserCredentials ResourceDoc should have UserXORApplication authMode", VersionOfApi) { val operationId = buildOperationId(ApiVersion.v6_0_0, "verifyUserCredentials") val docs = ResourceDoc.getResourceDocs(List(operationId)) docs should not be empty docs.foreach { doc => - doc.authMode should equal(UserOrApplication) + doc.authMode should equal(UserXORApplication) doc.errorResponseBodies should contain(ApplicationNotIdentified) } } diff --git a/obp-api/src/test/scala/code/api/v6_0_0/VerifyUserCredentialsTest.scala b/obp-api/src/test/scala/code/api/v6_0_0/VerifyUserCredentialsTest.scala index 60418036c1..1c17498a48 100644 --- a/obp-api/src/test/scala/code/api/v6_0_0/VerifyUserCredentialsTest.scala +++ b/obp-api/src/test/scala/code/api/v6_0_0/VerifyUserCredentialsTest.scala @@ -109,7 +109,7 @@ class VerifyUserCredentialsTest extends V600ServerSetup with DefaultUsers { Then("We should get a 401") response.code should equal(401) - And("The error message should indicate application not identified (UserOrApplication mode requires at least app auth)") + And("The error message should indicate application not identified (UserXORApplication mode requires at least app auth)") response.body.extract[ErrorMessage].message should include("OBP-20200") } @@ -129,8 +129,8 @@ class VerifyUserCredentialsTest extends V600ServerSetup with DefaultUsers { response.body.extract[ErrorMessage].message should equal(UserHasMissingRoles + CanVerifyUserCredentials) } - scenario("Successfully verify valid credentials with consumer scope (no user entitlement)", ApiEndpoint, VersionOfApi) { - // Add scope to consumer instead of entitlement to user — UserOrApplication should accept this + scenario("Successfully verify valid credentials with consumer scope, the Consumer acting as itself (no user entitlement)", ApiEndpoint, VersionOfApi) { + // Add scope to consumer instead of entitlement to user — UserXORApplication accepts it when the Consumer acts as itself val addedScope = Scope.scope.vend.addScope("", testConsumer.id.get.toString, ApiRole.CanVerifyUserCredentials.toString) When("We verify valid credentials using consumer with scope") @@ -139,13 +139,17 @@ class VerifyUserCredentialsTest extends V600ServerSetup with DefaultUsers { "password" -> testPassword, "provider" -> Constant.localIdentityProvider ) - val request = (v6_0_0_Request / "users" / "verify-credentials").POST <@ (user1) - val response = try { - makePostRequest(request, write(postJson)) + val request = (v6_0_0_Request / "users" / "verify-credentials").POST <@ (application1) + val personRequest = (v6_0_0_Request / "users" / "verify-credentials").POST <@ (user1) + val (response, personResponse) = try { + (makePostRequest(request, write(postJson)), makePostRequest(personRequest, write(postJson))) } finally { Scope.scope.vend.deleteScope(addedScope) } + And("a person calling through the same Consumer, without an Entitlement, is refused") + personResponse.code should equal(403) + Then("We should get a 200") response.code should equal(200) diff --git a/obp-api/src/test/scala/code/api/v7_0_0/AssetsEndpointTest.scala b/obp-api/src/test/scala/code/api/v7_0_0/AssetsEndpointTest.scala index f6eaaffc72..5789c8b898 100644 --- a/obp-api/src/test/scala/code/api/v7_0_0/AssetsEndpointTest.scala +++ b/obp-api/src/test/scala/code/api/v7_0_0/AssetsEndpointTest.scala @@ -26,19 +26,32 @@ TESOBE (http://www.tesobe.com/) */ package code.api.v7_0_0 -import code.api.util.ErrorMessages.{AssetNotFound, AssetNotFoundByChainIdentity, InvalidAssetQueryParameter} +import code.api.util.APIUtil.OAuth._ +import code.api.util.ApiRole.canCreateAsset +import code.api.util.ErrorMessages.{AssetAlreadyExists, AssetChainIdentityAlreadyExists, AssetDtiAlreadyExists, AssetNotFound, + AssetNotFoundByChainIdentity, AuthenticatedUserIsRequired, InvalidAsset, InvalidAssetQueryParameter, ProductNotFoundByProductCode, + UserHasMissingRoles} +import code.api.util.APIUtil import code.api.v6_0_0.V600ServerSetup import code.asset.{Asset, AssetSeed, AssetStatuses, AssetTypes, Assets, RestoresSeededAssetRegistry} +import code.bankconnectors.Connector +import code.entitlement.Entitlement import com.openbankproject.commons.model.ErrorMessage import com.openbankproject.commons.util.ApiVersion -import org.json4s.JsonAST.{JInt, JString, JValue} +import org.json4s.JsonAST.{JInt, JNothing, JNull, JString, JValue} +import org.json4s.native.Serialization.write import net.liftweb.mapper.By import org.scalatest.Tag +import scala.concurrent.Await +import scala.concurrent.duration._ + /** - * This suite checks the three read-only asset registry endpoints: they answer without - * authentication, list and filter the seeded registry, find one asset by code ignoring case, find a - * token by its chain identity, and refuse a bad query parameter with 400 rather than ignore it. + * This suite checks the asset registry endpoints. The three reads answer without authentication, + * list and filter the seeded registry, find one asset by code ignoring case, find a token by its + * chain identity, and refuse a bad query parameter with 400 rather than ignore it. Create Asset + * needs the CanCreateAsset Role at the bank in the path, records that bank as the issuer (or no + * issuer at SYS), and refuses a taken code, chain identity or DTI with 409. * * The test setup empties every table after each test, so each scenario seeds the registry itself, * and the suite puts the seeded registry back when it ends (RestoresSeededAssetRegistry). @@ -48,6 +61,7 @@ class AssetsEndpointTest extends V600ServerSetup with RestoresSeededAssetRegistr def v7_0_0_Request = baseRequest / "obp" / "v7.0.0" object VersionOfApi extends Tag(ApiVersion.v7_0_0.toString) object ApiEndpoint extends Tag("assets") + object CreateAssetEndpoint extends Tag("createAsset") private def assets = v7_0_0_Request / "assets" @@ -177,4 +191,147 @@ class AssetsEndpointTest extends V600ServerSetup with RestoresSeededAssetRegistr response.body.extract[ErrorMessage].message should startWith(AssetNotFoundByChainIdentity) } } + + feature(s"Create Asset - POST /obp/v7.0.0/banks/BANK_ID/assets - $VersionOfApi") { + + val bankId = testBankId1.value + def createAt(bank: String) = (v7_0_0_Request / "banks" / bank / "assets").POST + + def bondBody(code: String = "BANKBOND30", + assetType: String = AssetTypes.DEBT_SECURITY, + productCode: Option[String] = None, + chainScheme: Option[String] = None, + chainAssetId: Option[String] = None, + dti: Option[String] = None): String = + write(PostAssetJsonV700(code, assetType, "Bank Bond 2030", 0, productCode, chainScheme, chainAssetId, dti)) + + def grantAt(bank: String): Unit = + Entitlement.entitlement.vend.addEntitlement(bank, resourceUser1.userId, canCreateAsset.toString) + + def errorOf(body: JValue): String = body.extract[ErrorMessage].message + + scenario("without logging in, or without the Role, nothing is created", CreateAssetEndpoint, VersionOfApi) { + seedRegistry() + val anonymous = makePostRequest(createAt(bankId), bondBody()) + anonymous.code should equal(401) + errorOf(anonymous.body) should startWith(AuthenticatedUserIsRequired) + + val withoutRole = makePostRequest(createAt(bankId) <@ (user1), bondBody()) + withoutRole.code should equal(403) + errorOf(withoutRole.body) should startWith(UserHasMissingRoles) + errorOf(withoutRole.body) should include(canCreateAsset.toString) + Assets.getAsset("BANKBOND30").isEmpty shouldBe true + } + + scenario("a bank creates an asset it issues, and its code is accepted as a currency at once", CreateAssetEndpoint, VersionOfApi) { + seedRegistry() + grantAt(bankId) + val response = makePostRequest(createAt(bankId) <@ (user1), bondBody(code = "bankbond30")) + response.code should equal(201) + (response.body \ "asset_code") should equal(JString("BANKBOND30")) + (response.body \ "asset_type") should equal(JString(AssetTypes.DEBT_SECURITY)) + (response.body \ "issuer_bank_id") should equal(JString(bankId)) + (response.body \ "status") should equal(JString(AssetStatuses.ACTIVE)) + (response.body \ "created_by_user_id") should equal(JString(resourceUser1.userId)) + + makeGetRequest((assets / "BANKBOND30").GET).code should equal(200) + APIUtil.isValidCurrencyISOCode("BANKBOND30") shouldBe true + } + + scenario("a taken code is refused with 409", CreateAssetEndpoint, VersionOfApi) { + seedRegistry() + grantAt(bankId) + makePostRequest(createAt(bankId) <@ (user1), bondBody()).code should equal(201) + val again = makePostRequest(createAt(bankId) <@ (user1), bondBody(code = "BankBond30")) + again.code should equal(409) + errorOf(again.body) should startWith(AssetAlreadyExists) + } + + scenario("a taken chain identity or DTI is refused with 409", CreateAssetEndpoint, VersionOfApi) { + seedRegistry() + grantAt(bankId) + val first = makePostRequest(createAt(bankId) <@ (user1), + bondBody(chainScheme = Some("cardano_mainnet"), chainAssetId = Some(tokenChainAssetId), dti = Some("X9J9K872S"))) + first.code should equal(201) + (first.body \ "chain_scheme") should equal(JString("CARDANO_MAINNET")) + + val sameChainIdentity = makePostRequest(createAt(bankId) <@ (user1), + bondBody(code = "OTHERBOND", chainScheme = Some("CARDANO_MAINNET"), chainAssetId = Some(tokenChainAssetId))) + sameChainIdentity.code should equal(409) + errorOf(sameChainIdentity.body) should startWith(AssetChainIdentityAlreadyExists) + + val sameDti = makePostRequest(createAt(bankId) <@ (user1), bondBody(code = "OTHERBOND", dti = Some("x9j9k872s"))) + sameDti.code should equal(409) + errorOf(sameDti.body) should startWith(AssetDtiAlreadyExists) + + // The same identity on another network is another token + makePostRequest(createAt(bankId) <@ (user1), + bondBody(code = "OTHERBOND", chainScheme = Some("CARDANO_PREPROD"), chainAssetId = Some(tokenChainAssetId))).code should equal(201) + } + + scenario("an asset that breaks the registry's rules is refused with 400", CreateAssetEndpoint, VersionOfApi) { + seedRegistry() + grantAt(bankId) + val badBodies = List( + "a three-character code for a bank-issued asset" -> bondBody(code = "BND"), + "a currency type at a bank" -> bondBody(assetType = AssetTypes.FIAT), + "a crypto asset at a bank" -> bondBody(assetType = AssetTypes.CRYPTO), + "an unknown chain scheme" -> bondBody(chainScheme = Some("CARDANO"), chainAssetId = Some(tokenChainAssetId)), + "a chain scheme without a chain asset id" -> bondBody(chainScheme = Some("CARDANO_MAINNET")), + "a DTI that is not 9 characters" -> bondBody(dti = Some("X9J9K872")), + "more decimal places than the registry allows" -> + write(PostAssetJsonV700("BANKBOND30", AssetTypes.DEBT_SECURITY, "Bank Bond 2030", Assets.MaxDecimalPlaces + 1, None, None, None, None)) + ) + badBodies.foreach { case (clue, body) => + val response = makePostRequest(createAt(bankId) <@ (user1), body) + withClue(clue) { + response.code should equal(400) + errorOf(response.body) should startWith(InvalidAsset) + } + } + Assets.getAsset("BANKBOND30").isEmpty shouldBe true + // BND is in the registry already, as the seeded Brunei dollar: it must still be that, not a bank's bond + Assets.getAsset("BND").map(_.issuerBankId) should equal(net.liftweb.common.Full(None)) + } + + scenario("an issuer product must exist at the issuing bank", CreateAssetEndpoint, VersionOfApi) { + seedRegistry() + grantAt(bankId) + val missing = makePostRequest(createAt(bankId) <@ (user1), bondBody(productCode = Some("NO-SUCH-PRODUCT"))) + missing.code should equal(404) + errorOf(missing.body) should startWith(ProductNotFoundByProductCode) + + Await.result(Connector.connector.vend.createOrUpdateProduct(bankId, "BOND-TERMS-30", None, "Bond terms", "bonds", + "bonds", "bonds", "", "", "", "Terms of the 2030 bond", "", "", None), 30.seconds) + val found = makePostRequest(createAt(bankId) <@ (user1), bondBody(productCode = Some("BOND-TERMS-30"))) + found.code should equal(201) + (found.body \ "issuer_product_code") should equal(JString("BOND-TERMS-30")) + } + + scenario("at SYS a crypto asset is created with no issuer, and nothing else can be", CreateAssetEndpoint, VersionOfApi) { + seedRegistry() + val systemBank = "SYS" + makePostRequest(createAt(systemBank) <@ (user1), bondBody(code = "SOL", assetType = AssetTypes.CRYPTO)).code should equal(403) + + grantAt(systemBank) + val crypto = makePostRequest(createAt(systemBank) <@ (user1), + write(PostAssetJsonV700("SOL", "crypto", "Solana", 9, None, None, None, None))) + crypto.code should equal(201) + (crypto.body \ "asset_type") should equal(JString(AssetTypes.CRYPTO)) + (crypto.body \ "decimal_places").values should equal(9) + List(JNothing, JNull) should contain(crypto.body \ "issuer_bank_id") + + val issuedAtSys = makePostRequest(createAt(systemBank) <@ (user1), bondBody()) + issuedAtSys.code should equal(400) + errorOf(issuedAtSys.body) should startWith(InvalidAsset) + } + + scenario("the Role at one bank does not let the caller create an asset at another", CreateAssetEndpoint, VersionOfApi) { + seedRegistry() + grantAt(bankId) + val response = makePostRequest(createAt(testBankId2.value) <@ (user1), bondBody()) + response.code should equal(403) + errorOf(response.body) should startWith(UserHasMissingRoles) + } + } } diff --git a/obp-api/src/test/scala/code/api/v7_0_0/CustomerEmailsTest.scala b/obp-api/src/test/scala/code/api/v7_0_0/CustomerEmailsTest.scala new file mode 100644 index 0000000000..53d6185731 --- /dev/null +++ b/obp-api/src/test/scala/code/api/v7_0_0/CustomerEmailsTest.scala @@ -0,0 +1,156 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ +package code.api.v7_0_0 + +import java.util.Date + +import code.api.util.APIUtil.OAuth._ +import code.api.util.APIUtil.generateUUID +import code.api.util.ApiRole.CanSendEmailToCustomerAtOneBank +import code.api.util.ErrorMessages._ +import code.api.v6_0_0.V600ServerSetup +import code.customer.CustomerX +import code.entitlement.Entitlement +import code.messageoutbox.MessageOutbox +import code.model.dataAccess.SelfServiceBank +import code.setup.APIResponse +import com.openbankproject.commons.model.{BankId, CustomerFaceImage, ErrorMessage} +import com.openbankproject.commons.util.ApiVersion +import net.liftweb.mapper.By +import org.json4s.native.Serialization.write +import org.scalatest.Tag + +/** + * This suite checks Send Email to Customer: who may queue an email to a Customer, that only the space's own + * Customers are emailed, under which limits, that it lands in the message outbox addressed to the Customer, and that + * the address is never returned. + */ +class CustomerEmailsTest extends V600ServerSetup { + + def v7 = baseRequest / "obp" / "v7.0.0" + object VersionOfApi extends Tag(ApiVersion.v7_0_0.toString) + object CustomerEmails extends Tag("customerEmails") + + private lazy val bankId = testBankId1.value + + private def enable(extra: (String, String)*): Unit = { + setPropsValues((List("customer_emails.sender_address" -> "cohort@example.com") ++ extra): _*) + } + + private def withRole[T](f: => T): T = { + val grant = Entitlement.entitlement.vend.addEntitlement(bankId, resourceUser1.userId, CanSendEmailToCustomerAtOneBank.toString) + try f finally Entitlement.entitlement.vend.deleteEntitlement(grant) + } + + /** A fresh Customer of `bank`, created through the provider; returns its id and email address. */ + private def customer(bank: String = bankId, email: Option[String] = None): (String, String) = { + val address = email.getOrElse(s"applicant-${generateUUID().take(8)}@example.com") + val created = CustomerX.customerProvider.vend.addCustomer( + BankId(bank), generateUUID().take(12), "Applicant Test", "+968 1234 5678", address, + CustomerFaceImage(new Date(), "http://example.com/face.png"), + new Date(), "single", 0, Nil, "Bachelor", "employed", false, new Date(), + None, None, "", "", "" + ).openOrThrowException("expected the customer to be created") + (created.customerId, address) + } + + private def emails(customerId: String) = v7 / "banks" / bankId / "customers" / customerId / "emails" + private def body(subject: String = "Your application was received", reference: Option[String] = Some("app:1")): String = + write(PostCustomerEmailJsonV700(subject, "Thank you for applying.", reference)) + private def message(response: APIResponse): String = response.body.extract[ErrorMessage].message + + feature(s"Send Email to Customer - POST /banks/BANK_ID/customers/CUSTOMER_ID/emails - $VersionOfApi") { + + scenario("Customers of a self-service bank are not emailed", CustomerEmails, VersionOfApi) { + enable() + val (customerId, _) = customer() + SelfServiceBank.record(bankId, resourceUser1.userId) + val response = try withRole { makePostRequest(emails(customerId).POST <@ (user1), body()) } + finally SelfServiceBank.findAll(By(SelfServiceBank.BankId, bankId)).foreach(_.delete_!) + response.code should equal(403) + message(response) should include(CustomerEmailsNotInSelfServiceBank) + } + + scenario("It needs the Role", CustomerEmails, VersionOfApi) { + enable() + val (customerId, _) = customer() + val response = makePostRequest(emails(customerId).POST <@ (user1), body()) + response.code should equal(403) + message(response) should include(UserHasMissingRoles) + } + + scenario("It queues the email in the outbox, to the Customer's address, without returning it", CustomerEmails, VersionOfApi) { + enable() + val (customerId, address) = customer() + val response = withRole { makePostRequest(emails(customerId).POST <@ (user1), body()) } + withClue(response.body) { response.code should equal(201) } + val queued = response.body.extract[CustomerEmailJsonV700] + queued.customer_id should equal(customerId) + queued.status should equal(MessageOutbox.STATUS_PENDING) + queued.reference should equal(Some("app:1")) + response.body.toString should not include address + + And("the outbox row is an email to the Customer's address, saying which application sent it") + val row = MessageOutbox.find(By(MessageOutbox.id, queued.message_id.toLong)).openOrThrowException("row") + row.outboxType should equal(MessageOutbox.TYPE_EMAIL) + row.targetId should equal(address) + val payload = MessageOutbox.emailPayload(row) + payload.subject should equal("Your application was received") + payload.text_content.getOrElse("") should include("Thank you for applying.") + payload.text_content.getOrElse("") should include("Sent by") + row.MetadataJson.get should include("app:1") + } + + scenario("Only the space's own Customers are emailed", CustomerEmails, VersionOfApi) { + enable() + val (otherBanksCustomer, _) = customer(bank = testBankId2.value) + withRole { + makePostRequest(emails(otherBanksCustomer).POST <@ (user1), body()).code should equal(404) + makePostRequest(emails("no-such-customer").POST <@ (user1), body()).code should equal(404) + } + } + + scenario("A Customer without a usable address, and a subject over more than one line, are refused", CustomerEmails, VersionOfApi) { + enable() + val (noAddress, _) = customer(email = Some("")) + val (customerId, _) = customer() + withRole { + val response = makePostRequest(emails(noAddress).POST <@ (user1), body()) + response.code should equal(400) + message(response) should include(CustomerHasNoEmail) + makePostRequest(emails(customerId).POST <@ (user1), body(subject = "Hello\r\nBcc: someone@example.com")).code should equal(400) + } + } + + scenario("At most customer_emails.max_per_customer_per_day emails are queued for a Customer", CustomerEmails, VersionOfApi) { + enable("customer_emails.max_per_customer_per_day" -> "2") + val (customerId, _) = customer() + val codes = withRole { (1 to 3).map(_ => makePostRequest(emails(customerId).POST <@ (user1), body()).code) } + codes should equal(Seq(201, 201, 429)) + } + } +} diff --git a/obp-api/src/test/scala/code/api/v7_0_0/CustomerOnboardingTest.scala b/obp-api/src/test/scala/code/api/v7_0_0/CustomerOnboardingTest.scala new file mode 100644 index 0000000000..a106cab152 --- /dev/null +++ b/obp-api/src/test/scala/code/api/v7_0_0/CustomerOnboardingTest.scala @@ -0,0 +1,80 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ +package code.api.v7_0_0 + +import code.api.util.APIUtil.OAuth._ +import code.api.util.ApiRole.{CanCreateCustomer, CanCreateUserCustomerLink, CanSendEmailToCustomerAtOneBank} +import code.api.v2_0_0.CreateUserCustomerLinkJson +import code.api.v6_0_0.{CustomerJsonV600, PostCustomerJsonV600, V600ServerSetup} +import code.scope.Scope +import com.openbankproject.commons.util.ApiVersion +import org.json4s.native.Serialization.write +import org.scalatest.Tag + +/** + * This suite checks that an application acting as itself, such as a programme website, onboards a person as a + * Customer of its space with the Create Customer and Create User Customer Link Scopes, and can then email them. + */ +class CustomerOnboardingTest extends V600ServerSetup { + + def v7 = baseRequest / "obp" / "v7.0.0" + def v6 = baseRequest / "obp" / "v6.0.0" + def v4 = baseRequest / "obp" / "v4.0.0" + object VersionOfApi extends Tag(ApiVersion.v7_0_0.toString) + object Onboarding extends Tag("customerOnboarding") + + private lazy val bankId = testBankId2.value + + private def withScopes[T](roles: String*)(f: => T): T = { + val added = roles.map(role => Scope.scope.vend.addScope(bankId, testConsumer2.id.get.toString, role)) + try f finally added.foreach(Scope.scope.vend.deleteScope) + } + + feature("An application onboards a person as a Customer, and emails them") { + + scenario("With the Scopes, an application acting as itself creates a Customer, links a User, and emails the Customer", Onboarding, VersionOfApi) { + setPropsValues("customer_emails.sender_address" -> "cohort@example.com") + val body = write(PostCustomerJsonV600(legal_name = "Ali Example", mobile_phone_number = "+968 9000 0000", + email = Some("ali@example.com"))) + makePostRequest((v6 / "banks" / bankId / "customers").POST <@ (application2), body).code should equal(403) + + withScopes(CanCreateCustomer.toString, CanCreateUserCustomerLink.toString, CanSendEmailToCustomerAtOneBank.toString) { + val created = makePostRequest((v6 / "banks" / bankId / "customers").POST <@ (application2), body) + withClue(created.body) { created.code should equal(201) } + val customerId = created.body.extract[CustomerJsonV600].customer_id + + val link = makePostRequest((v4 / "banks" / bankId / "user_customer_links").POST <@ (application2), + write(CreateUserCustomerLinkJson(resourceUser2.userId, customerId))) + withClue(link.body) { link.code should equal(201) } + + val email = makePostRequest((v7 / "banks" / bankId / "customers" / customerId / "emails").POST <@ (application2), + write(PostCustomerEmailJsonV700("Your application was received", "Thank you.", None))) + withClue(email.body) { email.code should equal(201) } + } + } + } +} diff --git a/obp-api/src/test/scala/code/api/v7_0_0/DynamicEntityDefinitionTest.scala b/obp-api/src/test/scala/code/api/v7_0_0/DynamicEntityDefinitionTest.scala index 6bf83ae37a..69648bef33 100644 --- a/obp-api/src/test/scala/code/api/v7_0_0/DynamicEntityDefinitionTest.scala +++ b/obp-api/src/test/scala/code/api/v7_0_0/DynamicEntityDefinitionTest.scala @@ -205,8 +205,9 @@ class DynamicEntityDefinitionTest extends ServerSetupWithTestData { scenario("a Consumer holding the Role as a Scope may list and update definitions", ApiEndpoint1, ApiEndpoint3, VersionOfApi) { // A service (the Portal, the API Manager) ensures its entities at startup with a client-credentials - // token: no User, so the Roles must be accepted as Scopes on its Consumer (auth mode UserOrApplication). - // user2 holds none of these Roles and signs with testConsumer2, which gets the Scopes. + // token: no User, so the Roles must be accepted as Scopes on its Consumer (auth mode UserXORApplication). + // user2 holds none of these Roles and signs with testConsumer2, which gets the Scopes; application2 is + // testConsumer2 acting as itself. val entityName = newEntityName() val dynamicEntityId = createdAt(SYS, entityName) try { @@ -220,15 +221,19 @@ class DynamicEntityDefinitionTest extends ServerSetupWithTestData { val scopes = List(canGetDynamicEntityDefinitions, canUpdateDynamicEntityDefinition).map(role => Scope.scope.vend.addScope(SYS, testConsumer2.id.get.toString, role.toString)) try { - When("user2 lists them again") - val listed = makeGetRequest(definitionsAt(SYS).GET <@ (user2)) + When("user2, a person, lists them again through testConsumer2") + Then("the Consumer's Scope is not theirs to use") + makeGetRequest(definitionsAt(SYS).GET <@ (user2)).code should equal(403) + + When("testConsumer2 lists them as itself") + val listed = makeGetRequest(definitionsAt(SYS).GET <@ (application2)) Then("the Scope is enough") listed.code should equal(200) (listed.body \ "dynamic_entities").extract[List[JObject]] .map(e => (e \ "entity_name").extract[String]) should contain(entityName) - When("user2 updates the definition") - val updated = makePutRequest((definitionsAt(SYS) / dynamicEntityId).PUT <@ (user2), write(definition(entityName))) + When("testConsumer2 updates the definition as itself") + val updated = makePutRequest((definitionsAt(SYS) / dynamicEntityId).PUT <@ (application2), write(definition(entityName))) Then("the Scope is enough for that too") updated.code should equal(200) } finally scopes.foreach(scope => Scope.scope.vend.deleteScope(scope)) diff --git a/obp-api/src/test/scala/code/api/v7_0_0/FilesTest.scala b/obp-api/src/test/scala/code/api/v7_0_0/FilesTest.scala new file mode 100644 index 0000000000..9df41aad2c --- /dev/null +++ b/obp-api/src/test/scala/code/api/v7_0_0/FilesTest.scala @@ -0,0 +1,388 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ +package code.api.v7_0_0 + +import java.security.MessageDigest +import java.util.Base64 + +import code.api.Constant.DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID +import code.api.util.APIUtil.OAuth._ +import code.api.util.ApiRole.{CanCreateFileAtOneBank, CanCreatePublicFileAtOneBank, CanGetFileAtOneBank} +import code.api.util.ErrorMessages._ +import code.api.v6_0_0.V600ServerSetup +import code.entitlement.Entitlement +import code.model.dataAccess.SelfServiceBank +import net.liftweb.mapper.By +import code.scope.Scope +import code.setup.{APIResponse, OBPReq} +import com.openbankproject.commons.model.ErrorMessage +import com.openbankproject.commons.util.ApiVersion +import org.json4s.native.Serialization.write +import org.scalatest.Tag + +/** + * This suite checks the v7.0.0 file endpoints: who may upload, read, attach and share files, the checks + * on an upload (media type, content, size, sha256), the bytes and headers of the content endpoints, and + * that the public endpoints serve public files only, without logging in. + */ +class FilesTest extends V600ServerSetup { + + def v7_0_0_Request = baseRequest / "obp" / "v7.0.0" + + object VersionOfApi extends Tag(ApiVersion.v7_0_0.toString) + object Files extends Tag("files") + + private lazy val bankId = testBankId1.value + private val SYS = DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID + + private def files(bank: String = bankId) = v7_0_0_Request / "banks" / bank / "files" + private def publicFiles(bank: String = bankId) = v7_0_0_Request / "banks" / bank / "public-files" + + // The smallest content that passes the type check: the type's signature followed by some bytes. + private val pngBytes: Array[Byte] = Array(0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a).map(_.toByte) ++ "a test image".getBytes("UTF-8") + private val pdfBytes: Array[Byte] = "%PDF-1.7\n% a test document\n".getBytes("UTF-8") + + private def sha256Hex(bytes: Array[Byte]): String = + MessageDigest.getInstance("SHA-256").digest(bytes).map("%02x".format(_)).mkString + + private def body(fileName: String = "photo.png", mediaType: String = "image/png", bytes: Array[Byte] = pngBytes, + isPublic: Option[Boolean] = None, sha256: Option[String] = None): String = + write(PostFileJsonV700(fileName, mediaType, Base64.getEncoder.encodeToString(bytes), isPublic, sha256)) + + private def withEntitlements[T](grants: (String, String, String)*)(f: => T): T = { + val added = grants.map { case (bank, userId, role) => Entitlement.entitlement.vend.addEntitlement(bank, userId, role) } + try f finally added.foreach(Entitlement.entitlement.vend.deleteEntitlement) + } + + private def canCreate(userId: String = resourceUser1.userId, bank: String = bankId) = + (bank, userId, CanCreateFileAtOneBank.toString) + + /** user1 uploads a file and the test gets its id. */ + private def uploaded(isPublic: Boolean = false, bytes: Array[Byte] = pngBytes, mediaType: String = "image/png", + bank: String = bankId): FileJsonV700 = { + val grants = canCreate(bank = bank) +: (if (isPublic) Seq((bank, resourceUser1.userId, CanCreatePublicFileAtOneBank.toString)) else Nil) + withEntitlements(grants: _*) { + val response = makePostRequest(files(bank).POST <@ (user1), body(bytes = bytes, mediaType = mediaType, isPublic = Some(isPublic))) + withClue(response.body) { response.code should equal(201) } + response.body.extract[FileJsonV700] + } + } + + /** The raw response: the test client otherwise reads every body as text, which would spoil the bytes. */ + private def rawGet(req: OBPReq): (Int, Array[Byte], okhttp3.Headers) = { + val response = OBPReq.client.newCall(req.GET.toOkHttpRequest).execute() + try (response.code(), Option(response.body()).map(_.bytes()).getOrElse(Array.emptyByteArray), response.headers()) + finally response.close() + } + + private def message(response: APIResponse): String = response.body.extract[ErrorMessage].message + + feature(s"Create File - POST /banks/BANK_ID/files - $VersionOfApi") { + + scenario("Anonymous access fails with 401", Files, VersionOfApi) { + makePostRequest(files().POST, body()).code should equal(401) + } + + scenario("A user without CanCreateFileAtOneBank gets 403", Files, VersionOfApi) { + val response = makePostRequest(files().POST <@ (user2), body()) + response.code should equal(403) + message(response) should include(CanCreateFileAtOneBank.toString) + } + + scenario("A user with CanCreateFileAtOneBank stores a file and owns it", Files, VersionOfApi) { + val file = uploaded() + file.bank_id should equal(bankId) + file.file_name should equal("photo.png") + file.media_type should equal("image/png") + file.size_in_bytes should equal(pngBytes.length.toLong) + file.sha256 should equal(sha256Hex(pngBytes)) + file.is_public should equal(false) + file.owner_user_id should equal(resourceUser1.userId) + } + + scenario("In a self-service bank a file stored with the Role starts pending: the Role vouches for nothing there", Files, VersionOfApi) { + uploaded().scan_status should equal("clean") + SelfServiceBank.record(bankId, resourceUser1.userId) + try uploaded().scan_status should equal("pending") + finally SelfServiceBank.findAll(By(SelfServiceBank.BankId, bankId)).foreach(_.delete_!) + } + + scenario("The Role at another bank does not cover this bank, and there is no any-bank Role", Files, VersionOfApi) { + withEntitlements(canCreate(bank = SYS)) { + makePostRequest(files().POST <@ (user1), body()).code should equal(403) + } + } + + scenario("The Role at SYS lets a user store a file in the system space", Files, VersionOfApi) { + uploaded(bank = SYS).bank_id should equal(SYS) + } + + scenario("is_public needs CanCreatePublicFileAtOneBank as well", Files, VersionOfApi) { + withEntitlements(canCreate()) { + val response = makePostRequest(files().POST <@ (user1), body(isPublic = Some(true))) + response.code should equal(403) + message(response) should include(CanCreatePublicFileAtOneBank.toString) + } + uploaded(isPublic = true).is_public should equal(true) + } + + scenario("Uploads that fail the checks get 400", Files, VersionOfApi) { + withEntitlements(canCreate()) { + def post(json: String) = makePostRequest(files().POST <@ (user1), json) + + Then("a media type outside the allowed list") + message(post(body(fileName = "page.html", mediaType = "text/html", bytes = "".getBytes("UTF-8")))) should include(FileMediaTypeNotAllowed) + + And("content that is not of the media type it claims") + message(post(body(mediaType = "image/png", bytes = pdfBytes))) should include(InvalidFile) + + And("content that is not base64") + message(post(write(PostFileJsonV700("photo.png", "image/png", "not base64 !!", None, None)))) should include(InvalidFile) + + And("a sha256 that does not match the content") + message(post(body(sha256 = Some(sha256Hex(pdfBytes))))) should include(FileSha256Mismatch) + + And("a matching sha256 is accepted, in any letter case") + post(body(sha256 = Some(sha256Hex(pngBytes).toUpperCase))).code should equal(201) + } + } + + scenario("A file larger than files.max_size_in_bytes gets 400", Files, VersionOfApi) { + setPropsValues("files.max_size_in_bytes" -> "10") + withEntitlements(canCreate()) { + val response = makePostRequest(files().POST <@ (user1), body()) + response.code should equal(400) + message(response) should include(FileTooLarge) + } + } + + scenario("files.allowed_media_types can narrow the list but never extend it", Files, VersionOfApi) { + setPropsValues("files.allowed_media_types" -> "application/pdf,text/html") + withEntitlements(canCreate()) { + message(makePostRequest(files().POST <@ (user1), body())) should include(FileMediaTypeNotAllowed) + message(makePostRequest(files().POST <@ (user1), + body(fileName = "page.html", mediaType = "text/html", bytes = "".getBytes("UTF-8")))) should include(FileMediaTypeNotAllowed) + makePostRequest(files().POST <@ (user1), body(fileName = "doc.pdf", mediaType = "application/pdf", bytes = pdfBytes)).code should equal(201) + } + } + + scenario("A Consumer holding the Role as a Scope stores a file as itself, owned by its own User", Files, VersionOfApi) { + val scope = Scope.scope.vend.addScope(bankId, testConsumer2.id.get.toString, CanCreateFileAtOneBank.toString) + try { + makePostRequest(files().POST <@ (user2), body()).code should equal(403) + val response = makePostRequest(files().POST <@ (application2), body()) + response.code should equal(201) + val ownUser = code.model.UserX.findByProviderId(code.api.OAuth2Login.OBPOIDC.clientCredentialsProvider, testConsumer2.key.get) + .map(_.userId).openOrThrowException("own User") + response.body.extract[FileJsonV700].owner_user_id should equal(ownUser) + } finally Scope.scope.vend.deleteScope(scope) + } + } + + feature(s"Reading a file - Get File, Get File Content, Get My Files - $VersionOfApi") { + + scenario("The owner reads it; another user only with CanGetFileAtOneBank or access", Files, VersionOfApi) { + val file = uploaded() + val one = files() / file.file_id + + makeGetRequest(one.GET).code should equal(401) + makeGetRequest(one.GET <@ (user1)).body.extract[FileJsonV700].file_id should equal(file.file_id) + + val refused = makeGetRequest(one.GET <@ (user2)) + refused.code should equal(403) + message(refused) should include(CanGetFileAtOneBank.toString) + + withEntitlements((bankId, resourceUser2.userId, CanGetFileAtOneBank.toString)) { + makeGetRequest(one.GET <@ (user2)).code should equal(200) + } + + makePostRequest((one / "access").POST <@ (user1), write(PostFileAccessJsonV700(resourceUser2.userId))).code should equal(201) + makeGetRequest(one.GET <@ (user2)).code should equal(200) + makeGetRequest((v7_0_0_Request / "banks" / bankId / "my" / "files").GET <@ (user2)) + .body.extract[FilesJsonV700].files.map(_.file_id) should contain(file.file_id) + + makeDeleteRequest((one / "access" / resourceUser2.userId).DELETE <@ (user1)).code should equal(204) + makeGetRequest(one.GET <@ (user2)).code should equal(403) + } + + scenario("A file of another space is not found", Files, VersionOfApi) { + val file = uploaded() + makeGetRequest((files(SYS) / file.file_id).GET <@ (user1)).code should equal(404) + message(makeGetRequest((files() / "no-such-file").GET <@ (user1))) should include(FileNotFound) + } + + scenario("Get File Content returns the bytes with the file's headers", Files, VersionOfApi) { + val file = uploaded() + val (code, bytes, headers) = rawGet(files() / file.file_id / "content" <@ (user1)) + code should equal(200) + bytes.toSeq should equal(pngBytes.toSeq) + headers.get("Content-Type") should startWith("image/png") + headers.get("ETag") should equal("\"" + sha256Hex(pngBytes) + "\"") + headers.get("X-Content-Type-Options") should equal("nosniff") + headers.get("Content-Security-Policy") should include("sandbox") + headers.get("Cache-Control") should equal("private, no-store") + headers.get("Content-Disposition") should equal("attachment; filename=\"photo.png\"") + + rawGet((files() / file.file_id / "content" <@ (user1)) < "true"))._3 + .get("Content-Disposition") should startWith("inline") + + rawGet(files() / file.file_id / "content" <@ (user2))._1 should equal(403) + } + + scenario("Get My Files lists the files a user owns", Files, VersionOfApi) { + val file = uploaded() + makeGetRequest((v7_0_0_Request / "banks" / bankId / "my" / "files").GET <@ (user1)) + .body.extract[FilesJsonV700].files.map(_.file_id) should contain(file.file_id) + } + } + + feature(s"Get Files of a Space - GET /management/banks/BANK_ID/files - $VersionOfApi") { + + scenario("Lists every file of the space, newest first, for a holder of CanGetFileAtOneBank only", Files, VersionOfApi) { + val older = uploaded() + val newer = uploaded() + val request = v7_0_0_Request / "management" / "banks" / bankId / "files" + + makeGetRequest(request.GET).code should equal(401) + val refused = makeGetRequest(request.GET <@ (user2)) + refused.code should equal(403) + message(refused) should include(CanGetFileAtOneBank.toString) + + withEntitlements((bankId, resourceUser2.userId, CanGetFileAtOneBank.toString)) { + val page = makeGetRequest(request.GET <@ (user2)).body.extract[FilesPageJsonV700] + val ids = page.files.map(_.file_id) + ids should contain allOf (older.file_id, newer.file_id) + ids.indexOf(newer.file_id) should be < ids.indexOf(older.file_id) + page.pagination.total should be >= 2L + + val first = makeGetRequest(request.GET <@ (user2) < "1")).body.extract[FilesPageJsonV700] + first.files.size should equal(1) + first.pagination.limit should equal(1) + + message(makeGetRequest(request.GET <@ (user2) < "0"))) should include(InvalidFileQueryParameter) + } + } + } + + feature(s"Public files - $VersionOfApi") { + + scenario("A public file is served without logging in, and says nothing of who stored it", Files, VersionOfApi) { + val file = uploaded(isPublic = true, bytes = pdfBytes, mediaType = "application/pdf") + + val metadata = makeGetRequest((publicFiles() / file.file_id).GET) + metadata.code should equal(200) + metadata.body.extract[PublicFileJsonV700].sha256 should equal(sha256Hex(pdfBytes)) + (metadata.body \ "owner_user_id").toOption should equal(None) + + val (code, bytes, headers) = rawGet(publicFiles() / file.file_id / "content") + code should equal(200) + bytes.toSeq should equal(pdfBytes.toSeq) + headers.get("Content-Type") should startWith("application/pdf") + headers.get("Cache-Control") should equal("public, max-age=3600") + } + + scenario("A file that is not public is not found on the public endpoints, even for its owner", Files, VersionOfApi) { + val file = uploaded() + message(makeGetRequest((publicFiles() / file.file_id).GET)) should include(FileNotFound) + makeGetRequest((publicFiles() / file.file_id).GET <@ (user1)).code should equal(404) + rawGet(publicFiles() / file.file_id / "content")._1 should equal(404) + } + + scenario("Update File Is Public needs CanCreatePublicFileAtOneBank, and withdraws a file from the public endpoints", Files, VersionOfApi) { + val file = uploaded(isPublic = true) + val isPublic = files() / file.file_id / "is-public" + + val refused = makePutRequest(isPublic.PUT <@ (user1), write(PutFileIsPublicJsonV700(false))) + refused.code should equal(403) + message(refused) should include(CanCreatePublicFileAtOneBank.toString) + + withEntitlements((bankId, resourceUser1.userId, CanCreatePublicFileAtOneBank.toString)) { + val withdrawn = makePutRequest(isPublic.PUT <@ (user1), write(PutFileIsPublicJsonV700(false))) + withdrawn.code should equal(200) + withdrawn.body.extract[FileJsonV700].is_public should equal(false) + } + makeGetRequest((publicFiles() / file.file_id).GET).code should equal(404) + } + } + + feature(s"Attachments and access - $VersionOfApi") { + + scenario("The owner attaches a file to a record once; readers see it on the record", Files, VersionOfApi) { + val privateFile = uploaded() + val publicFile = uploaded(isPublic = true) + val record = PostFileAttachmentJsonV700("customer", s"customer-${java.util.UUID.randomUUID()}") + + makePostRequest((files() / privateFile.file_id / "attachments").POST <@ (user2), write(record)).code should equal(403) + + val attached = makePostRequest((files() / privateFile.file_id / "attachments").POST <@ (user1), write(record)) + attached.code should equal(201) + val attachment = attached.body.extract[FileAttachmentJsonV700] + message(makePostRequest((files() / privateFile.file_id / "attachments").POST <@ (user1), write(record))) should include(FileAttachmentAlreadyExists) + makePostRequest((files() / publicFile.file_id / "attachments").POST <@ (user1), write(record)).code should equal(201) + + val onRecord = List("record_type" -> record.record_type, "record_id" -> record.record_id) + makeGetRequest(files().GET <@ (user1) < CommBankId, CreditLimit, CreditRating, CustomerFaceImage} import fs2.Stream @@ -59,7 +59,7 @@ import org.typelevel.ci.CIString import java.util.Date import code.setup.ServerSetupWithTestData import org.json4s.JValue -import org.json4s.JsonAST.{JArray, JBool, JField, JInt, JNull, JObject, JString} +import org.json4s.JsonAST.{JArray, JBool, JField, JInt, JNothing, JNull, JObject, JString} import com.openbankproject.commons.util.JsonAliases.parse import org.scalatest.Tag import com.openbankproject.commons.util.JsonAliases.RichJField @@ -174,9 +174,7 @@ class Http4s700RoutesTest extends ServerSetupWithTestData { visibility: String = "public", status: String = "active" ): Unit = { - Organisations.organisation.vend.createOrganisation( - orgId, s"Test $orgId", None, None, status, visibility, resourceUser1.userId - ) + Tenants.tenant.vend.createTenant(orgId, s"Test $orgId", None, None, status, visibility, resourceUser1.userId) } // ─── root ──────────────────────────────────────────────────────────────────── @@ -589,6 +587,55 @@ class Http4s700RoutesTest extends ServerSetupWithTestData { } } + // ─── getAccountDirectory ────────────────────────────────────────────────────── + + feature("Http4s700 getAccountDirectory endpoint") { + + def accountDirectoryPath = s"/obp/v7.0.0/banks/${testBankId1.value}/account-directory" + + scenario("Reject an unauthenticated request", Http4s700RoutesTag) { + val (statusCode, json, _) = makeHttpRequest(accountDirectoryPath) + statusCode shouldBe 401 + messageOf(json) should include(AuthenticatedUserIsRequired) + } + + scenario("Return 403 without CanGetAccountDirectoryAtOneBank", Http4s700RoutesTag) { + val (statusCode, json, _) = makeHttpRequest(accountDirectoryPath, Map("DirectLogin" -> s"token=${token2.value}")) + statusCode shouldBe 403 + messageOf(json) should include(UserHasMissingRoles) + messageOf(json) should include(canGetAccountDirectoryAtOneBank.toString) + } + + scenario("List an account created through v7.0.0 with who created it, and one without a creation record without metadata", Http4s700RoutesTag) { + Given("resourceUser1 creates an account through v7.0.0 Create Account, and holds CanGetAccountDirectoryAtOneBank") + addEntitlement(testBankId1.value, resourceUser1.userId, canCreateAccount.toString) + addEntitlement(testBankId1.value, resourceUser1.userId, canGetAccountDirectoryAtOneBank.toString) + val headers = Map("DirectLogin" -> s"token=${token1.value}") + val createdId = s"v7-directory-${APIUtil.generateUUID().take(12)}" + val (createStatus, _, _) = makeHttpRequestWithBody( + "PUT", s"/obp/v7.0.0/banks/${testBankId1.value}/accounts/$createdId", createAccountBody(), headers) + createStatus shouldBe 201 + + When("GET /banks/BANK_ID/account-directory") + val (statusCode, json, _) = makeHttpRequest(s"$accountDirectoryPath?limit=1000", headers) + + Then("The created account carries metadata.created naming resourceUser1 as caller and as the User it was for") + statusCode shouldBe 200 + val accounts = (json \ "accounts").children + val created = accounts.find(a => (a \ "account_id") == JString(createdId)) + .getOrElse(fail(s"Expected $createdId in the directory")) + (created \ "bank_id") shouldBe JString(testBankId1.value) + (created \ "metadata" \ "created" \ "user_id") shouldBe JString(resourceUser1.userId) + (created \ "metadata" \ "created" \ "on_behalf_of_user_id") shouldBe JString(resourceUser1.userId) + (created \ "metadata" \ "created" \ "consent_reference_id") shouldBe JNothing + + And("An account set up by the test fixture, with no creation record, has no metadata") + val fixtureAccount = accounts.find(a => (a \ "account_id") == JString(testAccountId0.value)) + .getOrElse(fail(s"Expected ${testAccountId0.value} in the directory")) + (fixtureAccount \ "metadata") shouldBe JNothing + } + } + // ─── same-bank corridor guard ───────────────────────────────────────────────── feature("Http4s700 OPEN_CORRIDOR same-bank guard") { @@ -1656,12 +1703,12 @@ class Http4s700RoutesTest extends ServerSetupWithTestData { } } - feature("Http4s700 createOrganisation endpoint") { + feature("Http4s700 createTenant endpoint") { - scenario("Reject unauthenticated POST to /organisations", Http4s700RoutesTag) { - Given("POST /obp/v7.0.0/organisations with no auth") - val body = """{"organisation_id":"test-org-401","name":"X"}""" - val (statusCode, json, _) = makeHttpRequestWithBody("POST", "/obp/v7.0.0/organisations", body) + scenario("Reject unauthenticated POST to /tenants", Http4s700RoutesTag) { + Given("POST /obp/v7.0.0/tenants with no auth") + val body = """{"tenant_id":"test-org-401","name":"X"}""" + val (statusCode, json, _) = makeHttpRequestWithBody("POST", "/obp/v7.0.0/tenants", body) Then("Response is 401") statusCode shouldBe 401 @@ -1675,11 +1722,11 @@ class Http4s700RoutesTest extends ServerSetupWithTestData { } } - scenario("Return 403 when authenticated but missing canCreateOrganisation role", Http4s700RoutesTag) { + scenario("Return 403 when authenticated but missing canCreateTenant role", Http4s700RoutesTag) { Given("DirectLogin without the required role") - val body = """{"organisation_id":"test-org-403","name":"X"}""" + val body = """{"tenant_id":"test-org-403","name":"X"}""" val headers = Map("DirectLogin" -> s"token=${token1.value}") - val (statusCode, json, _) = makeHttpRequestWithBody("POST", "/obp/v7.0.0/organisations", body, headers) + val (statusCode, json, _) = makeHttpRequestWithBody("POST", "/obp/v7.0.0/tenants", body, headers) Then("Response is 403 with UserHasMissingRoles message") statusCode shouldBe 403 @@ -1693,67 +1740,67 @@ class Http4s700RoutesTest extends ServerSetupWithTestData { } } - scenario("Return 201 with organisation JSON when authenticated with role and valid body", Http4s700RoutesTag) { - Given("canCreateOrganisation granted to caller") - addEntitlement("", resourceUser1.userId, canCreateOrganisation.toString) + scenario("Return 201 with tenant JSON when authenticated with role and valid body", Http4s700RoutesTag) { + Given("canCreateTenant granted to caller") + addEntitlement("", resourceUser1.userId, canCreateTenant.toString) val orgId = s"test-org-${APIUtil.generateUUID().take(8)}" - val body = s"""{"organisation_id":"$orgId","name":"Test Org"}""" + val body = s"""{"tenant_id":"$orgId","name":"Test Org"}""" val headers = Map("DirectLogin" -> s"token=${token1.value}") - When(s"POST /obp/v7.0.0/organisations with organisation_id=$orgId") - val (statusCode, json, _) = makeHttpRequestWithBody("POST", "/obp/v7.0.0/organisations", body, headers) + When(s"POST /obp/v7.0.0/tenants with tenant_id=$orgId") + val (statusCode, json, _) = makeHttpRequestWithBody("POST", "/obp/v7.0.0/tenants", body, headers) Then("Response is 201 with the expected fields") statusCode shouldBe 201 json match { case JObject(fields) => val map = toFieldMap(fields) - map.keys should contain allOf ("organisation_id", "name", "status", "visibility", "created_by_user_id") - map.get("organisation_id") shouldBe Some(JString(orgId)) + map.keys should contain allOf ("tenant_id", "name", "status", "visibility", "created_by_user_id") + map.get("tenant_id") shouldBe Some(JString(orgId)) map.get("status") shouldBe Some(JString("active")) map.get("visibility") shouldBe Some(JString("public")) case _ => fail("Expected JSON object") } } - scenario("Return 400 when organisation_id format is invalid", Http4s700RoutesTag) { - Given("canCreateOrganisation granted; organisation_id contains an invalid character") - addEntitlement("", resourceUser1.userId, canCreateOrganisation.toString) - val body = """{"organisation_id":"bad id with spaces","name":"X"}""" + scenario("Return 400 when tenant_id format is invalid", Http4s700RoutesTag) { + Given("canCreateTenant granted; tenant_id contains an invalid character") + addEntitlement("", resourceUser1.userId, canCreateTenant.toString) + val body = """{"tenant_id":"bad id with spaces","name":"X"}""" val headers = Map("DirectLogin" -> s"token=${token1.value}") - When("POST /obp/v7.0.0/organisations with invalid id") - val (statusCode, json, _) = makeHttpRequestWithBody("POST", "/obp/v7.0.0/organisations", body, headers) + When("POST /obp/v7.0.0/tenants with invalid id") + val (statusCode, json, _) = makeHttpRequestWithBody("POST", "/obp/v7.0.0/tenants", body, headers) - Then("Response is 400 with InvalidOrganisationIdFormat message") + Then("Response is 400 with InvalidTenantIdFormat message") statusCode shouldBe 400 json match { case JObject(fields) => toFieldMap(fields).get("message") match { - case Some(JString(msg)) => msg should include(InvalidOrganisationIdFormat) + case Some(JString(msg)) => msg should include(InvalidTenantIdFormat) case _ => fail("Expected message field") } case _ => fail("Expected JSON object") } } - scenario("Return 409 when organisation already exists", Http4s700RoutesTag) { - Given("an organisation already exists; canCreateOrganisation granted") - addEntitlement("", resourceUser1.userId, canCreateOrganisation.toString) + scenario("Return 409 when tenant already exists", Http4s700RoutesTag) { + Given("an tenant already exists; canCreateTenant granted") + addEntitlement("", resourceUser1.userId, canCreateTenant.toString) val orgId = s"dup-org-${APIUtil.generateUUID().take(8)}" createTestOrg(orgId) - When("POST /obp/v7.0.0/organisations with the same organisation_id") - val body = s"""{"organisation_id":"$orgId","name":"X"}""" + When("POST /obp/v7.0.0/tenants with the same tenant_id") + val body = s"""{"tenant_id":"$orgId","name":"X"}""" val headers = Map("DirectLogin" -> s"token=${token1.value}") - val (statusCode, json, _) = makeHttpRequestWithBody("POST", "/obp/v7.0.0/organisations", body, headers) + val (statusCode, json, _) = makeHttpRequestWithBody("POST", "/obp/v7.0.0/tenants", body, headers) - Then("Response is 409 with OrganisationAlreadyExists message") + Then("Response is 409 with TenantAlreadyExists message") statusCode shouldBe 409 json match { case JObject(fields) => toFieldMap(fields).get("message") match { - case Some(JString(msg)) => msg should include(OrganisationAlreadyExists) + case Some(JString(msg)) => msg should include(TenantAlreadyExists) case _ => fail("Expected message field") } case _ => fail("Expected JSON object") @@ -1761,11 +1808,11 @@ class Http4s700RoutesTest extends ServerSetupWithTestData { } } - feature("Http4s700 getOrganisations endpoint") { + feature("Http4s700 getTenants endpoint") { - scenario("Reject unauthenticated GET to /organisations", Http4s700RoutesTag) { - Given("GET /obp/v7.0.0/organisations with no auth") - val (statusCode, json, _) = makeHttpRequest("/obp/v7.0.0/organisations") + scenario("Reject unauthenticated GET to /tenants", Http4s700RoutesTag) { + Given("GET /obp/v7.0.0/tenants with no auth") + val (statusCode, json, _) = makeHttpRequest("/obp/v7.0.0/tenants") Then("Response is 401") statusCode shouldBe 401 @@ -1779,33 +1826,33 @@ class Http4s700RoutesTest extends ServerSetupWithTestData { } } - scenario("Return 200 with organisations array for an authenticated user", Http4s700RoutesTag) { - Given("an organisation exists") + scenario("Return 200 with tenants array for an authenticated user", Http4s700RoutesTag) { + Given("an tenant exists") val orgId = s"list-org-${APIUtil.generateUUID().take(8)}" createTestOrg(orgId) val headers = Map("DirectLogin" -> s"token=${token1.value}") - When("GET /obp/v7.0.0/organisations") - val (statusCode, json, _) = makeHttpRequest("/obp/v7.0.0/organisations", headers) + When("GET /obp/v7.0.0/tenants") + val (statusCode, json, _) = makeHttpRequest("/obp/v7.0.0/tenants", headers) - Then("Response is 200 with an organisations array") + Then("Response is 200 with an tenants array") statusCode shouldBe 200 json match { case JObject(fields) => - toFieldMap(fields).get("organisations") match { + toFieldMap(fields).get("tenants") match { case Some(JArray(_)) => succeed - case _ => fail("Expected organisations array") + case _ => fail("Expected tenants array") } case _ => fail("Expected JSON object") } } } - feature("Http4s700 getOrganisation endpoint") { + feature("Http4s700 getTenant endpoint") { - scenario("Reject unauthenticated GET to /organisations/ORGANISATION_ID", Http4s700RoutesTag) { - Given("GET /obp/v7.0.0/organisations/anything with no auth") - val (statusCode, json, _) = makeHttpRequest("/obp/v7.0.0/organisations/anything") + scenario("Reject unauthenticated GET to /tenants/TENANT_ID", Http4s700RoutesTag) { + Given("GET /obp/v7.0.0/tenants/anything with no auth") + val (statusCode, json, _) = makeHttpRequest("/obp/v7.0.0/tenants/anything") Then("Response is 401") statusCode shouldBe 401 @@ -1819,52 +1866,52 @@ class Http4s700RoutesTest extends ServerSetupWithTestData { } } - scenario("Return 404 when organisation does not exist", Http4s700RoutesTag) { - Given("an authenticated user; organisation_id that does not exist") + scenario("Return 404 when tenant does not exist", Http4s700RoutesTag) { + Given("an authenticated user; tenant_id that does not exist") val headers = Map("DirectLogin" -> s"token=${token1.value}") - When("GET /obp/v7.0.0/organisations/no-such-org-xyz") - val (statusCode, json, _) = makeHttpRequest("/obp/v7.0.0/organisations/no-such-org-xyz", headers) + When("GET /obp/v7.0.0/tenants/no-such-org-xyz") + val (statusCode, json, _) = makeHttpRequest("/obp/v7.0.0/tenants/no-such-org-xyz", headers) - Then("Response is 404 with OrganisationNotFound message") + Then("Response is 404 with TenantNotFound message") statusCode shouldBe 404 json match { case JObject(fields) => toFieldMap(fields).get("message") match { - case Some(JString(msg)) => msg should include(OrganisationNotFound) + case Some(JString(msg)) => msg should include(TenantNotFound) case _ => fail("Expected message field") } case _ => fail("Expected JSON object") } } - scenario("Return 200 with organisation JSON for an existing public organisation", Http4s700RoutesTag) { - Given("a public organisation exists") + scenario("Return 200 with tenant JSON for an existing public tenant", Http4s700RoutesTag) { + Given("a public tenant exists") val orgId = s"get-org-${APIUtil.generateUUID().take(8)}" createTestOrg(orgId, visibility = "public") val headers = Map("DirectLogin" -> s"token=${token1.value}") - When(s"GET /obp/v7.0.0/organisations/$orgId") - val (statusCode, json, _) = makeHttpRequest(s"/obp/v7.0.0/organisations/$orgId", headers) + When(s"GET /obp/v7.0.0/tenants/$orgId") + val (statusCode, json, _) = makeHttpRequest(s"/obp/v7.0.0/tenants/$orgId", headers) Then("Response is 200 with the expected fields") statusCode shouldBe 200 json match { case JObject(fields) => val map = toFieldMap(fields) - map.get("organisation_id") shouldBe Some(JString(orgId)) + map.get("tenant_id") shouldBe Some(JString(orgId)) map.get("visibility") shouldBe Some(JString("public")) case _ => fail("Expected JSON object") } } } - feature("Http4s700 updateOrganisation endpoint") { + feature("Http4s700 updateTenant endpoint") { - scenario("Reject unauthenticated PUT to /organisations/ORGANISATION_ID", Http4s700RoutesTag) { - Given("PUT /obp/v7.0.0/organisations/anything with no auth") + scenario("Reject unauthenticated PUT to /tenants/TENANT_ID", Http4s700RoutesTag) { + Given("PUT /obp/v7.0.0/tenants/anything with no auth") val body = """{"name":"New Name"}""" - val (statusCode, json, _) = makeHttpRequestWithBody("PUT", "/obp/v7.0.0/organisations/anything", body) + val (statusCode, json, _) = makeHttpRequestWithBody("PUT", "/obp/v7.0.0/tenants/anything", body) Then("Response is 401") statusCode shouldBe 401 @@ -1878,11 +1925,11 @@ class Http4s700RoutesTest extends ServerSetupWithTestData { } } - scenario("Return 403 when authenticated but missing canUpdateOrganisation role", Http4s700RoutesTag) { + scenario("Return 403 when authenticated but missing canUpdateTenant role", Http4s700RoutesTag) { Given("DirectLogin without the required role") val body = """{"name":"New Name"}""" val headers = Map("DirectLogin" -> s"token=${token1.value}") - val (statusCode, json, _) = makeHttpRequestWithBody("PUT", "/obp/v7.0.0/organisations/anything", body, headers) + val (statusCode, json, _) = makeHttpRequestWithBody("PUT", "/obp/v7.0.0/tenants/anything", body, headers) Then("Response is 403 with UserHasMissingRoles message") statusCode shouldBe 403 @@ -1896,43 +1943,43 @@ class Http4s700RoutesTest extends ServerSetupWithTestData { } } - scenario("Return 200 with updated organisation JSON when authenticated with role", Http4s700RoutesTag) { - Given("an organisation exists; canUpdateOrganisation granted") - addEntitlement("", resourceUser1.userId, canUpdateOrganisation.toString) + scenario("Return 200 with updated tenant JSON when authenticated with role", Http4s700RoutesTag) { + Given("an tenant exists; canUpdateTenant granted") + addEntitlement("", resourceUser1.userId, canUpdateTenant.toString) val orgId = s"upd-org-${APIUtil.generateUUID().take(8)}" createTestOrg(orgId) - When(s"PUT /obp/v7.0.0/organisations/$orgId with a new name") + When(s"PUT /obp/v7.0.0/tenants/$orgId with a new name") val body = """{"name":"Updated Name"}""" val headers = Map("DirectLogin" -> s"token=${token1.value}") - val (statusCode, json, _) = makeHttpRequestWithBody("PUT", s"/obp/v7.0.0/organisations/$orgId", body, headers) + val (statusCode, json, _) = makeHttpRequestWithBody("PUT", s"/obp/v7.0.0/tenants/$orgId", body, headers) Then("Response is 200 and name reflects the update") statusCode shouldBe 200 json match { case JObject(fields) => val map = toFieldMap(fields) - map.get("organisation_id") shouldBe Some(JString(orgId)) + map.get("tenant_id") shouldBe Some(JString(orgId)) map.get("name") shouldBe Some(JString("Updated Name")) case _ => fail("Expected JSON object") } } } - feature("Http4s700 deleteOrganisation endpoint") { + feature("Http4s700 deleteTenant endpoint") { - scenario("Reject unauthenticated DELETE to /organisations/ORGANISATION_ID", Http4s700RoutesTag) { - Given("DELETE /obp/v7.0.0/organisations/anything with no auth") - val (statusCode, _, _) = makeHttpRequestWithMethod("DELETE", "/obp/v7.0.0/organisations/anything") + scenario("Reject unauthenticated DELETE to /tenants/TENANT_ID", Http4s700RoutesTag) { + Given("DELETE /obp/v7.0.0/tenants/anything with no auth") + val (statusCode, _, _) = makeHttpRequestWithMethod("DELETE", "/obp/v7.0.0/tenants/anything") Then("Response is 401") statusCode shouldBe 401 } - scenario("Return 403 when authenticated but missing canDeleteOrganisation role", Http4s700RoutesTag) { + scenario("Return 403 when authenticated but missing canDeleteTenant role", Http4s700RoutesTag) { Given("DirectLogin without the required role") val headers = Map("DirectLogin" -> s"token=${token1.value}") - val (statusCode, json, _) = makeHttpRequestWithMethod("DELETE", "/obp/v7.0.0/organisations/anything", headers) + val (statusCode, json, _) = makeHttpRequestWithMethod("DELETE", "/obp/v7.0.0/tenants/anything", headers) Then("Response is 403 with UserHasMissingRoles message") statusCode shouldBe 403 @@ -1946,21 +1993,93 @@ class Http4s700RoutesTest extends ServerSetupWithTestData { } } - scenario("Return 204 when authenticated with role and organisation exists", Http4s700RoutesTag) { - Given("an organisation exists; canDeleteOrganisation granted") - addEntitlement("", resourceUser1.userId, canDeleteOrganisation.toString) + scenario("Return 204 when authenticated with role and tenant exists", Http4s700RoutesTag) { + Given("an tenant exists; canDeleteTenant granted") + addEntitlement("", resourceUser1.userId, canDeleteTenant.toString) val orgId = s"del-org-${APIUtil.generateUUID().take(8)}" createTestOrg(orgId) - When(s"DELETE /obp/v7.0.0/organisations/$orgId") + When(s"DELETE /obp/v7.0.0/tenants/$orgId") val headers = Map("DirectLogin" -> s"token=${token1.value}") - val (statusCode, _, _) = makeHttpRequestWithMethod("DELETE", s"/obp/v7.0.0/organisations/$orgId", headers) + val (statusCode, _, _) = makeHttpRequestWithMethod("DELETE", s"/obp/v7.0.0/tenants/$orgId", headers) Then("Response is 204 with no body") statusCode shouldBe 204 } } + feature("Http4s700 Tenants belong to no bank") { + + def messageOf(json: JValue): String = json match { + case JObject(fields) => toFieldMap(fields).get("message") match { + case Some(JString(msg)) => msg + case _ => fail("Expected message field") + } + case _ => fail("Expected JSON object") + } + + def tenantIdsIn(json: JValue): List[String] = json match { + case JObject(fields) => toFieldMap(fields).get("tenants") match { + case Some(JArray(orgs)) => orgs.collect { case JObject(f) => toFieldMap(f).get("tenant_id") } + .collect { case Some(JString(id)) => id } + case _ => fail("Expected tenants array") + } + case _ => fail("Expected JSON object") + } + + val headers = Map("DirectLogin" -> s"token=${token1.value}") + + scenario("A Tenant Role granted at a bank authorises nothing: they are system Roles", Http4s700RoutesTag) { + Given("canCreateTenant granted at a bank only") + addEntitlement(testBankId2.value, resourceUser1.userId, canCreateTenant.toString) + val body = s"""{"tenant_id":"other-${APIUtil.generateUUID().take(8)}","name":"X"}""" + + When("POST /obp/v7.0.0/tenants") + val (statusCode, json, _) = makeHttpRequestWithBody("POST", "/obp/v7.0.0/tenants", body, headers) + + Then("Response is 403") + statusCode shouldBe 403 + messageOf(json) should include(UserHasMissingRoles) + } + + scenario("A Tenant names no bank", Http4s700RoutesTag) { + Given("a public Tenant") + val orgId = s"no-bank-${APIUtil.generateUUID().take(8)}" + createTestOrg(orgId) + + When("it is fetched") + val (statusCode, json, _) = makeHttpRequest(s"/obp/v7.0.0/tenants/$orgId", headers) + + Then("Response is 200 without a bank_id") + statusCode shouldBe 200 + json match { + case JObject(fields) => toFieldMap(fields).keys should not contain "bank_id" + case _ => fail("Expected JSON object") + } + } + + scenario("A private Tenant needs CanGetTenant", Http4s700RoutesTag) { + Given("a private Tenant") + val orgId = s"priv-org-${APIUtil.generateUUID().take(8)}" + createTestOrg(orgId, visibility = "private") + + When("it is fetched without the Role") + val (withoutRole, _, _) = makeHttpRequest(s"/obp/v7.0.0/tenants/$orgId", headers) + Then("Response is 403 and the list leaves it out") + withoutRole shouldBe 403 + val (_, listWithout, _) = makeHttpRequest("/obp/v7.0.0/tenants", headers) + tenantIdsIn(listWithout) should not contain orgId + + When("CanGetTenant is granted") + addEntitlement("", resourceUser1.userId, canGetTenant.toString) + val (withRole, _, _) = makeHttpRequest(s"/obp/v7.0.0/tenants/$orgId", headers) + Then("Response is 200 and the list includes it") + withRole shouldBe 200 + val (_, listWith, _) = makeHttpRequest("/obp/v7.0.0/tenants", headers) + tenantIdsIn(listWith) should contain(orgId) + } + } + // ─── Routing Schemes ────────────────────────────────────────────────────── /** Create a routing scheme directly via the model layer for test setup. */ @@ -2661,7 +2780,28 @@ class Http4s700RoutesTest extends ServerSetupWithTestData { statusCode shouldBe 401 } + scenario("Return 400 OpenCorridorDisabled when open_corridor_enabled is not set", Http4s700RoutesTag) { + val acctCurrency = code.bankconnectors.Connector.connector.vend + .getBankAccountLegacy(testBankId1, testAccountId0, None) + .map(_._1.currency).openOrThrowException("test account") + val headers = Map("DirectLogin" -> s"token=${token1.value}") + val (statusCode, json, _) = makeHttpRequestWithBody("POST", + openCorridorPromisePath(testBankId1.value, testAccountId0.value), + openCorridorPromiseBody(acctCurrency), headers) + statusCode shouldBe 400 + messageOf(json) should include(OpenCorridorDisabled) + } + + scenario("OPEN_CORRIDOR_PROMISE is advertised exactly when open_corridor_enabled is true", Http4s700RoutesTag) { + setPropsValues("transactionRequests_supported_types" -> "SIMPLE, OPEN_CORRIDOR_PROMISE,SEPA") + APIUtil.advertisedTransactionRequestTypes shouldBe List("SIMPLE", "SEPA") + + setPropsValues("open_corridor_enabled" -> "true") + APIUtil.advertisedTransactionRequestTypes shouldBe List("SIMPLE", "SEPA", "OPEN_CORRIDOR_PROMISE") + } + scenario("Return 400 InvalidJsonFormat when the originator block is missing", Http4s700RoutesTag) { + setPropsValues("open_corridor_enabled" -> "true") // Same shape but no `originator` field — extraction to the OPEN_CORRIDOR_PROMISE body class must fail. val body = """{ @@ -2691,6 +2831,7 @@ class Http4s700RoutesTest extends ServerSetupWithTestData { } scenario("Return 400 InvalidJsonValue when originator.name is empty", Http4s700RoutesTag) { + setPropsValues("open_corridor_enabled" -> "true") val headers = Map("DirectLogin" -> s"token=${token1.value}") val (statusCode, json, _) = makeHttpRequestWithBody("POST", openCorridorPromisePath(testBankId1.value, testAccountId0.value), @@ -2707,6 +2848,7 @@ class Http4s700RoutesTest extends ServerSetupWithTestData { } scenario("Return 400 InvalidJsonValue when originator.account_routing.address is empty", Http4s700RoutesTag) { + setPropsValues("open_corridor_enabled" -> "true") val headers = Map("DirectLogin" -> s"token=${token1.value}") val (statusCode, json, _) = makeHttpRequestWithBody("POST", openCorridorPromisePath(testBankId1.value, testAccountId0.value), @@ -2723,6 +2865,7 @@ class Http4s700RoutesTest extends ServerSetupWithTestData { } scenario("Return 201 with type OPEN_CORRIDOR_PROMISE and the originator echoed as explicit", Http4s700RoutesTag) { + setPropsValues("open_corridor_enabled" -> "true") val bankId = testBankId1.value val accountId = testAccountId0.value // Match the source account's currency so the payment path doesn't reject on currency. @@ -2775,6 +2918,7 @@ class Http4s700RoutesTest extends ServerSetupWithTestData { } scenario("Return 201 when the beneficiary account exists only at the far bank's CBS (not in OBP-API)", Http4s700RoutesTag) { + setPropsValues("open_corridor_enabled" -> "true") val acctCurrency = code.bankconnectors.Connector.connector.vend .getBankAccountLegacy(testBankId1, testAccountId0, None) .map(_._1.currency).openOrThrowException("test account") @@ -2807,6 +2951,7 @@ class Http4s700RoutesTest extends ServerSetupWithTestData { } scenario("Return 404 BankNotFound when the beneficiary bank is not registered", Http4s700RoutesTag) { + setPropsValues("open_corridor_enabled" -> "true") val acctCurrency = code.bankconnectors.Connector.connector.vend .getBankAccountLegacy(testBankId1, testAccountId0, None) .map(_._1.currency).openOrThrowException("test account") @@ -2878,6 +3023,7 @@ class Http4s700RoutesTest extends ServerSetupWithTestData { beneficiaryAccountId: String = testAccountId1.value, amount: String = "3.00" ): String = { + setPropsValues("open_corridor_enabled" -> "true") val acctCurrency = code.bankconnectors.Connector.connector.vend .getBankAccountLegacy(fromBankId, fromAccountId, None) .map(_._1.currency).openOrThrowException("test account") @@ -3016,6 +3162,43 @@ class Http4s700RoutesTest extends ServerSetupWithTestData { messageOf(json) should include("CanAttachOpenCorridorPromise") } + scenario("A Consumer holding CanAttachOpenCorridorPromise as a Scope may attach evidence as itself; its own User is recorded", Http4s700RoutesTag) { + Given("testConsumer2 holds the Scope at the bank; user2, a person, signs with it but holds no Entitlement") + val transactionRequestId = createPendingPromise() + val granted = code.scope.Scope.scope.vend.addScope( + testBankId1.value, testConsumer2.id.get.toString, canAttachOpenCorridorPromise.toString) + val path = promiseEvidencePath(testBankId1.value, testAccountId0.value, transactionRequestId) + val ((personCode, _, _), (statusCode, json, _)) = try { + (makeHttpRequestWithBody("POST", path, promiseEvidenceBody(), Map("DirectLogin" -> s"token=${token2.value}")), + makeHttpRequestWithBody("POST", path, promiseEvidenceBody(), Map("DirectLogin" -> s"token=${tokenApplication2.value}"))) + } finally code.scope.Scope.scope.vend.deleteScope(granted) + + Then("user2 does not get the Consumer's Scope") + personCode shouldBe 403 + + And("testConsumer2 acting as itself attaches the evidence, recorded as its own User") + statusCode shouldBe 201 + val applicationsOwnUserId = code.model.UserX.findByProviderId( + code.api.OAuth2Login.OBPOIDC.clientCredentialsProvider, testConsumer2.key.get).map(_.userId).openOrThrowException("own User") + json match { + case JObject(fields) => + toFieldMap(fields).get("reported_by_user_id") shouldBe Some(JString(applicationsOwnUserId)) + case _ => fail("Expected JSON object") + } + } + + scenario("A CanAttachOpenCorridorPromise Scope at another bank does not authorize this bank's URL", Http4s700RoutesTag) { + val headers = Map("DirectLogin" -> s"token=${tokenApplication2.value}") + val granted = code.scope.Scope.scope.vend.addScope( + testBankId2.value, testConsumer2.id.get.toString, canAttachOpenCorridorPromise.toString) + val (statusCode, json, _) = try makeHttpRequestWithBody("POST", + promiseEvidencePath(testBankId1.value, testAccountId0.value, "some-tr-id"), + promiseEvidenceBody(), headers) + finally code.scope.Scope.scope.vend.deleteScope(granted) + statusCode shouldBe 403 + messageOf(json) should include("CanAttachOpenCorridorPromise") + } + scenario("Attach evidence: 201, idempotent re-post, conflict refused", Http4s700RoutesTag) { Given("A PENDING OPEN_CORRIDOR_PROMISE and the role granted") addEntitlement(testBankId1.value, resourceUser1.userId, canAttachOpenCorridorPromise.toString) @@ -3297,6 +3480,36 @@ class Http4s700RoutesTest extends ServerSetupWithTestData { messageOf(json) should include("CanSettleOpenCorridor") } + scenario("A Consumer holding CanSettleOpenCorridor as a Scope at the bank passes the Role check as itself", Http4s700RoutesTag) { + Given("testConsumer2 holds the Scope at the bank; user2, a person, signs with it but holds no Entitlement") + val granted = code.scope.Scope.scope.vend.addScope( + testBankId1.value, testConsumer2.id.get.toString, canSettleOpenCorridor.toString) + val ((personCode, _, _), (statusCode, json, _)) = try { + (makeHttpRequestWithBody("POST", settlementsPath(testBankId1.value), settleBody("EUR"), + Map("DirectLogin" -> s"token=${token2.value}")), + makeHttpRequestWithBody("POST", settlementsPath(testBankId1.value), settleBody("EUR"), + Map("DirectLogin" -> s"token=${tokenApplication2.value}"))) + } finally code.scope.Scope.scope.vend.deleteScope(granted) + + Then("user2 does not get the Consumer's Scope") + personCode shouldBe 403 + + And("testConsumer2 acting as itself gets past the Role check and stops at open_corridor_enabled, which this test leaves off") + statusCode shouldBe 400 + messageOf(json) should include(OpenCorridorDisabled) + } + + scenario("A CanSettleOpenCorridor Scope at another bank does not authorize this bank's URL", Http4s700RoutesTag) { + val headers = Map("DirectLogin" -> s"token=${tokenApplication2.value}") + val granted = code.scope.Scope.scope.vend.addScope( + testBankId2.value, testConsumer2.id.get.toString, canSettleOpenCorridor.toString) + val (statusCode, json, _) = try makeHttpRequestWithBody("POST", + settlementsPath(testBankId1.value), settleBody("EUR"), headers) + finally code.scope.Scope.scope.vend.deleteScope(granted) + statusCode shouldBe 403 + messageOf(json) should include("CanSettleOpenCorridor") + } + scenario("The role is bank-scoped: a grant at another bank does not authorize this bank's URL", Http4s700RoutesTag) { addEntitlement(testBankId1.value, resourceUser1.userId, canSettleOpenCorridor.toString) val headers = Map("DirectLogin" -> s"token=${token1.value}") @@ -4569,6 +4782,8 @@ class Http4s700RoutesTest extends ServerSetupWithTestData { .shortBankName("Agent Made") .CreatedByUserId(agentUser.userId) .saveMe() + // What Create My Bank records, and what its quota counts. + code.model.dataAccess.SelfServiceBank.record(agentBankId, agentUser.userId) agentBankId } diff --git a/obp-api/src/test/scala/code/api/v7_0_0/PersonalFilesTest.scala b/obp-api/src/test/scala/code/api/v7_0_0/PersonalFilesTest.scala new file mode 100644 index 0000000000..16c47180a7 --- /dev/null +++ b/obp-api/src/test/scala/code/api/v7_0_0/PersonalFilesTest.scala @@ -0,0 +1,282 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ +package code.api.v7_0_0 + +import java.util.Base64 + +import code.api.util.APIUtil.OAuth._ +import code.api.util.APIUtil.generateUUID +import code.api.util.ApiRole.{CanCreateFileAtOneBank, CanGetFileAtOneBank, CanUpdateFileScanStatusAtOneBank, canCreateDynamicEntityDefinition} +import code.api.util.ErrorMessages._ +import code.api.v6_0_0.V600ServerSetup +import code.entitlement.Entitlement +import code.model.dataAccess.{AuthUser, ResourceUser} +import net.liftweb.mapper.By +import code.setup.{APIResponse, OBPReq} +import com.openbankproject.commons.model.ErrorMessage +import com.openbankproject.commons.util.ApiVersion +import org.json4s.JsonAST._ +import org.json4s.JsonDSL._ +import org.json4s.native.JsonMethods.{compact, render} +import org.json4s.native.Serialization.write +import org.scalatest.Tag + +/** + * This suite checks personal files and scan status: who may store a file in their own name without a Role and + * under which limits, that it goes only on the owner's own personal Dynamic Entity records, and that a file's + * content is served only once it is clean (to its owner and the scanner before that, to nobody when rejected). + */ +class PersonalFilesTest extends V600ServerSetup { + + def v7 = baseRequest / "obp" / "v7.0.0" + object VersionOfApi extends Tag(ApiVersion.v7_0_0.toString) + object PersonalFiles extends Tag("personalFiles") + + private lazy val bankId = testBankId1.value + // Set by defineEntities, which each scenario calls: dynamic entities do not outlive a scenario. + private var personalEntity = "" + private var sharedEntity = "" + + private val pdfBytes: Array[Byte] = "%PDF-1.7\n% a pitch deck\n".getBytes("UTF-8") + private val webpBytes: Array[Byte] = "RIFF\u0000\u0000\u0000\u0000WEBPVP8 ".getBytes("ISO-8859-1") + + /** + * The test Users are local Users with no login of their own, so their email addresses count as not validated. Each + * is given a validated login here, as a User who signed up and confirmed their address would have. + */ + private val loginsMadeHere = scala.collection.mutable.Set[Long]() + + private def setValidated(user: ResourceUser, validated: Boolean): Unit = { + val login = AuthUser.find(By(AuthUser.user, user.userPrimaryKey.value)).openOr { + // AuthUser.save copies username into the User's name and provider id, so it must be the provider id it has. + val made = AuthUser.create.user(user.userPrimaryKey.value).username(user.providerId.get).email(user.emailAddress) + .provider(code.api.Constant.localIdentityProvider).password("Aa1!" + generateUUID()).saveMe() + loginsMadeHere += made.id.get + made + } + login.validated(validated).saveMe() + } + + /** + * The logins made here go, so later suites see the test Users as they were. bulkDelete_!! removes the login rows + * only: AuthUser.delete_! would also delete the Users themselves. + */ + override def afterAll(): Unit = { + loginsMadeHere.foreach(id => AuthUser.bulkDelete_!!(By(AuthUser.id, id))) + super.afterAll() + } + + /** Personal files on in the test bank, for Users whose email addresses are validated. */ + private def enable(extra: (String, String)*): Unit = { + setPropsValues((List("files.personal.enabled_bank_ids" -> bankId) ++ extra): _*) + List(resourceUser1, resourceUser2, resourceUser3).foreach(setValidated(_, validated = true)) + } + + private def withEntitlements[T](grants: (String, String, String)*)(f: => T): T = { + val added = grants.map { case (bank, userId, role) => Entitlement.entitlement.vend.addEntitlement(bank, userId, role) } + try f finally added.foreach(Entitlement.entitlement.vend.deleteEntitlement) + } + + /** A personal and a shared dynamic entity, with fresh names, for this scenario. */ + private def defineEntities(): Unit = withEntitlements((bankId, resourceUser1.userId, canCreateDynamicEntityDefinition.toString)) { + val suffix = generateUUID().take(8) + personalEntity = s"pf_application_$suffix" + sharedEntity = s"pf_shared_$suffix" + for ((name, personal) <- List(personalEntity -> true, sharedEntity -> false)) { + val definition = compact(render(("entity_name" -> name) ~ ("has_personal_entity" -> personal) ~ + ("schema" -> (("description" -> "A test record files are attached to.") ~ ("required" -> List("title")) ~ + ("properties" -> ("title" -> (("type" -> "string") ~ ("example" -> "a title")))))))) + val response = makePostRequest((v7 / "management" / "banks" / bankId / "dynamic-entities").POST <@ (user1), definition) + withClue(response.body) { response.code should equal(201) } + } + } + + /** A personal record of `personalEntity` owned by whoever calls. */ + private def personalRecord(caller: Option[(Consumer, Token)]): String = { + val response = makePostRequest((v7 / "banks" / bankId / "dynamic-entities" / "my" / personalEntity).POST <@ (caller), + compact(render("title" -> "my application"))) + withClue(response.body) { response.code should equal(201) } + (response.body \ personalEntity \ s"${personalEntity}_id").extract[String] + } + + private def myFiles = v7 / "banks" / bankId / "my" / "files" + private def file(id: String) = v7 / "banks" / bankId / "files" / id + + private def body(recordId: String, recordType: String = personalEntity, mediaType: String = "application/pdf", + bytes: Array[Byte] = pdfBytes): String = + write(PostMyFileJsonV700("pitch_deck.pdf", mediaType, Base64.getEncoder.encodeToString(bytes), None, recordType, recordId)) + + private def message(response: APIResponse): String = response.body.extract[ErrorMessage].message + + /** The status of a content request. A served file is the file itself, which the JSON client cannot read. */ + private def contentStatus(req: OBPReq): Int = { + val response = OBPReq.client.newCall(req.GET.toOkHttpRequest).execute() + try response.code() finally response.close() + } + + private def uploaded(recordId: String): FileJsonV700 = { + val response = makePostRequest(myFiles.POST <@ (user1), body(recordId)) + withClue(response.body) { response.code should equal(201) } + response.body.extract[FileJsonV700] + } + + private def setStatus(fileId: String, status: String): APIResponse = + withEntitlements((bankId, resourceUser2.userId, CanUpdateFileScanStatusAtOneBank.toString)) { + makePutRequest((file(fileId) / "scan-status").PUT <@ (user2), write(PutFileScanStatusJsonV700(status, Some("test")))) + } + + feature(s"Create My File - POST /banks/BANK_ID/my/files - $VersionOfApi") { + + scenario("Personal files are off unless the space is listed", PersonalFiles, VersionOfApi) { + defineEntities() + setPropsValues("files.personal.enabled_bank_ids" -> "some-other-bank") + val response = makePostRequest(myFiles.POST <@ (user1), body(personalRecord(user1))) + response.code should equal(403) + message(response) should include(PersonalFilesNotEnabled) + } + + scenario("The owner's email address must be validated", PersonalFiles, VersionOfApi) { + defineEntities() + val recordId = personalRecord(user1) + enable() + setValidated(resourceUser1, validated = false) + val response = try makePostRequest(myFiles.POST <@ (user1), body(recordId)) finally setValidated(resourceUser1, validated = true) + response.code should equal(403) + message(response) should include(PersonalFileEmailNotValidated) + } + + scenario("A User stores a pending, private, personal file on their own record, without a Role", PersonalFiles, VersionOfApi) { + defineEntities() + enable() + val recordId = personalRecord(user1) + val stored = uploaded(recordId) + stored.is_personal shouldBe true + stored.is_public shouldBe false + stored.scan_status should equal("pending") + stored.owner_user_id should equal(resourceUser1.userId) + + And("it is attached to the record") + val attachments = makeGetRequest((file(stored.file_id) / "attachments").GET <@ (user1)) + attachments.body.extract[FileAttachmentsJsonV700].attachments.map(a => (a.record_type, a.record_id)) should + equal(List((personalEntity, recordId))) + } + + scenario("Only the owner's own personal records take a personal file", PersonalFiles, VersionOfApi) { + defineEntities() + enable() + val someoneElses = personalRecord(user2) + val notMine = makePostRequest(myFiles.POST <@ (user1), body(someoneElses)) + notMine.code should equal(403) + message(notMine) should include(PersonalFileRecordNotOwned) + val notPersonal = makePostRequest(myFiles.POST <@ (user1), body("any-id", recordType = sharedEntity)) + notPersonal.code should equal(403) + + And("the owner cannot attach it to another record later, but a holder of the Role can") + val stored = uploaded(personalRecord(user1)) + val attach = write(PostFileAttachmentJsonV700(personalEntity, someoneElses)) + makePostRequest((file(stored.file_id) / "attachments").POST <@ (user1), attach).code should equal(403) + withEntitlements((bankId, resourceUser1.userId, CanCreateFileAtOneBank.toString)) { + makePostRequest((file(stored.file_id) / "attachments").POST <@ (user1), attach).code should equal(201) + } + } + + scenario("The personal media types and quota are enforced", PersonalFiles, VersionOfApi) { + defineEntities() + enable("files.personal.max_files_per_user" -> "1") + val recordId = personalRecord(user1) + val webp = makePostRequest(myFiles.POST <@ (user1), body(recordId, mediaType = "image/webp", bytes = webpBytes)) + webp.code should equal(400) + message(webp) should include(FileMediaTypeNotAllowed) + // user3 has no personal files yet in this suite, so the quota of one is theirs to use. + val record3 = personalRecord(user3) + makePostRequest(myFiles.POST <@ (user3), body(record3)).code should equal(201) + val second = makePostRequest(myFiles.POST <@ (user3), body(record3)) + second.code should equal(400) + message(second) should include(PersonalFileQuotaExceeded) + } + } + + feature(s"Scan status - PUT /banks/BANK_ID/files/FILE_ID/scan-status - $VersionOfApi") { + + scenario("A pending file's content is served to its owner only, until it is clean", PersonalFiles, VersionOfApi) { + defineEntities() + enable() + val stored = uploaded(personalRecord(user1)) + val content = file(stored.file_id) / "content" + contentStatus(content <@ (user1)) should equal(200) + withEntitlements((bankId, resourceUser2.userId, CanGetFileAtOneBank.toString)) { + val pending = makeGetRequest(content.GET <@ (user2)) + pending.code should equal(403) + message(pending) should include(FileNotYetScanned) + + When("the scanner marks it clean") + val cleaned = setStatus(stored.file_id, "clean") + withClue(cleaned.body) { cleaned.code should equal(200) } + cleaned.body.extract[FileJsonV700].scan_status should equal("clean") + Then("staff who may read files get its content") + contentStatus(content <@ (user2)) should equal(200) + } + } + + scenario("A rejected file's content is served to nobody, not even its owner", PersonalFiles, VersionOfApi) { + defineEntities() + enable() + val stored = uploaded(personalRecord(user1)) + setStatus(stored.file_id, "rejected").code should equal(200) + val response = makeGetRequest((file(stored.file_id) / "content").GET <@ (user1)) + response.code should equal(403) + message(response) should include(FileScanRejected) + } + + scenario("Setting the status needs the Role and a valid status", PersonalFiles, VersionOfApi) { + defineEntities() + enable() + val stored = uploaded(personalRecord(user1)) + makePutRequest((file(stored.file_id) / "scan-status").PUT <@ (user2), write(PutFileScanStatusJsonV700("clean", None))) + .code should equal(403) + val invalid = setStatus(stored.file_id, "infected") + invalid.code should equal(400) + message(invalid) should include(InvalidFileScanStatus) + } + + scenario("A scanner lists the pending files of a space, and only those without CanGetFileAtOneBank", PersonalFiles, VersionOfApi) { + defineEntities() + enable() + val stored = uploaded(personalRecord(user1)) + val space = v7 / "management" / "banks" / bankId / "files" + withEntitlements((bankId, resourceUser2.userId, CanUpdateFileScanStatusAtOneBank.toString)) { + val pending = makeGetRequest((space < "pending")).GET <@ (user2)) + withClue(pending.body) { pending.code should equal(200) } + pending.body.extract[FilesPageJsonV700].files.map(_.file_id) should contain(stored.file_id) + pending.body.extract[FilesPageJsonV700].files.map(_.scan_status).distinct should equal(List("pending")) + makeGetRequest(space.GET <@ (user2)).code should equal(403) + And("the scanner may read a pending file's content to scan it") + contentStatus(file(stored.file_id) / "content" <@ (user2)) should equal(200) + } + } + } +} diff --git a/obp-api/src/test/scala/code/api/v7_0_0/RetentionTest.scala b/obp-api/src/test/scala/code/api/v7_0_0/RetentionTest.scala new file mode 100644 index 0000000000..dc8528a536 --- /dev/null +++ b/obp-api/src/test/scala/code/api/v7_0_0/RetentionTest.scala @@ -0,0 +1,832 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ +package code.api.v7_0_0 + +import java.util.{Base64, Date} + +import code.DynamicData.{DynamicData, DynamicDataAccess, DynamicDataProvider} +import code.api.Constant.DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID +import code.api.util.APIUtil.OAuth._ +import code.api.util.APIUtil.generateUUID +import code.api.util.ApiRole +import code.api.util.ApiRole._ +import code.api.util.ErrorMessages._ +import code.api.v6_0_0.V600ServerSetup +import code.chat.{ChatMessage, ChatRoom, Reaction} +import code.customer.MappedCustomer +import code.customerattribute.MappedCustomerAttribute +import code.dynamicEntity.{DynamicEntityCommons, DynamicEntityProvider} +import code.entitlement.Entitlement +import code.files.{File, FileAttachment, FileContent, MappedFilesProvider} +import code.retention.{RetentionArchive, RetentionPolicy, RetentionProps, RetentionRun} +import code.scheduler.{JobScheduler, RetentionScheduler} +import code.usercustomerlinks.MappedUserCustomerLink +import com.openbankproject.commons.model.ErrorMessage +import com.openbankproject.commons.util.ApiVersion +import net.liftweb.mapper.By +import org.json4s.JsonAST._ +import org.json4s.JsonDSL._ +import org.json4s.native.JsonMethods.{compact, render} +import org.scalatest.Tag + +/** + * This suite checks the v7.0.0 data retention endpoints: the Roles at SYS and at a bank, retention policy + * create, read, update and delete, the preview counts, the props that switch archiving and deleting off, + * approval by a second user, legal holds, archiving and deleting dynamic entity records with the files + * attached to them, the retention archive, the disposal register, anonymising a customer and deleting + * chat messages. + */ +class RetentionTest extends V600ServerSetup { + + object VersionOfApi extends Tag(ApiVersion.v7_0_0.toString) + object CreatePolicy extends Tag("createRetentionPolicy") + object GetPolicies extends Tag("getRetentionPolicies") + object GetPolicy extends Tag("getRetentionPolicy") + object UpdatePolicy extends Tag("updateRetentionPolicy") + object DeletePolicy extends Tag("deleteRetentionPolicy") + object Preview extends Tag("getRetentionPolicyPreview") + object CreateRun extends Tag("createRetentionRun") + object GetRuns extends Tag("getRetentionRuns") + object GetRun extends Tag("getRetentionRun") + object Approval extends Tag("updateRetentionRunApproval") + object RunItems extends Tag("getRetentionRunItems") + object CreateHold extends Tag("createRetentionHold") + object GetHolds extends Tag("getRetentionHolds") + object DeleteHold extends Tag("deleteRetentionHold") + object GetArchive extends Tag("getRetentionArchiveEntries") + object GetArchiveEntry extends Tag("getRetentionArchiveEntry") + + private val SYS = DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID + private def bankId = testBankId1.value + + private def v7 = baseRequest / "obp" / "v7.0.0" + private def space(bankId: String) = v7 / "management" / "banks" / bankId + private def policies(bankId: String) = space(bankId) / "retention-policies" + private def runs(bankId: String) = space(bankId) / "retention-runs" + private def holds(bankId: String) = space(bankId) / "retention-holds" + private def archive(bankId: String) = space(bankId) / "retention-archive" + + private def message(response: code.setup.APIResponse) = response.body.extract[ErrorMessage].message + private def str(json: JValue, field: String): String = (json \ field).extract[String] + private def int(json: JValue, field: String): Int = (json \ field).extract[Int] + + private val allRoles: List[ApiRole] = List(canManageRetentionPolicy, canGetRetentionPolicies, canRunRetentionPolicy, + canApproveRetentionRun, canGetRetentionRuns, canManageRetentionHold, canGetRetentionArchive) + + private def grant(bankId: String, userId: String, roles: ApiRole*): Unit = + roles.foreach(role => Entitlement.entitlement.vend.addEntitlement(bankId, userId, role.toString)) + + /** Most scenarios use short periods; the minimum retention period has scenarios of its own. */ + override def beforeEach(): Unit = { + super.beforeEach() + setPropsValues("retention.minimum_retain_days" -> "0") + } + + private def minimumRetainDays(days: Int): Unit = { + setPropsValues("retention.minimum_retain_days" -> days.toString) + } + + private def yearsAgo(years: Int): String = java.time.LocalDate.now(java.time.ZoneOffset.UTC).minusYears(years).toString + + private def allowArchiveAndDelete(archive: Boolean, delete: Boolean): Unit = { + setPropsValues("retention.archive.enabled" -> archive.toString, "retention.delete.enabled" -> delete.toString) + } + + /** A Dynamic Entity in the system space with a status and a date field. */ + private def entity(entityName: String): Unit = + DynamicEntityProvider.connectorMethodProvider.vend.createOrUpdate(DynamicEntityCommons( + entityName, + s"""{"$entityName":{"properties":{ + |"status":{"type":"string","example":"OPEN"}, + |"closed_on":{"type":"string","example":"2020-01-31"}}}}""".stripMargin, + None, "retention-test", None, hasPersonalEntity = false)).openOrThrowException("definition") + + /** A record of that entity in the system space; its id. */ + private def record(entityName: String, status: String, closedOn: String = "2020-01-31"): String = { + val id = generateUUID() + DynamicDataProvider.connectorMethodProvider.vend + .save(None, entityName, (s"${entityName}_id" -> id) ~ ("status" -> status) ~ ("closed_on" -> closedOn), None, false) + .openOrThrowException("record").dynamicDataId.get + } + + private def recordExists(entityName: String, id: String): Boolean = + DynamicData.count(By(DynamicData.BankId, SYS), By(DynamicData.DynamicEntityName, entityName), By(DynamicData.DynamicDataId, id)) > 0 + + private def policyBody(resourceType: String, action: String, entityName: Option[String] = None, retainDays: Int = 0, + archiveDays: Option[Int] = None, condition: Option[(String, String)] = None, + dateBasis: Option[String] = None, cascade: Boolean = false, requiresApproval: Boolean = false, + enabled: Boolean = true): String = + compact(render( + ("name" -> s"$resourceType $action") ~ + ("resource_type" -> resourceType) ~ + ("entity_name" -> entityName) ~ + ("date_basis" -> dateBasis) ~ + ("condition" -> condition.map { case (f, v) => ("field" -> f) ~ ("value" -> v) }) ~ + ("retain_days" -> retainDays) ~ + ("archive_days" -> archiveDays) ~ + ("action" -> action) ~ + ("cascade" -> cascade) ~ + ("requires_approval" -> requiresApproval) ~ + ("enabled" -> enabled))) + + /** Create a policy as user1, who holds every Role at the space; its id. */ + private def createPolicy(bankId: String, body: String): String = { + val created = makePostRequest(policies(bankId).POST <@ (user1), body) + withClue(created.body) { created.code should equal(201) } + str(created.body, "retention_policy_id") + } + + private def runNow(bankId: String, policyId: String, mode: String = "execute") = + makePostRequest((policies(bankId) / policyId / "runs").POST <@ (user1), compact(render("mode" -> mode))) + + private def runsOf(policyId: String): List[RetentionRun] = + RetentionRun.findAll(By(RetentionRun.RetentionPolicyId, policyId)) + + /** The scheduler runs every enabled policy in the database: switch off any left by other scenarios, and clear its lock. */ + private def schedulerSeesOnlyNewPolicies(): Unit = { + RetentionPolicy.findAll(By(RetentionPolicy.Enabled, true)).foreach(_.Enabled(false).saveMe()) + JobScheduler.findAll(By(JobScheduler.Name, "RetentionScheduler")).foreach(_.delete_!) + } + + private def items(bankId: String, runId: String): List[JValue] = + makeGetRequest((runs(bankId) / runId / "items").GET <@ (user1)).body \ "retention_run_items" match { + case JArray(values) => values + case _ => Nil + } + + feature("Retention policies") { + + scenario("the Roles are required, at the space named in the URL", CreatePolicy, GetPolicies, Preview, CreateRun, VersionOfApi) { + val body = policyBody("chat_message", "delete") + + When("an anonymous caller creates a policy") + makePostRequest(policies(SYS).POST, body).code should equal(401) + + When("user1 creates one without the Role") + val refused = makePostRequest(policies(SYS).POST <@ (user1), body) + refused.code should equal(403) + message(refused) should include(UserHasMissingRoles) + message(refused) should include("CanManageRetentionPolicy") + + When("user1 holds the Role at a bank but not at SYS") + grant(bankId, resourceUser1.userId, canManageRetentionPolicy) + makePostRequest(policies(SYS).POST <@ (user1), body).code should equal(403) + + Then("the Role at the bank is enough at that bank") + makePostRequest(policies(bankId).POST <@ (user1), body).code should equal(201) + + And("reading, previewing and running need their own Roles") + makeGetRequest(policies(SYS).GET <@ (user2)).code should equal(403) + makeGetRequest((runs(SYS)).GET <@ (user2)).code should equal(403) + makeGetRequest((holds(SYS)).GET <@ (user2)).code should equal(403) + makeGetRequest((archive(SYS)).GET <@ (user2)).code should equal(403) + } + + scenario("create, read, list, update and delete a policy at SYS, and refuse invalid ones", + CreatePolicy, GetPolicies, GetPolicy, UpdatePolicy, DeletePolicy, VersionOfApi) { + grant(SYS, resourceUser1.userId, allRoles: _*) + val entityName = s"application_${generateUUID().take(8).replace("-", "")}" + entity(entityName) + + When("a policy is created") + val created = makePostRequest(policies(SYS).POST <@ (user1), policyBody("dynamic_entity_record", "archive", + Some(entityName), retainDays = 730, archiveDays = Some(1825), condition = Some("status" -> "CLOSED"), + dateBasis = Some("updated"), cascade = true, requiresApproval = true)) + withClue(created.body) { created.code should equal(201) } + val policyId = str(created.body, "retention_policy_id") + str(created.body, "bank_id") should equal(SYS) + str(created.body, "resource_type") should equal("dynamic_entity_record") + str(created.body, "entity_name") should equal(entityName) + str(created.body, "date_basis") should equal("updated") + (created.body \ "condition") should equal(("field" -> "status") ~ ("value" -> "CLOSED")) + int(created.body, "retain_days") should equal(730) + int(created.body, "archive_days") should equal(1825) + str(created.body, "action") should equal("archive") + (created.body \ "cascade").extract[Boolean] should equal(true) + (created.body \ "requires_approval").extract[Boolean] should equal(true) + (created.body \ "enabled").extract[Boolean] should equal(true) + str(created.body, "created_by_user_id") should equal(resourceUser1.userId) + + Then("it is read and listed") + makeGetRequest((policies(SYS) / policyId).GET <@ (user1)).code should equal(200) + val listed = makeGetRequest(policies(SYS).GET <@ (user1)) + listed.code should equal(200) + (listed.body \ "retention_policies").children.map(p => str(p, "retention_policy_id")) should equal(List(policyId)) + And("it is not found in another space") + grant(bankId, resourceUser1.userId, allRoles: _*) + makeGetRequest((policies(bankId) / policyId).GET <@ (user1)).code should equal(404) + + When("it is updated to delete, without an archive stage, and the condition is removed") + val updated = makePutRequest((policies(SYS) / policyId).PUT <@ (user1), + compact(render(("action" -> "delete") ~ ("retain_days" -> 365) ~ ("condition" -> (("field" -> "") ~ ("value" -> "")))))) + withClue(updated.body) { updated.code should equal(200) } + str(updated.body, "action") should equal("delete") + int(updated.body, "retain_days") should equal(365) + (updated.body \ "archive_days") should equal(JNothing) + (updated.body \ "condition") should equal(JNothing) + + Then("invalid policies are refused") + val noArchiveDays = makePostRequest(policies(SYS).POST <@ (user1), policyBody("chat_message", "archive")) + noArchiveDays.code should equal(400) + message(noArchiveDays) should include(InvalidRetentionPolicy) + makePostRequest(policies(SYS).POST <@ (user1), policyBody("chat_message", "anonymise")).code should equal(400) + makePostRequest(policies(SYS).POST <@ (user1), policyBody("unknown", "delete")).code should equal(400) + makePostRequest(policies(SYS).POST <@ (user1), policyBody("dynamic_entity_record", "delete", Some("no_such_entity"))).code should equal(400) + makePostRequest(policies(SYS).POST <@ (user1), policyBody("customer", "anonymise")).code should equal(400) + + When("it is deleted") + makeDeleteRequest((policies(SYS) / policyId).DELETE <@ (user1)).code should equal(204) + makeGetRequest((policies(SYS) / policyId).GET <@ (user1)).code should equal(404) + } + + scenario("the preview counts what is due, what is held, and changes nothing", Preview, CreateHold, VersionOfApi) { + grant(SYS, resourceUser1.userId, allRoles: _*) + allowArchiveAndDelete(archive = false, delete = false) + val entityName = s"application_${generateUUID().take(8).replace("-", "")}" + entity(entityName) + val closed1 = record(entityName, "CLOSED", "2020-01-31") + val closed2 = record(entityName, "CLOSED", "2021-06-30") + val open = record(entityName, "OPEN", "2020-01-31") + val recent = record(entityName, "CLOSED", "2999-01-01") + + val policyId = createPolicy(SYS, policyBody("dynamic_entity_record", "delete", Some(entityName), + retainDays = 365, condition = Some("status" -> "CLOSED"), dateBasis = Some("field:closed_on"))) + + When("a record is put on hold") + val hold = makePostRequest(holds(SYS).POST <@ (user1), compact(render( + ("resource_type" -> "dynamic_entity_record") ~ ("resource_id" -> closed2) ~ ("entity_name" -> entityName) ~ ("reason" -> "Open complaint.")))) + withClue(hold.body) { hold.code should equal(201) } + + val preview = makeGetRequest((policies(SYS) / policyId / "preview").GET <@ (user1)) + withClue(preview.body) { preview.code should equal(200) } + Then("the two closed records past their date match, one of them on hold") + int(preview.body, "matched_count") should equal(2) + int(preview.body, "held_count") should equal(1) + int(preview.body, "due_count") should equal(1) + (preview.body \ "sample_resource_ids").extract[List[String]] should equal(List(closed1)) + str(preview.body, "oldest_trigger_date") should startWith("2020-01-31") + (preview.body \ "can_execute").extract[Boolean] should equal(false) + + And("a preview run records the counts and changes nothing") + val previewRun = runNow(SYS, policyId, "preview") + withClue(previewRun.body) { previewRun.code should equal(201) } + str(previewRun.body, "status") should equal("done") + int(previewRun.body, "matched_count") should equal(2) + List(closed1, closed2, open, recent).foreach(id => recordExists(entityName, id) should equal(true)) + + And("a policy with a longer period matches nothing yet") + val laterId = createPolicy(SYS, policyBody("dynamic_entity_record", "delete", Some(entityName), retainDays = 36500)) + int(makeGetRequest((policies(SYS) / laterId / "preview").GET <@ (user1)).body, "matched_count") should equal(0) + } + + scenario("a run is refused while the props switch archiving or deleting off", CreateRun, GetRun, VersionOfApi) { + grant(SYS, resourceUser1.userId, allRoles: _*) + allowArchiveAndDelete(archive = false, delete = false) + val entityName = s"application_${generateUUID().take(8).replace("-", "")}" + entity(entityName) + val id = record(entityName, "CLOSED") + + val archivePolicy = createPolicy(SYS, policyBody("dynamic_entity_record", "archive", Some(entityName), archiveDays = Some(30))) + val refusedArchive = runNow(SYS, archivePolicy) + refusedArchive.code should equal(400) + message(refusedArchive) should include(RetentionArchiveIsDisabled) + + val deletePolicy = createPolicy(SYS, policyBody("dynamic_entity_record", "delete", Some(entityName))) + val refusedDelete = runNow(SYS, deletePolicy) + refusedDelete.code should equal(400) + message(refusedDelete) should include(RetentionDeleteIsDisabled) + + Then("nothing was disposed of, and the refused runs are recorded") + recordExists(entityName, id) should equal(true) + val listed = makeGetRequest(runs(SYS).GET <@ (user1) < "refused")) + listed.code should equal(200) + (listed.body \ "retention_runs").children.size should equal(2) + + And("a preview still works") + makeGetRequest((policies(SYS) / deletePolicy / "preview").GET <@ (user1)).code should equal(200) + } + + scenario("a run that requires approval waits for a different user", CreateRun, Approval, GetRun, VersionOfApi) { + grant(SYS, resourceUser1.userId, allRoles: _*) + allowArchiveAndDelete(archive = false, delete = true) + val entityName = s"application_${generateUUID().take(8).replace("-", "")}" + entity(entityName) + val id = record(entityName, "CLOSED") + val policyId = createPolicy(SYS, policyBody("dynamic_entity_record", "delete", Some(entityName), requiresApproval = true)) + + When("user1 runs it") + val pending = runNow(SYS, policyId) + withClue(pending.body) { pending.code should equal(201) } + str(pending.body, "status") should equal("pending") + str(pending.body, "requested_by_user_id") should equal(resourceUser1.userId) + val runId = str(pending.body, "retention_run_id") + recordExists(entityName, id) should equal(true) + + val approve = compact(render(("decision" -> "approve") ~ ("comment" -> "Checked."))) + Then("user1 may not approve their own run") + val own = makePutRequest((runs(SYS) / runId / "approval").PUT <@ (user1), approve) + own.code should equal(403) + message(own) should include(RetentionRunApproverIsRequester) + + And("user2 needs the Role to approve it") + makePutRequest((runs(SYS) / runId / "approval").PUT <@ (user2), approve).code should equal(403) + grant(SYS, resourceUser2.userId, canApproveRetentionRun) + val approved = makePutRequest((runs(SYS) / runId / "approval").PUT <@ (user2), approve) + withClue(approved.body) { approved.code should equal(200) } + str(approved.body, "status") should equal("done") + str(approved.body, "approved_by_user_id") should equal(resourceUser2.userId) + str(approved.body, "approval_decision") should equal("approved") + int(approved.body, "deleted_count") should equal(1) + recordExists(entityName, id) should equal(false) + + And("a run that is no longer pending cannot be decided again") + makePutRequest((runs(SYS) / runId / "approval").PUT <@ (user2), approve).code should equal(400) + + And("a rejected run disposes of nothing") + val other = record(entityName, "CLOSED") + val second = str(runNow(SYS, policyId).body, "retention_run_id") + val rejected = makePutRequest((runs(SYS) / second / "approval").PUT <@ (user2), compact(render("decision" -> "reject"))) + rejected.code should equal(200) + str(rejected.body, "status") should equal("refused") + str(rejected.body, "approval_decision") should equal("rejected") + recordExists(entityName, other) should equal(true) + } + + scenario("a record on hold is skipped until the hold is released", CreateHold, GetHolds, DeleteHold, CreateRun, VersionOfApi) { + grant(SYS, resourceUser1.userId, allRoles: _*) + allowArchiveAndDelete(archive = false, delete = true) + val entityName = s"application_${generateUUID().take(8).replace("-", "")}" + entity(entityName) + val held = record(entityName, "CLOSED") + val free = record(entityName, "CLOSED") + val policyId = createPolicy(SYS, policyBody("dynamic_entity_record", "delete", Some(entityName))) + + When("every record of the entity is on hold") + val hold = makePostRequest(holds(SYS).POST <@ (user1), compact(render( + ("resource_type" -> "dynamic_entity_record") ~ ("entity_name" -> entityName) ~ ("reason" -> "Audit in progress.")))) + withClue(hold.body) { hold.code should equal(201) } + val holdId = str(hold.body, "retention_hold_id") + val first = runNow(SYS, policyId) + int(first.body, "held_count") should equal(2) + int(first.body, "deleted_count") should equal(0) + recordExists(entityName, held) should equal(true) + recordExists(entityName, free) should equal(true) + + When("that hold is released and one record is held by id") + makeDeleteRequest((holds(SYS) / holdId).DELETE <@ (user1)).code should equal(204) + makeDeleteRequest((holds(SYS) / holdId).DELETE <@ (user1)).code should equal(400) + makePostRequest(holds(SYS).POST <@ (user1), compact(render( + ("resource_type" -> "dynamic_entity_record") ~ ("resource_id" -> held) ~ ("reason" -> "Open complaint.")))).code should equal(201) + val active = makeGetRequest(holds(SYS).GET <@ (user1) < "true")) + (active.body \ "retention_holds").children.size should equal(1) + val released = makeGetRequest(holds(SYS).GET <@ (user1) < "false")) + (released.body \ "retention_holds").children.map(h => str(h, "released_by_user_id")) should equal(List(resourceUser1.userId)) + + Then("only the record still on hold stays") + val second = runNow(SYS, policyId) + int(second.body, "held_count") should equal(1) + int(second.body, "deleted_count") should equal(1) + recordExists(entityName, held) should equal(true) + recordExists(entityName, free) should equal(false) + } + } + + feature("Disposal") { + + scenario("archive a dynamic entity record with its file, read the archive, then delete it from the archive", + CreateRun, RunItems, GetArchive, GetArchiveEntry, VersionOfApi) { + grant(SYS, resourceUser1.userId, allRoles: _*) + allowArchiveAndDelete(archive = true, delete = false) + val entityName = s"application_${generateUUID().take(8).replace("-", "")}" + entity(entityName) + val id = record(entityName, "CLOSED") + DynamicDataAccess.create.BankId(SYS).EntityName(entityName).DynamicDataId(id).UserId(resourceUser2.userId) + .CanRead(true).GrantedBy(resourceUser1.userId).saveMe() + // One byte: the Files table keeps its bytes in a MappedBinary, which Lift maps to a one-byte BINARY on H2. + val bytes = Array[Byte](42) + val file = MappedFilesProvider.createFile(SYS, "0" * 64, "application/pdf", "evidence.pdf", false, resourceUser1.userId, bytes) + .openOrThrowException("file") + MappedFilesProvider.createAttachment(file, entityName, id, resourceUser1.userId).openOrThrowException("attachment") + + val policyId = createPolicy(SYS, policyBody("dynamic_entity_record", "archive", Some(entityName), + archiveDays = Some(0), cascade = true)) + val run = runNow(SYS, policyId) + withClue(run.body) { run.code should equal(201) } + str(run.body, "status") should equal("done") + int(run.body, "matched_count") should equal(1) + int(run.body, "archived_count") should equal(2) + val runId = str(run.body, "retention_run_id") + + Then("the record, its access list, the file and its bytes have left the live tables") + recordExists(entityName, id) should equal(false) + DynamicDataAccess.count(By(DynamicDataAccess.DynamicDataId, id)) should equal(0) + File.count(By(File.FileId, file.fileId)) should equal(0) + FileContent.count(By(FileContent.FileId, file.fileId)) should equal(0) + FileAttachment.count(By(FileAttachment.FileId, file.fileId)) should equal(0) + + And("the disposal register names both, the file under its record") + val register = items(SYS, runId) + register.map(i => (str(i, "resource_type"), str(i, "action"))).toSet should equal( + Set(("dynamic_entity_record", "archive"), ("file", "archive"))) + val fileItem = register.find(i => str(i, "resource_type") == "file").get + str(fileItem, "resource_id") should equal(file.fileId) + str(fileItem, "parent_resource_id") should equal(id) + + And("the archive holds both, and gives the copy and the bytes back") + val listed = makeGetRequest(archive(SYS).GET <@ (user1)) + listed.code should equal(200) + val entries = (listed.body \ "retention_archive_entries").children + entries.size should equal(2) + entries.foreach(e => (e \ "data") should equal(JNothing)) + val recordEntry = entries.find(e => str(e, "resource_type") == "dynamic_entity_record").get + val one = makeGetRequest((archive(SYS) / str(recordEntry, "retention_archive_id")).GET <@ (user1)) + one.code should equal(200) + str(one.body \ "data" \ "data", "status") should equal("CLOSED") + str(one.body, "resource_id") should equal(id) + val fileEntry = entries.find(e => str(e, "resource_type") == "file").get + val withContent = makeGetRequest((archive(SYS) / str(fileEntry, "retention_archive_id")).GET <@ (user1) < "true")) + Base64.getDecoder.decode(str(withContent.body, "content_base64")) should equal(bytes) + + When("deleting is switched on, the next run deletes what reached the end of its archive period") + allowArchiveAndDelete(archive = true, delete = true) + val purge = runNow(SYS, policyId) + withClue(purge.body) { purge.code should equal(201) } + int(purge.body, "deleted_count") should equal(2) + RetentionArchive.count(By(RetentionArchive.BankId, SYS)) should equal(0) + items(SYS, str(purge.body, "retention_run_id")).map(i => str(i, "action")).distinct should equal(List("delete_from_archive")) + } + + scenario("delete a dynamic entity record and cascade to its file; without cascade the file stays", CreateRun, VersionOfApi) { + grant(SYS, resourceUser1.userId, allRoles: _*) + allowArchiveAndDelete(archive = false, delete = true) + val entityName = s"application_${generateUUID().take(8).replace("-", "")}" + entity(entityName) + val id = record(entityName, "CLOSED") + val file = MappedFilesProvider.createFile(SYS, "1" * 64, "image/png", "photo.png", false, resourceUser1.userId, Array[Byte](1)) + .openOrThrowException("file") + MappedFilesProvider.createAttachment(file, entityName, id, resourceUser1.userId).openOrThrowException("attachment") + + val other = s"other_${generateUUID().take(8).replace("-", "")}" + entity(other) + val otherId = record(other, "CLOSED") + val kept = MappedFilesProvider.createFile(SYS, "2" * 64, "image/png", "kept.png", false, resourceUser1.userId, Array[Byte](4)) + .openOrThrowException("file") + MappedFilesProvider.createAttachment(kept, other, otherId, resourceUser1.userId).openOrThrowException("attachment") + + val cascading = createPolicy(SYS, policyBody("dynamic_entity_record", "delete", Some(entityName), cascade = true)) + val run = runNow(SYS, cascading) + int(run.body, "deleted_count") should equal(2) + recordExists(entityName, id) should equal(false) + File.count(By(File.FileId, file.fileId)) should equal(0) + RetentionArchive.count() should equal(0) + + val notCascading = createPolicy(SYS, policyBody("dynamic_entity_record", "delete", Some(other))) + int(runNow(SYS, notCascading).body, "deleted_count") should equal(1) + recordExists(other, otherId) should equal(false) + File.count(By(File.FileId, kept.fileId)) should equal(1) + FileAttachment.count(By(FileAttachment.FileId, kept.fileId)) should equal(0) + } + + scenario("anonymise a customer: personal fields cleared, links removed, the customer id kept", CreateRun, RunItems, VersionOfApi) { + grant(bankId, resourceUser1.userId, allRoles: _*) + allowArchiveAndDelete(archive = false, delete = true) + val customerId = generateUUID() + MappedCustomer.create.mCustomerId(customerId).mBank(bankId).mNumber(customerId.take(10)) + .mLegalName("Example Person").mEmail("person@example.com").mMobileNumber("+10000000000") + .mDateOfBirth(new Date(0L)).mTitle("Dr").saveMe() + MappedCustomerAttribute.create.mBankId(bankId).mCustomerId(customerId).mCustomerAttributeId(generateUUID()) + .mName("nickname").mType("STRING").mValue("Example").saveMe() + MappedUserCustomerLink.create.mUserCustomerLinkId(generateUUID()).mCustomerId(customerId) + .mUserId(resourceUser2.userId).mDateInserted(new Date()).mIsActive(true).saveMe() + + val policyId = createPolicy(bankId, policyBody("customer", "anonymise")) + val run = runNow(bankId, policyId) + withClue(run.body) { run.code should equal(201) } + int(run.body, "anonymised_count") should equal(1) + + val customer = MappedCustomer.find(By(MappedCustomer.mCustomerId, customerId)).openOrThrowException("customer") + customer.legalName should equal("ANONYMISED") + customer.email should equal("") + customer.mobileNumber should equal("") + customer.title should equal("") + customer.mNumber.get should equal(customerId.take(10)) + MappedCustomerAttribute.count(By(MappedCustomerAttribute.mCustomerId, customerId)) should equal(0) + MappedUserCustomerLink.count(By(MappedUserCustomerLink.mCustomerId, customerId)) should equal(0) + + val register = items(bankId, str(run.body, "retention_run_id")) + register.map(i => (str(i, "resource_id"), str(i, "action"), str(i, "owner_user_id"))) should equal( + List((customerId, "anonymise", resourceUser2.userId))) + + And("an anonymised customer is not anonymised again") + int(runNow(bankId, policyId).body, "matched_count") should equal(0) + } + + scenario("delete the chat messages of a bank, keeping the room", CreateRun, GetRuns, VersionOfApi) { + grant(bankId, resourceUser1.userId, allRoles: _*) + allowArchiveAndDelete(archive = false, delete = true) + val roomId = generateUUID() + ChatRoom.create.ChatRoomId(roomId).BankId(bankId).Name("General").CreatedByUserId(resourceUser1.userId) + .LastMessagePreview("second message").LastMessageAt(new Date()).saveMe() + val messageIds = List("first message", "second message").map { content => + val messageId = generateUUID() + ChatMessage.create.ChatMessageId(messageId).ChatRoomId(roomId).SenderUserId(resourceUser1.userId) + .Content(content).MessageType("text").saveMe() + messageId + } + Reaction.create.ReactionId(generateUUID()).ChatMessageId(messageIds.head).UserId(resourceUser2.userId).Emoji("+1").saveMe() + val otherBankRoom = generateUUID() + ChatRoom.create.ChatRoomId(otherBankRoom).BankId("another-bank").Name("Elsewhere").saveMe() + ChatMessage.create.ChatMessageId(generateUUID()).ChatRoomId(otherBankRoom).Content("elsewhere").MessageType("text").saveMe() + + val policyId = createPolicy(bankId, policyBody("chat_message", "delete")) + val run = runNow(bankId, policyId) + withClue(run.body) { run.code should equal(201) } + int(run.body, "deleted_count") should equal(2) + + ChatMessage.count(By(ChatMessage.ChatRoomId, roomId)) should equal(0) + Reaction.count(By(Reaction.ChatMessageId, messageIds.head)) should equal(0) + val room = ChatRoom.find(By(ChatRoom.ChatRoomId, roomId)).openOrThrowException("room") + room.lastMessagePreview should equal("") + ChatMessage.count(By(ChatMessage.ChatRoomId, otherBankRoom)) should equal(1) + items(bankId, str(run.body, "retention_run_id")).map(i => str(i, "resource_id")).toSet should equal(messageIds.toSet) + + val listed = makeGetRequest(runs(bankId).GET <@ (user1) < policyId)) + (listed.body \ "retention_runs").children.map(r => str(r, "status")) should equal(List("done")) + RetentionRun.count(By(RetentionRun.BankId, SYS)) should equal(0) + } + } + + feature("Retention runs only when it is switched on") { + + scenario("archiving, deleting and the scheduler are off by default", Preview, CreateRun, VersionOfApi) { + grant(SYS, resourceUser1.userId, allRoles: _*) + RetentionProps.archiveEnabled should equal(false) + RetentionProps.deleteEnabled should equal(false) + RetentionProps.schedulerEnabled should equal(false) + + Then("the scheduler is not started") + RetentionScheduler.startIfEnabled() should equal(false) + + And("a run is refused and the preview says it cannot execute") + val entityName = s"application_${generateUUID().take(8).replace("-", "")}" + entity(entityName) + val id = record(entityName, "CLOSED") + val policyId = createPolicy(SYS, policyBody("dynamic_entity_record", "delete", Some(entityName))) + (makeGetRequest((policies(SYS) / policyId / "preview").GET <@ (user1)).body \ "can_execute").extract[Boolean] should equal(false) + runNow(SYS, policyId).code should equal(400) + recordExists(entityName, id) should equal(true) + } + + scenario("each prop switches on only its own actions", Preview, CreateRun, VersionOfApi) { + grant(SYS, resourceUser1.userId, allRoles: _*) + grant(bankId, resourceUser1.userId, allRoles: _*) + val entityName = s"application_${generateUUID().take(8).replace("-", "")}" + entity(entityName) + val archivePolicy = createPolicy(SYS, policyBody("dynamic_entity_record", "archive", Some(entityName), archiveDays = Some(30))) + val deletePolicy = createPolicy(SYS, policyBody("dynamic_entity_record", "delete", Some(entityName))) + val anonymisePolicy = createPolicy(bankId, policyBody("customer", "anonymise")) + def canExecute(space: String, policyId: String) = + (makeGetRequest((policies(space) / policyId / "preview").GET <@ (user1)).body \ "can_execute").extract[Boolean] + + When("only archiving is switched on") + allowArchiveAndDelete(archive = true, delete = false) + canExecute(SYS, archivePolicy) should equal(true) + canExecute(SYS, deletePolicy) should equal(false) + canExecute(bankId, anonymisePolicy) should equal(false) + val kept = record(entityName, "CLOSED") + val refusedDelete = runNow(SYS, deletePolicy) + refusedDelete.code should equal(400) + message(refusedDelete) should include(RetentionDeleteIsDisabled) + val refusedAnonymise = runNow(bankId, anonymisePolicy) + refusedAnonymise.code should equal(400) + message(refusedAnonymise) should include(RetentionDeleteIsDisabled) + recordExists(entityName, kept) should equal(true) + Then("an archive run archives") + val archived = runNow(SYS, archivePolicy) + withClue(archived.body) { archived.code should equal(201) } + int(archived.body, "archived_count") should equal(1) + recordExists(entityName, kept) should equal(false) + + When("only deleting is switched on") + allowArchiveAndDelete(archive = false, delete = true) + canExecute(SYS, archivePolicy) should equal(false) + canExecute(SYS, deletePolicy) should equal(true) + canExecute(bankId, anonymisePolicy) should equal(true) + val other = record(entityName, "CLOSED") + val refusedArchive = runNow(SYS, archivePolicy) + refusedArchive.code should equal(400) + message(refusedArchive) should include(RetentionArchiveIsDisabled) + recordExists(entityName, other) should equal(true) + Then("a delete run deletes") + int(runNow(SYS, deletePolicy).body, "deleted_count") should equal(1) + recordExists(entityName, other) should equal(false) + } + + scenario("an approved run is refused when the props were switched off after it was requested", CreateRun, Approval, VersionOfApi) { + grant(SYS, resourceUser1.userId, allRoles: _*) + grant(SYS, resourceUser2.userId, canApproveRetentionRun) + allowArchiveAndDelete(archive = false, delete = true) + val entityName = s"application_${generateUUID().take(8).replace("-", "")}" + entity(entityName) + val id = record(entityName, "CLOSED") + val policyId = createPolicy(SYS, policyBody("dynamic_entity_record", "delete", Some(entityName), requiresApproval = true)) + val pending = runNow(SYS, policyId) + str(pending.body, "status") should equal("pending") + + When("deleting is switched off before user2 approves") + allowArchiveAndDelete(archive = false, delete = false) + val runId = str(pending.body, "retention_run_id") + val approved = makePutRequest((runs(SYS) / runId / "approval").PUT <@ (user2), compact(render("decision" -> "approve"))) + Then("the approval is refused, the run is recorded as refused, and the record stays") + approved.code should equal(400) + message(approved) should include(RetentionDeleteIsDisabled) + val run = makeGetRequest((runs(SYS) / runId).GET <@ (user1)) + str(run.body, "status") should equal("refused") + str(run.body, "approved_by_user_id") should equal(resourceUser2.userId) + int(run.body, "deleted_count") should equal(0) + recordExists(entityName, id) should equal(true) + } + + scenario("archived resources past their archive period stay in the archive while deleting is off", CreateRun, VersionOfApi) { + grant(SYS, resourceUser1.userId, allRoles: _*) + schedulerSeesOnlyNewPolicies() + allowArchiveAndDelete(archive = true, delete = false) + val entityName = s"application_${generateUUID().take(8).replace("-", "")}" + entity(entityName) + record(entityName, "CLOSED") + val policyId = createPolicy(SYS, policyBody("dynamic_entity_record", "archive", Some(entityName), archiveDays = Some(0))) + int(runNow(SYS, policyId).body, "archived_count") should equal(1) + val archivedCount = RetentionArchive.count(By(RetentionArchive.RetentionPolicyId, policyId)) + archivedCount should equal(1) + + When("the policy runs again, by hand and from the scheduler") + val again = runNow(SYS, policyId) + withClue(again.body) { again.code should equal(201) } + int(again.body, "deleted_count") should equal(0) + str(again.body, "message") should include("retention.delete.enabled is false") + RetentionScheduler.runOnce() should equal(true) + Then("the archive entry is still there") + RetentionArchive.count(By(RetentionArchive.RetentionPolicyId, policyId)) should equal(archivedCount) + } + } + + feature("The minimum retention period") { + + scenario("is five years by default", Preview, VersionOfApi) { + RetentionProps.MinimumRetainDaysDefault should equal(1825) + } + + scenario("a policy shorter than the minimum is refused", CreatePolicy, UpdatePolicy, Preview, VersionOfApi) { + grant(SYS, resourceUser1.userId, allRoles: _*) + minimumRetainDays(1825) + val entityName = s"application_${generateUUID().take(8).replace("-", "")}" + entity(entityName) + + val tooShort = makePostRequest(policies(SYS).POST <@ (user1), policyBody("dynamic_entity_record", "delete", Some(entityName), retainDays = 1824)) + tooShort.code should equal(400) + message(tooShort) should include(InvalidRetentionPolicy) + message(tooShort) should include("retention.minimum_retain_days") + makePostRequest(policies(SYS).POST <@ (user1), policyBody("dynamic_entity_record", "archive", Some(entityName), + retainDays = 365, archiveDays = Some(3650))).code should equal(400) + + val policyId = createPolicy(SYS, policyBody("dynamic_entity_record", "delete", Some(entityName), retainDays = 1825)) + val shortened = makePutRequest((policies(SYS) / policyId).PUT <@ (user1), compact(render("retain_days" -> 30))) + shortened.code should equal(400) + message(shortened) should include(InvalidRetentionPolicy) + + val preview = makeGetRequest((policies(SYS) / policyId / "preview").GET <@ (user1)) + int(preview.body, "minimum_retain_days") should equal(1825) + } + + scenario("a policy from before the minimum was raised disposes of nothing younger than the minimum", + Preview, CreateRun, VersionOfApi) { + grant(SYS, resourceUser1.userId, allRoles: _*) + allowArchiveAndDelete(archive = false, delete = true) + val entityName = s"application_${generateUUID().take(8).replace("-", "")}" + entity(entityName) + val old = record(entityName, "CLOSED", yearsAgo(6)) + val young = record(entityName, "CLOSED", yearsAgo(3)) + val policyId = createPolicy(SYS, policyBody("dynamic_entity_record", "delete", Some(entityName), + retainDays = 365, dateBasis = Some("field:closed_on"))) + int(makeGetRequest((policies(SYS) / policyId / "preview").GET <@ (user1)).body, "matched_count") should equal(2) + + When("the minimum is raised to five years") + minimumRetainDays(1825) + val preview = makeGetRequest((policies(SYS) / policyId / "preview").GET <@ (user1)) + int(preview.body, "matched_count") should equal(1) + (preview.body \ "sample_resource_ids").extract[List[String]] should equal(List(old)) + And("the policy cannot be changed until its retain_days meets the minimum") + makePutRequest((policies(SYS) / policyId).PUT <@ (user1), compact(render("name" -> "renamed"))).code should equal(400) + + Then("a run, by hand or from the scheduler, keeps the record younger than five years") + int(runNow(SYS, policyId).body, "deleted_count") should equal(1) + schedulerSeesOnlyNewPolicies() + RetentionPolicy.findAll(By(RetentionPolicy.RetentionPolicyId, policyId)).foreach(_.Enabled(true).saveMe()) + RetentionScheduler.runOnce() should equal(true) + recordExists(entityName, old) should equal(false) + recordExists(entityName, young) should equal(true) + } + } + + feature("The retention scheduler") { + + scenario("disposes of nothing, and records no runs, while the props are off", CreateRun, VersionOfApi) { + grant(SYS, resourceUser1.userId, allRoles: _*) + schedulerSeesOnlyNewPolicies() + allowArchiveAndDelete(archive = false, delete = false) + val entityName = s"application_${generateUUID().take(8).replace("-", "")}" + entity(entityName) + val id = record(entityName, "CLOSED") + val deletePolicy = createPolicy(SYS, policyBody("dynamic_entity_record", "delete", Some(entityName))) + val archivePolicy = createPolicy(SYS, policyBody("dynamic_entity_record", "archive", Some(entityName), archiveDays = Some(30))) + + RetentionScheduler.runOnce() should equal(true) + recordExists(entityName, id) should equal(true) + runsOf(deletePolicy) should equal(Nil) + runsOf(archivePolicy) should equal(Nil) + } + + scenario("runs enabled policies only, as the system, and leaves disabled ones alone", CreateRun, VersionOfApi) { + grant(SYS, resourceUser1.userId, allRoles: _*) + schedulerSeesOnlyNewPolicies() + allowArchiveAndDelete(archive = false, delete = true) + val enabledEntity = s"application_${generateUUID().take(8).replace("-", "")}" + val disabledEntity = s"other_${generateUUID().take(8).replace("-", "")}" + entity(enabledEntity) + entity(disabledEntity) + val gone = record(enabledEntity, "CLOSED") + val kept = record(disabledEntity, "CLOSED") + val enabledPolicy = createPolicy(SYS, policyBody("dynamic_entity_record", "delete", Some(enabledEntity))) + val disabledPolicy = createPolicy(SYS, policyBody("dynamic_entity_record", "delete", Some(disabledEntity), enabled = false)) + + RetentionScheduler.runOnce() should equal(true) + recordExists(enabledEntity, gone) should equal(false) + recordExists(disabledEntity, kept) should equal(true) + runsOf(enabledPolicy).map(r => (r.status, r.requestedByUserId)) should equal(List(("done", ""))) + runsOf(disabledPolicy) should equal(Nil) + + And("a disabled policy can still be run by hand") + int(runNow(SYS, disabledPolicy).body, "deleted_count") should equal(1) + recordExists(disabledEntity, kept) should equal(false) + } + + scenario("asks once for approval, rather than once per tick, and disposes of nothing until approved", CreateRun, VersionOfApi) { + grant(SYS, resourceUser1.userId, allRoles: _*) + schedulerSeesOnlyNewPolicies() + allowArchiveAndDelete(archive = false, delete = true) + val entityName = s"application_${generateUUID().take(8).replace("-", "")}" + entity(entityName) + val id = record(entityName, "CLOSED") + val policyId = createPolicy(SYS, policyBody("dynamic_entity_record", "delete", Some(entityName), requiresApproval = true)) + + RetentionScheduler.runOnce() should equal(true) + RetentionScheduler.runOnce() should equal(true) + runsOf(policyId).map(_.status) should equal(List("pending")) + recordExists(entityName, id) should equal(true) + } + + scenario("skips a tick while another holds the lock", CreateRun, VersionOfApi) { + grant(SYS, resourceUser1.userId, allRoles: _*) + schedulerSeesOnlyNewPolicies() + allowArchiveAndDelete(archive = false, delete = true) + val entityName = s"application_${generateUUID().take(8).replace("-", "")}" + entity(entityName) + val id = record(entityName, "CLOSED") + val policyId = createPolicy(SYS, policyBody("dynamic_entity_record", "delete", Some(entityName))) + + val lock = JobScheduler.create.JobId(generateUUID()).Name("RetentionScheduler").ApiInstanceId("another-node").saveMe() + RetentionScheduler.runOnce() should equal(false) + recordExists(entityName, id) should equal(true) + runsOf(policyId) should equal(Nil) + + Then("the next tick, with the lock gone, runs") + JobScheduler.delete_!(lock) + RetentionScheduler.runOnce() should equal(true) + recordExists(entityName, id) should equal(false) + } + } +} diff --git a/obp-api/src/test/scala/code/api/v7_0_0/TelemetryEndpointTest.scala b/obp-api/src/test/scala/code/api/v7_0_0/TelemetryEndpointTest.scala index 5fb2cc99b6..9b86d49b81 100644 --- a/obp-api/src/test/scala/code/api/v7_0_0/TelemetryEndpointTest.scala +++ b/obp-api/src/test/scala/code/api/v7_0_0/TelemetryEndpointTest.scala @@ -137,17 +137,22 @@ class TelemetryEndpointTest extends V600ServerSetup { all(names) should startWith("jvm.memory") } - scenario("A Consumer holding CanGetTelemetry as a Scope may read Telemetry without an Entitlement", ApiEndpoint, VersionOfApi) { - Given("user2 holds no CanGetTelemetry Entitlement and testConsumer2 no Scope") - makeGetRequest(telemetryRequest.GET <@ (user2)).code should equal(403) + scenario("A Consumer holding CanGetTelemetry as a Scope may read Telemetry as itself, without an Entitlement", ApiEndpoint, VersionOfApi) { + Given("testConsumer2 holds no CanGetTelemetry Scope") + makeGetRequest(telemetryRequest.GET <@ (application2)).code should equal(403) - When("testConsumer2, which user2 signs with, is granted CanGetTelemetry as a Scope") + When("testConsumer2 is granted CanGetTelemetry as a Scope") val granted = Scope.scope.vend.addScope("", testConsumer2.id.get.toString, CanGetTelemetry.toString) - val response = try makeGetRequest(telemetryRequest.GET <@ (user2)) finally Scope.scope.vend.deleteScope(granted) + val (response, personResponse) = try { + (makeGetRequest(telemetryRequest.GET <@ (application2)), makeGetRequest(telemetryRequest.GET <@ (user2))) + } finally Scope.scope.vend.deleteScope(granted) - Then("Telemetry is returned") + Then("Telemetry is returned to the Consumer acting as itself") response.code should equal(200) response.body.extract[TelemetryJsonV700].api_instance_id should equal(Constant.ApiInstanceId) + + And("a person calling through testConsumer2, without an Entitlement, is refused") + personResponse.code should equal(403) } } } diff --git a/obp-api/src/test/scala/code/scheduler/ConsentSchedulerTest.scala b/obp-api/src/test/scala/code/scheduler/ConsentSchedulerTest.scala new file mode 100644 index 0000000000..ea92fa3435 --- /dev/null +++ b/obp-api/src/test/scala/code/scheduler/ConsentSchedulerTest.scala @@ -0,0 +1,84 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ + +package code.scheduler + +import java.util.{Date, UUID} + +import code.consent.{ConsentStatus, MappedConsent} +import code.setup.ServerSetup +import com.openbankproject.commons.util.ApiStandards +import net.liftweb.mapper.{By, DB} + +/** + * The unfinishedObpConsents task: OBP consents still INITIATED after the time limit are rejected, so an app that + * reuses a pending consent is never stuck on one. Seeds real consent rows, backdates them with SQL (the ORM sets + * updatedAt itself on save) and runs the task directly. + */ +class ConsentSchedulerTest extends ServerSetup { + + private val sixHours = 21600 + private val hourMs = 3600000L + + private def consent(status: ConsentStatus.Value, apiStandard: String, hoursAgo: Long): String = { + val consentId = UUID.randomUUID.toString + MappedConsent.create + .mConsentId(consentId) + .mStatus(status.toString) + .mApiStandard(apiStandard) + .saveMe() + DB.runUpdate( + s"UPDATE ${MappedConsent.dbTableName} SET ${MappedConsent.updatedAt.dbColumnName} = ? " + + s"WHERE ${MappedConsent.mConsentId.dbColumnName} = ?", + List(new Date(new Date().getTime - hoursAgo * hourMs), consentId) + ) + consentId + } + + private def statusOf(consentId: String): String = + MappedConsent.find(By(MappedConsent.mConsentId, consentId)).map(_.mStatus.get).openOr("missing") + + feature("Rejecting OBP consents that stay INITIATED") { + + scenario("An OBP consent INITIATED for longer than the limit is rejected, with a note saying why") { + val id = consent(ConsentStatus.INITIATED, ApiStandards.obp.toString, hoursAgo = 7) + ConsentScheduler.unfinishedObpConsents(sixHours) + statusOf(id) shouldBe ConsentStatus.REJECTED.toString + MappedConsent.find(By(MappedConsent.mConsentId, id)).map(_.mNote.get).openOr("") should include("not accepted within 21600 seconds") + } + + scenario("A recent INITIATED consent, one already ACCEPTED and one of another standard are left alone") { + val recent = consent(ConsentStatus.INITIATED, ApiStandards.obp.toString, hoursAgo = 1) + val accepted = consent(ConsentStatus.ACCEPTED, ApiStandards.obp.toString, hoursAgo = 7) + val otherStandard = consent(ConsentStatus.INITIATED, "BG", hoursAgo = 7) + ConsentScheduler.unfinishedObpConsents(sixHours) + statusOf(recent) shouldBe ConsentStatus.INITIATED.toString + statusOf(accepted) shouldBe ConsentStatus.ACCEPTED.toString + statusOf(otherStandard) shouldBe ConsentStatus.INITIATED.toString + } + } +} diff --git a/obp-api/src/test/scala/code/setup/DefaultUsers.scala b/obp-api/src/test/scala/code/setup/DefaultUsers.scala index 5b0d2e196a..48844f7029 100644 --- a/obp-api/src/test/scala/code/setup/DefaultUsers.scala +++ b/obp-api/src/test/scala/code/setup/DefaultUsers.scala @@ -265,6 +265,36 @@ trait DefaultUsers { // prepare the OAuth users to login lazy val user1 = Some(consumer, token1) lazy val user2 = Some(consumer2, token2) + + // A Consumer acting as itself: the User an OBP-OIDC client-credentials token resolves to (provider + // obp-oidc#client, named after the client id), and a token for that User and the Consumer. + private def applicationToken(application: code.model.Consumer): Token = { + val ownUser = getOrCreateResourceUser( + provider = code.api.OAuth2Login.OBPOIDC.clientCredentialsProvider, + providerId = application.key.get, + createdByConsentId = None, + name = application.key.get, + email = "", + userId = None, + company = "" + ) + val token = Tokens.tokens.vend.createToken( + Access, + Some(application.id.get), + Some(ownUser.id.get), + Some(randomString(40).toLowerCase), + Some(randomString(40).toLowerCase), + Some(tokenDuration), + Some(TimeSpan(tokenDuration + System.currentTimeMillis())), + Some(new Date(System.currentTimeMillis())), + None + ).openOrThrowException(attemptedToOpenAnEmptyBox) + Token(token.key.get, token.secret.get) + } + lazy val tokenApplication1 = applicationToken(testConsumer) + lazy val tokenApplication2 = applicationToken(testConsumer2) + lazy val application1 = Some(consumer, tokenApplication1) + lazy val application2 = Some(consumer2, tokenApplication2) lazy val user3 = Some(consumer3, token3) lazy val userGatewayLogin = Some(consumer4, token4) diff --git a/obp-api/src/test/scala/code/tenant/TenantMigrationTest.scala b/obp-api/src/test/scala/code/tenant/TenantMigrationTest.scala new file mode 100644 index 0000000000..9660751303 --- /dev/null +++ b/obp-api/src/test/scala/code/tenant/TenantMigrationTest.scala @@ -0,0 +1,137 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ +package code.tenant + +import code.api.Constant.DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID +import code.api.util.APIUtil +import code.api.util.migration.MigrationOfOrganisationToTenant +import code.entitlement.MappedEntitlement +import code.group.Group +import code.setup.ServerSetup +import net.liftweb.db.DB +import net.liftweb.mapper.By +import net.liftweb.util.DefaultConnectionIdentifier +import org.scalatest.Tag + +/** + * The migration from Organisation to Tenant: rows of the old table are copied into Tenant without their bank id, the Organisation Roles held at SYS or the empty bank id become the Tenant system Roles, one held at a bank + * is left alone, and a Group at SYS holding only those Roles moves to the system level. + */ +class TenantMigrationTest extends ServerSetup { + + object TenantMigration extends Tag("TenantMigration") + + private val suffix = APIUtil.generateUUID().take(8) + private val userId = s"tenant-migration-user-$suffix" + + private def sql(statement: String, values: Any*): Unit = + DB.use(DefaultConnectionIdentifier) { connection => + val prepared = connection.prepareStatement(statement) + try { + values.zipWithIndex.foreach { + case (v: String, i) => prepared.setString(i + 1, v) + case (v: Boolean, i) => prepared.setBoolean(i + 1, v) + case (v, i) => prepared.setObject(i + 1, v) + } + prepared.executeUpdate() + } finally prepared.close() + } + + /** The old tables, as Organisation and OrganisationMember stored them. */ + private def oldTables(): Unit = { + sql("CREATE TABLE IF NOT EXISTS organisation (id BIGINT AUTO_INCREMENT PRIMARY KEY, organisationid VARCHAR(64), " + + "bankid VARCHAR(255), name VARCHAR(255), website VARCHAR(1024), logourl VARCHAR(1024), organisationtype VARCHAR(64), " + + "status VARCHAR(32), visibility VARCHAR(32), createdbyuserid VARCHAR(255), creationdate TIMESTAMP, lastupdate TIMESTAMP)") + sql("CREATE TABLE IF NOT EXISTS organisationmember (id BIGINT AUTO_INCREMENT PRIMARY KEY, organisationmemberid VARCHAR(36), " + + "organisationid VARCHAR(64), userid VARCHAR(255), memberrole VARCHAR(32), title VARCHAR(255), status VARCHAR(32), " + + "isprimary BOOLEAN, createdbyuserid VARCHAR(255), createdbyonbehalfofuserid VARCHAR(255), updatedbyuserid VARCHAR(255), " + + "updatedbyonbehalfofuserid VARCHAR(255), creationdate TIMESTAMP, lastupdate TIMESTAMP)") + } + + private def giveEntitlement(bankId: String, roleName: String): Unit = + MappedEntitlement.create.mBankId(bankId).mUserId(userId).mRoleName(roleName) + .mEntitlementId(APIUtil.generateUUID()).saveMe() + + private def entitlementsOf(roleName: String): List[String] = + MappedEntitlement.findAll(By(MappedEntitlement.mRoleName, roleName), By(MappedEntitlement.mUserId, userId)).map(_.mBankId.get) + + feature("Organisation becomes Tenant") { + + scenario("Organisations are copied into the Tenant table, once", TenantMigration) { + oldTables() + val orgId = s"old-org-$suffix" + val now = new java.sql.Timestamp(System.currentTimeMillis()) + sql("INSERT INTO organisation (organisationid, bankid, name, website, logourl, organisationtype, status, visibility, " + + "createdbyuserid, creationdate, lastupdate) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", + orgId, DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID, "Old Org", "https://example.com", "", "startup", "active", "public", userId, now, now) + + MigrationOfOrganisationToTenant.migrate(s"testTenantRows_$suffix") shouldBe true + val tenant = Tenants.tenant.vend.getTenant(orgId).openOrThrowException("tenant should exist") + tenant.name shouldBe "Old Org" + tenant.website shouldBe Some("https://example.com") + + And("running it again copies nothing twice") + MigrationOfOrganisationToTenant.migrate(s"testTenantRowsAgain_$suffix") shouldBe true + Tenant.findAll(By(Tenant.TenantId, orgId)).size shouldBe 1 + } + + scenario("Organisation Roles at SYS or the empty bank id become Tenant system Roles; one at a bank is left", TenantMigration) { + giveEntitlement(DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID, "CanCreateOrganisation") + giveEntitlement("", "CanGetAnyOrganisation") + giveEntitlement("some-bank", "CanUpdateOrganisation") + giveEntitlement("", "CanUpdateOrganisationMember") + giveEntitlement("", "CanGetAnyUser") + + MigrationOfOrganisationToTenant.migrate(s"testTenantRoles_$suffix") shouldBe true + entitlementsOf("CanCreateTenant") shouldBe List("") + entitlementsOf("CanGetTenant") shouldBe List("") + entitlementsOf("CanCreateOrganisation") shouldBe Nil + entitlementsOf("CanUpdateOrganisation") shouldBe List("some-bank") + entitlementsOf("CanUpdateTenant") shouldBe Nil + And("a member Role has no successor and is left as it was") + entitlementsOf("CanUpdateOrganisationMember") shouldBe List("") + entitlementsOf("CanGetAnyUser") shouldBe List("") + } + + scenario("A Group at SYS holding only Organisation Roles moves to the system level; a mixed one stays", TenantMigration) { + def makeGroup(name: String, roles: List[String]): Unit = + Group.create.GroupId(APIUtil.generateUUID()).GroupName(name).GroupDescription("tenant migration test") + .BankId(DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID).ListOfRoles(roles.mkString(",")).IsEnabled(true).saveMe() + def reload(name: String): Group = Group.find(By(Group.GroupName, name)).openOrThrowException(s"group $name") + val only = s"tenant-admins-$suffix" + val mixed = s"mixed-admins-$suffix" + makeGroup(only, List("CanGetOrganisation", "CanUpdateOrganisation")) + makeGroup(mixed, List("CanGetOrganisation", "CanCreateDynamicEntityDefinition")) + + MigrationOfOrganisationToTenant.migrate(s"testTenantGroups_$suffix") shouldBe true + reload(only).BankId.get shouldBe "" + reload(only).ListOfRoles.get shouldBe "CanGetTenant,CanUpdateTenant" + reload(mixed).BankId.get shouldBe DYNAMIC_ENTITY_SYSTEM_LEVEL_BANK_ID + reload(mixed).ListOfRoles.get shouldBe "CanGetTenant,CanCreateDynamicEntityDefinition" + } + } +} diff --git a/obp-api/src/test/scala/code/util/SelfServiceBanksMigrationTest.scala b/obp-api/src/test/scala/code/util/SelfServiceBanksMigrationTest.scala new file mode 100644 index 0000000000..b4ab98d740 --- /dev/null +++ b/obp-api/src/test/scala/code/util/SelfServiceBanksMigrationTest.scala @@ -0,0 +1,84 @@ +/** +Open Bank Project - API +Copyright (C) 2011-2026, TESOBE GmbH. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +Email: contact@tesobe.com +TESOBE GmbH. +Osloer Strasse 16/17 +Berlin 13359, Germany + +This product includes software developed at +TESOBE (http://www.tesobe.com/) + + */ +package code.util + +import code.api.util.APIUtil +import code.api.util.migration.MigrationOfSelfServiceBanks +import code.files.{File, Files, ScanStatus} +import code.model.dataAccess.{MappedBank, SelfServiceBank} +import code.setup.ServerSetup +import net.liftweb.db.DB +import net.liftweb.util.DefaultConnectionIdentifier +import org.scalatest.Tag + +/** + * Recognising the banks Create My Bank made before banks recorded it, and holding their unscanned files as pending. + */ +class SelfServiceBanksMigrationTest extends ServerSetup { + + object SelfServiceBanks extends Tag("SelfServiceBanks") + + private val suffix = APIUtil.generateUUID().take(8) + + feature("Self-service banks") { + + scenario("A bank id has the generated shape only with words from the lists and a hex suffix", SelfServiceBanks) { + BankNameGenerator.isGenerated(BankNameGenerator.generate().bankId) shouldBe true + BankNameGenerator.isGenerated("gh.29.uk") shouldBe false + BankNameGenerator.isGenerated("granite-astra-falcon") shouldBe false + BankNameGenerator.isGenerated("granite-astra-falcon-zzzz") shouldBe false + BankNameGenerator.isGenerated("notaword-astra-falcon-4f2a") shouldBe false + } + + scenario("A generated-looking bank is recorded as self-service and its unscanned files wait; other banks are left", SelfServiceBanks) { + val generated = BankNameGenerator.generate().bankId + val named = s"hand-named-$suffix" + List(generated, named).foreach { id => + MappedBank.create.permalink(id).fullBankName(id).shortBankName(id).saveMe() + } + def oldFile(bankId: String): String = { + val file = Files.provider.vend.createFile(bankId, "0" * 64, "image/png", "old.png", isPublic = false, + "someone", Array[Byte](1, 2, 3)).openOrThrowException("file") + // As stored before scanning existed: no status at all. + DB.use(DefaultConnectionIdentifier) { connection => + val statement = connection.prepareStatement( + s"UPDATE ${File._dbTableNameLC} SET ${File.ScanStatus.dbColumnName} = NULL WHERE ${File.FileId.dbColumnName} = ?") + try { statement.setString(1, file.fileId); statement.executeUpdate() } finally statement.close() + } + file.fileId + } + val heldFile = oldFile(generated) + val keptFile = oldFile(named) + + MigrationOfSelfServiceBanks.markAndHoldFiles(s"testSelfServiceBanks_$suffix") shouldBe true + SelfServiceBank.isSelfService(generated) shouldBe true + SelfServiceBank.isSelfService(named) shouldBe false + Files.provider.vend.getFile(generated, heldFile).map(_.scanStatus).openOr("") shouldBe ScanStatus.Pending + Files.provider.vend.getFile(named, keptFile).map(_.scanStatus).openOr("") shouldBe ScanStatus.Clean + } + } +} diff --git a/release_notes.md b/release_notes.md index 8381c983ea..6f5f276610 100644 --- a/release_notes.md +++ b/release_notes.md @@ -3,6 +3,140 @@ ### Most recent changes at top of file ``` Date Commit Action +01/11/2026 TBD CHANGED: with consents.allowed false (the default) consents are now refused when they are + created, not only when they are used: every endpoint that creates a consent or a consent + request (OBP v3.1.0, v5.0.0, v5.1.0 and v6.0.0, Berlin Group v1.3, UK Open Banking v3.1 and + v4.0.1) answers 401 OBP-35007 first, before a challenge is sent or anything is stored. + Before, such consents were created (and could even be accepted) but could never be used. +01/11/2026 TBD ADDED: OBP consents left INITIATED (their challenge never answered) are now rejected by + a new ConsentScheduler task, unfinishedObpConsents, once unchanged for + obp_unfinished_consents_time_in_seconds (default 21600, 6 hours). It runs every + obp_unfinished_consents_interval_in_seconds (default 593; 0 stops it) and notes the reason + on the consent. Before, an INITIATED OBP consent stayed INITIATED forever, so an app that + reuses a pending consent (as the Portal and other OBP-Frontend apps do for Opey) could be + stuck on one. Berlin Group consents already had the equivalent task. +01/11/2026 TBD CHANGED: v7.0.0 Organisation is now Tenant: a customer of the operator of this OBP + instance, which belongs to no bank (banks will later name their Tenant). The endpoints + move from /banks/BANK_ID/organisations[/ORGANISATION_ID] to /tenants[/TENANT_ID]. JSON + field organisation_id becomes tenant_id; bank_id and organisation_type are gone. The + Roles become system Roles: CanCreateTenant, CanGetTenant, CanUpdateTenant, + CanDeleteTenant. REMOVED: Organisation members (/banks/BANK_ID/organisations/ + ORGANISATION_ID/members[/USER_ID], GET /my/organisations), their four Roles and the + OrganisationMember table's use. Error codes OBP-30506..30513 keep their numbers with + Tenant wording; OBP-30553..30559 and OBP-30597 are gone. New table Tenant; the + migration renameOrganisationsToTenants copies the rows of Organisation (left in place, + unread) and moves the old Roles held at SYS or the empty bank id to the new system + Roles. This replaces moveOrganisationsIntoSpaces. +01/11/2026 TBD CHANGED: Create Customer (v6.0.0 POST /banks/BANK_ID/customers) and Create User Customer + Link (v4.0.0 POST /banks/BANK_ID/user_customer_links) are UserXORApplication: an + application acting as itself may call them with CanCreateCustomer and + CanCreateUserCustomerLink as Scopes. Called with no credentials at all, Create User + Customer Link now answers 401 OBP-20200 (application not identified) instead of 401 + OBP-20001. +01/11/2026 TBD ADDED: v7.0.0 Send Email to Customer. POST /banks/BANK_ID/customers/CUSTOMER_ID/emails + queues a plain-text email (subject, text, optional reference) to one of the bank's + Customers, at the address on the Customer record, through the message outbox; the + address is never returned. Needs the new Role CanSendEmailToCustomerAtOneBank (or the + Scope, for an application acting as itself). Only a Customer of BANK_ID is emailed, and + not in a self-service bank (made with Create My Bank); at most + customer_emails.max_per_customer_per_day (default 20) per Customer in 24 hours. From: + customer_emails.sender_address, else mail.api.consumer.registered.sender.address. + MessageOutbox.enqueueEmail takes optional metadata. +31/10/2026 TBD ADDED: v7.0.0 personal files and scan status. POST /banks/BANK_ID/my/files stores a + file in the caller's own name without a Role, attached to one of their own personal + Dynamic Entity records. Off unless the space is in files.personal.enabled_bank_ids; + needs the owner's email address validated (a User of another identity provider + needs only to have one); limited by + files.personal.allowed_media_types, files.personal.max_size_in_bytes, + files.personal.max_files_per_user and files.personal.max_total_bytes_per_user; never + public. Every file now has scan_status (pending, clean, rejected): personal files start + pending, files stored with CanCreateFileAtOneBank and existing files count as clean, + except in self-service banks (made with Create My Bank), where every file starts + pending. CHANGED: Create My Bank (POST /my/banks), still enabled by + self_service_bank_creation.limit > 0, is refused with OBP-30232 unless OBP stores banks + itself (the mapped connector, or star routing createOrUpdateBank to mapped), so a User + never creates a bank on a backend system. It records each bank in the new table + SelfServiceBank; its quota counts those rows, so banks made with CanCreateBank no longer + count. The migration markSelfServiceBanks records existing banks whose id has the + generated shape, and sets unscanned files in those banks to pending. + Until clean, content is served only to the owner and holders of the new Role + CanUpdateFileScanStatusAtOneBank, which PUT /banks/BANK_ID/files/FILE_ID/scan-status + needs; rejected content is not served. Get Files of a Space takes scan_status=pending. + File responses gain is_personal, scan_status, scan_status_reason and + scan_status_updated_at. New File columns: IsPersonal, ScanStatus, ScanStatusReason, + ScanStatusUpdatedAt, ScanStatusUpdatedByUserId. +30/10/2026 TBD ADDED: v7.0.0 Organisation members. POST/GET /banks/BANK_ID/organisations/ + ORGANISATION_ID/members and GET/PUT/DELETE .../members/USER_ID link Users to an + Organisation with a member_role (owner, admin, member), an optional title, a status + (invited, active, suspended, left) and is_primary. New Roles, granted at the space: + CanCreateOrganisationMember, CanGetOrganisationMembers, CanUpdateOrganisationMember, + CanDeleteOrganisationMember. An active owner or admin member manages members without a + Role, any active member reads the list, and a member may accept, leave and change their + own title. GET /my/organisations lists the current User's memberships. Each write + records both the caller and the person it acted for. New table: OrganisationMember. +30/10/2026 TBD CHANGED: v7.0.0 Organisations live in a space. The endpoints moved from + /obp/v7.0.0/organisations[/ORGANISATION_ID] to + /obp/v7.0.0/banks/BANK_ID/organisations[/ORGANISATION_ID], where BANK_ID is a bank or + SYS for the system space; responses carry bank_id, and an optional organisation_type + can be set. The four Roles now name a space (requiresBankId = true), and + CanGetAnyOrganisation is renamed CanGetOrganisation. The migration + moveOrganisationsIntoSpaces puts every existing Organisation in SYS and moves the + Roles' Entitlements, Entitlement Requests, Consumer Scopes and Groups from the empty + bank id to SYS, so existing grants keep exactly the access they had. A Group that also + holds non-Organisation Roles stays at the empty bank id and is named in the + migration log: grant its Organisation Roles at SYS another way. +29/10/2026 TBD CHANGED: Super admins (super_admin_user_ids) also hold the virtual Roles + CanGetEntitlementRequestsAtAnyBank and CanDeleteEntitlementRequestsAtAnyBank, + so on a fresh instance they can see and accept entitlement requests (accepting + deletes the request) without first granting themselves those Roles. No new + power: a super admin could already grant themselves any Role. +29/10/2026 TBD FIXED: Endpoints that check their Roles again in the handler + (hasAtLeastOneEntitlement, hasAllEntitlements, NewStyle hasEntitlement; e.g. Get + Entitlement Requests, v3.0.0 Get User by id / email / username, v3.1.0 Get Consumers) + now honour the virtual Roles of super_admin_user_ids and oidc_operator_user_ids, + as the ResourceDoc Role check already did, instead of answering 403. +29/10/2026 TBD ADDED: Files in v7.0.0: documents and photographs stored in a space (a bank, or + SYS) and attached to records. POST /banks/BANK_ID/files stores one, base64 in + JSON (Role CanCreateFileAtOneBank; is_public true also needs + CanCreatePublicFileAtOneBank). GET .../files/FILE_ID, .../content (the bytes, + with the file's Content-Type, ETag and Content-Disposition), .../attachments, + GET .../files?record_type=&record_id= and GET .../my/files, for the owner, a + User given access, or CanGetFileAtOneBank. GET /management/banks/BANK_ID/files + lists every file of a space, paginated (CanGetFileAtOneBank). PUT .../files/FILE_ID/is-public + (CanCreatePublicFileAtOneBank). POST/DELETE .../attachments and .../access. + Public files are served without login by GET .../public-files, + .../public-files/FILE_ID and .../public-files/FILE_ID/content, which serve + public files only. No AtAnyBank Roles; a Role at SYS covers the system space. + Media types are at most image/jpeg, image/png, image/webp, application/pdf + (files.allowed_media_types can narrow them); size limit files.max_size_in_bytes + (default 10 MB). New column File.IsPublic, new Roles, new tag File, new errors + OBP-30586 to OBP-30596, new Glossary item Files. +29/10/2026 TBD CHANGED: the endpoint auth mode UserOrApplication is replaced by UserXORApplication. + A request is either a person acting or an application acting as itself, + never a mix. The application acts as itself when there is no User, or the User + is its own, the one an OBP-OIDC client-credentials token resolves to (provider + obp-oidc#client, named after the client id): then only its Consumer's Scopes + count. Otherwise only the User's Entitlements count; a person calling through a + Consumer that holds a Scope no longer gets it (DirectLogin needs only the + consumer key, so anyone with an account and the key could). Who is let in is + now who is recorded as acting. Applies to every endpoint that was + UserOrApplication (log cache, telemetry, Platform Apps, Grants, OIDC / Consumer + bootstrap, Dynamic Entity definitions, Get My Role, verify credentials, ...). + require_scopes_for_all_roles / require_scopes_for_listed_roles no longer make + an application acting as itself on an ApplicationOnly or UserXORApplication + endpoint also need Entitlements. Dynamic Entity auth_mode accepts the former + value UserOrApplication and stores and returns it as UserXORApplication; rows + already stored with it read as UserXORApplication. Get My Role's + allowed_for_auth_modes names UserXORApplication, and application_only is true + only for a User of provider obp-oidc#client. +29/10/2026 TBD CHANGED: v7.0.0 Attach Open Corridor Promise Evidence, Create Open Corridor + Settlement, Create Open Corridor Fee Settlement and Get Open Corridor + Settlement are UserXORApplication: a person with the Role at BANK_ID as an + Entitlement, or an application acting as itself whose Consumer holds it as a + Scope at BANK_ID, such as a bank's Bank Node with a client-credentials token. + reported_by_user_id is the application's own User when it acted. A call with no + credentials still gets 401, now with ApplicationNotIdentified instead of + AuthenticatedUserIsRequired. Response bodies are unchanged. 13/10/2026 TBD ADDED: Domain APIs have is_publicly_discoverable (new column domain_api.is_publicly_discoverable, false by default). The new public GET /obp/v7.0.0/banks/BANK_ID/domain-apis lists, without authentication, the diff --git a/scripts/resource_doc_baseline/parity_allowlist.json b/scripts/resource_doc_baseline/parity_allowlist.json index 5e47f4f95b..fec398b409 100644 --- a/scripts/resource_doc_baseline/parity_allowlist.json +++ b/scripts/resource_doc_baseline/parity_allowlist.json @@ -1225,7 +1225,7 @@ "field": "description", "reason": "Additive clarification, nothing removed.", "lift_digest": "d92aec65916179b76ddfe8cc8040465ac06c69dff9fea5a0ed7aea28c0f926a3", - "http4s_digest": "16f6e6359454b25aeca6f6dd13945d5987010cce18385360cfed9308037d3e22" + "http4s_digest": "1064d2f7004266c3de9988421983a019cf3c4c14531365e5f18a4fe5cec179d6" }, { "version": "v6_0_0", @@ -1343,9 +1343,9 @@ "version": "v6_0_0", "endpoint": "createBankLevelDynamicEntity", "field": "description", - "reason": "Digest refresh: row-level access grant endpoint corrected from GET/POST to GET/PUT (matches Method.PUT route in Http4sDynamicEntity.scala), on top of the field-level access control and indexed/index docs already reviewed. Digest refresh 2026-10-02: documents the hide_field_from_public_access field flag.", + "reason": "Digest refresh: row-level access grant endpoint corrected from GET/POST to GET/PUT (matches Method.PUT route in Http4sDynamicEntity.scala), on top of the field-level access control and indexed/index docs already reviewed. Digest refresh 2026-10-02: documents the hide_field_from_public_access field flag. Digest refresh 2026-10-29: auth_mode UserOrApplication renamed to UserXORApplication (Scopes for the application acting as itself, Entitlements for a person); the former name is still accepted. Digest refresh 2026-10-29: auth_mode UserOrApplication renamed to UserXORApplication (Scopes for the application acting as itself, Entitlements for a person); the former name is still accepted. Digest refresh 2026-10-29: auth_mode UserOrApplication renamed to UserXORApplication (Scopes for the application acting as itself, Entitlements for a person); the former name is still accepted. Digest refresh 2026-10-29: auth_mode UserOrApplication renamed to UserXORApplication (Scopes for the application acting as itself, Entitlements for a person); the former name is still accepted. Digest refresh 2026-10-29: auth_mode UserOrApplication renamed to UserXORApplication (Scopes for the application acting as itself, Entitlements for a person); the former name is still accepted.", "lift_digest": "2e7c649041362776f7eec66ccbb9f025f3edfead992108d0fd5864af76e93961", - "http4s_digest": "6bf7d3278c575dfe04a64e3f20fe7bafbdfccfdc5aff61dafefd73563a20de56" + "http4s_digest": "d2a81d825f3e109ec7cb4e6aece220fe2334a924be6a4c57a8294a3bc8bab0f1" }, { "version": "v6_0_0", @@ -1353,7 +1353,7 @@ "field": "description", "reason": "Digest refresh: row-level access grant endpoint corrected from GET/POST to GET/PUT (matches Method.PUT route in Http4sDynamicEntity.scala), on top of the field-level access control and indexed/index docs already reviewed. Digest refresh 2026-10-02: documents the hide_field_from_public_access field flag.", "lift_digest": "81310cdd2be1c2b33d465cabd1604cedee38bfa1ecccae4b016d0f99aa443ab6", - "http4s_digest": "afe200b33fc38afdb0a212a229e6cec6ccf7caa77b2ac7ff23633fd16638432b" + "http4s_digest": "a54cf6b69e5dd2ea2ac99f012f4e2986adeb2f5c49b344c8984b21f4757b0110" }, { "version": "v6_0_0", @@ -1361,7 +1361,7 @@ "field": "description", "reason": "Digest refresh: row-level access grant endpoint corrected from GET/POST to GET/PUT (matches Method.PUT route in Http4sDynamicEntity.scala), on top of the field-level access control and indexed/index docs already reviewed. Digest refresh 2026-10-02: documents the hide_field_from_public_access field flag.", "lift_digest": "2d18665f9db52506a1b043ad7575b16ac52cedf0d11f036afd4a96b353eb8ddd", - "http4s_digest": "67aea2489cfbaf7b62ab7fe4c1a7fa7c5ad904bb4bd5db5a0977eeb37298a1cb" + "http4s_digest": "9cbf8c520307ca38d82c69d079504d19c344696220ec459b9ef38475adb55332" }, { "version": "v6_0_0", @@ -1369,7 +1369,7 @@ "field": "description", "reason": "Digest refresh: row-level access grant endpoint corrected from GET/POST to GET/PUT (matches Method.PUT route in Http4sDynamicEntity.scala), on top of the field-level access control and indexed/index docs already reviewed. Digest refresh 2026-10-02: documents the hide_field_from_public_access field flag.", "lift_digest": "97f0c97cc592f89f4dba8a6641ad4b41f6ffa6f4f6e78bf3d3cc34a6a11f76e7", - "http4s_digest": "b3d573ff89040298846a54f3ce897f96f6abac1de477c9ea4efe267e8c8f13ff" + "http4s_digest": "71c7c1036177280c60d568471033fc9dac7dbdc5ae171241920bd6bc49edf9ea" }, { "version": "v6_0_0", @@ -1377,7 +1377,7 @@ "field": "description", "reason": "Digest refresh: row-level access grant endpoint corrected from GET/POST to GET/PUT (matches Method.PUT route in Http4sDynamicEntity.scala), on top of the field-level access control and indexed/index docs already reviewed. Digest refresh 2026-10-02: documents the hide_field_from_public_access field flag.", "lift_digest": "e28c2996f3fd54842534fd54d45d7ba49df8b346e65f541353be4fb5311b550c", - "http4s_digest": "6f71dd7c0ad0fcbdc973f5bebbed11b75c57a7f6bef56613d92a999b12bf6821" + "http4s_digest": "03efe3fdb3fb7c924c0c880942931e6fe51dc889d288a52946ee7b84a25f95db" }, { "version": "v3_0_0", @@ -2666,6 +2666,30 @@ "reason": "Added ApiCollectionNotFound: 404 when the collection does not exist or belongs to another user, so only its owner can update it", "lift_digest": "ea8061d2189d61d4f84552f70fa4482c88aa1fb6d5f7692b7c3c6eae0b24f4e6", "http4s_digest": "181a990959c91843d2617852d9247a24ae916364b5b3afa58fab87ddd6692130" + }, + { + "version": "v4_0_0", + "endpoint": "createUserCustomerLinks", + "field": "description", + "reason": "Says who may call it: an application acting as itself may link a User to a Customer with CanCreateUserCustomerLink as a Scope (UserXORApplication)", + "lift_digest": "a8a1007d01ccd9676513bfbb2bcd793671473041930bcd25489f8e1a8bf37b02", + "http4s_digest": "6125d4f8108dda68e7c88dc280d2807534972e43d51728df40c705bc78ed0c02" + }, + { + "version": "v6_0_0", + "endpoint": "createCustomer", + "field": "description", + "reason": "Says who may call it: an application acting as itself may create a Customer with CanCreateCustomer as a Scope (UserXORApplication)", + "lift_digest": "56f72c04242cf24d496141ac1a6665508ae086553eaabd51c6064dad5b149a7d", + "http4s_digest": "f128cd0805126f789fa5c23585276034b7fc630570c34f5daa27c514e0a1dacd" + }, + { + "version": "v2_1_0", + "endpoint": "getTransactionRequestTypesSupportedByBank", + "field": "description", + "reason": "Explains that the list comes from the transactionRequests_supported_types prop and when OPEN_CORRIDOR_PROMISE is included. Additive, nothing removed.", + "lift_digest": "dea1479859f8f2e4eead46eb86caef95b294ee947771d5d0ef71df39d288cdae", + "http4s_digest": "77a01355ab52e9743c1d2a10bb40273ad237df04d45fdea42a23828437bb729d" } ] } diff --git a/todo/authmode_user_or_application_design.md b/todo/authmode_user_or_application_design.md new file mode 100644 index 0000000000..2bee00e705 --- /dev/null +++ b/todo/authmode_user_or_application_design.md @@ -0,0 +1,168 @@ +# AuthMode: review the design, especially UserOrApplication + +Status: UserXORApplication implemented in the working tree on 2026-10-29 (uncommitted; not yet compiled or +tested). Dynamic Entities accept the former name UserOrApplication as an alias. Written 2026-10-28. + +## Principle: attribution follows authorization + +Whoever's permission let an action through is the one responsible for it. A record of an +action should say whose grant was used, not only who was logged in: + + action -> grant used (the User's Entitlement, or the Consumer's Scope) -> who created that grant + +If an Entitlement let it through, the person acted on authority given to them by its grantor. +If a Scope let it through, the Consumer acted on authority given to it by the Scope's grantor; +for an application, that grantor (and whoever runs the application) is accountable. + +Platform App is not part of this. It is a marker plus a convenient way for an application to +declare the Scopes it needs; marking a Consumer grants nothing, and any Consumer can hold +Scopes marked or not. Rules about authority should not depend on the marker. + +## The four modes + +`APIUtil.handleAccessControlWithAuthMode`: + +- **UserOnly** (Entitlements). Clear: a person acts on their own authority. +- **UserAndApplication** (Entitlements AND Scopes). Clear: the person needs the Role, and the + Scope limits which applications may exercise it. Restricts the application's power. +- **ApplicationOnly** (Scopes). Clear for machine applications: the application acts as itself. +- **UserOrApplication** (Scopes OR Entitlements). Too loose. When both hold the Role nothing + says which grant was used; when only the Consumer holds it, a person acts on the + application's authority but is recorded as if it were their own (e.g. `reported_by_user_id`). + +## The underlying hole: people can log in through any Consumer + +DirectLogin needs only `username`, `password` and `consumer_key` (not the secret), and +consumer keys sit in `.env` files and config. Anyone with an OBP account and a Consumer's key +can call through that Consumer. So: + +- under **UserOrApplication**, they get the Consumer's Scopes; +- under **ApplicationOnly** as implemented today, they also pass (only Scopes are checked). + +Endpoints already on `UserOrApplication` include log cache (v5.1.0), telemetry +(`Http4s700Telemetry.scala`) and the OIDC / Consumer bootstrap endpoints. + +An OAuth2 client-credentials token from OBP-OIDC gets a User of its own (provider +`obp-oidc#client`, id = client id; `OAuth2.scala` `resolveProvider` / `clientCredentialsProvider`), +so an application acting as itself can be told apart from a person. + +## Direction to weigh + +1. **A Consumer that holds Scopes (a machine Consumer) does not let people log in.** + Only the client-credentials grant (no DirectLogin, no authorization_code, no redirect URIs); + the only User ever seen with it is its own `obp-oidc#client` User. This is a property of the + Consumer's credentials, not of the Platform App marker. Applications that people log in to + (Portal, API Manager, an operator UI) use other Consumers. +2. **Equivalently, in the check:** a Scope counts only when the Consumer is acting as itself + (its own client-credentials User, or no User). A person calling through it needs their own + Entitlement. With 1 in place, this always holds; checking it as well covers Consumers created + before the rule. +3. **Retire UserOrApplication**, or keep it only where 1/2 make it unambiguous (then the Scope + path is always the application and the Entitlement path always a person, and attribution is + clear). Move existing endpoints to ApplicationOnly, UserOnly or UserAndApplication as fits. +4. **Record the basis** with the action where it matters: user_id, consumer_id, and whether an + Entitlement or a Scope was used (ideally the grant's id), so the chain above can be followed. + +Later: delegation (an application acting for a person, `on_behalf_of` in v7) needs both +principals and the consent recorded. + +## Proposal: replace UserOrApplication with UserXORApplication (2026-10-29) + +Each request is either a person acting or an application acting as itself, never a mix: + +- **Application acting as itself:** no User, or the User is the application's own + client-credentials User (provider `obp-oidc#client`, id = client id). Only the Consumer's + Scopes count. +- **Otherwise a person:** only their own Entitlements count; the Consumer's Scopes are ignored. + +Decided by who is acting, not by which credentials are present: a person's request always comes +through a Consumer too, so "not both credentials" never happens. Write that definition next to +the mode, since the name alone can read that way. + +Gives: attribution always matches the grant used; a person who logs in with a machine +Consumer's key gets none of its Scopes; and the strict props can apply to the person path only. + +Rename rather than change UserOrApplication in place, so every current use (log cache, +telemetry, OIDC / Consumer bootstrap) is reviewed and moved to UserXORApplication, UserOnly or +ApplicationOnly, then remove the OR mode. Machine applications using client credentials (e.g. +OBP-Sentinel) keep working; only a person borrowing a Consumer's Scope stops working, which is +the intent. + +## Decided 2026-10-29: go with UserXORApplication. Implementation plan + +Checked first: a bare consumer key does not make a request "the application acting as itself". +With no Authorization header, `getUserAndSessionContextFuture` identifies a Consumer only by +certificate (mTLS / TPP signature); a Consumer-Key header is used only for consents, and only +with `consumer_validation_method_for_consent=CONSUMER_KEY_VALUE`. So the application path is +reached by a client-credentials token (its `obp-oidc#client` User) or a certificate. + +authMode does two jobs; XOR changes only the second: +1. Authentication: may a request with a Consumer but no User come in at all + (`ResourceDocMiddleware.authenticate` `isAppMode` -> `applicationAccess`). Unchanged. +2. Role check (`APIUtil.handleAccessControlWithAuthMode`): XOR = if the caller is the + application acting as itself, Scopes only; otherwise the User's Entitlements only. The + strict props make the person path Entitlement AND Scope and leave the application path alone. + +Steps: +- Add `UserXORApplication` to `EndpointAuthMode` (APIUtil); implement the role check with an + "is the application acting as itself" helper (no User, or User provider `obp-oidc#client` with + id = the Consumer's key / azp). Same logic in `Http4s700MyRoles` (its own copy of the check). +- Move every `authMode = UserOrApplication` to `UserXORApplication`. About 50 uses in main: + - Without Roles (Berlin Group AIS/PIS, UK OB v3.1.0 / v4.0.1, OIDC, Dynamic Entity + definitions, MyRoles): only job 1 matters, so behaviour is identical. Mechanical. + - With Roles (v5.1.0 log cache, telemetry, Platform Apps, Grants, some v6.0.0 and v7.0.0): + behaviour changes as intended (a person no longer uses the Consumer's Scope). Review each; + some may belong as ApplicationOnly or UserOnly instead. +- Then remove `UserOrApplication` from `EndpointAuthMode`. +- Dynamic Entities: `DynamicEntityAuthMode.UserOrApplication` is a stored, API-visible value + (`DynamicEntityProvider.scala`, `normalise`, v6.0.0 JSON). Contract decision needed: accept the + old string as an alias mapped to the new one, and say so in the release notes. +- Docs: Glossary (authMode item, Platform Apps), `OpenAPI31JSONFactory`, + `introductory_system_documentation.md`, release notes. The Lift baseline JSON + (`scripts/resource_doc_baseline/lift_resource_docs_v5_1_0.json`, `_v6_0_0.json`) mentions it: + do not hand-edit; use `parity_allowlist.json` via `allowlist_helper.py` if the audit flags it. +- Tests: ~35 references in 11 suites (`EndpointAuthModeTest`, `DynamicEntityAuthModeTest`, + BG/UK consent tests, ...). Add: person through a Consumer that holds the Scope -> 403; + application's own client-credentials User with the Scope -> allowed; person with own + Entitlement through that Consumer -> allowed and recorded as the person. +- The parked `open-corridor-user-or-application` branch: switch its four endpoints to + UserXORApplication once the mode exists (rebase onto the XOR branch). + +DirectLogin for Consumers with Scopes: not needed for safety under XOR (a person logging in +through such a Consumer gets only their own Entitlements). Optional hardening; separate decision. +Disabling redirect URLs for Consumers with Scopes: rejected (the Portal needs both). + +## Per-instance strictness + +Decided 2026-10-29: props are fine for this (boot-time is OK). But the override must not flatten +the application path: under XOR it should make the person path UserAndApplication (Entitlement +AND the Consumer's Scope) and leave the application path Scopes-only, so client-credentials +Scope access keeps working. + +One instance may want to be strict (e.g. UserAndApplication for people everywhere). Partly +possible today: `require_scopes_for_all_roles=true` and `require_scopes_for_listed_roles=CanX,...` +make the check `userHasTheRoles && hasScope` before the endpoint's authMode is looked at. +Limits: + +- Props, so set at boot; changing them needs a restart. +- They flatten every mode, ApplicationOnly included: a machine application's own + `obp-oidc#client` User would then also need Entitlements for every Role its Scopes cover, + which breaks machine applications or pushes admins to grant Entitlements to app users. +- They can only tighten, all endpoints or per Role; nothing like "this instance does not + allow UserOrApplication". + +Earlier idea, superseded by the XOR proposal and the props decision above: an instance policy +mapping each mode (UserOnly -> UserAndApplication; UserOrApplication -> UserAndApplication for +people, ApplicationOnly for machine Consumers; ApplicationOnly unchanged), possibly held in the +database and changeable at run time. + +## Work parked on this + +Branch `open-corridor-user-or-application` (uncommitted, tests not run): sets +`authMode = UserOrApplication` on v7.0.0 attachOpenCorridorPromise, +createOpenCorridorSettlement, createOpenCorridorFeeSettlement and getOpenCorridorSettlement, +with description text, four Scope tests in `Http4s700RoutesTest` and a release-notes entry. +Do not merge it until the rule above is decided. For OBP-Bank-Node the likely end state is a +machine Consumer per node (client-credentials only, Scopes at its own bank) calling these +endpoints, while a person's "Settle now" goes through a different Consumer on their own +Entitlement. diff --git a/todo/todo_account_access_for_consumers.md b/todo/todo_account_access_for_consumers.md new file mode 100644 index 0000000000..e376f8f2ad --- /dev/null +++ b/todo/todo_account_access_for_consumers.md @@ -0,0 +1,86 @@ +# TODO — Account access granted to a Consumer + +## Context + +OBP has two layers of permission, and only one of them can be granted to a Consumer today: + +| | Granted to a user | Granted to a Consumer | +|---|---|---| +| **What may be called** (Roles) | Entitlement | Scope | +| **Which accounts, through which view** | Account access | *missing* | + +Under `UserXORApplication` an application acting as itself (client credentials: no user, or its own +`obp-oidc#client` user, see `APIUtil.isApplicationActingAsItself`) is authorised by its Scopes. But an +endpoint authorised through a view, such as creating a Transaction Request on +`/banks/BANK_ID/accounts/ACCOUNT_ID/VIEW_ID/...`, has nothing for it to use: account access is held only by +users. + +The case that raised it: a bank-side service that initiates `OPEN_CORRIDOR_PROMISE` Transaction Requests on +the bank's corridor account, on behalf of the customers whose payments it relays. It needs the +report-back and settlement endpoints (bank Roles, available as Scopes) and the Transaction Request create +(the account's view). Today it runs as a DirectLogin service user who owns the account. + +Alternatives set aside: + +- **Account access for the Consumer's own `obp-oidc#client` user.** Works without changes, but the same + principal is the *application* in role checks and a *user* in account access. Its identity also follows + the consumer key (`idGivenByProvider`), so rotating the key loses the access. +- **A bank Role + Scope for the Transaction Request create.** Any Consumer granted the Scope could initiate + payments from any account at the bank. Too broad for payment initiation: who may initiate from an account + belongs on the account. +- **A Consent.** Consents are given by people, on whose behalf an app then acts. Here the Consumer itself + acts. + +## Wished-for direction + +Let account access be granted to a Consumer, next to account access granted to a user, so that +"who may use this view of this account" stays on the account and the actor is the Consumer itself. +Attribution follows authorization: action → account access → the Consumer → who granted it. + +**Additive and opt-in**, so existing user paths don't change: + +- A **separate table**, `consumer_account_access`, shaped like `AccountAccess` with the Consumer in place + of the user and the grantor recorded (unlike `MappedScope`, which doesn't record who granted it): + `consumer_account_access_id` (UUID), `consumer_id`, `bank_id`, `account_id`, `view_id`, + `granted_by_user_id`, created/updated; unique on (`consumer_id`, `bank_id`, `account_id`, `view_id`). + Not the existing + `AccountAccess` table: its `consumer_id` column already means something else, namely that a *user's* + grant may only be used through that Consumer (`ALL-CONSUMERS` by default; `User.hasAccountAccess`). A + user-less row there would change what lookups by `consumer_id` alone return + (`AccountAccess.findByBankIdAccountIdViewIdConsumerId`), and `user_fk` is part of the unique index. +- Consulted **only when the caller is an application acting as itself**, and only on endpoints that opt in + through their auth mode (`ApplicationOnly` / `UserXORApplication`). A person calling through the same + Consumer never picks up the Consumer's account access: that is the leak the old `UserOrApplication` had + with Scopes. +- Start with the endpoints that need it (the Transaction Request create), not the whole view-check + surface. + +## Open questions + +1. **Which views.** Not `owner`: account holding and "own account" permissions are user concepts, and a + Consumer should not become an account holder. A dedicated view holding only the permission needed (e.g. + creating Transaction Requests) is least privilege. +2. **Who may grant it.** The account holder, or someone with a grant-access permission on that view. The + grant records the grantor. +3. **Consumers that may hold it.** Client credentials only. A Consumer that holds account access, like one + that holds Scopes, should not let people log in through it. +4. **Declaring it.** The Platform App declaration (`PUT /consumers/current/platform-app`) could gain + `required_account_access` entries (bank, account, view, `needed_for`), approved by the account holder, + as Scopes are approved by administrators. +5. **SCA / challenges.** Transaction Request creation can raise a challenge above a threshold, and a + Consumer can't answer an OTP. Application-initiated requests need a rule (no challenge, or a limit on + the view). This is a policy question as much as a code one. +6. **Recording the initiator.** The Transaction Request and the Transaction it posts should show that a + Consumer initiated them. Check how the initiator is stored today; if it is only a user, this is a + data-model change. +7. **Listing and revoking.** Endpoints to grant, list and revoke, and the account's access list showing + Consumers next to users. + +## Size and risk + +The view check takes a user throughout: `checkViewAccessAndReturnView` has about 48 call sites and +`hasAccountAccess` about 22, and `AccountAccess` appears in about 58 files. Making all of that +Consumer-aware would be a large and risky change. Done as above (a new table, one extra branch in the view +check for an application acting as itself, opt-in per endpoint), existing behaviour is untouched, and the +risk sits in the new branch, which tests can cover. The larger work is the policy around it: questions 2, +5 and 6.