From 5b988d3944171e746f57f31e8d6c4bbc03b6d5d9 Mon Sep 17 00:00:00 2001 From: Dan Wolfson Date: Fri, 23 Oct 2026 14:14:34 -0400 Subject: [PATCH 01/17] Fix role-appointment class names and mismatched delete/detach bodies (ISSUE-163) - link_person/team/it_profile_role_to_*: default properties class is now Person/Team/ITProfileRoleAppointmentProperties (old names don't exist in Egeria; a no-body call got HTTP 400). - detach_it_profile_role_from_it_profile, detach_identity_from_profile: send DeleteRelationshipRequestBody (was DeleteElementRequestBody, 400 live). - delete_actor_role, delete_contribution_record: send DeleteElementRequestBody. - GlossaryManager.remove_is_context_definition: DeleteClassificationRequestBody. - Dr.Egeria Link IT Profile Role Appointment called a non-existent SDK method. Verified against ActorManagerResource.java / GlossaryManagerResource.java and live on quickstart 9443. 11 micro-tests, 9 fail on the old code. Signed-off-by: Dan Wolfson --- PYEGERIA_ISSUES.md | 34 +++++ md_processing/v2/actor_manager.py | 2 +- pyegeria/omvs/actor_manager.py | 44 +++---- pyegeria/omvs/glossary_manager.py | 6 +- .../test_role_appointment_class_names.py | 122 ++++++++++++++++++ 5 files changed, 182 insertions(+), 26 deletions(-) create mode 100644 tests/micro-tests/test_role_appointment_class_names.py diff --git a/PYEGERIA_ISSUES.md b/PYEGERIA_ISSUES.md index 3c267f2f..1f44cd15 100644 --- a/PYEGERIA_ISSUES.md +++ b/PYEGERIA_ISSUES.md @@ -286,6 +286,40 @@ they're holding real port materialization until this exists server-side --- +### ISSUE-163: Actor Manager role-appointment links send non-existent `*AppointmentProperties` classes, and five delete/detach methods send the wrong request-body class + +**Fixed 2026-10-23 on branch `fix/role-appointment-class-names` (not yet released).** **Live on quickstart 9443** (two runs, throwaway `pyeg-test::role-appt::` elements, all read back as gone by GUID afterwards, and a find on the prefix returned nothing). The old person-link class was refused with a 400. All three fixed links, called with no body, read back the relationship to the profile. The old IT detach body was refused with a 400; the fixed detach made the link unreadable. The fixed `delete_actor_role` deleted its role. + +**Layer:** Pyegeria (+ Dr.Egeria) · **Status:** fixed on branch · **Found:** 2026-10-23, reported by the owner ("pyegeria is using the wrong class name when linking a role to a profile"). + +**Root cause:** each method below either passes a properties class that doesn't exist to `_async_new_relationship_request`, or calls the shared delete helper for the wrong body class. Neither is caught client-side. A wrong `prop` list only matters when the caller passes no body: the helper then builds `{"class": "NewRelationshipRequestBody", "properties": {"class": prop[0]}}` itself. My Profile's "add role" calls `link_person_role_to_profile` that way. The ground truth is `ActorManagerResource.java` / `GlossaryManagerResource.java` (`@RequestBody` types) and `open-metadata-framework .../properties/actors/*RoleAppointmentProperties.java`. **The `.http` file is wrong for `actor-roles/{guid}/delete`:** it says `DeleteRelationshipRequestBody`, but Egeria declares `DeleteElementRequestBody`. + +| Method | Sent | Egeria expects | Live before fix | +|---|---|---|---| +| `link_person_role_to_profile` | `PersonAppointmentProperties` | `PersonRoleAppointmentProperties` | 400 | +| `link_team_role_to_profile` | `TeamAppointmentProperties` | `TeamRoleAppointmentProperties` | (same shape) | +| `link_it_profile_role_to_it_profile` | `ITProfileAppointmentProperties` | `ITProfileRoleAppointmentProperties` | (same shape) | +| `detach_it_profile_role_from_it_profile` | `DeleteElementRequestBody` | `DeleteRelationshipRequestBody` | 400 | +| `detach_identity_from_profile` (`user-identities/{g}/profile-identity/{g}/detach`) | `DeleteElementRequestBody` | `DeleteRelationshipRequestBody` | not run | +| `delete_actor_role` | `DeleteRelationshipRequestBody` | `DeleteElementRequestBody` | accepted (tolerated) | +| `delete_contribution_record` | `DeleteRelationshipRequestBody` | `DeleteElementRequestBody` | not run | +| `GlossaryManager.remove_is_context_definition` | `DeleteRelationshipRequestBody` | `DeleteClassificationRequestBody` | not run | + +**IT detach request and response (before the fix), Egeria quickstart 9443, 2026-10-23:** +``` +POST .../actor-manager/actor-roles/a6e578d4-…/it-profile-role-appointments/91cc35e6-…/detach +{"forLineage":false,"forDuplicateProcessing":false,"class":"DeleteElementRequestBody","cascadeDelete":false} +→ HTTP 400 (pyegeria reports it as CLIENT_ERROR_400, "The client is unable to connect to the Egeria platform", which is misleading for a 400) +``` + +**Dr.Egeria:** `ActorManagerLinkProcessor` (`md_processing/v2/actor_manager.py`) called `_async_link_it_profile_role_to_profile` for `Link IT Profile Role Appointment`. No such method exists (the real one is `_async_link_it_profile_role_to_it_profile`), so that command raised `AttributeError` on every Link. + +**Fix:** corrected the three `prop` lists; switched the two detaches to `_async_delete_relationship_request`, `delete_actor_role` and `delete_contribution_record` to `_async_delete_element_request`, and `remove_is_context_definition` to `_async_delete_classification_request`; aligned the type hints and docstrings; corrected the Dr.Egeria method name. Tests: `tests/micro-tests/test_role_appointment_class_names.py` (11 tests; 9 of them fail on the old code; the two that pass cover the person and team detaches, which were already correct). + +**Checked and left alone:** `MetadataExpert.delete_related_elements` sends `DeleteRelationshipRequestBody`, while Egeria declares `OpenMetadataDeleteRequestBody`. That's deliberate (ISSUE-63): the Egeria class can't carry `deleteMethod`. Decertify and unlicense (`GovernanceOfficerResource`: `DeleteRelationshipRequestBody`) are already correct. **Follow-up:** `scripts/omvs_audit.py --service actor-manager` on the old code reported 59 OK and 1 unrelated mismatch, so its BODY check flagged none of these. It takes the first `*RequestBody` identifier in the method, which is the signature annotation, so a wrong helper with a matching type hint looks clean. Comparing the helper called against the `.http` body class would catch this class of bug. + +--- + ### ISSUE-161: Dr.Egeria's two entry points resolve the Egeria user differently, and the CLI fixes its credential defaults at import time **Layer:** Pyegeria · **Status:** fixed on branch `fix/issue-161-dr-egeria-credentials` (owner approved 2026-10-21), not yet released · **Found:** 2026-10-21, while checking an RE report that `dr_egeria --process` ignored `EGERIA_USER`. **That report is withdrawn; RE was the cause:** RE's own `config.py:316` defaulted its daemon identity (`EGERIA_USER_ID`) to "erinoverview", so RE handed the child process erinoverview. The CLI correctly used it. RE is removing that default. diff --git a/md_processing/v2/actor_manager.py b/md_processing/v2/actor_manager.py index 435b4acc..b9e9a538 100644 --- a/md_processing/v2/actor_manager.py +++ b/md_processing/v2/actor_manager.py @@ -145,7 +145,7 @@ async def apply_changes(self) -> str: # Canonical: IT Profile Role, IT Profile role_guid = attributes.get("IT Profile Role", {}).get("guid") profile_guid = attributes.get("IT Profile", {}).get("guid") - link_coro = self.client.actor_manager._async_link_it_profile_role_to_profile + link_coro = self.client.actor_manager._async_link_it_profile_role_to_it_profile detach_coro = self.client.actor_manager._async_detach_it_profile_role_from_it_profile else: raise PyegeriaException(f"Unsupported role appointment type: {object_type}") diff --git a/pyegeria/omvs/actor_manager.py b/pyegeria/omvs/actor_manager.py index 68f13fd3..109dbfc8 100644 --- a/pyegeria/omvs/actor_manager.py +++ b/pyegeria/omvs/actor_manager.py @@ -1659,7 +1659,7 @@ async def _async_link_person_role_to_profile(self, person_role_guid: str, person url = url = ( f"{self.command_root}/actor-roles/{person_role_guid}/person-role-appointments/{person_profile_guid}/attach") - await self._async_new_relationship_request(url, ["PersonAppointmentProperties"], body) + await self._async_new_relationship_request(url, ["PersonRoleAppointmentProperties"], body) logger.debug(f"Linking person role {person_role_guid} to Person Profile {person_profile_guid}") @dynamic_catch @@ -1851,7 +1851,7 @@ async def _async_link_team_role_to_profile(self, team_role_guid: str, team_profi url = url = ( f"{self.command_root}/actor-roles/{team_role_guid}/team-role-appointments/{team_profile_guid}/attach") - await self._async_new_relationship_request(url, ["TeamAppointmentProperties"], body) + await self._async_new_relationship_request(url, ["TeamRoleAppointmentProperties"], body) logger.debug(f"Linking team role {team_role_guid} to Team Profile {team_profile_guid}") @dynamic_catch @@ -2041,7 +2041,7 @@ async def _async_link_it_profile_role_to_it_profile(self, it_profile_role_guid: url = url = ( f"{self.command_root}/actor-roles/{it_profile_role_guid}/it-profile-role-appointments/{it_profile_guid}/attach") - await self._async_new_relationship_request(url, ["ITProfileAppointmentProperties"], body) + await self._async_new_relationship_request(url, ["ITProfileRoleAppointmentProperties"], body) logger.debug(f"Linking IT Profile role {it_profile_role_guid} to IT Profile {it_profile_guid}") @dynamic_catch @@ -2137,7 +2137,7 @@ async def _async_detach_it_profile_role_from_it_profile(self, it_profile_role_gu url = ( f"{self.command_root}/actor-roles/{it_profile_role_guid}/it-profile-role-appointments/{it_profile_guid}/detach") - await self._async_delete_element_request(url, body) + await self._async_delete_relationship_request(url, body) logger.debug(f"Detached IT Profile Role {it_profile_role_guid} from IT Profile {it_profile_guid}") def detach_it_profile_role_from_it_profile(self, it_profile_role_guid: str, it_profile_guid: str, @@ -2188,7 +2188,7 @@ def detach_it_profile_role_from_it_profile(self, it_profile_role_guid: str, it_p @dynamic_catch async def _async_delete_actor_role(self, actor_role_guid: str, - body: Optional[dict | DeleteRelationshipRequestBody] = None, + body: Optional[dict | DeleteElementRequestBody] = None, cascade: bool = False) -> None: """ Delete an actor role. Async Version. @@ -2221,7 +2221,7 @@ async def _async_delete_actor_role(self, actor_role_guid: str, _____ JSON Structure looks like: { - "class": "DeleteRelationshipRequestBody", + "class": "DeleteElementRequestBody", "externalSourceGUID": "add guid here", "externalSourceName": "add qualified name here", "effectiveTime": "{{$isoTimestamp}}", @@ -2231,11 +2231,11 @@ async def _async_delete_actor_role(self, actor_role_guid: str, """ url = f"{self.command_root}/actor-roles/{actor_role_guid}/delete" - await self._async_delete_relationship_request(url, body, cascade) + await self._async_delete_element_request(url, body, cascade) logger.debug(f"Deleted actor role {actor_role_guid} with cascade {cascade}") @dynamic_catch - def delete_actor_role(self, actor_role_guid: str, body: Optional[dict | DeleteRelationshipRequestBody] = None, + def delete_actor_role(self, actor_role_guid: str, body: Optional[dict | DeleteElementRequestBody] = None, cascade: bool = False) -> None: """ Delete an actor role. Async Version. @@ -2247,7 +2247,7 @@ def delete_actor_role(self, actor_role_guid: str, body: Optional[dict | DeleteRe cascade: bool, optional, defaults to True If true, a cascade delete is performed. - body: dict | DeleteRelationshipRequestBody, optional, default = None + body: dict | DeleteElementRequestBody, optional, default = None A structure representing the details of the relationship. Returns @@ -2268,7 +2268,7 @@ def delete_actor_role(self, actor_role_guid: str, body: Optional[dict | DeleteRe _____ JSON Structure looks like: { - "class": "DeleteRelationshipRequestBody", + "class": "DeleteElementRequestBody", "externalSourceGUID": "add guid here", "externalSourceName": "add qualified name here", "effectiveTime": "{{$isoTimestamp}}", @@ -3328,7 +3328,7 @@ async def _async_detach_identity_from_profile(self, user_identity_guid: str, act """ url = (f"{self.command_root}/user-identities/{user_identity_guid}/profile-identity/{actor_profile_guid}/detach") - await self._async_delete_element_request(url, body) + await self._async_delete_relationship_request(url, body) logger.debug(f"Detached User Identity {user_identity_guid} from it a actor profile {actor_profile_guid}") @dynamic_catch @@ -4744,15 +4744,15 @@ def update_contribution_record(self, contribution_record_guid: str, body: Option return asyncio.run(self._async_update_contribution_record(contribution_record_guid, body)) @dynamic_catch - async def _async_delete_contribution_record(self, contribution_record_guid: str, body: Optional[dict | DeleteRelationshipRequestBody] = None) -> None: + async def _async_delete_contribution_record(self, contribution_record_guid: str, body: Optional[dict | DeleteElementRequestBody] = None) -> None: """ Delete a contribution record. Async version. Parameters ---------- contribution_record_guid: str The unique identifier of the contribution record to delete. - body: dict | DeleteRelationshipRequestBody, optional - A dict or DeleteRelationshipRequestBody representing the delete options. + body: dict | DeleteElementRequestBody, optional + A dict or DeleteElementRequestBody representing the delete options. Returns ------- @@ -4764,7 +4764,7 @@ async def _async_delete_contribution_record(self, contribution_record_guid: str, One of the pyegeria exceptions will be raised if there are issues in communications, message format, or Egeria errors. ValidationError - Pydantic validation errors are raised if the body does not conform to the DeleteRelationshipRequestBody. + Pydantic validation errors are raised if the body does not conform to the DeleteElementRequestBody. PyegeriaNotAuthorizedException The principle specified by the user_id does not have authorization for the requested action @@ -4772,7 +4772,7 @@ async def _async_delete_contribution_record(self, contribution_record_guid: str, ----- example: { - "class" : "DeleteRelationshipRequestBody", + "class" : "DeleteElementRequestBody", "cascadeDelete" : false, "externalSourceGUID": "add guid here", "externalSourceName": "add qualified name here", @@ -4782,18 +4782,18 @@ async def _async_delete_contribution_record(self, contribution_record_guid: str, } """ url = f"{self.command_root}/contribution-records/{contribution_record_guid}/delete" - await self._async_delete_relationship_request(url, body) + await self._async_delete_element_request(url, body) @dynamic_catch - def delete_contribution_record(self, contribution_record_guid: str, body: Optional[dict | DeleteRelationshipRequestBody] = None) -> None: + def delete_contribution_record(self, contribution_record_guid: str, body: Optional[dict | DeleteElementRequestBody] = None) -> None: """ Delete a contribution record. Parameters ---------- contribution_record_guid: str The unique identifier of the contribution record to delete. - body: dict | DeleteRelationshipRequestBody, optional - A dict or DeleteRelationshipRequestBody representing the delete options. + body: dict | DeleteElementRequestBody, optional + A dict or DeleteElementRequestBody representing the delete options. Returns ------- @@ -4805,7 +4805,7 @@ def delete_contribution_record(self, contribution_record_guid: str, body: Option One of the pyegeria exceptions will be raised if there are issues in communications, message format, or Egeria errors. ValidationError - Pydantic validation errors are raised if the body does not conform to the DeleteRelationshipRequestBody. + Pydantic validation errors are raised if the body does not conform to the DeleteElementRequestBody. PyegeriaNotAuthorizedException The principle specified by the user_id does not have authorization for the requested action @@ -4813,7 +4813,7 @@ def delete_contribution_record(self, contribution_record_guid: str, body: Option ----- example: { - "class" : "DeleteRelationshipRequestBody", + "class" : "DeleteElementRequestBody", "cascadeDelete" : false, "externalSourceGUID": "add guid here", "externalSourceName": "add qualified name here", diff --git a/pyegeria/omvs/glossary_manager.py b/pyegeria/omvs/glossary_manager.py index dce02163..09ba8729 100644 --- a/pyegeria/omvs/glossary_manager.py +++ b/pyegeria/omvs/glossary_manager.py @@ -2180,7 +2180,7 @@ def add_is_context_definition( @dynamic_catch async def _async_remove_is_context_definition( - self, term_guid: str, body: Optional[dict | DeleteRelationshipRequestBody] = None, + self, term_guid: str, body: Optional[dict | DeleteClassificationRequestBody] = None, ) -> None: """Remove the context definition designation from the glossary term. Async Version. @@ -2221,12 +2221,12 @@ async def _async_remove_is_context_definition( f"{self.platform_url}/servers/{self.view_server}/api/open-metadata/glossary-manager/glossaries/" f"terms/{term_guid}/is-context-definition/delete" ) - await self._async_delete_relationship_request(url, body) + await self._async_delete_classification_request(url, body) logger.info(f"Removed ContextDefinition classification to {term_guid}") @dynamic_catch def remove_is_context_definition( - self, term_guid: str, body: Optional[dict | DeleteRelationshipRequestBody] = None, + self, term_guid: str, body: Optional[dict | DeleteClassificationRequestBody] = None, ) -> None: """Remove the context definition designation from the glossary term. diff --git a/tests/micro-tests/test_role_appointment_class_names.py b/tests/micro-tests/test_role_appointment_class_names.py new file mode 100644 index 00000000..5fa196e6 --- /dev/null +++ b/tests/micro-tests/test_role_appointment_class_names.py @@ -0,0 +1,122 @@ +""" +Role-to-profile appointment links (PersonRoleAppointment, TeamRoleAppointment, +ITProfileRoleAppointment) must send Egeria's real properties class name when the +caller passes no body -- the SDK previously defaulted to PersonAppointmentProperties / +TeamAppointmentProperties / ITProfileAppointmentProperties, which don't exist +(see open-metadata-framework .../properties/actors/*RoleAppointmentProperties.java). +No live server: the HTTP call is captured and the URL and body asserted. +""" +import json + +import pytest + +from md_processing.v2.actor_manager import ActorManagerLinkProcessor +from pyegeria.omvs.actor_manager import ActorManager + + +class _Response: + def json(self): + return {"class": "GUIDResponse", "guid": "rel-guid", "relatedHTTPCode": 200} + + +def _actor_manager(calls, monkeypatch): + monkeypatch.setattr(ActorManager, "check_connection", lambda self: "test") + client = ActorManager("view-server", "https://localhost:1", "user", "pwd") + + async def fake_make_request(method, url, payload=None, *args, **kwargs): + calls.append({"method": method, "url": url, + "body": json.loads(payload) if isinstance(payload, str) else payload}) + return _Response() + + client._async_make_request = fake_make_request + return client + + +@pytest.mark.asyncio +@pytest.mark.parametrize("method, segment, expected_class", [ + ("_async_link_person_role_to_profile", "person-role-appointments", "PersonRoleAppointmentProperties"), + ("_async_link_team_role_to_profile", "team-role-appointments", "TeamRoleAppointmentProperties"), + ("_async_link_it_profile_role_to_it_profile", "it-profile-role-appointments", + "ITProfileRoleAppointmentProperties"), +]) +async def test_link_role_to_profile_without_body_sends_real_class_name(monkeypatch, method, segment, + expected_class): + calls: list = [] + client = _actor_manager(calls, monkeypatch) + + await getattr(client, method)("role-guid", "profile-guid") + + assert calls[0]["method"] == "POST" + assert calls[0]["url"].endswith(f"/actor-roles/role-guid/{segment}/profile-guid/attach") + assert calls[0]["body"]["properties"]["class"] == expected_class + + +@pytest.mark.asyncio +@pytest.mark.parametrize("method, url_tail", [ + ("_async_detach_person_role_from_profile", "/actor-roles/a-guid/person-role-appointments/b-guid/detach"), + ("_async_detach_team_role_from_profile", "/actor-roles/a-guid/team-role-appointments/b-guid/detach"), + ("_async_detach_it_profile_role_from_it_profile", + "/actor-roles/a-guid/it-profile-role-appointments/b-guid/detach"), + ("_async_detach_identity_from_profile", "/user-identities/a-guid/profile-identity/b-guid/detach"), +]) +async def test_detach_sends_a_relationship_body_not_an_element_body(monkeypatch, method, url_tail): + # ActorManagerResource.java declares DeleteRelationshipRequestBody for these + # /detach endpoints; a DeleteElementRequestBody got a live 400 (IT profile role). + calls: list = [] + client = _actor_manager(calls, monkeypatch) + + await getattr(client, method)("a-guid", "b-guid", + {"class": "DeleteRelationshipRequestBody", "forLineage": True}) + + assert calls[0]["url"].endswith(url_tail) + assert calls[0]["body"]["class"] == "DeleteRelationshipRequestBody" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("method, url_tail", [ + ("_async_delete_actor_role", "/actor-roles/a-guid/delete"), + ("_async_delete_contribution_record", "/contribution-records/a-guid/delete"), +]) +async def test_element_delete_sends_an_element_body(monkeypatch, method, url_tail): + # ActorManagerResource.java declares DeleteElementRequestBody for these /delete + # endpoints (the .http file says DeleteRelationshipRequestBody for actor-roles -- wrong). + calls: list = [] + client = _actor_manager(calls, monkeypatch) + + await getattr(client, method)("a-guid", {"class": "DeleteElementRequestBody", "forLineage": True}) + + assert calls[0]["url"].endswith(url_tail) + assert calls[0]["body"]["class"] == "DeleteElementRequestBody" + + +@pytest.mark.asyncio +async def test_remove_is_context_definition_sends_a_classification_body(monkeypatch): + # GlossaryManagerResource.java: .../is-context-definition/delete takes DeleteClassificationRequestBody. + from pyegeria.omvs.glossary_manager import GlossaryManager + monkeypatch.setattr(GlossaryManager, "check_connection", lambda self: "test") + client = GlossaryManager("view-server", "https://localhost:1", "user", "pwd") + calls: list = [] + + async def fake_make_request(method, url, payload=None, *args, **kwargs): + calls.append({"url": url, "body": json.loads(payload) if isinstance(payload, str) else payload}) + return _Response() + + client._async_make_request = fake_make_request + await client._async_remove_is_context_definition( + "term-guid", {"class": "DeleteClassificationRequestBody", "forLineage": True}) + + assert calls[0]["url"].endswith("/glossaries/terms/term-guid/is-context-definition/delete") + assert calls[0]["body"]["class"] == "DeleteClassificationRequestBody" + + +def test_dr_egeria_it_profile_role_appointment_targets_an_existing_sdk_method(): + # Dr.Egeria's Link IT Profile Role Appointment used to reference + # _async_link_it_profile_role_to_profile, which ActorManager doesn't define. + import inspect + source = inspect.getsource(ActorManagerLinkProcessor) + for name in ("_async_link_person_role_to_profile", "_async_link_team_role_to_profile", + "_async_link_it_profile_role_to_it_profile", + "_async_detach_it_profile_role_from_it_profile"): + assert name in source + assert hasattr(ActorManager, name) + assert "_async_link_it_profile_role_to_profile" not in source From 1a6db16f2fcf70e5234e53f44bfd1d97bb34056c Mon Sep 17 00:00:00 2001 From: Dan Wolfson Date: Fri, 23 Oct 2026 14:28:32 -0400 Subject: [PATCH 02/17] omvs_audit: check request bodies and class names against Egeria's Java source (ISSUE-164) - RENAMED (verb+path) matches now get the verb/body checks; they were skipped, which hid two ISSUE-163 detach bugs. - New Java ground truth (--java-dir / $EGERIA_JAVA_DIR, --no-java to skip): indexes Spring endpoints -> @RequestBody type and reports JAVA-BODY REJECTED (sibling class under a @JsonTypeInfo base: a 400), CLASS-IGNORED, UNKNOWN-CLASS (Properties/RequestBody names Egeria lacks, scanned in pyegeria/omvs, md_processing and commands), JAVA-MISSING and HTTP-vs-JAVA. A .http BODY mismatch Java contradicts no longer fails. - Strips the 'retrieve"}' junk on some .http lines. On the pre-ISSUE-163 code it flags all eight ISSUE-163 bugs. Signed-off-by: Dan Wolfson --- CLAUDE.md | 25 +- scripts/README.md | 4 + scripts/omvs_audit.py | 360 +++++++++++++++++- .../micro-tests/test_omvs_audit_java_truth.py | 83 ++++ 4 files changed, 451 insertions(+), 21 deletions(-) create mode 100644 tests/micro-tests/test_omvs_audit_java_truth.py diff --git a/CLAUDE.md b/CLAUDE.md index f9165315..5e0d612d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -177,12 +177,31 @@ after a real defect slipped through an earlier, weaker version of this script: trailing verbs (`/attach` vs `/detach`) or the leading service segment; an earlier version did, and that masked exactly the bugs worth finding. - **BODY** — compares the request-body class (`FilterRequestBody` vs - `ResultsRequestBody`, etc.). Heuristic: it takes the first `*RequestBody` - identifier in the method, which may be a signature annotation, so treat body - findings as lower-confidence than verb/path. + `ResultsRequestBody`, etc.). When the method delegates to a shared + `_async_*_request` helper it uses the classes that helper accepts (what is + actually sent); otherwise it falls back to the method's own `body` + annotation, which is lower-confidence. - **LINT** — structural URL defects detectable without ground truth: double slashes, a path parameter with no separator before it (`/agreements{guid}`), and underscores in path segments (Egeria uses hyphens). +- **RENAMED** — a method found by verb+path under an unrelated name now gets + the verb/body checks too. It used to be reported as RENAMED and skipped, + which is how the IT-profile-role and profile-identity detaches (ISSUE-163) + sent the wrong body unchecked. +- **Java checks** (needs `~/localGit/egeria-v6/egeria`, or `--java-dir` / + `$EGERIA_JAVA_DIR`; `--no-java` skips) — the Spring `@RequestBody` types are + a second ground truth, added after ISSUE-163, where the `.http` file named + the same wrong class as the SDK. **JAVA-BODY REJECTED** (defect) only fires + when the declared type sits under a `@JsonTypeInfo(property="class")` base. + Among request bodies that's only `DeleteRequestBody` → + `Delete{Element,Relationship,Classification}RequestBody`, where a sibling + class is a 400. Elsewhere Egeria ignores `"class"` and maps fields by name, + so a mismatch there is **CLASS-IGNORED** (review: only dropped fields + matter). **UNKNOWN-CLASS** (defect) flags any `"…Properties"` / + `"…RequestBody"` string in `pyegeria/omvs` that isn't an Egeria Java class. + The properties bases are polymorphic, so sending one is a 400; this is what + caught `PersonAppointmentProperties`. **JAVA-MISSING** (review) lists SDK + endpoints with no Spring mapping. URL extraction is AST-based, not regex — multi-line f-strings silently produced empty paths under the regex version, which made whole modules look clean. diff --git a/scripts/README.md b/scripts/README.md index 758c9dbe..66bfda58 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -11,6 +11,10 @@ package or the pytest suite. file (URL, HTTP verb, request-body class) and reports MISSING/VERB MISMATCH/PATH MISMATCH/BODY MISMATCH/ELSEWHERE/LINT findings (see `omvs_audit_report.md` at the repo root for the latest run's output). + When an Egeria Java checkout is present (`--java-dir`, `$EGERIA_JAVA_DIR`, + default `~/localGit/egeria-v6/egeria`) it also checks against the Spring + `@RequestBody` types: JAVA-BODY REJECTED / CLASS-IGNORED, UNKNOWN-CLASS, + JAVA-MISSING and HTTP-vs-JAVA (ISSUE-163). `--no-java` skips them. `python scripts/omvs_audit.py [--report PATH] [--service NAME] [--quiet]`; exits 1 on any confirmed defect, so it can gate CI. - `safe_git_sync.sh` — stashes all local changes (including untracked), diff --git a/scripts/omvs_audit.py b/scripts/omvs_audit.py index b5b2fcda..90ab1feb 100644 --- a/scripts/omvs_audit.py +++ b/scripts/omvs_audit.py @@ -15,6 +15,24 @@ @name (no amount of snake_case guessing would find it) * LINT - malformed URLs (``//``, missing separators) in the SDK +RENAMED matches get the verb and body checks too (they used to be skipped). + +With an Egeria Java checkout (``--java-dir``, ``$EGERIA_JAVA_DIR``, default +``~/localGit/egeria-v6/egeria``) the Spring resources are a second ground +truth, because the .http files can name the same wrong class the SDK sends: + + * JAVA-BODY REJECTED - the helper's body class is not assignable to the + declared @RequestBody type, and that type sits + under a @JsonTypeInfo(property="class") base, so + Egeria returns 400 (defect) + * JAVA-BODY CLASS-IGNORED - a different class, but Egeria ignores "class" + there and maps fields by name (review) + * UNKNOWN-CLASS - a "...Properties"/"...RequestBody" string in the + SDK that is no Egeria Java class (defect: under a + polymorphic properties base this is a 400) + * JAVA-MISSING - no Spring mapping for the SDK's verb+path (review) + * HTTP-vs-JAVA - the .http body class disagrees with Java (info) + Usage ----- python scripts/omvs_audit.py [--report PATH] [--service NAME] [--quiet] @@ -27,8 +45,8 @@ https://localhost:9443. Best-effort: an unreachable platform just skips live verification rather than failing the run. -Exit status is 1 when any confirmed defect (verb/path/body mismatch or lint) -is found, so this can gate CI. +Exit status is 1 when any confirmed defect (verb/path/body mismatch, lint, +JAVA-BODY REJECTED or UNKNOWN-CLASS) is found, so this can gate CI. Note: ``pyegeria/http-clients/`` is gitignored. It must be present locally or the audit cannot run - the script fails loudly rather than reporting a clean @@ -178,6 +196,7 @@ def canon_path(path: str) -> str: # Strip copy/paste junk seen on some .http lines. Deliberately excludes '}' # and ']' so a path ending in a parameter ("/{{guid}}") stays intact. path = path.rstrip('"\'`),;') + path = re.sub(r'"\}+$', "", path) # 'retrieve"}' junk seen on some .http lines path = re.sub(r"\{\{[^}]+\}\}", "{}", path) # {{guid}} -> {} path = re.sub(r"\{[^}]*\}", "{}", path) # {guid} -> {} path = re.sub(r"\{\}(?=\{\})", "", path) # collapse adjacent params @@ -403,6 +422,147 @@ def discover_helper_bodies(path: str) -> dict[str, frozenset[str]]: return bodies +# -------------------------------------------------------------------------- +# second ground truth: Egeria's Java source +# -------------------------------------------------------------------------- +# The .http collections are hand-maintained and can be wrong in exactly the +# way that hides an SDK bug: for actor-roles/{guid}/delete and +# is-context-definition/delete they named the same wrong body class the SDK +# sent, so the .http comparison reported OK (ISSUE-163). The Spring resource +# classes' @RequestBody types are what the server actually deserialises. + +JAVA_DIR_DEFAULT = os.environ.get("EGERIA_JAVA_DIR", + os.path.expanduser("~/localGit/egeria-v6/egeria")) +MAPPING_RE = re.compile(r"@(Get|Post|Put|Delete|Patch)Mapping\s*\(") +CLASS_MAPPING_RE = re.compile(r"@RequestMapping\s*\(") +REQUEST_BODY_RE = re.compile(r"@RequestBody\s*(?:\([^)]*\))?\s*([A-Z]\w*)") +EXTENDS_RE = re.compile(r"\bclass\s+(\w+)(?:<[^>]*>)?\s+extends\s+(\w+)") +# String constants in the SDK that name an Egeria Java class. +CLASS_CONST_RE = re.compile(r"^[A-Z][A-Za-z0-9]*(?:Properties|RequestBody)$") + + +def _balanced(text: str, open_at: int) -> str: + """Text between the '(' at ``open_at`` and its matching ')'.""" + depth = 0 + for i in range(open_at, len(text)): + if text[i] == "(": + depth += 1 + elif text[i] == ")": + depth -= 1 + if depth == 0: + return text[open_at + 1:i] + return text[open_at + 1:] + + +@dataclass +class JavaTruth: + # (verb, canonical path) -> {body class or None}. A generic resource + # mapped under {urlMarker} has "{}" as its service segment. + endpoints: dict[tuple[str, str], set[str | None]] + # class -> superclass, for every Java class seen (Jackson accepts a + # subclass of the declared @RequestBody type, nothing else). + parents: dict[str, str] + # Stems of every *Properties.java / *RequestBody.java file. + known_classes: set[str] + # Classes annotated @JsonTypeInfo(property="class"). Below one of these, + # Jackson resolves "class" as a type id and a non-assignable value is a + # 400 (DeleteRequestBody -> Delete{Element,Relationship,Classification}). + # Everywhere else "class" is an ignored unknown property and fields map + # by name, so a wrong class there is harmless unless fields are dropped. + polymorphic: set[str] = field(default_factory=set) + + def lookup(self, verb: str, path: str) -> set[str | None] | None: + hit = self.endpoints.get((verb, path)) + if hit is None: + parts = path.split("/") + if len(parts) > 2: + hit = self.endpoints.get((verb, "/".join(["", "{}"] + parts[2:]))) + return hit + + def ancestors(self, cls: str) -> list[str]: + chain, seen = [], set() + while cls and cls not in seen: + chain.append(cls) + seen.add(cls) + cls = self.parents.get(cls) + return chain + + def rejects(self, declared: str, sent: str) -> bool: + """True when Egeria refuses ``sent`` where ``declared`` is expected.""" + return (not self.accepts(declared, sent) + and any(a in self.polymorphic for a in self.ancestors(declared))) + + def accepts(self, declared: str, sent: str) -> bool: + seen = set() + while sent and sent not in seen: + if sent == declared: + return True + seen.add(sent) + sent = self.parents.get(sent) + return False + + +def load_java_truth(java_dir: str) -> JavaTruth | None: + impl = os.path.join(java_dir, "open-metadata-implementation") + if not os.path.isdir(impl): + return None + endpoints: dict[tuple[str, str], set[str | None]] = defaultdict(set) + parents: dict[str, str] = {} + known: set[str] = set() + polymorphic: set[str] = set() + for root, _dirs, files in os.walk(impl): + if {"test", "build", "target"} & set(os.path.relpath(root, impl).split(os.sep)): + continue + for fname in files: + if not fname.endswith(".java"): + continue + stem = fname[:-5] + if stem.endswith(("Properties", "RequestBody")): + known.add(stem) + if not (stem.endswith(("Properties", "RequestBody", "Options", "Resource")) + or "properties" in root or "rest" in root): + continue + text = open(os.path.join(root, fname), encoding="utf-8", errors="ignore").read() + for child, parent in EXTENDS_RE.findall(text): + parents.setdefault(child, parent) + if "@JsonTypeInfo" in text and 'property = "class"' in text.replace("property=", "property = "): + polymorphic.add(stem) + if not (stem.endswith("Resource") and "spring" in root.split(os.sep)): + continue + cm = CLASS_MAPPING_RE.search(text) + base = "" + if cm: + strs = re.findall(r'"([^"]*)"', _balanced(text, cm.end() - 1)) + base = strs[0] if strs else "" + for mm in MAPPING_RE.finditer(text): + verb = mm.group(1).upper() + args = _balanced(text, mm.end() - 1) + paths = [s for s in re.findall(r'"([^"]*)"', args) if s.startswith("/")] or [""] + sig_open = text.find("(", text.find("public", mm.end())) + body = None + if sig_open > 0: + rb = REQUEST_BODY_RE.search(_balanced(text, sig_open)) + body = rb.group(1) if rb else None + for p in paths: + endpoints[(verb, canon_path(base + p))].add(body) + return JavaTruth(dict(endpoints), parents, known, polymorphic) + + +def class_constants(filepath: str) -> list[tuple[int, str]]: + """(line, name) for every string constant that names a Properties / + RequestBody class - prop lists, default bodies, module-level lists.""" + try: + tree = ast.parse(open(filepath, encoding="utf-8").read()) + except SyntaxError as exc: + print(f" ! skipping {filepath}: {exc}", file=sys.stderr) + return [] + # The fixed tail of f"{type}StatusRequestBody" is not a class name on its own. + fragments = {id(v) for n in ast.walk(tree) if isinstance(n, ast.JoinedStr) for v in n.values} + return [(n.lineno, n.value) for n in ast.walk(tree) + if isinstance(n, ast.Constant) and isinstance(n.value, str) + and id(n) not in fragments and CLASS_CONST_RE.match(n.value)] + + @dataclass class PyMethod: name: str @@ -419,6 +579,11 @@ class PyMethod: # either branch under the same or a different @name, and `path` alone # (whichever assignment the walk sees first) isn't enough to match both. all_paths: list[str] = field(default_factory=list) + # "helper" when body_class is what a shared _async_*_request helper + # validates and sends; "signature" when it is only the method's own type + # hint (the method posts whatever the caller passes). + body_source: str | None = None + lineno: int = 0 def _flatten(node: ast.AST, roots: dict[str, str]) -> str: @@ -621,6 +786,8 @@ def parse_py_file(filepath: str, helper_verbs: dict[str, str], name=fn.name, verb=direct_verb or helper_verb, path=path, raw_url=raw_url, body_class=helper_body or body_class, lint=lint, all_paths=all_paths, + body_source="helper" if helper_body else ("signature" if body_class else None), + lineno=fn.lineno, ) for sm in sync_names: @@ -635,7 +802,7 @@ def parse_py_file(filepath: str, helper_verbs: dict[str, str], def audit(service_filter: str | None, report_path: str, quiet: bool, http_dir: str = HTTP_DIR, live_url: str | None = None, - live_insecure: bool = True) -> int: + live_insecure: bool = True, java_dir: str | None = JAVA_DIR_DEFAULT) -> int: if not os.path.isdir(http_dir): sys.exit( f"ERROR: '{http_dir}' not found.\n" @@ -658,6 +825,11 @@ def audit(service_filter: str | None, report_path: str, quiet: bool, helper_verbs = discover_helper_verbs(SERVER_CLIENT) helper_bodies = discover_helper_bodies(SERVER_CLIENT) + java = load_java_truth(java_dir) if java_dir else None + if java_dir and java is None: + print(f" ! Egeria Java source not found at '{java_dir}'; skipping the " + "JAVA-BODY / UNKNOWN-CLASS checks (--java-dir or $EGERIA_JAVA_DIR)", + file=sys.stderr) http_by_service = {} for hf in http_files: @@ -739,6 +911,32 @@ def audit(service_filter: str | None, report_path: str, quiet: bool, lines.extend(dupe_rows) lines.append("") + def compare(d: PyMethod, req: HttpRequest) -> list[str]: + issues = [] + if d.verb and d.verb != req.verb: + issues.append(f"VERB {d.verb} != {req.verb}") + candidate_paths = d.all_paths or ([d.path] if d.path else []) + if candidate_paths and not any(paths_compatible(p, req.path) for p in candidate_paths): + issues.append(f"PATH\n SDK: {d.path}\n API: {req.path}") + # Mismatch only when the documented class is not among those the + # SDK path accepts (several helpers accept a union). + if (d.body_class and req.body_class + and "RequestBody" in req.body_class + and req.body_class not in d.body_class + and not java_confirms(d)): + issues.append( + f"BODY sends {'|'.join(sorted(d.body_class))} != {req.body_class}") + return issues + + def java_confirms(d: PyMethod) -> bool: + """The Spring @RequestBody type accepts what the SDK sends, so a + disagreeing .http file is the stale side (listed under HTTP-vs-JAVA).""" + if java is None or not (d.verb and d.path and d.body_class): + return False + declared = {b for p in (d.all_paths or [d.path]) + for b in (java.lookup(d.verb, p) or ()) if b} + return bool(declared) and any(java.accepts(x, sent) for x in declared for sent in d.body_class) + # ---- coverage pass --------------------------------------------------- for svc, requests in sorted(http_by_service.items()): if svc in SKIP_SERVICES: @@ -797,7 +995,19 @@ def audit(service_filter: str | None, report_path: str, quiet: bool, elif by_path_hit: counts["renamed"] += 1 hits = ", ".join(f"`{s}.py`:`{m}`" for s, m in by_path_hit) - lines.append(f"- {name}: RENAMED -> {hits}{live_tag}") + # A path match says nothing about the body: the IT-profile + # role and profile-identity detaches (ISSUE-163) matched + # here and sent the wrong body class unchecked. + renamed_issues = [ + f"`{m}`: {i}" for s, m in by_path_hit + for i in compare(py_by_service[s][m], req)] + if renamed_issues: + counts["mismatch"] += 1 + defects += 1 + lines.append(f"- {name}: MISMATCH (renamed) -> {hits}{live_tag}") + lines += [f" - {i}" for i in renamed_issues] + else: + lines.append(f"- {name}: RENAMED -> {hits}{live_tag}") else: counts["missing"] += 1 lines.append(f"- {name}: MISSING (`{req.verb} {req.path}`){live_tag}") @@ -806,19 +1016,7 @@ def audit(service_filter: str | None, report_path: str, quiet: bool, check = match if match.startswith("_async_") else f"_async_{match}" d = py_methods.get(check) or py_methods[match] - issues = [] - if d.verb and d.verb != req.verb: - issues.append(f"VERB {d.verb} != {req.verb}") - candidate_paths = d.all_paths or ([d.path] if d.path else []) - if candidate_paths and not any(paths_compatible(p, req.path) for p in candidate_paths): - issues.append(f"PATH\n SDK: {d.path}\n API: {req.path}") - # Mismatch only when the documented class is not among those the - # SDK path accepts (several helpers accept a union). - if (d.body_class and req.body_class - and "RequestBody" in req.body_class - and req.body_class not in d.body_class): - issues.append( - f"BODY sends {'|'.join(sorted(d.body_class))} != {req.body_class}") + issues = compare(d, req) if issues: counts["mismatch"] += 1 @@ -830,6 +1028,108 @@ def audit(service_filter: str | None, report_path: str, quiet: bool, if not quiet or live_tag: lines.append(f"- {name}: OK{live_tag}") + # ---- Java ground-truth pass ------------------------------------------- + java_rows: list[str] = [] + java_soft: list[str] = [] + java_ignored: list[str] = [] + unknown_rows: list[str] = [] + http_vs_java: list[str] = [] + java_missing: list[str] = [] + if java is not None: + # Names of pyegeria's own Python classes (e.g. the SpecificationProperties + # OMVS client, or FindPropertyNamesRequestBody, a local model) appear as + # strings in __all__ lists; they are not wire class names. + py_classes = {n.name for f in glob.glob("pyegeria/**/*.py", recursive=True) + for n in ast.walk(ast.parse(open(f, encoding="utf-8").read())) + if isinstance(n, ast.ClassDef)} + for svc, ms in sorted(py_by_service.items()): + if service_filter and service_filter != svc: + continue + for m, d in sorted(ms.items()): + if not (m.startswith("_async_") and d.verb and d.path): + continue + declared: set[str] = set() + mapped = False + for p in (d.all_paths or [d.path]): + hit = java.lookup(d.verb, p) if p else None + if hit is not None: + mapped = True + declared |= {b for b in hit if b} + if not mapped and "{" not in d.path.split("/")[1] and d.path.count("/") > 1: + java_missing.append(f"- `{svc}.py:{d.lineno}` `{m}` `{d.verb} {d.path}`") + if not d.body_class: + continue + if not declared: + continue + if any(java.accepts(decl, sent) for decl in declared for sent in d.body_class): + counts["java_ok"] += 1 + continue + row = (f"- `{svc}.py:{d.lineno}` `{m}` `{d.verb} {d.path}`: sends " + f"{'|'.join(sorted(d.body_class))}, Java declares " + f"{'|'.join(sorted(declared))}") + rejected = all(java.rejects(decl, sent) for decl in declared for sent in d.body_class) + if d.body_source != "helper": + java_soft.append(row) + elif rejected: + java_rows.append(row) + else: + java_ignored.append(row) + + # Dr.Egeria and the CLI build bodies too (curation.py's Retention spec + # sent the non-existent RetentionClassificationProperties, ISSUE-163). + omvs_files = sorted(glob.glob(f"{OMVS_DIR}/*.py")) + [SERVER_CLIENT] + other_files = sorted(glob.glob("md_processing/**/*.py", recursive=True) + + glob.glob("commands/**/*.py", recursive=True)) + for pf in omvs_files + ([] if service_filter else other_files): + svc = os.path.basename(pf).replace(".py", "").replace("_omvs", "").replace("_", "-") + if service_filter and service_filter != svc: + continue + seen: set[str] = set() + for lineno, cls in class_constants(pf): + if cls in java.known_classes or cls in py_classes or cls in seen: + continue + seen.add(cls) + unknown_rows.append(f"- `{pf}:{lineno}`: `{cls}` is not an Egeria Java class") + + for svc, requests in sorted(http_by_service.items()): + if service_filter and service_filter != svc: + continue + for name, req in requests.items(): + if not (req.body_class and req.body_class.endswith("RequestBody")): + continue + hit = java.lookup(req.verb, req.path) + declared = {b for b in (hit or ()) if b} + if declared and not any(java.accepts(decl, req.body_class) for decl in declared): + http_vs_java.append( + f"- {svc}: {name} `{req.verb} {req.path}`: .http says " + f"{req.body_class}, Java declares {'|'.join(sorted(declared))}") + + lines.append("\n## Java ground truth (Spring @RequestBody types)\n") + lines.append(f"_Source: `{java_dir}`._\n") + lines.append("### JAVA-BODY REJECTED: the endpoint refuses the body class the SDK sends (HTTP 400)\n") + lines.append("_The declared class sits under a `@JsonTypeInfo(property=\"class\")` base, so " + "`class` is a type id and a non-assignable value fails deserialisation._\n") + lines.extend(java_rows or ["_none_"]) + lines.append("\n### JAVA-BODY CLASS-IGNORED: a different class, but Egeria ignores `class` here\n") + lines.append("_No `@JsonTypeInfo` on the declared class, so fields map by name. Harmless unless " + "the SDK relies on a field only its own class has; that field is silently dropped._\n") + lines.extend(java_ignored or ["_none_"]) + lines.append("\n### JAVA-BODY (low confidence): the method's own type hint disagrees; " + "it posts whatever the caller passes\n") + lines.extend(java_soft or ["_none_"]) + lines.append("\n### UNKNOWN-CLASS: SDK names a Properties/RequestBody class Egeria does not have\n") + lines.extend(unknown_rows or ["_none_"]) + lines.append("\n### JAVA-MISSING (review): no Spring mapping for this verb+path\n") + lines.append("_Removed or renamed in Egeria, or an endpoint the SDK guessed. Static path " + "resolution can also miss a runtime-built segment, so confirm before changing._\n") + lines.extend(java_missing or ["_none_"]) + lines.append("\n### HTTP-vs-JAVA (informational): the .http file disagrees with Java\n") + lines.extend(http_vs_java or ["_none_"]) + lines.append("") + defects += len(java_rows) + len(unknown_rows) + counts["java_body"] = len(java_rows) + counts["unknown_class"] = len(unknown_rows) + header = [ "# OMVS Audit Report", "", @@ -845,6 +1145,16 @@ def audit(service_filter: str | None, report_path: str, quiet: bool, f"| Found in another module | {counts['elsewhere']} |", f"| URL lint | {counts['lint']} |", ] + if java is not None: + header += [ + f"| Java-verified body classes | {counts['java_ok']} |", + f"| JAVA-BODY REJECTED (helper sends a class the endpoint refuses) | {len(java_rows)} |", + f"| JAVA-BODY CLASS-IGNORED (different class, `class` ignored by Egeria) | {len(java_ignored)} |", + f"| JAVA-BODY low confidence (type hint only) | {len(java_soft)} |", + f"| UNKNOWN-CLASS | {len(unknown_rows)} |", + f"| JAVA-MISSING (no Spring mapping; review) | {len(java_missing)} |", + f"| HTTP-vs-JAVA disagreements (informational) | {len(http_vs_java)} |", + ] if live_index is not None: header += [ f"| Live-verified against {live_url} | {counts['live_ok']} |", @@ -859,6 +1169,12 @@ def audit(service_filter: str | None, report_path: str, quiet: bool, print(f" OK={counts['ok']} mismatch={counts['mismatch']} " f"missing={counts['missing']} elsewhere={counts['elsewhere']} " f"lint={counts['lint']}") + if java is not None: + print(f" java_ok={counts['java_ok']} java_rejected={len(java_rows)} " + f"java_class_ignored={len(java_ignored)} " + f"java_soft={len(java_soft)} unknown_class={len(unknown_rows)} " + f"java_missing={len(java_missing)} " + f"http_vs_java={len(http_vs_java)}") if live_index is not None: print(f" live_ok={counts['live_ok']} live_stale={counts['live_stale']}") return 1 if defects else 0 @@ -891,9 +1207,17 @@ def main() -> int: help="verify TLS when fetching --live (default: off, for self-signed " "local dev platforms)", ) + ap.add_argument( + "--java-dir", default=JAVA_DIR_DEFAULT, + help="Egeria Java checkout used as a second ground truth for request-body " + "classes and class names (defaults to $EGERIA_JAVA_DIR or " + "~/localGit/egeria-v6/egeria; skipped with a warning if absent)", + ) + ap.add_argument("--no-java", action="store_true", help="skip the Java checks") args = ap.parse_args() return audit(args.service, args.report, args.quiet, args.http_dir, - live_url=args.live, live_insecure=not args.live_verify_tls) + live_url=args.live, live_insecure=not args.live_verify_tls, + java_dir=None if args.no_java else args.java_dir) if __name__ == "__main__": diff --git a/tests/micro-tests/test_omvs_audit_java_truth.py b/tests/micro-tests/test_omvs_audit_java_truth.py new file mode 100644 index 00000000..cec11222 --- /dev/null +++ b/tests/micro-tests/test_omvs_audit_java_truth.py @@ -0,0 +1,83 @@ +""" +omvs_audit's Java ground truth (ISSUE-163): endpoint -> @RequestBody index, the +polymorphic-base rule that decides REJECTED vs CLASS-IGNORED, and the class-name +scan. Runs against a tiny synthetic Java tree, not a real Egeria checkout. +""" +import sys +import textwrap +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "scripts")) + +import omvs_audit # noqa: E402 + +FILES = { + "frameworks/x/src/main/java/rest/DeleteRequestBody.java": """ + @JsonTypeInfo(use = JsonTypeInfo.Id.NAME, include = JsonTypeInfo.As.PROPERTY, property = "class") + @JsonSubTypes({@JsonSubTypes.Type(value = DeleteElementRequestBody.class, name = "DeleteElementRequestBody")}) + public class DeleteRequestBody extends DeleteOptions {} + """, + "frameworks/x/src/main/java/rest/DeleteElementRequestBody.java": + "public class DeleteElementRequestBody extends DeleteRequestBody {}", + "frameworks/x/src/main/java/rest/DeleteRelationshipRequestBody.java": + "public class DeleteRelationshipRequestBody extends DeleteRequestBody {}", + "frameworks/x/src/main/java/rest/ResultsRequestBody.java": + "public class ResultsRequestBody extends QueryOptions {}", + "frameworks/x/src/main/java/rest/FilterRequestBody.java": + "public class FilterRequestBody extends QueryOptions {}", + "frameworks/x/src/main/java/properties/PersonRoleAppointmentProperties.java": + "public class PersonRoleAppointmentProperties extends RelationshipBeanProperties {}", + "view/actor-manager-spring/src/main/java/server/spring/ActorManagerResource.java": """ + @RequestMapping("/servers/{serverName}/api/open-metadata/{urlMarker}") + public class ActorManagerResource { + @PostMapping(path = "/actor-roles/{itProfileRoleGUID}/it-profile-role-appointments/{itProfileGUID}/detach") + public VoidResponse detachITProfileRoleFromProfile(@PathVariable String serverName, + @RequestBody (required = false) + DeleteRelationshipRequestBody requestBody) { return null; } + @PostMapping(path = {"/actor-roles/{actorRoleGUID}/delete","/solution-roles/{actorRoleGUID}/delete"}) + public VoidResponse deleteActorRole(@RequestBody (required = false) DeleteElementRequestBody requestBody) { return null; } + @PostMapping(path = "/assets/{guid}/endpoints/retrieve") + public VoidResponse getEndpoints(@RequestBody ResultsRequestBody requestBody) { return null; } + } + """, +} + + +def _truth(tmp_path): + impl = tmp_path / "open-metadata-implementation" + for rel, src in FILES.items(): + f = impl / rel + f.parent.mkdir(parents=True, exist_ok=True) + f.write_text(textwrap.dedent(src)) + return omvs_audit.load_java_truth(str(tmp_path)) + + +def test_generic_url_marker_endpoints_resolve_for_any_service(tmp_path): + java = _truth(tmp_path) + assert java.lookup("POST", "/actor-manager/actor-roles/{}/it-profile-role-appointments/{}/detach") == { + "DeleteRelationshipRequestBody"} + # both paths of a multi-path mapping are indexed + assert java.lookup("POST", "/actor-manager/solution-roles/{}/delete") == {"DeleteElementRequestBody"} + assert java.lookup("POST", "/actor-manager/no-such/{}/thing") is None + + +def test_sibling_under_polymorphic_base_is_rejected_elsewhere_class_is_ignored(tmp_path): + java = _truth(tmp_path) + # the live 400 from ISSUE-163: DeleteElementRequestBody where DeleteRelationshipRequestBody is declared + assert java.rejects("DeleteRelationshipRequestBody", "DeleteElementRequestBody") + assert not java.rejects("DeleteRelationshipRequestBody", "DeleteRelationshipRequestBody") + # ResultsRequestBody has no @JsonTypeInfo: "class" is ignored, so not a rejection + assert not java.accepts("ResultsRequestBody", "FilterRequestBody") + assert not java.rejects("ResultsRequestBody", "FilterRequestBody") + + +def test_known_classes_and_class_constant_scan(tmp_path): + java = _truth(tmp_path) + assert "PersonRoleAppointmentProperties" in java.known_classes + py = tmp_path / "client.py" + py.write_text('def f():\n """Uses PersonAppointmentProperties."""\n' + ' return ["PersonAppointmentProperties"], {"class": "DeleteElementRequestBody"}\n') + found = {name for _, name in omvs_audit.class_constants(str(py))} + # docstring prose is not a bare class-name constant + assert found == {"PersonAppointmentProperties", "DeleteElementRequestBody"} + assert "PersonAppointmentProperties" not in java.known_classes From f3ea6711b36b7d84c094979d182f8a2c4f152dee Mon Sep 17 00:00:00 2001 From: Dan Wolfson Date: Fri, 23 Oct 2026 14:28:32 -0400 Subject: [PATCH 03/17] Document the omvs_audit Java sweep findings (ISSUE-164); correct ISSUE-163's audit note Signed-off-by: Dan Wolfson --- PYEGERIA_ISSUES.md | 57 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 56 insertions(+), 1 deletion(-) diff --git a/PYEGERIA_ISSUES.md b/PYEGERIA_ISSUES.md index 1f44cd15..18c954a8 100644 --- a/PYEGERIA_ISSUES.md +++ b/PYEGERIA_ISSUES.md @@ -286,6 +286,56 @@ they're holding real port materialization until this exists server-side --- +### ISSUE-164: `omvs_audit` now checks against Egeria's Java source. The first sweep finds 6 always-400 calls, 4 more 400s with a body, and 24 endpoints with no Spring mapping + +**Layer:** Pyegeria (+ Dr.Egeria, CLI) · **Status:** audit improved on branch `fix/role-appointment-class-names`; the findings below are **open, not fixed** (triaged, awaiting the owner's go) · **Found:** 2026-10-23, sweep requested by the owner before the next release, after ISSUE-163. + +**Audit changes** (`scripts/omvs_audit.py`, docs in CLAUDE.md and `scripts/README.md`, tests in `tests/micro-tests/test_omvs_audit_java_truth.py`): +- **RENAMED matches** now get the verb/body checks too; they used to be skipped. +- **Java ground truth** (`--java-dir`, `$EGERIA_JAVA_DIR`, default `~/localGit/egeria-v6/egeria`; `--no-java` skips it). The audit indexes 1,587 Spring endpoints → `@RequestBody` type. + - **JAVA-BODY REJECTED** (defect): fires only when the declared type sits under a `@JsonTypeInfo(property="class")` base. Among request bodies that's only `DeleteRequestBody` → `Delete{Element,Relationship,Classification}RequestBody`, where a sibling class is a 400. + - **CLASS-IGNORED** (review): everywhere else Egeria's DTOs are plain `@JsonIgnoreProperties(ignoreUnknown=true)`, so `"class"` is ignored and fields map by name. + - **UNKNOWN-CLASS** (defect): any `"…Properties"`/`"…RequestBody"` string in `pyegeria/omvs`, `md_processing` or `commands` that is no Egeria class and no pyegeria class. The properties bases (`RelationshipBeanProperties`, `ClassificationBeanProperties`, …) are polymorphic, so an unknown name there is a 400. + - **JAVA-MISSING** (review): an SDK verb+path with no Spring mapping. + - **HTTP-vs-JAVA** (info): 21 `.http` entries name a body class Java doesn't accept. When Java accepts what the SDK sends, the `.http` BODY mismatch is no longer counted. +- **Regression check:** on the pre-ISSUE-163 code the audit flags all eight ISSUE-163 bugs: 5 REJECTED and 3 UNKNOWN-CLASS, the detaches via the RENAMED path. + +**Findings, current branch** (`python scripts/omvs_audit.py --quiet`: java_ok=951, rejected=4, class_ignored=25, unknown_class=16, java_missing=24, http_vs_java=21): + +*Always fails: the SDK always sends this, so every call is a 400 or a client-side ValidationError.* + +| Where | What | Fix (from Java) | +|---|---|---| +| `glossary_manager.remove_is_abstract_concept` | element helper always sends `DeleteElementRequestBody`; endpoint takes `DeleteClassificationRequestBody` | `_async_delete_classification_request` | +| `runtime_manager.detach_cohort_member` | `_async_metadata_source_body_request` always sends `{"class":"MetadataSourceRequestBody"}`; endpoint takes `DeleteRelationshipRequestBody` | `_async_delete_relationship_request` | +| Dr.Egeria `Retention` classify (`md_processing/v2/curation.py:136`, `processors.py:237`) | body `class` is `RetentionClassificationProperties` | `RetentionProperties` | +| CLI todo status update (`commands/my/todo_actions.py:201`) | typo `UpdataElementRequestBody`; pyegeria's Literal validator rejects it before sending | `UpdateElementRequestBody` | +| `data_discovery` analysis-report methods (8) | `analysis-reports/...` has no mapping in `DataDiscoveryResource`; class `AnalysisReportProperties` doesn't exist | confirm whether Egeria still has the endpoint | +| `classification_explorer.add/remove_element_to/from_subject_area` | `elements/{g}/subject-area-member` has no mapping; class `SubjectAreaMemberProperties` doesn't exist | confirm | + +*Fails only when a no-body default or a docstring-style body is used. With the caller's own correct body these work.* + +| Where | What | Fix | +|---|---|---| +| `feedback_manager.remove_search_keyword` | relationship body to an element `/remove` | `_async_delete_element_request` | +| `glossary_manager.remove_activity_description` | relationship body; endpoint takes `DeleteClassificationRequestBody` | `_async_delete_classification_request` | +| `actor_manager.add_security_group_membership` default class | `SecurityGroupMembershipProperties` | `SecurityListMembershipProperties` | +| `classification_explorer.set_retention_classification` default class | `RetentionClassificationProperties` | `RetentionProperties` | +| `glossary_manager.add_is_data_value` default class | `DataValueProperties` | `DataValueMeaningProperties` | +| `solution_architect.create/update_solution_role` default class | `SolutionRoleProperties` | `SolutionActorRoleProperties` (extends `ActorRoleProperties`; the endpoint is `createActorRole`) | +| `data_designer.link_data_classification_to_data_field` | `data-classifications/.../data-fields/{g}/attach` has no mapping; `DataFieldClassificationProperties` doesn't exist. The create/update/detach data-classification methods are also unmapped | confirm | +| `GOV_DEF_PROPERTIES_LIST` (`governance_officer.py:40`) | `GovernanceProcessingPurposeProperties` | `DataProcessingPurposeProperties`; probably one of ISSUE-154's renames | + +*Harmless on the wire: an unknown top-level `class` that Egeria ignores.* `GovernanceActionRequestBody` (`automated_curation.initiate_engine_action`), `IntegrationServiceRequestBody` (`full_omag_server_config`), `AnyTimeRequestBody` (`metadata_expert.get_relationship_by_guid`, three Solution Architect retrieves) and `FindProperties` (`classification_explorer.retrieve_instance_for_guid`). Worth renaming for clarity only. + +*JAVA-MISSING, other entries to confirm.* `cancel_governance_action_process`, `get_search_keyword_by_guid`, `get_approved_engine_actions` (GET), `find_governance_action_processes`, `get_gov_def_in_context`, `get_platform_organization`, `new_subscription_type`, and `subject_area.link/detach_subject_area_hierarchy`. The last calls `collections/{g}/collection-hierarchies`, while Java has `subject-areas/{g}/subject-area-hierarchies/{g}/attach|detach`, so it is likely a real 404. `get_classified_elements_by` (`by-{}`) is a static-analysis false positive. + +*CLASS-IGNORED (25), review only.* Mostly Filter-vs-Results and MetadataSource-vs-DeleteClassification pairs that are harmless. Worth a field-level look: `metadata_expert.create_metadata_element_from_template` (Java `OpenMetadataTemplateRequestBody`) and `metadata_expert.classify_metadata_element` (Java `NewOpenMetadataClassificationRequestBody`). Both Java classes use typed `ElementProperties`, so the flat SDK shape may silently lose fields; see CLAUDE.md's note on `MetadataExpert` create. Also `project_manager.create_project_task` (Java `NewAttachmentRequestBody`). + +**Still not checked:** field-level drops on the CLASS-IGNORED pairs (Pydantic fields vs Java getters), which would be the next audit step. + +--- + ### ISSUE-163: Actor Manager role-appointment links send non-existent `*AppointmentProperties` classes, and five delete/detach methods send the wrong request-body class **Fixed 2026-10-23 on branch `fix/role-appointment-class-names` (not yet released).** **Live on quickstart 9443** (two runs, throwaway `pyeg-test::role-appt::` elements, all read back as gone by GUID afterwards, and a find on the prefix returned nothing). The old person-link class was refused with a 400. All three fixed links, called with no body, read back the relationship to the profile. The old IT detach body was refused with a 400; the fixed detach made the link unreadable. The fixed `delete_actor_role` deleted its role. @@ -316,7 +366,12 @@ POST .../actor-manager/actor-roles/a6e578d4-…/it-profile-role-appointments/91c **Fix:** corrected the three `prop` lists; switched the two detaches to `_async_delete_relationship_request`, `delete_actor_role` and `delete_contribution_record` to `_async_delete_element_request`, and `remove_is_context_definition` to `_async_delete_classification_request`; aligned the type hints and docstrings; corrected the Dr.Egeria method name. Tests: `tests/micro-tests/test_role_appointment_class_names.py` (11 tests; 9 of them fail on the old code; the two that pass cover the person and team detaches, which were already correct). -**Checked and left alone:** `MetadataExpert.delete_related_elements` sends `DeleteRelationshipRequestBody`, while Egeria declares `OpenMetadataDeleteRequestBody`. That's deliberate (ISSUE-63): the Egeria class can't carry `deleteMethod`. Decertify and unlicense (`GovernanceOfficerResource`: `DeleteRelationshipRequestBody`) are already correct. **Follow-up:** `scripts/omvs_audit.py --service actor-manager` on the old code reported 59 OK and 1 unrelated mismatch, so its BODY check flagged none of these. It takes the first `*RequestBody` identifier in the method, which is the signature annotation, so a wrong helper with a matching type hint looks clean. Comparing the helper called against the `.http` body class would catch this class of bug. +**Checked and left alone:** `MetadataExpert.delete_related_elements` sends `DeleteRelationshipRequestBody`, while Egeria declares `OpenMetadataDeleteRequestBody`. That's deliberate (ISSUE-63): the Egeria class can't carry `deleteMethod`. Decertify and unlicense (`GovernanceOfficerResource`: `DeleteRelationshipRequestBody`) are already correct. **Why the audit missed them:** `scripts/omvs_audit.py --service actor-manager` on the old code reported 59 OK and 1 unrelated mismatch. Three blind spots, all fixed in ISSUE-164: +- Both detaches matched only by verb+path (RENAMED), and RENAMED matches skipped the body check. +- Nothing compared the inner `properties` class, which is where the three link bugs were. +- For `delete_actor_role`, `delete_contribution_record` and `is-context-definition`, the `.http` file named the same wrong class the SDK sent. + +The improved audit flags all eight on the pre-fix code. --- From 42ef8b56c9405bca22ac9c3d9e1f3df2d738f82c Mon Sep 17 00:00:00 2001 From: Dan Wolfson Date: Fri, 23 Oct 2026 14:48:45 -0400 Subject: [PATCH 04/17] omvs_audit: field-level checks for silently dropped request fields (ISSUE-164) - FIELD-DROP: a method passes an argument whose JSON key the server-side class (with ancestors) does not declare; Egeria's ignoreUnknown=true discards it without error. Helper argument -> JSON key maps are read from each helper's dict literals, following local aliases; a parameter counts as dropped only if every key it can fill is dropped. - CLIENT-DROP: a dict naming a pyegeria model via "class" uses a key the model reads neither as alias nor field name, so Pydantic drops it before sending. - MODEL-FIELD gaps (review): model fields the Java class lacks, per pair, with the methods using it. - Java field index honours @JsonProperty, skips statics, and excludes adapters/ (connector DTOs reuse API class names). Signed-off-by: Dan Wolfson --- CLAUDE.md | 18 +- scripts/README.md | 3 +- scripts/omvs_audit.py | 223 +++++++++++++++++- .../micro-tests/test_omvs_audit_java_truth.py | 48 ++++ 4 files changed, 285 insertions(+), 7 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 5e0d612d..d549593d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -201,7 +201,23 @@ after a real defect slipped through an earlier, weaker version of this script: `"…RequestBody"` string in `pyegeria/omvs` that isn't an Egeria Java class. The properties bases are polymorphic, so sending one is a 400; this is what caught `PersonAppointmentProperties`. **JAVA-MISSING** (review) lists SDK - endpoints with no Spring mapping. + endpoints with no Spring mapping, meaning no URL, not a missing class. +- **Field-level checks** — these catch the "call succeeds but silently ignores + an option" bugs that no class check can see. + - **FIELD-DROP** (defect): a method passes an argument whose JSON key the + server-side class (and its ancestors) doesn't declare. The helper-argument + → JSON-key map is read from each helper's dict literals, following simple + local aliases, so a parameter counts as dropped only if *every* key it can + fill is dropped. + - **CLIENT-DROP** (defect): a dict in our own code names a pyegeria model + via `"class"` but uses a key that model reads neither as alias nor as field + name, so Pydantic discards it before sending (e.g. `"zoneFilter"` vs + `governanceZoneFilter`). + - **MODEL-FIELD gaps** (review): pyegeria model fields the Java class + lacks, with the methods that use each pair. A near-miss spelling there + (`anchorScopeGuids` vs `anchorScopeGUIDs`) is a bug. + - `adapters/` is excluded from the Java index, because connector-internal + DTOs reuse API class names. URL extraction is AST-based, not regex — multi-line f-strings silently produced empty paths under the regex version, which made whole modules look clean. diff --git a/scripts/README.md b/scripts/README.md index 66bfda58..d1d3530c 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -14,7 +14,8 @@ package or the pytest suite. When an Egeria Java checkout is present (`--java-dir`, `$EGERIA_JAVA_DIR`, default `~/localGit/egeria-v6/egeria`) it also checks against the Spring `@RequestBody` types: JAVA-BODY REJECTED / CLASS-IGNORED, UNKNOWN-CLASS, - JAVA-MISSING and HTTP-vs-JAVA (ISSUE-163). `--no-java` skips them. + JAVA-MISSING and HTTP-vs-JAVA, plus field-level FIELD-DROP / CLIENT-DROP / + MODEL-FIELD gaps (ISSUE-163/164). `--no-java` skips them. `python scripts/omvs_audit.py [--report PATH] [--service NAME] [--quiet]`; exits 1 on any confirmed defect, so it can gate CI. - `safe_git_sync.sh` — stashes all local changes (including untracked), diff --git a/scripts/omvs_audit.py b/scripts/omvs_audit.py index 90ab1feb..ce09c73d 100644 --- a/scripts/omvs_audit.py +++ b/scripts/omvs_audit.py @@ -32,6 +32,13 @@ polymorphic properties base this is a 400) * JAVA-MISSING - no Spring mapping for the SDK's verb+path (review) * HTTP-vs-JAVA - the .http body class disagrees with Java (info) + * FIELD-DROP - the method passes an argument whose JSON key the + server-side class does not declare; Egeria's + ignoreUnknown=true discards it, no error (defect) + * CLIENT-DROP - a dict naming a pyegeria model uses a key the model + reads neither as alias nor field name; Pydantic + discards it before sending (defect) + * MODEL-FIELD gaps - pyegeria model fields the Java class lacks (review) Usage ----- @@ -46,7 +53,7 @@ verification rather than failing the run. Exit status is 1 when any confirmed defect (verb/path/body mismatch, lint, -JAVA-BODY REJECTED or UNKNOWN-CLASS) is found, so this can gate CI. +JAVA-BODY REJECTED, UNKNOWN-CLASS, FIELD-DROP or CLIENT-DROP) is found, so this can gate CI. Note: ``pyegeria/http-clients/`` is gitignored. It must be present locally or the audit cannot run - the script fails loudly rather than reporting a clean @@ -168,6 +175,8 @@ # their own request, so they must not be mistaken for the method's own verb. LOOKUP_HELPER_RE = re.compile(r"get_guid__|__async_get_guid") +# helper -> (ordered params, {param: JSON key}); filled by audit() before parsing. +HELPER_FIELDS: dict[str, tuple[list[str], dict[str, set[str]]]] = {} # Known-duplicate endpoints: same verb+path by design, differing only by body. EXPECTED_DUPLICATE_PATHS = { "POST platform-services/server-platform/servers/{}/instance", @@ -437,6 +446,9 @@ def discover_helper_bodies(path: str) -> dict[str, frozenset[str]]: CLASS_MAPPING_RE = re.compile(r"@RequestMapping\s*\(") REQUEST_BODY_RE = re.compile(r"@RequestBody\s*(?:\([^)]*\))?\s*([A-Z]\w*)") EXTENDS_RE = re.compile(r"\bclass\s+(\w+)(?:<[^>]*>)?\s+extends\s+(\w+)") +FIELD_RE = re.compile(r'(?:@JsonProperty\s*\(\s*(?:value\s*=\s*)?"(\w+)"\s*\)\s*)?' + r'^[ \t]*(?:private|protected|public)\s+(?!static\b)(?:final\s+)?' + r'[\w.<>\[\],? ]+?\s+(\w+)\s*(?:=[^;]*)?;', re.M) # String constants in the SDK that name an Egeria Java class. CLASS_CONST_RE = re.compile(r"^[A-Z][A-Za-z0-9]*(?:Properties|RequestBody)$") @@ -470,6 +482,19 @@ class JavaTruth: # Everywhere else "class" is an ignored unknown property and fields map # by name, so a wrong class there is harmless unless fields are dropped. polymorphic: set[str] = field(default_factory=set) + # class -> JSON property names declared directly on it (instance fields, + # honouring @JsonProperty renames). Inherited ones come via json_fields(). + fields: dict[str, set[str]] = field(default_factory=dict) + + def json_fields(self, cls: str) -> set[str] | None: + """Every JSON property ``cls`` deserialises, or None if its own source + was not parsed (then nothing can be said about dropped fields).""" + if cls not in self.fields: + return None + out: set[str] = set() + for a in self.ancestors(cls): + out |= self.fields.get(a, set()) + return out def lookup(self, verb: str, path: str) -> set[str | None] | None: hit = self.endpoints.get((verb, path)) @@ -510,6 +535,7 @@ def load_java_truth(java_dir: str) -> JavaTruth | None: parents: dict[str, str] = {} known: set[str] = set() polymorphic: set[str] = set() + fields: dict[str, set[str]] = {} for root, _dirs, files in os.walk(impl): if {"test", "build", "target"} & set(os.path.relpath(root, impl).split(os.sep)): continue @@ -522,11 +548,19 @@ def load_java_truth(java_dir: str) -> JavaTruth | None: if not (stem.endswith(("Properties", "RequestBody", "Options", "Resource")) or "properties" in root or "rest" in root): continue + # Connector-internal DTOs reuse API class names (the Unity Catalog + # adapter has its own UpdateElementRequestBody); never let them + # shadow the REST API's classes. + if "adapters" in os.path.relpath(root, impl).split(os.sep): + continue text = open(os.path.join(root, fname), encoding="utf-8", errors="ignore").read() for child, parent in EXTENDS_RE.findall(text): parents.setdefault(child, parent) if "@JsonTypeInfo" in text and 'property = "class"' in text.replace("property=", "property = "): polymorphic.add(stem) + if stem.endswith(("Properties", "RequestBody", "Options")) and f"class {stem}" in text: + fields[stem] = {renamed or name for renamed, name in FIELD_RE.findall(text) + if name != "serialVersionUID"} if not (stem.endswith("Resource") and "spring" in root.split(os.sep)): continue cm = CLASS_MAPPING_RE.search(text) @@ -545,7 +579,7 @@ def load_java_truth(java_dir: str) -> JavaTruth | None: body = rb.group(1) if rb else None for p in paths: endpoints[(verb, canon_path(base + p))].add(body) - return JavaTruth(dict(endpoints), parents, known, polymorphic) + return JavaTruth(dict(endpoints), parents, known, polymorphic, fields) def class_constants(filepath: str) -> list[tuple[int, str]]: @@ -563,6 +597,108 @@ def class_constants(filepath: str) -> list[tuple[int, str]]: and id(n) not in fragments and CLASS_CONST_RE.match(n.value)] +def discover_helper_fields(path: str) -> dict[str, tuple[list[str], dict[str, set[str]]]]: + """For each ``_async_*_request`` helper that builds its own default body: + (ordered parameter names, {parameter: JSON keys it can fill}). + + A parameter can fill different keys in different branches + (``_async_find_request`` puts ``search_string`` in ``searchString`` or, with + property names, in ``propertyValue``), so the value is a set. + + Read from the helper's dict literals, e.g. ``"filter": filter_string`` + maps ``filter_string`` -> ``filter``. Lets the audit tell whether a method + actually sets a field the server will drop, not just whether it could. + """ + out: dict[str, tuple[list[str], dict[str, set[str]]]] = {} + if not os.path.exists(path): + return out + tree = ast.parse(open(path, encoding="utf-8").read()) + for fn in ast.walk(tree): + if not (isinstance(fn, (ast.FunctionDef, ast.AsyncFunctionDef)) + and fn.name.startswith("_async_") and fn.name.endswith("_request")): + continue + params = [a.arg for a in fn.args.args if a.arg != "self"] + mapping: dict[str, set[str]] = defaultdict(set) + # Locals derived from a parameter, e.g. ``effective_search_string = + # None if search_string == "*" else search_string`` -> search_string. + alias: dict[str, set[str]] = {} + for a in ast.walk(fn): + if (isinstance(a, ast.Assign) and len(a.targets) == 1 + and isinstance(a.targets[0], ast.Name) and a.targets[0].id not in params): + srcs = {n.id for n in ast.walk(a.value) if isinstance(n, ast.Name) and n.id in params} + if srcs: + alias.setdefault(a.targets[0].id, set()).update(srcs) + for d in ast.walk(fn): + if not isinstance(d, ast.Dict): + continue + for k, v in zip(d.keys, d.values): + if isinstance(k, ast.Constant) and isinstance(k.value, str) and k.value != "class": + for n in ast.walk(v): + if isinstance(n, ast.Name): + for src in ({n.id} if n.id in params else alias.get(n.id, set())): + mapping[src].add(k.value) + out[fn.name] = (params, dict(mapping)) + return out + + +def model_fields(name: str) -> set[str] | None: + """JSON keys a pyegeria request-body model can serialise (aliases), or + None if pyegeria has no model of that name.""" + try: + from pyegeria import models + except Exception: # pragma: no cover + return None + cls = getattr(models, name, None) + if cls is None or not hasattr(cls, "model_fields"): + return None + return {f.alias or k for k, f in cls.model_fields.items()} - {"class"} + + +def model_accepts(name: str) -> set[str] | None: + """Keys a pyegeria model reads from an input dict: aliases and, because + PyegeriaModel sets populate_by_name, field names. Anything else is + discarded by extra='ignore' before the request is sent.""" + try: + from pyegeria import models + except Exception: # pragma: no cover + return None + cls = getattr(models, name, None) + if cls is None or not hasattr(cls, "model_fields"): + return None + return {f.alias or k for k, f in cls.model_fields.items()} | set(cls.model_fields) | {"class"} + + +def client_drops(filepath: str) -> list[tuple[int, str, list[str]]]: + """(line, model, keys) for dict literals in code that name a pyegeria + request model via "class" but use keys that model does not read. + + These never leave the client: Pydantic validates the dict and silently + drops the unknown keys (e.g. "anchorScopeGuid" where the model reads + anchorScopeGUID). Docstring examples are strings, not dicts, so they are + not scanned.""" + try: + tree = ast.parse(open(filepath, encoding="utf-8").read()) + except SyntaxError: + return [] + out = [] + for d in ast.walk(tree): + if not isinstance(d, ast.Dict): + continue + keys = [k.value for k in d.keys if isinstance(k, ast.Constant) and isinstance(k.value, str)] + cls = next((v.value for k, v in zip(d.keys, d.values) + if isinstance(k, ast.Constant) and k.value == "class" + and isinstance(v, ast.Constant) and isinstance(v.value, str)), None) + if not cls or not cls.endswith("RequestBody"): + continue + accepted = model_accepts(cls) + if accepted is None: + continue + bad = sorted(k for k in keys if k not in accepted) + if bad: + out.append((d.lineno, cls, bad)) + return out + + @dataclass class PyMethod: name: str @@ -584,6 +720,10 @@ class PyMethod: # hint (the method posts whatever the caller passes). body_source: str | None = None lineno: int = 0 + # The body-building helper this method calls, and the helper parameters it + # passes explicitly with a value other than None. + helper: str | None = None + passed: frozenset[str] = frozenset() def _flatten(node: ast.AST, roots: dict[str, str]) -> str: @@ -717,6 +857,8 @@ def parse_py_file(filepath: str, helper_verbs: dict[str, str], raw_url = path = body_class = None direct_verb = helper_verb = helper_body = None + helper_name = None + helper_passed: frozenset[str] = frozenset() lint: list[str] = [] helper_bodies = helper_bodies or {} @@ -781,13 +923,21 @@ def parse_py_file(filepath: str, helper_verbs: dict[str, str], helper_verb = helper_verbs[fname] if helper_body is None and fname in helper_bodies: helper_body = helper_bodies[fname] + if helper_name is None: + helper_name = fname + order = HELPER_FIELDS.get(fname, ([], {}))[0] + given = {order[i]: a for i, a in enumerate(node.args) if i < len(order)} + given.update({k.arg: k.value for k in node.keywords if k.arg}) + helper_passed = frozenset( + k for k, v in given.items() + if not (isinstance(v, ast.Constant) and v.value is None)) methods[fn.name] = PyMethod( name=fn.name, verb=direct_verb or helper_verb, path=path, raw_url=raw_url, body_class=helper_body or body_class, lint=lint, all_paths=all_paths, body_source="helper" if helper_body else ("signature" if body_class else None), - lineno=fn.lineno, + lineno=fn.lineno, helper=helper_name, passed=helper_passed, ) for sm in sync_names: @@ -825,6 +975,7 @@ def audit(service_filter: str | None, report_path: str, quiet: bool, helper_verbs = discover_helper_verbs(SERVER_CLIENT) helper_bodies = discover_helper_bodies(SERVER_CLIENT) + HELPER_FIELDS.update(discover_helper_fields(SERVER_CLIENT)) java = load_java_truth(java_dir) if java_dir else None if java_dir and java is None: print(f" ! Egeria Java source not found at '{java_dir}'; skipping the " @@ -1035,6 +1186,41 @@ def java_confirms(d: PyMethod) -> bool: unknown_rows: list[str] = [] http_vs_java: list[str] = [] java_missing: list[str] = [] + field_rows: list[str] = [] + client_rows: list[str] = [] + model_gaps: dict[tuple[str, str], set[str]] = {} + gap_methods: dict[tuple[str, str], list[str]] = defaultdict(list) + + def check_fields(svc: str, m: str, d: PyMethod, declared: set[str]) -> None: + """Fields the SDK model can send that the class Egeria instantiates + does not declare are dropped silently (ignoreUnknown=true).""" + for sent in sorted(d.body_class): + sdk = model_fields(sent) + if not sdk: + continue + for decl in sorted(declared): + # Under a polymorphic base Jackson builds the class named by + # "class" when it is assignable; otherwise the declared one. + polymorphic = any(a in java.polymorphic for a in java.ancestors(decl)) + target = sent if (polymorphic and java.accepts(decl, sent)) else decl + known = java.json_fields(target) + if known is None: + continue + dropped = sdk - known + if not dropped: + continue + model_gaps.setdefault((sent, target), set()).update(dropped) + gap_methods[(sent, target)].append(f"{svc}.{m.removeprefix('_async_')}") + mapping = HELPER_FIELDS.get(d.helper or "", ([], {}))[1] + # A passed parameter is lost only if every key it can fill is dropped. + set_here = sorted({k for p in d.passed if mapping.get(p) and mapping[p] <= dropped + for k in mapping[p]}) + if set_here: + field_rows.append( + f"- `{svc}.py:{d.lineno}` `{m}` `{d.verb} {d.path}`: sets " + f"{', '.join(f'`{x}`' for x in set_here)}, which {target} does not have " + f"(sent as {sent})") + if java is not None: # Names of pyegeria's own Python classes (e.g. the SpecificationProperties # OMVS client, or FindPropertyNamesRequestBody, a local model) appear as @@ -1061,6 +1247,8 @@ def java_confirms(d: PyMethod) -> bool: continue if not declared: continue + if d.body_source == "helper": + check_fields(svc, m, d, declared) if any(java.accepts(decl, sent) for decl in declared for sent in d.body_class): counts["java_ok"] += 1 continue @@ -1084,6 +1272,9 @@ def java_confirms(d: PyMethod) -> bool: svc = os.path.basename(pf).replace(".py", "").replace("_omvs", "").replace("_", "-") if service_filter and service_filter != svc: continue + for lineno, cls, bad in client_drops(pf): + client_rows.append(f"- `{pf}:{lineno}` {cls}: " + f"{', '.join(f'`{k}`' for k in bad)} not read by the model") seen: set[str] = set() for lineno, cls in class_constants(pf): if cls in java.known_classes or cls in py_classes or cls in seen: @@ -1119,6 +1310,24 @@ def java_confirms(d: PyMethod) -> bool: lines.extend(java_soft or ["_none_"]) lines.append("\n### UNKNOWN-CLASS: SDK names a Properties/RequestBody class Egeria does not have\n") lines.extend(unknown_rows or ["_none_"]) + lines.append("\n### FIELD-DROP: the method sets a field the server-side class does not have\n") + lines.append("_Egeria's DTOs are `@JsonIgnoreProperties(ignoreUnknown=true)`, so the field is " + "discarded without an error and the call runs as if it had not been set._\n") + lines.extend(sorted(set(field_rows)) or ["_none_"]) + lines.append("\n### CLIENT-DROP: keys pyegeria's own model discards before sending\n") + lines.append("_A dict names a pyegeria request model via `class`, but a key matches neither a " + "field name nor its alias, so Pydantic (`extra='ignore'`) drops it silently._\n") + lines.extend(client_rows or ["_none_"]) + lines.append("\n### MODEL-FIELD gaps (review): pyegeria model fields the Java class lacks\n") + lines.append("_Dropped whenever a caller sets them. A near-miss spelling (`cascadeDelete` vs " + "`cascadedDelete`) is a bug; a pyegeria-only convenience field may be intended._\n") + for (sent, target), gap in sorted(model_gaps.items()): + users = sorted(set(gap_methods[(sent, target)])) + eg = ", ".join(users[:3]) + (f" (+{len(users) - 3} more)" if len(users) > 3 else "") + lines.append(f"- {sent} -> {target}: {', '.join(f'`{x}`' for x in sorted(gap))} \n" + f" used by {len(users)}: {eg}") + if not model_gaps: + lines.append("_none_") lines.append("\n### JAVA-MISSING (review): no Spring mapping for this verb+path\n") lines.append("_Removed or renamed in Egeria, or an endpoint the SDK guessed. Static path " "resolution can also miss a runtime-built segment, so confirm before changing._\n") @@ -1126,7 +1335,7 @@ def java_confirms(d: PyMethod) -> bool: lines.append("\n### HTTP-vs-JAVA (informational): the .http file disagrees with Java\n") lines.extend(http_vs_java or ["_none_"]) lines.append("") - defects += len(java_rows) + len(unknown_rows) + defects += len(java_rows) + len(unknown_rows) + len(set(field_rows)) + len(client_rows) counts["java_body"] = len(java_rows) counts["unknown_class"] = len(unknown_rows) @@ -1152,6 +1361,9 @@ def java_confirms(d: PyMethod) -> bool: f"| JAVA-BODY CLASS-IGNORED (different class, `class` ignored by Egeria) | {len(java_ignored)} |", f"| JAVA-BODY low confidence (type hint only) | {len(java_soft)} |", f"| UNKNOWN-CLASS | {len(unknown_rows)} |", + f"| FIELD-DROP (method sets a field Egeria discards) | {len(set(field_rows))} |", + f"| CLIENT-DROP (key pyegeria's model discards before sending) | {len(client_rows)} |", + f"| MODEL-FIELD gaps (model/Java class pairs; review) | {len(model_gaps)} |", f"| JAVA-MISSING (no Spring mapping; review) | {len(java_missing)} |", f"| HTTP-vs-JAVA disagreements (informational) | {len(http_vs_java)} |", ] @@ -1173,7 +1385,8 @@ def java_confirms(d: PyMethod) -> bool: print(f" java_ok={counts['java_ok']} java_rejected={len(java_rows)} " f"java_class_ignored={len(java_ignored)} " f"java_soft={len(java_soft)} unknown_class={len(unknown_rows)} " - f"java_missing={len(java_missing)} " + f"java_missing={len(java_missing)} field_drop={len(set(field_rows))} " + f"model_gaps={len(model_gaps)} client_drop={len(client_rows)} " f"http_vs_java={len(http_vs_java)}") if live_index is not None: print(f" live_ok={counts['live_ok']} live_stale={counts['live_stale']}") diff --git a/tests/micro-tests/test_omvs_audit_java_truth.py b/tests/micro-tests/test_omvs_audit_java_truth.py index cec11222..88ab8546 100644 --- a/tests/micro-tests/test_omvs_audit_java_truth.py +++ b/tests/micro-tests/test_omvs_audit_java_truth.py @@ -81,3 +81,51 @@ def test_known_classes_and_class_constant_scan(tmp_path): # docstring prose is not a bare class-name constant assert found == {"PersonAppointmentProperties", "DeleteElementRequestBody"} assert "PersonAppointmentProperties" not in java.known_classes + + +def test_inherited_json_fields_and_json_property_rename(tmp_path): + impl = tmp_path / "open-metadata-implementation" / "frameworks" / "src" / "main" / "java" / "rest" + impl.mkdir(parents=True) + (impl / "BaseOptions.java").write_text( + "public class BaseOptions extends Object {\n" + " private static final long serialVersionUID = 1L;\n" + " private boolean cascadedDelete = false;\n}\n") + (impl / "ThingRequestBody.java").write_text( + "public class ThingRequestBody extends BaseOptions {\n" + " @JsonProperty(\"class\")\n private String className;\n" + " private List anchorScopeGUIDs = null;\n}\n") + java = omvs_audit.load_java_truth(str(tmp_path)) + # inherited field included, serialVersionUID and statics excluded, rename honoured + assert java.json_fields("ThingRequestBody") == {"class", "anchorScopeGUIDs", "cascadedDelete"} + assert java.json_fields("NoSuchRequestBody") is None + + +def test_helper_field_map_follows_local_aliases(tmp_path): + src = tmp_path / "client.py" + src.write_text(textwrap.dedent(''' + class C: + async def _async_find_request(self, url, search_string="*", property_names=None): + effective = None if search_string == "*" else search_string + if property_names: + body = {"class": "FindPropertyNamesProperties", "propertyValue": search_string, + "propertyNames": property_names} + else: + body = {"class": "SearchStringRequestBody", "searchString": effective} + ''')) + params, mapping = omvs_audit.discover_helper_fields(str(src))["_async_find_request"] + assert params == ["url", "search_string", "property_names"] + # search_string reaches two keys (one via the `effective` local); a field is + # only "dropped" when every key it can fill is dropped + assert mapping["search_string"] == {"propertyValue", "searchString"} + assert mapping["property_names"] == {"propertyNames"} + + +def test_client_drops_flags_keys_the_pyegeria_model_does_not_read(tmp_path): + src = tmp_path / "client.py" + src.write_text('def f(x):\n' + ' return {"class": "SearchStringRequestBody", "searchString": x,\n' + ' "zoneFilter": x, "relationshipPageSize": 5, "start_from": 0}\n') + # zoneFilter / relationshipPageSize are near-misses of governanceZoneFilter / + # relationshipsPageSize; start_from is a field name, accepted via populate_by_name + assert omvs_audit.client_drops(str(src)) == [ + (2, "SearchStringRequestBody", ["relationshipPageSize", "zoneFilter"])] From abf71a5e82e86a31efae1ebeb553b0448dd1ce6a Mon Sep 17 00:00:00 2001 From: Dan Wolfson Date: Fri, 23 Oct 2026 14:48:45 -0400 Subject: [PATCH 05/17] ISSUE-164: record field-level sweep findings against Egeria main 4486d1a7f1 Signed-off-by: Dan Wolfson --- PYEGERIA_ISSUES.md | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/PYEGERIA_ISSUES.md b/PYEGERIA_ISSUES.md index 18c954a8..05dbb2d3 100644 --- a/PYEGERIA_ISSUES.md +++ b/PYEGERIA_ISSUES.md @@ -300,7 +300,7 @@ they're holding real port materialization until this exists server-side - **HTTP-vs-JAVA** (info): 21 `.http` entries name a body class Java doesn't accept. When Java accepts what the SDK sends, the `.http` BODY mismatch is no longer counted. - **Regression check:** on the pre-ISSUE-163 code the audit flags all eight ISSUE-163 bugs: 5 REJECTED and 3 UNKNOWN-CLASS, the detaches via the RENAMED path. -**Findings, current branch** (`python scripts/omvs_audit.py --quiet`: java_ok=951, rejected=4, class_ignored=25, unknown_class=16, java_missing=24, http_vs_java=21): +**Findings, current branch**, against Egeria `main` 4486d1a7f1 (2026-10-23). `python scripts/omvs_audit.py --quiet` reports java_ok=951, rejected=4, class_ignored=25, unknown_class=16, java_missing=22, field_drop=3, client_drop=13, model_gaps=27, http_vs_java=21. An earlier run against a 2026-10-06 checkout listed 24 JAVA-MISSING; `cancel_governance_action_process` and `get_approved_engine_actions` (ISSUE-153's Governance Engine paths) now resolve. *Always fails: the SDK always sends this, so every call is a 400 or a client-side ValidationError.* @@ -328,11 +328,34 @@ they're holding real port materialization until this exists server-side *Harmless on the wire: an unknown top-level `class` that Egeria ignores.* `GovernanceActionRequestBody` (`automated_curation.initiate_engine_action`), `IntegrationServiceRequestBody` (`full_omag_server_config`), `AnyTimeRequestBody` (`metadata_expert.get_relationship_by_guid`, three Solution Architect retrieves) and `FindProperties` (`classification_explorer.retrieve_instance_for_guid`). Worth renaming for clarity only. -*JAVA-MISSING, other entries to confirm.* `cancel_governance_action_process`, `get_search_keyword_by_guid`, `get_approved_engine_actions` (GET), `find_governance_action_processes`, `get_gov_def_in_context`, `get_platform_organization`, `new_subscription_type`, and `subject_area.link/detach_subject_area_hierarchy`. The last calls `collections/{g}/collection-hierarchies`, while Java has `subject-areas/{g}/subject-area-hierarchies/{g}/attach|detach`, so it is likely a real 404. `get_classified_elements_by` (`by-{}`) is a static-analysis false positive. +*JAVA-MISSING (meaning no such URL in Egeria; it is not about classes), other entries to confirm.* `get_search_keyword_by_guid`, `find_governance_action_processes`, `get_gov_def_in_context`, `get_platform_organization`, `new_subscription_type`, and `subject_area.link/detach_subject_area_hierarchy`. The last calls `collections/{g}/collection-hierarchies`, while Java has `subject-areas/{g}/subject-area-hierarchies/{g}/attach|detach`, so it is likely a real 404. `get_classified_elements_by` (`by-{}`) is a static-analysis false positive. *CLASS-IGNORED (25), review only.* Mostly Filter-vs-Results and MetadataSource-vs-DeleteClassification pairs that are harmless. Worth a field-level look: `metadata_expert.create_metadata_element_from_template` (Java `OpenMetadataTemplateRequestBody`) and `metadata_expert.classify_metadata_element` (Java `NewOpenMetadataClassificationRequestBody`). Both Java classes use typed `ElementProperties`, so the flat SDK shape may silently lose fields; see CLAUDE.md's note on `MetadataExpert` create. Also `project_manager.create_project_task` (Java `NewAttachmentRequestBody`). -**Still not checked:** field-level drops on the CLASS-IGNORED pairs (Pydantic fields vs Java getters), which would be the next audit step. +**Field-level findings.** These calls succeed but silently ignore an option. Egeria's DTOs are `ignoreUnknown=true`, and pyegeria's models are `extra='ignore'`, so a misnamed field disappears on either side with no error. + +*CLIENT-DROP: pyegeria's own model discards the key before sending.* + +| Where | Key sent | Model/Java field | Effect | +|---|---|---|---| +| `_async_find_request` (`_server_client.py:6543`, every `find_*` without property names) | `zoneFilter`, `relationshipPageSize`, `anchorScopeGuid`, `anchorTypeName` | `governanceZoneFilter`, `relationshipsPageSize`, `anchorScopeGUID`; `anchorTypeName` is not a field at all | `governance_zone_filter`, `relationship_page_size` and `anchor_scope_guid` never take effect on any find | +| My Profile to-do / meeting / review create (`my_profile.py:1314/1431/1523`) | `isOwnAnchor` | Java `ActionRequestBody` (via `AnchorOptions`) has it; **pyegeria's `ActionRequestBody` model lacks it**, and also `anchorGUID`, `anchorScopeGUIDs` and `initialClassifications`. It names Java's `newActionTargets` `actionTargetGUIDs` | not anchored as intended | +| Dr.Egeria `common_md_utils.py:263` | `endWith` | `endsWith` | the "ends with" search option is ignored | +| Dr.Egeria `common_md_utils.py:292/323/737`, `v2/schema_maker.py:28`, `v2/asset_maker.py:57`, CLI `glossary_actions.py:123` | `anchorScopeGUID(s)`, `governanceZoneFilter` on create/update bodies | the model reads `anchorScopeGuids` (see below); create/update bodies have no zone filter | anchor scope is dropped | +| Dr.Egeria `v2/asset_maker.py:57` | `parentRelationshipProperties`, `typeName` | Java `TemplateRequestBody` has `parentRelationshipProperties` (and `replacementClassifications`); **the pyegeria model lacks both** | parent-relationship properties are dropped | +| CLI `glossary_actions.py:178` | `elementProperties` | `properties` | update has no properties | +| `validate_delete_classification_request` default (`_server_client.py:6244`) | `cascadeDelete` | not on the model, and Java spells it `cascadedDelete` | ISSUE-132 | + +*Model spelling that Egeria discards. Confirmed: anchor scope cannot be set through pyegeria at all.* The `NewElementRequestBody`, `NewRelationshipRequestBody` and `TemplateRequestBody` models alias `anchor_scope_guids` as `anchorScopeGuids`, while Java's `MakeAnchorOptions` reads `anchorScopeGUIDs`. A dict using the documented `"anchorScopeGUIDs"` (as in `collection_manager.py`'s docstrings) is dropped client-side. `anchorScopeGuids` / `anchor_scope_guids` are sent, then dropped server-side. Checked by `model_validate` → `model_dump_json`. This affects 47 create, 157 link and 26 from-template methods; one alias fixes it. + +*FIELD-DROP: the method passes it, Egeria's class lacks it.* +- `get_tech_types_for_open_metadata_type(tech_name=...)` and `get_endpoints_for_asset(name=...)` send `filter` in a `FilterRequestBody`. The endpoints declare `ResultsRequestBody`, which has no `filter`, so they return everything unfiltered. +- `find_software_capabilities(property_names=...)` sends `propertyNames` where Java declares `SearchStringRequestBody`, so the property restriction is lost. + +*MODEL-FIELD gaps (review; 27 model→Java pairs, with users, in the report).* +- Most are benign. For example, `ResultsRequestBody → GetRequestBody` drops paging on 51 `get_*_by_guid` calls, where paging means nothing. +- Status-filter models carry both `activityStatus` and `activityStatusList`; only the `…List` form exists in Java. No current caller uses the singular form in a filter (My Profile's uses are inside the nested `…Properties`, where it is a real field). +- `UpdatePropertiesRequestBody.replaceProperties` (Java `UpdateOptions` has only `mergeUpdate`) is used by three Metadata Expert updates. --- From 3bb233c1d5c391ccc0c6db4f1f60b8c8275048c2 Mon Sep 17 00:00:00 2001 From: Dan Wolfson Date: Fri, 23 Oct 2026 14:54:43 -0400 Subject: [PATCH 06/17] Send anchor scope and find options under Egeria's field names (ISSUE-164) - NewElementRequestBody / NewRelationshipRequestBody / TemplateRequestBody: anchor_scope_guids now serialises as anchorScopeGUIDs (Egeria's MakeAnchorOptions name). The generated alias anchorScopeGuids was dropped by Egeria, and the documented anchorScopeGUIDs by extra='ignore', so anchor scope could never be set. All three spellings are still read. - _async_find_request (every find_*): send governanceZoneFilter, relationshipsPageSize, anchorScopeGUID and startFrom instead of zoneFilter, relationshipPageSize, anchorScopeGuid and start_from, which the model discarded; the property-names branch now sends the zone filter too. Removed anchorTypeName, which no Egeria options class has. 5 micro-tests, all fail on the old code. Signed-off-by: Dan Wolfson --- pyegeria/core/_server_client.py | 15 ++-- pyegeria/models/models.py | 31 ++++++- .../test_issue164_silent_field_drops.py | 81 +++++++++++++++++++ 3 files changed, 116 insertions(+), 11 deletions(-) create mode 100644 tests/micro-tests/test_issue164_silent_field_drops.py diff --git a/pyegeria/core/_server_client.py b/pyegeria/core/_server_client.py index 0f8e3b28..2b8f6d6a 100644 --- a/pyegeria/core/_server_client.py +++ b/pyegeria/core/_server_client.py @@ -6520,20 +6520,20 @@ async def _async_find_request(self, url: str, _type: str, _gen_output: Callable[ "limitResultsByStatus": limit_results_by_status, "sequencingOrder": sequencing_order, "sequencingProperty": sequencing_property, - "start_from": start_from, + "startFrom": start_from, "pageSize": page_size, "metadataElementSubtypeNames": metadata_element_subtypes, "skipRelationships": skip_relationships, "includeOnlyRelationships": include_only_relationships, - "relationshipPageSize": relationship_page_size, + "relationshipsPageSize": relationship_page_size, "skipClassifiedElements": skip_classified_elements, "includeOnlyClassifiedElements": include_only_classified_elements, "graphQueryDepth": graph_query_depth, "maxMermaidNodeCount": max_mermaid_node_count, "anchorGUID": anchor_guid, - "anchorTypeName": anchor_type_name, "anchorDomainName": anchor_domain, - "anchorScopeGuid": anchor_scope_guid, + "anchorScopeGUID": anchor_scope_guid, + "governanceZoneFilter": governance_zone_filter, **kwargs } if not metadata_element_type: @@ -6547,15 +6547,14 @@ async def _async_find_request(self, url: str, _type: str, _gen_output: Callable[ "endsWith": ends_with, "ignoreCase": ignore_case, "anchorGUID": anchor_guid, - "anchorTypeName": anchor_type_name, "anchorDomainName": anchor_domain, - "anchorScopeGuid": anchor_scope_guid, - "zoneFilter": governance_zone_filter, + "anchorScopeGUID": anchor_scope_guid, + "governanceZoneFilter": governance_zone_filter, "metadataElementTypeName": metadata_element_type, "metadataElementSubtypeNames": metadata_element_subtypes, "skipRelationships": skip_relationships, "includeOnlyRelationships": include_only_relationships, - "relationshipPageSize": relationship_page_size, + "relationshipsPageSize": relationship_page_size, "skipClassifiedElements": skip_classified_elements, "includeOnlyClassifiedElements": include_only_classified_elements, "graphQueryDepth": graph_query_depth, diff --git a/pyegeria/models/models.py b/pyegeria/models/models.py index 0dc4d88d..e57cbd04 100644 --- a/pyegeria/models/models.py +++ b/pyegeria/models/models.py @@ -278,7 +278,16 @@ class RequestBody(PyegeriaModel): class NewRelationshipRequestBody(RequestBody): class_: Annotated[Literal["NewRelationshipRequestBody"], Field(alias="class")] make_anchor: bool | None = False # do I make element at end2 an anchor to end1 - anchor_scope_guids: list[str] | None = None # in anchor classification for end2 + anchor_scope_guids: list[str] | None = Field( + # Egeria's MakeAnchorOptions reads "anchorScopeGUIDs"; the camel-case + # generator produced "anchorScopeGuids", which Egeria dropped (and the + # documented "anchorScopeGUIDs" was dropped here by extra='ignore'), so + # anchor scope could never be set (ISSUE-164). Old spellings still read. + # (in the anchor classification for end2) + default=None, + validation_alias=AliasChoices("anchorScopeGUIDs", "anchorScopeGuids", "anchor_scope_guids"), + serialization_alias="anchorScopeGUIDs", + ) properties: dict | None = None relationship_properties: dict | None = Field(default=None, alias="relationshipProperties") @@ -489,7 +498,15 @@ class NewElementRequestBody(RequestBody): class_: Annotated[Literal["NewElementRequestBody"], Field(alias="class")] anchor_guid: str | None = None is_own_anchor: bool | None = True - anchor_scope_guids: list[str] | None = None + anchor_scope_guids: list[str] | None = Field( + # Egeria's MakeAnchorOptions reads "anchorScopeGUIDs"; the camel-case + # generator produced "anchorScopeGuids", which Egeria dropped (and the + # documented "anchorScopeGUIDs" was dropped here by extra='ignore'), so + # anchor scope could never be set (ISSUE-164). Old spellings still read. + default=None, + validation_alias=AliasChoices("anchorScopeGUIDs", "anchorScopeGuids", "anchor_scope_guids"), + serialization_alias="anchorScopeGUIDs", + ) initial_classifications: Dict[str, InitialClassifications] | None = None parent_relationship_properties: dict | None = None properties: dict | None = None @@ -527,7 +544,15 @@ class TemplateRequestBody(RequestBody): class_: Annotated[Literal["TemplateRequestBody"], Field(alias="class")] anchor_guid: str | None = None is_own_anchor: bool | None = True - anchor_scope_guids: list[str] | None = None + anchor_scope_guids: list[str] | None = Field( + # Egeria's MakeAnchorOptions reads "anchorScopeGUIDs"; the camel-case + # generator produced "anchorScopeGuids", which Egeria dropped (and the + # documented "anchorScopeGUIDs" was dropped here by extra='ignore'), so + # anchor scope could never be set (ISSUE-164). Old spellings still read. + default=None, + validation_alias=AliasChoices("anchorScopeGUIDs", "anchorScopeGuids", "anchor_scope_guids"), + serialization_alias="anchorScopeGUIDs", + ) initial_classifications: dict | None = None # unsure parent_guid: str | None = None parent_relationship_type_name: str | None = None diff --git a/tests/micro-tests/test_issue164_silent_field_drops.py b/tests/micro-tests/test_issue164_silent_field_drops.py new file mode 100644 index 00000000..5351a99e --- /dev/null +++ b/tests/micro-tests/test_issue164_silent_field_drops.py @@ -0,0 +1,81 @@ +""" +ISSUE-164: request fields that were silently dropped, either by pyegeria's own +models (extra='ignore') or by Egeria (ignoreUnknown=true), so the call +succeeded but ignored the option. Each test captures the body actually sent. +No live server. +""" +import json + +import pytest + +from pyegeria.omvs.project_manager import ProjectManager + + +class _Response: + def json(self): + return {"class": "GUIDResponse", "guid": "g", "elements": [], "relatedHTTPCode": 200} + + +def _client(cls, calls, monkeypatch): + monkeypatch.setattr(cls, "check_connection", lambda self: "test") + client = cls("view-server", "https://localhost:1", "user", "pwd") + + async def fake_make_request(method, url, payload=None, *args, **kwargs): + calls.append({"url": url, "body": json.loads(payload) if isinstance(payload, str) else payload}) + return _Response() + + client._async_make_request = fake_make_request + return client + + +# --- group 1: anchor scope, and the find helper's keys ------------------------ + +@pytest.mark.parametrize("key", ["anchorScopeGUIDs", "anchorScopeGuids", "anchor_scope_guids"]) +def test_anchor_scope_is_sent_under_egerias_name_whichever_spelling_the_caller_uses(key): + # Egeria's MakeAnchorOptions reads anchorScopeGUIDs. The documented spelling + # used to be dropped client-side, and the model's own spelling server-side. + from pyegeria.models import NewElementRequestBody, NewRelationshipRequestBody, TemplateRequestBody + for model, extra in ((NewElementRequestBody, {}), (NewRelationshipRequestBody, {}), + (TemplateRequestBody, {"templateGUID": "t"})): + sent = model.model_validate({"class": model.__name__, key: ["scope-1"], **extra}).model_dump( + by_alias=True, exclude_none=True) + assert sent["anchorScopeGUIDs"] == ["scope-1"] + assert "anchorScopeGuids" not in sent + + +@pytest.mark.asyncio +async def test_find_forwards_zone_filter_relationship_page_size_and_anchor_scope(monkeypatch): + calls: list = [] + client = _client(ProjectManager, calls, monkeypatch) + + # find_projects forwards these through **kwargs to the shared find helper + await client._async_find_projects("clinical", governance_zone_filter=["quarantine"], + relationship_page_size=7, anchor_scope_guid="scope-1") + + body = calls[0]["body"] + assert body["class"] == "SearchStringRequestBody" + assert body["searchString"] == "clinical" + assert body["governanceZoneFilter"] == ["quarantine"] + assert body["relationshipsPageSize"] == 7 + assert body["anchorScopeGUID"] == "scope-1" + for dropped in ("zoneFilter", "relationshipPageSize", "anchorScopeGuid"): + assert dropped not in body + + +@pytest.mark.asyncio +async def test_find_with_property_names_also_sends_zone_filter_and_paging(monkeypatch): + calls: list = [] + client = _client(ProjectManager, calls, monkeypatch) + + await client._async_find_request( + "https://localhost:1/x", _type="Project", _gen_output=None, search_string="clinical", + property_names=["displayName"], governance_zone_filter=["quarantine"], + relationship_page_size=7, anchor_scope_guid="scope-1", start_from=5, page_size=10) + + body = calls[0]["body"] + assert body["class"] == "FindPropertyNamesProperties" + assert body["propertyValue"] == "clinical" + assert body["governanceZoneFilter"] == ["quarantine"] + assert body["relationshipsPageSize"] == 7 + assert body["anchorScopeGUID"] == "scope-1" + assert body["startFrom"] == 5 From 046b36e573f0c9bae4c6cc99093fb53c51ce0e8a Mon Sep 17 00:00:00 2001 From: Dan Wolfson Date: Fri, 23 Oct 2026 14:54:43 -0400 Subject: [PATCH 07/17] omvs_audit: read serialization/validation aliases; tag JAVA-MISSING with --live results Signed-off-by: Dan Wolfson --- scripts/omvs_audit.py | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/scripts/omvs_audit.py b/scripts/omvs_audit.py index ce09c73d..43a30b09 100644 --- a/scripts/omvs_audit.py +++ b/scripts/omvs_audit.py @@ -651,7 +651,7 @@ def model_fields(name: str) -> set[str] | None: cls = getattr(models, name, None) if cls is None or not hasattr(cls, "model_fields"): return None - return {f.alias or k for k, f in cls.model_fields.items()} - {"class"} + return {f.serialization_alias or f.alias or k for k, f in cls.model_fields.items()} - {"class"} def model_accepts(name: str) -> set[str] | None: @@ -665,7 +665,15 @@ def model_accepts(name: str) -> set[str] | None: cls = getattr(models, name, None) if cls is None or not hasattr(cls, "model_fields"): return None - return {f.alias or k for k, f in cls.model_fields.items()} | set(cls.model_fields) | {"class"} + keys = set(cls.model_fields) | {"class"} + for k, f in cls.model_fields.items(): + keys.add(f.alias or k) + va = f.validation_alias + if isinstance(va, str): + keys.add(va) + elif va is not None: # AliasChoices + keys |= {c for c in getattr(va, "choices", []) if isinstance(c, str)} + return keys def client_drops(filepath: str) -> list[tuple[int, str, list[str]]]: @@ -1242,7 +1250,14 @@ def check_fields(svc: str, m: str, d: PyMethod, declared: set[str]) -> None: mapped = True declared |= {b for b in hit if b} if not mapped and "{" not in d.path.split("/")[1] and d.path.count("/") > 1: - java_missing.append(f"- `{svc}.py:{d.lineno}` `{m}` `{d.verb} {d.path}`") + # With --live, the running platform's OpenAPI doc settles it: + # the Java scan can miss a mapping it cannot parse. + tag = "" + if live_index is not None: + tag = (" **[LIVE: served, so a Java-scan false positive]**" + if live_matches(d.verb, d.path, *live_index) + else " **[LIVE: absent, so a confirmed 404]**") + java_missing.append(f"- `{svc}.py:{d.lineno}` `{m}` `{d.verb} {d.path}`{tag}") if not d.body_class: continue if not declared: @@ -1330,7 +1345,8 @@ def check_fields(svc: str, m: str, d: PyMethod, declared: set[str]) -> None: lines.append("_none_") lines.append("\n### JAVA-MISSING (review): no Spring mapping for this verb+path\n") lines.append("_Removed or renamed in Egeria, or an endpoint the SDK guessed. Static path " - "resolution can also miss a runtime-built segment, so confirm before changing._\n") + "resolution can also miss a runtime-built segment, so confirm before changing. " + "Run with `--live` to check each one against the platform's OpenAPI doc._\n") lines.extend(java_missing or ["_none_"]) lines.append("\n### HTTP-vs-JAVA (informational): the .http file disagrees with Java\n") lines.extend(http_vs_java or ["_none_"]) From b8b4567b6201fec7834c7dcebe52b1f83edcb8c5 Mon Sep 17 00:00:00 2001 From: Dan Wolfson Date: Fri, 23 Oct 2026 15:00:56 -0400 Subject: [PATCH 08/17] Fix always-400 deletes and wrong body/properties class names (ISSUE-164) Each verified against Egeria main 4486d1a7f1's Spring resources and @JsonSubTypes registrations (registered names and aliases): - remove_is_abstract_concept, remove_activity_description: send DeleteClassificationRequestBody (was an element / relationship body). - detach_cohort_member: relationship delete; the metadata-source helper always sent MetadataSourceRequestBody, not a DeleteRequestBody subtype. - remove_search_keyword: element delete, as Egeria declares. - initiate_engine_action: InitiateEngineActionRequestBody with actionSourceGUIDs and startDate; requestSourceGUIDs and startTime were dropped, so a scheduled action started at once. - add/remove_is_data_value default DataValueMeaningProperties; solution roles SolutionActorRoleProperties; GOV_DEF_PROPERTIES_LIST DataProcessingPurposeProperties; CLI todo update UpdateElementRequestBody (typo failed client-side validation). Left alone on purpose: RetentionClassificationProperties and SecurityGroupMembershipProperties are registered aliases. Signed-off-by: Dan Wolfson --- commands/my/todo_actions.py | 4 +- md_processing/v2/engine_action.py | 4 +- pyegeria/core/_server_client.py | 2 +- pyegeria/omvs/automated_curation.py | 8 +- pyegeria/omvs/glossary_manager.py | 24 +++--- pyegeria/omvs/governance_officer.py | 2 +- pyegeria/omvs/runtime_manager.py | 8 +- pyegeria/omvs/solution_architect.py | 12 +-- .../test_automated_curation_engine_action.py | 5 +- .../test_issue164_silent_field_drops.py | 75 +++++++++++++++++++ 10 files changed, 111 insertions(+), 33 deletions(-) diff --git a/commands/my/todo_actions.py b/commands/my/todo_actions.py index bc907f8c..55222ec3 100644 --- a/commands/my/todo_actions.py +++ b/commands/my/todo_actions.py @@ -198,7 +198,7 @@ def change_todo_status(server, url, userid, password, timeout, todo_guid, new_st if new_status not in ACTIVITY_STATUS: raise PyegeriaInvalidParameterException(context={"reason":"Invalid activity status"}) body = { - "class": "UpdataElementRequestBody", + "class": "UpdateElementRequestBody", "properties": { "class": "ToDoProperties", "activityStatus": new_status @@ -234,7 +234,7 @@ def mark_todo_complete(server, url, userid, password, timeout, todo_guid): m_client.create_egeria_bearer_token() body = { - "class": "UpdataElementRequestBody", + "class": "UpdateElementRequestBody", "properties": { "class": "ToDoProperties", "activityStatus": "COMPLETED", diff --git a/md_processing/v2/engine_action.py b/md_processing/v2/engine_action.py index 83154224..29e65455 100644 --- a/md_processing/v2/engine_action.py +++ b/md_processing/v2/engine_action.py @@ -9,9 +9,9 @@ Both commands call pyegeria's `AutomatedCuration` client directly (`initiate_engine_action` / `cancel_engine_action`) rather than going through the generic AssetMaker/Referenceable create path: `initiate_engine_action`'s -request body (qualifiedName/domainIdentifier/requestSourceGUIDs/ +request body (qualifiedName/domainIdentifier/actionSourceGUIDs/ actionTargets/receivedGuards/requestType/requestParameters/processName/...) -is its own bespoke shape (`GovernanceActionRequestBody`), not a +is its own bespoke shape (`InitiateEngineActionRequestBody`), not a NewElementRequestBody-wrapped Properties object, and the element it creates is populated entirely server-side as the engine runs -- there is nothing to declare beyond the initiation request itself. `cancel_engine_action` takes diff --git a/pyegeria/core/_server_client.py b/pyegeria/core/_server_client.py index 2b8f6d6a..b319d73c 100644 --- a/pyegeria/core/_server_client.py +++ b/pyegeria/core/_server_client.py @@ -5652,7 +5652,7 @@ async def _async_remove_search_keyword( """ url = f"{self.command_root}classification-explorer/search-keywords/{keyword_guid}/remove" - await self._async_delete_relationship_request(url = url, body = None, cascade_delete = False) + await self._async_delete_element_request(url = url, body = None, cascade_delete = False) @dynamic_catch def remove_search_keyword( diff --git a/pyegeria/omvs/automated_curation.py b/pyegeria/omvs/automated_curation.py index 578b680d..63a89d09 100644 --- a/pyegeria/omvs/automated_curation.py +++ b/pyegeria/omvs/automated_curation.py @@ -3836,15 +3836,17 @@ async def _async_initiate_engine_action( url = f"{self.ref_curation_command_base}/governance-engines/{governance_engine_name}/engine-actions/initiate" body = { - "class": "GovernanceActionRequestBody", + "class": "InitiateEngineActionRequestBody", "qualifiedName": qualified_name + str(int(start_time.timestamp())), "domainIdentifier": domain_identifier, "displayName": display_name, "description": description, - "requestSourceGUIDs": request_source_guids, + # Egeria's InitiateEngineActionRequestBody names: requestSourceGUIDs and + # startTime were dropped silently, so the action started at once (ISSUE-164) + "actionSourceGUIDs": request_source_guids, "actionTargets": action_targets, "receivedGuards": received_guards, - "startTime": int(start_time.timestamp() * 1000), + "startDate": int(start_time.timestamp() * 1000), "requestType": request_type, "requestParameters": request_parameters, "processName": process_name, diff --git a/pyegeria/omvs/glossary_manager.py b/pyegeria/omvs/glossary_manager.py index 09ba8729..678bde28 100644 --- a/pyegeria/omvs/glossary_manager.py +++ b/pyegeria/omvs/glossary_manager.py @@ -1988,7 +1988,7 @@ def add_is_abstract_concept( @dynamic_catch async def _async_remove_is_abstract_concept( - self, term_guid: str, body: Optional[dict | DeleteElementRequestBody] = None, + self, term_guid: str, body: Optional[dict | DeleteClassificationRequestBody] = None, ) -> None: """Remove the abstract concept designation from the glossary term. Async Version. @@ -2030,12 +2030,12 @@ async def _async_remove_is_abstract_concept( f"{self.platform_url}/servers/{self.view_server}/api/open-metadata/glossary-manager/glossaries/" f"terms/{term_guid}/is-abstract-concept/delete" ) - await self._async_delete_element_request(url, body) + await self._async_delete_classification_request(url, body) logger.info(f"Removed AbstractConcept classification to {term_guid}") @dynamic_catch def remove_is_abstract_concept( - self, term_guid: str, body: Optional[dict | DeleteElementRequestBody] = None, + self, term_guid: str, body: Optional[dict | DeleteClassificationRequestBody] = None, ) -> None: """Remove the abstract concept designation from the glossary term. @@ -2301,7 +2301,7 @@ async def _async_add_is_data_value( "class" : "NewClassificationRequestBody", "effectiveTime" : {{@isoTimestamp}}, "properties" : { - "class" : "DataValueProperties" + "class" : "DataValueMeaningProperties" } } """ @@ -2315,11 +2315,11 @@ async def _async_add_is_data_value( "class": "NewClassificationRequestBody", "properties": { - "class": "DataValueProperties" + "class": "DataValueMeaningProperties" } } - await self._async_new_classification_request(url, "DataValueProperties", body) + await self._async_new_classification_request(url, "DataValueMeaningProperties", body) logger.info(f"Added DataValue classification to {term_guid}") @dynamic_catch @@ -2356,7 +2356,7 @@ def add_is_data_value( "class" : "NewClassificationRequestBody", "effectiveTime" : {{@isoTimestamp}}, "properties" : { - "class" : "DataValueProperties" + "class" : "DataValueMeaningProperties" } } """ @@ -2400,7 +2400,7 @@ async def _async_remove_is_data_value( "class" : "DeleteClassificationRequestBody", "effectiveTime" : {{@isoTimestamp}}, "properties" : { - "class" : "DataValueProperties" + "class" : "DataValueMeaningProperties" } } """ @@ -2446,7 +2446,7 @@ def remove_is_data_value( "class" : "DeleteClassificationRequestBody", "effectiveTime" : {{@isoTimestamp}}, "properties" : { - "class" : "DataValueProperties" + "class" : "DataValueMeaningProperties" } } """ @@ -2561,7 +2561,7 @@ def add_activity_description( @dynamic_catch async def _async_remove_activity_description( - self, term_guid: str, body: Optional[dict | DeleteRelationshipRequestBody] = None, + self, term_guid: str, body: Optional[dict | DeleteClassificationRequestBody] = None, ) -> None: """Remove the activity designation from the glossary term. Async Version. @@ -2603,12 +2603,12 @@ async def _async_remove_activity_description( f"{self.platform_url}/servers/{self.view_server}/api/open-metadata/glossary-manager/glossaries/" f"terms/{term_guid}/is-activity/remove" ) - await self._async_delete_relationship_request(url, body) + await self._async_delete_classification_request(url, body) logger.info(f"Removed ActivityDescription classification to {term_guid}") @dynamic_catch def remove_activity_description( - self, term_guid: str, body: Optional[dict | DeleteRelationshipRequestBody] = None, + self, term_guid: str, body: Optional[dict | DeleteClassificationRequestBody] = None, ) -> None: """Remove the activity designation from the glossary term. diff --git a/pyegeria/omvs/governance_officer.py b/pyegeria/omvs/governance_officer.py index dd7750a8..58beedea 100644 --- a/pyegeria/omvs/governance_officer.py +++ b/pyegeria/omvs/governance_officer.py @@ -37,7 +37,7 @@ GOV_DEF_PROPERTIES_LIST = ["GovernanceDefinitionProperties", "GovernanceStrategyProperties", "RegulationProperties", "GovernanceControlProperties", "SecurityAccessControlProperties", "NamingStandardRuleProperties", "CertificationTypeProperties", "LicenseTypeProperties", "GovernanceApproachProperties", - "GovernanceProcessingPurposeProperties","RegulationProperties","GovernanceControlProperties", + "DataProcessingPurposeProperties","RegulationProperties","GovernanceControlProperties", "BusinessImperativeProperties", "RegulationArticleProperties", "ThreatProperties", "DataLensProperties","NotificationTypeProperties", "ServiceAccessControlProperties", "GovernanceZoneProperties", "GovernancePolicyProperties", "GovernancePrincipleProperties", diff --git a/pyegeria/omvs/runtime_manager.py b/pyegeria/omvs/runtime_manager.py index 7e68f6be..72d87ca6 100644 --- a/pyegeria/omvs/runtime_manager.py +++ b/pyegeria/omvs/runtime_manager.py @@ -3732,7 +3732,7 @@ def link_cohort_member(self, cohort_guid: str, cohort_member_guid: str, @dynamic_catch async def _async_detach_cohort_member(self, cohort_guid: str, cohort_member_guid: str, - body: Optional[dict | MetadataSourceRequestBody] = None) -> None: + body: Optional[dict | DeleteRelationshipRequestBody] = None) -> None: """ Detach a cohort member from a cohort. Async version. Parameters @@ -3741,7 +3741,7 @@ async def _async_detach_cohort_member(self, cohort_guid: str, cohort_member_guid The unique identifier of the cohort. cohort_member_guid: str The unique identifier of the cohort member. - body: dict | MetadataSourceRequestBody, optional, default = None + body: dict | DeleteRelationshipRequestBody, optional, default = None A structure representing the details of the request. Returns @@ -3768,12 +3768,12 @@ async def _async_detach_cohort_member(self, cohort_guid: str, cohort_member_guid ``` """ url = f"{self.runtime_command_root}/metadata-repository-cohorts/{cohort_guid}/cohort-members/{cohort_member_guid}/detach" - await self._async_metadata_source_body_request(url, body) + await self._async_delete_relationship_request(url, body) logger.info(f"Detached cohort member {cohort_member_guid} from cohort {cohort_guid}") @dynamic_catch def detach_cohort_member(self, cohort_guid: str, cohort_member_guid: str, - body: Optional[dict | MetadataSourceRequestBody] = None) -> None: + body: Optional[dict | DeleteRelationshipRequestBody] = None) -> None: """ Detach a cohort member from a cohort.""" loop = asyncio.get_event_loop() loop.run_until_complete(self._async_detach_cohort_member(cohort_guid, cohort_member_guid, body)) diff --git a/pyegeria/omvs/solution_architect.py b/pyegeria/omvs/solution_architect.py index ade48f87..80250314 100644 --- a/pyegeria/omvs/solution_architect.py +++ b/pyegeria/omvs/solution_architect.py @@ -5395,7 +5395,7 @@ async def _async_create_solution_role(self, body: dict | NewElementRequestBody) }, "parentAtEnd1": false, "properties": { - "class" : "SolutionRoleProperties", + "class" : "SolutionActorRoleProperties", "qualifiedName": "add unique name here", "name": "add short name here", "description": "add description here", @@ -5417,7 +5417,7 @@ async def _async_create_solution_role(self, body: dict | NewElementRequestBody) url = (f"{self.platform_url}/servers/{self.view_server}/api/open-metadata/solution-architect/" f"solution-roles") - return await self._async_create_element_body_request(url, ["SolutionRoleProperties"], body) + return await self._async_create_element_body_request(url, ["SolutionActorRoleProperties"], body) @dynamic_catch def create_solution_role(self, body: dict| NewElementRequestBody) -> str: @@ -5466,7 +5466,7 @@ def create_solution_role(self, body: dict| NewElementRequestBody) -> str: }, "parentAtEnd1": false, "properties": { - "class" : "SolutionRoleProperties", + "class" : "SolutionActorRoleProperties", "qualifiedName": "add unique name here", "name": "add short name here", "description": "add description here", @@ -5688,7 +5688,7 @@ async def _async_update_solution_role(self, guid: str, body: dict |UpdateElement { "class" : "UpdateElementRequestBody", "properties": { - "class" : "SolutionRoleProperties", + "class" : "SolutionActorRoleProperties", "qualifiedName": "add unique name here", "name": "add short name here", "description": "add description here", @@ -5712,7 +5712,7 @@ async def _async_update_solution_role(self, guid: str, body: dict |UpdateElement url = (f"{self.platform_url}/servers/{self.view_server}/api/open-metadata/solution-architect/" f"solution-roles/{guid}/update") - await self._async_update_element_body_request(url, ["SolutionRoleProperties"], body) + await self._async_update_element_body_request(url, ["SolutionActorRoleProperties"], body) @dynamic_catch def update_solution_role(self, guid: str, body: dict | UpdateElementRequestBody) -> None: @@ -5748,7 +5748,7 @@ def update_solution_role(self, guid: str, body: dict | UpdateElementRequestBody) { "class" : "UpdateElementRequestBody", "properties": { - "class" : "SolutionRoleProperties", + "class" : "SolutionActorRoleProperties", "qualifiedName": "add unique name here", "name": "add short name here", "description": "add description here", diff --git a/tests/micro-tests/test_automated_curation_engine_action.py b/tests/micro-tests/test_automated_curation_engine_action.py index 762867ef..84a38bc6 100644 --- a/tests/micro-tests/test_automated_curation_engine_action.py +++ b/tests/micro-tests/test_automated_curation_engine_action.py @@ -79,9 +79,10 @@ async def test_initiate_engine_action_body_shape(): ) body = cap["body"] - assert body["class"] == "GovernanceActionRequestBody" + # Egeria declares InitiateEngineActionRequestBody (ISSUE-164) + assert body["class"] == "InitiateEngineActionRequestBody" assert body["requestType"] == "survey-file-folder" - assert body["requestSourceGUIDs"] == ["src-guid"] + assert body["actionSourceGUIDs"] == ["src-guid"] assert body["actionTargets"] == [{"actionTargetName": "resource", "actionTargetGUID": "tgt-guid"}] assert body["requestParameters"] == {"key": "value"} # qualifiedName has a timestamp suffix appended -- just confirm the prefix survives diff --git a/tests/micro-tests/test_issue164_silent_field_drops.py b/tests/micro-tests/test_issue164_silent_field_drops.py index 5351a99e..50c8a176 100644 --- a/tests/micro-tests/test_issue164_silent_field_drops.py +++ b/tests/micro-tests/test_issue164_silent_field_drops.py @@ -79,3 +79,78 @@ async def test_find_with_property_names_also_sends_zone_filter_and_paging(monkey assert body["relationshipsPageSize"] == 7 assert body["anchorScopeGUID"] == "scope-1" assert body["startFrom"] == 5 + + +# --- group 2: always-400 calls and wrong class names -------------------------- + +@pytest.mark.asyncio +@pytest.mark.parametrize("method, args, expected", [ + # endpoint declares DeleteClassificationRequestBody (a sibling class is a 400) + ("_async_remove_is_abstract_concept", ("term-guid",), "DeleteClassificationRequestBody"), + ("_async_remove_activity_description", + ("term-guid", {"class": "DeleteClassificationRequestBody", "forLineage": True}), + "DeleteClassificationRequestBody"), + # search keywords are elements: the endpoint declares DeleteElementRequestBody + ("_async_remove_search_keyword", ("kw-guid",), "DeleteElementRequestBody"), +]) +async def test_glossary_and_keyword_removals_send_the_declared_delete_class(monkeypatch, method, args, expected): + from pyegeria.omvs.glossary_manager import GlossaryManager + calls: list = [] + client = _client(GlossaryManager, calls, monkeypatch) + await getattr(client, method)(*args) + assert calls[0]["body"]["class"] == expected + + +@pytest.mark.asyncio +async def test_detach_cohort_member_no_longer_sends_a_metadata_source_body(monkeypatch): + # The metadata-source helper always sent {"class": "MetadataSourceRequestBody"}, which + # is not a registered DeleteRequestBody subtype, so every call was a 400. + from pyegeria.omvs.runtime_manager import RuntimeManager + calls: list = [] + client = _client(RuntimeManager, calls, monkeypatch) + await client._async_detach_cohort_member("cohort-guid", "member-guid") + assert calls[0]["url"].endswith("/metadata-repository-cohorts/cohort-guid/cohort-members/member-guid/detach") + assert calls[0]["body"] is None + await client._async_detach_cohort_member("cohort-guid", "member-guid", + {"class": "DeleteRelationshipRequestBody", "forLineage": True}) + assert calls[1]["body"]["class"] == "DeleteRelationshipRequestBody" + + +@pytest.mark.asyncio +async def test_initiate_engine_action_uses_egerias_field_names(monkeypatch): + # InitiateEngineActionRequestBody has actionSourceGUIDs and startDate; the old + # requestSourceGUIDs / startTime were dropped, so a scheduled action started at once. + from datetime import datetime + from pyegeria.omvs.automated_curation import AutomatedCuration + calls: list = [] + client = _client(AutomatedCuration, calls, monkeypatch) + when = datetime(2030, 1, 2, 3, 4, 5) + await client._async_initiate_engine_action( + "engine", "qn::action", 0, "Action", "desc", ["source-guid"], [], [], start_time=when) + body = calls[0]["body"] + assert body["class"] == "InitiateEngineActionRequestBody" + assert body["actionSourceGUIDs"] == ["source-guid"] + assert body["startDate"] == int(when.timestamp() * 1000) + assert "requestSourceGUIDs" not in body and "startTime" not in body + + +@pytest.mark.asyncio +async def test_add_is_data_value_default_class_is_the_one_the_handler_accepts(monkeypatch): + # GlossaryManager's setTermAsDataValue checks `instanceof DataValueMeaningProperties`; + # DataValueProperties is no registered subtype (a 400 when no body is given). + from pyegeria.omvs.glossary_manager import GlossaryManager + calls: list = [] + client = _client(GlossaryManager, calls, monkeypatch) + await client._async_add_is_data_value("term-guid") + assert calls[0]["body"]["properties"]["class"] == "DataValueMeaningProperties" + + +def test_solution_role_and_governance_purpose_use_registered_class_names(): + from pyegeria.models import NewElementRequestBody + from pyegeria.omvs.governance_officer import GOV_DEF_PROPERTIES_LIST + # create_solution_role posts to createActorRole, which takes ActorRoleProperties subtypes + assert NewElementRequestBody.model_validate({ + "class": "NewElementRequestBody", + "properties": {"class": "SolutionActorRoleProperties", "qualifiedName": "q"}}) + assert "DataProcessingPurposeProperties" in GOV_DEF_PROPERTIES_LIST + assert "GovernanceProcessingPurposeProperties" not in GOV_DEF_PROPERTIES_LIST From a2db0266616f3326f19ef8703a3866e269f78b85 Mon Sep 17 00:00:00 2001 From: Dan Wolfson Date: Fri, 23 Oct 2026 15:00:57 -0400 Subject: [PATCH 09/17] omvs_audit: accept registered @JsonSubTypes names and aliases as valid classes Signed-off-by: Dan Wolfson --- scripts/omvs_audit.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/scripts/omvs_audit.py b/scripts/omvs_audit.py index 43a30b09..4626f86e 100644 --- a/scripts/omvs_audit.py +++ b/scripts/omvs_audit.py @@ -450,6 +450,11 @@ def discover_helper_bodies(path: str) -> dict[str, frozenset[str]]: r'^[ \t]*(?:private|protected|public)\s+(?!static\b)(?:final\s+)?' r'[\w.<>\[\],? ]+?\s+(\w+)\s*(?:=[^;]*)?;', re.M) # String constants in the SDK that name an Egeria Java class. +# Wire names Jackson accepts: @JsonSubTypes.Type(value = X.class, name = "N", +# names = {"Alias", ...}). The alias list keeps old names working, e.g. +# RetentionProperties is also registered as "RetentionClassificationProperties". +SUBTYPE_RE = re.compile(r'@JsonSubTypes\.Type\(\s*value\s*=\s*\w+\.class\s*,\s*name\s*=\s*"(\w+)"' + r'(?:\s*,\s*names\s*=\s*\{([^}]*)\})?') CLASS_CONST_RE = re.compile(r"^[A-Z][A-Za-z0-9]*(?:Properties|RequestBody)$") @@ -556,6 +561,9 @@ def load_java_truth(java_dir: str) -> JavaTruth | None: text = open(os.path.join(root, fname), encoding="utf-8", errors="ignore").read() for child, parent in EXTENDS_RE.findall(text): parents.setdefault(child, parent) + for name, aliases in SUBTYPE_RE.findall(text): + known.add(name) + known.update(re.findall(r'"(\w+)"', aliases)) if "@JsonTypeInfo" in text and 'property = "class"' in text.replace("property=", "property = "): polymorphic.add(stem) if stem.endswith(("Properties", "RequestBody", "Options")) and f"class {stem}" in text: From 13c17e3737f694a2950f30f1e195a16ddf92f306 Mon Sep 17 00:00:00 2001 From: Dan Wolfson Date: Fri, 23 Oct 2026 15:05:00 -0400 Subject: [PATCH 10/17] Add missing Egeria fields to request models; fix Dr.Egeria/CLI body keys (ISSUE-164) - ActionRequestBody: anchorGUID, isOwnAnchor, anchorScopeGUIDs, initialClassifications and newActionTargets (Egeria's names; the old actionTargetGUIDs key is still read). My Profile's isOwnAnchor on to-dos/meetings/reviews was being dropped client-side. - TemplateRequestBody: parentRelationshipProperties added; initial_classifications now serialises as replacementClassifications, Egeria's template name for it (the old key is still read). - Dr.Egeria set_find_body: endsWith (was endWith, ignored); set_create_body: anchorScopeGUIDs list (was a singular key, dropped); create/update bodies no longer send governanceZoneFilter, which Egeria's create/update bodies lack. - CLI glossary actions: anchorScopeGUIDs list; update sends properties (was elementProperties, so the update carried no properties). Signed-off-by: Dan Wolfson --- commands/cat/glossary_actions.py | 4 +- .../md_processing_utils/common_md_utils.py | 8 ++-- pyegeria/models/models.py | 28 ++++++++++- .../test_issue164_silent_field_drops.py | 48 +++++++++++++++++++ 4 files changed, 80 insertions(+), 8 deletions(-) diff --git a/commands/cat/glossary_actions.py b/commands/cat/glossary_actions.py index a8245ed0..e165036f 100644 --- a/commands/cat/glossary_actions.py +++ b/commands/cat/glossary_actions.py @@ -124,7 +124,7 @@ def create_term(server, url, userid, password, glossary_name, term_name, summary "class": "NewElementRequestBody", "parentGUID" : glossary_guid, "parentRelationshipTypeName" : "CollectionMembership", - "anchorScopeGUID" : glossary_guid, + "anchorScopeGUIDs" : [glossary_guid], "isOwnAnchor": True, "parentAtEnd1": True, "properties": { @@ -178,7 +178,7 @@ def update_term(server, url, userid, password, term_name, summary, description, body = { "class": "UpdateElementRequestBody", "mergeUpdate": True, - "elementProperties": { + "properties": { "class": "GlossaryTermProperties", "displayName": term_name, "summary": summary, diff --git a/md_processing/md_processing_utils/common_md_utils.py b/md_processing/md_processing_utils/common_md_utils.py index 6c76c8a8..eefcce94 100644 --- a/md_processing/md_processing_utils/common_md_utils.py +++ b/md_processing/md_processing_utils/common_md_utils.py @@ -264,7 +264,7 @@ def set_find_body(object_type: str, attributes: dict)->dict: "class": "SearchStringRequestBody", "searchString": search_string, "startsWith": attributes.get('Start With', {}).get('value', True), - "endWith": attributes.get('End With', {}).get('value', False), + "endsWith": attributes.get('End With', {}).get('value', False), "ignoreCase": attributes.get('Ignore Case', {}).get('value', False), "limitResultsByStatus": attributes.get('Limit Results By Status', {}).get('value', []), "startFrom": int(attributes.get('Start From', {}).get('value', 0)), @@ -294,7 +294,6 @@ def set_create_body(object_type: str, attributes: dict)->dict: "externalSourceGUID": attributes.get('External Source GUID', {}).get('guid', None), "externalSourceName": attributes.get('External Source Name', {}).get('value', None), "effectiveTime": attributes.get('Effective Time', {}).get('value', None), - "governanceZoneFilter" : attributes.get('Governance Zone Filter', {}).get('value', None), "forLineage": attributes.get('For Lineage', {}).get('value', False), "forDuplicateProcessing": attributes.get('For Duplicate Processing', {}).get('value', False), "anchorGUID": attributes.get('Anchor ID', {}).get('guid', None), @@ -303,7 +302,9 @@ def set_create_body(object_type: str, attributes: dict)->dict: "parentRelationshipTypeName": attributes.get('Parent Relationship Type Name', {}).get('value', None) or attributes.get('Parent Relationship Type', {}).get('value', None), "parentRelationshipProperties": attributes.get('Parent Relationship Attributes', {}).get('value', None) or attributes.get('Parent Relationship Properties', {}).get('value', None), "parentAtEnd1": attributes.get('Parent at End1', {}).get('value', True), - "anchorScopeGUID": attributes.get('Anchor Scope ID', {}).get('guid', None), + # Egeria reads a list, anchorScopeGUIDs; the singular key was dropped (ISSUE-164) + "anchorScopeGUIDs": ([attributes['Anchor Scope ID']['guid']] + if attributes.get('Anchor Scope ID', {}).get('guid') else None), "properties": "", "initialClassifications": {}} @@ -324,7 +325,6 @@ def set_update_body(object_type: str, attributes: dict)->dict: "class" : "UpdateElementRequestBody", "externalSourceGUID": attributes.get('External Source GUID', {}).get('guid', None), "externalSourceName": attributes.get('External Source Name', {}).get('value', None), - "governanceZoneFilter": attributes.get('Governance Zone Filter', {}).get('value', None), "effectiveTime": attributes.get('Effective Time', {}).get('value', None), "forLineage": attributes.get('For Lineage', {}).get('value', False), "forDuplicateProcessing": attributes.get('For Duplicate Processing', {}).get('value', False), diff --git a/pyegeria/models/models.py b/pyegeria/models/models.py index e57cbd04..d7ca9bd0 100644 --- a/pyegeria/models/models.py +++ b/pyegeria/models/models.py @@ -553,9 +553,18 @@ class TemplateRequestBody(RequestBody): validation_alias=AliasChoices("anchorScopeGUIDs", "anchorScopeGuids", "anchor_scope_guids"), serialization_alias="anchorScopeGUIDs", ) - initial_classifications: dict | None = None # unsure + # Egeria's TemplateRequestBody calls this replacementClassifications ("classifications + # to include in the entity creation request"); initialClassifications was dropped by + # the server (ISSUE-164). The old key is still read. + initial_classifications: dict | None = Field( + default=None, + validation_alias=AliasChoices("replacementClassifications", "initialClassifications", + "initial_classifications"), + serialization_alias="replacementClassifications", + ) parent_guid: str | None = None parent_relationship_type_name: str | None = None + parent_relationship_properties: dict | None = None parent_at_end_1: bool | None = True template_guid: str replacement_properties: dict[str, Any] | None = None @@ -809,7 +818,22 @@ class ActionRequestBody(RequestBody): action_sponsor_guid: Optional[str] = Field(None, alias="actionSponsorGUID") originator_guid: Optional[str] = Field(None, alias="originatorGUID") assign_to_actor_guid: Optional[str] = Field(None, alias="assignToActorGUID") - action_target_guids: Optional[list[dict[str,str]]] = Field(None, alias="actionTargetGUIDs") + # Egeria's ActionRequestBody (via AnchorOptions/MakeAnchorOptions) also takes these; the + # model lacked them, so e.g. My Profile's "isOwnAnchor" was dropped client-side (ISSUE-164). + anchor_guid: Optional[str] = Field(None, alias="anchorGUID") + is_own_anchor: Optional[bool] = None + anchor_scope_guids: Optional[list[str]] = Field( + default=None, + validation_alias=AliasChoices("anchorScopeGUIDs", "anchorScopeGuids", "anchor_scope_guids"), + serialization_alias="anchorScopeGUIDs", + ) + initial_classifications: Optional[dict] = None + # [{"actionTargetName": ..., "actionTargetGUID": ...}]; Egeria calls it newActionTargets. + new_action_targets: Optional[list[dict[str, str]]] = Field( + default=None, + validation_alias=AliasChoices("newActionTargets", "actionTargetGUIDs", "action_target_guids"), + serialization_alias="newActionTargets", + ) diff --git a/tests/micro-tests/test_issue164_silent_field_drops.py b/tests/micro-tests/test_issue164_silent_field_drops.py index 50c8a176..1e4a3be6 100644 --- a/tests/micro-tests/test_issue164_silent_field_drops.py +++ b/tests/micro-tests/test_issue164_silent_field_drops.py @@ -154,3 +154,51 @@ def test_solution_role_and_governance_purpose_use_registered_class_names(): "properties": {"class": "SolutionActorRoleProperties", "qualifiedName": "q"}}) assert "DataProcessingPurposeProperties" in GOV_DEF_PROPERTIES_LIST assert "GovernanceProcessingPurposeProperties" not in GOV_DEF_PROPERTIES_LIST + + +# --- group 3: model gaps and Dr.Egeria / CLI keys ------------------------------ + +def test_action_request_body_carries_egerias_anchor_fields_and_new_action_targets(): + # My Profile sends isOwnAnchor for to-dos/meetings/reviews; the model lacked it, so + # Pydantic dropped it. Egeria's field for targets is newActionTargets. + from pyegeria.models import ActionRequestBody + targets = [{"actionTargetName": "subject", "actionTargetGUID": "t-1"}] + for key in ("newActionTargets", "actionTargetGUIDs"): + sent = ActionRequestBody.model_validate({ + "class": "ActionRequestBody", "isOwnAnchor": True, "anchorGUID": "a-1", + "anchorScopeGUIDs": ["s-1"], key: targets}).model_dump(by_alias=True, exclude_none=True) + assert sent["isOwnAnchor"] is True + assert sent["anchorGUID"] == "a-1" + assert sent["anchorScopeGUIDs"] == ["s-1"] + assert sent["newActionTargets"] == targets + assert "actionTargetGUIDs" not in sent + + +def test_template_request_body_sends_replacement_classifications_and_parent_relationship_properties(): + from pyegeria.models import TemplateRequestBody + cls = {"Confidentiality": {"class": "ConfidentialityProperties", "confidentialityLevel": 1}} + rel = {"class": "CollectionMembershipProperties"} + for key in ("initialClassifications", "replacementClassifications"): + sent = TemplateRequestBody.model_validate({ + "class": "TemplateRequestBody", "templateGUID": "t", key: cls, + "parentRelationshipProperties": rel}).model_dump(by_alias=True, exclude_none=True) + assert sent["replacementClassifications"] == cls + assert sent["parentRelationshipProperties"] == rel + assert "initialClassifications" not in sent + + +def test_dr_egeria_find_create_update_bodies_use_keys_the_models_read(): + from md_processing.md_processing_utils.common_md_utils import set_create_body, set_find_body, set_update_body + from pyegeria.models import NewElementRequestBody, SearchStringRequestBody + + find = set_find_body("Glossary Term", {"Search String": {"value": "x"}, "End With": {"value": True}}) + assert find["endsWith"] is True and "endWith" not in find + assert SearchStringRequestBody.model_validate(find).ends_with is True + + create = set_create_body("Glossary Term", {"Anchor Scope ID": {"guid": "scope-1"}}) + # callers fill "properties" in afterwards (two-layer convention) + assert NewElementRequestBody.model_validate({**create, "properties": None}).model_dump( + by_alias=True, exclude_none=True)["anchorScopeGUIDs"] == ["scope-1"] + # create/update bodies have no zone filter in Egeria + assert "governanceZoneFilter" not in create + assert "governanceZoneFilter" not in set_update_body("Glossary Term", {}) From e11dc1e0772741088362fac2a2f77147422884c3 Mon Sep 17 00:00:00 2001 From: Dan Wolfson Date: Fri, 23 Oct 2026 15:10:43 -0400 Subject: [PATCH 11/17] ISSUE-164: record live verification and the Egeria 6 ElementHeader shape Signed-off-by: Dan Wolfson --- PYEGERIA_ISSUES.md | 42 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 41 insertions(+), 1 deletion(-) diff --git a/PYEGERIA_ISSUES.md b/PYEGERIA_ISSUES.md index 05dbb2d3..bfbfa7bd 100644 --- a/PYEGERIA_ISSUES.md +++ b/PYEGERIA_ISSUES.md @@ -288,7 +288,38 @@ they're holding real port materialization until this exists server-side ### ISSUE-164: `omvs_audit` now checks against Egeria's Java source. The first sweep finds 6 always-400 calls, 4 more 400s with a body, and 24 endpoints with no Spring mapping -**Layer:** Pyegeria (+ Dr.Egeria, CLI) · **Status:** audit improved on branch `fix/role-appointment-class-names`; the findings below are **open, not fixed** (triaged, awaiting the owner's go) · **Found:** 2026-10-23, sweep requested by the owner before the next release, after ISSUE-163. +**Layer:** Pyegeria (+ Dr.Egeria, CLI) · **Status:** audit improved, and the findings were fixed on branch `fix/role-appointment-class-names` (owner approved 2026-10-23; not yet released), except the items under *Not fixed* at the end · **Found:** 2026-10-23, sweep requested by the owner before the next release, after ISSUE-163. + +**Fixed 2026-10-23, in the owner's order:** +1. **Anchor scope and find keys** (3bb233c1): `anchorScopeGUIDs` serialisation on the New/Relationship/Template models; the `_async_find_request` keys. +2. **Always-400 deletes and wrong class names** (b8b4567b): `remove_is_abstract_concept`, `remove_activity_description`, `detach_cohort_member`, `remove_search_keyword`, `initiate_engine_action` (`InitiateEngineActionRequestBody`, `actionSourceGUIDs`, `startDate`), `DataValueMeaningProperties`, `SolutionActorRoleProperties`, `DataProcessingPurposeProperties`, and the CLI todo typo. +3. **Model gaps and Dr.Egeria/CLI keys** (13c17e37): `ActionRequestBody` anchor fields and `newActionTargets`; `TemplateRequestBody` `parentRelationshipProperties` and `replacementClassifications`; `endsWith`; Dr.Egeria/CLI `anchorScopeGUIDs`; the CLI update `properties`. + +Tests: `tests/micro-tests/test_issue164_silent_field_drops.py` (15 tests). + +**Live on quickstart 9443, 2026-10-23** (three peer-cleared runs, throwaway `pyeg-test::issue164::` glossary/terms/solution role; every GUID read back as gone and the prefix find was empty after each run): +- **Anchor scope stored:** a term created with `anchorScopeGUIDs: []` reads back with `elementHeader.anchor.classificationProperties.anchorScopeGUIDs == []`; a term created without it has none. +- **Classification add/remove round trips:** AbstractConcept, ContextDefinition, ActivityDescription and DataValue (stored as `DataValueMeaning`, matching the corrected class) each appear in `elementHeader.glossaryTermKinds` after the add and are gone after the remove. Before the fix, `remove_is_abstract_concept` was a 400 on every call. +- **Solution role:** `create_solution_role` with `SolutionActorRoleProperties` created, then deleted. +- **Not verified live:** + - The find zone filter: an unzoned element matches any zone filter, and zoning a throwaway element could make it undeletable for the test user. Verified at the wire level only. + - `remove_search_keyword`: my test body was wrong (`keyword`; Egeria's field is `displayName`, as the docstring says). + - `detach_cohort_member`, `initiate_engine_action`: not exercised, by design. + +**Egeria 6 element shape (for anyone reading classifications back; RE's read-backs too).** `get_*_by_guid(output_format="JSON")` returns the element with `elementHeader` and `properties`. **There is no `classifications` list.** Each classification is a named camelCase slot on `elementHeader` (`anchor`, `zoneMembership`, `retention`, `glossaryTermKinds`, `projectClassification`, `memento`, … per Java `ElementHeader`), and anything unslotted is in `elementHeader.otherClassifications`. A slot holds an `ElementClassification`, or a list of them for the `…Kinds` slots, each with `classificationName` and `classificationProperties`. Trimmed raw sample from this run: +```json +"elementHeader": { + "guid": "97581996-622d-46fe-82e0-0feaeea17132", + "anchor": {"class": "ElementClassification", "classificationName": "Anchors", + "classificationProperties": {"class": "AnchorsProperties", "anchorTypeName": "GlossaryTerm", + "anchorDomainName": "AuthoredReferenceable", + "anchorScopeGUIDs": ["057d4bd3-62bc-4591-8777-410f7528d867"]}}, + "glossaryTermKinds": [{"class": "ElementClassification", "classificationName": "AbstractConcept", "...": "..."}] +} +``` +Note that the anchor property is the plural `anchorScopeGUIDs`, a list. Code that looks for a generic `classifications` list, or for a singular `anchorScopeGUID`, finds nothing. That is how this run's first two read-backs reported false absences. + +**Correction to the triage below:** `RetentionClassificationProperties` and `SecurityGroupMembershipProperties` are **not** bugs. Egeria registers them as `@JsonSubTypes` aliases (`names = {...}`): `RetentionProperties` is also accepted as `"RetentionClassificationProperties"`, and `processors.py` records a 2026-08-03 live 400 for `RetentionProperties` on an older build. Left unchanged; the audit now reads registered names and aliases. **Audit changes** (`scripts/omvs_audit.py`, docs in CLAUDE.md and `scripts/README.md`, tests in `tests/micro-tests/test_omvs_audit_java_truth.py`): - **RENAMED matches** now get the verb/body checks too; they used to be skipped. @@ -352,6 +383,15 @@ they're holding real port materialization until this exists server-side - `get_tech_types_for_open_metadata_type(tech_name=...)` and `get_endpoints_for_asset(name=...)` send `filter` in a `FilterRequestBody`. The endpoints declare `ResultsRequestBody`, which has no `filter`, so they return everything unfiltered. - `find_software_capabilities(property_names=...)` sends `propertyNames` where Java declares `SearchStringRequestBody`, so the property restriction is lost. +**Not fixed (owner's call or Egeria's):** +- **19 endpoints Egeria doesn't serve.** Confirmed absent from the live platform's OpenAPI doc (`omvs_audit.py --live`; GET `/v3/api-docs` on 9443) and from Egeria `main` 4486d1a7f1: the 8 Data Discovery analysis-report methods, data-classification create/update/link/detach, subject-area member add/remove, subject-area hierarchy link/detach (the SDK uses `collections/{g}/collection-hierarchies`; Egeria has `subject-areas/{g}/subject-area-hierarchies`), `find_governance_action_processes`, `get_gov_def_in_context` and `get_platform_organization`. Raise with the Egeria team, or remove from the SDK. Three other JAVA-MISSING entries are served live, so they were Java-scan false positives. +- **No server-side filter exists:** `get_tech_types_for_open_metadata_type(tech_name=)` and `get_endpoints_for_asset(name=)`, whose endpoints take `ResultsRequestBody` (no `filter`), and `find_software_capabilities(property_names=)` (`SearchStringRequestBody`). Options: filter client-side, or drop and document the argument. +- **Dr.Egeria Asset Maker template `typeName`:** Egeria's nearest field is `TemplateOptions.metadataElementSubtypeName` ("create an element of a subtype to the template type"). That isn't clearly the same as the spec's "type name of the element to create", so it needs a live check before mapping. Part of ISSUE-155 item 4 (Metadata Expert create-from-template) is covered by the `TemplateRequestBody` fields added above. +- **Harmless top-level names Egeria ignores:** `FindProperties`, `AnyTimeRequestBody` (Java: `GetRequestBody`), `IntegrationServiceRequestBody` (legacy admin-services config). +- `cascadeDelete` stays as it is: ISSUE-132, on hold for Egeria. +- **New `.http` coverage, not part of this sweep:** the refreshed collections document usage-activity, member-popularity-profile and metadata undo/restore, which pyegeria lacks (ISSUE-155 style). +- **Dead legacy path:** `pyegeria/view/md_processing_utils.py:1539` (behind the `dr_egeria_jupyter` CLI) still sends the sunset `initialStatus`, but it calls `create_controlled_glossary_term`, which no longer exists. + *MODEL-FIELD gaps (review; 27 model→Java pairs, with users, in the report).* - Most are benign. For example, `ResultsRequestBody → GetRequestBody` drops paging on 51 `get_*_by_guid` calls, where paging means nothing. - Status-filter models carry both `activityStatus` and `activityStatusList`; only the `…List` form exists in Java. No current caller uses the singular form in a filter (My Profile's uses are inside the nested `…Properties`, where it is a real field). From 340c2a019965c4839ed378135682a11a701a52a4 Mon Sep 17 00:00:00 2001 From: Dan Wolfson Date: Fri, 23 Oct 2026 15:11:31 -0400 Subject: [PATCH 12/17] ISSUE-164: scope the ElementHeader note to the view-service shape; Metadata Expert still has a classifications list Signed-off-by: Dan Wolfson --- PYEGERIA_ISSUES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/PYEGERIA_ISSUES.md b/PYEGERIA_ISSUES.md index bfbfa7bd..90dc9284 100644 --- a/PYEGERIA_ISSUES.md +++ b/PYEGERIA_ISSUES.md @@ -306,7 +306,7 @@ Tests: `tests/micro-tests/test_issue164_silent_field_drops.py` (15 tests). - `remove_search_keyword`: my test body was wrong (`keyword`; Egeria's field is `displayName`, as the docstring says). - `detach_cohort_member`, `initiate_engine_action`: not exercised, by design. -**Egeria 6 element shape (for anyone reading classifications back; RE's read-backs too).** `get_*_by_guid(output_format="JSON")` returns the element with `elementHeader` and `properties`. **There is no `classifications` list.** Each classification is a named camelCase slot on `elementHeader` (`anchor`, `zoneMembership`, `retention`, `glossaryTermKinds`, `projectClassification`, `memento`, … per Java `ElementHeader`), and anything unslotted is in `elementHeader.otherClassifications`. A slot holds an `ElementClassification`, or a list of them for the `…Kinds` slots, each with `classificationName` and `classificationProperties`. Trimmed raw sample from this run: +**Egeria 6 element shape (for anyone reading classifications back; RE's read-backs too).** The view-service `get_*_by_guid(output_format="JSON")` calls (glossary, collection, actor manager, …) return the element with `elementHeader` and `properties`, and that header **has no `classifications` list**. The lower-level Metadata Expert / OMF shape (`get_metadata_element_by_guid`, an OpenMetadataElement) is different: it still has a top-level `classifications` list (Thursday Coordinator's read-only probe, 2026-10-23). Readers must know which shape they hold. Each classification is a named camelCase slot on `elementHeader` (`anchor`, `zoneMembership`, `retention`, `glossaryTermKinds`, `projectClassification`, `memento`, … per Java `ElementHeader`), and anything unslotted is in `elementHeader.otherClassifications`. A slot holds an `ElementClassification`, or a list of them for the `…Kinds` slots, each with `classificationName` and `classificationProperties`. Trimmed raw sample from this run: ```json "elementHeader": { "guid": "97581996-622d-46fe-82e0-0feaeea17132", From 85e54cd5a8ddc76f635124c436d4bfd6af7eec61 Mon Sep 17 00:00:00 2001 From: Dan Wolfson Date: Fri, 23 Oct 2026 15:18:22 -0400 Subject: [PATCH 13/17] Add new Egeria APIs: usage activity, member popularity, undo/restore, product catalog From the refreshed .http collections, checked against Egeria main 4486d1a7f1 (Spring @RequestBody types and registered properties classes): - ClassificationExplorer: add/update/clear_usage_activity and add/update/clear_member_popularity_profile (New/Update/Delete classification bodies; UsageActivityProperties, MemberPopularityProfileProperties). - MetadataExpert: undo_metadata_element_update, restore_metadata_element, undo_related_elements_update, restore_related_elements (MetadataSourceRequestBody; Egeria returns VoidResponse). - New ProductCatalog client (pyegeria, EgeriaTech "product_catalog"): find_digital_product_catalogs on the product-catalog URL marker. It inherits CollectionManager without changing its root, so facade resolution of inherited methods is unchanged. omvs_audit now reports 0 MISSING (was 13). 9 micro-tests. Signed-off-by: Dan Wolfson --- pyegeria/__init__.py | 2 + pyegeria/egeria_tech_client.py | 2 + pyegeria/omvs/__init__.py | 2 + pyegeria/omvs/classification_explorer.py | 194 ++++++++++++++++++ pyegeria/omvs/metadata_expert.py | 186 +++++++++++++++++ pyegeria/omvs/product_catalog.py | 111 ++++++++++ .../test_new_egeria_apis_2026_10_23.py | 116 +++++++++++ 7 files changed, 613 insertions(+) create mode 100644 pyegeria/omvs/product_catalog.py create mode 100644 tests/micro-tests/test_new_egeria_apis_2026_10_23.py diff --git a/pyegeria/__init__.py b/pyegeria/__init__.py index cf61f9f9..1a3fb10b 100644 --- a/pyegeria/__init__.py +++ b/pyegeria/__init__.py @@ -81,6 +81,7 @@ from pyegeria.omvs.notification_manager import NotificationManager from pyegeria.omvs.people_organizer import PeopleOrganizer from pyegeria.omvs.product_manager import ProductManager +from pyegeria.omvs.product_catalog import ProductCatalog from pyegeria.omvs.project_manager import ProjectManager from pyegeria.omvs.reference_data import ReferenceDataManager from pyegeria.omvs.registered_info import RegisteredInfo @@ -189,6 +190,7 @@ "NotificationManager", "PeopleOrganizer", "ProductManager", + "ProductCatalog", "ProjectManager", "ReferenceDataManager", "RegisteredInfo", diff --git a/pyegeria/egeria_tech_client.py b/pyegeria/egeria_tech_client.py index 9cf39114..439a10db 100644 --- a/pyegeria/egeria_tech_client.py +++ b/pyegeria/egeria_tech_client.py @@ -31,6 +31,7 @@ from pyegeria.omvs.notification_manager import NotificationManager from pyegeria.omvs.people_organizer import PeopleOrganizer from pyegeria.omvs.product_manager import ProductManager +from pyegeria.omvs.product_catalog import ProductCatalog from pyegeria.omvs.project_manager import ProjectManager from pyegeria.omvs.reference_data import ReferenceDataManager from pyegeria.omvs.registered_info import RegisteredInfo @@ -123,6 +124,7 @@ def __init__( "connection_maker": ConnectionMaker, "time_keeper": TimeKeeper, "product_manager": ProductManager, + "product_catalog": ProductCatalog, "location_arena": LocationArena, "data_discovery": DataDiscovery, "data_engineer": DataEngineer, diff --git a/pyegeria/omvs/__init__.py b/pyegeria/omvs/__init__.py index 7f110307..e841ba21 100644 --- a/pyegeria/omvs/__init__.py +++ b/pyegeria/omvs/__init__.py @@ -25,6 +25,7 @@ from pyegeria.omvs.notification_manager import NotificationManager from pyegeria.omvs.people_organizer import PeopleOrganizer from pyegeria.omvs.product_manager import ProductManager +from pyegeria.omvs.product_catalog import ProductCatalog from pyegeria.omvs.project_manager import ProjectManager from pyegeria.omvs.reference_data import ReferenceDataManager from pyegeria.omvs.registered_info import RegisteredInfo @@ -65,6 +66,7 @@ "NotificationManager", "PeopleOrganizer", "ProductManager", + "ProductCatalog", "ProjectManager", "ReferenceDataManager", "RegisteredInfo", diff --git a/pyegeria/omvs/classification_explorer.py b/pyegeria/omvs/classification_explorer.py index 42725805..9a1b1610 100644 --- a/pyegeria/omvs/classification_explorer.py +++ b/pyegeria/omvs/classification_explorer.py @@ -13533,6 +13533,200 @@ def clear_element_as_promise(self, element_guid: str, loop = asyncio.get_event_loop() loop.run_until_complete(self._async_clear_element_as_promise(element_guid, body)) + # + # UsageActivity + # + + @dynamic_catch + async def _async_add_usage_activity(self, element_guid: str, + body: Optional[dict | NewClassificationRequestBody] = None) -> None: + """Add the UsageActivity classification to an element. + It records how often the element has been read, found, updated and had attachments or + feedback changed or deleted. Async version. + + With no body, the classification is added with no counts set. Sample body: + + { + "class" : "NewClassificationRequestBody", + "properties" : { + "class" : "UsageActivityProperties", + "explicitGetCount" : 0, + "findCount" : 0, + "updatePropertyCount" : 0, + "updateAttachmentCount" : 0, + "updateFeedbackCount" : 0, + "deleteAttachmentCount" : 0, + "deleteFeedbackCount" : 0 + } + } + """ + url = f"{self.classification_command_root}/elements/{element_guid}/usage-activity" + if body is None: + body = {"class": "NewClassificationRequestBody", "properties": {"class": "UsageActivityProperties"}} + await self._async_new_classification_request(url, prop=["UsageActivityProperties"], body=body) + logger.info(f"Added UsageActivity classification to {element_guid}") + + @dynamic_catch + def add_usage_activity(self, element_guid: str, body: Optional[dict | NewClassificationRequestBody] = None) -> None: + """Add the UsageActivity classification to an element. + It records how often the element has been read, found, updated and had attachments or + feedback changed or deleted. + See _async_add_usage_activity for the body.""" + loop = asyncio.get_event_loop() + loop.run_until_complete(self._async_add_usage_activity(element_guid, body)) + + @dynamic_catch + async def _async_update_usage_activity(self, element_guid: str, + body: dict | UpdateClassificationRequestBody) -> None: + """Update the properties of an element's UsageActivity classification. Set + "mergeUpdate": true to change only the properties supplied. Async version. Sample body: + + { + "class" : "UpdateClassificationRequestBody", + "mergeUpdate" : true, + "properties" : { + "class" : "UsageActivityProperties", + "explicitGetCount" : 0, + "findCount" : 0, + "updatePropertyCount" : 0, + "updateAttachmentCount" : 0, + "updateFeedbackCount" : 0, + "deleteAttachmentCount" : 0, + "deleteFeedbackCount" : 0 + } + } + """ + url = f"{self.classification_command_root}/elements/{element_guid}/usage-activity/update" + await self._async_update_element_body_request(url, prop=["UsageActivityProperties"], body=body) + logger.info(f"Updated UsageActivity classification on {element_guid}") + + @dynamic_catch + def update_usage_activity(self, element_guid: str, body: dict | UpdateClassificationRequestBody) -> None: + """Update the properties of an element's UsageActivity classification. + See _async_update_usage_activity for the body.""" + loop = asyncio.get_event_loop() + loop.run_until_complete(self._async_update_usage_activity(element_guid, body)) + + @dynamic_catch + async def _async_clear_usage_activity(self, element_guid: str, + body: Optional[dict | DeleteClassificationRequestBody] = None) -> None: + """Remove the UsageActivity classification from an element. Async version.""" + url = f"{self.classification_command_root}/elements/{element_guid}/usage-activity/remove" + if body is None: + body = {"class": "DeleteClassificationRequestBody"} + await self._async_delete_classification_request(url, body) + logger.info(f"Removed UsageActivity classification from {element_guid}") + + @dynamic_catch + def clear_usage_activity(self, element_guid: str, body: Optional[dict | DeleteClassificationRequestBody] = None) -> None: + """Remove the UsageActivity classification from an element.""" + loop = asyncio.get_event_loop() + loop.run_until_complete(self._async_clear_usage_activity(element_guid, body)) + + # + # MemberPopularityProfile + # + + @dynamic_catch + async def _async_add_member_popularity_profile(self, element_guid: str, + body: Optional[dict | NewClassificationRequestBody] = None) -> None: + """Add the MemberPopularityProfile classification to an element. + It holds the same activity counts as UsageActivity for a collection, as a total plus a + per-member map of member GUID to count. Async version. + + With no body, the classification is added with no counts set. Sample body: + + { + "class" : "NewClassificationRequestBody", + "properties" : { + "class" : "MemberPopularityProfileProperties", + "explicitGetCountTotal" : 0, + "explicitGetCountProfile" : {"add member GUID" : 0}, + "findCountTotal" : 0, + "findCountProfile" : {}, + "updatePropertyCountTotal" : 0, + "updatePropertyCountProfile" : {}, + "updateAttachmentCountTotal" : 0, + "updateAttachmentCountProfile" : {}, + "updateFeedbackCountTotal" : 0, + "updateFeedbackCountProfile" : {}, + "deleteAttachmentCountTotal" : 0, + "deleteAttachmentCountProfile" : {}, + "deleteFeedbackCountTotal" : 0, + "deleteFeedbackCountProfile" : {} + } + } + """ + url = f"{self.classification_command_root}/elements/{element_guid}/member-popularity-profile" + if body is None: + body = {"class": "NewClassificationRequestBody", "properties": {"class": "MemberPopularityProfileProperties"}} + await self._async_new_classification_request(url, prop=["MemberPopularityProfileProperties"], body=body) + logger.info(f"Added MemberPopularityProfile classification to {element_guid}") + + @dynamic_catch + def add_member_popularity_profile(self, element_guid: str, body: Optional[dict | NewClassificationRequestBody] = None) -> None: + """Add the MemberPopularityProfile classification to an element. + It holds the same activity counts as UsageActivity for a collection, as a total plus a + per-member map of member GUID to count. + See _async_add_member_popularity_profile for the body.""" + loop = asyncio.get_event_loop() + loop.run_until_complete(self._async_add_member_popularity_profile(element_guid, body)) + + @dynamic_catch + async def _async_update_member_popularity_profile(self, element_guid: str, + body: dict | UpdateClassificationRequestBody) -> None: + """Update the properties of an element's MemberPopularityProfile classification. Set + "mergeUpdate": true to change only the properties supplied. Async version. Sample body: + + { + "class" : "UpdateClassificationRequestBody", + "mergeUpdate" : true, + "properties" : { + "class" : "MemberPopularityProfileProperties", + "explicitGetCountTotal" : 0, + "explicitGetCountProfile" : {"add member GUID" : 0}, + "findCountTotal" : 0, + "findCountProfile" : {}, + "updatePropertyCountTotal" : 0, + "updatePropertyCountProfile" : {}, + "updateAttachmentCountTotal" : 0, + "updateAttachmentCountProfile" : {}, + "updateFeedbackCountTotal" : 0, + "updateFeedbackCountProfile" : {}, + "deleteAttachmentCountTotal" : 0, + "deleteAttachmentCountProfile" : {}, + "deleteFeedbackCountTotal" : 0, + "deleteFeedbackCountProfile" : {} + } + } + """ + url = f"{self.classification_command_root}/elements/{element_guid}/member-popularity-profile/update" + await self._async_update_element_body_request(url, prop=["MemberPopularityProfileProperties"], body=body) + logger.info(f"Updated MemberPopularityProfile classification on {element_guid}") + + @dynamic_catch + def update_member_popularity_profile(self, element_guid: str, body: dict | UpdateClassificationRequestBody) -> None: + """Update the properties of an element's MemberPopularityProfile classification. + See _async_update_member_popularity_profile for the body.""" + loop = asyncio.get_event_loop() + loop.run_until_complete(self._async_update_member_popularity_profile(element_guid, body)) + + @dynamic_catch + async def _async_clear_member_popularity_profile(self, element_guid: str, + body: Optional[dict | DeleteClassificationRequestBody] = None) -> None: + """Remove the MemberPopularityProfile classification from an element. Async version.""" + url = f"{self.classification_command_root}/elements/{element_guid}/member-popularity-profile/remove" + if body is None: + body = {"class": "DeleteClassificationRequestBody"} + await self._async_delete_classification_request(url, body) + logger.info(f"Removed MemberPopularityProfile classification from {element_guid}") + + @dynamic_catch + def clear_member_popularity_profile(self, element_guid: str, body: Optional[dict | DeleteClassificationRequestBody] = None) -> None: + """Remove the MemberPopularityProfile classification from an element.""" + loop = asyncio.get_event_loop() + loop.run_until_complete(self._async_clear_member_popularity_profile(element_guid, body)) + if __name__ == "__main__": print("Main-Classification Manager") diff --git a/pyegeria/omvs/metadata_expert.py b/pyegeria/omvs/metadata_expert.py index 9220b526..b22d3ec7 100644 --- a/pyegeria/omvs/metadata_expert.py +++ b/pyegeria/omvs/metadata_expert.py @@ -601,6 +601,100 @@ def archive_metadata_element(self, metadata_element_guid: str, body: Optional[di loop = asyncio.get_event_loop() return loop.run_until_complete(self._async_archive_metadata_element(metadata_element_guid, body)) + @dynamic_catch + async def _async_undo_metadata_element_update(self, metadata_element_guid: str, body: Optional[dict | MetadataSourceRequestBody] = None) -> None: + """ + Undo the last update to a metadata element, restoring its previous version. Async version. + + Egeria keeps the element's version history; this reverts one update. + + Parameters + ---------- + metadata_element_guid : str + Unique identifier of the metadata element. + body : dict | MetadataSourceRequestBody, optional + Request options. + + Notes + ----- + Sample JSON body (optional; with none, an empty MetadataSourceRequestBody is sent): + { + "class": "MetadataSourceRequestBody", + "externalSourceGUID": "", + "externalSourceName": "", + "forLineage": false, + "forDuplicateProcessing": false, + "effectiveTime": "{{$isoTimestamp}}" + } + """ + url = f"{self.command_root}/metadata-elements/{metadata_element_guid}/undo" + await self._async_metadata_source_body_request(url, body) + + @dynamic_catch + def undo_metadata_element_update(self, metadata_element_guid: str, body: Optional[dict | MetadataSourceRequestBody] = None) -> None: + """ + Undo the last update to a metadata element, restoring its previous version. + + Egeria keeps the element's version history; this reverts one update. + + Parameters + ---------- + metadata_element_guid : str + Unique identifier of the metadata element. + body : dict | MetadataSourceRequestBody, optional + Request options; see the async version for a sample. + """ + loop = asyncio.get_event_loop() + return loop.run_until_complete(self._async_undo_metadata_element_update(metadata_element_guid, body)) + + @dynamic_catch + async def _async_restore_metadata_element(self, metadata_element_guid: str, body: Optional[dict | MetadataSourceRequestBody] = None) -> None: + """ + Restore a deleted or archived metadata element. Async version. + + Egeria also restores everything the same delete or archive changed (for example + the anchored elements removed with it). + + Parameters + ---------- + metadata_element_guid : str + Unique identifier of the metadata element. + body : dict | MetadataSourceRequestBody, optional + Request options. + + Notes + ----- + Sample JSON body (optional; with none, an empty MetadataSourceRequestBody is sent): + { + "class": "MetadataSourceRequestBody", + "externalSourceGUID": "", + "externalSourceName": "", + "forLineage": false, + "forDuplicateProcessing": false, + "effectiveTime": "{{$isoTimestamp}}" + } + """ + url = f"{self.command_root}/metadata-elements/{metadata_element_guid}/restore" + await self._async_metadata_source_body_request(url, body) + + @dynamic_catch + def restore_metadata_element(self, metadata_element_guid: str, body: Optional[dict | MetadataSourceRequestBody] = None) -> None: + """ + Restore a deleted or archived metadata element. + + Egeria also restores everything the same delete or archive changed (for example + the anchored elements removed with it). + + Parameters + ---------- + metadata_element_guid : str + Unique identifier of the metadata element. + body : dict | MetadataSourceRequestBody, optional + Request options; see the async version for a sample. + """ + loop = asyncio.get_event_loop() + return loop.run_until_complete(self._async_restore_metadata_element(metadata_element_guid, body)) + @dynamic_catch async def _async_classify_metadata_element(self, metadata_element_guid: str, classification_name: str, body: Optional[dict | NewClassificationRequestBody] = None) -> None: """ @@ -1040,6 +1134,98 @@ def delete_related_elements(self, relationship_guid: str, body: Optional[dict | loop = asyncio.get_event_loop() return loop.run_until_complete(self._async_delete_related_elements(relationship_guid, body)) + @dynamic_catch + async def _async_undo_related_elements_update(self, relationship_guid: str, body: Optional[dict | MetadataSourceRequestBody] = None) -> None: + """ + Undo the last update to a relationship, restoring its previous version. Async version. + + Egeria keeps the relationship's version history; this reverts one update. + + Parameters + ---------- + relationship_guid : str + Unique identifier of the relationship. + body : dict | MetadataSourceRequestBody, optional + Request options. + + Notes + ----- + Sample JSON body (optional; with none, an empty MetadataSourceRequestBody is sent): + { + "class": "MetadataSourceRequestBody", + "externalSourceGUID": "", + "externalSourceName": "", + "forLineage": false, + "forDuplicateProcessing": false, + "effectiveTime": "{{$isoTimestamp}}" + } + """ + url = f"{self.command_root}/related-elements/{relationship_guid}/undo" + await self._async_metadata_source_body_request(url, body) + + @dynamic_catch + def undo_related_elements_update(self, relationship_guid: str, body: Optional[dict | MetadataSourceRequestBody] = None) -> None: + """ + Undo the last update to a relationship, restoring its previous version. + + Egeria keeps the relationship's version history; this reverts one update. + + Parameters + ---------- + relationship_guid : str + Unique identifier of the relationship. + body : dict | MetadataSourceRequestBody, optional + Request options; see the async version for a sample. + """ + loop = asyncio.get_event_loop() + return loop.run_until_complete(self._async_undo_related_elements_update(relationship_guid, body)) + + @dynamic_catch + async def _async_restore_related_elements(self, relationship_guid: str, body: Optional[dict | MetadataSourceRequestBody] = None) -> None: + """ + Restore a deleted or archived relationship. Async version. + + Egeria also restores everything the same delete or archive changed. + + Parameters + ---------- + relationship_guid : str + Unique identifier of the relationship. + body : dict | MetadataSourceRequestBody, optional + Request options. + + Notes + ----- + Sample JSON body (optional; with none, an empty MetadataSourceRequestBody is sent): + { + "class": "MetadataSourceRequestBody", + "externalSourceGUID": "", + "externalSourceName": "", + "forLineage": false, + "forDuplicateProcessing": false, + "effectiveTime": "{{$isoTimestamp}}" + } + """ + url = f"{self.command_root}/related-elements/{relationship_guid}/restore" + await self._async_metadata_source_body_request(url, body) + + @dynamic_catch + def restore_related_elements(self, relationship_guid: str, body: Optional[dict | MetadataSourceRequestBody] = None) -> None: + """ + Restore a deleted or archived relationship. + + Egeria also restores everything the same delete or archive changed. + + Parameters + ---------- + relationship_guid : str + Unique identifier of the relationship. + body : dict | MetadataSourceRequestBody, optional + Request options; see the async version for a sample. + """ + loop = asyncio.get_event_loop() + return loop.run_until_complete(self._async_restore_related_elements(relationship_guid, body)) + @dynamic_catch async def _async_detach_related_elements_in_store( self, diff --git a/pyegeria/omvs/product_catalog.py b/pyegeria/omvs/product_catalog.py new file mode 100644 index 00000000..f016f520 --- /dev/null +++ b/pyegeria/omvs/product_catalog.py @@ -0,0 +1,111 @@ +""" +SPDX-License-Identifier: Apache-2.0 +Copyright Contributors to the ODPi Egeria project. + +This module provides access to the Product Catalog OMVS module. + +The Product Catalog view service supports browsing digital product catalogs. Egeria +serves it through the generic collection endpoints under the "product-catalog" URL +marker, so it inherits CollectionManager; the methods here use that marker. +""" +import asyncio +from typing import Optional + +from pyegeria.core.utils import dynamic_catch +from pyegeria.models import SearchStringRequestBody +from pyegeria.omvs.collection_manager import CollectionManager + + +class ProductCatalog(CollectionManager): + def __init__( + self, + view_server: str = None, + platform_url: str = None, + user_id: str = None, + user_pwd: Optional[str] = None, + token: Optional[str] = None, + timeout: int = None): + CollectionManager.__init__(self, view_server, platform_url, user_id, user_pwd, token, timeout=timeout) + # Only the methods below use this root; inherited CollectionManager methods keep their + # own collection-manager root, so they behave exactly as on CollectionManager. + self.product_catalog_command_root: str = ( + f"{self.platform_url}/servers/{self.view_server}/api/open-metadata/product-catalog" + ) + + @dynamic_catch + async def _async_find_digital_product_catalogs( + self, + search_string: str = "*", + metadata_element_type_name: str = "DigitalProductCatalog", + starts_with: bool = True, + ends_with: bool = False, + ignore_case: bool = True, + start_from: int = 0, + page_size: int = 100, + output_format: str = "JSON", + report_spec: Optional[str | dict] = None, + body: Optional[dict | SearchStringRequestBody] = None, + **kwargs, + ) -> list | str: + """Find digital product catalogs, or the products in them, through the product-catalog + view service. Async version. + + Parameters + ---------- + search_string : str, default "*" + String to match; "*" matches every element of the requested type. + metadata_element_type_name : str, default "DigitalProductCatalog" + Type to search for; use "DigitalProduct" to find products. + starts_with, ends_with, ignore_case : bool + How the search string is matched. + start_from, page_size : int + Paging. + output_format : str, default "JSON" + JSON, DICT, LIST, MD, REPORT, ... + report_spec : str | dict, optional + Report spec for non-JSON output. + body : dict | SearchStringRequestBody, optional + A full request body, used instead of the arguments above. + + Returns + ------- + list | str + The matching elements, or NO_ELEMENTS_FOUND. + + Notes + ----- + Sample body: + { + "class" : "SearchStringRequestBody", + "searchString" : "Open Metadata Digital Product Catalog", + "metadataElementTypeName" : "DigitalProductCatalog" + } + """ + url = f"{self.product_catalog_command_root}/collections/by-search-string" + return await self._async_find_request( + url, _type=metadata_element_type_name, _gen_output=self._generate_collection_output, + search_string=search_string, starts_with=starts_with, ends_with=ends_with, ignore_case=ignore_case, + metadata_element_type=metadata_element_type_name, start_from=start_from, page_size=page_size, + output_format=output_format, report_spec=report_spec, body=body, **kwargs) + + @dynamic_catch + def find_digital_product_catalogs( + self, + search_string: str = "*", + metadata_element_type_name: str = "DigitalProductCatalog", + starts_with: bool = True, + ends_with: bool = False, + ignore_case: bool = True, + start_from: int = 0, + page_size: int = 100, + output_format: str = "JSON", + report_spec: Optional[str | dict] = None, + body: Optional[dict | SearchStringRequestBody] = None, + **kwargs, + ) -> list | str: + """Find digital product catalogs, or the products in them ("DigitalProduct"), through the + product-catalog view service. See _async_find_digital_product_catalogs for the parameters.""" + loop = asyncio.get_event_loop() + return loop.run_until_complete(self._async_find_digital_product_catalogs( + search_string, metadata_element_type_name, starts_with, ends_with, ignore_case, + start_from, page_size, output_format, report_spec, body, **kwargs)) diff --git a/tests/micro-tests/test_new_egeria_apis_2026_10_23.py b/tests/micro-tests/test_new_egeria_apis_2026_10_23.py new file mode 100644 index 00000000..ba0d24a0 --- /dev/null +++ b/tests/micro-tests/test_new_egeria_apis_2026_10_23.py @@ -0,0 +1,116 @@ +""" +New Egeria APIs from the refreshed .http collections (2026-10-23), checked against +Egeria main 4486d1a7f1 (Spring @RequestBody types, registered properties classes): + classification-explorer: UsageActivity and MemberPopularityProfile add/update/clear + metadata-expert: undo / restore for metadata elements and relationships + product-catalog: find digital product catalogs (generic collection find) +No live server: the HTTP call is captured and the verb, URL and body asserted. +""" +import json + +import pytest + +from pyegeria.omvs.classification_explorer import ClassificationExplorer +from pyegeria.omvs.metadata_expert import MetadataExpert +from pyegeria.omvs.product_catalog import ProductCatalog + + +class _Response: + def json(self): + return {"class": "VoidResponse", "elements": [], "relatedHTTPCode": 200} + + +def _client(cls, calls, monkeypatch): + monkeypatch.setattr(cls, "check_connection", lambda self: "test") + client = cls("view-server", "https://localhost:1", "user", "pwd") + + async def fake_make_request(method, url, payload=None, *args, **kwargs): + calls.append({"method": method, "url": url, + "body": json.loads(payload) if isinstance(payload, str) else payload}) + return _Response() + + client._async_make_request = fake_make_request + return client + + +CE = "/servers/view-server/api/open-metadata/classification-explorer" +ME = "/servers/view-server/api/open-metadata/metadata-expert" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("slug, props_class", [ + ("usage_activity", "UsageActivityProperties"), + ("member_popularity_profile", "MemberPopularityProfileProperties"), +]) +async def test_activity_classification_add_update_clear(monkeypatch, slug, props_class): + calls: list = [] + client = _client(ClassificationExplorer, calls, monkeypatch) + seg = slug.replace("_", "-") + + await getattr(client, f"_async_add_{slug}")("el-guid") + await getattr(client, f"_async_update_{slug}")("el-guid", { + "class": "UpdateClassificationRequestBody", "mergeUpdate": True, + "properties": {"class": props_class}}) + await getattr(client, f"_async_clear_{slug}")("el-guid") + + add, update, clear = calls + assert add["url"].endswith(f"{CE}/elements/el-guid/{seg}") + assert add["body"]["class"] == "NewClassificationRequestBody" + assert add["body"]["properties"]["class"] == props_class + assert update["url"].endswith(f"{CE}/elements/el-guid/{seg}/update") + assert update["body"]["class"] == "UpdateClassificationRequestBody" + assert update["body"]["mergeUpdate"] is True + assert clear["url"].endswith(f"{CE}/elements/el-guid/{seg}/remove") + assert clear["body"]["class"] == "DeleteClassificationRequestBody" + assert {c["method"] for c in calls} == {"POST"} + + +@pytest.mark.asyncio +async def test_usage_activity_counts_reach_the_wire(monkeypatch): + calls: list = [] + client = _client(ClassificationExplorer, calls, monkeypatch) + await client._async_add_usage_activity("el-guid", { + "class": "NewClassificationRequestBody", + "properties": {"class": "UsageActivityProperties", "explicitGetCount": 3, "findCount": 5}}) + props = calls[0]["body"]["properties"] + assert props["explicitGetCount"] == 3 and props["findCount"] == 5 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("method, url_tail", [ + ("_async_undo_metadata_element_update", "/metadata-elements/g1/undo"), + ("_async_restore_metadata_element", "/metadata-elements/g1/restore"), + ("_async_undo_related_elements_update", "/related-elements/g1/undo"), + ("_async_restore_related_elements", "/related-elements/g1/restore"), +]) +async def test_metadata_expert_undo_and_restore(monkeypatch, method, url_tail): + calls: list = [] + client = _client(MetadataExpert, calls, monkeypatch) + await getattr(client, method)("g1") + assert calls[0]["method"] == "POST" + assert calls[0]["url"].endswith(f"{ME}{url_tail}") + # Egeria declares MetadataSourceRequestBody for all four + assert calls[0]["body"]["class"] == "MetadataSourceRequestBody" + + +@pytest.mark.asyncio +async def test_find_digital_product_catalogs_uses_the_product_catalog_marker(monkeypatch): + calls: list = [] + client = _client(ProductCatalog, calls, monkeypatch) + await client._async_find_digital_product_catalogs("Open Metadata Digital Product Catalog") + await client._async_find_digital_product_catalogs(metadata_element_type_name="DigitalProduct") + first, second = calls + assert first["url"].endswith("/api/open-metadata/product-catalog/collections/by-search-string") + assert first["body"]["class"] == "SearchStringRequestBody" + assert first["body"]["searchString"] == "Open Metadata Digital Product Catalog" + assert first["body"]["metadataElementTypeName"] == "DigitalProductCatalog" + assert second["body"]["metadataElementTypeName"] == "DigitalProduct" + assert "searchString" not in second["body"] # "*" means every element of the type + + +def test_product_catalog_keeps_collection_manager_behaviour_for_inherited_methods(monkeypatch): + # EgeriaTech resolves a method on whichever sub-client has it; an inherited + # find_collections on ProductCatalog must still call collection-manager. + monkeypatch.setattr(ProductCatalog, "check_connection", lambda self: "test") + pc = ProductCatalog("view-server", "https://localhost:1", "user", "pwd") + assert pc.collection_command_root.endswith("/api/open-metadata/collection-manager/collections") From 58dfaeff9e1b23d75c25381eaec61de4e1cd762c Mon Sep 17 00:00:00 2001 From: Dan Wolfson Date: Fri, 23 Oct 2026 15:21:20 -0400 Subject: [PATCH 14/17] ISSUE-165: document the 19 SDK methods calling endpoints Egeria doesn't serve Validated against Egeria main 4486d1a7f1, the live platform's OpenAPI doc, and git history: 13 removed upstream (with removing commit), 6 never existed, plus the Dr.Egeria command affected and proposed handling. Signed-off-by: Dan Wolfson --- PYEGERIA_ISSUES.md | 41 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 40 insertions(+), 1 deletion(-) diff --git a/PYEGERIA_ISSUES.md b/PYEGERIA_ISSUES.md index 90dc9284..6aa51e08 100644 --- a/PYEGERIA_ISSUES.md +++ b/PYEGERIA_ISSUES.md @@ -286,6 +286,45 @@ they're holding real port materialization until this exists server-side --- +### ISSUE-165: 19 SDK methods call endpoints Egeria doesn't serve: 13 removed upstream, 6 never existed + +**Layer:** Pyegeria · **Status:** open, documented, not changed (owner to decide: remove/deprecate, repoint, or raise with the Egeria team) · **Found:** 2026-10-23 by `scripts/omvs_audit.py` JAVA-MISSING (ISSUE-164). + +**Validation.** Each URL was checked three ways: +1. **No Spring mapping** in Egeria `main` 4486d1a7f1 (2026-10-23), the checkout `omvs_audit.py` reads. +2. **Absent from the live platform's OpenAPI doc** (`GET https://localhost:9443/v3/api-docs`, which `omvs_audit.py --live` reads; it marks these rows "[LIVE: absent, so a confirmed 404]"). +3. **Git history:** `git log -S` on the path fragment, to find the commit that removed it, or to show it was never in Egeria's Java. + +Three other JAVA-MISSING rows are served live, so they are Java-scan false positives and are excluded: `get_classified_elements_by` (runtime `by-{}` segment), `get_search_keyword_by_guid` and `new_subscription_type`. + +**Removed from Egeria:** + +| SDK method (file:line) | Calls | Removed by Egeria | Replacement in Egeria now | +|---|---|---|---| +| `data_discovery`: `create/update/delete_analysis_report`, `get_analysis_reports_by_name`, `find_analysis_reports`, `get_analysis_report_by_guid`, `link/detach_analysis_report_to/from_asset` (`data_discovery.py:1692-1773`, 8 methods) | `data-discovery/analysis-reports/...` | `e17cc09d44`, 2025-08-04 "Radical simplification of platform" | None as a CRUD API. Data Discovery now serves only `survey-reports/{g}/new-annotations[/{g}/attach\|detach]`; survey reports are produced by survey engines. The `AnalysisReportProperties` class these methods use doesn't exist either. | +| `classification_explorer.add_element_to_subject_area` / `remove_element_from_subject_area` (`classification_explorer.py:12293/12405`) | `classification-explorer/elements/{g}/subject-area-member[/remove]` | `6a49a3e23f`, 2026-01-09, which also removed `elements/by-subject-area-membership` | Not confirmed. Likely collection membership on the subject area (`{urlMarker}/collections/{g}/members/{g}/attach\|detach`, served by the generic CollectionManager resource); ask the Egeria team. `SubjectAreaMemberProperties` doesn't exist either. | +| `governance_officer.find_governance_action_processes` (`governance_officer.py:2310`) | `governance-officer/governance-action-processes/by-search-string` | `d299b120f4`, 2025-10-16, which removed all of Governance Officer's process create/update/publish/withdraw/remove/search endpoints | Not confirmed; a generic search by type `GovernanceActionProcess` is the likely route. `initiate` and `cancel` live on Automated Curation / Governance Engine (ISSUE-153). | +| `platform_services.get_platform_organization` (`platform_services.py:1083`) | `GET platform-services/server-platform/organization-name` | `816076ab50`, 2026-06-05 "Improve OMAGServerPlatform properties…" | Not confirmed; presumably folded into the platform properties. `GET server-platform/origin` remains. | + +**Never in Egeria's Java (the SDK URL was a guess):** + +| SDK method (file:line) | Calls | What Egeria has | +|---|---|---| +| `data_designer.create_data_classification`, `update_data_classification`, `link_data_classification_to_data_field`, `detach_data_classification_from_data_field` (`data_designer.py:3298/3465/3601/3709`) | `data-designer/data-classifications/...` | Only `data-classes/{g}/nested-data-classes/{g}/attach\|detach`; there is no "data classification" element API. The `DataFieldClassificationProperties` class doesn't exist either. | +| `subject_area.link_subject_area_hierarchy` / `detach_subject_area_hierarchy` (`subject_area.py:408/495`) | `subject-area/collections/{g}/collection-hierarchies/{g}/attach\|detach` | `subject-area/subject-areas/{parent}/subject-area-hierarchies/{nested}/attach\|detach`. **A straightforward URL fix**: same verbs and GUID pair, different segments. | +| `governance_officer.get_gov_def_in_context` (`governance_officer.py:2796`) | `governance-officer/governance-definitions/{g}/retrieve-context` | Only `governance-definitions/{g}/retrieve` (plus update/delete and the peer/supporting links). | + +**Known consumer:** Dr.Egeria's `View Governance Definition Context` command (`GovernanceContextProcessor`, `md_processing/v2/governance.py:775`, registered in `dr_egeria.py:207`) calls `get_gov_def_in_context`, so that command 404s too. No other Dr.Egeria or CLI code calls these methods. + +**Every call above returns a 404 today.** Before ISSUE-164, pyegeria reported a 404 as "unable to connect to the Egeria platform", so a caller could not tell "this API is gone" from "the server is down". + +**Proposed (not made):** +1. Repoint `link/detach_subject_area_hierarchy` to the `subject-areas/.../subject-area-hierarchies/...` paths and live-test them. +2. Mark the analysis-report, data-classification and `get_gov_def_in_context` methods deprecated, raising a clear "not supported by Egeria" error, before removing them in a later release. Dr.Egeria/CLI usage needs checking first. +3. Ask the Egeria team for the intended replacements for subject-area membership, governance-action-process search and the platform organization name. + +--- + ### ISSUE-164: `omvs_audit` now checks against Egeria's Java source. The first sweep finds 6 always-400 calls, 4 more 400s with a body, and 24 endpoints with no Spring mapping **Layer:** Pyegeria (+ Dr.Egeria, CLI) · **Status:** audit improved, and the findings were fixed on branch `fix/role-appointment-class-names` (owner approved 2026-10-23; not yet released), except the items under *Not fixed* at the end · **Found:** 2026-10-23, sweep requested by the owner before the next release, after ISSUE-163. @@ -384,7 +423,7 @@ Note that the anchor property is the plural `anchorScopeGUIDs`, a list. Code tha - `find_software_capabilities(property_names=...)` sends `propertyNames` where Java declares `SearchStringRequestBody`, so the property restriction is lost. **Not fixed (owner's call or Egeria's):** -- **19 endpoints Egeria doesn't serve.** Confirmed absent from the live platform's OpenAPI doc (`omvs_audit.py --live`; GET `/v3/api-docs` on 9443) and from Egeria `main` 4486d1a7f1: the 8 Data Discovery analysis-report methods, data-classification create/update/link/detach, subject-area member add/remove, subject-area hierarchy link/detach (the SDK uses `collections/{g}/collection-hierarchies`; Egeria has `subject-areas/{g}/subject-area-hierarchies`), `find_governance_action_processes`, `get_gov_def_in_context` and `get_platform_organization`. Raise with the Egeria team, or remove from the SDK. Three other JAVA-MISSING entries are served live, so they were Java-scan false positives. +- **19 endpoints Egeria doesn't serve**, documented in full with provenance in **ISSUE-165**. Confirmed absent from the live platform's OpenAPI doc (`omvs_audit.py --live`; GET `/v3/api-docs` on 9443) and from Egeria `main` 4486d1a7f1: the 8 Data Discovery analysis-report methods, data-classification create/update/link/detach, subject-area member add/remove, subject-area hierarchy link/detach (the SDK uses `collections/{g}/collection-hierarchies`; Egeria has `subject-areas/{g}/subject-area-hierarchies`), `find_governance_action_processes`, `get_gov_def_in_context` and `get_platform_organization`. Raise with the Egeria team, or remove from the SDK. Three other JAVA-MISSING entries are served live, so they were Java-scan false positives. - **No server-side filter exists:** `get_tech_types_for_open_metadata_type(tech_name=)` and `get_endpoints_for_asset(name=)`, whose endpoints take `ResultsRequestBody` (no `filter`), and `find_software_capabilities(property_names=)` (`SearchStringRequestBody`). Options: filter client-side, or drop and document the argument. - **Dr.Egeria Asset Maker template `typeName`:** Egeria's nearest field is `TemplateOptions.metadataElementSubtypeName` ("create an element of a subtype to the template type"). That isn't clearly the same as the spec's "type name of the element to create", so it needs a live check before mapping. Part of ISSUE-155 item 4 (Metadata Expert create-from-template) is covered by the `TemplateRequestBody` fields added above. - **Harmless top-level names Egeria ignores:** `FindProperties`, `AnyTimeRequestBody` (Java: `GetRequestBody`), `IntegrationServiceRequestBody` (legacy admin-services config). From 238aeebe11c950c675dd589e163b66bd561d555f Mon Sep 17 00:00:00 2001 From: Dan Wolfson Date: Fri, 23 Oct 2026 15:22:17 -0400 Subject: [PATCH 15/17] omvs_audit: tag JAVA-MISSING rows whose path moved to another service; ISSUE-165 move check Signed-off-by: Dan Wolfson --- CLAUDE.md | 4 +++- PYEGERIA_ISSUES.md | 1 + scripts/omvs_audit.py | 8 ++++++++ 3 files changed, 12 insertions(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index d549593d..84c1050d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -201,7 +201,9 @@ after a real defect slipped through an earlier, weaker version of this script: `"…RequestBody"` string in `pyegeria/omvs` that isn't an Egeria Java class. The properties bases are polymorphic, so sending one is a 400; this is what caught `PersonAppointmentProperties`. **JAVA-MISSING** (review) lists SDK - endpoints with no Spring mapping, meaning no URL, not a missing class. + endpoints with no Spring mapping, meaning no URL, not a missing class. Each row is tagged + "MOVED?" when the same path tail is served under another service, and with `--live` + as served or absent on the running platform. ISSUE-165 records the validated list. - **Field-level checks** — these catch the "call succeeds but silently ignores an option" bugs that no class check can see. - **FIELD-DROP** (defect): a method passes an argument whose JSON key the diff --git a/PYEGERIA_ISSUES.md b/PYEGERIA_ISSUES.md index 6aa51e08..f6b1de16 100644 --- a/PYEGERIA_ISSUES.md +++ b/PYEGERIA_ISSUES.md @@ -294,6 +294,7 @@ they're holding real port materialization until this exists server-side 1. **No Spring mapping** in Egeria `main` 4486d1a7f1 (2026-10-23), the checkout `omvs_audit.py` reads. 2. **Absent from the live platform's OpenAPI doc** (`GET https://localhost:9443/v3/api-docs`, which `omvs_audit.py --live` reads; it marks these rows "[LIVE: absent, so a confirmed 404]"). 3. **Git history:** `git log -S` on the path fragment, to find the commit that removed it, or to show it was never in Egeria's Java. +4. **Not moved to another module** (owner's question): the same path tail is not served under any other view service, in Java or live. The mappings added by each removing commit show no renamed replacement, with one exception. `d299b120f4` moved the governance-action-process **step-link** endpoints to Action Author; pyegeria already calls them there (`action_author.py`, `ActionProcessStepLinkProcessor`). Process search was not re-added anywhere. The audit now tags a JAVA-MISSING row "MOVED?" when the tail is served elsewhere; there are 0 such rows today. Three other JAVA-MISSING rows are served live, so they are Java-scan false positives and are excluded: `get_classified_elements_by` (runtime `by-{}` segment), `get_search_keyword_by_guid` and `new_subscription_type`. diff --git a/scripts/omvs_audit.py b/scripts/omvs_audit.py index 4626f86e..a575271d 100644 --- a/scripts/omvs_audit.py +++ b/scripts/omvs_audit.py @@ -1265,6 +1265,14 @@ def check_fields(svc: str, m: str, d: PyMethod, declared: set[str]) -> None: tag = (" **[LIVE: served, so a Java-scan false positive]**" if live_matches(d.verb, d.path, *live_index) else " **[LIVE: absent, so a confirmed 404]**") + # An endpoint can move between view services with the same tail + # (e.g. process-step links went to action-author); say where. + tail = "/" + "/".join(d.path.split("/")[2:]) + moved = sorted({p for (v, p) in java.endpoints + if v == d.verb and p.endswith(tail) and p != d.path}) + if moved: + tag += f" **[MOVED? Egeria serves `{moved[0]}`" + ( + f" (+{len(moved) - 1} more)" if len(moved) > 1 else "") + "]**" java_missing.append(f"- `{svc}.py:{d.lineno}` `{m}` `{d.verb} {d.path}`{tag}") if not d.body_class: continue From 3debc7d782f789fb1ea9565b7ce930a98fdfa227 Mon Sep 17 00:00:00 2001 From: Dan Wolfson Date: Fri, 23 Oct 2026 15:28:09 -0400 Subject: [PATCH 16/17] Repoint subject-area hierarchy link/detach to Egeria's URLs (ISSUE-165) link_subject_area_hierarchy / detach_subject_area_hierarchy called subject-area/collections/{g}/collection-hierarchies/{g}/..., which Egeria never served (404). SubjectAreaResource maps subject-area/subject-areas/{parent}/subject-area-hierarchies/{nested}/...; link_subject_areas / detach_subject_areas already used it, so these were broken duplicates. Both names kept. Signed-off-by: Dan Wolfson --- PYEGERIA_ISSUES.md | 6 +++--- pyegeria/omvs/subject_area.py | 4 ++-- .../test_new_egeria_apis_2026_10_23.py | 16 ++++++++++++++++ 3 files changed, 21 insertions(+), 5 deletions(-) diff --git a/PYEGERIA_ISSUES.md b/PYEGERIA_ISSUES.md index f6b1de16..82928913 100644 --- a/PYEGERIA_ISSUES.md +++ b/PYEGERIA_ISSUES.md @@ -286,7 +286,7 @@ they're holding real port materialization until this exists server-side --- -### ISSUE-165: 19 SDK methods call endpoints Egeria doesn't serve: 13 removed upstream, 6 never existed +### ISSUE-165: 19 SDK methods called endpoints Egeria doesn't serve: 13 removed upstream, 6 never existed (2 since repointed) **Layer:** Pyegeria · **Status:** open, documented, not changed (owner to decide: remove/deprecate, repoint, or raise with the Egeria team) · **Found:** 2026-10-23 by `scripts/omvs_audit.py` JAVA-MISSING (ISSUE-164). @@ -312,7 +312,7 @@ Three other JAVA-MISSING rows are served live, so they are Java-scan false posit | SDK method (file:line) | Calls | What Egeria has | |---|---|---| | `data_designer.create_data_classification`, `update_data_classification`, `link_data_classification_to_data_field`, `detach_data_classification_from_data_field` (`data_designer.py:3298/3465/3601/3709`) | `data-designer/data-classifications/...` | Only `data-classes/{g}/nested-data-classes/{g}/attach\|detach`; there is no "data classification" element API. The `DataFieldClassificationProperties` class doesn't exist either. | -| `subject_area.link_subject_area_hierarchy` / `detach_subject_area_hierarchy` (`subject_area.py:408/495`) | `subject-area/collections/{g}/collection-hierarchies/{g}/attach\|detach` | `subject-area/subject-areas/{parent}/subject-area-hierarchies/{nested}/attach\|detach`. **A straightforward URL fix**: same verbs and GUID pair, different segments. | +| `subject_area.link_subject_area_hierarchy` / `detach_subject_area_hierarchy` (`subject_area.py:408/495`) | `subject-area/collections/{g}/collection-hierarchies/{g}/attach\|detach` | `subject-area/subject-areas/{parent}/subject-area-hierarchies/{nested}/attach\|detach`. **Fixed 2026-10-23** (owner approved): repointed. The same module already had `link_subject_areas` / `detach_subject_areas` on the correct URLs, so these were broken duplicates; both names are kept for existing callers. | | `governance_officer.get_gov_def_in_context` (`governance_officer.py:2796`) | `governance-officer/governance-definitions/{g}/retrieve-context` | Only `governance-definitions/{g}/retrieve` (plus update/delete and the peer/supporting links). | **Known consumer:** Dr.Egeria's `View Governance Definition Context` command (`GovernanceContextProcessor`, `md_processing/v2/governance.py:775`, registered in `dr_egeria.py:207`) calls `get_gov_def_in_context`, so that command 404s too. No other Dr.Egeria or CLI code calls these methods. @@ -320,7 +320,7 @@ Three other JAVA-MISSING rows are served live, so they are Java-scan false posit **Every call above returns a 404 today.** Before ISSUE-164, pyegeria reported a 404 as "unable to connect to the Egeria platform", so a caller could not tell "this API is gone" from "the server is down". **Proposed (not made):** -1. Repoint `link/detach_subject_area_hierarchy` to the `subject-areas/.../subject-area-hierarchies/...` paths and live-test them. +1. ~~Repoint `link/detach_subject_area_hierarchy`~~ (done). 2. Mark the analysis-report, data-classification and `get_gov_def_in_context` methods deprecated, raising a clear "not supported by Egeria" error, before removing them in a later release. Dr.Egeria/CLI usage needs checking first. 3. Ask the Egeria team for the intended replacements for subject-area membership, governance-action-process search and the platform organization name. diff --git a/pyegeria/omvs/subject_area.py b/pyegeria/omvs/subject_area.py index 793748bf..d3404d7d 100644 --- a/pyegeria/omvs/subject_area.py +++ b/pyegeria/omvs/subject_area.py @@ -443,7 +443,7 @@ async def _async_link_subject_area_hierarchy( } ``` """ - url = f"{self.platform_url}/servers/{self.view_server}/api/open-metadata/subject-area/collections/{parent_subject_area_guid}/collection-hierarchies/{nested_subject_area_guid}/attach" + url = f"{self.platform_url}/servers/{self.view_server}/api/open-metadata/subject-area/subject-areas/{parent_subject_area_guid}/subject-area-hierarchies/{nested_subject_area_guid}/attach" await self._async_new_relationship_request(url, ["SubjectAreaHierarchyProperties"], body) def link_subject_area_hierarchy( @@ -527,7 +527,7 @@ async def _async_detach_subject_area_hierarchy( } ``` """ - url = f"{self.platform_url}/servers/{self.view_server}/api/open-metadata/subject-area/collections/{parent_subject_area_guid}/collection-hierarchies/{nested_subject_area_guid}/detach" + url = f"{self.platform_url}/servers/{self.view_server}/api/open-metadata/subject-area/subject-areas/{parent_subject_area_guid}/subject-area-hierarchies/{nested_subject_area_guid}/detach" await self._async_delete_relationship_request(url, body) def detach_subject_area_hierarchy( diff --git a/tests/micro-tests/test_new_egeria_apis_2026_10_23.py b/tests/micro-tests/test_new_egeria_apis_2026_10_23.py index ba0d24a0..172bf8f7 100644 --- a/tests/micro-tests/test_new_egeria_apis_2026_10_23.py +++ b/tests/micro-tests/test_new_egeria_apis_2026_10_23.py @@ -114,3 +114,19 @@ def test_product_catalog_keeps_collection_manager_behaviour_for_inherited_method monkeypatch.setattr(ProductCatalog, "check_connection", lambda self: "test") pc = ProductCatalog("view-server", "https://localhost:1", "user", "pwd") assert pc.collection_command_root.endswith("/api/open-metadata/collection-manager/collections") + + +@pytest.mark.asyncio +@pytest.mark.parametrize("method, verb, body_class", [ + ("_async_link_subject_area_hierarchy", "attach", "NewRelationshipRequestBody"), + ("_async_detach_subject_area_hierarchy", "detach", "DeleteRelationshipRequestBody"), +]) +async def test_subject_area_hierarchy_uses_egerias_url(monkeypatch, method, verb, body_class): + # ISSUE-165: the SDK called subject-area/collections/{g}/collection-hierarchies/{g}/..., which + # Egeria never served; SubjectAreaResource maps subject-areas/{g}/subject-area-hierarchies/{g}/... + from pyegeria.omvs.subject_area import SubjectArea + calls: list = [] + client = _client(SubjectArea, calls, monkeypatch) + await getattr(client, method)("parent-guid", "nested-guid", {"class": body_class}) + assert calls[0]["url"].endswith( + f"/api/open-metadata/subject-area/subject-areas/parent-guid/subject-area-hierarchies/nested-guid/{verb}") From 3d7cfd547a3239f637454cd6254988087215a133 Mon Sep 17 00:00:00 2001 From: Dan Wolfson Date: Fri, 23 Oct 2026 15:34:11 -0400 Subject: [PATCH 17/17] Record live verification of the new APIs; document count-reset on activity updates UsageActivity / MemberPopularityProfile, undo/restore (element and relationship; restore also reverses a Memento archive, keeping the GUID and original qualifiedName), subject-area hierarchy and product-catalog find verified on quickstart 9443. Update docstrings now say to send every count: Egeria's counts are primitive longs, so an omitted count resets to 0 even with mergeUpdate. Marks ISSUE-155 items 4 (partly), 5, 6 and ISSUE-159's restore proposal. Signed-off-by: Dan Wolfson --- PYEGERIA_ISSUES.md | 17 +++++++++++++---- pyegeria/omvs/classification_explorer.py | 14 ++++++++++---- 2 files changed, 23 insertions(+), 8 deletions(-) diff --git a/PYEGERIA_ISSUES.md b/PYEGERIA_ISSUES.md index 82928913..f75bd188 100644 --- a/PYEGERIA_ISSUES.md +++ b/PYEGERIA_ISSUES.md @@ -423,6 +423,15 @@ Note that the anchor property is the plural `anchorScopeGUIDs`, a list. Code tha - `get_tech_types_for_open_metadata_type(tech_name=...)` and `get_endpoints_for_asset(name=...)` send `filter` in a `FilterRequestBody`. The endpoints declare `ResultsRequestBody`, which has no `filter`, so they return everything unfiltered. - `find_software_capabilities(property_names=...)` sends `propertyNames` where Java declares `SearchStringRequestBody`, so the property restriction is lost. +**New APIs (2026-10-23, from the refreshed `.http`; ISSUE-155 items 5 and 6), live on 9443.** One peer-cleared run with throwaway `pyeg-test::newapis::` elements; all GUIDs read back as gone and the prefix find was empty. Raw dumps were analysed afterwards: +- **`add/update/clear_usage_activity`:** add stored `explicitGetCount=1, findCount=2`; update set `findCount=9`; clear removed the `elementHeader.usageActivity` slot. **Egeria behaviour:** that update also reset `explicitGetCount` to 0 despite `mergeUpdate: true`. The count fields are primitive `long`s (Java `UsageActivityProperties`), so an omitted count deserialises as 0. The docstrings say to send every count. +- **`add/update/clear_member_popularity_profile`** (on a glossary): the total and the per-member map were stored; update set `findCountTotal=4` and kept the map; clear removed it. +- **`undo_metadata_element_update`:** a term description `v2` went back to `v1`. **`undo_related_elements_update`:** a RelatedTerm description `r2` went back to `r1`. +- **`restore_related_elements`:** a removed RelatedTerm came back with its properties. +- **`restore_metadata_element`:** a soft-deleted term came back. **An archived (Memento) term also came back, with the same GUID, the Memento removed, and its qualifiedName restored** from `…t3_archivedOn_Fri Oct 23 19:29:39 GMT 2026` to the original. This answers ISSUE-159's open question: restore is not soft-delete-only. Anchor-scoped restore (anchored descendants) was not tested. A plain get of the archived term (no `forLineage`) returned HTTP 500, not 404. +- **`link/detach_subject_area_hierarchy`** (repointed, ISSUE-165): the child was linked, then gone after detach. +- **`ProductCatalog.find_digital_product_catalogs`:** a named search returned 2 results in 6.2s (`ProductManager`'s product-manager-marker version returned 13 in 41s). An unfiltered "*" search timed out at the helper's 90s. + **Not fixed (owner's call or Egeria's):** - **19 endpoints Egeria doesn't serve**, documented in full with provenance in **ISSUE-165**. Confirmed absent from the live platform's OpenAPI doc (`omvs_audit.py --live`; GET `/v3/api-docs` on 9443) and from Egeria `main` 4486d1a7f1: the 8 Data Discovery analysis-report methods, data-classification create/update/link/detach, subject-area member add/remove, subject-area hierarchy link/detach (the SDK uses `collections/{g}/collection-hierarchies`; Egeria has `subject-areas/{g}/subject-area-hierarchies`), `find_governance_action_processes`, `get_gov_def_in_context` and `get_platform_organization`. Raise with the Egeria team, or remove from the SDK. Three other JAVA-MISSING entries are served live, so they were Java-scan false positives. - **No server-side filter exists:** `get_tech_types_for_open_metadata_type(tech_name=)` and `get_endpoints_for_asset(name=)`, whose endpoints take `ResultsRequestBody` (no `filter`), and `find_software_capabilities(property_names=)` (`SearchStringRequestBody`). Options: filter client-side, or drop and document the argument. @@ -583,7 +592,7 @@ The OMF store's delete (`OpenMetadataStoreRESTServices`, around `:4118`) also ro **Possible route today (unverified):** `ProjectManager.delete_project(guid, body={"class": "DeleteElementRequestBody", "deleteMethod": "ARCHIVE"})`. pyegeria passes `deleteMethod` through, but whether the project-manager delete path honours ARCHIVE as the OMF store delete does was not traced. **Proposed (not made):** -1. Add `MetadataExpert.restore_metadata_element` / `restore_related_elements` (and undo), per ISSUE-155 item 5. +1. ~~Add `MetadataExpert.restore_metadata_element` / `restore_related_elements` (and undo), per ISSUE-155 item 5.~~ Done 2026-10-23 (branch `fix/role-appointment-class-names`). Items 2 and 3 are still open. 2. Add `ProjectManager.archive_project` and `CollectionManager.archive_collection`. These can be thin wrappers that send `deleteMethod: ARCHIVE` once that path is confirmed live, or call Metadata Expert archive. 3. Live-test archive then restore of a throwaway Project, including the `_archivedOn_` rename and its reversal. @@ -635,9 +644,9 @@ Egeria (`AssetMakerResource.addCatalogTarget`, served under every generic URL ma 1. **Engine-action calls moved to the Governance Engine view service (PR #9373).** Automated Curation no longer has claim, status update, action-targets update, completion-status or active-claimed. On `main` the audit reports them ELSEWHERE, plus `getApprovedEngineActions` and `cancelGovernanceActionProcess` MISSING. **Already fixed on branch `fix/issue-153-governance-engine` (ISSUE-153), not yet merged or released.** The only action is to merge and release that branch. 2. **`cascadeDelete` alias (PR #9365, `DeleteOptions.setCascadedDelete` now has `@JsonAlias("cascadeDelete")`).** pyegeria's cascade flag now reaches the server, so ISSUE-154 item 1 applies: switch to `cascadedDelete` and re-test cascades. 3. **Zones fail closed (PR #9367).** A GovernanceZone with no security access control now admits no-one; before, it was ignored. This is not a REST change. The new `MISSING_ZONE_ACCESS_CONTROL` exception type is raised by the platform cataloguer, not on a REST path. Callers will see 403s or empty results for elements whose only zones are unsecured. **pyegeria:** no code change. Expect more 403s on quickstart content, as with the digital-products zone (2026-10-14). The 403 error message already reads clearly since ISSUE-142. -4. **`createMetadataElementFromTemplate` body class.** The `.http` now shows `OpenMetadataTemplateRequestBody`. The Java parameter type was already that at V6.2, so only the sample changed. pyegeria sends its `TemplateRequestBody` model (`metadata_expert.py:159-207`, `models.py:526`). That model has no `typeName`, `replacementClassifications` or `parentRelationshipProperties`, so the PyegeriaModel `extra='ignore'` rule silently drops those fields if a caller passes them. **Proposed:** add an `OpenMetadataTemplateRequestBody` model with those fields and use it here. Medium value: it is the only way to create a subtype from a template through Metadata Expert. -5. **Undo and restore (PR #9365/#9373, new).** Metadata Expert `POST /metadata-elements/{guid}/undo` restores the previous version and fails if there is none. `/metadata-elements/{guid}/restore` reverses a delete or archive, including what that delete cascaded, but not a purge. `related-elements/{guid}/undo` and `/restore` do the same for relationships. Each takes an optional `MetadataSourceRequestBody`. **Proposed:** four methods on `MetadataExpert` via the existing `_async_metadata_source_body_request` (`_server_client.py:6395`). This pairs well with ISSUE-117 and RE's archive walk: restore is the safety net for a mistaken archive. -6. **Usage Activity and Member Popularity Profile classifications (Classification Explorer, new, probably for the Lovelace Popularity Profiler in #9369).** Each has add (`POST /elements/{guid}/usage-activity`), `/update` and `/remove`, and the same for `/member-popularity-profile`. The bodies are New/Update/DeleteClassificationRequestBody with `UsageActivityProperties` or `MemberPopularityProfileProperties`. **Proposed:** six methods, and the property names in `element_status_fields.py` if Dr.Egeria should show them. Low priority until something writes them. +4. **`createMetadataElementFromTemplate` body class.** The `.http` now shows `OpenMetadataTemplateRequestBody`. The Java parameter type was already that at V6.2, so only the sample changed. pyegeria sends its `TemplateRequestBody` model (`metadata_expert.py:159-207`, `models.py:526`). That model has no `typeName`, `replacementClassifications` or `parentRelationshipProperties`, so the PyegeriaModel `extra='ignore'` rule silently drops those fields if a caller passes them. **Proposed:** add an `OpenMetadataTemplateRequestBody` model with those fields and use it here. Medium value: it is the only way to create a subtype from a template through Metadata Expert. **Partly done 2026-10-23:** pyegeria's `TemplateRequestBody` now sends `replacementClassifications` and `parentRelationshipProperties` (ISSUE-164). `typeName` (on `OpenMetadataTemplateRequestBody` only) is still open. +5. **Undo and restore (PR #9365/#9373, new).** Metadata Expert `POST /metadata-elements/{guid}/undo` restores the previous version and fails if there is none. `/metadata-elements/{guid}/restore` reverses a delete or archive, including what that delete cascaded, but not a purge. `related-elements/{guid}/undo` and `/restore` do the same for relationships. Each takes an optional `MetadataSourceRequestBody`. **Proposed:** four methods on `MetadataExpert` via the existing `_async_metadata_source_body_request` (`_server_client.py:6395`). This pairs well with ISSUE-117 and RE's archive walk: restore is the safety net for a mistaken archive. **Done 2026-10-23 (ISSUE-164 follow-up, branch `fix/role-appointment-class-names`):** `MetadataExpert.undo_metadata_element_update`, `restore_metadata_element`, `undo_related_elements_update`, `restore_related_elements`. +6. **Usage Activity and Member Popularity Profile classifications (Classification Explorer, new, probably for the Lovelace Popularity Profiler in #9369).** Each has add (`POST /elements/{guid}/usage-activity`), `/update` and `/remove`, and the same for `/member-popularity-profile`. The bodies are New/Update/DeleteClassificationRequestBody with `UsageActivityProperties` or `MemberPopularityProfileProperties`. **Proposed:** six methods, and the property names in `element_status_fields.py` if Dr.Egeria should show them. Low priority until something writes them. **Done 2026-10-23 (same branch):** `ClassificationExplorer.add/update/clear_usage_activity` and `add/update/clear_member_popularity_profile`. 7. **`GetOptions.viewElementGUID` (new).** It identifies the element whose view a request is assembling, so related elements fetched for that view are not reported as "found" (usage tracking). **Proposed:** an optional field on the get-request models, for the Resource Browser and the explorers. Low priority. 8. **`relationshipPurpose` on RelationshipDef (PR #9365).** Type definitions now carry a relationship purpose used for anchor processing. Nothing in pyegeria reads it. **Proposed:** none now. Possibly show it in the Egeria Explorer type views later. 9. **`finalAnalysisStep` (PR #9366).** This is internal to the survey connectors (a provider/service API) and not a REST field. **pyegeria:** none. diff --git a/pyegeria/omvs/classification_explorer.py b/pyegeria/omvs/classification_explorer.py index 9a1b1610..0a6cbd79 100644 --- a/pyegeria/omvs/classification_explorer.py +++ b/pyegeria/omvs/classification_explorer.py @@ -13578,8 +13578,11 @@ def add_usage_activity(self, element_guid: str, body: Optional[dict | NewClassif @dynamic_catch async def _async_update_usage_activity(self, element_guid: str, body: dict | UpdateClassificationRequestBody) -> None: - """Update the properties of an element's UsageActivity classification. Set - "mergeUpdate": true to change only the properties supplied. Async version. Sample body: + """Update the properties of an element's UsageActivity classification. Async version. + + Send every count, even with "mergeUpdate": true. Egeria's count fields are primitive longs, + so an omitted count arrives as 0 and overwrites the stored value (seen live: explicitGetCount + 1 -> 0 on a findCount-only merge update). Profile maps that are omitted are kept. Sample body: { "class" : "UpdateClassificationRequestBody", @@ -13675,8 +13678,11 @@ def add_member_popularity_profile(self, element_guid: str, body: Optional[dict | @dynamic_catch async def _async_update_member_popularity_profile(self, element_guid: str, body: dict | UpdateClassificationRequestBody) -> None: - """Update the properties of an element's MemberPopularityProfile classification. Set - "mergeUpdate": true to change only the properties supplied. Async version. Sample body: + """Update the properties of an element's MemberPopularityProfile classification. Async version. + + Send every count, even with "mergeUpdate": true. Egeria's count fields are primitive longs, + so an omitted count arrives as 0 and overwrites the stored value (seen live: explicitGetCount + 1 -> 0 on a findCount-only merge update). Profile maps that are omitted are kept. Sample body: { "class" : "UpdateClassificationRequestBody",