Skip to content

Dr.Egeria updates no longer reset properties the document doesn't set (ISSUE-162) #482

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dwolfson merged 3 commits into from
Oct 23, 2026
Merged

Dr.Egeria updates no longer reset properties the document doesn't set (ISSUE-162) #482

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
24af3ed
docs(issues): ISSUE-162 Update Glossary Term resets four boolean flag…
dwolfson Oct 23, 2026
aec2755
fix: Dr.Egeria updates no longer reset properties the document doesn'…
dwolfson Oct 23, 2026
4ac8859
docs(issues): ISSUE-162 fixed (glossary term flags and Data Designer …
dwolfson Oct 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions PYEGERIA_ISSUES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -286,6 +286,28 @@ they're holding real port materialization until this exists server-side

---

### ISSUE-162: Dr.Egeria "Update Glossary Term" resets isAbstractConcept / isActivityDescription / isContext / isDataValue to False whenever the document doesn't mention them

**Layer:** Pyegeria (Dr.Egeria) · **Status:** fixed on branch `fix/issue-162-update-resets-booleans` (owner approved 2026-10-23), not yet released · **Found:** 2026-10-23 by RE's question-term update batch builder, reading `md_processing/v2` (RE term batch fix round 3). Reported via the Thursday Coordinator; confirmed here in source on `main` (6.2.9), not run live.

**What:** `TermProcessor.apply_changes` (`md_processing/v2/glossary.py:145-148`) builds the term properties with `attributes.get('Is Abstract Concept', {}).get('value', False)`, and likewise for `Is Activity Description`, `Is Context` and `Is Data Value`. On update, the body goes through `filter_update_properties` (`md_processing/v2/processors.py:1957`). For a merge update that drops only `None` values, so the `False` defaults are always sent. Every "Update Glossary Term", even one that changes only the description, therefore writes `False` to all four flags and clears any that were `True`.

**Same pattern, likely affected (not confirmed):** `md_processing/v2/data_designer.py:54-55` (`DataValueSpecificationProcessor`) and `:547-549` (`DataClassProcessor`) default `allowsDuplicateValues` / `isNullable` to `True`. Both processors appear to build one property body for create and update, so an unrelated update would reset those to `True`. Other `.get('value', <default>)` uses checked in `md_processing` are request options (`forLineage`, `startsWith`, ...) or create-only values, not stored properties.

**Impact:** a term with any of the four flags set loses it on an unrelated update. RE's 96-term question update batch will check the terms before applying.

**Fix (2026-10-23):**
- **Glossary terms:** the four flags now default to `None`, so a merge update omits them unless the document sets them. Create still sets them to `False` as before.
- **Data Designer: confirmed and wider than first logged.** Both `DataValueSpecificationProcessor` and `DataClassProcessor` updates send their property body straight to `_async_update_element_body_request`, where `body_slimmer` drops only `None`. So an unrelated update also reset `matchPropertyNames`, `sampleValues` and `dataPatterns` to `[]`, `matchThreshold` to `0`, and `specificationDetails` and `additionalProperties` to `{}`, besides `allowsDuplicateValues` / `isNullable` to `True`. All of these now default to `None`, and the old defaults are applied only on Create (`_with_create_defaults`). Egeria's `mergeUpdate` defaults to `true` (`UpdateOptions.java:23`), so omitted properties keep their values.
- Tests: `tests/micro-tests/test_issue_162_update_does_not_reset_properties.py`, 7 cases. The 4 update cases fail on the old code; the 3 create cases guard the unchanged create defaults. The full micro-test suite (1,436) passes in a clean python:3.13-slim container. Not live-tested.

**Originally proposed:**
- Default these attributes to `None` (`.get('value')`), so a merge update omits them unless the document sets them. Keep `False` / `True` only on the create path, or leave creation to Egeria's own defaults.
- Check the Data Designer cases the same way.
- Add a micro-test: an update with no flag attributes sends none of the four keys.

---

### ISSUE-161: Dr.Egeria's two entry points resolve the Egeria user differently, and the CLI fixes its credential defaults at import time

**Layer:** Pyegeria · **Status:** fixed on branch `fix/issue-161-dr-egeria-credentials` (owner approved 2026-10-21), not yet released · **Found:** 2026-10-21, while checking an RE report that `dr_egeria --process` ignored `EGERIA_USER`. **That report is withdrawn; RE was the cause:** RE's own `config.py:316` defaulted its daemon identity (`EGERIA_USER_ID`) to "erinoverview", so RE handed the child process erinoverview. The CLI correctly used it. RE is removing that default.
Expand Down
57 changes: 41 additions & 16 deletions md_processing/v2/data_designer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,29 @@
from pyegeria.core.utils import body_slimmer

# --- DataValueSpecificationProcessor ---

# ISSUE-162: defaults applied only when creating a data value specification / data class.
# On update, unset attributes stay None and are dropped (Egeria's mergeUpdate defaults to
# true), so an update no longer resets these properties.
_DATA_VALUE_SPEC_CREATE_DEFAULTS = {
"matchPropertyNames": [],
"matchThreshold": 0,
"specificationDetails": {},
"allowsDuplicateValues": True,
"isNullable": True,
"sampleValues": [],
"dataPatterns": [],
"additionalProperties": {},
}


def _with_create_defaults(props: dict) -> dict:
out = dict(props)
for key, default in _DATA_VALUE_SPEC_CREATE_DEFAULTS.items():
if key in out and out[key] is None:
out[key] = default
return out

class DataValueSpecificationProcessor(AsyncBaseCommandProcessor):
"""
Processor for Data Value Specification create/update with upsert logic.
Expand All @@ -43,24 +66,24 @@ async def apply_changes(self) -> str:
"displayName": display_name,
"description": attributes.get('Description', {}).get('value'),
"namespacePath": attributes.get('Namespace Path', {}).get('value'),
"matchPropertyNames": attributes.get('Match Property Names', {}).get('value', []),
"matchThreshold": attributes.get('Match Threshold', {}).get('value', 0),
"matchPropertyNames": attributes.get('Match Property Names', {}).get('value'),
"matchThreshold": attributes.get('Match Threshold', {}).get('value'),
"specification": attributes.get('Specification', {}).get('value'),
"specificationDetails": attributes.get('Specification Details', {}).get('value', {}),
"specificationDetails": attributes.get('Specification Details', {}).get('value'),
"dataType": attributes.get('Data Type', {}).get('value'),
"units": attributes.get('Units', {}).get('value'),
"absoluteUncertainty": attributes.get('Absolute Uncertainty', {}).get('value'),
"relativeUncertainty": attributes.get('Relative Uncertainty', {}).get('value'),
"allowsDuplicateValues": attributes.get('Allow Duplicates', {}).get('value', True),
"isNullable": attributes.get('Is Nullable', {}).get('value', True),
"allowsDuplicateValues": attributes.get('Allow Duplicates', {}).get('value'),
"isNullable": attributes.get('Is Nullable', {}).get('value'),
"defaultValue": attributes.get('Default Value', {}).get('value'),
"averageValue": attributes.get('Average Value', {}).get('value'),
"valueList": attributes.get('Value List', {}).get('value'),
"valueRangeFrom": attributes.get('Value Range From', {}).get('value'),
"valueRangeTo": attributes.get('Value Range To', {}).get('value'),
"sampleValues": attributes.get('Sample Values', {}).get('value', []),
"dataPatterns": attributes.get('Data Patterns', {}).get('value', []),
"additionalProperties": attributes.get('Additional Properties', {}).get('value', {})
"sampleValues": attributes.get('Sample Values', {}).get('value'),
"dataPatterns": attributes.get('Data Patterns', {}).get('value'),
"additionalProperties": attributes.get('Additional Properties', {}).get('value')
}

# "In Data Value Specification"/"Specializes Data Value Specification"
Expand Down Expand Up @@ -98,6 +121,7 @@ async def apply_changes(self) -> str:
return await self.render_result_markdown(guid)

elif verb == "Create":
props = _with_create_defaults(props)
self.last_body = body = {"class": "NewElementRequestBody", "properties": props}
raw_guid = await self.client.data_designer._async_create_data_value_specification(body)
guid = self.extract_guid_or_raise(raw_guid, "Create Data Value Specification")
Expand Down Expand Up @@ -539,22 +563,22 @@ async def apply_changes(self) -> str:
"displayName": display_name,
"description": attributes.get('Description', {}).get('value'),
"namespacePath": attributes.get('Namespace', {}).get('value'),
"matchPropertyNames": attributes.get('Match Property Names', {}).get('value', []),
"matchThreshold": attributes.get('Match Threshold', {}).get('value', 0),
"matchPropertyNames": attributes.get('Match Property Names', {}).get('value'),
"matchThreshold": attributes.get('Match Threshold', {}).get('value'),
"specification": attributes.get('Specification', {}).get('value'),
"specificationDetails": attributes.get('Specification Details', {}).get('value', {}),
"specificationDetails": attributes.get('Specification Details', {}).get('value'),
"dataType": attributes.get('Data Type', {}).get('value'),
"allowsDuplicateValues": attributes.get('Allow Duplicate Values', {}).get('value', True),
"allowsDuplicateValues": attributes.get('Allow Duplicate Values', {}).get('value'),
"isCaseSensitive": attributes.get('Is Case Sensitive', {}).get('value'),
"isNullable": attributes.get('Is Nullable', {}).get('value', True),
"isNullable": attributes.get('Is Nullable', {}).get('value'),
"defaultValue": attributes.get('Default Value', {}).get('value'),
"averageValue": attributes.get('Average Value', {}).get('value'),
"valueList": attributes.get('Value List', {}).get('value'),
"valueRangeFrom": attributes.get('Value Range From', {}).get('value'),
"valueRangeTo": attributes.get('Value Range To', {}).get('value'),
"sampleValues": attributes.get('Sample Values', {}).get('value', []),
"dataPatterns": attributes.get('Data Patterns', {}).get('value', []),
"additionalProperties": attributes.get('Additional Properties', {}).get('value', {})
"sampleValues": attributes.get('Sample Values', {}).get('value'),
"dataPatterns": attributes.get('Data Patterns', {}).get('value'),
"additionalProperties": attributes.get('Additional Properties', {}).get('value')
}

# 2. Relationships
Expand Down Expand Up @@ -600,6 +624,7 @@ async def apply_changes(self) -> str:
return await self.render_result_markdown(guid)

elif verb == "Create":
props = _with_create_defaults(props)
self.last_body = body = {"class": "NewElementRequestBody", "properties": props}
raw_guid = await self.client.data_designer._async_create_data_class(body)
guid = self.extract_guid_or_raise(raw_guid, "Create Data Class")
Expand Down
13 changes: 9 additions & 4 deletions md_processing/v2/glossary.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,10 +142,10 @@ async def apply_changes(self) -> str:
prop_body["user_defined_status"] = attributes.get('UserDefinedStatus', {}).get('value', None)
prop_body["contextDescription"] = attributes.get('Context Description', {}).get('value', None)
prop_body["contextScope"] = attributes.get('Context Scope', {}).get('value', None)
prop_body["isAbstractConcept"] = attributes.get('Is Abstract Concept', {}).get('value', False)
prop_body["isActivityDescription"] = attributes.get('Is Activity Description', {}).get('value', False)
prop_body["isContext"] = attributes.get('Is Context', {}).get('value', False)
prop_body["isDataValue"] = attributes.get('Is Data Value', {}).get('value', False)
prop_body["isAbstractConcept"] = attributes.get('Is Abstract Concept', {}).get('value')
prop_body["isActivityDescription"] = attributes.get('Is Activity Description', {}).get('value')
prop_body["isContext"] = attributes.get('Is Context', {}).get('value')
prop_body["isDataValue"] = attributes.get('Is Data Value', {}).get('value')
prop_body["termActivityType"] = attributes.get('Term Activity Type', {}).get('value', None)

# 2. Extract collection GUIDs
Expand Down Expand Up @@ -193,6 +193,11 @@ async def apply_changes(self) -> str:

elif verb == "Create":
self.last_body = body = set_create_body(om_type or "GlossaryTerm", attributes)
# ISSUE-162: the term flags default to False only on create; on update an unset
# flag stays None so filter_update_properties drops it and Egeria keeps the value.
for flag in ("isAbstractConcept", "isActivityDescription", "isContext", "isDataValue"):
if prop_body.get(flag) is None:
prop_body[flag] = False
body["properties"] = prop_body

# Anchor Scope check
Expand Down
116 changes: 116 additions & 0 deletions tests/micro-tests/test_issue_162_update_does_not_reset_properties.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
"""
ISSUE-162: an Update that doesn't mention a property must not change it.
- Update Glossary Term sent isAbstractConcept/isActivityDescription/isContext/isDataValue
= False whenever the document omitted them.
- Update Data Value Specification / Data Class sent allowsDuplicateValues/isNullable = True
and empty lists/maps/0 for several properties, so an update reset them.
Defaults now apply only on Create. No live server: the client is mocked.
"""
from unittest.mock import AsyncMock

import pytest

from md_processing.v2.data_designer import DataClassProcessor, DataValueSpecificationProcessor
from md_processing.v2.extraction import DrECommand
from md_processing.v2.glossary import TermProcessor

GUID = "11111111-1111-1111-1111-111111111111"
TERM_FLAGS = ("isAbstractConcept", "isActivityDescription", "isContext", "isDataValue")
DV_DEFAULTED = {
"matchPropertyNames": [], "matchThreshold": 0, "specificationDetails": {},
"allowsDuplicateValues": True, "isNullable": True, "sampleValues": [],
"dataPatterns": [], "additionalProperties": {},
}


def _processor(cls, verb, object_type, attributes, client, om_type=None, guid=None):
cmd = DrECommand(verb=verb, object_type=object_type, attributes=attributes, raw_block=f"{verb} {object_type}")
proc = cls(client, cmd)
proc.parsed_output = {"attributes": attributes, "qualified_name": attributes["Qualified Name"]["value"],
"valid": True}
if guid:
proc.parsed_output["guid"] = guid
if om_type:
proc.get_command_spec = lambda: {"OM_TYPE": om_type}
proc.render_result_markdown = AsyncMock(return_value="# Result")
return proc


def _term_attrs(**extra):
attrs = {"Display Name": {"value": "T"}, "Qualified Name": {"value": "GlossaryTerm::T"},
"Description": {"value": "new description"}}
attrs.update(extra)
return attrs


async def test_term_update_without_flags_sends_none_of_them():
client = AsyncMock()
client._async_get_related_elements.return_value = []
proc = _processor(TermProcessor, "Update", "Glossary Term", _term_attrs(), client, guid=GUID)
await proc.apply_changes()

body = client._async_update_glossary_term.call_args.args[1]
for flag in TERM_FLAGS:
assert flag not in body["properties"], flag
assert body["properties"]["description"] == "new description"


async def test_term_update_sends_a_flag_the_document_sets():
client = AsyncMock()
client._async_get_related_elements.return_value = []
proc = _processor(TermProcessor, "Update", "Glossary Term", _term_attrs(**{"Is Context": {"value": True}}),
client, guid=GUID)
await proc.apply_changes()

props = client._async_update_glossary_term.call_args.args[1]["properties"]
assert props["isContext"] is True
assert "isAbstractConcept" not in props


async def test_term_create_still_defaults_flags_to_false():
client = AsyncMock()
client._async_create_glossary_term.return_value = GUID
proc = _processor(TermProcessor, "Create", "Glossary Term", _term_attrs(), client)
await proc.apply_changes()

call = client._async_create_glossary_term.call_args
props = (call.kwargs.get("body") or call.args[0])["properties"]
for flag in TERM_FLAGS:
assert props[flag] is False, flag


def _dv_attrs():
return {"Display Name": {"value": "D"}, "Qualified Name": {"value": "DataValueSpecification::D"},
"Description": {"value": "changed"}}


@pytest.mark.parametrize("cls, om_type", [(DataValueSpecificationProcessor, "DataValueSpecification"),
(DataClassProcessor, "DataClass")])
async def test_data_designer_update_does_not_send_defaults(cls, om_type):
client = AsyncMock()
proc = _processor(cls, "Update", "Data Value Specification", _dv_attrs(), client, om_type=om_type, guid=GUID)
proc._sync_value_spec_parent = AsyncMock()
proc._sync_all_rels = AsyncMock()
await proc.apply_changes()

props = client.data_designer._async_update_data_value_specification.call_args.args[1]["properties"]
for key in DV_DEFAULTED:
# None is dropped by body_slimmer before sending; anything else would overwrite Egeria's value
assert props.get(key) is None, key
assert props["description"] == "changed"


@pytest.mark.parametrize("cls, om_type, create_method", [
(DataValueSpecificationProcessor, "DataValueSpecification", "_async_create_data_value_specification"),
(DataClassProcessor, "DataClass", "_async_create_data_class")])
async def test_data_designer_create_keeps_defaults(cls, om_type, create_method):
client = AsyncMock()
getattr(client.data_designer, create_method).return_value = GUID
proc = _processor(cls, "Create", "Data Value Specification", _dv_attrs(), client, om_type=om_type)
proc._sync_value_spec_parent = AsyncMock()
proc._sync_all_rels = AsyncMock()
await proc.apply_changes()

props = getattr(client.data_designer, create_method).call_args.args[0]["properties"]
for key, default in DV_DEFAULTED.items():
assert props[key] == default, key
Loading