[FScholPer]

📌 Description

Centralize metrics calculation as addon to the existing schema and remove inline calculation

🚨 Impact Analysis

• This change does not violate any tool requirements and is covered by existing tool requirements
• This change does not violate any design decisions
• Otherwise I have created a ticket for new tool qualification

✅ Checklist

• Added/updated documentation for new or changed features
• Added/updated tests to cover the changes
• Followed project coding standards and guidelines

[github-actions]

Documentation preview for this pull request is available at:
pr-949: https://eclipse-score.github.io/docs-as-code/pr-949/

[antonkri]

Hi @FScholPer, thanks for centralizing the metrics — the direction makes a lot of sense. One issue though: the version scoping for the Component Requirements pie charts does not work as intended and silently regresses current behaviour.

What happens

module_verification_report.need now passes version="{{ module_version or '' }}" into the filter funcs, and sphinx_filters.get_component_status_metrics / get_component_coverage_metrics look up:

metrics_key = f"verification_metrics_v{version}"
metrics = CALCULATED_METRICS.get(metrics_key, {})
if not metrics:
    metrics = CALCULATED_METRICS.get("verification_metrics", {})

But nothing ever writes a verification_metrics_v* key. The only producer call in calculate_full_need_metrics is:

"verification_metrics": calculate_verification_metrics(all_needs),

i.e. without report_version, so calculate_verification_metrics(all_needs, report_version=None) always produces a single unversioned block. The fallback therefore always kicks in, and the version argument is a no-op.

Effect

For a report with applies_to_module_version set, the Component Requirements Status / Test Coverage pies now count all requirements instead of only those in scope for the report version. Previously this was handled by component_req_filter (id in [...], built from req_in_report_version). The requirements needtable right below still uses component_req_filter, so chart and table of the same component will show contradicting numbers.

The architecture pies are unaffected in outcome — req_in_version ends in return True for anything that is not a comp_req and has no valid_from, so they stay unversioned as before. That is the correct result, but it reads as accidental: the call site passes version_filter=report_version for arch as well, which suggests filtering is intended there.

Suggestion — one of the two, not both:

1. Actually produce versioned metrics: collect the applies_to_module_version values present in the build and store calculate_verification_metrics(all_needs, v) under verification_metrics_v<v>, then drop the silent fallback so a missing key surfaces instead of returning wrong numbers.
2. Or drop the version/report_version parameter entirely and document the pies as unscoped — but then the requirements needtable needs to be unscoped too, otherwise chart and table disagree.

One more thing: req_in_version in traceability_metrics.py re-implements _RequirementInReportVersion from score_sphinx_needs_templates/__init__.py. The two already differ in the architecture case (return True vs. any([]) → False). Two sources of truth for the same rule will drift — could the existing helper be reused instead?

Minor: component_ids is collected from every belongs_to in the graph, so feat__* IDs (from feat_arc_sta/feat_arc_dyn) also end up as keys with empty requirements/architecture blocks in metrics.json.

[antonkri]

see comment

[antonkri]

Thanks for the rework in e8d1371 — routing the version scoping through the now-public req_in_report_version is exactly the right call, and it removes the dead verification_metrics_v* lookup I flagged earlier. Two follow-ups though, this one is a blocker.

Architecture pies go empty as soon as a report version is set

RequirementInReportVersion.__call__ excludes any Need that has neither a valid_from of its own nor a resolvable derived_from:

valid_from = need.get("valid_from")
if valid_from:
    ...
linked_feat_reqs = _linked_needs_callable(need["id"], "derived_from")
return any(self(feat_req, report_version) for feat_req in linked_feat_reqs)

That rule is correct for requirements, but architecture Needs satisfy neither condition. Per metamodel.yaml, comp_arc_sta / comp_arc_dyn only have belongs_to (mandatory), uses and fulfils — no valid_from, no derived_from. So any([]) is False and every architecture element is dropped.

The template passes the version to those pies as well:

:filter-func: ...get_component_status_metrics(component_id="...", category="architecture", version="{{ module_version or '' }}")

Effect: for any report with applies_to_module_version set, Component Architecture Statistics renders 0/0/0 in both pies, while the Architectural Elements needtable right below still lists every element — the filter there is not version-scoped. Without a version the pies take the CALCULATED_METRICS branch and look fine, so this only shows up on versioned reports.

Worth noting that your own earlier req_in_version in traceability_metrics.py had return True in this branch, so the switch to the shared helper is what introduced it.

Architecture elements have never been version-scoped — neither in the pies nor in the tables — and there is no attribute to scope them by. Simplest fix is to only apply the version for category == "requirements", or to not pass version for the architecture pies at all, so chart and table keep agreeing.

[antonkri]

The "Test Coverage" pie changes meaning depending on whether a version is set

There are two independent traceability axes in this repo, and the two branches of get_component_coverage_metrics now use a different one each.

(a) Verification coverage — fully_verifies_back / partially_verifies_back: backlinks from test case Needs that declare fully_verifies / partially_verifies. This answers "is this requirement verified by a test, fully or only partially?"

(b) Link coverage — testlink / source_code_link: both populated by score_source_code_linker, pointing at implementation code and at a test case harvested from the JUnit XMLs. This answers "is this requirement linked to code and/or to a test?" It already has its own chart — see the docstring of get_non_overlapping_link_metrics, which lists exactly these four buckets, including "Fully linked — has both testlink and source_code_link".

Where each one is used today:

path	code	axis
template on main	... and fully_verifies_back / ... and partially_verifies_back and not fully_verifies_back / ... and not ... and not ...	(a)
calculate_status_and_coverage_metrics (taken when no version is set)	has_fully = bool(need.get("fully_verifies_back"))	(a)
_count_coverage_filtered (taken when a version is set)	has_testlink and has_code_link -> fully, or -> partially	(b)

So the same pie, with the same title and the same fully covered / partially covered / not covered labels, measures verification coverage on unversioned reports and link coverage on versioned ones.

Concrete consequences:

1. Numbers jump when a version is set. A requirement verified by a fully_verifies test case but without annotated implementation code counts as fully covered without a version and as partially covered with one.
2. partially_verifies disappears. Axis (b) has no notion of it — "partially" there means "only one of the two links present". A requirement that is only partially verified but has both links gets promoted to fully covered.
3. Dependence on build inputs. testlink / source_code_link are only populated when the source code linker has data (tests-report / bazel-testlogs). Without it the versioned pies would read 100% not covered while the unversioned ones still look plausible.

To be explicit about the intent: in this chart we only want the test/verification metric, i.e. axis (a). Link coverage is a separate concern and already has its own visualisation, so it should not leak into the verification report. Switching _count_coverage_filtered to fully_verifies_back / partially_verifies_back makes all three paths agree again.

[MaximilianSoerenPollak]

I think we need a meeting before we put too much work into the wrong direction that then in the end is wasted.
We currently have the metrics.json and also the mod_ver_report etc. We need to decide and agree on where the info is coming from how to deal with etc.

So I wouldn't put too much effort into any one direction now without having a decision internally in Docs-As-Code of how we want to do this.

[FScholPer]

I think we need a meeting before we put too much work into the wrong direction that then in the end is wasted. We currently have the metrics.json and also the mod_ver_report etc. We need to decide and agree on where the info is coming from how to deal with etc.

So I wouldn't put too much effort into any one direction now without having a decision internally in Docs-As-Code of how we want to do this.

So this is exactly the point enrich the metrics.json with verification metrics and move the calculation outside to helpers so that this can be used in both areas

[FScholPer]

@antonkri can you please check again if it fits now?

[Copilot]

🟡 Changes recommended

Callback argument handling produces zero-count charts, and inherited version filtering accesses needs after they become read-only.

2 open findings

• Read-only chart evaluation fails on mutable needs access · New
• Quoted filter arguments cause all chart metrics to return zero · New

3 resolved since last review

• Collect component IDs from component Needs
• Decode filter arguments using arg1, arg2, and arg3
• Add regression tests for per-component metrics

🧠 Review effort: Balanced

[antonkri]

Hi @FScholPer, thanks for the rework — the two points from my last review (architecture pies filtered by version, coverage axis) are fixed. 👍

While re-checking I found a few more things and, to keep this moving, pushed two commits on top of your branch. Short reasoning:

481318c – restore the ASIL/QM module report + decode filter-func arguments

• The main merge (a539af8) kept this branch's module_verification_report.need, which silently reverted feat: split module report pies into ASIL and QM #946 (ASIL B/QM split, legend, sorting). Restored main's template.
• sphinx-needs passes :filter-func: arguments verbatim after split(","), so ("comp__x", "requirements", "") arrived as '"comp__x"', ' "requirements"', ' ""' → every pie counted 0. Added _arg() to strip whitespace/quotes.
• sphinx-needs resolves the function with rsplit(".", 1) over the whole call, so an argument with a dot (v1.0) makes the call fail ("malformed function signature") and the pie is dropped. Documented + test.
• Removed the unused version_filter path in calculate_status_and_coverage_metrics (it duplicated req_in_report_version).

553147b – count the module report pies with your central filter functions

• The module report now uses get_component_status_metrics / get_component_coverage_metrics for its pies, wired through safety_pie_row(component=…, category=…, version=…), so the feat: split module report pies into ASIL and QM #946 layout stays.
• Added arg4 = safety class to the filter functions and safety to the shared count helpers. QM counts valid_inspected as valid, ASIL lists valid_inspected first — same as the existing labels/colours.
• The version is passed as v1_0 and decoded back to v1.0 (dot problem above).
• The filter functions now always count the needs handed in by needpie and no longer read metrics.json, so the report does not depend on the open metrics.json discussion. Your verification_metrics block in metrics.json is untouched; whether it stays here or moves to a follow-up is up to you / the meeting @MaximilianSoerenPollak proposed.
• A test renders safety_pie_row both ways and asserts the filter functions count exactly what the inline filters count (per safety class, with version scope).

Locally: 48 tests green, ruff clean. The platform report still uses inline filters; it could follow later. Happy to adjust if you see it differently.