UN-4387 [FEAT] Dashboard API for failing pipelines and deployments, and failed pages by source

[muhammad-ali-e]

What

• GET /api/v2/unstract/<org>/metrics/attention/ lists the items that need attention on the dashboard. OSS ships two kinds:
  • failing_pipeline: active ETL and Task pipelines whose last 5 finished runs all ended in ERROR.
  • failing_deployment: active API deployments whose last 5 finished runs all ended in ERROR.
• GET /api/v2/unstract/<org>/metrics/attribution/ gives failed pages for the range, split by source (API deployment, ETL or Task pipeline, workflow, deleted) and by day.
• Both responses have a JSON Schema contract in backend/dashboard_metrics/contract/.

Why

• PB-T14 of the Frontend v1 epic (UN-4257). The new dashboard's "Needs attention" panel and its failure breakdown need this data, and no endpoint returns it today.

How

• workflow_manager/workflow_v2/failing_streaks.py finds failing rows in one query. The failing test and the date range run in SQL first, on the (pipeline_id, -created_at) index, so the streak lookups only run for rows that are failing. For each failing row it reports:
  • how many finished runs in a row have failed (since the newest finished run that was not an ERROR)
  • when that streak began
  • the latest failure and the last success
• Runs still pending or executing are skipped, so a queued retry does not hide a row that keeps failing. A STOPPED run ends a streak.
• The rule is ERROR only, like the list status dots, not is_failure_run (which also counts STOPPED and partly failed runs). The module docstring says why.
• dashboard_metrics/attention.py has a provider registry. OSS registers its two providers in DashboardMetricsConfig.ready(). Cloud apps register their own kinds from their own ready() in a separate unstract-cloud PR. Each provider's items are checked against the contract's types and serialized inside its own guard. If a provider raises or returns a malformed item, the feed still returns, and that provider's kind appears in failed_kinds, so "nothing needs attention" is never a silent error.
• Items carry a route of {key, params}: etl_pipeline.detail, task_pipeline.detail and api_deployment.detail, the keys proposed by the global search work (UN-4402 [FEAT] Global search API #2343, not yet merged).
• dashboard_metrics/attribution.py groups MetricsQueryService.failed_file_executions_with_pages(). get_failed_pages now builds the same queryset with the same SQL, so attribution totals always equal the failed_pages series.
• Both actions reuse MetricsQuerySerializer, so they take start_date/end_date (default 30 days, max 365), and keep the existing permission and throttle classes. Neither response is cached.

Defaults chosen where the spec was open:

• The params are start_date/end_date, matching the other metrics endpoints, rather than the spec's from/to.
• Inactive pipelines and deployments are excluded.
• An item appears when its latest failure falls in the range.
• Visibility is org-wide, as in the existing dashboard endpoints.

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

• No. The PR adds two read-only endpoints and new modules. The only existing code it changes is get_failed_pages, which now builds its queryset through a shared helper. The SQL is unchanged and every existing dashboard_metrics test passes.

Database Migrations

• None.

Env Config

• None.

Relevant Docs

• backend/dashboard_metrics/README.md, section "Attention and attribution (PB-T14)".

Related Issues or PRs

• UN-4387 (PB-T14), epic UN-4257.
• Route keys proposed in UN-4402 [FEAT] Global search API #2343.

Dependencies Versions

• None.

Notes on Testing

• dashboard_metrics/tests/test_attention_attribution.py adds 19 tests covering:
  • the streak rules: lower bound, stops, unfinished runs, range bounds, and a check that it runs as one query
  • provider order in the feed
  • the provider registry, on test-only kinds with the registry restored after each test: sorting, a provider that raises, and malformed items
  • attribution totals matching get_failed_pages
  • 403 with no organization
• Responses are validated against the schemas with a format checker.
• dashboard_metrics: 212 passed.
• The repo's pre-commit hooks pass on the PR's files.

🤖 Generated with Claude Code

https://claude.ai/code/session_019Ja7fuC73jWYRnA1ttdLD1

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[muhammad-ali-e]

Self-review of #2352 — round 1 (INITIAL), tip cb92b16

Verdict — REQUEST CHANGES

Summary — Critical: 0 · High: 1 · Medium: 9 · Low: 4 · Lenses run: 19/19 (5, 6, 11, 12, 14, 18, 19 N/A with reason)

Security deep-dive: done (no findings at confidence >= 8)

Findings

1. [High] [Lens 3] — An in-flight run hides a failing row from the feed

• Location: backend/workflow_manager/workflow_v2/failing_streaks.py:L56-L70
• Failure mode: recent is the last five runs of any status. A pipeline that has failed 20 times and has one PENDING/EXECUTING run drops out of the feed until that run ends, and _last_failure becomes the in-flight run's time.
• Suggested fix: restrict recent to terminal statuses and flip test_not_failing's in-flight case.
• Confidence: High · VERIFIED — read the query; the existing test pins the behaviour.

2. [Medium] [Lens 3] — "Failures in a row" counts across STOPPED runs

• Location: backend/workflow_manager/workflow_v2/failing_streaks.py:L63-L67
• Failure mode: the streak is every ERROR after the last COMPLETED, so ERROR, STOPPED, ERROR reports 2 in a row and since predates the STOPPED run.
• Suggested fix: bound the streak by the newest terminal non-ERROR run.
• Confidence: High · VERIFIED — read the filter.

3. [Medium] [Lens 2] — The failure rule differs from is_failure_run without saying why

• Location: backend/workflow_manager/workflow_v2/failing_streaks.py:L1-L6
• Failure mode: the canonical rule (unstract/core/data_models.py:626) counts STOPPED and partial-file failures; this counts ERROR only. A maintainer could "fix" one toward the other.
• Suggested fix: document that this follows the list status dots, not is_failure_run.
• Confidence: Medium (product call) · VERIFIED — compared both definitions.

4. [Medium] [Lens 9] — Every active row pays five correlated subqueries; the range filter runs in Python

• Location: failing_streaks.py:L68-L83; backend/dashboard_metrics/attention.py:L141-L148
• Failure mode: the uncached endpoint computes streak count/start (unbounded scans for never-succeeded rows) for healthy rows, then discards them in Python.
• Suggested fix: filter on the five-ERROR array and the range in SQL before the costly annotations.
• Confidence: Medium · UNVERIFIED — would EXPLAIN ANALYZE on a large org confirm the cost?

5. [Medium] [Lens 8] — One bad provider item can 500 the whole feed

• Location: attention.py:L113-L121; backend/dashboard_metrics/views.py:L980
• Failure mode: to_dict() runs in the view, outside the per-provider guard; order is never validated.
• Suggested fix: serialize inside the try; validate order at registration.
• Confidence: High · VERIFIED — read both sites.

6. [Medium] [Lens 10] — The provider-failure log has no org or range — attention.py:L119. Add both. VERIFIED.

7. [Medium] [Lens 16, 17] — "Shared with PB-T13" is false — failing_streaks.py:L8-L9, README.md:L728-L729. The lists still use get_last_run_statuses. Delete the claim. VERIFIED by grep.

8. [Medium] [Lens 7, 16] — Cloud kind names and the route registry are stated as existing — attention.py:L4-L6, L35-L36; README.md:L724, L730. reviews_waiting / challenges_unresolved exist nowhere; route keys exist only in unmerged #2343. VERIFIED by grep across both repos.

9. [Medium] [Lens 13] — Registry tests use real cloud kind names on a process-global registry and assert the full list — tests/test_attention_attribution.py:L222-L270. Once a cloud app registers reviews_waiting, setUp raises and cleanup removes the real provider. Use test-only kinds, snapshot/restore, relative order. VERIFIED.

10. [Medium] [Lens 13] — The streak's lower bound is untested — tests/test_attention_attribution.py:L110-L140. No fixture has ERRORs before the last success; the ordering test can't tell grouping from a global sort. VERIFIED.

11. [Low] [Lens 13] — Schema checks ignore format — tests/test_attention_attribution.py:L160. Pass format_checker. VERIFIED.

12. [Low] [Lens 13] — end_date bound untested — tests/test_attention_attribution.py:L213. VERIFIED.

13. [Low] [Lens 16] — README "same defaults and 365-day cap as above" has no antecedent — README.md:L691. VERIFIED.

14. [Low] [Lens 15] — ruff 0.3.4 would reformat the test file — VERIFIED with ruff format --check --diff.

Open questions

• Visibility is org-wide (any member sees every pipeline name), as in existing dashboard endpoints (services.py:726-739). UNVERIFIED — is that intended for non-admin members?

Lens checklist

1 Clean · 2 #3 · 3 #1, #2 · 4 Clean (security pass done) · 5 N/A no writes · 6 N/A read-only · 7 #8 (no consumers yet) · 8 #5 · 9 #4 · 10 #6 · 11 N/A additive · 12 N/A no model calls · 13 #9–#12 · 14 N/A no new deps · 15 #14 · 16 #7, #8, #13 · 17 #7 · 18 N/A · 19 N/A

Fixes follow in the next push.

---

Generated by Claude Code

[muhammad-ali-e]

Self-review of #2352 — round 2 (FOLLOWUP), delta cb92b16..da7c2fe

Verdict — APPROVE WITH NITS

All round 1 findings are resolved:

• #1, #2: only finished runs count, and the streak ends at the newest finished non-ERROR run. Pinned by test_unfinished_runs_are_skipped and the "a stop" subtest (old code gives 6 failures since 9h, new gives 5 since 7h).
• #4: the five-ERROR array and range filters are in the WHERE clause (str(qs.query) checked; Django casts to varchar[]), so the streak subqueries run only for failing rows.
• #3, #5–#14: fixed as described in 09a1272. Every new guard was mutation-checked.
• pre-commit.ci's pyupgrade rewrite (da7c2fe) is accepted as is.

New findings

1. [Medium] [Lens 13] — test_display_order passes even when providers are not sorted

• Location: backend/dashboard_metrics/tests/test_attention_attribution.py:L339-L345
• Failure mode: every test provider is registered in ascending order, so a registry that returned insertion order would pass. Replacing sorted with list kept all 19 tests green.
• Suggested fix: register a low-order provider last and assert it sorts first.
• VERIFIED by mutation.

2. [Low] [Lens 16] — _FailingRowsProvider docstring still states the old rule — backend/dashboard_metrics/attention.py:L153. VERIFIED.

3. [Low] [Lens 16] — collect_attention docstring omits the count check; _serialize does not check id or since types — backend/dashboard_metrics/attention.py:L118-L150. VERIFIED.

Fixes follow in the next push.

---

Generated by Claude Code

[muhammad-ali-e]

Self-review of #2352 — round 3 (FOLLOWUP), delta da7c2fe..6bd9bd6

Verdict — APPROVE

Round 2 findings resolved:

• #1: a test provider with order = 5 is registered last and must sort first. Replacing sorted with insertion order now fails the test.
• #2: the failing-rows provider docstring now points at failing_streaks for the rule.
• #3: _serialize rejects a non-string id, a non-string name, and a since that is not a datetime (a date included), and the docstring says so. Each check was mutation-tested.

No new findings at Medium or above. One nit fixed in 6bd9bd6: the type error now names both id and name.

Checks: dashboard_metrics 212 passed; the repo's pre-commit hooks pass on the PR's files.

The loop stops here; the PR stays in draft.

---

Generated by Claude Code

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud