fix(evaluations): stop live eval billing for ignored signals by cursor[bot] · Pull Request #4224 · latitude-dev/latitude-llm

cursor · 2026-07-25T08:17:12Z

What does this PR do?

Fixes ongoing live-eval billing for ignored signals after the #4132 lifecycle restore.

The muted→ignored backfill migration (20260720152609) stamped ignored_at but did not archive linked evaluations the way a user-initiated ignore does via softDeleteBySignalId. Backfilled signals kept matching and billing live evals while appearing archived in the UI.

This PR adds a follow-up migration to soft-delete evaluations for ignored signals, and skips live eval execution when the linked signal has ignoredAt set (covers in-flight jobs enqueued before ignore).

Related issue (if applicable)

Follow-up to #4132 (specs/signal-resolve-ignore.md §6.5 backfill parity).

How was this tested?

Added regression test: live eval skips with reason: "signal-ignored" and records no billing events when the linked signal is ignored.
pnpm --filter @domain/evaluations test -- run-live-evaluation.test.ts (181 tests passed).

Checklist

Lint, type-checking, and tests pass locally
PR title follows Conventional Commits
I have signed the CLA

Summary by CodeRabbit

Bug Fixes
- Evaluations linked to ignored signals are now skipped before billing, execution, metering, or scoring.
- Existing evaluations associated with ignored signals are marked as deleted to keep their status consistent.
Tests
- Added coverage confirming ignored-signal evaluations are skipped and generate no billing usage events.

The muted→ignored backfill migration stamped ignored_at without archiving linked evaluations, so backfilled signals kept running live evals. Add a follow-up migration to soft-delete those evaluations, and skip live eval execution when the linked signal is ignored (including in-flight jobs). Co-authored-by: Gerard <gerard@latitude.so>

vercel · 2026-07-25T08:17:18Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
design-system	Skipped		Jul 25, 2026 8:17am

coderabbitai · 2026-07-25T08:17:20Z

📝 Walkthrough

Walkthrough

Changes

Ignored signal evaluation handling

Layer / File(s)	Summary
Skip ignored signals before billing `packages/domain/evaluations/src/use-cases/live/run-live-evaluation.ts`, `packages/domain/evaluations/src/use-cases/live/run-live-evaluation.test.ts`	Live evaluations return `signal-ignored` before billing, session loading, script execution, metering, or scoring; the test verifies the skip result and absence of billing events.
Backfill soft-deleted evaluations `packages/platform/db-postgres/drizzle/20260725081621_backfill-ignored-signal-eval-soft-delete/migration.sql`, `packages/platform/db-postgres/drizzle/.migration-lock`	A migration soft-deletes active evaluations linked to ignored, non-deleted signals and updates migration metadata.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: neoxelox, geclos

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: live evaluation billing is stopped for ignored signals.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch cursor/critical-bug-management-b7b6

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/domain/evaluations/src/use-cases/live/run-live-evaluation.ts`:
- Around line 355-363: Move the signal lookup and the issue.ignoredAt guard
before the embedding-readiness wait in the live evaluation flow, preserving the
existing signal-ignored result. Ensure ignored evaluations return immediately
with action "skipped" even when embeddings are unavailable, and add coverage for
an ignored evaluation whose script requires embeddings.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: fda95a90-015b-4429-9a0d-6b7af34db557

📥 Commits

Reviewing files that changed from the base of the PR and between e80aed6 and 629f6be.

📒 Files selected for processing (4)

packages/domain/evaluations/src/use-cases/live/run-live-evaluation.test.ts
packages/domain/evaluations/src/use-cases/live/run-live-evaluation.ts
packages/platform/db-postgres/drizzle/.migration-lock
packages/platform/db-postgres/drizzle/20260725081621_backfill-ignored-signal-eval-soft-delete/migration.sql

coderabbitai · 2026-07-25T14:52:18Z

+    if (issue.ignoredAt !== null) {
+      return {
+        action: "skipped",
+        reason: "signal-ignored",
+        evaluationId: input.evaluationId,
+        traceId: input.traceId,
+      } satisfies RunLiveEvaluationResult
+    }
+


🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Check ignoredAt before the embedding wait.

For an ignored evaluation whose script requires embeddings, this branch is not reached while embeddings are unavailable; the job is requeued and returns awaiting-embeddings instead. Move the signal lookup/guard before the readiness gate and add coverage for this path.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/domain/evaluations/src/use-cases/live/run-live-evaluation.ts` around lines 355 - 363, Move the signal lookup and the issue.ignoredAt guard before the embedding-readiness wait in the live evaluation flow, preserving the existing signal-ignored result. Ensure ignored evaluations return immediately with action "skipped" even when embeddings are unavailable, and add coverage for an ignored evaluation whose script requires embeddings.

claude · 2026-07-25T14:53:13Z

📄 Generated an HTML walkthrough of this PR's diff against development (background on the signal resolve/ignore restore + why the muted→ignored backfill left evaluations un-archived, an intuition section, a code walkthrough of the follow-up migration and the new signal-ignored skip gate, and a 5-question interactive quiz).

Published artifact: https://artifact-pr-4224-diff-review.magnetic-toothpaste.workers.dev

Claim within 60 minutes: https://dash.cloudflare.com/claim-preview?claimToken=e-MWG31ED71v2LsCed08AcTuwh9EvgV75aMzV49ZNvU

Self-contained artifact, no external requests. Generated for review purposes only.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 629f6befe3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-25T14:53:23Z

+    if (issue.ignoredAt !== null) {
+      return {
+        action: "skipped",
+        reason: "signal-ignored",
+        evaluationId: input.evaluationId,
+        traceId: input.traceId,
+      } satisfies RunLiveEvaluationResult


Move ignored-state check before embedding readiness

When an ignored signal uses semanticSimilarity() and the triggering trace lacks a current-model embedding, the readiness gate returns before reaching this check. Each queued evaluation can therefore publish up to three delayed retries and report awaiting-embeddings/embeddings-unavailable instead of signal-ignored, continuing unnecessary queue and database work for roughly six minutes. Load the signal and apply this guard before the embedding-readiness gate.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-25T14:53:23Z

+-- The muted→ignored backfill (20260720152609) stamped ignored_at but did not archive linked
+-- evaluations the way applySignalLifecycleCommand("ignore") does via softDeleteBySignalId.


Remove the transient migration commentary

These two lines narrate the preceding backfill and the implementation path that motivated this migration rather than documenting a lasting SQL constraint. That context becomes stale as migrations accumulate, and the repository explicitly requires comments to be rare, one line, and not anchored to transient development history; remove it or replace it with a single-line invariant that the SQL itself cannot express.

AGENTS.md reference: AGENTS.md:L14-L14

Useful? React with 👍 / 👎.

github-project-automation Bot added this to Latitude LLM Roadmap Jul 25, 2026

github-project-automation Bot moved this to Inbox in Latitude LLM Roadmap Jul 25, 2026

geclos marked this pull request as ready for review July 25, 2026 14:47

coderabbitai Bot reviewed Jul 25, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Jul 25, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(evaluations): stop live eval billing for ignored signals - #4224

fix(evaluations): stop live eval billing for ignored signals#4224
cursor[bot] wants to merge 1 commit into
developmentfrom
cursor/critical-bug-management-b7b6

cursor Bot commented Jul 25, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

vercel Bot commented Jul 25, 2026

Uh oh!

coderabbitai Bot commented Jul 25, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Uh oh!

coderabbitai Bot Jul 25, 2026

Uh oh!

claude Bot commented Jul 25, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jul 25, 2026

Uh oh!

chatgpt-codex-connector Bot Jul 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		-- The muted→ignored backfill (20260720152609) stamped ignored_at but did not archive linked
		-- evaluations the way applySignalLifecycleCommand("ignore") does via softDeleteBySignalId.

Uh oh!

Conversation

cursor Bot commented Jul 25, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Related issue (if applicable)

How was this tested?

Checklist

Summary by CodeRabbit

Uh oh!

vercel Bot commented Jul 25, 2026

Uh oh!

coderabbitai Bot commented Jul 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jul 25, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot commented Jul 25, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 25, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 25, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

cursor Bot commented Jul 25, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 25, 2026 •

edited

Loading