feat(retry): Retry webhooks, client LLM proxy and guardrails calls by Prajna1999 · Pull Request #1275 · ProjectTech4DevAI/kaapi-backend

Prajna1999 · 2026-10-19T12:27:57Z

Summary

Kaapi's own outbound HTTP calls now retry transient failures 3 times (4 attempts) using tenacity's @retry directly. Provider SDK calls keep each SDK's default retry: OpenAI, Anthropic and Sarvam retry twice, and ElevenLabs doesn't retry.

Secondary; lock litellm version >=1.83.10,<1.90to prevent future security exploits.

 send_callback                        # webhooks
+  @retry: any non-2xx or RequestException, HMAC re-signed per attempt
 execute_llm_call
   _execute_proxy_call
+    _post_to_proxy                   # client LLM proxy
+      @retry: 5xx / connect failure (never ReadTimeout — may already be billed)
   provider.execute                   # unchanged: SDK default retries
 run_guardrails_validation
+  @retry: 5xx / connect failure → then existing fail-open
 list_validators_config
+  @retry: 5xx / connect failure
 proxy_guardrails_request
+  @retry: connect failure only (a POST is never re-sent after it landed)

The shared predicate is app/core/retry.py:

def is_transient(error: BaseException) -> bool:
    """5xx from upstream, or a connection that never reached it."""

Never retried: 4xx, 429, read timeouts, guardrails auth errors (401/403/422 still fail closed).

A proxy failure that was already retried returns BlockResult.retryable=False, so fast-eval retry_llm_call doesn't stack 4 more attempts on top.

Evidence

Before: send_callback against a receiver returning 503 → 1 POST, then a failure callback / False.
After: TestSendCallbackRetry: 302 → ConnectionError → 500 → 200 → True after 4 POSTs; a constant 503 → False after 4 POSTs; each attempt has a fresh X-Webhook-Timestamp.
Before: guardrails validation 503 → bypassed after 1 call.
After: TestGuardrailsRetry: 503 → 200 → validated result; four 503s → bypassed=True; 401 → 1 call, auth_error=True.
Before: proxy ConnectError → job failed after 1 call, retryable=True.
After: TestPostToProxyRetry: ConnectError → 502 → 200 → success on call 3; ReadTimeout and 400 → 1 call.
Affected suites pass with uv run pytest -n 0: 1222 tests (core, test_utils, services/llm, services/guardrails, api/routes/test_guardrails). Pre-commit is clean.

Merge Danger

Door: two-way

The change is decorators plus one small module; no schema, migration or config changes. Reverting the PR restores single-attempt behaviour.

Blast Radius: latency

Callers wait longer on failure. Worst case, a call adds ~3 backoff waits (exponential, capped at 10s each) before it fails.
- On request-path calls, the guardrails management proxy can hold an API response longer.
- On Celery jobs, webhooks and validation hold a worker longer.
Receivers may get duplicate webhooks. A webhook whose response timed out after the receiver processed it is re-sent, so receivers should de-duplicate (at-least-once delivery).
Fast-eval proxy calls hold a DB transaction during waits. Proxy retries run inside execute_llm_call's session. With record_call=False (fast evals), a read transaction stays open across the backoff waits.

🤖 Generated with Claude Code

Add a thin tenacity decorator, retry_thrice, and a shared is_transient predicate in app/core/retry.py. - send_callback retries any non-2xx response or request error and re-signs the payload on each attempt. - The provider call in execute_llm_call retries 5xx errors. - The client-LLM proxy and guardrails calls retry 5xx and connect failures. Already-retried failures are marked retryable=False so fast-eval retries don't stack. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

coderabbitai · 2026-10-19T12:28:27Z

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one): 1

ready-for-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: ProjectTech4DevAI/kaapi-backend/.coderabbit.yaml
Review profile: CHILL
Plan: Advanced
Run ID: 3d0679b9-c8be-420d-931b-aa4da7b293df

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-10-19T12:28:52Z

OpenAPI changes ⚪ No API surface changes

Note

This PR does not modify the API contract.

_{main ↔ 8308aee8 · generated by oasdiff}

@Retry

Drop the retry_thrice wrapper. Call sites now use tenacity's @Retry with stop_after_attempt(3) and exponential backoff. app/core/retry.py keeps only the shared is_transient predicate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Remove the provider-call retry from execute_llm_call. Provider SDK retries move to a follow-up PR stacked on the google-genai SDK update. Provider failures keep retryable=True as before, and is_transient now only classifies HTTP exceptions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Use each SDK's built-in retry: max_retries=3 on the OpenAI and Anthropic clients, and request_options={"max_retries": 3} on Sarvam STT/TTS calls. Verified against a stub server: 4 attempts on 503, and retried Sarvam uploads still carry the file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Match the provider SDKs' max_retries=3 so webhooks, the LLM proxy and guardrails also retry three times after the first attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

codecov · 2026-10-19T13:12:04Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@Retry

The SDK's max_retries is unusable: its counter starts at 2, and streaming TTS never retries. Wrap STT convert and the TTS stream consumption in a tenacity @Retry (4 attempts, 5xx/connect only). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

vprashrex · 2026-10-21T09:46:10Z

-                f"[run_guardrails_validation] Response received | job_id={job_id}, "
-                f"status={response.status_code}, elapsed_ms={elapsed_ms}, "
+
+            @retry(


can combine this into one single decorator I guess, instead of repeating everywhere

It's a decorator. So thought of keeping it staright forward

That was Claude's suggestion. But overrode it to make the retry straight forward

…ElevenLabs Drop the max_retries=3 overrides and the ElevenLabs tenacity wrapper; provider calls go back to each SDK's default retry behaviour. This PR now only retries webhooks, the client LLM proxy and guardrails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

_execute_proxy_call ran inside execute_llm_call's session, so the read transaction (or the one reopened by create_llm_call's refresh) stayed open across the proxy request and its retry backoff, pinning a pooled connection per concurrent fast-eval call. Commit before the request. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

github-actions · 2026-10-22T07:02:32Z

🎉 This PR is included in version 1.8.0-main.31 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

github-actions Bot changed the title ~~feat(retry): Retry webhooks and transient upstream failures~~ feat(retry): Add shared retry layer Oct 19, 2026

Prajna1999 and others added 2 commits October 19, 2026 18:06

Prajna1999 changed the title ~~feat(retry): Add shared retry layer~~ feat(retry): Retry webhooks, client LLM proxy and guardrails calls Oct 19, 2026

Prajna1999 changed the title ~~feat(retry): Retry webhooks, client LLM proxy and guardrails calls~~ feat(retry): Retry webhooks, LLM proxy, guardrails and provider SDK calls Oct 19, 2026

fix(retry): Use 3 retries (4 attempts) for tenacity call sites

0a16118

Match the provider SDKs' max_retries=3 so webhooks, the LLM proxy and guardrails also retry three times after the first attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Prajna1999 mentioned this pull request Oct 20, 2026

fix(deps): Bump certifi version #1280

Closed

2 of 4 tasks

Prajna1999 self-assigned this Oct 21, 2026

Merge branch 'main' into feat/retry-webhooks-and-provider-calls

faf6dc6

Prajna1999 linked an issue Oct 21, 2026 that may be closed by this pull request

Provider Calls: Implement retry mechanism #1263

Closed

Prajna1999 requested review from AkhileshNegi, Ayush8923 and vprashrex and removed request for Ayush8923 October 21, 2026 04:00

vprashrex approved these changes Oct 21, 2026

View reviewed changes

Prajna1999 and others added 3 commits October 22, 2026 09:54

Merge branch 'main' into feat/retry-webhooks-and-provider-calls

c151a85

Merge branch 'main' into feat/retry-webhooks-and-provider-calls

0dd873e

Prajna1999 changed the title ~~feat(retry): Retry webhooks, LLM proxy, guardrails and provider SDK calls~~ feat(retry): Retry webhooks, client LLM proxy and guardrails calls Oct 22, 2026

Prajna1999 and others added 3 commits October 22, 2026 11:29

chore: version lock to litellm

c8b56bd

Merge branch 'main' into feat/retry-webhooks-and-provider-calls

dd4348c

Prajna1999 enabled auto-merge (squash) October 22, 2026 06:48

Prajna1999 merged commit 11ce211 into main Oct 22, 2026
5 checks passed

Prajna1999 deleted the feat/retry-webhooks-and-provider-calls branch October 22, 2026 07:01

github-actions Bot added the released on @main label Oct 22, 2026

Navigation Menu

Repository navigation

feat(retry): Retry webhooks, client LLM proxy and guardrails calls - #1275

feat(retry): Retry webhooks, client LLM proxy and guardrails calls#1275
Prajna1999 merged 13 commits into
mainfrom
feat/retry-webhooks-and-provider-calls

Prajna1999 commented Oct 19, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 19, 2026 •

edited

Loading

Review skipped

Uh oh!

github-actions Bot commented Oct 19, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Oct 19, 2026

Uh oh!

vprashrex Oct 21, 2026

Uh oh!

Prajna1999 Oct 22, 2026

Uh oh!

Prajna1999 Oct 22, 2026

Uh oh!

Uh oh!

github-actions Bot commented Oct 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Prajna1999 commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Evidence

Merge Danger

Uh oh!

coderabbitai Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

github-actions Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

OpenAPI changes ⚪ No API surface changes

Uh oh!

codecov Bot commented Oct 19, 2026

Codecov Report

Uh oh!

vprashrex Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

Prajna1999 Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

Prajna1999 Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Oct 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Prajna1999 commented Oct 19, 2026 •

edited

Loading

coderabbitai Bot commented Oct 19, 2026 •

edited

Loading

github-actions Bot commented Oct 19, 2026 •

edited

Loading