[rockhunters08]

Why

flavour-extractor OOMKilled across the whole fleet on 2026-10-14 (32 restarts over 16 pods). Root cause is in this repo.

get() and _mutate() retried a failed request by calling themselves:

except PyBaseException as e:
    e._handle_exception(self, dest_region=dest_region)
    return self._mutate(table, key, values, rq_type)

Python has no tail-call optimization, so each retry adds a live stack frame holding values plus a freshly built protobuf request. A row that never succeeds keeps every attempt alive at once.

In the extractor the document being written is 1.4 MB as JSON (markup 674 KB, content_text 365 KB, four embedding vectors), roughly 3 MB live per frame. The stock recursion limit is 1000, so the ceiling is about 3.1 GiB allocated in 3 seconds, which OOMs the container before RecursionError can unwind. Raising the container limit from 3Gi to 4Gi did not help, because the recursion is unbounded.

Caught with py-spy dump on a live pod: hundreds of stacked _mutate frames at client.py:265.

What changed

Both retry sites become bounded loops that raise the last exception once attempts are exhausted. The loop also lets each attempt's request be collected, so memory stays flat regardless of the bound.

MAX_REQUEST_ATTEMPTS = 10 is generous for the transient cases these retries exist for (region moved, splitting, server reestablished), which normally clear in one or two attempts. The behavior change is that a permanently failing row now raises instead of retrying forever. That is strictly better than exhausting memory, and callers already handle the exception: flavour catches it as HBaseWriteException and requeues.

Unrecoverable exceptions are unaffected, since _handle_exception re-raises them itself.

Tests

Added tests/test_retry_bounds.py, which runs without a live HBase (the existing integration tests need one):

• _mutate and get stop after exactly MAX_REQUEST_ATTEMPTS and raise
• _mutate still returns normally when a retry succeeds

Verified these are meaningful: against the pre-fix code with the recursion limit lowered to 200, _mutate dies with RecursionError after 198 attempts. After the fix it stops at 10.

flake8 reports the same issue set before and after on client.py (2 E128, 1 F821, 2 W293, all pre-existing and untouched); the new test file is clean.

Note on deploy

flavour pins 393cb97 in requirements-source.txt, which is two commits behind master. That pin needs bumping separately for this to reach production.

One thing this does not address: why those particular rows fail their HBase write in the first place. That is still open. This change contains the blast radius either way, turning an OOM into a logged, catchable error.

[Copilot]

🟡 Changes recommended

Saved exceptions retain large requests after successful retries until cyclic garbage collection runs.

1 open finding

• Clear last_exception after successful requests to prevent retained references · New

What changed in this PR

Replaces recursive HBase request retries with bounded loops to prevent memory exhaustion from permanently failing requests.

Changes:

• Limits get() and _mutate() to 10 attempts before raising the last exception.
• Adds offline tests for retry limits and successful recovery.

File	Description
tests/​test_retry_bounds.py	Tests retry exhaustion and mutation recovery.
pybase/​client.py	Replaces recursive retries with bounded loops.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟢 Approval recommended

The bounded loops preserve exception handling, address recursive memory growth, and have focused regression coverage with no unresolved findings.

0 open findings

1 resolved since last review

• Clear last_exception after successful requests to prevent retained references

🧠 Review effort: Balanced