Skip to content

[backend:fix] fix same-kernel Python UDF registration and invocation MCDB-103581 - #156

Open
ShubMishra wants to merge 1 commit into
singlestore-labs:mainfrom
ShubMishra:skumar/mcdb-103581-notebook-udf-loop
Open

ShubMishra wants to merge 1 commit into
singlestore-labs:mainfrom
ShubMishra:skumar/mcdb-103581-notebook-udf-loop

Conversation

@ShubMishra

@ShubMishra ShubMishra commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Disambiguate error 1347 using routine/table catalogs and SHOW AGGREGATES. Reject conflicting names before stopping the live server; keep non-replacing CREATE and published/other-session ownership checks.
  • Move interactive UDF HTTP serving off the notebook event loop so synchronous same-kernel SQL calls can complete. Preserve context, startup errors, re-registration, cancellation, and graceful shutdown.
  • Leave published UDF serving, notebook image pins, package version, and public APIs unchanged.

Test Plan

  • Staging (user-provided evidence): both UDFs registered successfully; same-notebook SELECT pulse.double_it_sh_test(2.0); returned 4.0. Cell duration: 529 ms.
  • Local: PYTHONASYNCIODEBUG=1 .venv/bin/python -m pytest -q -n 0 singlestoredb/tests/test_function_url.py singlestoredb/tests/test_udf_event_loop.py - 56 tests and 19 subtests passed on Python 3.11.12 / Uvicorn 0.34.3 against an isolated SingleStore 9.1.1 container. Covers actual 1347/1982/1050 responses, conflicting object types, case variants, schemas, and lifecycle behavior.
  • Checks: changed-file pre-commit checks (including flake8/mypy) and git diff --check passed. Full engine-protocol suites were not run locally.

The pure-Python fallback warning is expected with SINGLESTOREDB_BUILD_EXTENSION=0; it did not prevent registration or invocation.

Runtime Impact

Interactive apps gain one persistent HTTP serving thread/event loop, not a new server thread per query. Startup/shutdown coordination is outside the request path; existing UDF dispatch remains unchanged. The new loop can own an executor for synchronous UDF work. Latency, throughput, and memory overhead have not been benchmarked; the screenshot is a functional smoke test.

Deployment Plan

  • No shared staging/production image is changed by this PR or the manual runtime installation.
  • After SDK review/release and remaining staging lifecycle checks, update the Helios notebook image's SDK pin in a separate rollout.
  • Roll back the manual test by reinstalling the recorded prior SDK version/revision and restarting the kernel, or by recreating the disposable runtime from its unchanged image. A kernel restart alone does not restore the old package.
  • Python cannot forcibly stop arbitrary synchronous UDF code. Shutdown stops the serving loop without waiting indefinitely for such worker code; the existing limitation on cancelling the user code itself remains.

JIRA Issues

Resolves MCDB-103581.


Note

Medium Risk
Changes interactive UDF server threading and registration preflight against the database catalog; incorrect conflict detection or shutdown could break notebook UDF registration or leave listeners open, though published UDF paths are unchanged.

Overview
Fixes same-kernel interactive Python UDF registration and invocation by serving HTTP on a dedicated thread (ThreadedUvicornServer) instead of the notebook asyncio loop, so synchronous SQL-in-notebook calls no longer deadlock the cell. Startup/shutdown now support both async tasks and thread Futures, including cleanup when startup fails or is cancelled, and updated docs that interactive shutdown may run from a different loop in the same kernel.

Interactive preflight now treats SHOW CREATE FUNCTION wrong-object (1347) as a possible name clash: it checks information_schema and SHOW AGGREGATES, raises a clear RuntimeError when the name is taken by a procedure/table/view/aggregate, and does not tear down the live UDF server when preflight fails. Free names still register with non-replacing CREATE.

Published UDF serving on the notebook loop is unchanged. Tests cover threaded lifecycle, context propagation, catalog conflicts, and integration against a real engine.

Reviewed by Cursor Bugbot for commit 1ad4347. Bugbot is set up for automated code reviews on this repo. Configure here.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The unshielded startup cleanup wait allows a second cancellation to invalidate the server’s shared completion Future, causing errors during completion and subsequent shutdown or re-registration.

1 open issue
What changed in this PR

This PR fixes interactive Python UDF registration and allows synchronous SQL calls from the same notebook kernel by moving HTTP serving to a dedicated thread.

Changes:

  • Treat error 1347 from SHOW CREATE FUNCTION as a missing function while preserving non-replacing registration.
  • Add threaded interactive serving with startup and shutdown coordination; retain caller-loop serving for published UDFs.
  • Expand regression tests for registration, same-kernel invocation, context propagation, and server lifecycle.
File Description
singlestoredb/​tests/​test_function_url.py Adds registration and threaded-server regression tests.
singlestoredb/​functions/​ext/​function_url.py Recognizes WRONG_OBJECT during missing-function checks.
singlestoredb/​apps/​_uvicorn_util.py Implements threaded serving and loop cleanup.
singlestoredb/​apps/​_python_udfs.py Selects interactive threaded serving and coordinates its lifecycle.

Review effort: Balanced

Comment thread singlestoredb/apps/_python_udfs.py Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block the interactive Python UDF registration and serving changes.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@kesmit13 kesmit13 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Threaded serving looks solid; one real gap in the 1347 handling, plus two small items.

Also: the shutdown_udf_app docstring still says to call it "on the same event loop as run_udf_app", which no longer applies to the threaded server.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit ef78a7a. Configure here.

Comment thread singlestoredb/functions/ext/asgi.py Resolved
@ShubMishra
ShubMishra force-pushed the skumar/mcdb-103581-notebook-udf-loop branch from ef78a7a to 334fcfe Compare November 3, 2026 04:01
@ShubMishra
ShubMishra force-pushed the skumar/mcdb-103581-notebook-udf-loop branch from 334fcfe to 88c4575 Compare November 3, 2026 04:21
…MCDB-103581

Disambiguate SHOW CREATE error 1347 with catalog checks before stopping a live interactive server. Preserve non-replacing registration and reject procedure, aggregate, table, and view conflicts. Serve interactive UDF callbacks on a dedicated event-loop thread with safe startup, cancellation, and shutdown; retain published serving behavior. Cover registration and lifecycle regressions, including real-engine catalog and error-code checks.

This branch had an error being deployed

1 failed deployment
Base — 1ad4347a Deployed Nov 3, 2026 by ShubMishra via test-coverage #615
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants