Repository navigation
feat(auth)!: add OIDC and rotating bearer tokens - #133
glasstiger wants to merge 430 commits into
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 Walkthrough
Pre-merge checks:
|
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (1)
src/questdb/auth/_questdb.py (1)
82-94: 💤 Low valueChain the import exception for better diagnostics.
When both
psycopgandpsycopg2fail to import, the raisedImportErrorloses the underlying cause. Chaining the exception preserves the diagnostic context.Suggested fix
def _pg_module(): try: import psycopg # type: ignore # psycopg v3 return psycopg except ImportError: pass try: import psycopg2 # type: ignore return psycopg2 - except ImportError: + except ImportError as e: raise ImportError( 'A PostgreSQL driver is required: install `psycopg` (v3) or ' - '`psycopg2-binary`.') + '`psycopg2-binary`.') from e🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/questdb/auth/_questdb.py` around lines 82 - 94, The second except ImportError block in the _pg_module() function should preserve the exception chain for better diagnostics. Capture the ImportError exception in the second except block using `as e` syntax, then use `raise ... from e` syntax when raising the new ImportError to chain the exception, which preserves the underlying cause and provides better debugging context.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.claude/skills/review-pr/SKILL.md:
- Around line 157-162: The Step 3 preamble at lines 157-162 states that every
agent receives the full change-surface map, but Agent 10's input contract at
lines 231-240 explicitly limits it to only the PR diff and changed file names.
Add an explicit exception clause in the Step 3 preamble section to clarify that
Agent 10 does not receive the full change-surface map and instead receives only
the diff and changed file names, ensuring consistency between the general rule
and the specific Agent 10 behavior documented at lines 231-240.
- Around line 39-42: The table at lines 39-42 states that level 0 (default)
skips Step 2.5, but there is a later reference (around lines 144-155) that marks
Step 2.5e as mandatory at every level, creating a contradiction. Fix this
inconsistency by clarifying the level 0 description in the table: either modify
the "Skip Step 2.5" statement to specify that Step 2.5e is an exception and is
still performed, or restructure it to list exactly which substeps (2.5a, 2.5b,
2.5c, 2.5d, 2.5e) are executed or skipped. Ensure the description at level 0 and
the mandatory requirement stated at lines 144-155 are aligned so the default
path is no longer self-contradictory.
In `@docs/auth.rst`:
- Around line 65-76: The code example in the documentation is missing the import
statement for TimestampNanos, which is used in the sender.row() method call. Add
an import statement at the top of the code snippet to include TimestampNanos
from the questdb.ingress module (or appropriate module) so that the example can
be executed successfully when copied verbatim.
In `@src/questdb/auth/__init__.py`:
- Around line 75-92: The `__all__` list in the module needs to be sorted
alphabetically to comply with Ruff's RUF022 rule and maintain consistency in the
public API exports. Reorder all the items in the `__all__` list (which includes
'connect', 'QuestDB', 'OidcDeviceAuth', 'OidcConfig', 'TokenCache', 'TokenSet',
'MemoryCache', 'FileCache', 'NullCache', and the various OidcError types) in
alphabetical order while ensuring all items remain in the list.
In `@src/questdb/auth/_http.py`:
- Around line 123-129: The `_opener()` function doesn't receive the `insecure`
parameter, so it cannot enforce HTTPS-only redirect policies. To fix this, add
an `insecure` parameter to the `_opener()` function signature. When
`insecure=False`, create a custom redirect handler that validates redirects do
not downgrade from HTTPS to HTTP for non-loopback hosts (similar to the
validation in `_require_secure()`), and pass this handler to
`urllib.request.build_opener()` instead of using the default. When
`insecure=True`, use urllib's default redirect handling. Update all callers of
`_opener()` to pass the `insecure` parameter value.
In `@test/test_auth.py`:
- Around line 600-605: The timed joins with a 10-second timeout do not verify
that threads actually completed, which can cause non-daemon threads to remain
running and hang the test process if a regression causes a deadlock. After
calling t.join(10) on each thread in both the loop at lines 600-605 and at lines
828-835, add a verification check that confirms each thread has terminated by
asserting that t.is_alive() returns False, or raise an exception if any thread
is still alive after the timeout. This ensures the test fails cleanly rather
than leaving hanging threads.
---
Nitpick comments:
In `@src/questdb/auth/_questdb.py`:
- Around line 82-94: The second except ImportError block in the _pg_module()
function should preserve the exception chain for better diagnostics. Capture the
ImportError exception in the second except block using `as e` syntax, then use
`raise ... from e` syntax when raising the new ImportError to chain the
exception, which preserves the underlying cause and provides better debugging
context.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: bf4af383-9022-4ae5-b189-73691f4ccd00
📒 Files selected for processing (18)
.claude/skills/review-pr/SKILL.mdCHANGELOG.rstdocs/api.rstdocs/auth.rstdocs/index.rstdocs/installation.rstexamples/oidc_device_auth.pysetup.pysrc/questdb/auth/__init__.pysrc/questdb/auth/_cache.pysrc/questdb/auth/_device.pysrc/questdb/auth/_discovery.pysrc/questdb/auth/_errors.pysrc/questdb/auth/_http.pysrc/questdb/auth/_questdb.pysrc/questdb/auth/_render.pytest/test.pytest/test_auth.py
Address moderate findings from the PR #133 review: - Persist and re-check `issuer` in the token store so a token pinned to one issuer is never served from disk to a session pinned to another. Kept out of the file-name hash, so the cross-language file contract and existing token files stay byte-stable; isolation is enforced via the on-load identity re-check, matching the in-memory cache_key. - FileTokenStore.load() now returns None (not raises) for an unreadable entry: a directory/symlink-loop at the token path (stat.S_ISREG/errno guard) and a deeply-nested attacker file (RecursionError), restoring its documented contract and matching every other json.loads site. - _display_url escapes a host that still carries a URL-structural char after IDNA (e.g. the U+FF3C/U+FE68 reverse-solidus fold) as a visible \uXXXX, instead of a bare backslash a browser would read as a slash. - CHANGELOG: relabel the Python 3.8->3.10 bump as a breaking change with the whole-package blast radius, EOL rationale, and a migration note. Adds regression tests for each; full auth suite passes (308 tests). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Code review —
|
Code review —
|
Code review —
|
Tandem review: py-questdb-client #133 + c-questdb-client #183Level 3, multi-agent, adversarial. These two PRs are one change: py #133 adds the native-backed OIDC device-flow binding and pins the Verdict: Approve. No Critical issues — no memory-unsafety, refcount imbalance, double-free, GIL violation, ABI mismatch, auth-bypass, secret-leak, or terminal-injection. This is an unusually well-hardened change. The two Moderate items are now resolved on the branch; the remainder are Minor. ✅ Resolved in
|
|
Note: this PR temporarily carries UUID / fixed-size-binary changes that belong to #140. Commit It appears here because this PR bumps the To be precise about what did and did not change: the QWP wire bytes are unchanged and still match the Java client — Merge order: #140 should land first. This branch's copy then collapses into the merge base and this PR's diff reduces to the OIDC feature alone. Watch the |
Picks up three fixes on the c-questdb-client branch: a misconfigured provider stays terminal instead of driving an endless reconnect, a failed token pull is reported to the connection listener instead of failing silently, and close drops the in-memory credential on every path rather than only when it drains. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GjKmdBcwtTjoSb8mZwv4xz
The event handler's user_data was an INCREF'd weakref, so the final native release ran a Py_DECREF -- and that release can run on a thread the interpreter neither manages nor joins. A token provider's acquisition is isolated onto its own worker so shutdown can abandon it, and that worker keeps the provider closure, and so the callback state, until its own HTTP call returns: potentially long after every handle is gone and after finalization has begun. The guard was a Py_IsFinalizing check, and no such check can be made safe -- finalization can begin between the test and PyGILState_Ensure, which then hangs the thread or faults. Weakening the closure's handle does not help either: the auth owns its renderer, which is the same CEventHandler, so anything holding the auth holds the callback. Hand native an opaque integer key instead and resolve the provider through a module-level registry under the GIL. Native then owns no Python reference, the release callback has nothing to free and never needs the GIL, and the entry is dropped in __dealloc__, which always runs on a managed thread. This removes the hazard rather than narrowing its window. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GjKmdBcwtTjoSb8mZwv4xz
Nothing but __dealloc__ removes a registry entry now that native owns no Python reference, so a stale one would be a slow leak that no weakref assertion catches. Asserts the entry appears with the provider, does not keep it alive, and is gone once it is collected -- and that the registry does not grow across 200 create/destroy cycles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GjKmdBcwtTjoSb8mZwv4xz
Picks up two fixes on the c-questdb-client branch: a long Retry-After is waited out in slices so the prompt keeps counting down and Ctrl-C stays deliverable, and the in-process token-store lock is bounded by the acquire budget instead of spinning without a deadline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GjKmdBcwtTjoSb8mZwv4xz
The Renderer docstring said only that a callback must not call back into the same instance. It omitted that the guard applies to any thread, and now that token() succeeds from a valid cache but is refused when a fresh acquisition would be needed. States both, and points at signing in before putting a provider behind a connection pool so pooled checkouts hit that cache. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GjKmdBcwtTjoSb8mZwv4xz
Picks up the c-questdb-client fix that stops the OIDC callback guard failing unrelated callers: a valid cached token is now served to other threads during a sign-in prompt, and the two refusal cases carry distinct, accurate diagnostics. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GjKmdBcwtTjoSb8mZwv4xz
Picks up two c-questdb-client fixes: a future-dated lock file no longer disables the token store permanently, and a credential is no longer persisted into a directory whose permissions could not be restricted and which the client will therefore never read back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GjKmdBcwtTjoSb8mZwv4xz
client_id, token_endpoint and device_authorization_endpoint are required positionals but went through the optional-string helper, which drops a None and leaves the field unset. Native then reported the field as missing from QuestDB's /settings and told the caller to pass it explicitly -- advice that makes no sense for a constructor that never contacts /settings, and that names a builder method this API does not have. from_questdb keeps the optional path, where the three are genuine overrides. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GjKmdBcwtTjoSb8mZwv4xz
Only native enforced that oidc_auth excludes a fixed credential, and it reports the internal config key it knows -- "http_token_provider" or "qwp_ws_token_provider" -- which exists in no public API, so a caller who passed oidc_auth= together with token= was told about a symbol they cannot find. Reject it in terms of the parameters actually written, on both the Sender and the pool path, keeping the established QuestDBError(ConfigError) type. Also documents the 65536 ceiling on the two callback inbox capacities, which is enforced at connect time but was mentioned nowhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GjKmdBcwtTjoSb8mZwv4xz
Asserts a credential conflict names oidc_auth and token rather than an internal provider key, and that each required constructor argument is rejected as required instead of being reported as missing from /settings. Adds the changelog entry for the callback inbox cap, which rejects a previously accepted value at connect time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GjKmdBcwtTjoSb8mZwv4xz
Picks up three c-questdb-client fixes: advertised endpoints are confirmed against the IdP so Entra ID and Google work with issuer=, a 401 from a token that expired mid-retry is retried once with a rotated credential, and a persisted JWT honours the same one-hour rotation cap as one off the wire. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GjKmdBcwtTjoSb8mZwv4xz
OidcDeviceAuth is an extension type, so help() and the rendered API reference cannot introspect its signature -- Sphinx shows __init__(*args, **kwargs) and none of the sixteen parameters. Four of them (ca_bundle, interactive, default_interval, timeout) were described nowhere at all, and two carry contracts worth stating: insecure relaxes plaintext for the QuestDB discovery request only, never the identity provider, and timeout is a per-request timeout rather than a deadline for the sign-in. Documents all of them in the docstring, which is what the reference renders. token() and headers() now name OidcInteractionRequired, which both raise before the first sign_in. The renderer docs told custom implementations to sanitize untrusted identity-provider text without exporting anything to do it with, while oidc.pxi reached across the extension boundary for the private _strip_control. Exports sanitize_display_text and points both the Renderer docstring and docs/auth.rst at it. Also records why the event trampoline may enter Python when the release callback must not: renderer events are emitted only by the device flow, reached only through the foreground sign_in(), so no abandoned worker can reach it after finalization begins. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GjKmdBcwtTjoSb8mZwv4xz
- The object-dtype uuid.UUID column asserts concrete wire bytes. It drives qwp_numpy_s16, a different native entry point from the Arrow routes every other UUID test uses, and it is the one changed to hand over canonical RFC 4122 bytes and rely on native to reverse them. A round-trip through one symmetric transform would pass either way. - OidcError's control-character stripping is asserted through a real flow with a hostile error_description. Native does not sanitize this path, so the Python layer is the only sink guard, yet every existing test used clean strings yet -- deleting it passed the whole suite. - The SUCCESS/FAILURE non-cancellation guard: an interrupt raised from on_success must not discard the token just acquired. - Cancellation tests now advertise a short device-code lifetime. They assert promptness after the blocking call, so a regressed cancellation stalled for ten minutes rather than failing, and the CI watchdog re-arms past that. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GjKmdBcwtTjoSb8mZwv4xz
Picks up the c-questdb-client test work: the hand-maintained OIDC and connect-options headers are now guarded against drifting from the Rust definitions they mirror, and the C++ event handler's ownership contract is covered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GjKmdBcwtTjoSb8mZwv4xz
Picks up the c-questdb-client CI work: the OIDC device-flow test that reached for the WS builder API is now gated on the WS feature, so the HTTP-only leg of the cargo matrix compiles again; two clang-format violations are cleared; and the token-store interop goldens are pinned against line-ending conversion, which was corrupting the byte-exact comparison on Windows checkouts alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GjKmdBcwtTjoSb8mZwv4xz
`_finish_builder` inserts the provider into `_OIDC_PROVIDERS` before installing the native event handler, but only the handler-failure path dropped that entry. A failed `questdb_oidc_builder_build` left it behind with `_raw` still NULL -- and the already-initialized guard keys on `_raw != NULL`, so it does not fire on a retry. A caller that re-runs `__init__` on the same object, such as a subclass catching OidcConfigError and calling `super().__init__()` again with a corrected endpoint, then overwrote `_provider_id` and stranded the previous key as a dead weakref in a module-global dict for the life of the process, once per retry. The post-registration block is now wrapped so every failure unwinds the entry, not just the one that happened to be handled, restoring "registered <=> built" against future raise sites too. `test_event_handler_weakref_released_on_failed_build` could not have caught this: its empty client_id is rejected by `_oidc_required_utf8` before the builder is populated, so it never reached a build failure at all, and its rationale described the pre-integer-key ownership model this branch replaced. Two tests take its place, using an unreadable ca_bundle -- stored by the setter, opened by build() -- and asserting registry size rather than a weakref proxy. One holds the half-built object alive, because asserting after it is dropped proves nothing: __dealloc__ pops the entry either way. Both fail without the fix. Also pins the causal-chain contract from the Python side: a token failure reaching a transport arrives as a typed OidcInteractionRequired that is still a QuestDBError and still carries the transport's own SocketError code, which is what retry logic keys on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012QoXxcyoqduoW9Mi3ZSnyy
`_direct_dataframe_run` rebuilt the error as a bare QuestDBError when a one-shot stream had already been consumed. That flattened every subclass that can reach the branch: an OIDC token failure on an `oidc_auth` transport arrives as OidcNetworkError, OidcDeviceFlowError or OidcTimeoutError -- each carries a retryable code by design, so it clears both the terminal-OIDC gate and the code check above it -- and rebuilding stopped `except OidcError` matching while discarding .status, .retry_after, .error and .error_description. Those are the attributes `c_err_to_py` promises are catchable. It now appends the note to the original exception and re-raises it. `args` is what Exception.__str__ reads and every field lives on the instance, so code, in_doubt and sender_error survive for free. The existing consumed-stream test gains the discriminating assertion: `raise X from exc` set __cause__, re-raising the original leaves it None. Also corrects the `c_err_to_py` / `c_err_to_py_fmt` docstrings, which asserted the native view means the error is an OIDC failure. It means one is in its causal chain, so `c_err_to_py_fmt` dropping its format string on that branch was reasoning from a premise that does not hold -- the caller cannot tell a pure auth failure from a transport failure an auth error caused, and dropping the transport's context is only ever wrong for the second. That branch is in fact unreachable today (its sole caller is the TCP flush path, and native rejects oidc_auth on TCP); the formatting is made correct anyway rather than left as a trap, and the unreachability is now stated where the next reader will look. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012QoXxcyoqduoW9Mi3ZSnyy
`proj.py doc` builds with `-nW --keep-going`, and this branch failed it: "build finished with problems, 6 warnings (with warnings treated as errors)". Every warning came from new content. Five were `:mod:`questdb.auth`` cross-references -- in docs/auth.rst, docs/installation.rst, CHANGELOG.rst (twice, reached through docs/changelog.rst) and the OidcError docstring rendered by docs/api.rst -- with no `py:module` target anywhere under docs/ for them to resolve to. Declaring the module in api.rst gives them one; the auth entries there are all fully qualified, so the directive changes nothing else. The links now resolve to a real anchor rather than merely stopping the warning. The sixth was `sqlalchemy.engine.Engine`, the return annotation of the autodoc'd `questdb.auth.sqlalchemy_engine`. It goes in `nitpick_ignore` rather than intersphinx on purpose: SQLAlchemy is an optional, lazily imported integration, and a mapping for it would make every docs build fetch the inventory of a dependency the package does not require, and fail when that host is unreachable. Readthedocs sets no `fail_on_warning`, so these degraded to unlinked text there while breaking the project's own doc command. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012QoXxcyoqduoW9Mi3ZSnyy
Keep callback targets on GC-visible owners and retain bearer holders on query readers until teardown. Honor permanent dataframe failures, retain typed auto-flush errors, validate snapshot expiry ranges, and reset the Python callback lock after fork. Correct source, adapter and snapshot typing and strengthen lifecycle, GC, LONG256 and mypy regressions. Pin c-questdb-client to 0211c8d4 for native provider fixes.
Pin c-questdb-client to 3dc38c73, including the token provider header formatting fix required by CI.
Keep bearer-provider ownership in GC-visible instance dictionaries so PyPy can collect sources that own their providers, pools or senders. Remove the redundant application-wrapper source reference. Use dictionary-field access during native-handle teardown to avoid recreating dying cpyext wrappers. Extend the ownership-cycle regression to standalone senders as well as pools. Validated full CPython and PyPy aggregates, focused provider suites, strict Sphinx documentation and diff checks.
test_the_gil_is_released_while_waiting failed on the cp314 leg of the
macOS arm64 wheels job ("5 not greater than 5"). It counted how often a
second Python thread woke from 5 ms sleeps during a 0.2 s wait, which
measures scheduling, not the GIL: token() releases the GIL for the whole
native wait. Locally, CPU-bound Python threads in the same process
reproduce the failure (9 of 20 runs with eight of them), while the
GIL-release behaviour is unchanged.
Make the check causal instead: the source returns only after another
Python thread has slept and re-acquired the GIL ten times and then
opened the gate. If token() held the GIL, that thread could not run, and
token() fails after cold_wait, as a mutant build without the nogil
section confirms. Under load the test only runs slower.
|
/azp run questdb.py-questdb-client-release |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Preserve main-thread interrupts and signal-handler exceptions across token callbacks, propagate foreground-terminal OIDC dataframe failures, and share application-provider isolation across client attachments without rooting ownership cycles. Add bounded cooperative source draining via RefreshingTokenProvider.close(wait=True, timeout=...), reject cursor reentry while decoding borrowed batches, and document the shutdown and token-store coordination requirements. Add regression tests and typing coverage, and pin the native OIDC fixes at 426fe4c8f4d1097ecf43f334318506de564625f7. Validation: broader Python run (477 passed, 2 skipped); final focused run (233 passed, 15 skipped, 105 subtests passed).
Changes extracted from the OIDC / rotating bearer token PR (#133) that do not depend on either feature, so that PR can shrink to the feature itself. The c-questdb-client pin is unchanged. Breaking: - QuestDB.close() raises QuestDBError(InvalidApiCall) instead of waiting for a pool operation or lease held by the calling thread. Uses are counted per thread-state token, so a recycled thread ID cannot inherit a departed thread's lease; a sender lease follows the thread that last used it, a reader lease stays with the thread that borrowed it. - A sender's own connection_listener / error_handler can no longer call row(), dataframe(), new_buffer(), establish(), transaction() or the SenderTransaction operations on it. The guard is keyed on the owning sender, so peers sharing the default handler are unaffected. - A Buffer that a native call is modifying rejects reads and writes from any thread; a kept flush only borrows it for reading. - SenderTransaction.commit() after close raises QuestDBError instead of TypeError; a failed commit discards its rows itself. - An abandoned QueryResult reclaims its cursor before warning, and a ResourceWarning promoted to an error reaches sys.unraisablehook. Fixes: - Finalizers and GC never block on, or free, a cursor another operation is using; cursor close/cancel/pulls reject same-thread re-entry. - An unknown connection-event ordinal is dropped with one warning instead of being delivered as Connected; event and server-role enums bind the header constants. - DataFrame errors keep their class, in_doubt and sender_error when a hint or note is appended; the batch-too-large hint also covers the trailing sync. - A close waiting on a concurrent close that gives up closes the pool. - Small fixes: dataframe() buffer leak on a closed handle, PyWeakref_GetRef error handling, numpy enum names in the .pxd, connect() rephrases duplicate keys only for ConfigError, .pyi constructors. CI and tests: wheel jobs check out the submodule, PyPy pinned to 3.10 and 3.11, Windows cp314t wheels, pandas 3 only on 64-bit, pyarrow<26 with NumPy 1.x, PyYAML, authenticated questdb clone, JDK 25 module flags, quieter mock server, pandas 3 dtype fix, release-download retry, leak harness settling windows, manifest and suite-wiring guards. Docs: 6.0.0 changelog entries, 5.0-to-6.0 migration section, free-threaded wheel note, cross-repo merge procedure, polars.DataFrame(result).
Stack this PR on #160, which carries the changes extracted from it that do not depend on OIDC, token providers or #140. The extracted code now matches #160; what remains in this PR's diff against #160 is the feature, #140's UUID / fixed-size-binary work, and what #160 leaves here because it needs the newer native client (the config-string and callback-inbox caps, CredentialUnavailable, the native fixes reaching Python through the pin). Conflicts: took #160's version of the extracted code and wording, plus this PR's OIDC additions. Changelog and migration guide: #160's entries in #160's order, with this PR's own entries around them; dropped this PR's copies of the cursor re-entry and close-retry bullets that #160 now carries. Bump c-questdb-client to the head of questdb/c-questdb-client#183, now stacked on questdb/c-questdb-client#226.
test_far_future_source_results_are_classified_not_broken_snapshots read status.last_failure right after token() raised. With cold_wait=0.02, token() can give up before the refresher thread has run the source at all; it then raises the generic "no usable token" error and no failure is recorded yet, so last_failure was None (seen on the windows_i686 wheel job). Wait for the failure to be recorded before reading it, and do the same in the three sibling tests that share the race with a 200 ms cold_wait. test_a_classified_failure_reaches_the_caller also asserts the raised error's classification, so it gets a cold_wait that outlasts a slow first fetch: below the source's retry_after, token() still returns as soon as that fetch fails.
Bring in #140 (UUID, IPV4, BINARY, CHAR, DATE, LONG256 and GEOHASH columns). Both branches had reworked the same lifecycle, callback and UUID code independently, so most conflicts were semantic: - Callback dispatch: main's owner-token dispatch contexts replace this branch's _DispatchTarget stack. Kept this branch's global active-dispatch counter as the fast path for the per-row callback check, and its drop-and-warn for unknown connection-event kinds. - QuestDB.close(): main's bounded, one-way close (timeout, scoped-call refusal, Ctrl-C hardening) replaces this branch's per-thread lease attribution. Ported the OIDC rules on top: refused inside the pool's own persistence-warning callback, waits for in-flight diagnostics first, every wait bounded to two seconds inside any OIDC callback, and the provider released with the callbacks. The renderer-close test now expects the handle to stay closing after that bound. - Re-entry guards: this branch's native borrows and in-flight flags still refuse other threads and OIDC callbacks. Python that a dataframe() frame, or a reader query's bind values, runs on its own thread follows main's row and lease rules instead, as the re-entrancy and concurrency grids pin. SenderTransaction checks main's state first. - SenderTransaction.commit(): main's outcome rules (over once the flush was attempted, still open if refused before it), with this branch's callback guard. - DataFrames: main's planner and UUID object builder. This branch's tests that expected unlabelled 16/32-byte pandas columns to land as BINARY, or a short int.to_bytes to be refused, now expect main's behaviour. _direct_dataframe_run keeps the OIDC fail-fast gate under main's call-level in_doubt aggregation; widened errors keep their class and chain the original. - Kept this branch's col_target_t layout (no reserved LONG256 target). - Type stubs: SchemaOverride gains 'binary' and SupportsIndex bits. - CI: pipeline variables in list form, so the variable group and main's defaults coexist. OIDC and token-provider classes are excused from the re-entrancy and concurrency grids (test/api_surface.py). - Changelog: one 6.0.0 section under main's headings; dropped this branch's UUID and close() entries that main's supersede. Migration guide updated to match. Bump c-questdb-client to 456e8e72 (rotating-provider attach checks).
Native-backed OIDC device-flow sign-in and rotating Bearer-token providers for the Python client (`questdb.auth`, `oidc_auth=`, `bearer_token_provider=`), split out of #133 so the feature can be reviewed against `main` without the unrelated fixes of #160. The feature runs user callbacks (token providers, sign-in renderers) inside native calls, so this also carries the re-entry guards from #160 that keep such a callback from using an object a native call holds: - a sender rejects its methods while a native call borrows it, and from its own connection listener or error handler; - a `Buffer` that a native call is modifying rejects reads and writes; - a query cursor rejects same-thread re-entry during a native read, and finalizers reclaim cursors without blocking; - a DataFrame error keeps its class when a note is appended, so `except OidcError` still matches. Bumps c-questdb-client to the matching native change (questdb/c-questdb-client#228). The full history is on branch `ia_oidc_device_flow` (#133).
|
superseded by #165 |
Summary
Adds native-backed OIDC device-flow authentication for QuestDB Enterprise. The
Python API runs OAuth 2.0 Device Authorization Grant (RFC 8628) through the
native QuestDB client, including from remote Jupyter kernels where the browser
and Python process are on different machines.
Discovery, endpoint validation, token selection, caching, refresh, persistence,
and concurrency control use the native implementation. Python provides the
public authentication API, Jupyter/terminal rendering, PG-wire adapters, and
integration with QuestDB transports.
It also adds rotating Bearer-token providers, for machine credentials that
rotate: Microsoft Entra ID managed identities and service principals, OAuth2
client credentials, a secrets vault.
bearer_token_provider=accepts a nativeRefreshingTokenProvideraround a token source you write, or any callable orobject that returns the current token. They follow the cross-language
dynamic-credential specification shared with the Java client; see Rotating
Bearer tokens.
Merge order
This branch temporarily carries the UUID / fixed-size-binary changes that belong
to #140 (
a363777). They are here because bumping thec-questdb-clientsubmodule past questdb/c-questdb-client#186 — which added
reverse_uuid_bytesand has been on
mainsince 18 Aug — obliges the Python side to supplycanonical RFC 4122 bytes; without that adaptation every UUID would be silently
stored byte-reversed. The QWP wire bytes themselves are unchanged and still
match the Java client.
#140 should land first. This branch's copy then collapses into the merge
base and this PR reduces to the OIDC and token-provider features alone. This
branch currently documents the combined changes under
6.0.0, with a5.0-to-6.0 migration guide; reconcile those headings when merging #140.
User-visible breaking changes
The 5.0-to-6.0 migration guide records eleven changes that may require action:
Arrow-backed query results; remove any application-side byte reversal.
use the
arrow.uuidlabel, preserved whole-table/stream Arrow fieldmetadata (
questdb.column_type=uuidorquestdb.column_type=long256),or
schema_overridesas documented.schema_overridestuple is accepted only for('geohash', bits); values such as('symbol', 16), whose argument waspreviously discarded, must become
'symbol'. The publicquestdb.SchemaOverridealias describes accepted values.error_inbox_capacityconfig-string key,reject capacities above 65,536.
QuestDBError(InvalidApiCall), on every entry point:Sender.from_conf,Sender.from_env/QDB_CLIENT_CONF,questdb.connectandQuestDB.from_conf.SenderTransactionafter its sender was closed now raisesQuestDBError(InvalidApiCall)instead of an internalTypeError. A commitwhose flush fails leaves the sender's buffer empty and the transaction
completed, exactly as in 5.0 -- that is not a change.
QueryResult'sResourceWarningto an exception, the finalizer now lets it reach
sys.unraisablehookinsteadof swallowing it, after releasing the cursor. Otherwise normal warning filters
apply as before. Close results deterministically (preferably with
with).already admitted, including those paused before acquiring the GIL. Renderer
and persistence-warning callbacks must return promptly: a blocked callback
can delay process exit, even on a daemon sign-in thread.
failover_max_duration_mswhile attemptsremain, after a last round rejected on role, at the WebSocket upgrade or at
TLS, now raises that
RoleMismatch/HandshakeError/TlsErrorinsteadof the connection failure that started the failover.
QuestDB.close()raisesQuestDBError(InvalidApiCall)instead of waitingwhile the calling thread holds a pool operation or lease. That includes a
sender lease handed to a worker that has not used it yet, which 5.0 waited
for; have the worker use the lease before closing on the borrowing thread.
sender's own
connection_listenerorerror_handler,row(),dataframe(),new_buffer(),establish(),transaction()and theSenderTransactionoperations raiseQuestDBError(InvalidApiCall), asflush()andclose()already did. 5.0 acceptedrow()there; handlistener events to another sender or to the owning thread instead.
rejects its credential with HTTP 401 or 403 on a reconnect, a fixed
token=orusername=/password=included: it keeps reconnecting withcapped backoff, keeps its unacknowledged rows, and reports each attempt as
ConnectionEventKind.AuthFailed. 5.0 stopped the sender at the firstrejection and raised
AuthErrorfrom its next call. A connection that iswaited for -- a sender's first connection, an eager
questdb.connect--still fails at once. Code that relied on the sender stopping must close it
itself.
See
CHANGELOG.rstanddocs/migration.rstfor exact affected entry points andmigration examples.
Usage
Interactive sign-in is explicit.
sign_in()is the only operation that mayprompt or open a browser;
token(),headers(), and transport connect/reconnectpaths only use cached or silently refreshable credentials and otherwise raise
OidcInteractionRequired.questdb.connect,Sender, andSender.from_confretain the shared nativeprovider and request a current token on every connect or reconnect. A fixed
token=is not required and is mutually exclusive withoidc_auth=.For PG-wire,
sqlalchemy_engine()injects a current token as the_ssopassword on each new pooled connection, while
psycopg_connect()captures thecurrent token for one connection. Other clients can use
auth.token()orauth.headers()after explicit sign-in.Configuration and lifecycle
OidcDeviceAuth.from_questdb()discovers configuration from QuestDB's public/settingsendpoint; explicit constructor arguments can override discovery,and the direct constructor supports fully explicit IdP endpoints.
groups_in_token=Trueselects the ID token; otherwise the access token isused. It does not modify
scope: the configured scope is sent verbatim onthe initial device-authorization request, including in groups mode. Refresh
requests intentionally omit
scope, matching the Java client, so the IdPpreserves the scope originally granted. Include
openidexplicitly when theidentity provider requires it to issue an ID token.
FileTokenStore.at_default_location()enables opt-in native persistence withatomic files, owner-only permissions, and cross-process refresh coordination.
On non-Unix platforms the bundled store rejects mutation-required operations
before creating credentials or transient coordination metadata; use
memory-only authentication there. Custom Python token stores are not
supported (the native Rust API alone accepts a custom/keychain-backed store).
terminals, with optional QR output. Renderer callbacks receive bounded,
native display-normalized but still untrusted text. Custom renderers must
encode it for their output sink and use only the native-vetted
browser_targetfor actionable URLs.questdblogger,often on a token-refresh thread. A QWP/WebSocket ACK wait made inside such a
handler raises
OidcInteractionRequiredpromptly once its sender'sreconnect needs a token the provider cannot supply until the handler
returns; the frame stays queued. ACK waits on other threads are not
rejected: they wait for the handler to return.
close()permanently closes a provider and cancels a device flow, refreshcoordination, or token-store lock wait running on another thread; it is safe
from any thread, including a renderer callback, so a notebook "Cancel" button
can call it.
OidcDeviceAuthis a context manager.Ctrl-Cduringsign_in()cancels only the current attempt and raisesKeyboardInterrupt;the provider and its attached transports remain usable. Operations on a
closed provider -- including attaching it to a new
Sender, pool or reader-- raise
OidcCancelledError, exceptclear(), which staysavailable so the persisted credential can still be removed;
cancel_sign_in(), which remains an idempotent no-op when no flow isrunning; and
config, which remains readable.OidcErrorsubclasses, includingOidcConfigError,OidcNetworkError,OidcInteractionRequired,OidcDeviceFlowError,OidcTimeoutError, andOidcCancelledError, allQuestDBErrorsubclasses. Theircodereports the failing call's nativecategory and is not a recovery instruction: with no usable credential,
auth.token()reportsAuthErrorwhile an HTTP sender flush reportsSocketError, yet retrying cannot succeed until someone signs in. CatchOidcInteractionRequiredbefore any retry logic keyed oncode; itsacquisition_busyflag tells a wait-and-retry case (another thread isacquiring) from one that needs
sign_in().insecure=Trueapplies only to QuestDB discovery transport. Attach theprovider to
https::/wss::transports: over plaintexthttp::/ws::to a non-loopback host the token travels in cleartext.
The public surface comprises
OidcDeviceAuth,OidcConfig,FileTokenStore,Renderer, the PG-wire adapter functions, and the typed exceptions. There is nopublic
TokenSet;token()returns the selected token string.OIDC itself adds no optional Python dependency. SQLAlchemy, psycopg/psycopg2,
qrcode, Pillow (for notebook PNG QR output), and IPython are imported lazily
for their respective integrations.
Rotating Bearer tokens
bearer_token_provider=works onquestdb.connect,QuestDB.from_conf,Sender,Sender.from_confandSender.from_env, and is mutuallyexclusive with
oidc_auth=,token=,username=andpassword=.Passing an
OidcDeviceAuththere raisesTypeErrorpointing atoidc_auth=, which is unchanged.questdb.auth.RefreshingTokenProvideris the native refreshing cache. Itssource -- a callable or an object with
fetch_token()-- returns anExpiringToken, a(token, expires_at)tuple, or any object withtokenand
expires_on. Azure Core'sAccessTokenandAccessTokenInfoare suchobjects, so Azure Identity works without this package importing it. The
source runs only on the provider's own thread;
token(),await_ready()and
on_rejected()release the GIL.token()and an optionalon_rejected(token, http_status), works as an application provider. It iscalled on every connection attempt, possibly on a native thread.
questdb.auth.TokenUnavailableErrorclassifies a source's failure asretryable or permanent; any other exception counts as retryable. A
connection that is waited for retries only a retryable failure, while a
connected sender keeps retrying either kind. A provider's failure surfaces
as a
TokenUnavailableError, aQuestDBErrorthat keeps the transport'serror code.
the endpoint once with a new one, unless the
WWW-Authenticatechallengenames an error a new token cannot fix. A rejection that persists is handled
by phase: a connected sender rides it out (see the breaking change above),
a store-and-forward slot left by an earlier sender rides it out and is then
left on disk for the next session, and a query fails, with its pooled
connection replaced for the next one.
live in an id-keyed registry owned by their Python wrappers, which clients
keep alive, and every callback passes the process-wide callback gate that
is drained before interpreter finalization. A provider is not usable across
fork(). A provider used overhttp::orws::, anoidc_auth=oneincluded, is warned about once per process on the
questdblogger.token_providerand
azure_*keys are rejected on every protocol, and there is noazureextra. The token-provider guide shows Entra ID and OAuth2
client-credentials recipes in application code instead. This deviates from
the specification, which proposes
token_provider=azurewith an optionalPython extra; the note asking it to make connect-string providers optional
is still to be raised on feat(qwp): add refreshing token providers for Entra ID authentication java-questdb-client#102.
Documentation and example
docs/auth.rstand API reference entries indocs/api.rst.docs/token_providers.rst, withEntra ID and OAuth2 client-credentials recipes and a section on rejections
that persist, plus API reference entries.
examples/oidc_device_auth.py, including native transport, PG-wire,manual-token, and opt-in persistence examples. The example requires a live
OIDC-enabled QuestDB Enterprise deployment and a human sign-in, so it is not
part of the automated example suite.
Test scope
test/test_auth.pycontains the Python binding/integration test suite andrequires the compiled
questdb._clientextension. It is imported bytest/test.pyfor the aggregate CI run.The focused tests cover:
views, typed errors, and the interactive/non-interactive lifecycle split;
renderer interface validation, and provider/renderer ownership and GC;
FileTokenStoreacceptance and default-location configuration;Sender,Sender.from_conf, and lazyquestdb.connect, including invalid types and fixed-token conflicts;before token retrieval or connection creation;
close(), closing from a renderercallback,
Ctrl-Cduringsign_in(),clear()afterclose(), andrejection of an explicitly closed provider at attach;
Spyder, and stderr-vs-stdout redirection), token-store path resolution, and
propagation of the native error code onto
OidcError.These Python tests use
OidcTestServerto simulate discovery, deviceauthorization, polling, token acquisition, and refresh. The human-in-the-loop
live-IdP example remains intentionally outside the automated suite.
test/test_token_provider.py, also imported bytest/test.py, covers thetoken providers:
ExpiringTokenforms and validation, andTokenUnavailableErrorclassification;
RefreshingTokenProvider: every source shape, classified and unclassifiedfailures, a forced refresh through
on_rejected,close()failing everyholder, the GIL released while waiting, the source released with the
provider, argument validation, and the protocol classes;
a
Senderand to a pool against a QWP/WebSocket mock that checks theAuthorizationheader: a 401 heard byon_rejectedand retried once, apermanent failure ending a synchronous connect and a retryable one
retried, conflicting arguments, the reserved configuration keys, the
holder's lifetime, and the cleartext warning;
reprof a token, a provideror its status.
Additional maintenance
32-bit wheels, and makes the parquet dtype expectation compatible with pandas
3's string dtype.
JDK 25.
server errors.
Interactive detection and browser opening
sign_in()prompts by default, wherever it is called from. There is deliberatelyno TTY check: a missing TTY is not evidence of a missing human, and refusing on
it turned away sign-ins that work —
prog 2>&1 | tee logat a real terminal, aprocess supervisor, an IDE run configuration. The one exception is a notebook
executed headlessly (papermill,
nbclient,nbconvert --execute), whose kernelstates at protocol level that it accepts no input; there
sign_in()raisesOidcInteractionRequiredimmediately. Passinteractive=Falseto get thatfail-fast behaviour anywhere else, such as cron or CI.
open_browseris tri-state for the same reason: the defaultNoneopens abrowser except inside a Jupyter kernel, where it may be on a different machine
from the reader;
Trueopens one anyway — correct for a localjupyter lab,where the previous unconditional suppression was simply wrong and had no
override — and
Falsenever does.Submodule pin
This branch pins
c-questdb-clientat3dc38c73896fa05623aba1c3870888cd5c7a2743,the current head of questdb/c-questdb-client#183. It adds the refreshing token
providers and phase-specific credential-failure handling on top of the OIDC
integration. Compared with the earlier
c3c58866pin, it also includes the C++test and gitleaks updates and a merge of native
main(6317e5d3, #195:maximum GEOHASH values and Arrow FFI validation). The two must merge together:
line_sender.pxdbindsthat PR's
oidc.h,client.handtoken_provider.hsurface and does notcompile against the current
main. That pin also carries changes to APIalready present on native
main:ReaderConfig::upgrade_headersnow returnsResult<Vec<_>>rather thanVec<_>for direct Rust callers, callback inboxcapacities are capped at 65,536, config strings are capped at 1 MiB, a
QWP/WebSocket sender that has connected rides out an HTTP 401 or 403 instead
of becoming terminal, and the reserved token-provider configuration keys are
rejected on every protocol. All but the Rust signature change reach Python;
CHANGELOG.rstdocuments the caps and the 401/403 change under Breakingchanges and the reserved keys under Other changes. The TTY behavior and
non-copyable C++
device_authconcern APIs introduced inside the native PRitself, so they do not break a pre-existing API.