Follow the SQL cursor across pages instead of returning only the first - #123
Open
rusackas wants to merge 1 commit into
Open
Follow the SQL cursor across pages instead of returning only the first#123rusackas wants to merge 1 commit into
rusackas wants to merge 1 commit into
Conversation
Cursor.execute() sent one POST to _sql with fetch_size and returned whatever came back, ignoring the `cursor` field the response includes when more rows remain. Any result set larger than fetch_size silently lost every row past the first page, with no error. Follow the cursor to fetch all remaining pages, matching the SQL pagination protocol: https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-pagination.html Also close the cursor (best-effort, logged not raised) if pagination exits early via an exception, so a mid-stream failure doesn't leak server-side cursor state. Fixes preset-io#88 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
rusackas
requested review from
Vitor-Avila,
eschutho,
mistercrunch and
sadpandajoe
July 31, 2026 18:13
sadpandajoe
reviewed
Aug 20, 2026
| "Missing columns field, maybe it's an opendistro sql ep" | ||
| ) | ||
| self._results = rows | ||
| self.description = get_description_from_columns(columns) |
There was a problem hiding this comment.
If a cursor has results from an earlier query and a follow-up page request fails, _results is not cleared before this updates description. A subsequent fetchall() can then return rows from the earlier query under the new description. Could this clear pending results before pagination starts and add a regression test for the failure path?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cursor.execute()sends one POST to_sqlwithfetch_size, takesrows/columnsfrom that single response, and returns. Elasticsearch's SQL API includes acursorfield in the response whenever more rows remain pastfetch_size— this driver never follows it. Any result set larger thanfetch_sizesilently loses every row past the first page, with no error raised.Fixes #88.
What changed
Cursor.execute()now loops on thecursorfield the initial (and each subsequent) response returns, POSTing{"cursor": ...}back to the same_sqlendpoint until a response comes back with no cursor (the documented signal that the result set is exhausted: https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-pagination.html). Rows from every page are concatenated intoself._results, sofetchone/fetchmany/fetchallbehave exactly as before from the caller's perspective — just with the full result set now.Also added
close_elastic_cursor, called (best-effort, logged not raised) if pagination exits early via an exception, so a mid-stream failure doesn't leak a still-open server-side cursor.Testing
es/tests/test_cursor_pagination.py(new): mocks the transport layer to test the pagination loop directly — no live cluster needed. Confirmed these fail against the pre-fix code (only the first page's rows come back) and pass with the fix. Covers: multi-page accumulation, the single-page case is unaffected, and the cursor gets closed if a follow-up request raises mid-pagination.es/tests/test_dbapi.py::test_execute_fetchall_paginates_past_fetch_size(new): a live-cluster integration test matching the existing style in this file — connects with a smallfetch_sizeagainst theflightsfixture (31 rows) and asserts all 31 come back. I don't have Docker available in my environment to run this one myself; it should run in this repo's existing CI matrix (test-elasticsearch7/test-elasticsearch8) the same way the rest oftest_dbapi.pydoes. Scoped to skip on the OpenSearch/opendistro driver since this is specifically the Elasticsearch_sqlcursor protocol.black,flake8,mypyall pass on the changed files.Scope note
This PR only touches
es/elastic/api.py(the plain Elasticsearch driver), notes/opendistro/api.py. I didn't investigate whether OpenDistro/OpenSearch's SQL plugin has an analogous pagination gap — that'd be a separate change if it does.