Skip to content

Follow the SQL cursor across pages instead of returning only the first - #123

Open
rusackas wants to merge 1 commit into
preset-io:masterfrom
rusackas:fix/sql-cursor-pagination
Open

Follow the SQL cursor across pages instead of returning only the first#123
rusackas wants to merge 1 commit into
preset-io:masterfrom
rusackas:fix/sql-cursor-pagination

Conversation

@rusackas

Copy link
Copy Markdown
Member

Summary

Cursor.execute() sends one POST to _sql with fetch_size, takes rows/columns from that single response, and returns. Elasticsearch's SQL API includes a cursor field in the response whenever more rows remain past fetch_size — this driver never follows it. Any result set larger than fetch_size silently loses every row past the first page, with no error raised.

Fixes #88.

What changed

Cursor.execute() now loops on the cursor field the initial (and each subsequent) response returns, POSTing {"cursor": ...} back to the same _sql endpoint until a response comes back with no cursor (the documented signal that the result set is exhausted: https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-pagination.html). Rows from every page are concatenated into self._results, so fetchone/fetchmany/fetchall behave exactly as before from the caller's perspective — just with the full result set now.

Also added close_elastic_cursor, called (best-effort, logged not raised) if pagination exits early via an exception, so a mid-stream failure doesn't leak a still-open server-side cursor.

Testing

  • es/tests/test_cursor_pagination.py (new): mocks the transport layer to test the pagination loop directly — no live cluster needed. Confirmed these fail against the pre-fix code (only the first page's rows come back) and pass with the fix. Covers: multi-page accumulation, the single-page case is unaffected, and the cursor gets closed if a follow-up request raises mid-pagination.
  • es/tests/test_dbapi.py::test_execute_fetchall_paginates_past_fetch_size (new): a live-cluster integration test matching the existing style in this file — connects with a small fetch_size against the flights fixture (31 rows) and asserts all 31 come back. I don't have Docker available in my environment to run this one myself; it should run in this repo's existing CI matrix (test-elasticsearch7/test-elasticsearch8) the same way the rest of test_dbapi.py does. Scoped to skip on the OpenSearch/opendistro driver since this is specifically the Elasticsearch _sql cursor protocol.
  • black, flake8, mypy all pass on the changed files.

Scope note

This PR only touches es/elastic/api.py (the plain Elasticsearch driver), not es/opendistro/api.py. I didn't investigate whether OpenDistro/OpenSearch's SQL plugin has an analogous pagination gap — that'd be a separate change if it does.

Cursor.execute() sent one POST to _sql with fetch_size and returned
whatever came back, ignoring the `cursor` field the response includes
when more rows remain. Any result set larger than fetch_size silently
lost every row past the first page, with no error.

Follow the cursor to fetch all remaining pages, matching the SQL
pagination protocol:
https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-pagination.html

Also close the cursor (best-effort, logged not raised) if pagination
exits early via an exception, so a mid-stream failure doesn't leak
server-side cursor state.

Fixes preset-io#88

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Comment thread es/elastic/api.py
"Missing columns field, maybe it's an opendistro sql ep"
)
self._results = rows
self.description = get_description_from_columns(columns)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a cursor has results from an earlier query and a follow-up page request fails, _results is not cleared before this updates description. A subsequent fetchall() can then return rows from the earlier query under the new description. Could this clear pending results before pagination starts and add a regression test for the failure path?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

No pagination support?

2 participants