Skip to content

Support file objects in Polars JSON and NDJSON readers and writers - #1763

Open
digit50 wants to merge 1 commit into
apache:mainfrom
digit50:fix/polars-json-file-objects
Open

digit50 wants to merge 1 commit into
apache:mainfrom
digit50:fix/polars-json-file-objects

Conversation

@digit50

@digit50 digit50 commented Oct 15, 2026

Copy link
Copy Markdown

closes #1758

PolarsJSONReader, PolarsJSONWriter, PolarsNDJSONReader and PolarsNDJSONWriter declare that they accept file objects (and the readers raw bytes), but they passed source/file straight to utils.get_file_metadata, which only handles paths. A BytesIO raised AttributeError: '_io.BytesIO' object has no attribute 'decode', and on the writers this happened after the data had already been written.

Changes

  • The two readers return file metadata only when source is a str or Path, as before; for file objects and bytes they return dataframe metadata from utils.get_dataframe_metadata.
  • The two writers return file and dataframe metadata only when file is a str or Path, as before; for file objects they return dataframe metadata.
  • hamilton/io/utils.py is unchanged, so metadata for paths is the same as before.

How I tested this

  • Added test_polars_json_file_objects in tests/plugins/test_polars_extensions.py, parametrized over the JSON and NDJSON pairs. It saves a frame to a BytesIO, loads it back from a BytesIO and from raw bytes, checks the frame round-trips, and checks that each returned metadata dict has the dataframe metadata (rows and column names) and no file_metadata entry. Both cases fail on main with the AttributeError above and pass with this change.
  • pytest tests/function_modifiers tests/plugins/test_polars*.py passes (450 passed, 2 skipped), the existing path-based JSON and NDJSON tests included.
  • ruff check and ruff format --check pass on the changed files.

Notes

Metadata for file objects and bytes now has no file_metadata entry, since there is no path, size or modification time to report. Previously raw bytes produced a file_metadata entry whose path was the whole file content.

Checklist

  • PR has an informative and human-readable title (this will be pulled into the release notes)
  • Changes are limited to a single goal (no scope creep)
  • Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output) (no new functions)
  • Placeholder code is flagged / future TODOs are captured in comments (no placeholder code)
  • Project documentation has been updated if adding/changing functionality. (no documentation change needed)

The Polars JSON and NDJSON loaders and savers accept file objects and bytes, but passed them to get_file_metadata, which only handles paths, so a BytesIO raised AttributeError (on the writer after the data was written). File metadata is now computed only for str and Path, and dataframe metadata is returned for file objects and bytes.

Closes apache#1758

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PolarsNDJSONWriter/Reader crash on BytesIO, writer after the data is written

1 participant