Repository navigation
Conversation
The Polars JSON and NDJSON loaders and savers accept file objects and bytes, but passed them to get_file_metadata, which only handles paths, so a BytesIO raised AttributeError (on the writer after the data was written). File metadata is now computed only for str and Path, and dataframe metadata is returned for file objects and bytes. Closes apache#1758
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #1758
PolarsJSONReader,PolarsJSONWriter,PolarsNDJSONReaderandPolarsNDJSONWriterdeclare that they accept file objects (and the readers rawbytes), but they passedsource/filestraight toutils.get_file_metadata, which only handles paths. ABytesIOraisedAttributeError: '_io.BytesIO' object has no attribute 'decode', and on the writers this happened after the data had already been written.Changes
sourceis astrorPath, as before; for file objects andbytesthey return dataframe metadata fromutils.get_dataframe_metadata.fileis astrorPath, as before; for file objects they return dataframe metadata.hamilton/io/utils.pyis unchanged, so metadata for paths is the same as before.How I tested this
test_polars_json_file_objectsintests/plugins/test_polars_extensions.py, parametrized over the JSON and NDJSON pairs. It saves a frame to aBytesIO, loads it back from aBytesIOand from rawbytes, checks the frame round-trips, and checks that each returned metadata dict has the dataframe metadata (rows and column names) and nofile_metadataentry. Both cases fail on main with theAttributeErrorabove and pass with this change.pytest tests/function_modifiers tests/plugins/test_polars*.pypasses (450 passed, 2 skipped), the existing path-based JSON and NDJSON tests included.ruff checkandruff format --checkpass on the changed files.Notes
Metadata for file objects and
bytesnow has nofile_metadataentry, since there is no path, size or modification time to report. Previously rawbytesproduced afile_metadataentry whosepathwas the whole file content.Checklist