Repository navigation
Conversation
PolarsJSONReader and PolarsNDJSONReader annotated schema and schema_overrides with polars' SchemaDefinition, whose forward references cannot be resolved by typing.get_type_hints in this module. That broke @load_from.json/ndjson and from_.json/ndjson with a NameError. The fields now use a runtime-resolvable Mapping or Sequence annotation, and the values are still passed to polars unchanged. Closes apache#1757
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #1757
PolarsJSONReaderandPolarsNDJSONReaderannotatedschemaandschema_overrideswithpolars._typing.SchemaDefinition. That alias is built from forward references that polars only imports underTYPE_CHECKING, sotyping.get_type_hintson either reader raisedNameError: name 'DataTypeClass' is not defined. Sinceget_required_argumentsandget_optional_argumentscallget_type_hints,@load_from.json,@load_from.ndjson,from_.jsonandfrom_.ndjsonall failed before reading any file.Changes
hamilton/plugins/polars_post_1_0_0_extensions.py,schemaandschema_overrideson both readers are now annotatedMapping[str, Any] | Sequence[str | tuple[str, Any]] | None. This resolves at runtime and still covers what polars accepts: a mapping of column name to dtype, or a sequence of names or (name, dtype) pairs. I followed the existingPolarsSpreadsheetReader.schema_overridesannotation. The values are still passed to polars unchanged.SchemaDefinitionimport because nothing else uses it.How I tested this
I added the following to
tests/plugins/test_polars_extensions.py:test_getting_type_hints_json_readers, parametrized over both readers. It checks thattyping.get_type_hintssucceeds and thatget_required_argumentsandget_optional_argumentsreturn the expected fields.test_polars_json_readers_materialize, forjsonandndjson. It writes a file, materializes it throughfrom_.json/from_.ndjsonon a driver and checks that a downstream node sees both rows.test_polars_ndjson_load_from. It uses@load_from.ndjsonwithschema_overrides=value({"a": pl.Float64})and checks that the override reaches polars.All five test cases fail on main with the
NameErrorfrom the issue and pass with this change. With polars 1.44.2,pytest tests/function_modifiers tests/plugins/test_polars*.pygives 453 passed and 2 skipped. With polars 2.0.0, the new tests also pass. The csv, parquet and feather reader tests fail on 2.0.0 both with and without this change, because of arguments that polars 2.0 removed (rechunk,row_count_offset).Notes
polars_pre_1_0_0_extension.pyhas the same annotation on its JSON reader. I left it alone because that module targets polars versions before 1.0 and this issue is about the post-1.0 readers.Checklist