Repository navigation
Conversation
The pandas serializer named the parquet file after a hash of the columns, shape and dtypes only, so two same-shape DataFrames serialized to one path shared a file and the earlier state loaded the later data. The hash now also covers the DataFrame content, using pandas row hashes and falling back to the parquet bytes for cells pandas cannot hash, such as lists. Closes apache#961
enekoreto
reviewed
Oct 17, 2026
enekoreto
left a comment
There was a problem hiding this comment.
Looks good to me. I pulled the branch and the repro from #961 now gives back [1, 2] for the first state, and the new test fails on main and passes here.
I also tried a few other frames (dict cells, NaN, datetimes, categorical, empty, string index) and they all round-trip fine. Identical frames still end up in the same file, and the name is the same across runs.
One small thing, not blocking:
- For list cells the fallback calls
to_parquet()just to get the hash, and then the frame is written again a few lines below, so that path serializes twice. Probably fine since it's only the fallback.
I only ran it on Python 3.13, not the other versions.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #961
The pandas serializer named the parquet file after a hash of the columns, shape and dtypes only. Two DataFrames with the same schema and shape, serialized with the same
pandas_kwargspath, were written to the same file, so deserializing the earlier state returned the later DataFrame's data. Persisters and the tracking client serialize state at every step, so loading or forking from an earlier step could silently return the latest DataFrame.Changes
serialize_pandas_dfinburr/integrations/serde/pandas.pynow also feeds the DataFrame content into the SHA-256 that names the file, usingpd.util.hash_pandas_object(value, index=True).hash_pandas_objectraisesTypeError; in that case the parquet bytes of the DataFrame are hashed instead, so those DataFrames keep serializing as before.df_<64 hex chars>.parquet) is unchanged. Identical DataFrames still map to the same file.How I tested this
test_serde_of_same_shape_dataframes_keeps_each_dataframeintests/integrations/serde/test_pandas.py, parametrized over scalar cells and list cells. It serializes two same-shape DataFrames with different values to the same path, checks they get different file paths, and checks thatState.deserializeof each returns its own DataFrame. Both cases fail on the current code (same file path) and pass with the change.pytest tests/integrations/serde/test_pandas.py tests/integrations/serde/test_pickle.py tests/integrations/serde/test_pydantic.py: 14 passed.pytest tests/core: 396 passed, 4 failed. The 4 failures aretest_graphviz_display.py::test_visualize_dot_output, which needs the Graphvizdotbinary that is not installed on my machine; they fail the same way without this change.Notes
Files written by earlier versions keep their old names and are still read through the path stored in the serialized state, so existing persisted state loads as before.
Checklist