Skip to content

fix: include DataFrame content in pandas serde file name - #963

Open
digit50 wants to merge 1 commit into
apache:mainfrom
digit50:fix/pandas-serde-content-hash
Open

digit50 wants to merge 1 commit into
apache:mainfrom
digit50:fix/pandas-serde-content-hash

Conversation

@digit50

@digit50 digit50 commented Oct 15, 2026

Copy link
Copy Markdown

closes #961

The pandas serializer named the parquet file after a hash of the columns, shape and dtypes only. Two DataFrames with the same schema and shape, serialized with the same pandas_kwargs path, were written to the same file, so deserializing the earlier state returned the later DataFrame's data. Persisters and the tracking client serialize state at every step, so loading or forking from an earlier step could silently return the latest DataFrame.

Changes

  • serialize_pandas_df in burr/integrations/serde/pandas.py now also feeds the DataFrame content into the SHA-256 that names the file, using pd.util.hash_pandas_object(value, index=True).
  • For cells pandas cannot hash, such as lists, hash_pandas_object raises TypeError; in that case the parquet bytes of the DataFrame are hashed instead, so those DataFrames keep serializing as before.
  • The file name format (df_<64 hex chars>.parquet) is unchanged. Identical DataFrames still map to the same file.

How I tested this

  • Added test_serde_of_same_shape_dataframes_keeps_each_dataframe in tests/integrations/serde/test_pandas.py, parametrized over scalar cells and list cells. It serializes two same-shape DataFrames with different values to the same path, checks they get different file paths, and checks that State.deserialize of each returns its own DataFrame. Both cases fail on the current code (same file path) and pass with the change.
  • pytest tests/integrations/serde/test_pandas.py tests/integrations/serde/test_pickle.py tests/integrations/serde/test_pydantic.py: 14 passed.
  • pytest tests/core: 396 passed, 4 failed. The 4 failures are test_graphviz_display.py::test_visualize_dot_output, which needs the Graphviz dot binary that is not installed on my machine; they fail the same way without this change.

Notes

Files written by earlier versions keep their old names and are still read through the path stored in the serialized state, so existing persisted state loads as before.

Checklist

  • PR has an informative and human-readable title (this will be pulled into the release notes)
  • Changes are limited to a single goal (no scope creep)
  • Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future TODOs are captured in comments
  • Project documentation has been updated if adding/changing functionality.

The pandas serializer named the parquet file after a hash of the columns, shape and dtypes only, so two same-shape DataFrames serialized to one path shared a file and the earlier state loaded the later data. The hash now also covers the DataFrame content, using pandas row hashes and falling back to the parquet bytes for cells pandas cannot hash, such as lists.

Closes apache#961
@github-actions github-actions Bot added the area/integrations External integrations (LLMs, frameworks) label Oct 15, 2026

@enekoreto enekoreto left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. I pulled the branch and the repro from #961 now gives back [1, 2] for the first state, and the new test fails on main and passes here.

I also tried a few other frames (dict cells, NaN, datetimes, categorical, empty, string index) and they all round-trip fine. Identical frames still end up in the same file, and the name is the same across runs.

One small thing, not blocking:

  • For list cells the fallback calls to_parquet() just to get the hash, and then the frame is written again a few lines below, so that path serializes twice. Probably fine since it's only the fallback.

I only ran it on Python 3.13, not the other versions.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/integrations External integrations (LLMs, frameworks)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pandas serde overwrites same-shape DataFrames, earlier state loads later data

2 participants