feat: add built-in frozendict support for Python 3.15+ - #274
Open
XuehaiPan wants to merge 34 commits into
Open
Conversation
XuehaiPan
added
enhancement
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #274 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 15 15
Lines 1623 1655 +32
Branches 218 225 +7
=========================================
+ Hits 1623 1655 +32 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. |
XuehaiPan
force-pushed
the
frozendict
branch
4 times, most recently
from
March 25, 2026 15:25
3426921 to
12a3276
Compare
XuehaiPan
force-pushed
the
frozendict
branch
4 times, most recently
from
April 3, 2026 06:14
fe42e10 to
0faccef
Compare
XuehaiPan
force-pushed
the
frozendict
branch
5 times, most recently
from
May 11, 2026 17:39
19952e3 to
7656326
Compare
Register `frozendict` (Python 3.15.0a7+) as a built-in PyTree node type with key-sorted traversal, matching the behavior of `dict` and `collections.defaultdict`. The `dict_insertion_ordered` context manager also affects `frozendict`. Changes across 29 files: - C++ enum `PyTreeKind::FrozenDict` and `PyFrozenDictTypeObject` macro (guarded by `PY_VERSION_HEX >= 0x030F00A7`) - All treespec switch statements updated (flatten, unflatten, serialize, hash, compare, traverse, construct, paths, accessors, entries) - Python registry with sorted and insertion-ordered flatten/unflatten - `treespec_frozendict()` constructor, `STANDARD_DICT_TYPES` expansion, `PyTree[T]` type union update - `optree.treespec.frozendict` namespace alias - README, Sphinx docs, and spelling wordlist - Test data extended (3 frozendict entries in TREES fixture), subinterpreter test tree includes frozendict, registry size checks - Minor: empty deque repr improved (`deque()` instead of `deque([])`)
Sphinx autodoc/autosummary imports symbols during the reading phase regardless of ``.. only::`` directives, causing import errors on Python < 3.15 where ``treespec_frozendict`` does not exist. Remove the entries for now with TODO comments to add them back when building with Python 3.15+.
PyPy/Windows jobs intermittently fail in pip steps with transient network errors (e.g. `OSError: Connection broken: IncompleteRead`) while a nested build-isolation pip downloads build dependencies such as `scikit-build-core` for nightly `pybind11`. Set `PIP_RETRIES=10` (default 5) and `PIP_TIMEOUT=120` (default 15s) in the `env:` block of all test/build/lint workflows. As environment variables these propagate into nested build-isolation pip subprocesses (unlike CLI flags on the outer `pip install`), covering connection setup, DNS, and metadata-fetch hiccups that account for most CI pip failures. Also add both vars to cibuildwheel's `environment-pass` so they cross into the isolated (manylinux) wheel-build containers, which would otherwise not inherit the host environment. Note: a mid-download `IncompleteRead` can still escape pip's internal retry; a command-level retry would be the next lever if it recurs.
The pyenv-win bundled `.versions_cache.xml` lags behind python.org, and the `pyenv update` command is unusable on GitHub runners: its VBScript updater parses release listings via the Internet Explorer `htmlfile` COM object, which modern Windows images reject with "This command is not supported". Run the pyenv-win stdlib updater (`.github/scripts/update_versions_cache.py`) right after cloning so `pyenv install` can resolve the newest CPython release (3.15.0b3), which ships the gh-149786 fix for the free-threaded venv launcher that previously broke the 3.15 free-threaded debug build venv/ensurepip.
…urface Extend three existing tests with `frozendict` cases (gated on Python 3.15+) to exercise paths that were previously untested: - `test_tree_broadcast_common`: the `PyTreeKind::FrozenDict` arm in `BroadcastToCommonSuffix`. - `test_different_metadata`: the `frozendict:` key-mismatch message emitted by `FlattenUpTo`. - `test_pytree_typing`: the `optree.typing.FrozenDict` alias, its `__all__` position, and the `frozendict[Any, ...]` member of the `PyTree` generic union.
XuehaiPan
force-pushed
the
main
branch
2 times, most recently
from
July 8, 2026 10:51
51ec4bb to
b5d273b
Compare
Integrate the correctness sweep from metaopt#290 with the built-in `frozendict` support for Python 3.15+ (PEP 814). Explicit conflicts: - `optree/ops.py`: `tree_broadcast_common()` was rewritten upstream to delegate to the new `_tree_broadcast_common_with_treespec()` helper. Took the upstream refactor and dropped our now-superseded inline body. - `src/treespec/serialization.cpp`: `PyTreeSpec::FromPicklable()` was rewritten upstream into a validating, exhaustive per-kind switch. Wove `PyTreeKind::FrozenDict` through the new structure instead of the old `t[...]` accessors, and moved our cross-version rejection guard below upstream's new kind-range check. Implicit conflicts (no textual overlap, both directions): - `src/treespec/constructors.cpp`: upstream's new `depends_on_namespace` predicate listed only `Dict`/`DefaultDict`. `frozendict` is equally affected by `dict_insertion_ordered`, so a `frozendict` treespec built by `treespec_from_collection()` under an insertion-ordered namespace had its namespace erased, yielding a spec with insertion-ordered keys that claims the global namespace and that no `tree_flatten()` can produce. - `src/treespec/serialization.cpp`: upstream's `ToPicklable()` now copies the node's mutable key list for `Dict`/`OrderedDict`/`DefaultDict` so the pickled state cannot alias the immutable spec. `FrozenDict` stores the same `py::list` and was left aliasing, reintroducing the bug upstream had just fixed. - `optree/ops.py`, `optree/typing.py`: upstream introduced two new locals named `sentinel`, which Python 3.15 added as a builtin (PEP 661); the new comprehension variable is renamed and the new local suppressed. Confirmed load-bearing: pylint running on 3.15 reports `W0622 redefined-builtin` for a bare `sentinel`. Follow-up fixes from the post-merge review: - `tests/helpers.py`: `TREE_PATHS` and `TREE_ACCESSORS` were built BEFORE the version-gated block that rebinds the per-mode lists, so they kept the pre-`frozendict` 40-element objects while `TREES` grew to 43. Every `zip(TREES, ...)` consumer silently truncated, leaving `PyTreeKind.FROZENDICT` with zero path and accessor coverage. The tables are now built after the block, and an import-time assertion pins their lengths to `TREES`. - `tests/helpers.py`: `is_dict()` used a bare `isinstance(dct, dict)`, but `frozendict` is not a `dict` subclass, so the `is_leaf` matrix skipped the `frozendict` trees entirely. - `optree/typing.py`: the `PyTree` and `PyTreeTypeVar` docstrings pin the union rendering and omit the `frozendict` member inserted on 3.15+; documented via a version note rather than in the doctest output, which is exercised on 3.13 where `frozendict` does not exist. - `optree/ops.py`: the `treespec_frozendict()` fallback `raise` lacked a coverage pragma, so it read as uncovered on 3.15+ where it is unreachable. - `optree/registry.py`: `DictMetaData` also backs `frozendict` nodes. Tests: - `test_treespec_frozendict_from_collection_retains_insertion_ordered_namespace` covers the namespace-retention fix. - `test_treespec_frozendict_pickled_state_does_not_alias_keys` covers the `__getstate__`/`__setstate__` aliasing fix, mirroring the upstream tests for the other dict kinds. It compares the repr of the STATE, not of the treespec, since the latter never reads `original_keys`. - `test_treespec_frozendict_setstate_rejected_on_build_without_support` covers the cross-version pickle guard, via both `__setstate__` and a hand-assembled pickle payload. It runs on every build WITHOUT `frozendict` support, i.e. the majority of the CI matrix. - `test_treespec_setstate_rejects_builtin_custom_type` now includes `frozendict` in its built-in type enumeration, which its own comment declares exhaustive. Validation, across three interpreters: - CPython 3.15.0b4 (release, `frozendict` enabled): 100839 passed, 7 skipped, 1 xfailed, no failures. `pylint optree` rates 10.00/10. - CPython 3.14.6 (`frozendict` unavailable): 94197 passed, 5 skipped, 1 xfailed. `pre-commit run --all-files` clean. - CPython 3.16.0a0 free-threaded debug (`frozendict` enabled): 100838 passed, 7 skipped, 1 xfailed, plus one pre-existing unrelated failure. The 3.16-only failure is `test_dataclasses.py::test_same_signature`: CPython 3.16 added a `qualname` parameter to `dataclasses.make_dataclass()` that `optree.dataclasses` has not yet mirrored. It does not occur on 3.15 or 3.14, and `qualname` is absent at the merge base and on both parents, so it is unrelated to this merge.
A four-way audit of the repository -- build/CI config, test suite,
library API surface, and documentation -- for places where a `dict` /
`OrderedDict` / `defaultdict` peer exists but `frozendict` was missing.
Correctness:
- `optree/registry.py`: `_frozendict_flatten()` and
`_frozendict_insertion_ordered_flatten()` delegate to `_dict_flatten()`
and `_dict_insertion_ordered_flatten()`, whose parameters were
annotated `dict[KT, VT]`. `frozendict` is NOT a `dict` subclass, so
this was a genuine type error; both now take `Mapping[KT, VT]`, which
they already satisfied (the bodies only call `.items()`). Matches
typeshed, which declares `frozendict(Mapping[_KT, _VT])`.
- `include/optree/pymacros.h`: `OPTREE_HAS_FROZENDICT` lacked the
`!defined(PYPY_VERSION)` guard that `OPTREE_HAS_SUBINTERPRETER_SUPPORT`
carries. `PyFrozenDict_Type` lives in CPython's
`Include/cpython/dictobject.h`, and PyPy reports the `PY_VERSION_HEX`
of the CPython level it emulates, so a future PyPy claiming 3.15 would
enable the macro and fail to link.
- Six stale `# type: ignore[import]` / `[type-arg]` comments on
`frozendict` lines: typeshed already declares `frozendict`, and
declares it generic, so these were unnecessary and would become hard
errors under `warn_unused_ignores`.
- `optree/typing.py`: the `FrozenDict` union member carried
`# type: ignore[valid-type]` where mypy reports `arg-type` (it goes
through `list.append`, not a list literal like its neighbours).
Together these take `mypy --python-version=3.15 optree` from 15 errors
to 5; the remaining 5 are pre-existing and unrelated. They were hidden
because `[tool.mypy] python_version = "3.9"` is rejected by mypy >= 2.3,
which falls back to 3.10 and prunes every `sys.version_info >= (3, 15)`
block as unreachable. That setting is deliberately left alone here:
correcting it also surfaces the 5 unrelated errors and is a project-wide
decision, not a `frozendict` one.
Tests. Three gaps were proven undetected by mutation testing -- each
mutation passed the entire suite before these were added, and is caught
after (1, 4 and 1 failures respectively):
- `_frozendict_unflatten()` rebuilding in sorted rather than original
insertion order, now covered by
`test_python_frozendict_unflatten_preserves_original_insertion_order`
and `test_python_frozendict_flatten_returns_dict_metadata`.
- `register_pytree_node.get(frozendict)` ignoring
`dict_insertion_ordered()`, now covered in
`test_pytree_node_registry_get`.
- `AutoEntry` dispatching `frozendict` to `FlattenedEntry` instead of
`MappingEntry`. `frozendict` is a virtual `Mapping`, not a `dict`
subclass, so it relies on the `issubclass(type, Mapping)` branch; a
user registering a `frozendict` subclass was silently mis-dispatched.
Cross-version pickling, in `test_treespec_frozendict_pickle_cross_version`:
a treespec pickled on a 3.15+ build names `PyTreeKind::FrozenDict`, whose
enum value exists everywhere. The test runs on every interpreter and
asserts whichever half applies -- rejection without `frozendict` support,
faithful reconstruction with it. The payload is built by retagging a
plain `dict` spec's state rather than checked in as an opaque blob, and
on supporting builds the retagged state is asserted equal to the one that
build emits natively, so the stand-in is provably faithful. Verified by
deleting the `#if !defined(OPTREE_HAS_FROZENDICT)` guard from
`FromPicklable` and rebuilding: the test fails.
Further coverage, correct today but previously unpinned:
- The negative contract on builds WITHOUT `frozendict` support, which is
most of the CI matrix: `test_frozendict_unsupported_build_surface`
pins that no dict-family surface claims `frozendict`, that no tree can
flatten to a `FROZENDICT` node, and that the always-importable
constructors raise rather than silently degrading (verified by
mutation). `test_frozendict_kind_is_defined_on_every_build` pins the
enum value at 11 on every build, since it is part of the pickle
format. `test_pytree_type` gained the matching `else` branch.
- A `FROZENDICT` arm in the malformed-`__setstate__` matrix.
`FrozenDict` rides on shared `||` conditions with `Dict` at ~10 sites
in `PyTreeSpec::FromPicklable`; dropping one would let a crafted
pickle build a corrupt spec unnoticed.
- `test_treespec_pickle_all_protocols_roundtrip` covered only a plain
`dict`, despite protocols 0/1 taking the `copyreg.__newobj__` path
that upstream had just repaired; it now covers all four dict kinds and
asserts the restored spec still unflattens identically.
- Cross-kind broadcasting. The existing assertions were
frozendict-vs-frozendict only; `frozendict` now broadcasts against
`dict` / `OrderedDict` / `defaultdict` in both operand orders with the
result types asserted explicitly -- `frozendict({'a': 1}) == {'a': 1}`
is `True`, so `==` alone would pass even if the node kinds collapsed.
Documentation:
- `dict_insertion_ordered()` -- the canonical doc for the feature -- said
only "a dictionary", which reads as including `OrderedDict`
(unaffected) and omits `frozendict` (affected).
- `treespec_frozendict()` had 2 doctest examples where every sibling
constructor has 6; the four missing ones are added, each executed on
3.15 first. `+SKIP` is retained: the symbol is always defined but only
callable on 3.15+, and `make doctest` runs on every supported Python.
- `PyTreeKind.FROZENDICT`'s enum docstring in `src/optree.cpp` renders
into the docs on every Python and now carries the same `(Python 3.15+)`
caveat as `optree/_C.pyi`.
- `DictMetaData`, `optree.treespec`'s module docstring, `CHANGELOG.md`,
and the `Returns:` role in `treespec_frozendict()`.
Validation: 100858 passed / 7 skipped / 1 xfailed on CPython 3.15.0b4;
94199 passed / 22 skipped / 1 xfailed on CPython 3.14.6.
`pre-commit run --all-files` clean.
Comment review across the files this branch changes. Trimmed, all of them cases where the comment restated the code below it or repeated a rationale already given elsewhere: - `test_treespec_frozendict_pickle_cross_version`: a 10-line header with a bulleted list, plus the payload-construction rationale stated twice. - Four comments that paraphrased the assertion immediately following them. - The `FromPicklable` cross-version guard and the `OPTREE_HAS_FROZENDICT` PyPy guard, both longer than the surrounding house style. Corrected, not merely shortened: the `MakeNode` and `GetType` guards claimed they were "reachable only via a treespec deserialized from a Python 3.15+ build". That stopped being true once `FromPicklable` gained its own guard, which now rejects those states first, so both are unreachable defence in depth. The comments say so. Em-dashes and ASCII `--` used as dashes are replaced with ordinary punctuation, including the pre-existing occurrences in `README.md`, `optree/registry.py` and `src/CMakeLists.txt`. No functional change. 100858 passed on CPython 3.15.0b4, 94199 on 3.14.6, `pre-commit run --all-files` clean.
The Windows jobs wrap pytest in `cdb -gG -o -c "..."`. cdb's default for the `eh` event is "notify", so it halts and resumes the debuggee, and prints a line, for every first-chance C++ exception. optree raises a great many of those on entirely successful paths: `PyTreeIter::Next()` ends iteration with `throw py::stop_iteration()` (`src/treespec/traversal.cpp:164`), which is the only way a native `__next__` can terminate a Python loop. On the 3.13td Windows run of metaopt#274 that produced 175,715 notifications out of 274,593 log lines, 59% of them from `test_tree_iter_thread_safe` alone and a further 17% from one `range(5000)` unregistration stress test. Error-path tests using `pytest.raises` account for well under 1%. That job reached 99% of the suite before hitting the 3h cap. It is chronically close to the limit: 123 min on `main`, 153 on another PR branch, 174 and then 180 on this one. `-xi eh` sets first-chance handling for C++ exceptions to "ignore". Access violations and other genuine faults are separate events and still break, dump and report, so crash diagnostics are unaffected. The flag cannot be moved into the `-c` string, which `-g -G` defer until the first break. Applied to the five invocations that run a program. The two `cdb -z` post-mortem analyzers are left alone: they read an existing dump and need full exception detail for `!analyze -vv` and `.ecxr`.
cibuildwheel 4.2.0 builds CPython 3.15 by default, bundling 3.15.0rc1, which upstream guarantees is ABI compatible with the final release. That removes the reason this branch set `ALLOW_PRERELEASES: "true"`. The variable does double duty: it adds `cpython-prerelease` to `CIBW_ENABLE`, which the cp315/cp315t targets needed, but it also gates off the TestPyPI and PyPI publish steps, sets `CIBW_ALLOW_EMPTY` and downgrades `if-no-files-found` to a warning. A tag push from this branch would therefore have run the whole matrix and uploaded nothing, and any target producing zero wheels would have passed silently. Verified with `cibuildwheel --print-build-identifiers` against this `pyproject.toml`, with no `CIBW_ENABLE` set: 4.1.1 cp313, cp314, cp314t 4.2.0 cp313, cp314, cp314t, cp315, cp315t and that no target goes empty under the stricter settings: `cp315` identifiers are produced for the ios and android platforms too. The separate cp313t job stays on `cibuildwheel~=3.4` (4.0 dropped that target) and still resolves `cp313t-*` from `pypy cpython-freethreading` alone, so it is unaffected by dropping the prerelease enable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Register
frozendict(PEP 814, Python 3.15.0a7+) as a built-in PyTree node type with key-sorted traversal, matching the behavior ofdictandcollections.defaultdict. Thedict_insertion_orderedcontext manager also affectsfrozendict.Also improves the empty deque representation:
deque()instead ofdeque([]).Motivation and Context
Python 3.15 introduces
frozendictas a built-in immutable mapping type (PEP 814). As a fundamental container type, it should be supported as a built-in PyTree node — just likedict,OrderedDict, anddefaultdict.Types of changes
Implemented Tasks
PyTreeKind::FrozenDictenum andPyFrozenDictTypeObjectmacro (guarded byPY_VERSION_HEX >= 0x030F00A7)treespec_frozendict()constructor,STANDARD_DICT_TYPESexpansion,PyTree[T]type unionoptree.treespec.frozendictnamespace aliasChecklist
make format. (required)make lint. (required)make testpass. (required)