Support CLDR 49 by tomasr8 · Pull Request #1427 · python-babel/babel

tomasr8 · 2026-11-03T20:21:46Z

CDLR 49 is out, let's see how much stuff broke :D https://cldr.unicode.org/downloads/cldr-49

For the record, this PR is to just make the test suite pass and patch any breaking changes. CLDR 49 is including some new data which we can make use of (e.g. #1383) but that is better left to a follow up.

The main changes here are twofold:

Changes to zones/metazones -> they can now contain aliases and the import script could not handle them
Locales below Basic coverage have been split to a separate file. I think this warrants a discussion on how we want to deal with this. Do we keep the locales for now (the plan is to remove any below-Basic locales in CLDR 50) or do we already remove them? I did decide to keep them here since some of them might survive beyond CLDR 50 so removing them now would be a bit premature, but as you can see it adds some complications to the import script. Would like to know what you think.

metazones can now contain aliases, which the parser currently cannot handle. This updates the import script to return an `Alias` instance for aliased metazones.

codecov · 2026-11-03T20:24:05Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.66%. Comparing base (7576e5d) to head (18ce81e).
⚠️ Report is 8 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1427      +/-   ##
==========================================
+ Coverage   93.57%   93.66%   +0.08%     
==========================================
  Files          27       27              
  Lines        4951     4971      +20     
==========================================
+ Hits         4633     4656      +23     
+ Misses        318      315       -3

Flag	Coverage Δ
macos-14-3.10	`92.69% <ø> (+0.09%)`	⬆️
macos-14-3.11	`92.63% <ø> (+0.09%)`	⬆️
macos-14-3.12	`92.89% <ø> (+0.08%)`	⬆️
macos-14-3.13	`92.89% <ø> (+0.08%)`	⬆️
macos-14-3.14	`92.87% <ø> (+0.08%)`	⬆️
macos-14-3.15	`92.87% <ø> (+0.08%)`	⬆️
macos-14-3.8	`92.57% <ø> (+0.09%)`	⬆️
macos-14-3.9	`92.63% <ø> (+0.09%)`	⬆️
macos-14-pypy3.10	`92.69% <ø> (+0.09%)`	⬆️
ubuntu-24.04-3.10	`92.71% <ø> (+0.09%)`	⬆️
ubuntu-24.04-3.11	`92.65% <ø> (+0.09%)`	⬆️
ubuntu-24.04-3.12	`92.91% <ø> (+0.08%)`	⬆️
ubuntu-24.04-3.13	`92.91% <ø> (+0.08%)`	⬆️
ubuntu-24.04-3.14	`92.89% <ø> (+0.08%)`	⬆️
ubuntu-24.04-3.15	`92.89% <ø> (+0.08%)`	⬆️
ubuntu-24.04-3.8	`92.59% <ø> (+0.09%)`	⬆️
ubuntu-24.04-3.9	`92.65% <ø> (+0.09%)`	⬆️
ubuntu-24.04-pypy3.10	`92.71% <ø> (+0.09%)`	⬆️
windows-2022-3.10	`92.71% <ø> (+0.09%)`	⬆️
windows-2022-3.11	`92.65% <ø> (+0.09%)`	⬆️
windows-2022-3.12	`92.91% <ø> (+0.08%)`	⬆️
windows-2022-3.13	`92.91% <ø> (+0.08%)`	⬆️
windows-2022-3.14	`92.89% <ø> (+0.08%)`	⬆️
windows-2022-3.15	`92.89% <ø> (+0.08%)`	⬆️
windows-2022-3.8	`92.68% <ø> (+0.09%)`	⬆️
windows-2022-3.9	`92.64% <ø> (+0.09%)`	⬆️
windows-2022-pypy3.10	`92.71% <ø> (+0.09%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

codspeed · 2026-11-03T20:24:18Z

Merging this PR will not alter performance

✅ 88 untouched benchmarks

_{Comparing tomasr8:cldr49 (18ce81e) with master (63c168e)}

tomasr8 · 2026-11-03T20:23:06Z

    'M': [1, 2, 3, 4, 5], 'L': [1, 2, 3, 4, 5],                         # month
    'w': [1, 2], 'W': [1],                                              # week
-    'd': [1, 2], 'D': [1, 2, 3], 'F': [1], 'g': None,                   # day
+    'd': [1, 2, 3], 'D': [1, 2, 3], 'F': [1], 'g': None,                # day


There's a new ddd (dayOfMonth) pattern now. This is just to make the import script not bail, the ddd pattern does not do anything useful (yet)

tomasr8 · 2026-11-03T20:23:32Z

    '1,099.98 US dollars'
-    >>> format_currency(1099.98, 'USD', locale='ee', format_type='name')
-    'us ga dollar 1,099.98'
+    >>> format_currency(1099.98, 'USD', locale='my', format_type='name')


my is one of the few locales that still put the amount at the end

tomasr8 · 2026-11-03T20:24:39Z

+COMMON_FILESUM = 'c704a2baf9ce4e28f40e242f8b2fbd0123579bb2c56daaaf8a92d72abd02485a1f60095c3cc4fe89eaf2d5c9f5fd07954c2eaee88839f97e18b1ff37cd62e2d8'
+EXEMPLARS_FILESUM = '8c0331f4927833499d24a716e2d072a87ae169098d41886d3ae51028e4f5033f8c139339c2c8b073b59da8431663c91ed15d24a61aaa78305994e22326521836'
+
+# In CLDR 49, the common zip no longer contains the locales below Basic


We need to decide if we want to keep the locales with low coverage or get rid of them.. This is the alternative where we keep them for now until CLDR 50 removes them.

Yeah, let's keep them for now, I think. I'd maybe preferably like an RFC issue where the user community could chip in, but I don't think people who rely on the low-covered locales would ever find it.

Yeah I prefer "doing nothing" (i.e. keeping all the locales as is) until CLDR decides what to do.. Perhaps if it was just a few locales it'd be fine, but with this many there's a good chance people are using some of them.

tomasr8 · 2026-11-03T20:26:06Z

+        meta_zones[mzid] = _parse_zone_names(elem, ['meta_zones', mzid])
+
+
+def _parse_zone_names(elem, ctxt):


metazones can now have aliases, e.g.:

<metazone type="British"><long> <alias source="locale" path="../../metazone[@type='GMT']/long"/> </long></metazone>

tomasr8 · 2026-11-03T20:26:26Z



-@pytest.mark.parametrize("test_case", toml_test_cases_path.glob("bad.*.toml"), ids=lambda p: p.name)
+@pytest.mark.parametrize("test_case", tuple(toml_test_cases_path.glob("bad.*.toml")), ids=lambda p: p.name)


this is just to silence a pytest deprecation warning

tomasr8 · 2026-11-03T20:26:58Z

        ("Europe/Kiev", {"locale": "en"}, "Eastern European Time"),
+        # Since CLDR 49, the `British` and `Irish` metazones alias the long names
+        # they don't define themselves to the `GMT` metazone (in root.xml)
+        ("Europe/London", {"locale": "en", "zone_variant": "generic"}, "British Time"),


Additional test cases for the British/Irish metazone changes

tomasr8 · 2026-11-03T20:28:27Z



+def test_locale_count():
+    assert len(localedata.locale_identifiers()) == 1147


Now that some locales might be removed, a simple smoke test to keep track of the number of locales could be handy?

Might want to add a docstring explaining what this test is for? We have the context here in the PR, but it'd be convenient for it to be in the code too.

Added in 5f7c842

tomasr8 · 2026-11-03T20:41:32Z

cc @akx :)

tomasr8 · 2026-11-04T08:08:30Z

(not entirely sure why the Windows test failed, especially since one of the Ubuntu tests passed..)

akx · 2026-11-04T08:39:38Z

(not entirely sure why the Windows test failed, especially since one of the Ubuntu tests passed..)

It was a checksum failure, restarting the job fixed it.

akx

Very small comments, looks great. Thanks for doing the work!

akx · 2026-11-04T08:47:13Z

-        >>> Locale('en', 'US').time_zones['Europe/London']['long']['daylight']
-        'British Summer Time'


Review note to self: this is now in meta_zones -> British -> long.

akx · 2026-11-04T08:48:43Z

+# In CLDR 49, the common zip no longer contains the locales below Basic
+# coverage. See: https://cldr.unicode.org/downloads/cldr-49#migration. The
+# stated plan is to remove all locales below Basic coverage in CLDR 50. Since
+# presumably some locales will achieve Basic coverage in CLDR 50, we should keep
+# all the existing below-Basic coverage locales as is until CLDR 50 makes it
+# official.
+#
+# The below-Basic locales have been moved to a separate file
+# ("cldr-exemplars-delta-49.zip"). This file however contains also ~1500 files
+# from the CLDR repository's `exemplars/` directory. These are locales that
+# don't have data in CLDR and were never available in Babel. We only want to
+# merge below-Basic locales with the common data while keeping the exemplars out.
+# To do so, we generate an explicit list of all the below-Basic locales.
+#
+# To regnerate the list, run:
+#
+#   git clone --depth 1 --branch release-49 --filter=blob:none --no-checkout https://github.com/unicode-org/cldr
+#   comm -13 <(unzip -Z1 cldr-common-49.zip 'common/main/*.xml' | sort) \
+#            <(git -C cldr ls-tree --name-only release-49 common/main/ | grep '\.xml$' | sort) \
+#     | sed 's|common/main/||; s|\.xml$||'
+#
+BELOW_BASIC_LOCALES = frozenset(


Excellent comment, thanks!

akx · 2026-11-04T08:50:47Z

+COMMON_FILESUM = 'c704a2baf9ce4e28f40e242f8b2fbd0123579bb2c56daaaf8a92d72abd02485a1f60095c3cc4fe89eaf2d5c9f5fd07954c2eaee88839f97e18b1ff37cd62e2d8'
+EXEMPLARS_FILESUM = '8c0331f4927833499d24a716e2d072a87ae169098d41886d3ae51028e4f5033f8c139339c2c8b073b59da8431663c91ed15d24a61aaa78305994e22326521836'
+
+# In CLDR 49, the common zip no longer contains the locales below Basic


Yeah, let's keep them for now, I think. I'd maybe preferably like an RFC issue where the user community could chip in, but I don't think people who rely on the low-covered locales would ever find it.

akx · 2026-11-04T08:54:09Z

+        for child in width_elem:
+            if child.tag == 'alias':
+                info[width] = Alias(_translate_alias([*ctxt, width], child.attrib['path']))
+                break
+            names[child.tag] = str(child.text)
+        else:
+            if names:
+                info[width] = names


The for: else: construct took me a couple of reads. Could you add a comment clarifying the else branch (which, IIUC, we take if we didn't encounter an alias for this width?)

Sure! I'll either add a comment or if removing the for-else construct makes it simpler, I'll do that

Added a comment in 18ce81e

akx · 2026-11-04T08:55:53Z



+def test_locale_count():
+    assert len(localedata.locale_identifiers()) == 1147


Might want to add a docstring explaining what this test is for? We have the context here in the PR, but it'd be convenient for it to be in the code too.

tomasr8 · 2026-11-04T19:48:57Z

I'm curious, do you want to make a new release before merging this? That way we get a release with CLDR 48 (the latest babel release has CLDR 47) or do we go straight to 49?

tomasr8 added 12 commits November 3, 2026 10:22

Add new date pattern

cff985f

Update test_get_cldr_version

98cb2eb

Update download script to fetch exemplars

62fb973

Fix get_cldr_version doctest

9b711ac

Fix test_list_currencies

928256c

Fix format_currency doctest

8d084d5

Fix test_format_currency_long_display_name

8c944ae

Fix test_get_currency_unit_pattern

0f84b67

Fix datetime_skeletons doctest

d765632

Fix zone and metazone parsing

b34feac

metazones can now contain aliases, which the parser currently cannot handle. This updates the import script to return an `Alias` instance for aliased metazones.

Filter out exemplars and keep only below-Basic locales

2bb76dc

Fix pytest deprecation warning

2f18f08

tomasr8 commented Nov 3, 2026

View reviewed changes

Make linter happy

51392e8

akx self-assigned this Nov 4, 2026

akx requested review from akx and a lite review from Copilot and removed request for Copilot November 4, 2026 07:22

akx added enhancement area/CLDR feature labels Nov 4, 2026

akx requested changes Nov 4, 2026

View reviewed changes

akx added this to the Babel 2.19 milestone Nov 4, 2026

tomasr8 added 2 commits November 4, 2026 20:38

Add docstring to test_locale_count

5f7c842

Add a comment to explain a for-else block

18ce81e

		meta_zones[mzid] = _parse_zone_names(elem, ['meta_zones', mzid])


		def _parse_zone_names(elem, ctxt):



		@pytest.mark.parametrize("test_case", toml_test_cases_path.glob("bad.*.toml"), ids=lambda p: p.name)
		@pytest.mark.parametrize("test_case", tuple(toml_test_cases_path.glob("bad.*.toml")), ids=lambda p: p.name)



		def test_locale_count():
		assert len(localedata.locale_identifiers()) == 1147

		>>> Locale('en', 'US').time_zones['Europe/London']['long']['daylight']
		'British Summer Time'

Conversation

tomasr8 commented Nov 3, 2026

Uh oh!

codecov Bot commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codspeed Bot commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tomasr8 commented Nov 3, 2026

Uh oh!

tomasr8 commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akx commented Nov 4, 2026

Uh oh!

akx left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tomasr8 commented Nov 4, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Nov 3, 2026 •

edited

Loading

codspeed Bot commented Nov 3, 2026 •

edited

Loading

tomasr8 commented Nov 4, 2026 •

edited

Loading