Repository navigation
Fix opening non-ASCII file names with an ASCII locale - #203
Merged
Merged
Conversation
Member
It was always a bit messy there, and if you are motivated to have something more coherent, I am fine with that. |
sbraz
added a commit
to sbraz/MediaInfoLib
that referenced
this pull request
Nov 3, 2026
The MediaInfoA_* and MediaInfoListA_* functions decoded file names with the locale encoding, so under the C locale, non-ASCII names couldn't be opened [1]. Decode them with FileName_From_Native(), which uses UTF-8 under an ASCII locale and on macOS and Android, as ZenLib does when opening files [2]. Other strings, such as option values, still use the locale encoding. [1] MediaArea#2464 [2] MediaArea/ZenLib#203 Assisted-by: Claude:claude-opus-5-5
sbraz
added a commit
to sbraz/MediaInfo
that referenced
this pull request
Nov 3, 2026
Command line arguments were decoded with the locale encoding, so under the C locale, non-ASCII file names couldn't be opened. Decode them with FileName_From_Native(), which uses UTF-8 under an ASCII locale and on macOS and Android, as ZenLib does when opening files [1]. The output still uses the locale encoding. [1] MediaArea/ZenLib#203 Assisted-by: Claude:claude-opus-5-5
This was referenced Nov 3, 2026
Contributor
Author
|
Thanks! I opened MediaArea/MediaInfoLib#2767 and MediaArea/MediaInfo#1331. Testing them showed that this PR should be merged together with them. On macOS, the C locale maps bytes 0x80–0xFF to U+0080–U+00FF and back, so the narrow API and the CLI could open files with UTF-8 names before. With this PR alone, these names are encoded to UTF-8 twice and the files can't be opened. With all three PRs, they open on every platform I tested. |
There was a problem hiding this comment.
🟡 Changes recommended
An empty filename conversion can cause the default recursive directory scan to revisit the same directory indefinitely, exhausting the stack or file descriptors.
1 open issue
What changed in this PR
This PR centralizes native filename conversion so ZenLib can access non-ASCII paths under ASCII locales while preserving locale-based conversion for other Unix encodings and leaving Windows behavior unchanged.
Changes:
- Add filename conversion helpers with UTF-8 handling for macOS, Android, and ASCII Unix locales.
- Apply the helpers to file operations and metadata queries.
- Use consistent encoding conversion for directory enumeration and creation.
| File | Description |
|---|---|
Source/ZenLib/OS_Utils.h |
Declare native filename conversion helpers. |
Source/ZenLib/OS_Utils.cpp |
Implement platform- and locale-aware filename conversion. |
Source/ZenLib/File.cpp |
Apply conversion helpers to file paths. |
Source/ZenLib/Dir.cpp |
Apply conversion helpers to directory paths and returned filenames. |
Review effort: Balanced
Dir::GetAllFileNames() decoded names with From_UTF8(), which returns an empty string for names that aren't valid UTF-8, such as Latin-1 names. The entry then pointed to the directory itself, which was listed again and again until file descriptors ran out. Skip these entries instead. Assisted-by: Claude:claude-opus-5-5
File names were converted with wcstombs(), which depends on the LC_CTYPE locale. Programs that never call setlocale() and macOS apps launched from the Finder use the C locale, where non-ASCII characters can't be converted, or become Latin-1 bytes on macOS and FreeBSD. These files couldn't be opened [1]. On macOS and Android, file names are always UTF-8, so use UTF-8 there. On other Unix-like systems, keep the locale encoding but use UTF-8 when the locale is ASCII, as Python does for file names: its UTF-8 Mode is enabled automatically under the C locale [2]. Decode directory listings the same way. [1] sbraz/pymediainfo#121 [2] https://peps.python.org/pep-0540/ Assisted-by: Claude:claude-opus-5-5
FileName_To_Native() and FileName_From_Native() used UTF-8 under an ASCII locale, but the other conversions kept the locale encoding. Under the C locale, programs could open files with non-ASCII names but showed these names with '?', or with Latin-1 bytes on macOS and FreeBSD. On macOS and FreeBSD, programs decoding their command line arguments with From_Local() could no longer open files with UTF-8 names, which were encoded to UTF-8 twice. Use UTF-8 under an ASCII locale in Ztring::From_Local() and Ztring::To_Local() instead, so that all conversions agree. File names still use UTF-8 on macOS and Android whatever the locale, and there, FileName_From_Native() falls back to the locale encoding for strings which aren't valid UTF-8, such as arguments typed in a terminal using another encoding. Also skip directory entries whose decoded name doesn't give back the same bytes, such as overlong UTF-8 sequences decoded to "/", since they can't be opened with this name, and free the glob() results.
JeromeMartinez
added a commit
that referenced
this pull request
Nov 4, 2026
File names and text conversions: follow-up of #203
sbraz
added a commit
to sbraz/MediaInfo
that referenced
this pull request
Nov 4, 2026
Command line arguments were decoded with the locale encoding, so under the C locale, non-ASCII file names couldn't be opened. Decode them with FileName_From_Native(), which uses UTF-8 under an ASCII locale and on macOS and Android, as ZenLib does when opening files [1]. The output still uses the locale encoding. [1] MediaArea/ZenLib#203 Assisted-by: Claude:claude-opus-5-5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hello,
This fixes sbraz/pymediainfo#121.
Before this change, file names were converted with
wcstombs(), so under the C locale, which programs get when they never callsetlocale(), non-ASCII file names couldn't be opened. This PR uses UTF-8 for file names on macOS and Android, where they are always UTF-8, and on other Unix-like systems when the locale is ASCII. Other locales, such as ISO-8859-1 or EUC-JP, still usewcstombs(). Windows behavior is unchanged.Tested on Linux, macOS (ARM64 and x86-64) and Windows (x86-64 and x86) by building libmediainfo against this branch and opening
café.mkv,Åsnan é 测试.mkvand🎵.mkv:On Linux under en_US.ISO-8859-1, a
café.mkvwith a Latin-1 name opened before and after, butDir::GetAllFileNames()only listed it after this change.The macOS and Windows tests ran on GitHub Actions: https://github.com/sbraz/pymediainfo/actions/runs/41442332292
The
MediaInfoA_*functions and the CLI, which decodes its arguments withFrom_Local(), still use the locale. Because of the former, this PR alone doesn't fix MediaArea/MediaInfoLib#2464. @JeromeMartinez, if you agree with these ZenLib changes, would you like me to make similar PRs for the CLI and MediaInfoLib, usingFileName_From_Native()?It may also help with MediaArea/MediaInfo#134 and MediaArea/MediaInfo#349.