Repository navigation
File names and text conversions: follow-up of #203 - #204
Merged
JeromeMartinez merged 8 commits intoNov 4, 2026
Merged
Conversation
From_Local() gave an empty string for such bytes, so names in another encoding could no more be opened or listed with the byte-transparent C locale of macOS and FreeBSD: they are now kept as U+DC80 to U+DCFF, and To_Local() and FileName_To_Native() give them back, without opening a file with the UTF-8 name of the same characters. Without Unicode, strings are the bytes of the names, so there is no specific encoding of file names.
A name ending with an EUC lead byte is decoded with the end of the string as second byte, and To_Local() stopped at this 0 byte, so the name looked valid and each file of this directory was seen as the directory itself.
mbsrtowcs() of Android 7 sets the source pointer to NULL at the end of the string also when there is no destination, so the second call, which does the conversion, read a NULL pointer.
Text is UTF-8 in this build, the one of MediaInfo.js: only the low byte of each UTF-16 value was kept, From_Unicode() depended on the locale and gave an empty string with an ASCII locale, From_CC3() and From_CC4() added bytes which are not UTF-8 (e.g. the 0xA9 of MP4 tags), and From_UTF8() accepted everything, so parsers trying another encoding when it fails never did it. To_UTF8() now always gives a text which can be decoded, as documents are built with it and decoded as a whole. UTF-16 surrogate pairs are also one character where wchar_t is 32-bit in all builds, characters outside of the BMP were lost.
The first byte of 2-byte characters lost a bit, so U+0400 to U+07FF were wrong, bytes which are not UTF-8 were modified, and U+D801 to U+FFFF were wrong with a 16-bit wchar_t; overlong sequences, e.g. of "/", UTF-16 surrogates and values after U+10FFFF are no longer decoded. The comparison of URL_Encoded_Encode() which is always true with an unsigned char is also removed, the result is the same.
The copy constructors of int128 and uint128 are user-provided, so their implicit copy assignments are deprecated: they are declared as defaulted, they stay trivial, without ABI change. Parameters of the functions which are not supported on this platform are not used.
make check runs test/filename.sh, with ASCII, UTF-8, Latin-1 and EUC-JP locales if available, with and without Unicode: conversions, functions of File and Dir, directory listing with names which are not valid UTF-8, and text in other encodings. The test files are in the source archive of the release.
Linux, macOS, Alpine, FreeBSD and Windows, with and without Unicode, sanitizers, and Android emulators of the API levels with different code, as these systems have their own code for file names; the Unix jobs also check the library with -Wextra.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up of #203.
To_Local()andFileName_To_Native()give them back, without opening a file with the UTF-8 name of the same characters. They can now also be opened and listed on Linux with an ASCII locale.To_Local()stopped at this 0 byte, so the name looked valid.From_Local()on Android 7:mbsrtowcs()sets the source pointer to NULL also when there is no destination.From_Unicode()depended on the locale and gave an empty string with an ASCII locale, it now always gives UTF-8, as on Windows.From_UTF8()accepted everything, so parsers trying another encoding when it fails never did it, it now rejects invalid text.To_UTF8()now always gives a text which can be decoded, as documents are built with it and decoded as a whole. Without Unicode, a file name which is not valid UTF-8 has U+FFFD in such documents.From_CC3()andFrom_CC4()added bytes which are not UTF-8, e.g. the 0xA9 of MP4 tags.wchar_tis 32-bit, in all builds: characters outside of the BMP were lost.URL_Encoded_Decode():%D0%B0gave0.wchar_t./, UTF-16 surrogates and values after U+10FFFF are no longer decoded.-Wextrawarnings are fixed, the Unix jobs of the new workflow check the library with it.make checkand new workflow for the file names and the conversions, with and without Unicode, on the systems which have their own code for file names, also on Android emulators. The test files are added to the source archive of the release.