Repository navigation
Texts and file names with all the locales, same strings with and without Unicode - #211
Open
JeromeMartinez wants to merge 2 commits into
Open
JeromeMartinez wants to merge 2 commits into
JeromeMartinez wants to merge 2 commits into
Conversation
From_Local() kept the bytes which are not valid in the encoding of the locale only with an ASCII locale. With a UTF-8 locale, the most common one, a name which is not valid UTF-8 (e.g. Latin-1 names of old archives or of volumes using another encoding) gave an empty string, so it could not be opened or listed, and To_Local() gave "?" for the U+DC80 to U+DCFF of the names given by Python, which keeps the bytes this way with all the locales (os.fsdecode() with "surrogateescape"). The bytes are now kept with all the locales, as Python does: with the UTF-8 decoder of ZenLib for ASCII and UTF-8 locales, and character per character with the C library for the other ones, with a conversion state for each call. A character which does not give the same bytes back (e.g. with two encodings) is also kept as bytes, so a name always gives the same bytes back. ASCII text is converted without the C library. To_UTF8() gives U+FFFD for these bytes and for the other values which are not characters, as the documents built with it must be valid UTF-8, and it no longer reads after the end of a string which ends with the first part of a UTF-16 surrogate pair. On macOS, FreeBSD and other systems, wchar_t is not Unicode with some locales (e.g. EUC-JP, GB18030, Shift_JIS, Big5, ISO-8859-15, KOI8-R), so the text converted by the C library was wrong, e.g. the title of a file shown in a terminal using such a locale, or a name typed in it and converted to the UTF-8 names of macOS. iconv of the system is used instead on these systems (also elsewhere if ZENLIB_ICONV is defined, e.g. for tests): it is found at run time in the program (e.g. in the C library of FreeBSD) or in libiconv of macOS, only for a string which is not ASCII with a locale which is neither ASCII nor UTF-8, and the C library is used if it is not available. Known names of the encodings of the locales are given to iconv if it doesn't know the ones of the system (e.g. "eucJP" with macOS 14 and more). The file names of macOS without Unicode use it too, so ZenLib is no longer linked with libiconv. APFS and HFS+ accept only UTF-8 names, but network volumes (e.g. NFS, AFS, SMB) keep the bytes of the names created by other systems with another encoding: with a locale which is neither ASCII nor UTF-8, such a name was decoded with the encoding of the locale and given back as UTF-8, which is another name, and a name typed in a terminal using this locale may exist only with the bytes of this encoding, for all the path or a part of it. The UTF-8 name is used if it exists, else for each part of the path the first one which exists among the UTF-8 one, the bytes of the encoding of the locale and the UTF-8 name of these bytes decoded with this encoding. A new file in an existing directory gets the UTF-8 name, the names of the file system are decoded as strict UTF-8 with the bytes which are not valid kept, and the directory listing checks that the names give the same bytes back with the directory. Also on other systems if ZENLIB_FILENAME_UTF8 is defined (e.g. for tests).
Without Unicode, From_Local() kept the bytes as they are and To_Local() gave them back, so the strings were in the encoding of the locale (the code page on Windows) and the file names were the bytes of the file system, while the other texts were UTF-8 (e.g. From_UTF8(), To_UTF8()): a program compiled without Unicode got other strings than the same program with Unicode, and on Windows the names with characters which are not in the code page could not be opened. The strings are now UTF-8 text in all cases without Unicode: From_Local() decodes the encoding of the locale (the code page on Windows) and To_Local() encodes it, the bytes which are not valid in it kept as with Unicode. To_Unicode() gives these bytes as U+DC80 to U+DCFF, as with Unicode, so they are 3 bytes in the UTF-8 text, the encoding of a UTF-16 surrogate which is not in a pair (as WTF-8, also from From_Unicode()): they never form a valid UTF-8 character with the next bytes, and To_UTF8() replaces them by U+FFFD, as with Unicode. To_Unicode(), To_UTF8() and To_Local() give the same strings with and without Unicode, and MakeLowerCase() and MakeUpperCase() change the characters as with Unicode, not the bytes of the UTF-8 text (e.g. with a Latin-1 locale). From_UTF8() accepts the same sequences with and without Unicode, also when they are not valid UTF-8: a UTF-16 surrogate is kept (e.g. a byte kept, in a std::string made from a Ztring) and a pair is one character for To_UTF8(), an overlong sequence is the character, and a value after U+10FFFF is U+FFFD (with Unicode on Windows, the overlong sequences of 4 bytes and these values gave other values). The file names have the same conversions as with Unicode (also the names of network volumes on macOS). On Windows, File, Dir, the temporary file names, Shell_Execute() and the folder selection use the wide functions in both builds. ZtringListListF reads its files as UTF-8 first, as it writes them, as with Unicode. A program which puts the bytes of the encoding of the locale in a Ztring without From_Local() (e.g. the arguments of main() or a name from the narrow functions of the system), as it could with the previous versions, still opens these names. On Windows, a string which is not UTF-8 text (bytes which are neither valid UTF-8 nor the UTF-16 surrogates of From_Unicode(), so never a string made by ZenLib) is read as the bytes of the code page: To_Unicode() and To_UTF8() decode it as the narrow functions of Windows do, and To_Local() gives the same bytes back. On the other systems, the bytes which are not valid are kept as before. When such bytes are also valid UTF-8, the functions which read names (File::Open() for reading, File::Exists(), Dir::Exists(), Dir::GetAllFileNames()) use the name in this encoding if only it exists, on Windows with the wide functions, on the other systems with the candidates of FileName_To_Native() (as for the network volumes of macOS), and Dir::GetAllFileNames() converts the name of the directory to UTF-8 text first, so the listed names are UTF-8 text. The byte after a lead byte of a double-byte code page (932, 936, 949, 950) is skipped when looking for a path separator only in the bytes of the code page: "\" and "/" are never in a UTF-8 character, and the separator after a directory name in Chinese, Japanese or Korean was not found in UTF-8 text. OS_Utils.h tells that the names given by ZenLib are UTF-8 text without Unicode, to be converted with FileName_To_Native() or To_Local() for the narrow functions of the system or to be shown. The functions shared by the files of ZenLib are hidden where the compiler supports it, so the exported symbols don't change.
This was referenced Nov 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up of #204 and #206, in 2 commits.
Texts and file names with all the locales
From_Local()kept the bytes which are not valid in the encoding of the locale only with an ASCII locale: with a UTF-8 locale, the most common one, a name which is not valid UTF-8 (e.g. Latin-1 names of old archives or of volumes using another encoding) could not be opened or listed (question of sbraz in MediaArea/MediaInfo#1331), andTo_Local()gave "?" for the U+DC80 to U+DCFF of the names given by Python.To_UTF8()gives U+FFFD for these bytes and for the other values which are not characters, as the documents built with it must be valid UTF-8, and no longer reads after the end of a string which ends with the first part of a UTF-16 surrogate pair.wchar_tis not Unicode with some locales (macOS, FreeBSD and other systems, e.g. EUC-JP, GB18030, Shift_JIS, Big5, ISO-8859-15, KOI8-R), iconv of the system is used instead of the C library. It is found at run time (dlopen()), only for a string which is not ASCII with a locale which is neither ASCII nor UTF-8, so ZenLib is no longer linked with libiconv on macOS. The characters of GB18030 after U+FFFF are converted without iconv, which does not know them on macOS.ZENLIB_ICONVandZENLIB_FILENAME_UTF8enable this code on other systems.Same strings with and without Unicode
Without Unicode, the strings from
From_Local()and from the file system were in the encoding of the locale (the code page on Windows), while the other texts were UTF-8 (e.g.From_UTF8(),To_UTF8()): a program built without Unicode got other strings than the same program with Unicode, and on Windows the names with characters which are not in the code page could not be opened. The build with Unicode and its ABI don't change.From_Local()decodes the encoding of the locale andTo_Local()encodes it, the bytes which are not valid in it kept as with Unicode. These bytes are 3 bytes in the UTF-8 text, the encoding of a UTF-16 surrogate which is not in a pair (as WTF-8, also fromFrom_Unicode()), so they never form a valid character with the next bytes.To_Unicode(),To_UTF8()andTo_Local()give the same strings with and without Unicode, andMakeLowerCase()andMakeUpperCase()change the characters, not the bytes of the UTF-8 text.From_UTF8()accepts the same sequences with and without Unicode, also when they are not valid UTF-8: a UTF-16 surrogate is kept and a pair is one character forTo_UTF8(), an overlong sequence is the character, and a value after U+10FFFF is U+FFFD (with Unicode on Windows, the overlong sequences of 4 bytes and these values gave other values).File,Dir, the temporary file names,Shell_Execute()and the folder selection use the wide functions in both builds.ZtringListListFreads its files as UTF-8 first, as it writes them.ZtringwithoutFrom_Local()(e.g. the arguments ofmain()), as it could before, still opens these names: on Windows, a string which is not UTF-8 text (never a string made by ZenLib) is read as the bytes of the code page, as the narrow functions of Windows do; on the other systems, the bytes which are not valid UTF-8 are kept. When such bytes are also valid UTF-8, the functions which read names use the name in this encoding if only it exists.FileName_To_Native()orTo_Local()for the narrow functions of the system or to be shown (seeOS_Utils.h). The functions shared by the files of ZenLib are hidden where the compiler supports it, so the exported symbols don't change.To be merged before MediaArea/MediaInfoLib#2822 and MediaArea/MediaInfo#1338, which need these strings without Unicode.