14
$\begingroup$

In many languages such as Java, strings and character arrays are distinct types. In Java one must use toCharArray() in order to use array semantics on strings.

In C, strings and character arrays are more or less the same. A C string is just a character array with a null terminator. Otherwise there is no internal distinction between them.

What are the advantages or disadvantages of strings being distinct types from character arrays?

$\endgroup$

12 Answers 12

8
$\begingroup$

Separate string methods or going via arrays

A disadvantage of a string type is that the methods available to manipulate arrays cannot just be used for strings. Instead, the language must either provide a separate set of functionality (which means more to document, learn, etc.) or force the programmer to split a string into an array whenever character manipulations are needed.

Array programming languages like APL, J, and K, make extensive use of array functions to manipulate strings, with great effect.

$\endgroup$
9
  • $\begingroup$ Ideally I'd like a string not to share methods with an array. What is "\U0001F1F8\u200d\U0001F1FA".map(println) supposed to do? Query the OS' default font to figure it out at runtime? $\endgroup$
    – Longinus
    Commented Jul 3, 2023 at 18:48
  • 3
    $\begingroup$ @Longinus I'd expect it to loop over Unicode codepoints. That's what Scala does. $\endgroup$
    – user
    Commented Jul 3, 2023 at 18:50
  • $\begingroup$ @user That would break any text editor GUI using that code. $\endgroup$
    – Longinus
    Commented Jul 3, 2023 at 18:51
  • $\begingroup$ @Longinus I don't see why it would. A text editor specifically working with Unicode would use a separate method that's aware of the fact that the "array" passed into it is really a string in UTF-8 or whatever, to be displayed in a specific font. $\endgroup$
    – user
    Commented Jul 3, 2023 at 18:53
  • 4
    $\begingroup$ @user We can, but when mapping over an Array<T> we know we deal with Ts, whereas when mapping over a String, nothing tells us what we're actually doing - code units? Code points? Graphemes? Grapheme clusters? Glyphs? Composed emojis? Emoji components? ZWJ sequence components? All of which fit one of the many definitions of the word "character" - so I don't think we should. Also whether one option is more "obvious" than another depends on whether you're dealing with text (which is graphical) or memory (which is not). $\endgroup$
    – Longinus
    Commented Jul 3, 2023 at 21:03
20
$\begingroup$

Different definitions of “character”

In Swift, the Character type isn’t a single Unicode codepoint (that’s UnicodeScalar), nor is it a single byte (that’s CChar). Instead, it represents an “extended grapheme cluster”, which includes combining diacritics. This makes some things more straightforward — é is one Unicode codepoint and is two, but they’re both a single character. However, it means that characters aren’t a fixed size, and for String to just be [Character] would be a massive waste of memory. Don’t get me wrong, it is a Collection<Character>, just not a random-access collection the way an array is.

$\endgroup$
10
$\begingroup$

An array must have a fixed stride, so that the memory address of the ith element equals the address of the start of the array plus i times the stride. If you really want an array of characters, so that each logical character (i.e. each Unicode symbol) occupies one slot in the array, then the array's stride would need to be 32 bits.

That is a waste of space compared to UTF-8 encoding, in which the most common characters take 8 or 16 bits each. On the other hand, if you encode strings as UTF-8 then it's no longer true that the ith character occurs at a predictable index, so what you have is not an array of characters. (Instead, it is an array of bytes, but that's the internal implementation ─ the API need not expose it to the user as such.)

$\endgroup$
12
  • 1
    $\begingroup$ Technically, you only need 21 bits. That doesn't exactly optimize well, though, as loading an odd number of bits is even worse than a regular unaligned load. $\endgroup$
    – Bbrk24
    Commented Jul 3, 2023 at 18:34
  • $\begingroup$ A "Unicode symbol" can be more than 32bits. National flags are 64bits, and both combining characters and ZWJ sequences allow for "symbols" to be arbitrarily long. $\endgroup$
    – Longinus
    Commented Jul 3, 2023 at 18:35
  • 6
    $\begingroup$ A Unicode code point must be between 0 and 0x10FFFF, hence "21 bits". It's true that a user-perceived "character" can contain multiple code points, though. $\endgroup$
    – dan04
    Commented Jul 3, 2023 at 18:43
  • $\begingroup$ Arrays cannot have a stride of 21 bits; it would be rounded up to 32. $\endgroup$
    – kaya3
    Commented Jul 3, 2023 at 18:55
  • $\begingroup$ @Longinus To my knowledge, the national flag symbols are not "Unicode symbols" ─ they are "individual symbols" because applications render them as such, but Unicode doesn't define them as far as I know. Really they are sequences of Unicode symbols which are interpreted as national flags by convention rather than by the Unicode standard. $\endgroup$
    – kaya3
    Commented Jul 3, 2023 at 19:00
10
$\begingroup$

Encapsulation

Treating strings as character arrays entails a belief that a character array is an adequate model for a string - not just good enough now, but good enough that you won't want to be able to swap it out later while keeping existing interfaces.

In reality, there are many operations on strings that don't make a lot of sense on character arrays, and vice-versa. It's questionable, for example, whether strings should provide random access at all. If you index into a string, should this produce a "character" that is an instance of a separate type? What should one be able to do with that object? Different languages have different answers to this question.

Another problem is that characters are completely different from graphemes. Consider this Python example:

>>> 'fiancée'[::-1]
'eécnaif'
>>> 'fiancée'[::-1]
'éecnaif'

The first string uses a precomposed character to represent the grapheme é with a single code point:

>>> import unicodedata
>>> unicodedata.name('fiancée'[-2])
'LATIN SMALL LETTER E WITH ACUTE'

The second "composes" the symbol using two separate characters:

>>> unicodedata.name('fiancée'[-2])
'COMBINING ACUTE ACCENT'
>>> unicodedata.name('fiancée'[-3])
'LATIN SMALL LETTER E'

Similarly with flags:

>>> '🇦🇺'[::-1]
'🇺🇦'
>>> '🇧🇬'[::-1]
'🇬🇧'

Your terminal (like mine) might not support the composition of flag emojis from their components (regional indicator symbols), so you might see something more like a fancy AU turning into a fancy UA, or BG into GB - which goes a long way towards explaining the result.

There are much stranger phenomena than that:

>>> calendars = "🗓️" + "🗓"
>>> len(calendars)
3

Depending on your environment, the two calendars might look the same or different - for me, they are different in my browser, but the same in my terminal. The length is 3 because the first one is composed of two characters: the calendar emoji followed by a "variation selector". There isn't a "precomposed version" of that; the variation selector is something like a control character that tells the text engine to draw a different glyph (image) for the previous character, while still treating it as the same character.

It is left as a (hopefully simple) exercise to explain why calendars[::-1] will produce a result that looks the same as calendars, regardless of the environment.

Don't even get me started on the bidirectional text layout algorithm and associated control codes. You understood correctly: the people responsible for standardizing Unicode are currently on revision 46 of a document about 18 thousand words long just to explain how to decide when to write text left to right and when to write it right to left.

But, sure, reversing a string is a toy problem that also turns out not to be very important in realistic applications. What about the real meat and potatoes of string manipulation?

  • Did you know that some languages have a concept of uppercase and lowercase, but don't have separate uppercase and lowercase versions of every letter? For example, the German letter ß is considered lowercase; when a word containing it is set in uppercase, the ß is supposed to be replaced with SS (two uppercase Ses)? So it's not actually possible to uppercase words in a locale-aware way by simply iterating over characters and replacing them separately. The length of the string might need to change. I'm not even sure that each character can be considered independently, for every language.

  • Did you know that many widely-used written languages are only ever intended to appear in a "cursive" form (i.e. there are strokes that visually continue between letters), so there are corresponding variant characters with different appearances (assigned separate code points - not distinguished with variation selectors - perhaps because they had to solve the problem before anyone came up with the idea of variation selectors)? Did you know that you might be expected to substitute the glyphs from those variant characters to render a string, while still treating the string as being composed of whatever was actually put into it? And, if you're the poor sap tasked with implementing text rendering for an operating system, you might be expected to ensure that the characters have their variant appearances when selected, but copying and pasting them gives the original "general" characters?

    >>> # Try highlighting, copying and pasting individual letters of the output.
    >>> # (You might find it tricky because of the LTR rendering.)
    >>> print('\u0628\u0628\u0628')
    ببب
    
  • Did you know that the order in which people expect strings to be sorted, is not necessarily based on Unicode code point order at all? And that it might be different for different languages (that could validly have the same two strings in them)? And that it might not be possible to apply naive lexicographical ordering at all (i.e. considering the first character first, breaking ties with the second character etc.)? And that this will get completely broken by combining characters anyway? And that you can't actually necessarily determine the language of text just by examining some characters (chat can validly be an English verb or a French noun)? And that there's a whole other essay on Unicode.org about how to do it properly? (About 30 thousand words, by the way.)

  • Did you know that for the East Asian languages that use ideographs and/or logographs (Chinese, Japanese and occasionally Korean - and even Vietnamese, which I didn't know before writing this answer), Unicode provides "unified" characters - such that a symbol might be considered "the same character" in different languages, but is supposed to be displayed differently depending on what language is being written? And that you can't rely on variation selectors to tell you which glyph to use? (And, again, for short excerpts, you can't reliably determine the language by examining the characters - after all, it often takes only a single character to express an entire word/concept?) And that the process of doing this was an absolute ton of work, and the results are still contentious in some places? And that sometimes "traditional Chinese" characters and their "simplified Chinese" counterparts use the same code point (as in the table at the previous link), and sometimes they're separate?

    • And that the entire topic can be really politically sensitive?
  • Did you know that the text in a string could be in more than one language? That it could be in no languages? That it could try to express an idea, despite not consisting of words in a language? That it could try to express an idea that isn't linguistic? That it could try to decorate meaningful text with meaningless symbols?

  • Did you know about the hack used to cram the Unicode code space into 16-bit encodings? (Or, relatedly, the reason why the allowed number of distinct code points isn't a power of 2?)

Text is really, really complicated. Attempting to model it as merely an array of characters is hopelessly naive. While, yes, it is conceptually composed of a sequence of characters (in the only models of it we have that are viable to work with on computers), it makes next to no sense to expose a plain array interface to that sequence (whether or not mutable). Arrays have the conceptual affordance of algorithms that just don't work on text.


This stands completely independent of the question of whether a byte is an adequate model for a character. Yes, it's true that UTF-8 uses a varying number of bytes to encode each character, such that a string implementation relying on a storage buffer in UTF-8 encoding could not be random access. But this is an absolutely trifling issue compared to ones created by naive ideas about what a "character" actually is, and about how text is written, and about what manipulations make sense for that data.

Sure, we have a standard that precisely, unambiguously and permanently maps abstract numbers to "characters". We have a virtual "code space" for those numbers, a commitment not to ever reassign them even if it would make things look much more elegant, and a reasonable assurance that we won't overflow that space for a very, very long time, even as we strain to come up with bizarre pseudo-textual things to fill that space with.

And, sure, there are many plausible ways to represent "a character", depending on system requirements:

  • A byte whose values correspond to the first 256 Unicode code points (obviously restricted to text that only uses those characters)

  • A byte whose values are mapped to Unicode code points in an instance-specific manner, according to encoding metadata stored along with the array (restricted to text that only uses characters from one of such single-byte encodings - which can only exist for languages that fit in such a character limit, and don't deal well with mixed-language text)

  • An unsigned 16-bit value representing a Unicode code point from the Basic Multilingual Plane (only represents those characters - so no emoji, very limited ideographic support, many world languages missing, no private use area)

  • The same, but using surrogate pairs to represent characters outside the BMP

  • An 32-bit value, which might as well be signed I suppose

  • A 21-bit value, bit-packed three to each 64-bit machine word (now that 64-bit architectures are the norm)

And, sure, there might not be any need to use a linked list, or other data structure, to store the underlying character data. Simply writing each character's corresponding number sequentially in memory - i.e., using an array - is perfectly adequate for the simple task of remembering which characters are in the string, in what order.

But... none of these things actually address the fundamental complexity of text. While it's easy to laugh at the previous generation of Western programmers and their naive faith placed in "code pages", the "array" part of "array of characters" is also problematic - perhaps much more so.

Exposing such an interface to the raw storage raises many questions. Simply equating the string type with an array-of-characters type - leaving all that string functionality outside the type itself - causes innumerable problems. Please don't do it in your language.

$\endgroup$
2
  • $\begingroup$ +1000 Excellent answer addressing the (huge) gap from arrays to human language text. And you didn't even get to normalization. :-) $\endgroup$
    – Pablo H
    Commented Aug 25, 2023 at 17:09
  • 1
    $\begingroup$ Although I didn't explicitly mention normalization, I touch on it in a few places. $\endgroup$ Commented Aug 25, 2023 at 17:42
8
$\begingroup$

Resizing with Bounds Checking and Constant-Time Size Lookup

The most common alternatives do not have all three. A fixed-length array is not resizable. And C-style arbitrary-length, null-terminated strings are the biggest bug attractor in history. Anyone reading this far down on this site knows why.

It’s also faster on modern vector CPUs to copy or scan a memory range whose length is known in advance than to search for the terminating character.

Short-String Optimization

Most strings are short (less than 32 bytes or so), and String implementations almost always optimize them so they can be statically-allocated, and not use the heap.

Standard libraries want this optimization to be transparent, and they do not want this optimization for a generic Vector, which would meet the first three requirements.

Variable-Width Encodings

Several other answers have mentioned this, but the specific problem with making a String an array of Char is that arbitrary indexing of a variable-width encoding is both unsafe and useless. It might point to a location that is a UTF-8 continuation byte or UTF-16 surrogate (or, previously, turn Shift-JIS into mojibake).

Changing arbitrary bytes could produce an invalid string. Even the “wide-character strings” that were promised to be fixed-width in the ’90s have turned out not to be. The C and C++ standards could say how wide strings “MUST” be a fixed-width encoding all they wanted. It didn’t get Microsoft to break the Windows API. They made their wide encoding variable-width anyway, and that was that.

Even changing one codepoint in the middle of a string to another valid codepoint could force the entire tail of the string to shift in memory, a linear-time operation that might even force the entire string to be reallocated and invalidate all current pointers to its data, so using array notation for replacing characters with other characters is actively misleading about the semantics.

It would be possible to use arrays of UCS-4, but that wastes too much memory when storing English, and often the user will want to iterate over graphemes instead of codepoints anyway. Some implementations add “breadcrumbs” (the offsets of the characters at certain positions within the string) to speed up linear-time operations, which also requires a special type.

Validation

Related to the above, a blob of character data is not necessarily a valid Unicode string. And in some languages, it’s common to use strings as blobs of arbitrary bytes of data. But if a String class validates its input, the type system can guarantee that any String object always contains a valid string. It could also perform normalization, if desired. Some algorithms run faster on canonically-decomposed Unicode.

Arrays are a Leaky Abstraction

If you look at the history of the C++ std::string, you see it starting out trying to offer implementers some flexibility. That didn’t work, because the API exposed a pointer to the bytes of a C-style, null-terminated array of char.

So, it became official: std::string must be implemented that way. It could not, for example, be a rope, because it must be laid out in memory as a single contiguous array with random access. It could not be a copy-on-write class, because writing through a pointer within the array doesn’t invalidate pointers to the data. It could not save a byte of memory by removing the redundant terminating null when it already stored the length, because you had to be able to pass the address of the first char to a function that expected a C-style char*.

A class that did a better job of hiding its implementation details would have much more freedom to implement something stringy in a way that suits the application better.

$\endgroup$
2
  • $\begingroup$ It didn’t get Microsoft to break the Windows API. They made their wide encoding variable-width anyway, and that was that. I'm no MS fan but this is not the sequence of events as I recall it. It actually took MS an annoyingly long time to switch Windows from UCS-2 to UTF-16. It looks like you're suggesting they forced that change on everyone else. If I'm wrong please point me somewhere I can read up and learn. $\endgroup$ Commented Aug 2, 2023 at 5:28
  • 1
    $\begingroup$ @hippietrail All I’m saying is that their switching wchar_t from UCS-2 to UTF-16 ignored the part of the Standard that said wchar_t must be a fixed-width encoding. The Windows API was defined back when UCS-2 was, and MS refused to make breaking changes, but this meant that wchar_t became a variable-width encoding on Windows. This made all code that uses wchar_t non-portable, and not all compilers support std::ctype<char32_t>. $\endgroup$
    – Davislor
    Commented Aug 3, 2023 at 17:20
7
$\begingroup$

Handling Encoding

Arrays of raw data don't, generally speaking, have an encoding. Strings can be in UTF-8 or any number of other formats. In Ruby, a String object is an array of bytes together with its Encoding. If you're handling data in different formats (such as data coming in from the network and then entering your main program's file system or database), it can be invaluable having the language keep track of this for you.

$\endgroup$
1
  • 2
    $\begingroup$ In Dyalog APL, there are only character arrays, but they are stored with their encoding, namely 1-, 2-, or 4-byte unsigned integer Unicode code points. $\endgroup$
    – Adám
    Commented May 16, 2023 at 18:17
2
$\begingroup$

Performance

In Java, a char array can be expensive. If what we need is a pure ASCII string, then using a char array will waste most of the memory. We may want to use a byte array, but it's not the same type as char arrays.

Encodings

In Java, strings are stored in UTF-16 encoding, but it is explicitly known by all users. If we use a char array, it's less clear which encoding it is using.

$\endgroup$
2
$\begingroup$

If Java had included a family of immutable-array and immutable-array-segment types, each with a constructor that could accept a single array (mutable or not) and construct an immutable array (copying the data if the original was mutable), a constructor that could accept an array of arrays (mutable or not) and produce a composite immutable array, and a factory method to produce an immutable array segment from an immutable array, then it would make sense to use immutable arrays and array segments of char to represent strings. Having an immutable string type is extremely useful in a language that's based around a tracing garbage collector, and if there are no other immutable array types, making strings immutable would make it necessary for them to be distinct from other array types.

Note that in scenarios where an immutable array of char, short, or int would often not contain any values outside the range of a signed or unsigned byte, it may be worthwhile to store instances meeting that criterion using one byte per element. Code using elements of such arrays would be more complicated, but the reduction in cache footprint would often overcome that disadvantage.

$\endgroup$
1
$\begingroup$

In C, strings and character arrays are more or less the same. A C string is just a character array with a null terminator. Otherwise there is no internal distinction between them.

The only distinction between a string and a character array is whether the data is handled as a scalar or not.

C basically handles all strings usually as variable-length arrays, and C has so little facility for working with VLAs that its treatment of strings (whilst very simple to grasp in principle) tends to be regarded as hard work compared to more modern languages.

What are the advantages or disadvantages of strings being distinct types from character arrays?

Alternatively put, you are asking what is the advantage of arrays being treated as scalars?

The advantage of treating arrays as scalars is that you don't have to treat everything constantly as an array of bits! The bit being the only fundamental scalar in computing, with everything else being an array of bits.

The main problem with strings is that they tend to be variable length. Other bit arrays which are treated as scalars - like integers - were reducible to fixed-width forms at a relatively early stage of electronic computing, so that all hardware treats them as scalars even. Never so with strings, not even today.

At the time of C's design, and still today, there isn't a well worked-out scheme for how type systems cope with arrays in a fully general, systematic, and happy manner.

Even in APL, where arrays are king (and already were in 1972), the problem is solved largely by dispensing with types and a variety of other extravagances which make it completely unsuitable to the same purposes as C.

Nevertheless, most languages more modern than C, have implemented some kind of special treatment for string types and for operators which manipulate strings as if they are scalars, because there aren't any disadvantages to being able to treat strings as scalars, none whatsoever.

$\endgroup$
0
$\begingroup$

Java strings are immutable, making possible to share them. But this is very bad for security, as data like passwords may stay in the memory for very long with no way to clear them. Due that, character arrays must be used for all sensitive data that are discarded (overwritten) asap after using them. See here for more on this topic.

Immutable strings also mean that a new string is created each time you modify it. A heavily multi-threaded applications avoided touching the heap as much as possible because the new object allocation scales very badly with the number of threads: the memory pool is shared and all access there must be synchronized. I had applications where adding more threads after the first 20 or about actually brought the performance down, even if it was on the 64 core high performance computer.

$\endgroup$
2
  • 1
    $\begingroup$ "Due that, character arrays must be used for all sensitive data that are discarded asap after using them." Explicitly cleared, not necessarily "discarded" (i.e. abandoned to the garbage collector, which will only "deallocate" memory but not actually make it unreadable). On the other hand, this is only important in a context where an attacker can already read process memory. Also, last I checked the String type is still immutable (it would be a huge, breaking change), it's just that there are also other textual types that aren't. $\endgroup$ Commented Jul 10, 2023 at 14:07
  • $\begingroup$ It is commonly seen as a security flaw to use strings for passwords in Java. Please read the provided source for details. $\endgroup$ Commented Jul 10, 2023 at 16:17
0
$\begingroup$

While there may be some common features of strings and arrays, there are also some differences.

Rather than making strings a subtype of arrays, it would make more sense for them both to be a subtype of an abstract type (e.g. "indexed sequence") that represents that commonality. This would allow you to use the same syntax and methods for accessing elements and slices.

But they can then each have unique methods for aspects that don't make sense for the other.

Forcing strings to be a type of array makes it difficult to enforce these disjoint aspects.

$\endgroup$
0
$\begingroup$

UTF-8 Packing

So, what C calls a "character" isn't really a character in the typical Unicode sense. It is an 8-bit integer: a byte. The character 'c' takes one byte to represent, but the character 'ć' is not representable - it is too large to fit in C's char type. It is instead encoded as multiple bytes:

  • "c" = [0b01100011, 0x0]
  • "ć" = [0b11000100, 0b10000111, 0b0]

So a string in C is perhaps better thought of as an array of bytes.

In the original design of UTF-8, characters may be represented with any number of bytes: the number of 1s before the first 0 read in the first byte indicates the number of subsequent bytes to read. I believe subsequent iterations restricted the maximum size of a character to four bytes.

First code point Last code point Byte 1 Byte 2 Byte 3 Byte 4
U+0000 U+007F 0xxxxxxx
U+0080 U+07FF 110xxxxx 10xxxxxx
U+0800 U+FFFF 1110xxxx 10xxxxxx 10xxxxxx
U+10000 U+1FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

(chart adapted from Wikipedia)

This design allows for characters of different sizes to be placed contiguously in a file, and is conveniently backwards-compatible with ASCII (ASCII characters are 7-bit: all of the form 0xxxxxxx). It does, however, mean that arbitrarily splitting a UTF-8 encoded sequence at a byte boundary can produce invalid UTF-8. This, and subsequently that individual bytes do not necessarily correspond to a printable notion of a character, is why a representation of (UTF-8 encoded) strings as an array of bytes is undesirable.

Naively implementing strings as an array of characters, however, has considerable overhead. Arrays require the types of their elements to have a known size - as UTF-8 characters can vary from one to four bytes, typical modern representations of char types must compromise on this and represent all Unicode characters with a consistent size of four bytes. This is incredibly inefficient for characters in the ASCII range - taking up four times as much space! Implementing strings as arrays of chars only compounds this inefficiency.

Further, while all Unicode characters can be represented by four bytes - not all sequences of four bytes are valid characters (in any encoding). An array-based implementation may allow direct access to the individual characters as bytes: which gets rid of any invariants you may like a string type to hold, like being valid UTF-8. A distinct char type that is not convertable to from bytes can uphold them, but is yet another design consideration to watch out for.

This is not without tradeoffs, of course - but they're mostly very much worth it. Modification of a char is no longer an O(1) operation, as it has to shift every subsequent byte to make room if the char takes more bytes to represent. Any sort of indexing is also necessarily O(n), even if you store the length of the array. All methods defined on arrays no longer carry over to strings - though many would say this is a benefit, despite some code duplication, as strings are semantically distinct from character arrays.

$\endgroup$
4
  • $\begingroup$ Surely nobody is using arrays of 4-byte UTF-8, but UTF-32? There’s no packing involved there. The last two paragraphs I don’t understand: the answer to that point has wanted fixed-size characters to support indexing, but now individual bytes are indexed. Maybe there’s a transition missing. The last paragraph on tradeoffs maybe is missing a negation somewhere, though I’m not sure where (and it’s not clear what the comparison point where insertion was O(1) is). $\endgroup$
    – Michael Homer
    Commented May 30 at 19:33
  • $\begingroup$ A char array in Rust (and any other language where chars are four bytes) is an array of 4-byte UTF-8 characters. "Insertation (or removal)" is wrong and should read "modification", thanks for the catch. The point in the last two paragraphs is that representation as a char array does support indexing - but the individual chars themselves can be incorrectly modified, which is undesirable. $\endgroup$
    – apropos
    Commented May 30 at 20:57
  • $\begingroup$ That is certainly not true; every single implementation I can think of with 4-byte chars uses UTF-32, not UTF-8, for all the obvious reasons that make processing much easier. I can’t imagine why anyone would use UTF-8 there, although maybe Rust does (but the documentation implies but doesn’t outright state otherwise). I think the overarching answer here needs some structuring to set out when these considerations apply, sourcing ideally. $\endgroup$
    – Michael Homer
    Commented May 30 at 21:13
  • $\begingroup$ Ah, I see. I suppose "4-byte UTF-8 characters" is a misnomer to begin with since it's variable-length... I've updated my answer to be more pedantic on those fronts and elaborated on what C's actually doing. Hopefully that's clearer. $\endgroup$
    – apropos
    Commented May 31 at 2:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .