What Unicode character categories should be allowed in identifiers?

Question

When implementing Unicode identifiers, I'm never sure what characters should be allowed. For example, this is the list of categories:

• Control: C
• Letters: L
• Marks: M
• Numbers: N
• Punctuation: P
• Symbols: S
• Whitespace: Z

Each of these have different subcategories. What's a sensible policy for what Unicode characters to allow in identifiers? Letters and numbers seem to map obviously to the typical allowed Latin-1 characters, but what about "marks"? What policies do existing languages have for Unicode identifiers, and what should be taken into consideration when deciding whether something should be allowed?

Answer 1

https://unicode.org/reports/tr31/ is the Unicode standard for which characters should be available for use in identifiers. Any identifier system for a new language should make that its basis for identifiers.

If you're making a language where every program has to be cleanly designed for maximum reusability, the next COBOL, go ahead and use ASCII only identifiers. If you're making the new Perl or C or C++, worrying about whether that's an a or an а, which many code editors will tell you (since both are legal in virtually all commonly used languages), seems the least of your troubles in confusability. Non-ASCII identifiers are trivially found by code checkers and are more friendly to programmers whose native tongues aren't supported by ASCII. Why should a code by a Chinese student in a Chinese classroom have to use ASCII identifiers for your convenience?

Answer 2

If your goal is to let users name identifiers using words from any natural language, you pretty much need to permit anything from "L" (the letters) and "M" (the modifiers -- things like acute accents and vowel marks). From the "N" (numbers) category, "Nd" (decimal numbers) and "Nl" (letterlike numbers) should be fine, but some of the characters in "No" (other numbers) have an ambiguity problem: is a superscript digit part of an identifier, or does it represent exponentiation?

The big problems you'll run into with Unicode are normalization forms, homoglyphs, and duplicate characters. These are situations where visually-identical characters have different byte values, and can't be solved by restricting character categories.

Answer 3

My suggestion:

Initial characters:
Ll Lower case letter
Lo Other letter
Lt Title case letter
Lu Upper case letter
Ni Letter number
_ Low line

Additional continuation characters:
Lm Modifier letter
Mn Non-spacing mark
N Number
Pc Connector punctuation
Mc Spacing combining mark

JavaScript allows a little more than this, including beginning with an Lm (Modifier letter) but that doesn't really make sense as such letters are like combining accents. However, JavaScript only allows decimal digits, not other digits. I don't see any reason for that restriction, as surely, if A2 and Ah and Aʰ are allowed, A² should be too.

Raku allows all L (letters) and (for continuation) N (numbers).

Answer 4

Be inclusive

I'm going to suggest allowing the widest range of characters you can, and let the programmer make good choices for themselves. I'm also going to suggest allowing an even wider range than you can, later on.

Basic identifiers

I've previously allowed identifiers to start with letters (uppercase, lowercase, titlecase, modifier, other), underscores, and all "Symbol, other"s that weren't operator characters (discussed below). Subsequent characters could also be numbers (decimal digits, letter, other), non-spacing marks, spacing combining marks, and enclosing marks, the same other symbols, and the apostrophe.

This is a very broad group and permits text in natural languages as well as mathematical symbols, emoji characters, regional indicators (flags), and other multi-codepoint grapheme clusters. These are a generalisation of specific identifiers that different people wanted available, leaning towards inclusiveness where it wasn't unreasonable.

It's definitely possible to make unwise identifier choices within this, but ultimately that's true even with a narrow range of characters. Likely most programs will use only a tiny part of this range, but different users are interested in different parts of that space. In particular, the emoji characters while seeming "unserious" to some were pretty popular as mnemonic tools for exploring novices, and variable names in CJK characters too. A collaborative coding style guide could well forbid them, but the language doesn't have to make that choice for them.

Operator identifiers

In languages with user-defined operators, these are generally a distinct kind of identifier character — and the suitable characters don't really coincide with the Unicode categories, in both directions. Instead, a better granularity is to pick out specific Unicode blocks (256-codepoint ranges), along with some individual characters from ASCII and Latin-1. These operator codepoints should be excluded from being part of regular identifiers (unless all identifiers are usable as operators, as in some languages).

We used the remaining ASCII keyboard symbols, the symbols from Latin-1, and the entire blocks of (supplemental) mathematical operators and symbols, miscellaneous symbols, (supplemental) arrows, supplemental technical, currency symbols, and geometric shapes. All of these were excluded from ordinary identifiers explicitly, because many of them are in the catch-all Symbol, other category that also includes lots of "noun" codepoints.

Normalise

At the very least, normalise identifiers to either fully-composed (NFC) or fully-decomposed (NFD) form for comparison. 한국 and 한국 should be the same identifier and so should kākāpō and kākāpō, but they're not byte-for-byte identical. It's possibly good to normalise further cases away as well, but they mostly head into "malicious" rather than accidental territory and may not be worth the costs. I have not bothered for program source code, but do for network and file tasks.

Allow an escape hatch that permits any character

This isn't something you'll want to use all the time or encourage, but there are times when languages need to interact with other languages or systems with different choices, perhaps directly via a foreign function interface, or as a reification of an external namespace like a database table, or directly wired to a visual user interface. It's helpful to have a way to represent these identifiers on those occasions even if they clash with a language keyword, start with a number, have spaces, contain a character you use for something else, or one you wouldn't use at all.

It's probably good to have this escape hatch be a bit out of the way, but accessible for the cases that really do need it. F# uses double backticks for this purpose: let ``this, here, is a single identifier in F#! 😹`` = true. Something else awkward-but-available might work for your purposes.

Answer 5

When programming mathematical formulas, non-Roman characters are handy.

// From https://www.movable-type.co.uk/scripts/latlong.html
const distance = function(latitude, longitude) {
    const φ1 = latitude * Math.PI/180 // φ, λ in radians
    const φ2 = home.latitude * Math.PI/180
    const Δφ = (latitude - home.latitude) * Math.PI/180
    const Δλ = (longitude - home.longitude) * Math.PI/180
    const a = ( Math.sin(Δφ/2) * Math.sin(Δφ/2) )
            + ( Math.cos(φ1) * Math.cos(φ2) * Math.sin(Δλ/2) * Math.sin(Δλ/2) )
    return 6371e3 * 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1-a)) // in metres
}

Answer 6

From a usability-based perspective, one might decide that identifiers should match [a-zA-Z_][a-zA-Z0-9_]*, i.e. a Latin alphabet letter or underscore, followed by 0 or more Latin alphabet letters, digits or underscores. Other characters should not be permitted in identifiers. (There are some languages which allow $ or - in names too, which is probably fine depending on what the language is for.)

My reasoning for this is that I think source code should be editable by other programmers. While these characters aren't common to every keyboard layout, they are generally already required by other things in the language, particularly the keywords and the names used in the standard library. If e.g. Greek letters are allowed in identifiers, then those identifiers can be typed by programmers with a Greek keyboard layout but not an English keyboard layout. On the other hand, any keyboard layout used by Greek programmers must already be able to type if, while, System, out, println and so on, so identifiers using only the Latin alphabet can be typed by both English and Greek programmers. Likewise, the language already requires Arabic numerals for numeric literals, so these too can be assumed to be on all programmers' keyboard layouts.

To be clear, this reasoning only applies assuming that the language already privileges the Latin alphabet in its choice of keywords and the names in its standard library. I'm not advocating against other languages like قلب which are designed not to use the Latin alphabet at all, and I think it is good that such languages exist.

Answer 7

Unless you're writing for a specific niche that would benefit from non-English identifiers, don't allow non-ASCII characters in identifiers. Non-ASCII identifiers cause many problems for several reasons: not everyone will be able to tell what some of the characters are, let alone type them; not everyone can tell visually whether two identifiers are the same, and the conditions under which two identifiers are the same are very tricky.

Non-ASCII identifiers require display and input support. People in different locales will have different font sets. Few systems have really thorough coverage of Unicode, especially with a font that's comfortable for writing code. And the situation with screen readers is even worse: they're optimized for reading human-readable text, not arbitrary sequences of characters. Input is also difficult: unless everyone who maintains the program is from the same culture, forget about inputting identifers with anything other than copy-paste or autocompletion (if you manage to type the first character somehow).

Unicode includes a lot of characters that have very similar appearance, if not identical in typical fonts (homoglyphs). Can you distinguish between A, Α and А? (Respectively a Latin letter, a Greek letter and a Cyrillic letter.) This has been a well-known security concern for a long time with domain names, and awareness has recently spread to the programming world since the Trojan Source publication. Even where the characters are visually distinct and present in fonts, readers who are unfamiliar with the writing system will have trouble pattern-matching on the visual appearance.

Two Unicode strings can be “the same” even if they aren't made from the same sequences of code points. Unicode even defines two notions of equivalence: canonical equivalence and compatibility. For example, the strings Å (U+0065 U+030A), Å (U+00C5) and Å (U+212B) are equivalent — they are, respectively, A with a ring “accent”, a legacy code point which has the two pre-combined, and the angstrom symbol which is semantically distinct but has the same visual appearance. Some software will automatically canonicalize such text, so your language had better follow suit. But the rules are complicated and they change over time. So if you upgrade your Unicode support library, programs might not compile anymore!

Unicode support is ok only if you avoid these pitfalls.

• To mitigate the purely human side of the problems, Unicode support should be limited to application domains where the authors of a given program are likely to share the same culture. For example, a programming environment for beginners should allow identifiers written in the users' script (which would generally be the one used in the user interface) in addition to ASCII; in that case there would be no expectation of sharing a program across cultures. A programming environment for scientists should allow mathematical notation.
• To mitigate accessibility problems, if the language has broad Unicode support, a code base (or at least a file) should explicitly declare which Unicode segments the program uses. This gives readers a chance to ensure that their system has the required fonts and input methods.
• To mitigate both accessibility and security, as well as keep the cognitive load manageable for humans, the language should impose the same kind of rules as internationalized domain names: typically, that any given identifier should only use characters from a single script (plus some “neutral” characters like _ and digits), or only from a set of punctuation signs. And avoid any parts of Unicode where you aren't sure that canonicalization issues may lurk. Note that right-to-left support is especially tricky since it essentially allows hiding text.