10
$\begingroup$

As a followup to PLDI's first question, about horizontal whitespace, what vertical whitespace (newlines) should be supported? I know of at least the following:

  • Line feed (\n)
  • CRLF (\r\n)
  • Carriage return only (\r)
  • Form feed (\f, U+000C)
  • Next line (U+0085)
  • Vertical tab (\v, U+000B)

For many languages newlines are just treated like other whitespace, while for others they have special syntactical meaning (typically those which don't require line-ending characters like ;). What non-horizontal-whitespace characters should be considered whitespace, and what are the pros and cons of allowing each as newlines when this has a special syntactic meaning (or for counting line numbers)?

$\endgroup$
2
  • $\begingroup$ One thing which may be worth taking into consideration is IDEs and code editors: How do they count lines? If edge cases like NELs or form feeds are counted differently in editors vs. interpreters/compilers, debugging could get annoying (this could maybe even be used maliciously somehow?) $\endgroup$ Commented May 18, 2023 at 18:47
  • 3
    $\begingroup$ Voting to leave open, it seems to me that this is a real issue since AFAIK Unicode doesn't have a categorization for vertical separators. $\endgroup$
    – kouta-kun
    Commented May 18, 2023 at 19:39

3 Answers 3

8
$\begingroup$

Unicode fully defines the set of characters that should be regarded as terminating lines in Annex 14, with more in section 5.8 of the standard. These are

  • U+000A Line Feed (LF)
  • U+000D Carriage Return (CR) alone
  • CRLF as one indivisible sequence
  • U+000B Line Tabulation (VT) — supporting this is explicitly optional, and the main standard's newline function definition does not include it (so probably don't)
  • U+000C Form Feed (FF)
  • U+0085 Next Line (NEL), an EBCDIC round-trip compatibility character
  • U+2028 Line Separator (LS), not on your list, nor
  • U+2029 Paragraph Separator (PS)

As a practical matter, however, most of these will never be encountered in source text for a language developed today — they are backwards-compatibility encodings for EBCDIC and physical teleprinters.

The most inclusive approach is to take all of these, but there's unlikely to be any ill effect from taking only the standard (CR)?LF pairing, and editors generally treat the rest as control characters, not line endings.

The only other one you'll plausibly encounter is LS, which is very occasionally used for unambiguous line marking. Even that is generally ignored — VS Code actually warns you when you load the file and offers to remove (not replace) them.

If you're working on an EBCDIC machine then you can take NEL too, but line endings are a more complex question on those platforms and so something more tailored will be better.

Of special note is bare CR. Most code editors will still accept this, but don't create new files in that format, so code in a language you're creating now isn't going to use it. The most common way to end up with a lone carriage return these days is a LF-based editor that has mangled CRLF and inserted something in the middle. Treating it as a line increment only compounds the problem here, as the line number won't match how the file is displayed anywhere. Sniffing for line endings to guess is not constructive for a programming language; if you're targetting classic Mac OS or Oberon, you'll know it already.


My suggested line endings are those that match /\r?\n|\u2028/ at most, or even just /\r?\n/, and to make the presence of any of the other characters in the file an immediate error. If it's to be any broader than that, the only consistent choice is to match the full Unicode list, but for human-edited source code they won't actually appear.

$\endgroup$
5
$\begingroup$

Given that \r and other line endings now only exists as a cultural artifacts, there is only really \r\n and \n to consider. If we now decide to filter \r we end up with a single character that represents going to a new line, namely \n. If you pre-filter all other vertical whitespace, you'll end up with a much simpler line counting (and debug printing).

Whether this is the best method or not depends on language syntax (for example, does that language encourage Unicode characters in the source code?)

$\endgroup$
5
$\begingroup$

Also consider LS (Line Separator, U+2028) and PS (Paragraph Separator, U+2029) supported by JavaScript. These, in combination with the ones you listed, are matched by PCRE's \v (vertical white space character).

QNX used to use RS (Record Separator, U+1E)

That said, \r?\n is the only thing that matters today, so unless you need compatibility with old or strange systems, just go with that.

$\endgroup$
6
  • $\begingroup$ Is this meant to be an answer to the question, or just a comment about some missing items in the question's body? $\endgroup$ Commented May 18, 2023 at 19:19
  • 1
    $\begingroup$ @NikeDattani For now, it answers the first half of "What non-horizontal-whitespace characters should be considered whitespace" which is also the title. $\endgroup$
    – Adám
    Commented May 18, 2023 at 19:23
  • $\begingroup$ Okay, but the answer is not clear to me. Are you saying that everything listed in the question should be supported, as well as the two examples in your first paragraph? What about U+1E? That seems like an incomplete paragraph. Also, what is meant by "For now?" $\endgroup$ Commented May 18, 2023 at 19:27
  • $\begingroup$ @NikeDattani Somtimes the baby on my arm causes me to leave an MVP answer and come back later to explain futher. $\endgroup$
    – Adám
    Commented May 18, 2023 at 23:01
  • $\begingroup$ Congratulations on the baby :) Mine turned 5 months old this week! $\endgroup$ Commented May 19, 2023 at 1:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .