What are some different approaches to raw string syntax, and what are their pros and cons?

Question

Many languages include a "raw string" literal type, intended for arbitrary (or as close as possible to arbitrary) strings of characters (and/or bytes), ignoring escape sequences and the like. This can be handy for things like regular expressions, where using typical string literals would require unreadable and bug-inviting double escaping (e.g., "a\\?\\\\b" for a regex matching a?\b).

I've seen a ton of variations in raw string syntax, with varying capabilities and readability:

• JavaScript: String.raw`"I'm tired of escaping things like '\'"`
• Python: r"I can include \ without worries!"
• Lua: [[I can hold arbitrary strings! "'\[] ]]
• Rust: r#"String includes quotes? No worries, wrap it in potato shreds"#

What are the advantages of these options over one another, or other alternatives?

Answer 1

No end in sight

As used by Zig.

Many (all?) of the raw-string formats adopted by mainstream languages still keep to the form start/end delimiter. This evidently immediately creates the issue that the end delimiter must not appear in the string -- else the end is erroneously inferred -- and thereby limits the ability to copy/paste arbitrary segments of text -- especially segments of code -- in the raw-string. Or should I say, make it painful.

We can compare the virtues of {id|..|id} vs r##".."## vs R"id(..)id" but in the end they all suffer from the same drawback, and neither is that much more readable than the other. They're also hell to tokenize.

Yet, at the same time, almost all mainstream languages already have syntax support for an implicit end delimiter: comment-until-end-of-line.

As such, after many years of mulling this over and over, I would argue that a syntax with "no end marker" should be considered:

let paragraph = #"In Rust, a raw-string is delimited by `r"..."`,
    #"and a matching number of # signs can be added before the opening
    #"quote and after the closing one, in case quotes appears in the
    #"string.
    #"
    ;

The raw string, here, is composed of everything following the #" sequence, including newlines, with the exception of the last newline.

Advantages:

• Multi-lines support.
• Nested indentation support, without meaningful indentation necessary.
• Consistency: always the same starting delimiter, no arbitrary ID/number of pound signs/... to avoid the token sequence in the string itself.
• Ease of copy/paste: paste, indent all lines, prefix all lines. Even in a simple text editor, as long as vertical cursors are supported it's a simple operation, and IDEs support block comments already, which are very similar.
• Ease of tokenizing (1): No substring search necessary => SIMD tokenizers possible.
• Ease of tokenizing (2): Parallel tokenization of files, chunked arbitrarily, is possible.

Disadvantage:

• Vertical Space.

Illustration of Vertical Space issue:

//  Rust
string.push_str(r#""Hello, world""#);

//  Hypothetical
string.push_str(#""Hello, world"
    );

The matter, obviously, compounds if pushing many fragments of raw strings.

Choices:

• Interpolation vs Copy/Paste: Interpolation requires treating certain characters specially, so a choice between Interpolation vs Verbatim text must be made. Or no choice, and having two start sequences depending on the case. It's an orthogonal issue.

Answer 2

Repeat Quote

Repeat quote three times or more as STRING_QUOTE_TOKEN until the same token is encountered.

Should report a parse error if not found.

Similar to ` in markdown

'''raw string'''

""""
multiline * 4
raw string
""""

The disadvantage is that it is a bit long, at least 3 characters.

The advantage is that you don't need to consider any escaping, you can even dynamically interprete escape.

For simplicity, you can also specify that 'raw' is the raw string '''raw'literal''' is used to deal with the occurrence of ' in the raw string.

The fatal problem is what to do if your first character is ', at this point you have to choose another character like " to form.

---

The dilemma with escaping is that the more complex your escape, the more complex the unescaped form used to express that character.

If you have written tmLanguage, you may have encountered the situation where you have to write \\\\\\\\.

Therefore, the best way is not to escape, and directly capture the content between the two tokens as a raw string.

Answer 3

Go style: backticks only

Similar to Javascript, Go uses a pair of backticks for raw strings:

"regular"
`raw`

"regular\n    indented\newlines"
`raw
    indented
newlines`

Which is exceptionally useful for regexes, as you don't need to escape anything, except for the usual escaping of special regex characters:

regexp.MustCompile(`(\w)(\w+)(\w)`)

Pros

• Extremely simple, no new parsing required (just an additional token kind)
• Can be used anywhere a regular string literal is used
• You cannot escape anything in these, so \n is the literal characters \ and n.

Cons

• You cannot escape anything in these. This leads to backticks being the only disallowed character because it would end the literal, and you can't escape it because you can't escape a raw string. This can be worked around by using a regular string literal:

query := `SELECT * FROM` + "`table`"

• You need to keep track of indention level relative to the start of the raw string for indented parts to have the correct indentation.

Answer 4

OCaml

OCaml supports raw string string literals in addition to classic double-quoted strings with escapes:

• The OCaml Manual has the details.
• A raw string is enclosed by {id| and |id} where id is an optional character sequence that must match at the opening and closing delimiter. Most code uses simply {|...|} for raw strings (which now can't contain |}).
• The id can be used by the macro (PPX) system to give special meaning to particular strings. You could imagine using {sql|...|sql} for example.

Answer 5

I see no mention of Zig so far, so I think it's worth mentioning it as another point in the space.

Zig has regular inline strings with escape sequences like most languages, and multiline strings which have no escapes and require each line to be prefixed with \\.

const hello_world_in_c =
    \\#include <stdio.h>
    \\
    \\int main(int argc, char **argv) {
    \\    printf("hello world\n");
    \\    return 0;
    \\}
;

Although this means that you cannot paste in a block of text directly as a "raw string" without prefixing the lines, it has some advantages:

• The grammar remains context-free, unlike Swift, Rust, and others which require matching but arbitrary open/close pairs.
• It's impossible to forget to close the string, so in theory this means less catastrophic parse/highlighting failures in an IDE with incremental parsing of broken code.
• As Alex Kladov notes, it's possible to lex the file looking at chunks in parallel or incrementally, since the tokens on one line can never affect the tokens on another line.

Answer 6

Python style

Pros: Low friction syntax-wise, close enough to existing string syntax
Cons: being closed by a regular quotechar still requires you to include a \ for including a quote value

JavaScript style

Probably the most confusing of all the syntaxes, a very special case in parsing, requires you to recognize String.raw`str` as a function call.

Lua style

Possible confusion with array syntax in other languages (although in lua's case Arrays are defined by {} so not in this particular case)

Rust style

Maybe the most robust of all syntaxes, it's far enough from other rust syntaxes to be confusing or to be accidentally closed by another value, can't think of a downside

C++

I'd like to add another case of raw strings: C++. It allows the user to define their own syntax for their specific use case, for example

auto rawstr = R"rawstring(
text
text
text
)rawstring";

auto rawstr2 = R"aaaaa(
text
text
text
aaaaa)";

are both raw strings, and the user can choose a termination value that does not conflict with their expected values and for which the start/end of the string is clearly readable.

Answer 7

Swift: Raw strings with interpolation

Swift, like Rust, uses hash signs for raw strings. However, you can also interpolate into raw strings by adding the hashes to the interpolation marker:

"Regular string interpolation \(x)"
#"Raw string with "quotes" in it"#
#"Raw string with interpolation \#(x)"#
##"Not an interpolation \#(x)"##
##"But this is \##(x)"##

You can have arbitrarily many hash signs in the literal, but the opening and closing quote marks must match.

Answer 8

Some languages that don't have a explicit syntax still have ways of creating raw strings in other ways.

Depending on your language it may not be worth supporting raw strings if you already have something like this:

<?php
ob_start();
?>your string that may contains " quotes and \ backslashes<?php
$string = ob_get_clean();

This technique is commonly used when needing to embed other languages like CSS and Javascript in PHP.

If you have a templating language that already has a builtin for capturing output raw strings may be unnecessary.

Answer 9

Recognizing the delimiter

The examples given in the question obscure an important mechanical detail relevant to Python's implementation (and possibly others). A Python raw string literal uses the same lexing/tokenization rule as other string literals: a string consists of sequences of (anything except the closing quote) or (a backslash followed by a single character). It only has different rules for interpreting that token (everything is treated literally, including backslashes).

This has two important consequences:

1. A Python raw string literal can including the closing quote character, but only immediately after a backslash (which the resulting string will contain literally).

2. A Python raw string literal cannot end with an odd number of backslash characters.

An alternative rule (apparently used in Go, as seen in bigyihsuan's answer) is to just lex the string a character at a time, looking for the closing quote mark. This has the consequence that the literal can't include the closing quote at all, but then backslashes behave completely normally and are easy to understand.

(The above description deliberately glosses over the complication introduced by Python's "triple-quoted" string literals.)

Making raw strings primary

Following the Pythonic principle that "explicit is better than implicit", one could imagine a language in which "raw" strings are the default while strings that use backslash escapes require some special markup (or even an explicit call to an unescaping function, which might be applied at compile-time). This is the approach I am taking in my own design.

With this approach, the standard unescaping scheme would still not be able to use \' or \" escapes (since they could not be present to begin with); but this is not strictly necessary anyway. There are already other commonly understood ways to escape these characters, e.g. with hex codes (\x22 for a double quote and \x27 for a single quote); or one can invent new ones (perhaps \q for a single quote and \Q for a double quote).

Making the unescaping process explicit also offers the opportunity to specify error handling (should a literal like '\c', where there is no such escape sequence, produce a string that actually has a backslash in it? Produce a string with only the c? Raise an exception / cause a compilation error? Something else?).

Adding quotes

There are several possible workarounds for the inability to include the closing quote mark in such a raw string literal. Being able to choose between ' and " delimiters is already helpful, of course. If a string needs to contain both, this can be handled by concatenation with another literal, which could be explicit (using whatever string concatenation operator) or implicit (like "a string with both '" ' and " in it'). Another option is special rules for implicit concatenation - for example, it could be understood that implicit concatenation of two raw literals using the same quote type, like 'I' 'll', implicitly adds an intervening quote of that type (to make I'll). This is, in effect, functionally equivalent to "escaping" that quote character by doubling it up (as seen in some CSV dialects).

In my design, raw literals with the same quote type are implicitly concatenated with that quote when they appear on the same line of the source file, and with a newline when they appear on consecutive lines. The latter makes it easy to write multi-line raw-literal text without needing to worry about leading indentation.