2
$\begingroup$

In template languages, most of the content is written out as-is, with only certain special markers for expression evaluation or flow control. In my attempts to develop a language of this style, I've used regex and enforced that flow control (if, for, etc) must be on a line of their own to simplify detection of structures. Is there a better way to do so? I would expect that Jinja/Freemarker/other languages don't use regex and somehow parse the template. Do they just generate huge tokens for whatever is literal text until a control structure is found?

$\endgroup$

3 Answers 3

6
$\begingroup$

This is a fairly broad category and there certainly are (limited) systems that use regular-expression matching or string-splitting, but yes, of course any serious system uses a real parser. It's necessary to do so if, for example, the closing delimiter might appear within a code block as part of a string literal, because regular expression captures can't handle that.

Whether they "generate huge tokens" is more of an implementation choice; you don't have to have a token phase at all, they can break into smaller tokens arbitrarily, or may be passing through any literal text immediately as it's seen, but most do. Producing a long token is not an issue, and no different to a long string literal. They're not really going to be that long to be a problem, and this all don't present any sort of parsing difficulty.

For example, some production systems:

  • Here is the code of the lexer for the Go template library (as used in Hugo), emitting an itemText token for literal text outside template tags. It absorbs all text up until seeing {{.
  • Here is the same for PHP, producing a T_INLINE_HTML token. It's interwoven with some logic for different delimiters and some PHP filtering features.
  • Jinja calls this TOKEN_DATA as the default state, and deletes empty instances.
  • Freemarker uses a few separate STATIC_TEXT_* token types, including a "false alarm" token when a partial delimiter was seen, and parses all of these using a PCDATA production.

The parsers for each of these handle these tokens as passthrough data, with trivial handling and the bulk of their work going into handling the delimited code. They expect to see interleaved literal tokens and sequences of code tokens, with delimiter tokens between them.

You will find similar arrangements in most comparable template languages too. They do less work than backtracking regular expressions, and parsers are generally trivial systems that don't take up a lot of the developers' effort, so they're not going to be worth avoiding. The task isn't meaningfully different than a typical parser for any other language, despite looking like it's inverted somehow.

$\endgroup$
1
$\begingroup$

One option if your "start code" token is of length n > 1 is to only check every nth character and see if it is part of the start token, if not continue checking.

If the character is part of the start token, then check whether it is actually part of the token. Dump whatever came before verbatim to the output and then swap over to the parser/interpreter for the actual language.

When compiling down a template to code to be executed then everything between code sections is going to get converted to a write equivalent with the bit between code sections as the string to be written to the output.

Making the part between code section one giant token is a valid option when dealing with the tokenized source. Keep in mind that you don't need to make a copy of the text the token refers to when tokenizing, instead you can make the token have an offset and length referring to the part of the file it is from, while keeping the file in memory as you deal with it.

$\endgroup$
1
$\begingroup$

My programmable markup language Papyri is not a templating language, but it has the same property that literal content can be mixed with "code".

Papyri doesn't have "open-code" and "close-code" tags like <% ... %> or <?php ... ?>; this would create too much syntactic noise for how Papyri is meant to be used. Instead, each token is parsed depending on whether "literal content" or "code" (i.e. an expression) is syntactically expected in the current position. So for example the token . may be parsed as text or the "empty" value.

Since the parsing context isn't fed back into the lexer, this means that a sequence of text like foo bar. must be lexed as NAME WHITESPACE NAME DOT. If the parser expects literal content then this would be parsed as a sequence of text elements, whereas if an expression is expected then the token foo would be consumed (and treated as a bare string) and then parsing of the expression is complete, so the remaining tokens (including the whitespace) are consumed by the next rule up (which would typically expect content, but not necessarily).

This makes lexing simpler compared to sometimes treating the source string foo bar. as a single token, which would require feedback from the parser to the lexer. As a bonus, it is straightforward to apply some useful text transformations such as collapsing sequences of whitespace, or replacing -> with a Unicode arrow symbol.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .