Skip to main content
16 votes
Accepted

Why can C not be lexed without resolving identifiers?

The short answer is yes, you could do it all in the parser (lexers are optional), but historically it's been easier the other way. It's mostly a historical accident that this all exists at all. This ...
Michael Homer's user avatar
  • 13k
8 votes

Why can C not be lexed without resolving identifiers?

This is my best guess because I haven't looked deeply into this issue. If an expert has a better answer, go with that. You are correct that the lexer could lex things like that and not need context. ...
Gavin D. Howard's user avatar
8 votes

What vertical whitespace should be supported?

Unicode fully defines the set of characters that should be regarded as terminating lines in Annex 14, with more in section 5.8 of the standard. These are U+000A Line Feed (LF) U+000D Carriage Return (...
Michael Homer's user avatar
  • 13k
8 votes

How do lexers/parsers distinguish between nested generics and bitshifts?

There is actually no reason to assume that a file is tokenized until the end before parsing begins. There are frameworks allowing you to generate parsers and tokenizers that use tokens streams that ...
feldentm's user avatar
  • 1,779
6 votes

How to determine the meaning of a polymorphic operator?

Typically, this is not something that the lexer is concerned with in systems with a traditional lexer-parser sequence. Instead, it produces a token representing the symbol that appeared in the source, ...
Michael Homer's user avatar
  • 13k
5 votes

What vertical whitespace should be supported?

Given that \r and other line endings now only exists as a cultural artifacts, there is only really \r\n and ...
Nuoji's user avatar
  • 1,105
5 votes

What vertical whitespace should be supported?

Also consider LS (Line Separator, U+2028) and PS (Paragraph Separator, U+2029) supported by JavaScript. These, in combination with the ones you listed, are matched by PCRE's ...
Adám's user avatar
  • 3,156
4 votes

How would you implement a language in which the function-name could be separated from function arguments two different ways?

What you’re really facing here is a parsing issue, rather than a tokenising one. You would tokenise this language the same way that you tokenise any other language — give each of those distinct ...
Michael Homer's user avatar
  • 13k
4 votes

How can I track source spans during parsing and interpretation?

My usual strategy for this is that every parse tree node holds on to the first token of the parse element from which it was constructed, and sometimes the last too. The initial token is definitely ...
Michael Homer's user avatar
  • 13k
3 votes

Why can C not be lexed without resolving identifiers?

It all depends on what your parser needs from your lexer. If all it wants is a stream of language tokens, you're right: the lexer doesn't need to know that stuff. If you want it to identify the types ...
RLWatkins's user avatar
2 votes

Why can C not be lexed without resolving identifiers?

Here's one way of looking at it. Lexer and parser generators normally take as input a list of rules, and each rule is associated with a chunk of arbitrary code that runs when the rule matches. The ...
benrg's user avatar
  • 479
2 votes

How can I track source spans during parsing and interpretation?

Have you considered ditching lexing altogether? It makes things significantly easier. Using a lexerless parsing, e.g., PEG, you can simply embed location information into every AST node you're ...
SK-logic's user avatar
  • 1,310
1 vote

How can I track source spans during parsing and interpretation?

Source locations are typically represented to the user as a line and column number, but they don't need to be stored internally that way; it's faster to represent a source location as a simple index ...
kaya3's user avatar
  • 19.7k
1 vote

How would I allow a variety of different syntaxes for passing arguments into MathJax or LaTeX commands?

I would parse the LearnTeX syntax using a parsing library, such as Rust's combine or Haskell's parsec, or a parser generator, ...
user570286's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible