Questions tagged [lexing]

For questions related to the lexing stage of code compilation or interpretation that turns the source code into recognized kinds of tokens. Lexing, which is short for lexical analysis, is also known as scanning or tokenization.

Filter by
Sorted by
Tagged with
17 votes
4 answers

Why can C not be lexed without resolving identifiers?

I’ve seen it said before that C and C++ cannot be lexed without resolving identifiers to an extent. A common example is (a)*b, which could be either multiplication <...
Bbrk24's user avatar
  • 8,907
17 votes
2 answers

How do lexers/parsers distinguish between nested generics and bitshifts?

The lexical grammar of Java has a special case for the the > character. Normally, tokens are formed based on the longest-match rule, so that an input string of <...
kaya3's user avatar
  • 18.4k