Skip to main content

Questions tagged [lexing]

For questions related to the lexing stage of code compilation or interpretation that turns the source code into recognized kinds of tokens. Lexing, which is short for lexical analysis, is also known as scanning or tokenization.

Filter by
Sorted by
Tagged with
17 votes
4 answers
2k views

Why can C not be lexed without resolving identifiers?

I’ve seen it said before that C and C++ cannot be lexed without resolving identifiers to an extent. A common example is (a)*b, which could be either multiplication <...
Bbrk24's user avatar
  • 8,937
17 votes
2 answers
3k views

How do lexers/parsers distinguish between nested generics and bitshifts?

The lexical grammar of Java has a special case for the the > character. Normally, tokens are formed based on the longest-match rule, so that an input string of <...
kaya3's user avatar
  • 19k
9 votes
3 answers
245 views

What vertical whitespace should be supported?

As a followup to PLDI's first question, about horizontal whitespace, what vertical whitespace (newlines) should be supported? I know of at least the following: Line feed (...
Rydwolf Programs's user avatar
6 votes
2 answers
175 views

How to determine the meaning of a polymorphic operator?

Some operators have multiple meanings. For example: - represents the prefix negative, or it can represent the infix ...
Aster's user avatar
  • 3,177
5 votes
1 answer
96 views

How would I allow a variety of different syntaxes for passing arguments into MathJax or LaTeX commands? [closed]

What is LaTeX If you already know what LaTeX is, then skip down the the section labeled "sloppy syntax". There is a type-setting language known as LaTeX. LaTeX even has its very own Stack ...
Samuel Muldoon's user avatar
3 votes
3 answers
207 views

How can I track source spans during parsing and interpretation?

I want to implement error messages that point out where in your source code a parse error or runtime error is. However, when I convert the token stream to my AST, I lose the information about the ...
bigyihsuan's user avatar
  • 1,821
0 votes
1 answer
361 views

How would you implement a language in which the function-name could be separated from function arguments two different ways?

Suppose that there was a programming language in which there were two different syntaxes for function calls. An example of this is shown below: ...
Samuel Muldoon's user avatar