Questions tagged [tokens]

For questions relating to tokens, or the data structure that contains information about the raw components of source code

Filter by
Sorted by
Tagged with
0 votes
1 answer
344 views

How would you implement a language in which the function-name could be separated from function arguments two different ways?

Suppose that there was a programming language in which there were two different syntaxes for function calls. An example of this is shown below: ...
Samuel Muldoon's user avatar
10 votes
4 answers
1k views

Do parsers typically need access to all tokens?

Do parsers typically operate on the entire array/list of tokens available in memory, or are the tokens often streamed one by one as they are recognized? What influences the decision?
Lazar Ljubenović's user avatar
4 votes
0 answers
79 views

What would be the steps to implementing a formatter for your programming language? [duplicate]

Right now, I am using ClangFormat to format code written in my programming language, but that's not ideal, among other things, because ClangFormat treats := (...
FlatAssembler's user avatar
25 votes
6 answers
3k views

Negative literals, or unary negated positive literals?

Are there benefits to number literal tokens including negative numbers? Or is the string -1234 being parsed as unary negation of positive number literal ...
bigyihsuan's user avatar
  • 1,663
7 votes
7 answers
1k views

Pros and cons of whitespace or comments being included as tokens

What are the pros and cons of including whitespace and comments as tokens in the tokenizer output? For example: value= 6+ 7 // A comment Would result in these ...
MarcellPerger's user avatar
23 votes
5 answers
811 views

What options in Unicode are there for balanced pairs of brackets like `(){}[]<>`?

My esolang involves the use of balanced brackets (){}[]<> to denote different scopes. However, unlike practical languages, all pairs of identical brackets ...
bigyihsuan's user avatar
  • 1,663
20 votes
3 answers
532 views

Which horizontal whitespace should be supported?

There seems to be three approaches to horizontal whitespace for separating tokens: Only tab and space: Bash, C, D, Dart, Go, Java, Lua, OCaml, Python, PHP, Perl, Ruby, Rust, Scala, Swift (also ...
Adám's user avatar
  • 3,118