Questions tagged [tokens]

For questions relating to tokens, or the data structure that contains information about the raw components of source code

Filter by
Sorted by
Tagged with
0 votes
1 answer

How would you implement a language in which the function-name could be separated from function arguments two different ways?

Suppose that there was a programming language in which there were two different syntaxes for function calls. An example of this is shown below: ...
Samuel Muldoon's user avatar
10 votes
4 answers

Do parsers typically need access to all tokens?

Do parsers typically operate on the entire array/list of tokens available in memory, or are the tokens often streamed one by one as they are recognized? What influences the decision?
Lazar Ljubenović's user avatar
4 votes
0 answers

What would be the steps to implementing a formatter for your programming language? [duplicate]

Right now, I am using ClangFormat to format code written in my programming language, but that's not ideal, among other things, because ClangFormat treats := (...
FlatAssembler's user avatar
25 votes
6 answers

Negative literals, or unary negated positive literals?

Are there benefits to number literal tokens including negative numbers? Or is the string -1234 being parsed as unary negation of positive number literal ...
bigyihsuan's user avatar
  • 1,663
7 votes
7 answers

Pros and cons of whitespace or comments being included as tokens

What are the pros and cons of including whitespace and comments as tokens in the tokenizer output? For example: value= 6+ 7 // A comment Would result in these ...
MarcellPerger's user avatar
23 votes
5 answers

What options in Unicode are there for balanced pairs of brackets like `(){}[]<>`?

My esolang involves the use of balanced brackets (){}[]<> to denote different scopes. However, unlike practical languages, all pairs of identical brackets ...
bigyihsuan's user avatar
  • 1,663
20 votes
3 answers

Which horizontal whitespace should be supported?

There seems to be three approaches to horizontal whitespace for separating tokens: Only tab and space: Bash, C, D, Dart, Go, Java, Lua, OCaml, Python, PHP, Perl, Ruby, Rust, Scala, Swift (also ...
Adám's user avatar
  • 3,118